If they're enjoying the garden, I presume the game will behave as if the garden exists - it's likely only a visual glitch.