I don't have a problem with the garden. Try dropping your villagers just below the garden so they're still interested but not so close to the apple tree.