Yeah, since I don't pause the game when I'm away (unless there's something I particularly want to "oversee"), even if nobody is clicked off as a farmer, when I come back the hive and the fruit trees are usually either empty or close to it.

They sneak out when you're sleeping and pick fruit. How strange is that!