I'm guessing then that it's my fault. What I saw in the beginning is; when a fruit tree was fully harvested, it would go back to being small and say the 3 required hours to harvest.
This is absolutely not normal; trees, once grown, never un-grow themselves; they just regenerate a little fruit over time.
This is most likely due to your manipulation of the system clock -- as CFII says, it's very easy for even a single farmer to strip trees and honey so thoroughly that you'll never see them at levels higher than 7 or 8 fruit. (At which point my villagers seem to stop harvesting fruit.)