I had my villagers fishing but they would still occasionally mosey on back to the orchard and pick fruit off the poor trees. I did a few rain dances and changed ALL of my farmers to builders. So now, slowly but surely, the fruit is coming back!

Thank you!