I've had that problem too in the beginning of the game but my solution was to make sure that the "Trainee Farmers" have had at least 1/3 of their skill bar increase before I let them do their own thing.

I also kept my population really low in the beginning and then only had 2 Farmers (Maximum) harvesting crops so that by the time the crops dwindled it will be time for the next crop to come. I hope this helps....