Are your farmers all adept? They do take 'breaks' but they should stick with farming pretty well. It always seems worse when there's a looming disaster. I usually just turn into a slave driver when that happens, and make sure that their action shows that they are harvesting before I go on to the next villager!