Jami, if the food in the bin is low and there is anything to harvest, any villager with even a bit of training in farming will go and get it, while the others will worry.

Once you get more food in the bin, they tend to stay with their preferred task.