your right, but I don't recommend letting your food stores deplete...I just walk them through harvest when ever I'm around and keep the saved food at around 15K.
You can maximize harvest by making sure everyone returns to the patch after they deliver to the food bin.
This method got me through the beginning when I overbred and couldn't meet the food needs of the village.