I don't think that really saves any food by cutting down on the number harvesting, it just cuts down on the wait time. Having fewer farmers also helps spread out the food stocks, make them more easily manageable. Instead of your food jumping to quite a high figure then dwindling to almost nothing while there are no crops, you can keep the food level almost steady. It also helps get master farmers more quickly. Instead of many, only one or two are doing the work, so they will learn faster.