The faster your farmers clear away the crops the longer the wait will be until the next crop. You might consider switching one or more of your farmers to other work, especially during the fallow period, unless farming is the only skill they need to gain Elderhood.
Thanks for the tip Arnie, I usually have 3 or 4 people clearing crops because of the large time gap I have no food... but this is a good idea I'll try it