The crops are mostly only able to feed a village of about 20.As you have over 100,that food source dies out pretty quickly.And,as LadyCFII put,it is possible you accidentally have too few villagers on farming.