The crops are mostly only able to feed a village of about 20.As you have over 100,that food source dies out pretty quickly.And,as LadyCFII put,it is possible you accidentally have too few villagers on farming.

I had this Event yester day flash flood, so only 750 food, so I thought not bad soon get more food, how wrong I was, after I seen to my other v games, I came back to my village and the food was on 0 I had to get what builders I had on fishing and the food still kept been eaten, So I made a stew to fully restore all the villagers health line hoping the food would not disapear so fast. Then I discovered why I had least on farming, I saw a line of at least 29 villagers walking back and forth on the beach where they dance all healing some one. I must of got so many doctors I put all the doctors on fishing except for 3 trainee doctors. I was over 2 hours doing all this.

I got another flashflood event top layer edible then click the ok food went straight to 0. I wondered why the food is not building up as quick. the reason is I counted 59 kids in detail screen and 3 mothers nursing babies only leaving 50 abled villagers which 6 of them are elderly and 3 of who are trips don't like fish and two don't like swimming 1 dislikes work.

thats really strange that the food is very slow to go up

I notice the little guy just sitting there in the village center, Not Helping! When zero food happens in my villages, I have lots of farmers, but most insist on watering the empty garden, instead of fishing. I have to keep re-directing them. And so many of them wander around aimlessly, "worried about food". Way to use your heads, people!

Whenever I saw "worried about food" ... I said "now you can stop worrying", picked them up and plopped them in the ocean to fish, no matter WHAT vocation they were in ... until they got it and started fishing ... if things get THAT tight, everyone but the children can learn with a little patience how to fish.