I don't usually have more than about ten or so villagers until I have level 3 of harvesting, because food becomes a problem - children eat as much as adults do. You've started to push the limits of how many villagers you can feed from the farm, so food will probably be a little tight at this point. The best thing to do is to keep as many villagers as possible at the research table until you can purchase level 3 of harvesting, and don't make any more baby villagers yet. Once your villagers can fish, food won't be a problem.