It sounds like your villagers are still unable to fish, which could be a huge problem with a population that large. The initial food supplies in all VV games are limited and won't support a very large population. In VV3, if you grow the population beyond about 10-12 villagers before you have an unlimited food supply, you will need to hunt mushrooms a lot to prevent starvation.