If you're increasing your population too much, there won't be enough food to support them. Until you get sustainable fishing, you really can't support more than 15-20 villagers.