Have you maxed out the population based on the number of huts you have? You will not be able to increase the population if you don't have enough housing. For example, to go beyond 7 villagers, you need to have built the first hut, to go beyond 15, you need to ahve built the second hut, etc.