The maximum population depends upon the device you have (some devices can handle more villagers than others), but 3 huts is the limit (and they show up on the puzzle screen)!

You'll know you've reached the population limit when you can't breed any more because there's not enough housing (and you have built all of the huts).