If you havent gotten all your huts built yet you will not be able to have more then the limit, I do believe if memory serves right the first hut allows you 25 and then they will start building the second hut, but only if you have a builder, and at 50 they will start the third hut, If im wrong somone please say, but im pretty sure thats the ammounts. Once you have all three built the limit is 100 if youve reached the limit, not much you can do till somone dies, which sucks i know, but its the way it is, if you havent gotten 100 and you have all the huts, just keep trying, villagers can be stubborn.