I'm pretty sure 2 huts will only allow 30 villagers.
Try assigning a few villagers to building and that should solve the problem. =]