Be sure they are checked as builders and look at their likes/dislikes. Maybe they likes laundry!!
And also, it's possible that there is no hurry, so if you make more babies, they'll build one hut more easily!