Have you got any other puzzles left to solve that will take building? If so, I wouldn't worry too much about them fixing the huts. Even one more puzzle to be constructed will give you a master in no time, as long as you don't put a heap of people on the task.
If you've completed every building task, then make only one person a builder, and let him/her do all the fixing.