I did find that they wanted to clear rubble so I moved them all over there and am going to make them finish clearing it before we go anywhere near the hut again. I am trying to get my people masters in at least 3 areas so I am really wanting them to work on the building. I had some die out on me in VV2 and took forever to get them back to Masters so I make sure I have them Masters in more than one area!
I also found that they DO want to clear the rubble. I found this on my first tribe, so on all the other tribes that I have started since, I wouldn't let them touch the rubble until the second hut was finished being built. I also don't let them touch any of the other projects until it is finished. Make sure that you have your Chief directing work on this hut. That helps too.