I had it happen 3 times with the 3rd hut. All the villagers started dancing around the first hut completed. Still let me finish the 3rd hut though so I didn't worry to much about it.