My builders were somehow reluctant to stay and work on the ancient floor. They did a couple of turns and then went away and did something else, like fixing huts or admiring something, before returning. It's similar to the hidden gold chest in VV1 - they were equally slow and unwilling in that project. The other projects were much more interesting, as it seemed. The game speed didn't matter; the behaviour was the same.