The first time I tried the build-off puzzle I put the villager on the building supplies before using the time warp. When I used it on him while he was in the build-off he jetted off towards the huts to repair them. I grabbed him and took him back to continue but he just stood there and shook his head. When the time warp wore off and I had enough energy I took the builder back to try it again. Before placing the builder on the supplies to start I used the time warp on him then placed him on the supplies ad it worked like a charm. I guess thats what you have to do use the time warp on the builder then put him/her on the supplies.