What speed are you running the game at? When I start work on the statue I run the game at Fast speed, and usually see around 20-10 hours, depending on how many builders I've put on the job. In fact, once the job is underway I'll close the game and it tends to take far less time to complete than the timer indicates. I generally use more builders than just six, though.
If you have no more building projects you can train new builders by getting them to 'fix huts', but that takes time and patience. Another way is to feed them an 'exercise' potion, but that is also hit and miss.
Too late now, I know, but I always train builders on the clearing of the leaves and interrupt them before they deposit their load of leaves so the progress remains at 0%, and the ROD won't be completed. That gives me an easy way to train builders as needed.
_________________________
To err is human; to arr is pirate.