It was my experience that the only way to keep the villagers working was to leave my game open. Otherwise they just wandered off.

But with leaving it open, and with the maximum number of builders working on it, I got it done over one night.