Even at 2X Speed, it does seem like a VERY long time especially when you're desperate to feed your villagers.

Make sure your builders are "on the job"; I tend to watch the trainee builders because they will eventually wander off and do their own thing until they become adept builders.