I've found it best to only set the builders one task at a time. Otherwise, as you've found out, they'll wander between tasks, and as Murphy's Law says the one you want them to finish first will be the one they spend the least time on. At least the two tasks you mention are close together so the builders don't spend a lot of valuable time wandering from job to job, as would be the case (say) for the rubble clearance and hut building.