What speed are you running the game at? What are the levels skill of your builders? What is your food situation? If you have plenty of food they are more likely to wander off and do other things. Similarly, if you have too little food they will start 'worrying about food' instread of working steadily.