i only had one adept builder and little food also and there was no problem with him staying on the job or finishing the task quite quickly.
i guess your suggestions are helpful - just wanted to point out, there might be some other problem as well.

do any of your builder dislike work or like resting? this could also hint to some possible problems