I understand that if I disrupt a farmer when he/she's got some produce on the way to the food bin it will be lost.

What if I stop my builder from going to n fro from the rocks and wood, will it disrupt the building process?