I understand that if I disrupt a farmer when he/she's got some produce on the way to the food bin it will be lost.

What if I stop my builder from going to n fro from the rocks and wood, will it disrupt the building process?

The produce is only "lost" in that he doesn't get it to the bin, but it's not lost from your total crops.. but the process of going for the rocks and wood is part of the building cycle. The builder will not "increase his skill" until he's brought back the wood/rock and put it into place successfully

Thank you....