If you have finished all the building why do you need more builders? I get mine by ticking the building box and sitting with the circle on them, throwing them back on every time they wander until they are trainees - a hassle in time and effort but it works for me.

I did wonder if the huts collapsed/deteriorated or looked unkempt if they weren't 'fixed', but have never seen any sign of this. So unless I need a jack-of-all-trades late in the game I don't bother with them once all the buildings are complete.

Amaranth [hopefully never fading]