They won't build any more huts after that. Just put them to work doing something else and they will stop their "worrying."