The short answer is NO! Your villagers will continually go and clear the stream; they will do it one or two times and then walk away to another project. Oh, and look at your trophies, you will find one for clearing the stream of 1000 pieces of debris.
Clearing the stream is a good thing, it helps to provide work for your builders.
And, remember later after all the building projects are done, you will want to produce master builders to help in getting the coveted "Scholar" titles.
_________________________
Margi