I like to get as many villagers to esteemed elder as I can, so I do have builders even though all of my structures are built and the puzzles are finished. BUT it is very frustrating because they will not do any building task unless you are constantly overseeing the game and putting them on a hut to fix it, They go get a drink, take baths, look at fish, wash laundry ... everything but fix huts on their own. Having more than 3 or 4 working on building at a time is impossible, you have to constantly track down the ones who've wandered off and get them back on task.