The best way to do that is to choose a villager at random, then scroll through the details screen one villager by one until you find one you want. Return to the main screen and they'll be selected. If you want three builders together, pause the game, then scroll through until all three have been found, dropping them together. Then unpause. That way they don't wander off while you're looking for the next one.