"masters" of a single craft? I was needing more farmers than builders (built everything already) and now I need a master builder. I tried taking one who was a master farmer, but he seems stuck at a high level of building mastery, without getting the title "master builder".

Do I need to start with someone who hasn't gotten master of anything yet?

All villagers can be masters of several or all skills. But remember, the skill meter goes up at a slower pace as in increases. So just keep him on that skill and he will become a master at it after awhile.