I have noticed that when a villager is skilled in multiple areas they tend to focus on the area they have the most skill in, no matter what you set their focus to. If he has more knowledge in breeding than building, he'll probably build for a little while then go back to breeding whether you like or not.

Good thing for the ladies though!