Have you already completed hut #3? If not, your game is not a lost cause by a long shot! Builders (like other villagers) tend to want to goof off alot, so you must be persistent. When you get a message that reads "There's not enough housing!" when you try and breed villagers, you can start on hut #3. I like to pick a younger villager and stick with it until they become at least a trainee builder.

Don't give up! There's still lots of stuff for you to build.