Do you have any younger female children? If so, you can wait until they get to 18 and then start breeding as soon as you can. Guys can father kids at any age. While you're waiting, you can build up the villagers skills as much as possible. It's actually a good thing to keep the number of villagers low if you just started the game.