I have asked myself the same thing.
I suggest each villager should have a family tree, so that you could see the parents and eventually grandparents on both sides.
At least there should be some kind of gamelog to check upon each time you return to the game.