well, honestly, it doesn't really matter who you breed with. I know there are a few users that have tried to keep track of lineage, and just gave up when babies were born overnight. No way to know then!
I've been told that some Villagers will run from others- perhaps this is a in-breeding defense mechanism?

I just wouldn't worry about it... have fun!