It's likely that the females all had some breeding skill (a villager will often do anything that they know how to do, whether you select that skill or not). When a female chooses to breed, they will seek out the alpha male of the tribe, which is why all the children have the same father.
I just had an example of "Popular Guy". I had set several women to parent, to increase population (don't know how Igot so many blonds) When I came back to the village, Coconut Head had become a "papa of many". Coconut Head is no longer in the village. But interesting that all the kids had the same color hair.