I have lost 2 VV1 tribes and 1 VV2 tribe to this phenomenon. All I can suggest is to keep trying, and hope for the best. As in real life, some of the villager women seem prone to having boys. I would suggest mating different men with your women to see if that helps.