That has happened to me in my VV2 tribe. My only woman was over 50, and my kids were boys. I did get the barrel 'o babies event and that brought my tribe back.

Like HPVV said, don't give up on your tribe. They may surprise you.