Both times it happened to me the village was 'mature', near to or on the maximum population. The villagers soon made up for the missing members. There must be some way of the program making sure that it doesn't happen early on; imagine if it occurred right at the start of the game.

I suppose in some games it could be considered a good thing; rather than waiting for the more elderly villagers to die off, there is a sudden injection of youth to make sure the tribe will continue.