I've managed with ten villagers before, forgetting to keep an eye on the woman with parenting skill.

The difference, perhaps, is that I don't have the inclination to mushroom-hunt; I leave them to their own devices, generally.