They don't come back, as far as I know. This is just one of the ways the game randomly adds or removes villagers to and from your tribe.