It seems that the tribes (when puzzles and tech are completed) have a growth going up to the max and down to 20 or 30 several times in their life! It's what I have noticed with mine.
Did you select some villagers (between 18 and 50 years old for female) on parenting skill?