It must have been a glitch. I went back on the game to check and I'd lost 3 more villagers (from old age). It's now letting me breed them back up to my limit, currently at 97 and he was still going when I last checked.