It may be because game speed usually doesn't slow down the villagers' movements on the screen, only the rate at which tasks are completed, and the calculations done when the game is re-opened to figure out where your villagers are.

I would assume this is the root of the problem, that aging doesn't stop, on pause, until you close out. Lady CFII would be able to give better feedback here.