onica, I can't really answer that without more info.

What's your population when they die, what tech levels do you have and what's your food situation at?

Just a guess without the above info, but is it possible (since it takes a while to get tech levels) that you are doing no breeding at all and your villagers are dying of old age?

It's true that you need all your farming before you can grow a large population in this game but the villagers do age so make sure you have at least a few little rugrats running around.