How many villagers do you have? If it's more than a dozen or so, they can't survive on just the farm and berry bush. You can't kill your villagers, though.