I think the other difference in difficulty levels is the time it takes for crops to grow and stuff????? I'm not 100% sure on that but I'm pretty sure I've heard that mentioned in other threads.

The difference in the time for the crops to grow (and mothers caring fro their children) is a function of the speed setting, not difficulty.

The Difficulty setting determines how many villagers you start with, plus, I believe, how easy it is for a villager to learn a new task.