The biggest difference between the difficulty settings for both VV1 and VV2 is with the resources that you start the game with and how quickly those resources are depleted. On the easy setting you will start the game with more food (and more villagers in VV1, as well).
One thing to be aware of, though, is that changing the difficulty after a game has been started does not adjust those resources.
This post in the Technical Issues forum discusses how to truly start a new game at a different difficulty setting.