You did it right, SP. The number of villagers doesn't change with the difficulty setting in VV2. It's VV1 (and Village Sim for Palm) that has 5 villagers on the hard setting.