Do you have enough food for them to eat? It sounds like they're starving. There are some strategies for getting started successfully in the Virtual Villagers Strategy Guide (for Windows or Mac) or the Village Sim Guides (for the Palm version).