At the beginning the builders are not very important. You must set your villagers at research and try to gain 6000 tech points to buy the level of farming and you'll have some crops to harvest. If they don't want to do it, bring them to the well for drinking or to the bin for eating and back to the research table. (They must be set in the "detail" screen as "scientist") And, by waiting, take care of the mushrooms that your children may pick up. Don't let them do every thing they want, especially "embrace" or "going in door", you don't need to increase the population!! And forget the rocks for this moment, it's too early.
PS: excuse-me if my English is not very correct, in spite of, I hope it will help you