Yep. I usually start at a slow speed (normal speed in my case) to minimise the villagers' food consumption and allow my only farmer to build up a decent stock of food. When I've around 500 I'll switch the farmer to research or building the dam. Food levels can often be nearly maintained at that level with help from the mushrooms. Once the dam is finished I'll switch to a faster speed (usually 2X) to speed the appearance of the crops. Often I can leave the original farmer on research for a while longer, but after a while farming will again become a necessity.

Only after the dam has been completed will I even think about having a first baby.