The crops are on a cycle, and always grow at the same rate. The number of minutes it takes for them to grow depends on how quickly you harvested them the last time.

The time also adjusts if you change the time setting on the options screen from faster to slower or the other way around.

Your tribe doesn't require a large amount of food while your population is low. You can really get by with the minimum of 300-500 pieces for a small tribe under 10. I would just rotate a single farmer on the crops, as you need one.

I hope that helps!