The problem that you're having with food is caused by the fact that the farm won't comfortably support the population that you currently have. I don't normally have more than a dozen or so villagers until I've gotten the ability to fish.

I've explained how game speed works in this post.