You don't say what speed you are running. That not only makes a difference with how long the crop cycle is, but how much is collected (fish, coconuts or crops) each time.

I find that playing on "normal" speed is about right for keeping the early village well fed.

I'll also only put 2-3 villagers on the farm when the crops come in. They will keep busy, and not have too much time left in the "growing" season. As you discovered, if you harvest everything too soon, then it'll be a long wait to the next cycle.

Except for the dam and the briar thorns (so that more collectibles will appear) I also don't worry about building huts early in the game. Better to keep the villagers at the research table to earn the points.