Ooooh... that could do it. Especially at the beginning of the game, you need to carefully assign your villagers to make sure that everyone is contributing to survival, both short-term and long-term. For example, if your only farmer gets sick, and you don't have a healer, they probably won't survive long without your intervention (especially on fast speed).

Yes, you can have your farmers doing other things when there are no crops to harvest. They won't gain as much farming skill that way, but they will help get to the next tech level. It's a trade-off, but it can be a useful strategy.

You should check out this post for some helpful tips on useful strategies. There aren't any spoilers in there, so it's safe to look at.