I have had this problem also, throughout the game. Currently I have a population of 22, of whom 15 are farming (I took it down to the wire to get enough tech for Farming 3 ). Coconuts are more or less gone, but as soon as they get up to 20 or 30, everyone rushes over there to pick them, despite the fact that I have cleared the algae (thus leaving them a pristine fishing spot) and of course the irrigated cropland.

The cropland, though, is the real problem. As a couple people said above, it seems like all the farmers want to do is water the crops. Now again, I have 15 villagers farming. When the crops were growing, I had to keep babysitting them, dragging them back to the ocean -- at one point I'd stepped away and returned to find 8 of the 15 watering the crops! Now, the crops have come in, and suddenly nobody wanted anything to do with the farming patch. I had four people there, when they first came in, which has rapidly dwindled to one, and the rest of my farmers have suddenly decided they really like to fish!