You need level 3 of farming and will have to complete a certain puzzle before your villagers will be able to fish.

The cave is a puzzle, as well, that's solved very late in the game.