In fact, they will usually go fishing even if they dislike swimming. If you drop the villager in the water they will often run away. If left alone, however, they will often voluntarily go fishing. They don't seem to make the connection between swimming and fishing.