I've found that sometimes it helps to temporarily put their Preference checkmark on Farming/Fishing. It seems to help cut down on the wandering off. Also, after catching a few fish, they seem to like doing it on their own more.