I dress my villagers in different clothes so I can tell what their skills are (Elders of both sexes are in white). I found a couple of villagers (one male and one female) who disliked fish and swimming so I made sure they were the only ones in a particular outfit (yellow with red spots for the female and a green skirt for the male) and then I could see them in the crowd and not select them to go fishing.