The villagers require unobstructed access to the ocean or they won't go fishing!

Don't worry too much about how often they wander off to do other things. Although you can micromanage your villagers, and sometimes I still do that if I have a specific short-term goal in mind, most of the time I've taken a more relaxed attitude toward their wanderings and let them have breaks! I'd hate it if some giant with a stylus made me stand in the kitchen all day!