My villagers do the same thing, I'm constantly asking them where they're going! I also can't seem to keep them out of the stream, all they want to do is cool their feet or swim, makes for slow progress!