Unfortunately, rather when a villager wants to chat, argue, complain, tell a story, etc. rather than pick a random other villager (or the closest one available) they will pick the village "elder", the first one on the list. It does get frustrating when their work is always interupted (in one game he was also a runner!)

If the enler is female, I'll set her to breeding. That way she can sit there for two years listening to all the other villagers and it doesn't matter as much.