I think that's a different situation, Brenda. The situation I described involves villagers who act as if they've been dropped in the middle of nowhere, rather than at a spot where they could actually work. They don't refuse to build, because they don't even know that they've been asked to build.
In the situation that you're describing, they know they're supposed to do something - they just won't do it!

Some villagers just don't seem to have much aptitude for certain tasks. For example, I've seen villagers (I call them "farm dummies") who refuse many, many times to forage or farm. But, when I move them to the research table, they "get it" on the first try and stick to it like a burr on a dog. These little villagers are fascinating!
