@Manaiacal Zebra: What I do is zap it with lightning, pause after a few seconds to let any yellow masks get some distance, then put all available villagers onto the totem. It can take as many as three tries to completely dismantle it.

Still, the fact that you had zero progress is odd. Maybe it's your timing?

@Misskathy: The heathens don't try to rebuild the pain totem. They would get zapped by the totem if they tried anyway.