you need to use the lightning on it

hit it with the lightning bolt

paused my game and set the villagers on the right-side of the stream so that they're out of the way when I unpause the game and hit the pain totem with the lightening, electrocuting the heathens so they run away temporarily. I then wait just a few seconds as the heathens run away and place my villagers on the totem right away.

As soon as the totem is re-energized and the heathens come back, I hit it again with lightening (making sure once again that my villagers are out of the way). I can usually take out the totem with 2-3 hits of the lightening to scare the heathens away, using at least 5 villagers.

I haven't had an issue with the heathens rebuilding the partially dismantled totem since I've always had enough power to take the totem out, but I bet that they do rebuild it if you don't dismantle it completely and then you'd have to start dismantling it all over again.

Try again with at least 5 villagers (more is better). Make sure you have enough energy for a minimum of 2 hits of lightening, 3 would be better, or a large enough population to recharge your power quickly. I hope this helps.