Lightning NEVER, EVER scares them off. Ever.
Bees will make them move, but only once out of three attempts will get them to move far enough away to actually work on dismantling the totem.

I just don't get it. Every walkthrough says to just use lightning on them, but they just keep bathing and won't move at all for that.

Hello, Knot4Him! There are several different ways to complete this puzzle. Although multiple lightning bolts can be useful to keep the heathens at a distance while your villagers dismantle the Rainbow Totem, there's a much more useful power, if you have enough energy. Check out this discussion (read through both pages).