I used the bee power on the red masked heathen on an area of the springs far away enough from the Rainbow Totem to make him temporarily go far away enough from the totem, so the villagers can dismantle it. I didn't even have to upgrade Science to level 2 in order get the clothing hut to dismantle the totem. It took many uses of the bee power in order to do this, but it was worth it.

After dismantling the Rainbow Totem, whenever I tap on the dismantled totem, it says "100% of the Rainbow Totem dismantled" but when I tap on the dismantled Hungry Totem and Knowledge Totem, no message shows up.