i am able to move my totems every time, all the time. i don't know what's wrong with your game but if you really can't, you can easily make more totems before you have to deal with the gong piece that requires a totem.