Mine got stuck when I had the totems all stacked on top of each other in section #1 (Nature is Calling!). So it's definitely not the totems. Also, as far as I can tell, the villagers don't seek out the totems.

I think the villagers getting stuck is just a _bug_ in the software. Too many people have reported the same problem in the same places.

I think the reason it seems the villagers are seeking out the totems is because the totems just happen to be placed where the villagers naturally go. It's not the villagers who are seeking out the totems, but the player who has put the totems in an area where the villagers like to go.