I had the same happen, but when I picked up my villager and set him back down there was a totem...