The limit is 50 Esteemed Elders. Once you reach 50 elders, you cannot create any more Esteemed Elders (with or without totems) without encountering assistance from things outside your control. You can still have villagers master 3 or more skills, but they will receive neither the Esteemed Elder title nor the accompanying totem. The people on the leader boards with more than 50 elders have either run into some fortunate circumstances with Island Events or the gong (or they've cheated).