Shoot, just shouted about this same thing! I also have all my esteemed elders dressed in white, and I then retire them by unclicking any favorites, or putting the men on parenting detail. This way, I can see at a glance who needs to stop slacking off! I have about 17 villagers very close to elder status, but have reached my 50 with 1 who did not advance. I was looking forward to all the totems, I've been having fun with them.
Only 50 elders, not fair!