the glow light is what you want to get. You don't have to talk to them after that. Just use fog or earthquakes to get the glow light, then wait. When the light goes out hit them again. And again, and again, until they finally convert. Using the fog or earthquakes for red and yellow is like talking to the blue guys.

I tried fog once and got one glow light on one yellow guy. Nothing else. And fog has done nothing since then. I've used fog on several occasions, mostly so the villagers can fish, but nothing ever happens. I don't have any other options, because I don't have the powers.