I think it's caused by the low food. I accidentally left my second tribe on 2x speed overnight so when I woke up, I found they had 0 food with a number of villagers' health being lowered. I had just 2 kids still 13 years old so I managed to make 1 healing stew before they turned 14.

But then they turned 14 so I had no children left, 0 food, and not enough tech points to get rid of the algae. The next crops weren't going to be ready for a couple of hours and there were 0 coconuts left. Very dire. The only thing that could save them was to build up the tech points.

So I put every villager on research. To my surprise, each and every villager worked constantly on research. Not a single one wondered off for any reason. They didn't try to eat or go to the pond or cool their feet or do another skill; nothing. Just research. This is the only time I have ever seen 23 villagers stick to one skill for a long time without ever wondering off. The only thing I could contribute this to was the dire situation with 0 food left.

They finally made it far enough for me to buy the 3rd level of farming. I then changed them all to farming and made them get the algae fish. Some villagers who had 0 experience with farming had a little trouble, but for the most part, the villagers again were very dedicated to their job without wondering off.

However, once the crises was over and there was plenty of food again, the villagers began wondering off again.

My only conclusion is that they become very dedicated to their jobs when they're about to starve.