We'll start with the easy things that won't require you to start over. It would help to know exactly which version of the game that you're playing. There are many places to buy the game, and they don't all have exactly the same version. So, to pinpoint which version you have, I would need to know where and approximately when you purchased the game, as well as the exact version number that's displayed on the menu screen.

The next step is to do a little experimenting. Find a few villagers who have skill levels that you can keep track of (either by screen shots or making written notes). Then keep an eye on what they do, checking their skill levels after different activities. It might be the most productive to start with feeding them various stews and check their skills each time they eat (since that's the one "lead" that we have). If that doesn't cause a reduction in their skills, then try other things. Since we've never been able to find this problem, I can't give you any other specific guidance.