He is gaining skill; the difference is that as he rises in skill, gaining further skill has less effect. Think of it as a role-playing game. You may need 100 experience points to get to level 1 (trainee). You might need 250 to get to level 2 (adept). Level three might take 500 points, and level four might take 1000 - a total of 1,850.

This is only an example, but the point is that as your villagers gain skill, they need to repeat the action more often to make the same amount of progress. Just keep up the storytelling.