With that stew, I have notice what happen following one villager:
(It's valid for all kind of exercises except when he bangs the sewing-hut) when he finishes to exercise, sometime he says "no" with his head and I suppose that it means he doesn't increase his building skill, and sometime he sits down and there is a notification: "this villager improves at building" (on the bottom of the screen). I don't know if it is the same when he exercises by his own but, next time I see that, I'll try to follow him.