I've noticed that too. I have a few villagers that I've checked to do one thing but when they get bored they go off and do their own thing. I'm just glad that its productive and not sitting there eating me out of house and home.

(Which I believe they try to do, I'm living off seasonal food and in between harvest berries.. and there has been some really close calls.)