I agree with all of the above responses.
For me, the most annoying villager is the one that refuses to learn anything and you spend 10 minutes training them at research, no mind the farming and building. All they do is look at you and say "Uh Uh". Excuse me... it's not that hard!