My tiny annoyance is villagers who mate without my permission. This wouldn't bother me, except that I'm trying to promote monogamy in my village (something that being able to track parentage has made much easier). There's nothing in this game that annoys me more than starting the game to find one of my female villagers nursing a baby and me not knowing who the father is (until it's two years old obviously).