What little things about your villagers drive you nuts?
Mine is that they don't take the shortest route from the fields to the food bin. Even if they're at the end closest to it, they walk down, around in a big loop, then to the bin.
I also can't seem to keep them on one farming task. They're either harvesting coconuts when I want them to harvest crops, or watering the field when I want them to fish.