Sometimes it works better if you drop a villager near, but not directly on, the herbs. You can also use kids to gather herbs, and they do it much faster than most adults.