Are you putting your person over the right plant? When you drag them over a special plant the bottom line of the screen should say "(NAME) sees a (PLANT NAME)" (where NAME and PLANT NAME are the names of the villagers and the plant - if you then drop the villager on this plant it should then say "Gathering herbs" in the Action line.