It only takes one trip to each plant.
And it doesn't matter who you use.
Watch the villager carefully, don't take them away until they walk away completely, takes them a while. If you have that villager selected, you will see a message that says "your villagers now understand this plant (or herb or something like that)"
Let them walk away completely from all four plants and you'll be fine.
But also make sure you are putting them on the right plants also. There are five plants, and it takes a particular four.
_________________________
My VV2 story, "A Twist of Fate" And don't miss Krystal's stories, all three of them!
"Life is too fast... unless you're in traffic."