Keep him selected for a while and see if you get the message 'This villager improved at farming' while he's working. The bar moves SO much more slowly when they start to get close to master!
I'm having the same problem with my villager, but I'm getting the message so all I can do is wait and hope.
(You might be able to speed it up slightly by picking him up and moving him back into the field whenever he drops off the food, so he doesn't have to walk.)