Apparently, when you click the villager to use the ability on, you have to keep the villager selected, hovering with her/him in mid-air.
At least, this is what finally worked for me.
Before, I was simply clicking on the villager and then clicking the Time Warp icon.