You can yank them off the trees, but you have to grab the base of the tree to do it (the villager's selection point stays on the ground).