It increases by 4 and the increase happens as soon as you drop your villager on the fruit, not when it is deposited.