When a villager is selected, clicking on anything else has always deselected them in all of the VV games, and the focus camera in VV4 has not changed that - it's just more obvious now, with the addition of that feature. There are many different uses for the focus camera, but leaving a villager selected while you scroll around the map or work with other villagers is not possible.