Yeah, but you can scare them off by, dropping a villager near them.