Yes, I gave up because Rascal was the only one left on the island. I did wait for a while hoping for an event where some other villagers or barrel of babies might arrive, but when that didn't happen. I bid Rascal a fair farewell and gave him the island.