I guess I was lucky that my peep was able to get the remote out without too much trouble (took a few tries).

Try moving the peep using different methods. Like, try moving it very, very slowly toward the edge, and then gently letting go. Or try kind of throwing it at the edge. Sometimes they "snap" backwards when they're close to the edge, and moving them toward the edge from a different direction can help.