Sometimes I actually wish they would "slip away"... the really stubborn and slow ones, so I can have room for more helpful villagers.