I'd say clearing the leaves is pretty non-essential. As Elizabeth suggests for the rubble, stop them once they succeed in picking up a handful of leaves and drop them back again. You won't (hopefully) need the leaves cleared until the game is pretty well advanced as no-one will have died yet. Clearing the rubble does make the tablet pieces available to be found.