It depends upon how often you want to check on your villagers. The faster the game speed, the more quickly the food will be eaten, so I would not increase the game speed until your crops come in.