From what I've read on the forum, having villagers plant or water does nothing for the time it takes to grow crops -- if you are babysitting your village, then it doesn't hurt to change all your farmers to another task for a bit.
If you are turning off the game, though, you should keep a couple set to farm so they can harvest the crops while you are away -- your villagers won't spontaneously harvest until the bin is really low, and by then it's too late to build back the stocks by themselves.
-- jen