I generally have little difficulty using one farmer for up to about twenty people (at 2 X speed) but it's important to keep a close eye on the situation. I try to train villagers in a variety of skills so that they can slot into another job easily if need be.