I've found you need about one adept farmer per six to eight people including children and babies. So you can set any of the others as scientists and fast-track your technology.
One thing I recommend is give everyone basic training. Make sure everyone over 14 years old has at least one point of skill in each field (except breeding.) This way, if there ever is a food crisis, more people will collect food, and if there's an abundance of food, your farmers will spend some time researching.
I'm playing on hard at the moment, and I found that I couldn't support more that about six or seven villagers before I learned Farming Level 3.