The only time I took my main farmer off of farming was when enough food was stored up. Otherwise, that's all he did. He's now a Master Farmer, with a son (looks exactly like him) also working as a farmer.

So, I would suggest concentrating on one farmer and make sure all he does is farm.

Now, my problem is getting a Master Builder. Do I just keep one builder fixing huts until more are needed to be built?