I have 11 villagers and an infant, and have completed puzzles 1-3 and 6 and 10 and working on several other puzzles.
Maybe this is too many people for this stage, but I have a healthy and growing food supply.