Yup, like you, I found magic faction a lot easier too.
With 11 villagers, no chief and enough tech points to start fishing. No need to harvest any fruit, thank you very much. Although my lazy farmer would rather pick fruit than swim around catching and eating raw fish.