If you stop yourself from building the sewing hut, but drop it as close as you can to the left corner of the mud hut area, it will block most of the villagers from walking directly through the crowd of heathens while fishing.
Also, if you don't bring the lake back to life, your only food source (at least from adults) will be the farm, which you can Bloom at will. That will completely eliminate the problem of walking through the heathens. You will have to pay attention to the farm, but it's possible.
I assume that you've fully researched food. If not, there's your problem.
If you're playing at 2x speed it is harder to sustain a large population, but it should never be that hard.