I have also completed all the puzzles and got all the collectibles and rung the gong a few times now and am no rapidly ramping up number of villagers...
There is just one thing that has been bugging me for days. It has to do with the brick wall in the top left corner. Every so often when I drag an adult villager into this position something will fly by rapidly - like a large bird. Has anyone else noticed this? Any hints or clues? Is it part of the game? A separate puzzle?