Seems to me that as the village gets "older" and you have finished most of the techs and puzzles - that is when I have a problem getting anyone to want to work!

They all "enjoy the pond" or "relaxing" whenever I look at them!

My "newer" villages have more dedicated workers LOL