The more villagers you have, the more influence your chief will be when he/she directs work.

So if you have only a handful of villagers, they may wander off (maybe because their preference was checked for something else). But, as you increase the number of villagers that you have, the chief seems to get more influential. I've even simply put my chief on directing a task (mid and late game) and watched my villagers wander over to where he was to get to work.