Arnie's answer was correct. The builders don't have to be anywhere near the statue for that to happen. The villagers "decide" what to do next while they are still at the site of their last task. If they decide to work on the statue, and all of the available 'pushing slots' are full, they will be confused, because there is no more room to help. Then they will choose a new task (which could also be pushing the statue, in which case they'll be confused again, unless a slot has become available).