Be sure to drag ALL of your villagers around to check spots, not just the useful ones.