You can try pausing the game and then pile all your villagers on the robe and then unpause the game.