If you look you'll see a color is missing. Have two villagers bring the two colors of dye that make the missing color to the pool at the same. when you make that color the heathen(s) will leave the pool giving you some time to dismantle it. You'll have to do it several times depending on how many villagers you have working on it.