They will decompose after awhile, but there is another way to get rid of them! It is one of the puzzles, so I will not post how. Just take one of your villagers and go exploring.