I assume that you dragged an adult villager into the cave after it was opened. If not, you should at least do that.

There is a discussion about what to do after you've solved the puzzles here. You might find it interesting.