Dang edit timeout, anyways....
I don't actually know whether the heathens are from the lost civilization as VV5 gives no hints in that direction, plus it has been many generations since the refugees arrived on VV1. In fact, the intro for VV4 says that many generations have passed. So there could be some sort of conflict that happened elsewhere between VV4 and VV5 or between VV3 and VV4.