Did you have a villager man set to Breeding? He might of bred with all the other females.