Do you have your villiagers set to parenting? If you want them to do it on their own, you have to check mark next to parenting in the detail screen.