More info, Lady CFII:

It might be a 'sequencing' issue that I created. We started on the rubble first. Then, my village population topped out so we had to start working on the 2nd hut.

Now that the 2nd hut is completed (which took the game being on all the time and the chief directing them), they are back to working on the rubble pile. Some are also fishing and a few are researching. Most are working on rubble. The chief is no longer directing 'cuz she had a baby (had to - she was one of my few child bearing women left and my tribe was dying off).

So maybe it was because they were 'assigned' to rubble first and then I switched them to the hut - and they kept going back to the rubble (though as I mentioned, they didn't make much progress on that either - it's still slow progress but at least when I start the game again, they are already AT the rubble pile (vs. walking towards what they had been assigned as they were with the hut).

Thanks.