No, age shouldn't matter in terms of gaining skill. Even though your builder's bar doesn't move, he is still gaining building skill each time you see the message "improved in construction". If you want to stick with this tribe, then it sounds like you will have to keep putting him on the huts to fix them to become a master and a lot more often than 10 times.

I think once you select easy, normal, or hard from the options, you are stuck and cannot change it mid-game but I could be wrong.