Yes, that is correct. Builders only gain skill when fixing or repairing the huts. The exercise potion only works sometimes. Same like the fixing the huts.