That's just how the game works when you have multiple building projects started at the same time. The villagers are free to choose any available work that is consistent with their skills (guided by, but not slavish to, your selection of their preferred skill). If you want to make sure that your builders work on a specific project, make sure that's the only thing they have to work on. With the exception of huts, they will not work on something that you haven't initiated (although they may occasionally be curious about things). You gave them "permission" when you gave them options!