You'll have to find a way to keep the red and orange/yellow masks off of the worksite. OR you can try to attack the totem.

In my current game I got the mausoleum early on and only had to dispatch the blue guys while my kid distracted the two yellow masks trying to get there. Although the heathen master builder did decide to try and work against me near the end, but I was able to interrupt him and I don't think he affected progress much.