Put your peeps to building the statue, "not long after" you tear the fence down.
Take a grown-up/nursing mom and get "rascal" in the area between the garden and the steps to the forest. "Not" too close to the statue. Keep clicking on him/her and the statue to check on progress of statue. When they workers quit working that is the farthest you can get til you reach L3 in Spiritual, and I think "all the relics" to finish it. I "haven't gotten all yet in either of my 1st tribe or 2nd tribe.
Then you will have the wood for "all" the building projects "without" worrying about the 2 guards, only rascal.