As you have figured out you need builders. Unless you get a strom to come by and help you out (or maybe another event), you might be stuck. I have found keeping one of the ladies trained in building helps solve this, as it would me more likely that her offspring will have a little building skill when he/she is born.

As far as starting over, that's your call if you wont to wait for an event to happen or not.