They won't work on the buildings until they are trained and told what you want them to do. Make sure that you have trained one or more villagers as builders and then tap on the check box next to "Building" on the details screen for each builder, so that they will continue to work on the building instead of doing something else.