Achievements, Opportunities, and Challenges in Using AI for an Improved Building Enclosure Assessment Practice William Wilkins, Jr. SeekNow 75 5th Street NW, Suite 2140, Atlanta, GA (678) 801-6223 | home@wgwjr.com Nonpresenting Coauthor Habib Fathi No v e m b e r 9 , 1 0 , 1 2 , 1 6 , 1 7 , 1 9 , 2 0 2 0 | 2 0 2 0 I I BE C B u i l d in g E nc l o s u r e S y m p o s i u m Wi l kin s | 1 0 9 ABSTRACT SPEAKER William Wilkins, Jr. SeekNow | Atlanta, GA William Wilkins has been responsible for sales and business development to propel drone-enabled 3-D AI services across the roofing and facility management industries. He has experience running a drone startup company that capitalized on powerful analytics using drone-based imagery, and has worked with advanced computer vision and AI analytics in roofing for over 10 years. Wilkins has a passion for changing industries for the better, and regularly speaks at trade shows across the country, including IRE, GARCA, RT3, IIBEC, AIA, SRWA, and Metalcon. Nonpresenting Coauthor: Habib Fathi Visual assessment of building enclosures can be repetitive and time consuming. The industry has also faced numerous challenges in recent years due to shortages in skilled labor, inefficiencies in process, and an increasing focus on worker safety. Artificial intelligence (AI) applied to drone-captured imagery is emerging as an answer for these problems. Although this seems like a straightforward process (capture drone imagery, train machine-learning models, and predict objects and deficiencies), we have discovered, through extensive research and onsite tests, that this generic approach often fails to provide reliable outcomes in a practical setting. This presentation explores, in detail, several factors which significantly affect the results—including distance and angle of drone camera to each surface; practicality of a capture plan in terms of time, accessibility, and obstacle avoidance; differences in surface materials; a large number of assets and deficiencies; and the necessity for large training datasets—especially for less-common inventories and conditions. Case studies will be presented showing significant achievements in automation and efficiency when these challenges are overcome. For example, a drone-based data capture of a large hospital facility, coupled with a novel 3-D AI solution, led to a more than 84% savings in worker hours compared to the manual assessment scenario. 1 1 0 | Wi l kin s 2 0 2 0 I I BE C B u i l d in g E nc l o s u r e S y m p o s i u m | No v e m b e r 9 , 1 0 , 1 2 , 1 6 , 1 7 , 1 9 , 2 0 2 0 No v e m b e r 9 , 1 0 , 1 2 , 1 6 , 1 7 , 1 9 , 2 0 2 0 | 2 0 2 0 I I BE C B u i l d in g E nc l o s u r e S y m p o s i u m W i l kin s | 1 1 1 INTRODUCTION Building enclosure inspection or assessment at regular intervals is an important part of building ownership and management. It is defined as an in-depth analysis based on information captured from a visual-only survey—or sometimes a more destructive method—to document the current state of the facility and/or identify potential problems with the building enclosure. A visual inspection typically leads to identifying a symptom, but sometimes a more in-depth and destructive evaluation is needed to identify the root cause or precise location of the underlying problem. As an example, it is often not straightforward to identify the source of a visible leak; a leak in the roof could be related to a defect in a wall. Efficient building management also necessitates prioritizing repair and maintenance while being able to anticipate the service life of building enclosure components. This is a challenging task, as knowing how to best allocate limited resources involves difficult decision making that is often complicated by insufficient information on building conditions. However, if it is done properly, repair and maintenance provide a written record of the facility’s condition at a given moment in time, which includes detailed building component information, comparative condition evaluations, itemized repair recommendations, photographic documentation of the entire facility and key problem areas, construction cost estimates, etc. Such a document has the potential to serve both as a roadmap for the potential repair and ongoing maintenance of the building and as a snapshot of the state of the building. The ultimate goal is to create a living document that is continually updated, amended, referenced, consulted, and shared. A visual inspection is the first step in conducting a building enclosure assessment. It involves heavily manual and time-consuming steps which lead to a report including a summary, mapping, and analysis of observable conditions. These manual steps, however, are often repetitive in nature and, hence, are suitable for automation using artificial intelligence (AI). One of the primary objectives of AI is to automatically recognize patterns and regularities in data; this, in turn, results in identifying anomalies and allowing for the development of computer algorithms able to perform the same tasks that are carried out by human intelligence. It should be noted that in the case of visual building enclosure inspection, AI is limited to analyzing observable conditions and cannot assist in determining the existing composition and configuration of concealed assembly conditions. Therefore, AI functions as a tool that provides human-like inference from visual data, offers decision support for specific tasks, and enables a more efficient and less error-prone turnaround; however, it is not a replacement for a comprehensive onsite inspection by an expert inspector. This paper describes real-life applications of AI in the building enclosure inspection field. It highlights where AI helps in the process; why it is so hard to achieve a reliable AI-assisted framework; what limitations are expected in the automation, training, and prediction phases; and why opensource or general-purpose AI and machine learning tools/libraries are not performing well in this field. It provides several case studies that have shown significant achievements in automation and efficiency when these challenges are overcome. It also predicts the role that AI will play in the coming years. The content of this paper is organized to share with the intended audience a clear understanding of a) challenges in automating repetitive, time-consuming tasks around measurements, as well as asset identification; b) which deficiencies can be identified automatically using AI from drone imagery; c) how to present information gained from an assessment in an intuitive format; and d) opportunities in tracking and comparing data over time. INSPECTION CHALLENGES IN THE BUILDING ENCLOSURE SPACE The building enclosure industry has been facing several challenges in recent years due to a shortage in skilled inspectors and labor, a push for increased productivity, tightened safety rules, and increasingly strict requirements concerning energy consumption. In order to perform an inspection, traditionally, an inspector conducted an onsite visit and performed the inspection using a variety of methods which were often visual (using a clipboard, pen, and camera), but sometimes expensive and inconvenient. This included a number of repetitive and time-consuming tasks that would eventually generate massive data which could easily become intractable and require a significant amount of office work to organize, link, and present in a report format. Several other factors exacerbate the situation, including the existence of hard-to-access or expensive-toaccess areas (like those necessitating the use of scaffolding), fatigue from repetitive inspection, harsh weather conditions, unsafe locations, and multiple elevations and access types (including ladders, hatches, doors, etc.) which complicate accessibility. From time to time, these may even cause some people, especially those who have to meet the cost of the inspections, to avoid the actual exercise. There has therefore been a push in the whole industry to adopt digital technology and embrace innovative, but affordable, technological advancements which facilitate performing comprehensive inspections and reduce the monetary burden of doing so. Thus far, this has manifested in the form of mobile technology and software, personnel management, building information modeling, augmented and virtual reality technologies, analytics, AI and machine learning, customer relationship management software, digital documentation, etc. More recently, drones have been gaining popularity in the building enclosure industry. They enable inspectors to Achievements, Opportunities, and Challenges in Using AI for an Improved Building Enclosure Assessment Practice access any part of a building and capture high-quality images and videos. This is particularly useful for assessing fire or storm damages, or when access is difficult or unsafe. If equipped with a thermal camera, drones can also capture thermal imagery that helps in identifying possible leakages in insulations. Furthermore, if the imagery is captured while adequate forward and side overlap is enforced, photogrammetry software can be employed to generate topography maps, 3-D models of buildings and sites, and orthomosaic imagery. This all works well when inspectors fly the drone themselves and have access to a live stream of the visual data so that they can point the drone towards any areas of interest for more in-depth and up-close inspection. It also works well if the captured drone data is intended to only be used for marketing and sales. Usability, reliability, and practicality of the drone data decline significantly if the drone pilot is hired to visit the site and capture imagery using automated flight software with a generic flight plan. As a result, the adoption of drone technology for data capture and back-end analytics software has been limited in the industry. However, major players are now engaged to change this path. WHAT IS AI, WHAT ARE ITS MANY VARIANTS, AND WHERE IS IT USED TODAY? AI is commonly defined as a field in computer science that aims to solve cognitive problems by imitating human intelligence in learning, problem solving, and pattern recognition. A primary discipline in this field with tremendous potential to change the status quo in multiple industries, is machine learning (ML), which stems from the science of statistics and probability. ML is a technique for realizing AI, and empowers computer systems with the ability to learn by themselves using the provided data and make accurate predictions. The underlying algorithms are capable of ingesting an enormous amount of data (which might seem unstructured as far as a human can comprehend) and extracting hidden structures and patterns from the data by identifying concise descriptions to classify the data. Once the algorithms recognize the right pattern from the learning data, a series of statistical codes is generated (i.e., a model) which can make predictions under uncertainty implied in a new input data that had not been seen before. This is usually linked with statistical confidence in individual predictions as well as on the overall performance. It needs to be mentioned that the accuracy of the generated model is dependent on the quality and quantity of the training data. Furthermore, deep learning (DL) is a branch of ML and is usually characterized as the next evolution of ML. It is capable of analyzing high-dimensional problems with extraordinarily large numbers of data points due to advancements in network computations. DL is a powerful tool that can, in theory, identify hidden relationships and patterns in input data that could not even be modeled or perceived by a human-written code. This is all possible due to layering algorithms that could go beyond establishing explainable relationships in the dataset and instead try to create a distributed representation of the statistical information extracted from data patterns. In this paper, we refer to all three of these categories under the generic term of AI. At its core, AI can be divided into supervised and unsupervised learning techniques which primarily differ from each other by whether or not they have access to a desired output. In supervised learning, the desired output from the training data is known; for example, if an image of a blistered roof surface is being used in the training process, the roof damage type information is provided to the algorithms via a proper label. The algorithms will then learn from the image content and its associated label to predict the correct roof damage type if such a pattern is seen in a new image. AI has been utilized for several years by many industries to predict future outcomes based on historical data. A number of vastly successful use cases include personalized content recommendation engines, fraud detection, natural language processing, speech recognition, anomaly detection, risk assessment, healthcare, and visual data segmentation and classification. The construction industry has also started to embrace AI-enabled technologies to automate repetitive tasks to a large extent and mitigate the scope of human errors in the process. Managers and supervisors are empowered to get alerts on problematic issues and hence take proactive or corrective measures to minimize errors and risk factors. There is a potential to also realize all these benefits in the building enclosure industry. A significant amount of data is recorded in the process of inspecting different surfaces of a building enclosure. Thousands of images are captured by inspectors every year. The majority of this data is labeled and categorized in the process of generating an inspection report; this would, therefore, be an excellent source of training data for AI algorithms if a standard taxonomy and practice were used in that process. With the current widespread use of drones, high-resolution imagery captured using drones from a specific distance, angle, and location could unlock this tremendous potential with the promise of automating most of what building enclosure inspectors do every day. Although this seems to be a straightforward process (i.e., capture drone imagery, train a predictive model, and make predictions), it is shown that a generic approach often fails to provide a reliable outcome in practical settings. What differentiates the AI application in the building enclosure inspection field is that the traditional approach of making predictions based on individual 2-D images has failed to produce promising results because buildings are three-dimensional assets, each with their own unique location, geometry, layout, texture, and semantics. A successful application of AI in this field needs to employ a holistic view by considering a combination of historical, textual, 2-D, 3-D, temporal, and cost information. Such an approach will have the highest chance of success for automatically extracting measurements, identifying material types, producing an inventory of objects on the building, locating and classifying damages on building surfaces, etc. As can be inferred, drones seem to be a key component of the solution, but not a differentiating component. HOW AI IS USED TODAY AI-enabled technologies are currently being used in different stages of a building lifecycle. Risk assessment is usually done before the construction process even starts. AI compiles huge amounts of data and analyzes construction-relevant risk factors. AI has also begun to profoundly impact the way architects envision and create the built environment. Additionally, significant impact can be seen during the construction process. AI-enabled construction management software is being used to enhance productivity by monitoring and supervising daily operations. Safety stan112 | WilkinKINs 2020 IIBEbeC BuildINg ENClosure Symposium | November 9, 10, 12, 16, 17, 19, 2020 dards at job sites have risen to new levels thanks to continuous monitoring of safety risks by real-time processing of the visual data captured via surveillance cameras and other sensors. Storage and retrieval of massive amounts of data that are collected on a daily basis have become hassle free. Autonomous equipment and robots powered by AI are also deployed on more and more projects. The most consequential impact can be seen once the building is in place and the facility management process begins. This is due to the duration of this process (typically decades), number of people that it affects, amount of capital needed for proper maintenance, etc. AI-enabled technologies can be used on a wide variety of domains in this phase, such as optimizing serviceability, managing energy consumption, minimizing operating cost, and making decisions on maintenance operations. This section focuses on use cases most relevant to the building assessment process, while assuming that a set of overlapping imagery is captured using a drone following a properly designed flight plan. Figure 1 illustrates five major areas in which AI can be of great help if such a dataset is available. Measurements A 3-D representation of a building can be generated using overlapping imagery and off-the-shelf photogrammetry software. This 3-D representation, however, will not display any high-level characterization of the geometry that is understandable by computer systems, despite the fact that the represented geometry is easily comprehensible by a human. To a computer system, the given 3-D representation is a collection of individual colored points or textured triangular faces. AI could be utilized to analyze the imagery along with the 3-D representation of the scene and to infer the 3-D geometry and topology. It starts by identifying building surfaces with primitive geometry and then combining/connecting them to deduce the overall complex geometry of the building. Once the geometry is defined, vast amounts of information can be automatically extracted, including: corner location, elevation, and type; edge location, type, and length; surface boundaries, slope; orientation; and connectivity; a 3-D wireframe can be generated, as can 2-D wireframes from specific viewpoints, and site plans. Inventory An inventory, in the context of a building enclosure, is a list of all building components, in which each component has general information about its geometry and condition. A proper inventory provides a vital set of information that will place the on-going maintenance program upon a firm foundation. Maintaining an up-to-date inventory of all building components facilitates any preventive maintenance program, and these two coupled together can extend the service life of the building by 30% to 200%. AI is capable of scanning through the 3-D representation of the building and its corresponding overlapping 2-D imagery in order to not only identify but also organize the data according to roofing and façade systems. Different components on each system are identified, segmented, and labeled, including any objects on a roof/façade system, obstructions, openings, doors, windows, pipes, etc. The layout and topology of these components can also be extracted. The inventory is then augmented by any semantic or conditional information that can be detected from the visual data. Damage assessment Damage assessment of a building enclosure system is a process by which the building enclosure is assessed by an expert to identify regions or components that demonstrate abnormal visual and structural characteristics. There are a wide variety of surface damages that are sought in generating a damage assessment report ranging from moisture, air leakage, and heat loss to blistering, bulging, and cracking. Most of these damage types have unique characteristics (e.g., size, shape, pattern, extent, color) that would necessitate intricate flight planning ahead of the data acquisition event. This planning should be based on specific damage types that are sought for this specific inspection. For example, if one of the goals in this inspection is to detect cracks on a particular roof surface, the distance of the drone to the target surface and its viewing angle at the time of capture should be designed carefully in order to provide data with enough resolution (i.e., distance covered by one pixel) so that the AI can perform. November 9, 10, 12, 16, 17, 19, 2020 | 2020 IIBEbeC BuildINg ENClosure Symposium W WilKINs | 113 Figure 1 – Five major areas AI can currently help in the building enclosure industry. Change detection Change detection implies identifying and quantifying temporal effects using a multi-temporal dataset. The dataset in this context refers to overlapping drone imagery, and its corresponding 3-D representation of building components, captured in multiple missions. The primary goal here is to determine geographical location and types of changes, quantify those changes, and extract semantic information that can be translated into actionable data. Automatic change detection can provide stakeholders with an abstract representation of multiple temporal datasets and is proven to be extremely valuable in tracking the progress of different damages in time, saving time in prioritizing preventive maintenance actions, and highlighting regions that need more attention. An exemplary list of potential changes in a structure could include deterioration, surface damages, addition of structural elements, addition of accessories, occlusion status, etc. AI can automate this process and generate change-detection reports at predefined intervals. Overall health grading A building enclosure inspection results in a significant amount of data, which translates into a long report. These detailed findings need to be summarized for stakeholders so that they can make informed decisions without getting distracted by the sheer volume of technical details. A building enclosure rating system can be of great help, and AI has shown promising capabilities to achieve that goal. Such a system should be able to objectively identify the level of quality of a specific roof or façade system. According to Federal Transit Administration’s (FTA’s) Transit Economic Requirements Model (TERM), a general condition assessment rating scale can be defined as follows. It needs to be noted that the scale is categorical and hence only integer values are defined. • Excellent (5): no visible defects, new or near new condition • Good (4): good condition, but no longer new, may have some slightly defective or deteriorated component(s), but is overall functional • Adequate (3): moderately deteriorated or defective components; but has not exceeded useful life • Marginal (2): defective or deteriorated component(s) in need of replacement; exceeded useful life • Poor (1): critically damaged component(s) or in need of immediate repair; well past useful life ASSESSMENTS AND OPPORTUNITY FOR AI AI has proven to be a very effective tool for automation in so many industries over the past years and has the opportunity to influence the building enclosure industry, especially the assessment process, as well as to achieve higher profit margins, reduced costs, improved time efficiency, better collaboration, increased productivity, and enhanced safety. In order to illustrate such capabilities, this section focuses on commercial roof assessment as an example and dives deeper into what it takes to perform such a task. Figure 2a shows details of the current practice for assessing a commercial roof system while highlighting different steps and information flow among those. As can be seen, this process includes multiple planning steps prior to the site visit, followed by a number of time-consuming on-site activities, and concluded by a series of post-visit office tasks to organize data and generate the final report. This entire process can take multiple days to weeks for a complex roof system. Figure 2b shows that AI has the potential to impact this process by completely automating the steps that are not dependent on an inspector’s expertise, and assisting on some steps that heavily rely on said expertise. This is called an AI-assisted inspection and is intended to reduce costs and inspection time by up to 80% while keeping an inspector’s feet on the ground as much as possible. It is not, however, intended to eliminate the need for an on-site visit by an inspector, as that currently seems to be impossible for multiple reasons that will be discussed in the next sections. 114 | WilkinKINs 2020 IIBEbeC BuildINg ENClosure Symposium | November 9, 10, 12, 16, 17, 19, 2020 Figure 2 – Flowchart of activities involved in a commercial roof assessment. (A) Current practice. (B) Practice improved by AI. WHY A GENERAL-PURPOSE AI UNDERPERFORMS General-purpose AI is an artificial self-learning mechanism that aims to simulate human thinking and analysis capabilities with computer systems. Although it is proven that a general-purpose AI solution is a powerful tool that could make successful predictions based on historical data in many applications, there are still problems that look very easy for humans but pose a significant challenge to AI; this is often attributed to the extra context and bias to which we have access. Image segmentation is such an example. Almost all adult people can easily segment a dog in an image while even the most powerful AI model underperforms in comparison. This is because people have lots of external contextual knowledge to solve this problem. Visual inspection of a building enclosure system seems to belong to this category of problems. An inspector is able to identify elements or regions at risk not only because of past training and all previously gathered experiences but also because of the contextual knowledge about the built environment to which they have been exposed over the years. Building inspection is not only about making decisions based on visual features of a surface, element, or component; it requires knowledge about how different components are connected to each other, and how they perform their tasks. For example, a visible leak in a roof could be because of a disjointed flashing on a parapet wall rather than a torn roofing membrane. Humans intuitively assume the correct metric while computers do not have an arbitrary preference (e.g., we have bias towards locally or globally symmetric patterns). Furthermore, general-purpose AI requires tens of thousands of images to be annotated in order to successfully build a model with good prediction accuracy. This imposes a challenge in the building enclosure inspection scenario as it is sometimes impractical to gather such data due to a large number of assets and deficiencies that could exist in different building enclosure systems. The problem is even worse for less common deficiencies or inventory conditions. It is important to note that the aforementioned challenges do not mean AI is incapable of solving such problems. They do, however, indicate that designing and implementing a successful AI solution needs to go above and beyond the generic approach. Any contextual knowledge or bias that can be translated into computer logic should be considered. A well-thought-out flight plan is a crucial step that could significantly limit the search space for the algorithms and hence maximize the probability for success. A number of important factors are distance and angle of camera to each building enclosure surface, practicality of a flight/capture plan in terms of time, cost, safety, accessibility, obstacle avoidance, weather conditions, and time of the capture during a day (e.g., shadows could impact an object’s appearance). Such a plan could also dictate the quality and quantity of training data that needs to be collected and hence it is necessary to resolve these issues even before a training process begins. SOLUTION: AI IN 3-D SPACE (I.E., SPATIAL IQ) Humans perceive the surrounding environment by interpreting its features into concise descriptors that represent geometry, topology, and semantics. This section presents a novel 2-D/3-D AI platform, called Spatial IQ, that follows the same logic for scene interpretation. It is capable of not only detecting and identifying the existence and type of a building enclosure component but also highlighting any potential damage or deficiency therein. The platform is also able to isolate accurate boundaries of such damage so its size and location can be measured with respect to a local or global coordinate system. Spatial IQ outperforms state-of-the-art general-purpose ML systems in terms of a) model training efficiency and convergence speed, and b) prediction precision and recall. This is due to the combined use of 2-D and 3-D data (Figures 3a through 3e) for narrowing down the search space and cross-referencing any detection, segmentation, and identification, November 9, 10, 12, 16, 17, 19, 2020 | 2020 IIBEbeC BuildINg ENClosure Symposium W WilKINs | 115 (A) 2-D Data. (B) 3-D Scene Data. Figure 3 – 2-D and 3-D data. (C) 3-D Object Data. and labeling in multiple overlapping sensor data types through optimizing a joint probability distribution function. Multiple AI models, each optimally trained for one and/or a combination of input data types and a specific range of parameters are used in parallel and/or sequentially to maximize the probability of a confident prediction both in terms of accuracy and recall. The architecture of this system is designed based on specific applications and expected output data types. Its benefits include ensuring full coverage of deficiencies from every angle using overlapping drone imagery, minimizing required annotation efforts as one annotation can be automatically converted into tens of other annotations in other images, cross- referencing, and augmenting by computer vision and image-processing techniques to guide any inference. Any raw data that is intended to be used for training or prediction initially goes through a processing step to calculate a number of quantitative descriptors which are crucial for ensuring the quality of the data and optimum path for data in the AI platform. Some of these parameters include ground sample distance, surface sample distance, angle between an image plane and a 2-D/3-D surface of interest, angle between an image view direction and a 2-D/3-D surface of interest, Euclidean distance between a camera center and a 2-D/3-D surface of interest, percentage of occlusion in an image, and material type. These parameters allow determining the best model among existing models with a maximum probability of success. Intelligent partitioning of training and prediction data minimizes noise and randomness, ensures well-defined boundaries for the search space, and minimizes the probability of converging into a suboptimal solution, which in turn leads to faster convergence in the training phase and better accuracy and recall in the prediction phase. In parallel to the mentioned data partitioning process, a coarse-to-fine training and/or prediction step is run. The goal is to hierarchize information that is visually analyzed. On one hand, the coarse information is perceived in terms of shapes and colors. On the other hand, detailed information is perceived via several stages of convolution and max-pooling in order to obtain fine-level information. As an example, the existence of a roof corner could be determined via a coarse prediction step. Once a rough neighborhood is identified, exact pixel coordinates of the corner can be detected in multiple overlapping images through a fine-level tuning step. This platform has also been augmented with a principal component analysis step. It employs a transformation to convert a set of possibly correlated observations into a set of uncorrelated values. This helps to determine a canonical representation for the given dataset. The dataset is then transformed into its canonical representation followed by training and prediction. Predictions can then be transformed into the original space while adjusting the confidence values accordingly. Practical limitations Even with all the improvements that are inherent to the Spatial IQ platform, there are still practical limitations that would necessitate a human’s (i.e., inspector’s) intervention and expertise to be able to achieve a reliable assessment report. The two most important factors affecting the magnitude of such an intervention are: a) practicality of the optimum flight plan for the particular building under assessment; and b) possibility of detecting a deficiency or damage from only visual data even by an expert human inspector. The resolution of the drone camera and the distance between the drone and a surface of interest determine what humans can see in the resulting image. As a rule of thumb, if a human can see an object or deficiency in an image, there is a good chance that the Spatial IQ platform can make a successful prediction. As an illustrative example, Table 1 presents the maximum allowable distance between a drone and an object or deficiency on a roofing system assuming a resolution of 20 megapixels. As can be inferred, this could be a limiting factor for an optimum flight plan especially if the roofing system contains tall obstructions. However, having this information beforehand allows realistic expectations to be set regarding what is and is not achievable through the AI-assisted drone assessment in a particular event. Accordingly, the inspector would be informed to pay extra attention to detecting objects or deficiencies that are known to be left unrecognized by the AI platform. There are also categories of deficiencies that cannot be detected from visual data alone no matter what the resolution of the drone camera is or how close the drone is to the target surface. Examples of these deficiencies would include open-lap seams 116 | WilkinKINs 2020 IIBEbeC BuildINg ENClosure Symposium | November 9, 10, 12, 16, 17, 19, 2020 Example Deficiency Approx. Size (cm) Max Distance (ft) Data Capture Method Flashing, Open Lap Seam *Requires manual probing Parapet, Coping, Separated Sealant *Requires specialized viewing angle Roof Membrane, Small Puncture (~1/2”) 1 4.1 Mobile / Drone Roof Membrane, Large Puncture (~1”) 3 12.3 Exhaust Fan, Damaged (Hail) 5 20.5 Penetration, Damaged Boot 8 32.8 Drain, Clogged 20 82.0 Drone Vegetation, Tree Encroachment 40 164.0 Roof, Water Ponding (~6 sq. ft.) 80 328.1 Roof, Water Ponding (~36 sq. ft.) ~200 ~800 Manned Aerial Catastrophic Roof Damage, Flooding, etc. ~1000 ~4000 Table 1 – Maximum allowable camera distance for reliable AI detection. or a parapet wall lacking structural integrity due to loose bricks and missing mortar. These deficiencies require manual probing, which means using a simple handheld tool (e.g., a utility knife) or a small power tool (e.g., a hammer drill) to remove the outer layer of material to reveal underlying construction, material, and waterproofing. The number of these probes depends on the size of the building, scope of the inspection event, and the building condition. Approximately two to five probes are standard for a roof replacement project. Core cuts belong to another category of information that is provided in some assessment reports but cannot be gathered through a visual-only AI-assisted drone assessment. They provide valuable information for accurate estimation of repair or replacement cost and scope while preventing unexpected expenditures due to unforeseen conditions. Core cuts allow an assessor to examine a roofing system for moisture, determine type and characteristics of roofing material, assess the structural integrity of a roof, and discover the slope and thickness of insulation. In conclusion, the presented AI-assisted building enclosure assessment solution rarely eliminates boots on the roof, but is able to significantly cut the time needed for pre-planning, site visit, and post-visit office work. CASE STUDY – AN AI-DRIVEN ASSESSMENT Details of a comparative case study performed with the help of an experienced assessment crew are presented in this section. The scope of this study was limited to the roofing system. Two similarly sized buildings with matching geometry and age were selected; one for performing a traditional assessment process and the other for an AI-assisted approach. The reason for refusing to perform both assessment approaches on the same building was to prevent any biases or efficiency gain due to a priori information about the conditions in the second assessment. The scope was limited to a visual-only assessment and hence no core cuts or destructive testing was performed. A traditional assessment journey map consists of four phases, including planning, information gathering, information organization, and estimation and delivery. Details of each phase are demonstrated in Figures 4a to 4d. The crew started with a planning phase which included preparing an up-to-scale 2-D sketch of roofing systems and evaluating accessibility for each one of them. This was followed by an onsite visit with a clipboard, pen, camera, and paperwork with pre-designed templates. The crew hen walked each roofing system in a November 9, 10, 12, 16, 17, 19, 2020 | 2020 IIBEbeC BuildINg ENClosure Symposium W WilKINs | 117 (D) Estimation and delivery phase. (B) Information gathering phase. (C) Information organization phase. (A) Planning phase. Figure 4 – A traditional assessment journey map. methodical way to ensure all areas were covered and the recorded data was retrievable with no confusion when generating an assessment report. Every surface and all of the equipment on top of the roofing systems were inspected, and images were captured to document conditions where necessary. This also included uncovering hidden defects through minor investigative probes to check underlying conditions of some components. The crew used handheld tools such as a utility knife and hammer for this purpose. Afterwards, post-visit office work began. This work consisted of organizing the recorded data; importing them into a series of roofing softwares for the purpose of planning, estimation, digitization, and documentation; and finally generating an assessment report. The AI-assisted drone inspection process was performed according to a different journey map (Figure 5a). It started by submitting the address of the property to an online interface enabling the user to also determine the region of interest and level of inspection. The request was processed and a double-grid flight inspection pattern (Figure 5b) was designed, taking into consideration the rough geometry of the structure extractable from publicly available data. A drone-flight service API was then called to schedule a flight by a drone pilot using the designed flight plan. Drone pilots were selected from a nationwide pool of over 15,000 (FAA Certified Part 107) people. Drone imagery was captured using automated drone-flight software without the need for a pilot to put boots on the roof. The data was uploaded to the cloud and photogrammetry, computer vision, 118 | WilkinKINs 2020 IIBEbeC BuildINg ENClosure Symposium | November 9, 10, 12, 16, 17, 19, 2020 Figure 5a – AI-assisted drone inspection journey map. Figure 5b – Double-grid flight inspection pattern. and AI processes were started immediately. After several hours of processing in the cloud to generate all necessary output files, a message was sent to the inspection crew containing take-off/measurement information, site plan, inventory report, assessment report, and AI predictions along with a link to a web-based tool that would allow the crew to visualize all types of output information and approve or reject elements of AI-generated results. Once approved, the crew printed out the desired data, sketches, and reports, and visited the building to complete the inspection process. Findings from the onsite visit were used to mark up and update sketches and reports. Upon arrival at the office, updates were transferred into the web-based tool. The cloud software automatically updated all related data, organized information, and generated the final assessment report. A shareable link was ultimately generated and sent to the inspection crew and customer. Some of the intermediate results are illustrated in Figures 6a through 6d, and outputs are illustrated in Figures 7a to 7e. Table 2 compares the required amount of manual office/onsite work by the crew in these two scenarios: traditional and AI-assisted drone inspection. This comparison is made separately on the four phases of the journey maps explained above: Planning, information gathering, information organization, and estimation and delivery. As can be seen, an approximately 80% No v e m b e r 9 , 1 0 , 1 2 , 1 6 , 1 7 , 1 9 , 2 0 2 0 | 2 0 2 0 I I BE C B u i l d in g E nc l o s u r e S y m p o s i u m W i l kin s | 1 1 9 Every surface and all of the equipment on top of the roofing systems were inspected, and images were captured to document conditions where necessary. (B) AI-generated roofing systems. (D) AI-generated roof assessment. (A) Request form. (C) AI-generated predictions. Figure 6 – AI-assisted drone inspection intermediate results. savings in time can be achieved via utilizing the AI-assisted drone inspection program and Spatial IQ platform; this translates not only into significant savings in terms of time and cost but also results in a very detailed digital documentation of the building that lasts forever and could be used in other maintenance and/or management software via direct integration of the data. FUTURE OF AI IN THE BUILDING ENCLOSURE FIELD AI-driven technologies are advancing fast and have already shown promising results in improving efficiency and reliability in different aspects of the building enclosure inspection field. These improvements are expected to get more radical than what they are now considering the huge amount of data collected every day. These large volumes of historical data need to be harnessed in order to build more powerful predictive models, empowering stakeholders to make more informed decisions on the fly. Managers and inspectors are likely to become more reliant on AI solutions in the near future and deployment of drones and robots for automating their tasks. One of the most promising emerging technologies, which could revolutionize the way a building enclosure inspection is performed today, is real-time analysis and documentation of textual, vocal, visual, geo-locational, and historical data using local or cloud processing power accessible by mobile devices such as smartphones or tablets. An inspector could perform an onsite assessment while having access 120 | WilkinKINs 2020 IIBEbeC BuildINg ENClosure Symposium | November 9, 10, 12, 16, 17, 19, 2020 Figure 7 – AI-Generated outputs. (A) AI-generated site plan with overlaid information. (C) AI-generated damage assessment report. (B) AI-generated takeoff report. Planning Information Information Estimation Gathering Organization & Delivery Traditional 4 hrs 4 days 3 days 1.5 days AI-Assisted 3 hrs 1 day 0.5 day 0 days Table 2 – A comparison of required manual time for traditional and AI-assisted drone inspection. to all relevant data and processing power on the fly, allowing them to make real-time informed decisions, which could, in turn, result in significant improvements in automation, reliability, and safety of the process. Virtual reality and augmented reality (VR/AR) can also play a significant role in this field. These technologies allow an inspector to have a virtual presence at the building prior to, during, or after a site visit. This virtual presence augmented by real-time access to most relevant data empowers the inspector to explore building components from viewpoints that were impossible before; this could unlock an untapped potential and change the way various stakeholders interact. AI-driven technologies are also emerging for holistic life-cycle management of a portfolio of physical assets instead of considering them as individual properties. This has the potential to impact the performance and serviceability of these assets in a way that it has never seen before. Coupled with building information modeling (BIM), this will allow the management of all aspects of the portfolio from a centralized portal providing services to all levels of stakeholders. November 9, 10, 12, 16, 17, 19, 2020 | 2020 IIBEbeC BuildINg ENClosure Symposium W WilKINs | 121
