Next Article in Journal
Impact of Oat Husk Extracts on Mid-Stage Cement Hydration and the Mechanical Strength of Mortar
Previous Article in Journal
Thermal Analysis of Concrete Blocks and Stack-Bond Prisms under Different Boundary Conditions
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Review

A Comprehensive Analysis of the Integration of Deep Learning Models in Concrete Research from a Structural Health Perspective

by
Ayesha Munira Chowdhury
1 and
Rashed Kaiser
2,*
1
Department of Civil & Environmental Engineering, Pusan National University, Busan 46241, Republic of Korea
2
Department of Naval Architecture & Ocean Engineering, Pusan National University, Busan 46241, Republic of Korea
*
Author to whom correspondence should be addressed.
Constr. Mater. 2024, 4(1), 72-90; https://doi.org/10.3390/constrmater4010005
Submission received: 27 November 2023 / Revised: 22 December 2023 / Accepted: 29 December 2023 / Published: 3 January 2024

Abstract

:
Concrete stands as the most widely used construction material globally due to its versatility, encompassing applications ranging from pavement, multifloor structures, and bridges to dams. However, these concrete structures endure structural stress and require close monitoring to prevent accidents and ensure sustainability throughout their complete life cycle. In recent years, artificial intelligence (AI) and computer vision (CV) have demonstrated considerable potential in diverse applications within construction engineering, including structural health monitoring (SHM) and inspection processes such as crack and damage detection, as well as rebar exposure. While it is undeniable that CV and deep learning models are transforming the construction industry by offering robust solutions for complex scenarios, there remain numerous challenges pertinent to their applications that require attention. This paper aims to systematically and critically review the literature of the past decade on the application of deep learning models in the construction industry for SHM purposes in concrete structures. The review delves into proposed methodologies and technologies while identifying opportunities and challenges associated with these applications in practice. Additionally, the paper provides insights to bridge the gap between theory and application.

1. Introduction

Concrete is the most important and demanding construction material [1], and concrete structures have been influencing the construction industry for decades [2]. However, an ever-growing number of concrete structures worldwide are entering the aging phase [3]. Due to various factors, such as weather and environmental conditions, chemical reactions, and external and internal stresses, concrete structures are often subject to defects such as cracks, efflorescence, spalling, bar exposure, etc., and fail to meet the expected life cycle, aging earlier than expected [2].
The idea of structural health monitoring (SHM) first emerged in the early 2000s. Although initially, the sole focus of SHM was to monitor concrete bridges, in present times, it is defined as the method of continuously evaluating and assessing the condition and performance of any concrete structures, such as buildings, bridges, dams, pipelines, and other infrastructure, throughout their operational lifespan [4,5,6]. The objective of SHM is to detect any damage, deterioration, or changes in structural properties that could potentially compromise the safety, functionality, or longevity of the concrete structure and is crucial in maintaining structures in optimal condition [2]. The traditional methods for the SHM process primarily involve manual inspection, which is heavily dependent on the expertise of the inspector. However, these methods present various challenges, including time-consuming operation, varying subjectivity, or difficulties in inspecting components at elevated heights in tunnels/road pavement in busy traffic conditions [2,7]. Therefore, there is a pressing need for an innovative and precise inspection approach to effectively monitor the health condition of structures that can overcome the mentioned limitations.
The advancement of artificial intelligence (AI) and computer vision (CV) has enabled their various applications in the construction industry, ranging from automatic surface crack detection to tunnel lining defect detection [8,9]. In recent years, there has been a notable progression in the application of AI methodologies, particularly machine learning and deep learning techniques, for activities such as data diagnostics, data interpretation, and feature extraction in the realm of SHM for infrastructure systems. Recent advances in deep learning technologies have allowed for state-of-the-art performance in visual recognition problems such as image classification, object detection, and object segmentation. Thus, such technologies hold significant promise, with an extensive array of practical applications in the SHM area [10].
Boccagna et al. [10] stated that the primary challenges in the application of deep-learning-based technologies for automated SHM processes pertain to the establishment of a resilient framework that encompasses all the intervening stages, from data collection to result generation and analysis. Therefore, the objective of this study is to conduct a comparative analysis of existing and proposed deep-learning-based models and convolutional neural network (CNN) techniques in the field of SHM. This comparison considers various factors, including the type of concrete structure, the type of input data, the size of the data; the type of damage, and the accuracy achieved by each proposed model, as well as their limitations in certain applications and suggestions for future research. The aim is for this study to serve as a foundational reference for the selection of deep-learning-based approaches in future SHM problem-solving scenarios. Particularly, it can guide researchers and practitioners in choosing appropriate deep learning methodologies when encountering situations similar to those reported in prior studies and can assist in identifying systems that have demonstrated notable performance in structural defect detection in those cases.
To accomplish these goals, only studies published after 2017 were meticulously arranged to incorporate the most up-to-date developments and cutting-edge technologies. The selection process adhered to specific criteria established by prior review articles in the field of construction engineering research that employed CV and deep learning techniques [11]:
  • The chosen studies were centered on deep-learning- and CV-based applications, irrespective of their outcomes, including aspects like visualization or quantification;
  • The studies were focused on the SHM domain, spanning various construction stages and varying physical and environmental conditions.
Studies utilizing alternative methodologies like machine learning techniques such as artificial neural networks (ANNs), genetic algorithms (GAs), or support vector machines (SVMs) [12], as well as those emphasizing sensor-based approaches [13], were not included in the selection process. Figure 1 shows an outline of the research methodology. This study focuses solely on SHM application to concrete structures; hence, steel structures and other types are not considered. The chosen studies underwent a systematic analysis based on their methodologies and technologies, as well as their respective applications. Subsequently, a more in-depth examination identified those studies that warranted further exploration, guided by their methodological frameworks and specific areas of application.

2. Overview of Artificial Intelligence and Deep Learning

In 1955, one of the pioneers of AI, John McCarthy, defined AI as a means to develop machines that behave intelligently [14]. In present times, artificial intelligence is characterized as the “exploration and creation of intelligent agents” [1]. AI empowers computers to achieve human-like capabilities such as perception, knowledge representation, reasoning, problem solving, and planning. This enables them to address intricate and ambiguous problems in a purposeful, intelligent, and adaptable manner.
Investment in AI is experiencing swift expansion, with branches like machine learning and deep learning notably playing a significant role [15]. According to a study by Purdy and Daugherty [16], AI is in the process of reshaping all aspects of society, and this transformation might increase employee productivity by 40% and can increase annual economic growth rate by two times by the year 2035.
Although AI still lacks appropriate applications in the construction industry compared to other sectors [15], researchers have been attempting to implement different AI technologies in different areas of construction, such as cost prediction in the real estate business [17], SHM at construction sites [18], and supply chain management [19]. Chu et al. [20] carried out studies to investigate the applications of robotics in the construction industry. Chen et al. [21] proposed a fuzzy cognitive map model that can assess risk casualties and predict risk for construction projects. Zaira and Hadikusumo [22] adopted a knowledge-based AI approach and conducted structural equation modeling (SEM) to identify the factors with the most significant impact in terms of improving workers’ safety behavior. Artificial neural networks, and other machine learning approaches such as random forest, SVM, etc., are also popular AI techniques in the construction industry [23].
CV-based AI applications were initially constrained in the construction domain due to the complexity of data acquisition process and the requirement for computers with high levels of computing power. Nevertheless, as the prevalence of such devices and high-performance computing resources has grown, CV technologies have find applications in various construction domains, such as labor and asset monitoring, progress assessment, quality control and assurance, etc. [24]. Advancements in CNN architectures are currently revolutionizing the CV field. Convolutional neural networks (CNNs), which represent an essential part of deep learning techniques, are widely renowned for their excellence in top-tier CV projects.

History and Development of Deep Learning

Convolutional neural networks (CNNs) are advanced feedforward neural networks. A feedforward neural network operates by transmitting information from the input layer to the output layer in a unidirectional manner, without forming cyclic connections among its neurons [25,26]. When an image is introduced as input to the network, it traverses a sequence of convolutional and pooling layers before reaching the final stage of fully connected layers, usually concluding with a SoftMax layer, which yields the output [25] (Figure 1).
The concept of convolutional neural networks (CNNs) first came to light in the late 1990s, thanks to the work of LeCun et al. [27]. As noted by Sultana et al. [28], the success of LeNet-5 sparked researchers’ interest in exploring the vast potential of CNNs, particularly in tasks that involve pixel-level operations like detection and segmentation. Object classification by CNNs has found applications in categorizing various elements at construction sites, including workers, risk zones, and machinery, as well as safety measures or accident risk [29,30]. These classifications are typically conducted using images collected from sources like CCTV footage, unmanned aerial vehicles (UAVs), mobile robots, etc. [30,31,32]. Classifying workers with or without personal protective equipment is another popular application of CNNs in the construction engineering area. Such classification is achieved using histogram of oriented gradients (HOG) features extracted from hardhats, which efficiently describe detailed shape information [33].
CNNs excel over traditional machine learning techniques due to their adaptive nature, learning feature representations through end-to-end training, thereby eliminating the necessity for hand-engineered features and expert knowledge [34]. Additionally, deep learning models can fine tune themselves, making deep learning algorithms and CNNs popular in various sectors of computer vision, including image segmentation [35,36].
Image segmentation, a fundamental technique in computer vision, involves the partitioning a digital image into distinct sections based on various pixel characteristics [37]. Given the critical role of spatial information in semantically segmenting different regions of an image, image segmentation is typically considered a detail-level or pixel-level vision task, in contrast to tasks like classification and object recognition [28]. The primary objective of image segmentation is to extract specific information for further analysis. This process involves separating specific regions in an image in such a way that all pixels within each area share common attributes such as color, intensity, texture, etc. [38,39]. Chi and Caldas [40] were among the first to apply object segmentation in identifying equipment and workers at construction sites using a background subtraction algorithm. Recently, segmentation techniques were applied to the detection of workers around a fall pit in an attempt to reduce fall-induced hazards among workers, as well as to the detection of risk zones such as elevated edges or roofs [41,42].
In practice, there are three primary types of segmentation, as depicted in Figure 2:
  • Semantic segmentation: This involves labeling image pixels with object categories [43];
  • Instance segmentation: Here, the goal is to separate and identify all instances of objects within an image individually and independently [44,45];
  • Panoptic segmentation (PS): Panoptic segmentation, as defined by Kirillov et al. [46], combines both semantic and instance segmentation, offering a comprehensive view of the scene by identifying object categories and individual instances.
The applications of convolutional neural networks (CNNs) in image segmentation are incredibly diverse. Farabet et al. [47] introduced a multiscale CNN designed for scene-labeling tasks. They trained and tested their model on three distinct datasets: Sift flow [48], the Barcelona dataset [49], and the Stanford background dataset [50]. Girshick et al. [51] proposed region-based CNN (R-CNN) in the PASCAL VOC semantic segmentation challenge, which was part of the Pascal Visual Object Classes Challenge in 2007. This approach employs region proposals in a CNN to localize and segment individual objects.
The concept of a fully convolutional network (FCN) was introduced by Long et al. [52] to address semantic segmentation tasks. For the generation of region proposals, Faster R-CNN utilizes a CNN-based region proposal network (RPN) that employs bounding boxes—a technique shared with instance segmentation models. Mask R-CNN, introduced by He et al. [53], closely resembles Faster R-CNN but includes a binary mask prediction branch to facilitate instance segmentation in addition to object detection. Huang et al. [54] presented Mask Scoring R-CNN, which enhances Mask R-CNN by incorporating a network block to improve the qualitative aspect of predicted masks. In later development, Kirillov et al. [46] introduced a cutting-edge instance segmentation model that utilizes point-based rendering within the framework of Mask R-CNN.
The application of such instance segmentation models has promising implications in construction science research; for example, Teizer and Vela [55] proposed a segmentation-based tracking system for workers at construction sites. Wang et al. [56] employed a two-level approach that combines Faster R-CNN and Mask R-CNN to identify the morphological aspects of damage. This includes recognizing features such as damage topology, area, and ratio.

3. SHM System Based on Deep Learning Models

Researchers in the construction engineering field have recognized the immense potential and innovative technological strides resulting from the utilization of deep learning methods [57,58]. Consequently, numerous initiatives have been undertaken to apply deep learning techniques to structural health monitoring (SHM) of concrete infrastructure [59]. In this section, we delve into deep-learning-based research in the SHM domain, with a specific focus on two facets: (1) damage identification and (2) concrete condition assessment.

3.1. Damage Identification

At the heart of any SHM system lies its capacity to conduct damage identification. Damage refers to alterations in a material’s physical characteristics caused by ongoing deterioration or a singular event affecting a structure. Such changes have the potential to compromise the performance and structural integrity of the concrete [60]. One limitation of applying deep learning techniques is that they require a large and annotated database, which is not always available, especially in the concrete research area. However, the application of transfer learning can eliminate this problem, allowing an existing deep learning model to be retrained with smaller amounts of new data [61]; accordingly, an increasing number of applications of deep learning models in concrete research and SHM have been reported.
For example, Gopalakrishnan et al. [62] applied transfer learning to a pretrained VGG-16 model for crack detection in hot-mix asphalt and Portland cement concrete-based pavement. Kolar et al. [63] also applied transfer learning to VGG-16 model to detect safety guardrails to promote on-site safety inspection.
Real-world scenarios often limit the applications of deep learning models at actual construction sites because of lighting and shadow issues. Cha et al. [64] trained a CNN with a large database of 40k images under various lighting conditions and achieved 98% accuracy in detecting concrete cracks. The authors later compared the performance of the proposed CNN using Canny and Sobel edge detection methods. Tong et al. [65] integrated three CNNs to perform recognition, localization, and feature extraction tasks, enabling the 3D reconstruction of hidden pavement cracks with images of cracks collected using ground-penetrating radar (GPR). Figure 3 demonstrates the proposed pipeline of 3D reconstruction pavement cracks with GPR data. Gibert et al. [66] combined multiple detectors for automatic inspection of railway tracks.
Assessment of post-disaster damage in concrete to provide valuable insights for necessary follow-up actions is another application of deep learning in the SHM area. Davoudi et al. [67] applied image segmentation to determine the state of the damage in reinforced concrete beams and slabs. Lattanzi et al. [68] also applied image segmentation via the MATLAB Image Processing Toolbox for the extraction of features from images of damaged reinforcement columns to estimate the maximum lateral displacement using a regression model. Spalling, a common type of damage in concrete structures, is another area of application of deep learning in SHM practice. For example, Dawood et al. [69] presented a hybrid model combining image processing and machine learning techniques to identify spalling distress in subway stations. Yeum et al. [70] applied AlexNet to both collapse classification and spalling detection in post-disaster analysis of concrete structures.
Kim and Cho [71] introduced a method utilizing unmanned aerial vehicles (UAVs) and R-CNN to detect cracks in old concrete bridges. They applied transfer learning to R-CNN, using crack images to enhance crack detection, and later, image processing was employed to quantify the identified cracks. Kang and Cha [72] also applied UAV-based damage detection with deep learning; however, they addressed one important issue, which is that UAVs often require a skilled pilot and autonomous flight with GPS in certain complex locations of structures, such as indoors or beneath bridges. The authors proposed an ultrasonic beacon for UAV navigation in GPS-incompatible environments.
Xue and Li [73] devised a three-tiered deep learning framework including an FCN, RPN, and position-sensitive region of interest pooling for identification of damage in tunnel linings. Hoang et al. [74] also compared the performance of a CNN with Sobel and Canny edge detection algorithms, as previously reported by Cha et al. [64], for a cyclic survey of pavement cracks. Similarly, Dorafshan et al. [75] compared the performance of four edge detection methods with CNNs in detail for crack detection in concrete. Four common edge detection methods in the spatial domain (Roberts, Prewitt, Sobel, and Laplacian of Gaussian) and two in the frequency domain (Butterworth and Gaussian), as well as the AlexNet model in three modes of training (trained, transfer learning, and without training), were compared, and the authors concluded that AlexNet showed superiority over other methods.
AlexNet was used by Wang et al. [76] as well. The authors utilized both AlexNet and GoogLeNet for the detection of various types of damage to masonry walls in historic structures, using sliding-window techniques to pinpoint concrete damage. Motivated by the ImageNet Challenge, Gao and Mosalam [77] proposed the concept of Structural ImageNet, with four intended tasks: component identification, spalling detection, damage condition evaluation, and damage type determination in concrete through the application of transfer learning in VGGNet (Visual Geometry Group).
Wu et al. [78] applied transfer learning to VGG16 and ResNet18 to detect two types of prevalent concrete surface defects, namely cracks and corrosion. Zhang et al. [79] proposed Faster R-CNN to determine the spatiotemporal information of the vehicles on bridges in order to determine the stress state and traffic densities. Wang and Cheng [80] proposed DilaSeg-CRF by integrating a CNN with a dense conditional random field (CRF) to improve the segmentation accuracy in sewer pipe defect detection, whereas Li et al. [81] addressed the issue of data imbalance in sewer damage detection by introducing a hierarchical classification approach to supervise the learning process at different levels. Zha et al. [82] applied transfer learning to ResNet (deep residual neural network) for eight types post-disaster concrete damage detection: scenario classification, damage detection, spalling detection, material identification, collapse detection, effected component identification, and damage level and type determination, which were categorized into binary or multiclasses according to the conditions. The authors used the 2018 PEER Hub ImageNet Challenge distributed by the Pacific Earthquake Engineering Research Center to evaluate the proposed methodology.
Jang et al. [83] used transfer learning in GoogLeNet with hybrid images, combining vision and infrared thermography images to enhance crack detection in concrete structures. The authors suggested the use of a UAV-mounted hybrid system comprising a vision camera, an infrared camera, and a continuous-wave line laser to capture images, particularly for large-scale structures, then used them for inspection of the respective structures. U-Net, which is famous for applications in biomedical image segmentation, was first applied by Liu et al. [84] to concrete crack detection and later compared with FCN using evaluation metrics such as precision and the size of the training set. The authors applied U-Net for localization of concrete cracks under various lighting and background conditions. Khani et al. [85] investigated the impact of preprocessing on a concrete crack detection pipeline based on a CNN trained with 700 labelled gas turbine images. The authors concluded that bilateral filtering improves the generalization ability of the suggested framework in cases with cracks on complex structures. Zhang et al. [86] argued that two-stage detectors such as Faster R-CNN and ResNet-101 have limited practical applications due to their slow speeds. The authors used a single-stage detector (SSD), YOLOv3 (You Only Look Once), to detect multiple types of concrete bridge damage, such as cracks, pop-outs, spalling, exposed rebar, etc.
Liu et al. [87] argued that the motion blur from excessive vibration in UAVs limits the accuracy of crack detection in high-rise buildings. The authors introduced a generative adversarial network (GAN) that incorporates the concept of localized skip connections that recognize the correlation between blurred and sharpened crack images. The proposed method was validated through experiments involving the investigation of skip connections in deblurring and compared with a state-of-the-art deblurring model. Kim et al. [88] applied transfer learning to Mask R-CNN for automatic concrete damage detection and localization in four classes—cracks, efflorescence, rebar exposure, and spalling—using an instance segmentation approach.
Mondol et al. [89] applied Faster R-CNN to detect post-disaster damage like surface cracks, exposed rebar, and buckled rebar using image data collected from concrete structures damaged during past earthquakes in Nepal (2015), Taiwan (2016), Ecuador (2016), Erzincan (1992), Duzce (1999), Bingol (2003), Peru (2007), Wenchuan (2008), and Haiti (2010). Deng et al. [90] introduced LinkASSPNet (LinkNet with atrous spatial pyramid pooling) and conducted a performance comparison with U-Net and LinkNet in the context of concrete bridge surface damage detection. Notably, this study stands out, as the models were trained on a relatively small dataset. It purports to address the challenge of variations in labeling areas among labelers in pixel-wise image segmentation tasks.
Zheng and Zhang [91] proposed a crack detection model for concrete based on image segmentation tasks and the FCN, R-CNN, and RFCN (Richer Fully Convolutional Networks) models. The training included a wide range of image data, including images of buildings, bridges, dams, roads, etc. Karaaslan et al. [92] proposed a combination of an SSD-based VGG-16 model and a modified SegNet, where the former detects regions of interest related to damage, such as cracks or spalling, upon verification by the respective inspector, and the latter then applies segmentation to the damage for further analysis. Miao et al. [93] proposed U-Net-based Damage-Net for semantic segmentation of seismic damage in reinforced concrete structures, where the authors adjusted the padding size and stride size to ensure that the input and output size were the same, which is usually not the case in U-Net. The proposed Damage-Net receives its encoder from the convolutional layers of VGG-16, allowing it to adapt transfer learning and to be trained on a comparatively smaller dataset. Based on this architecture, two individual models were proposed: Crack-Net for detecting cracks, and 4Category-Net for identifying four additional damage categories, namely concrete spalling and crushing, reinforcement exposure, buckling, and fracture.
Qiao et al. [94] proposed EMA-DenseNet, a combination of densely connected convolutional networks (DenseNet) integrated with an expected maximum attention (EMA) module in the last pooling layer for the detection of surface damage in concrete bridges in a set of images collected from multiple bridges located in Zhejiang (China). The authors claimed that the proposed model performs better than FCN, SegNet, DeepLab v3+, and SDDNet. Huang et al. [95] proposed a software system for damage detection in subway tunnels by integrating four separate functions: image fusion to splice the images acquired by different cameras, image preprocessing to remove background noise and other preprocessing tasks, damage identification performed by the R-CNN model and a data platform for evaluation by the respective personnel. Arya et al. [96] proposed a concrete pavement damage dataset consisting of 26,620 data point from multiple countries and investigated how the demographics of the damage data affect the model performance based on a YOLO-v5/YOLO-v4/cascade R-CNN-based ensemble model. Cui et al. [97] proposed an improved YOLO-v3 model for the detection of erosion damage that achieved up to a 75% mean average precision value.
Pozzer et al. [98] compared the performance of different models, i.e., VGG-16, ResNet-18, ResNet-50, MobileNet-V2, Xception, etc., in detecting concrete defects such as delamination, cracks, spalling, and patches in thermographic and regular images at varying distances and under varying conditions using semantic segmentation. Andrushia et al. [99] implied that most research on damage detection in concrete structures does not consider the complex background or environmental effects and therefore proposed a U-Net with an encoder–decoder framework for thermal damage detection in concrete structures in the event of fires.
Munawar et al. [100] introduced a cycle generative adversarial network (CycleGAN) with 16 convolution layers, providing additional support to refine predictions through guided filtering (GF) and conditional random fields (CRFs). The authors applied this model to inspect mid- to high-rise concrete structures constructed during the 2000s using segmentation techniques and drones. Zou et al. [101] proposed a YOLOv4-based approach to the detection of multiple types of damage, including both fine and wide cracks, spalling, exposed and bucking rebars, etc., that was integrated in a graphical user interface (GUI) to streamline the assessment of structural damage in reinforced concrete (RC) buildings following an earthquake. Han et al. [102] proposed the use of a transfer-learning-based AlexNet and threshold segmentation to precisely locate cracks in concrete structures.
Tanveer et al. [103] compared and analyzed the performance of five semantic segmentation models (ENet, CGNet, ESNet, DDRNet-Slim23, and DeepLabV3+ (ResNet-50)). These models were categorized as lightweight and heavyweight based on the parameter count. The evaluation focused on on-site damage detection in concrete structures using edge computing devices such as smartphones, tablets, etc. Bai et al. [104] proposed an EfficientNet-V2-based model for component damage recognition, serving both structural health monitoring (SHM) and post-disaster assessment purposes. They also investigated the relationship between damage type, component damage level, and the structural safety state. Crognale [105] compared four different image processing techniques, namely Otsu-method thresholding, Markov random field segmentation, the RGB color detection technique, and the K-means clustering algorithm, in corrosion and crack detection based on a case study. Chen et al. [106] proposed an AlexNet-based multiclass damage detection method for reinforced concrete bridges in high-speed rail systems.
Wan et al. [107] proposed a BR-DETR model, a concrete bridge damage detection model based on detection transformers (DETR), with deformable Conv2D in place of convolution, as well as with an additional convolutional project attention layer after the self-attention layer. Zhu and Tang [108] introduced a DeepLabV3+ network architecture with Xception as the backbone to automatically estimate detailed crack information in hydraulic concrete structures. Huang et al. [109] proposed a Faster R-CNN with Res-Net101 as the backbone for detection of damage like cracks, spalling, and precipitates in hydraulic concrete structures.

3.2. Damage Quantification

Damage quantification is the next step after damage identification. Concrete damage quantification aims to determine the extent, severity, and specific characteristics of damage, such as cracks, spalling, corrosion, or other forms of deterioration. Although using deep learning for concrete damage quantification is still a relatively new concept in SHM, researchers are continuously generating new ideas to automate the quantification process, given the inherent challenges associated with this topic.
Kim et al. [110] proposed a UAV-based digital image processing system integrated with imaging and distance-sensing technology to determine the width and length of the cracks in concrete surfaces. Tong et al. [111] proposed a CNN-based method to calculate the mean texture depth (MTD) of pavement surfaces from 3D scan data, which was tested on four different highways in Shanxi, China. Huang et al. [112] studied lining damage in tunnels with a rapid detection and assessment analysis system developed by Nanjing HuoYang Hou Mdt InfoTech Ltd. The system includes a multichannel array of high-speed CCD (charged couple device) cameras to obtaining image data, multiple sensors to mitigate the impact of vehicle vibration on the tunnel, a multilayer lighting system, multiple positioning technology (reference object positioning technology + image positioning technology + mileage positioning technology + infrared laser positioning technology), and a computer vision approach for damage identification and analysis. Tayo et al. [113] presented a device capable of portable crack width calculation in concrete road pavement using pattern recognition based on multiple image processing technologies, such as graying, enhancement, filtering and denoising, binarization, segmentation, etc. Kim and Cho [114] proposed Mask R-CNN+image processing techniques for successful detection and quantification of concrete cracks with widths of 0.3 mm or more. Wei et al. [115] applied the same approach to concrete surface bughole segmentation and diameter measurement.
Beckman [116] applied Faster R-CNN to automatically and simultaneously detect and quantify concrete spalling in multiple locations within the same surface. The authors used a depth camera to obtain the volume quantifications of the spalling damage. Park et al. [117] applied YOLO for both concrete crack detection and quantification (i.e., to determine the size of the cracks) in real time. The authors used laser beams with integrated distance sensors for accurate measurement of the crack size. Bhowmick et al. [118] applied U-Net-based segmentation for concrete crack localization and binarization to estimate quantitating properties of cracks, like length, width, area, orientation, etc., from video data collected by a camera mounted on a UAV. Flah et al. [119] applied a deep learning technique to identify both structural and durability-related damage in structural members and assess the condition in a short time span by combing a Keras classifier with Otsu image processing. The proposed method can classify cracks; quantify them in terms of length, width, and angular orientation; and evaluate the severity of the damage.
Yuan et al. [120] proposed an inspection robot that transforms the quantification of concrete damage from a 2D plane to 3D space with stereo vision and a Mask R-CNN approach. The robot is based on four different sensors, with a monocular camera as a visual sensor, a stereo camera with a sensor for inertial measurement (IMU) of six degrees of freedom that can be mapped for panoramic image stitching, and a LiDAR sensor to measure the distance between the RC structure and the camera. Miao et al. [121] proposed a GoogLeNet-based transfer learning approach incorporating a novel sliding technique known as neighborhood scanning. This method aims at the detection, segmentation, and quantification of concrete cracks, achieving an average relative error of 14.58% in crack calculation.
Song et al. [122] introduced a deep learning approach for crack segmentation and quantification utilizing MobileNetV1 and ResNet50, along with DeeplabV3+ and U-Net. MobileNetV1 and ResNet50 handle crack classification, while DeeplabV3+ and U-Net manage panoramic crack segmentation (Figure 4). The quantitative information of the crack was subsequently acquired by multiplying the actual physical size corresponding to the unit pixel, assuming the length of a single pixel as the unit length. Kumarapu et al. [123] introduced UAVIC, a system that integrates UAVs with an image processing technique, i.e., digital image correlation. This approach is employed for damage quantification on scaled bridge girders.
Bae et al. [124] proposed a computer-vision-based crack quantification algorithm using decision making based on statistical methods for accurate estimation and quantification of damage based on an image dataset of concrete building structures in South Korea. Li et al. [125] proposed a ResNet50-based improved You Only Look At CoefficienTs for Edge devices (YolactEdge) combined with digital image processing techniques for damage identification and quantification in hydraulic tunnels.

4. Strengths, Weaknesses, Opportunities, and Threats (SWOT) Analysis

SWOT analysis is a strategic tool utilized to assess the strengths, weaknesses, opportunities, and threats within a system or situation [126]. To facilitate future planning, decision making, and strategic development in construction engineering, a SWOT analysis was conducted to evaluate the deep-learning-based SHM systems discussed in this study.
  • Strengths: The implementation of deep learning models can assist in improving the SHM process by systematically predicting patterns and anomalies in image data [127]. This results in more accurate damage identification in concrete and concrete structures. One advantage of deep learning is its integration capability with multiple other systems, such as sensors (LiDAR and IMU); UAVs or drones; and depth/stereo or infrared camera, which, in turn, aids in the quantification process. Combining deep learning with computer vision applications can not only eliminate the unsafe and lengthy manual inspection process but also enable the automation of the entire SHM system in real time [128].
  • Weaknesses: The first weakness in the implementation of deep learning models is their requirement for large-scale annotated data. Obtaining high-quality labeled data, specifically related to concrete health conditions, remains a persistent challenge [61]. Another potential limitation is the requirement for extensive knowledge. Those aiming to implement a deep-learning-based SHM process must be familiar with both deep learning and structural engineering, posing an additional challenge.
  • Opportunities: An SHM system based on deep learning models offers early damage identification and real-time continuous monitoring [2]. When integrated with an alarm system, it can promptly notify authorities during the early stages of damage. This enables rapid action, preventing the escalation of severity and reducing additional costs related to maintenance and damage repair. The field of construction engineering also often struggles with complexities in data, and deep learning has proven to be an effective solution to address these challenges [62].
  • Threats: While deep learning applications can undoubtedly aid in the SHM process, the reliability of results becomes questionable without proper validation or practical testing of the trained models. Additionally, a deep-learning-based system should undergo regular updates with new data or guidelines to effectively tackle emerging challenges [61].
According to the SWOT analysis, the application of deep learning in SHM systems offers a vast array of opportunities; however, further research and studies are required to understand its limitations and threats. The next section describes some practical recommendations regarding training data requirements, model reliability, and other issues found in the studies published to date.

5. Discussion and Suggested Frameworks for the Future

The number of deep-learning-based applications in concrete research is rapidly growing, especially in the SHM area. Numerous applications have been reported with respect to both concrete damage identification and quantification. However, based on the abovementioned trends, the number of automatic concrete damage quantification studies in the SHM area was comparatively less before 2019. The application and integration of stereo cameras and sensors, such as LiDAR and laser sensors, have made deep learning applications for damage quantification. Many researchers have applied various image processing techniques rather than integration with depth cameras or sensors. However, concrete cracks are very fine, so whichever system is adopted must precisely quantify a particular property (either length, width, or diameter). AlexNet, GoogleNet, Faster R-CNN, Mask R-CNN, U-Net, VGG, and YOLO models seem to be popular choices for damage identification. However, compared to other industries, construction falls behind in terms of adopting digitalization; therefore the application of deep-learning- and vision-based systems to monitor concrete health in the SHM area is still not sufficient in real practice, mostly due to the following issues:
  • Data shortage: Although transfer learning has made the adaptation of deep learning easier, there is still a lack of publicly available datasets in the construction domain. Raw data often need to go through many stages of post processing, which is very time-consuming and labor-intensive. Also, there is a need for annotated datasets, which are essential for any deep learning training [129]. Most studies have been conducted using private datasets; making such datasets public would open multiple doors for researchers in the SHM domain for multiple applications. Although data augmentation plays an important role in dataset incrementation, applying various transformations to existing data, such as rotating, scaling, flipping, or cropping images, is insufficient for research in the SHM area. An alternative method involves utilizing generative adversarial networks (GANs), where a deep learning model comprising two distinct networks (namely a generator and a discriminator) is employed to generate synthetic image data instead of relying on real-world camera inputs only, as reported in [87,100]. Deng et al. [130] implied that GANs trained on synthetic data often perform well in real-world scenarios.
  • Impact of the training data on overfitting: Transfer learning has undeniably simplified the application of deep learning models in structural health monitoring (SHM). However, the persistent challenge of overfitting can arise, particularly in instances where there is a paucity of image data. Deep learning models characterized by multiple layers and millions of parameters demand extensive tuning, as illustrated, for example, by the necessity of adjusting at least 100 million parameters in VGG-16 for crack detection [61]. The insufficiency of training data, both in terms of quantity and quality, poses a significant obstacle, rendering a model incapable of performing effectively in real-world applications. It is imperative that the training data encompass diverse real-world scenarios, accounting for variations in background, lighting, and weather conditions, to ensure the model’s robustness and applicability.
  • Requirement for high-performance computers: Many deep learning techniques necessitate several days for training due to the extensive calculations involved in computing related training parameters, such as loss functions. Adequate hardware, including high-capacity hard disks, multiple GPUs/CPUs, and substantial memory, is essential for storing these calculations. Researchers should prioritize discovering optimized model structures with fewer parameters, facilitating their seamless adaptation in structural health monitoring (SHM) applications. An attempt to address this concern was made by Zang et al. [86] with an SSD-based model.
  • Dealing with background noise: On the other hand, in addressing various background noises in images, researchers have implemented different morphological changes in the CNN architecture [80,87,90,93,94,100,107] to increase the detection accuracy. However, the source code is typically not publicly available. Researchers should be encouraged to make their source code publicly accessible, enabling other researchers to enhance the architecture further and, consequently, increase its applicability in actual practice. Due to the image resizing requirement of deep learning models to be trained on computers with average computing capacities, generalization abilities are often lost. For example, stains are a common issue in concrete structures and often incorrectly identified as cracks. To solve this issue, stains and similar defects could be categorized as another class [131] to improve the generalization abilities.
Despite the challenges and limitations, the use of deep learning and computer vision technologies holds significant promise in structural health monitoring (SHM) and concrete research. Integrating deep learning applications into smartphones or tablets for on-site inspections, as well as utilizing UAV-based approaches for the inspection of high-rise buildings and long-span bridges, not only facilitates the whole inspections process but also saves time and effort while promoting workplace safety. Additionally, the integration of big data and data mining technologies with cloud computing can enhance data management. A collaborative effort from researchers, scholars, and engineers in the construction, computer science, and civil engineering domains can establish more effective deep -earning-based SHM inspection systems for both damage identification and quantification, regardless of severity, delicacy of the damage, and the influence of the surrounding environment.

6. Conclusions

The aim of this research was to conduct a systematic review of the utilization of deep learning in the identification and quantification of concrete damage for SHM purposes. This study delved into the concepts and historical development of artificial intelligence (AI), computer vision (CV), and deep learning. With the aim of including the latest advancements in concrete research, the analysis was focused on studies spanning from 2017 to 2023, particularly those addressing vision-based crack identification, categorization, and measurement analysis. Our comprehensive discussion of the applications, purposes, and limitations of deep-learning-based SHM research yielded the following key points:
  • Although deep-learning-based damage identification is subject to multiple challenges regarding data acquisition, processing, training, and testing issues, it has demonstrated significant promise. The requirement for specific dataset preparation and strategic approaches during training could help the researchers overcome overfitting issues encountered as a result of limited resources.
  • The integrations of deep learning in concrete damage quantification research is challenging due to the fact it can only provide pixel-based measurement, not an actual measurement. While unit conversion and image processing techniques can be applied to smaller cracks, large cracks may require depth or stereo cameras and remote sensing systems. However, most available depth cameras on the market have a short range (>10 m), so efforts should be made to develop longer-range depth cameras.
In this study, we addressed four critical issues related to the application of deep learning to concrete damage identification and quantification and suggested potential frameworks to deal with these issues. This research provides helpful insights that can aid in future applications and studies regarding deep learning in the SHM area.

Author Contributions

Conceptualization, A.M.C. and R.K.; methodology, A.M.C.; software, A.M.C.; validation, A.M.C. and R.K.; formal analysis, A.M.C. and R.K.; investigation, A.M.C.; resources, A.M.C. and R.K.; data curation, A.M.C. and R.K.; writing—original draft preparation, A.M.C.; writing—review and editing, R.K.; visualization, A.M.C. and R.K.; supervision, R.K. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Data Availability Statement

No new data were created or analyzed in this study. Data sharing is not applicable to this article.

Acknowledgments

This research was conducted independently by the authors.

Conflicts of Interest

The authors declare no conflicts of interest.

References

  1. Moein, M.M.; Saradar, A.; Rahmati, K.; Mousavinejad, S.H.G.; Bristow, J.; Aramali, V.; Karakouzian, M. Predictive models for concrete properties using machine learning and deep learning approaches: A review. J. Build. Eng. 2023, 63, 105444. [Google Scholar] [CrossRef]
  2. Wang, W.; Su, C.; Fu, D. Automatic detection of defects in concrete structures based on deep learning. Structures 2022, 43, 192–199. [Google Scholar] [CrossRef]
  3. Pu, R.; Ren, G.; Li, H.; Jiang, W.; Zhang, J.; Qin, H. Autonomous Concrete Crack Semantic Segmentation Using Deep Fully Convolutional Encoder–Decoder Network in Concrete Structures Inspection. Buildings 2022, 12, 2019. [Google Scholar] [CrossRef]
  4. Shibu, M.; Kumar, K.P.; Pillai, V.J.; Murthy, H.; Chandra, S. Structural health monitoring using AI and ML based multimodal sensors data. Meas. Sens. 2023, 27, 100762. [Google Scholar] [CrossRef]
  5. Yuan, F.G.; Zargar, S.A.; Chen, Q.; Wang, S. Machine learning for structural health monitoring: Challenges and opportunities. Sens. Smart Struct. Technol. Civ. Mech. Aerosp. Syst. 2020, 11379, 1137903. [Google Scholar]
  6. Yoon, J.; Lee, J.; Kim, G.; Ryu, S.; Park, J. Deep neural network-based structural health monitoring technique for real-time crack detection and localization using strain gauge sensors. Sci. Rep. 2022, 12, 20204. [Google Scholar] [CrossRef] [PubMed]
  7. Zhou, Z.; Yan, L.; Zhang, J.; Zheng, Y.; Gong, C.; Yang, H.; Deng, E. Automatic segmentation of tunnel lining defects based on multiscale attention and context information enhancement. Constr. Build. Mater. 2023, 387, 131621. [Google Scholar] [CrossRef]
  8. Mundt, M.; Majumder, S.; Murali, S.; Panetsos, P.; Ramesh, V. Meta-learning convolutional neural architectures for multi-target concrete defect classification with the concrete defect bridge image dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 16–17 June 2019; pp. 11196–11205. [Google Scholar]
  9. Lei, M.F.; Zhang, Y.B.; Deng, E.; Ni, Y.Q.; Xiao, Y.Z.; Zhang, Y.; Zhang, J.J. Intelligent recognition of joints and fissures in tunnel faces using an improved mask region-based convolutional neural network algorithm. Comput.-Aided Civ. Infrastruct. Eng. 2023, 1–20. [Google Scholar] [CrossRef]
  10. Boccagna, R.; Bottini, M.; Petracca, M.; Amelio, A.; Camata, G. Unsupervised Deep Learning for Structural Health Monitoring. Big Data Cogn. Comput. 2023, 7, 99. [Google Scholar] [CrossRef]
  11. Reja, V.K.; Varghese, K.; Ha, Q.P. Computer vision-based construction progress monitoring. Autom. Constr. 2022, 138, 104245. [Google Scholar] [CrossRef]
  12. Gui, G.; Pan, H.; Lin, Z.; Li, Y.; Yuan, Z. Data-driven support vector machine with optimization techniques for structural health monitoring and damage detection. KSCE J. Civ. Eng. 2017, 21, 523–534. [Google Scholar] [CrossRef]
  13. Jeong, S.; Zhang, Y.; O’Connor, S.; Lynch, J.P.; Sohn, H.; Law, K.H. A NoSQL data management infrastructure for bridge monitoring. Smart Struct. Syst. 2016, 17, 669–690. [Google Scholar] [CrossRef]
  14. Ertel, W. Introduction to Artificial Intelligence, 1st ed.; Springer: Weingarten, Germany, 2018; pp. 1–2. [Google Scholar]
  15. Pan, Y.; Zhang, L. Roles of artificial intelligence in construction engineering and management: A critical review and future trends. Autom. Constr. 2021, 122, 103517. [Google Scholar] [CrossRef]
  16. Purdy, M.; Daugherty, P. Why Artificial Intelligence is the Future of Growth. 2016. Available online: www.accenture.com/t20170524T055435__w__/ca-en/_acnmedia/PDF-52/Accenture-Why-AI-isthe-Future-of-Growth.pdf (accessed on 29 October 2023).
  17. Khobragade, A.N.; Maheswari, N.; Sivagami, M. Analyzing the housing rate in a real estate informative system: A prediction analysis. Int. J. Civil Engin. Technol. 2018, 9, 1156–1164. [Google Scholar]
  18. Poh, C.Q.; Ubeynarayana, C.U.; Goh, Y.M. Safety leading indicators for construction sites: A machine learning approach. Autom. Constr. 2018, 93, 375–386. [Google Scholar] [CrossRef]
  19. Chen, C.; Huy, D.T.; Tiong, L.K.; Chen, I.M.; Cai, Y. Optimal facility layout planning for AGV-based modular prefabricated manufacturing system. Autom. Constr. 2019, 98, 310–321. [Google Scholar] [CrossRef]
  20. Chu, B.; Jung, K.; Han, C.S.; Hong, D. A survey of climbing robots: Locomotion and adhesion. Int. J. Precis. Eng. Manuf. 2010, 11, 633–647. [Google Scholar] [CrossRef]
  21. Chen, H.; Zhang, L.; Wu, X. Performance risk assessment in public–private partnership projects based on adaptive fuzzy cognitive map. Appl. Soft Comput. 2020, 93, 106413. [Google Scholar] [CrossRef]
  22. Zaira, M.M.; Hadikusumo, B.H. Structural equation model of integrated safety intervention practices affecting the safety behaviour of workers in the construction industry. Saf. Sci. 2017, 98, 124–135. [Google Scholar] [CrossRef]
  23. Han, T.; Jiang, D.; Zhao, Q.; Wang, L.; Yin, K. Comparison of random forest, artificial neural networks and support vector machine for intelligent diagnosis of rotating machinery. Trans. Inst. Meas. Control. 2018, 40, 2681–2693. [Google Scholar] [CrossRef]
  24. Paneru, S.; Jeelani, I. Computer vision applications in construction: Current state, opportunities & challenges. Autom. Constr. 2021, 132, 103940. [Google Scholar]
  25. Rawat, W.; Wang, Z. Deep convolutional neural networks for image classification: A comprehensive review. Neural Comput. 2017, 29, 2352–2449. [Google Scholar] [CrossRef]
  26. Habibi, A.H.; Heravi, E.J. Convolutional Neural Networks. In Guide to Convolutional Neural Networks; Springer: Cham, Switzerland, 2017. [Google Scholar]
  27. LeCun, Y.; Boser, B.; Denker, J.S.; Henderson, D.; Howard, R.E.; Hubbard, W.; Jackel, L.D. Back-propagation applied to handwritten zip code recognition. Neural Comput. 1989, 1, 541–551. [Google Scholar] [CrossRef]
  28. Sultana, F.; Sufian, A.; Dutta, P. Evolution of image segmentation using deep convolutional neural network: A survey. Knowl.-Based Syst. 2020, 201, 106062. [Google Scholar] [CrossRef]
  29. Yu, Y.; Guo, H.; Ding, Q.; Li, H.; Skitmore, M. An experimental study of real-time identification of construction workers’ unsafe behaviors. Autom. Constr. 2017, 82, 193–206. [Google Scholar] [CrossRef]
  30. Kim, K.; Kim, H.; Kim, H. Image-based construction hazard avoidance system using augmented reality in wearable device. Autom. Constr. 2017, 83, 390–403. [Google Scholar] [CrossRef]
  31. Fang, Q.; Li, H.; Luo, X.; Ding, L.; Luo, H.; Rose, T.M.; An, W. Detecting non-hardhat-use by a deep learning method from far-field surveillance videos. Autom. Constr. 2018, 85, 1–9. [Google Scholar] [CrossRef]
  32. Ardiny, H.; Witwicki, S.; Mondada, F. Construction automation with autonomous mobile robots: A review. In Proceedings of the 2015 3rd RSI International Conference on Robotics and Mechatronics (ICROM), Tehran, Iran, 7–9 October 2015; pp. 418–424. [Google Scholar]
  33. Park, M.W.; Elsafty, N.; Zhu, Z. Hardhat-wearing detection for enhancing on-site safety of construction workers. J. Constr. Eng. Manag. 2015, 141, 04015024. [Google Scholar] [CrossRef]
  34. Gu, J.; Wang, Z.; Kuen, J.; Ma, L.; Shahroudy, A.; Shuai, B.; Liu, T.; Wang, X.; Wang, G.; Cai, J.; et al. Recent advances in convolutional neural networks. Pattern Recognit. 2018, 77, 354–377. [Google Scholar] [CrossRef]
  35. Srinivas, S.; Sarvadevabhatla, R.K.; Mopuri, K.R.; Prabhu, N.; Kruthiventi, S.S.; Babu, R.V. A taxonomy of deep convolutional neural nets for computer vision. Front. Robot. AI 2016, 2, 36. [Google Scholar] [CrossRef]
  36. Chen, L.C.; Papandreou, G.; Kokkinos, I.; Murphy, K.; Yuille, A.L. Semantic image segmentation with deep convolutional nets and fully connected crfs. In Proceedings of the 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7–9 2015. [Google Scholar]
  37. Ahn, J.; Kwak, S. Learning pixel-level semantic affinity with image-level supervision for weakly supervised semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–22 June 2018; pp. 4981–4990. [Google Scholar]
  38. Pal, N.R.; Pal, S.K. A review on image segmentation techniques. Pattern Recognit. 1993, 26, 1277–1294. [Google Scholar] [CrossRef]
  39. Zaitoun, N.M.; Aqel, M.J. Survey on image segmentation techniques. Procedia Comput. Sci. 2015, 65, 797–806. [Google Scholar] [CrossRef]
  40. Chi, S.; Caldas, C.H. Automated object identification using optical video cameras on construction sites. Comput. Aided Civ. Infrastruct. Eng. 2011, 26, 368–380. [Google Scholar] [CrossRef]
  41. Fang, W.; Zhong, B.; Zhao, N.; Love, P.E.; Luo, H.; Xue, J.; Xu, S. A deep learning-based approach for mitigating falls from height with computer vision: Convolutional neural network. Adv. Eng. Inform. 2019, 39, 170–177. [Google Scholar] [CrossRef]
  42. Siddula, M.; Dai, F.; Ye, Y.; Fan, J. Classifying construction site photos for roof detection: A machine-learning method towards automated measurement of safety performance on roof sites. Constr. Innov. 2016, 16, 368–389. [Google Scholar] [CrossRef]
  43. Wei, Y.; Liang, X.; Chen, Y.; Jie, Z.; Xiao, Y.; Zhao, Y.; Yan, S. Learning to segment with image level annotations. Pattern Recognit. 2016, 59, 234–244. [Google Scholar] [CrossRef]
  44. Wu, Z.; Shen, C.; van den Hengel, A. Bridging Category-level and Instance-level Semantic Image Segmentation. arXiv 2016, arXiv:1605.06885. [Google Scholar]
  45. Hariharan, B.; Arbeláez, P.; Girshick, R.; Malik, J. Simultaneous detection and segmentation. In Proceedings of the Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, 6–12 September 2014; pp. 297–312. [Google Scholar]
  46. Kirillov, A.; He, K.; Girshick, R.; Rother, C.; Dollár, P. Panoptic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 15–20 June 2019; pp. 9404–9413. [Google Scholar]
  47. Farabet, C.; Couprie, C.; Najman, L.; LeCun, Y. Learning hierarchical features for scene labeling. IEEE Trans. Pattern Anal. Mach. Intell. 2012, 35, 1915–1929. [Google Scholar] [CrossRef]
  48. Liu, C.; Yuen, J.; Torralba, A. Sift flow: Dense correspondence across scenes and its applications. IEEE Trans. Pattern Anal. Mach. Intell. 2010, 33, 978–994. [Google Scholar] [CrossRef]
  49. Russell, B.; Torralba, A.; Liu, C.; Fergus, R.; Freeman, W. Object recognition by scene alignment. Adv. Neural Inf. Process. Syst. 2007, 20, 1–8. [Google Scholar]
  50. Gould, S.; Fulton, R.; Koller, D. September. Decomposing a scene into geometric and semantically consistent regions. In Proceedings of the 2009 IEEE 12th International Conference on Computer Vision, Kyoto, Japan, 29 September–2 October 2009; pp. 1–8. [Google Scholar]
  51. Girshick, R.B.; Donahue, J.; Darrell, T.; Malik, J. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA, 23–28 June 2014; pp. 580–587. [Google Scholar]
  52. Long, J.; Shelhamer, E.; Darrell, T. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 7–12 June 2015; pp. 3431–3440. [Google Scholar]
  53. He, K.; Gkioxari, G.; Dollár, P.; Girshick, R. Mask r-cnn. In Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 22–29 October 2017; pp. 2961–2969. [Google Scholar]
  54. Huang, Z.; Huang, L.; Gong, Y.; Huang, C.; Wang, X. Mask scoring r-cnn. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 15–20 June 2019; pp. 6409–6418. [Google Scholar]
  55. Teizer, J.; Vela, P.A. Personnel tracking on construction sites using video cameras. Adv. Eng. Inform. 2009, 23, 452–462. [Google Scholar] [CrossRef]
  56. Wang, N.; Zhao, X.; Zou, Z.; Zhao, P.; Qi, F. Autonomous damage segmentation and measurement of glazed tiles in historic buildings via deep learning. Comput. Aided Civ. Infrastruct. Eng. 2020, 35, 277–291. [Google Scholar] [CrossRef]
  57. DeVries, P.M.; Viégas, F.; Wattenberg, M.; Meade, B.J. Deep learning of aftershock patterns following large earthquakes. Nature 2018, 560, 632–634. [Google Scholar] [CrossRef] [PubMed]
  58. Spencer, B.F., Jr.; Hoskere, V.; Narazaki, Y. Advances in computer vision-based civil infrastructure inspection and monitoring. Engineering 2019, 5, 199–222. [Google Scholar] [CrossRef]
  59. Vodrahalli, K.; Bhowmik, A.K. 3D computer vision based on machine learning with deep neural networks: A review. J. Soc. Inf. Disp. 2017, 25, 676–694. [Google Scholar] [CrossRef]
  60. Gomez-Cabrera, A.; Escamilla-Ambrosio, P.J. Review of machine-learning techniques applied to structural health monitoring systems for building and bridge structures. Appl. Sci. 2022, 12, 10754. [Google Scholar] [CrossRef]
  61. Ye, X.W.; Jin, T.; Yun, C.B. A review on deep learning-based structural health monitoring of civil infrastructures. Smart Struct. Syst 2019, 24, 567–585. [Google Scholar]
  62. Gopalakrishnan, K.; Khaitan, S.K.; Choudhary, A.; Agrawal, A. Deep convolutional neural networks with transfer learning for computer vision-based data-driven pavement distress detection. Constr. Build. Mater. 2017, 157, 322–330. [Google Scholar] [CrossRef]
  63. Kolar, Z.; Chen, H.; Luo, X. Transfer learning and deep convolutional neural networks for safety guardrail detection in 2d images. Autom. Constr. 2018, 89, 58–70. [Google Scholar] [CrossRef]
  64. Cha, Y.J.; Choi, W.; Suh, G.; Mahmoudkhani, S.; Büyüköztürk, O. Autonomous structural visual inspection using region-based deep learning for detecting multiple damage types. Comput. Aided Civ. Infrastruct. Eng. 2018, 33, 731–747. [Google Scholar] [CrossRef]
  65. Tong, Z.; Gao, J.; Zhang, H.T. Recognition, location, measurement, and 3D reconstruction of concealed cracks using convolutional neural networks. Constr. Build. Mater. 2017, 146, 775–787. [Google Scholar] [CrossRef]
  66. Gibert, X.; Patel, V.M.; Chellappa, R. Deep multitask learning for railway track inspection. IEEE Trans. Intell. Transp. Syst. 2016, 18, 153–164. [Google Scholar] [CrossRef]
  67. Davoudi, R.; Miller, G.R.; Kutz, J.N. Computer vision based inspection approach to predict damage state and load level for RC members. In Proceedings of the 12th International Workshop on Structural Health Monitoring, Stanford, CA, USA, 10–12 September 2019. [Google Scholar]
  68. Lattanzi, D.; Miller, G.R.; Eberhard, M.O.; Haraldsson, O.S. Bridge column maximum drift estimation via computer vision. J. Comput. Civ. Eng. 2016, 30, 04015051. [Google Scholar] [CrossRef]
  69. Dawood, T.; Zhu, Z.; Zayed, T. Machine vision-based model for spalling detection and quantification in subway networks. Autom. Constr. 2017, 81, 149–160. [Google Scholar] [CrossRef]
  70. Yeum, C.M.; Dyke, S.J.; Ramirez, J. Visual data classification in post-event building reconnaissance. Eng. Struct. 2018, 155, 16–24. [Google Scholar] [CrossRef]
  71. Kim, B.; Cho, S. Automated vision based detection of cracks on concrete surfaces using a deep learning technique. Sensors 2018, 18, 3452. [Google Scholar] [CrossRef] [PubMed]
  72. Kang, D.; Cha, Y.J. Autonomous UAVs for structural health monitoring using deep learning and an ultrasonic beacon system with geo-tagging. Comput. Aided Civ. Infrastruct. Eng. 2018, 33, 885–902. [Google Scholar] [CrossRef]
  73. Xue, Y.D.; Li, Y.C. A fast detection method via region based fully convolutional neural networks for shield tunnel lining defects. Comput. -Aided Civ. Infrastruct. Eng. 2018, 33, 638–654. [Google Scholar] [CrossRef]
  74. Hoang, N.D.; Nguyen, Q.L.; Tran, V.D. Automatic recognition of asphalt pavement cracks using metaheuristic optimized edge detection algorithms and convolution neural network. Automat. Constr. 2018, 94, 203–213. [Google Scholar]
  75. Dorafshan, S.; Thomas, R.J.; Maguire, M. Comparison of deep convolutional neural networks and edge detectors for image-based crack detection in concrete. Constr. Build. Mater. 2018, 186, 1031–1045. [Google Scholar] [CrossRef]
  76. Wang, N.; Zhao, Q.; Li, S.; Zhao, X.; Zhao, P. Damage classification for masonry historic structures using convolutional neural networks based on still images. Comput. Aided Civ. Infrastruct. Eng. 2018, 33, 1073–1089. [Google Scholar] [CrossRef]
  77. Gao, Y.; Mosalam, K.M. Deep transfer learning for image-based structural damage recognition. Comput. Aided Civ. Infrastruct. Eng. 2018, 33, 748–768. [Google Scholar] [CrossRef]
  78. Wu, R.T.; Singla, A.; Jahanshahi, M.R.; Bertino, E.; Ko, B.J.; Verma, D. Pruning deep convolutional neural networks for efficient edge computing in condition assessment of infrastructures. Comput. Aided Civ. Infrastruct. Eng. 2019, 34, 774–789. [Google Scholar] [CrossRef]
  79. Zhang, B.; Zhou, L.; Zhang, J. A methodology for obtaining spatiotemporal information of the vehicles on bridges based on computer vision. Comput. Aided Civ. Infrastruct. Eng. 2019, 34, 471–487. [Google Scholar] [CrossRef]
  80. Wang, M.; Cheng, J.C. A unified convolutional neural network integrated with conditional random field for pipe defect segmentation. Comput. Aided Civ. Infrastruct. Eng. 2020, 35, 162–177. [Google Scholar] [CrossRef]
  81. Li, D.; Cong, A.; Guo, S. Sewer damage detection from imbalanced CCTV inspection data using deep convolutional neural networks with hierarchical classification. Autom. Constr. 2019, 101, 199–208. [Google Scholar] [CrossRef]
  82. Zha, B.; Bai, Y.; Yilmaz, A.; Sezen, H. November. Deep Convolutional Neural Networks for Comprehensive Structural Health Monitoring and Damage Detection. In Proceedings of the 12th International Workshop on Structural Health Monitoring, Stanford, CA, USA, 10–12 September 2019. [Google Scholar]
  83. Zha, B.; Bai, Y.; Yilmaz, A.; Sezen, H. Deep learning–based autonomous concrete crack evaluation through hybrid image scanning. Struct. Health Monit. 2019, 18, 1722–1737. [Google Scholar]
  84. Liu, Z.; Cao, Y.; Wang, Y.; Wang, W. Computer vision-based concrete crack detection using U-net fully convolutional networks. Autom. Constr. 2019, 104, 129–139. [Google Scholar] [CrossRef]
  85. Mohtasham Khani, M.; Vahidnia, S.; Ghasemzadeh, L.; Ozturk, Y.E.; Yuvalaklioglu, M.; Akin, S.; Ure, N.K. Deep-learning-based crack detection with applications for the structural health monitoring of gas turbines. Struct. Health Monit. 2020, 19, 1440–1452. [Google Scholar] [CrossRef]
  86. Zhang, C.; Chang, C.C.; Jamshidi, M. Concrete bridge surface damage detection using a single-stage detector. Comput.-Aided Civ. Infrastruct. Eng. 2020, 35, 389–409. [Google Scholar] [CrossRef]
  87. Liu, Y.; Yeoh, J.K.; Chua, D.K. Deep learning–based enhancement of motion blurred UAV concrete crack images. J. Comput. Civ. Eng. 2020, 34, 04020028. [Google Scholar] [CrossRef]
  88. Kim, B.; Cho, S. Automated multiple concrete damage detection using instance segmentation deep learning model. Appl. Sci. 2020, 10, 8008. [Google Scholar] [CrossRef]
  89. Ghosh Mondal, T.; Jahanshahi, M.R.; Wu, R.T.; Wu, Z.Y. Deep learning-based multi-class damage detection for autonomous post-disaster reconnaissance. Struct. Control Health Monit. 2020, 27, e2507. [Google Scholar] [CrossRef]
  90. Deng, W.; Mou, Y.; Kashiwa, T.; Escalera, S.; Nagai, K.; Nakayama, K.; Matsuo, Y.; Prendinger, H. Vision based pixel-level bridge structural damage detection using a link ASPP network. Autom. Constr. 2020, 110, 102973. [Google Scholar] [CrossRef]
  91. Zheng, M.; Lei, Z.; Zhang, K. Intelligent detection of building cracks based on deep learning. Image Vis. Comput. 2020, 103, 103987. [Google Scholar] [CrossRef]
  92. Karaaslan, E.; Bagci, U.; Catbas, F.N. Attention-guided analysis of infrastructure damage with semi-supervised deep learning. Autom. Constr. 2021, 125, 103634. [Google Scholar] [CrossRef]
  93. Miao, Z.; Ji, X.; Okazaki, T.; Takahashi, N. Pixel-level multicategory detection of visible seismic damage of reinforced concrete components. Comput. Aided Civ. Infrastruct. Eng. 2021, 36, 620–637. [Google Scholar] [CrossRef]
  94. Qiao, W.; Ma, B.; Liu, Q.; Wu, X.; Li, G. Computer vision-based bridge damage detection using deep convolutional networks with expectation maximum attention module. Sensors 2021, 21, 824. [Google Scholar] [CrossRef]
  95. Huang, Z.; Fu, H.L.; Fan, X.D.; Meng, J.H.; Chen, W.; Zheng, X.J.; Wang, F.; Zhang, J.B. Rapid surface damage detection equipment for subway tunnels based on machine vision system. J. Infrastruct. Syst. 2021, 27, 04020047. [Google Scholar] [CrossRef]
  96. Arya, D.; Maeda, H.; Ghosh, S.K.; Toshniwal, D.; Mraz, A.; Kashiyama, T.; Sekimoto, Y. Deep learning-based road damage detection and classification for multiple countries. Autom. Constr. 2021, 132, 103935. [Google Scholar] [CrossRef]
  97. Cui, X.; Wang, Q.; Dai, J.; Zhang, R.; Li, S. Intelligent recognition of erosion damage to concrete based on improved YOLO-v3. Mater. Lett. 2021, 302, 130363. [Google Scholar] [CrossRef]
  98. Pozzer, S.; Rezazadeh Azar, E.; Dalla Rosa, F.; Chamberlain Pravia, Z.M. Semantic segmentation of defects in infrared thermographic images of highly damaged concrete structures. J. Perform. Constr. Facil. 2021, 35, 04020131. [Google Scholar] [CrossRef]
  99. Andrushia, A.D.N.A.; Lubloy, E. Deep learning based thermal crack detection on structural concrete exposed to elevated temperature. Adv. Struct. Eng. 2021, 24, 1896–1909. [Google Scholar] [CrossRef]
  100. Munawar, H.S.; Ullah, F.; Heravi, A.; Thaheem, M.J.; Maqsoom, A. Inspecting buildings using drones and computer vision: A machine learning approach to detect cracks and damages. Drones 2021, 6, 5. [Google Scholar] [CrossRef]
  101. Zou, D.; Zhang, M.; Bai, Z.; Liu, T.; Zhou, A.; Wang, X.; Cui, W.; Zhang, S. Multicategory damage detection and safety assessment of post-earthquake reinforced concrete structures using deep learning. Comput. Aided Civ. Infrastruct. Eng. 2022, 37, 1188–1204. [Google Scholar] [CrossRef]
  102. Han, X.; Zhao, Z.; Chen, L.; Hu, X.; Tian, Y.; Zhai, C.; Wang, L.; Huang, X. Structural damage-causing concrete cracking detection based on a deep-learning method. Constr. Build. Mater. 2022, 337, 127562. [Google Scholar] [CrossRef]
  103. Tanveer, M.; Kim, B.; Hong, J.; Sim, S.H.; Cho, S. Comparative Study of Lightweight Deep Semantic Segmentation Models for Concrete Damage Detection. Appl. Sci. 2022, 12, 12786. [Google Scholar] [CrossRef]
  104. Bai, Z.; Liu, T.; Zou, D.; Zhang, M.; Zhou, A.; Li, Y. Image-based reinforced concrete component mechanical damage recognition and structural safety rapid assessment using deep learning with frequency information. Autom. Constr. 2023, 150, 104839. [Google Scholar] [CrossRef]
  105. Crognale, M.; De Iuliis, M.; Rinaldi, C.; Gattulli, V. Damage detection with image processing: A comparative study. Earthq. Eng. Eng. Vib. 2023, 22, 333–345. [Google Scholar] [CrossRef]
  106. Chen, L.; Chen, W.; Wang, L.; Zhai, C.; Hu, X.; Sun, L.; Tian, Y.; Huang, X.; Jiang, L. Convolutional neural networks (CNNs)-based multi-category damage detection and recognition of high-speed rail (HSR) reinforced concrete (RC) bridges using test images. Eng. Struct. 2023, 276, 115306. [Google Scholar] [CrossRef]
  107. Wan, H.; Gao, L.; Yuan, Z.; Qu, H.; Sun, Q.; Cheng, H.; Wang, R. A novel transformer model for surface damage detection and cognition of concrete bridges. Expert Syst. Appl. 2023, 213, 119019. [Google Scholar] [CrossRef]
  108. Zhu, Y.; Tang, H. Automatic damage detection and diagnosis for hydraulic structures using drones and artificial intelligence techniques. Remote Sens. 2023, 15, 615. [Google Scholar] [CrossRef]
  109. Huang, B.; Zhao, S.; Kang, F. Image-based automatic multiple-damage detection of concrete dams using region-based convolutional neural networks. J. Civ. Struct. Health Monit. 2023, 13, 413–429. [Google Scholar] [CrossRef]
  110. Kim, H.; Lee, J.; Ahn, E.; Cho, S.; Shin, M.; Sim, S.H. Concrete crack identification using a UAV incorporating hybrid image processing. Sensors 2017, 17, 2052. [Google Scholar] [CrossRef] [PubMed]
  111. Tong, Z.; Gao, J.; Sha, A.; Hu, L.; Li, S. Convolutional neural network for asphalt pavement surface texture analysis. Comput. Aided Civ. Infrastruct. Eng. 2018, 33, 1056–1072. [Google Scholar] [CrossRef]
  112. Huang, Z.; Fu, H.; Chen, W.; Zhang, J.; Huang, H. Damage detection and quantitative analysis of shield tunnel structure. Autom. Constr. 2018, 94, 303–316. [Google Scholar] [CrossRef]
  113. Tayo, C.O.; Linsangan, N.B.; Pellegrino, R.V. Portable crack width calculation of concrete road pavement using machine vision. In Proceedings of the 2019 IEEE 11th International Conference on Humanoid, Nanotechnology, Information Technology, Communication and Control, Environment, and Management (HNICEM), Laoag, Philippines, 29 November–1 December 2019; pp. 1–5. [Google Scholar]
  114. Kim, B.; Cho, S. Image-based concrete crack assessment using mask and region-based convolutional neural network. Struct. Control Health Monit. 2019, 26, e2381. [Google Scholar] [CrossRef]
  115. Wei, F.; Yao, G.; Yang, Y.; Sun, Y. Instance-level recognition and quantification for concrete surface bughole based on deep learning. Autom. Constr. 2019, 107, 102920. [Google Scholar] [CrossRef]
  116. Beckman, G.H.; Polyzois, D.; Cha, Y.J. Deep learning-based automatic volumetric damage quantification using depth camera. Autom. Constr. 2019, 99, 114–124. [Google Scholar] [CrossRef]
  117. Park, S.E.; Eem, S.H.; Jeon, H. Concrete crack detection and quantification using deep learning and structured light. Constr. Build. Mater. 2020, 252, 119096. [Google Scholar] [CrossRef]
  118. Bhowmick, S.; Nagarajaiah, S.; Veeraraghavan, A. Vision and deep learning-based algorithms to detect and quantify cracks on concrete surfaces from UAV videos. Sensors 2020, 20, 6299. [Google Scholar] [CrossRef] [PubMed]
  119. Flah, M.; Suleiman, A.R.; Nehdi, M.L. Classification and quantification of cracks in concrete structures using deep learning image-based techniques. Cem. Concr. Compos. 2020, 114, 103781. [Google Scholar] [CrossRef]
  120. Yuan, C.; Xiong, B.; Li, X.; Sang, X.; Kong, Q. A novel intelligent inspection robot with deep stereo vision for three-dimensional concrete damage detection and quantification. Struct. Health Monit. 2022, 21, 788–802. [Google Scholar] [CrossRef]
  121. Miao, P.; Srimahachota, T. Cost-effective system for detection and quantification of concrete surface cracks by combination of convolutional neural network and image processing techniques. Constr. Build. Mater. 2021, 293, 123549. [Google Scholar] [CrossRef]
  122. Song, L.; Sun, H.; Liu, J.; Yu, Z.; Cui, C. Automatic segmentation and quantification of global cracks in concrete structures based on deep learning. Measurement 2022, 199, 111550. [Google Scholar] [CrossRef]
  123. Kumarapu, K.; Mesapam, S.; Keesara, V.R.; Shukla, A.K.; Manapragada, N.V.S.K.; Javed, B. RCC Structural deformation and damage quantification using unmanned aerial vehicle image correlation technique. Appl. Sci. 2022, 12, 6574. [Google Scholar] [CrossRef]
  124. Bae, H.; An, Y.K. Computer vision-based statistical crack quantification for concrete structures. Measurement 2023, 211, 112632. [Google Scholar] [CrossRef]
  125. Li, Y.; Bao, T.; Huang, X.; Wang, R.; Shu, X.; Xu, B.; Tu, J.; Zhou, Y.; Zhang, K. An integrated underwater structural multi-defects automatic identification and quantification framework for hydraulic tunnel via machine vision and deep learning. Struct. Health Monit. 2023, 22, 2360–2383. [Google Scholar] [CrossRef]
  126. Milosevic, I.N. Practical Application of SWOT Analysis in the Management of a Construction Project. Leadersh. Manag. Eng. 2010, 10, 78–86. [Google Scholar] [CrossRef]
  127. Bao, Y.Q.; Tang, Z.Y.; Li, H.; Zhang, Y.F. Computer vision and deep learning based data anomaly detection method for structural health monitoring. Struct. Health Monit. 2019, 18, 401–421. [Google Scholar] [CrossRef]
  128. Gkoumas, K.; Gkoktsi, K.; Bono, F.; Galassi, M.C.; Tirelli, D. The way forward for indirect structural health monitoring (iSHM) using connected and automated vehicles in Europe. Infrastructures 2021, 6, 43. [Google Scholar] [CrossRef]
  129. Chowdhury, A.M.; Moon, S. Generating integrated bill of materials using mask R-CNN artificial intelligence model. Autom. Constr. 2023, 145, 104644. [Google Scholar] [CrossRef]
  130. Deng, J.; Singh, A.; Zhou, Y.; Lu, Y.; Lee, V.C.S. Review on computer vision-based crack detection and quantification methodologies for civil structures. Constr. Build. Mater. 2022, 356, 129238. [Google Scholar] [CrossRef]
  131. Yokoyama, S.; Matsumoto, T. Development of an automatic detector of cracks in concrete using machine learning. Procedia Eng. 2017, 171, 1250–1255. [Google Scholar] [CrossRef]
Figure 1. Outline of the research methodology.
Figure 1. Outline of the research methodology.
Constrmater 04 00005 g001
Figure 2. Overview of CNN and deep learning techniques.
Figure 2. Overview of CNN and deep learning techniques.
Constrmater 04 00005 g002
Figure 3. Pipeline for the 3D reconstruction models of pavement cracks [65].
Figure 3. Pipeline for the 3D reconstruction models of pavement cracks [65].
Constrmater 04 00005 g003
Figure 4. Before and after concrete crack segmentation [122].
Figure 4. Before and after concrete crack segmentation [122].
Constrmater 04 00005 g004
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Chowdhury, A.M.; Kaiser, R. A Comprehensive Analysis of the Integration of Deep Learning Models in Concrete Research from a Structural Health Perspective. Constr. Mater. 2024, 4, 72-90. https://doi.org/10.3390/constrmater4010005

AMA Style

Chowdhury AM, Kaiser R. A Comprehensive Analysis of the Integration of Deep Learning Models in Concrete Research from a Structural Health Perspective. Construction Materials. 2024; 4(1):72-90. https://doi.org/10.3390/constrmater4010005

Chicago/Turabian Style

Chowdhury, Ayesha Munira, and Rashed Kaiser. 2024. "A Comprehensive Analysis of the Integration of Deep Learning Models in Concrete Research from a Structural Health Perspective" Construction Materials 4, no. 1: 72-90. https://doi.org/10.3390/constrmater4010005

Article Metrics

Back to TopTop