Abstract
Bridge defects are important indicator for the bridge safety assessment. Considering the cost and inefficiency of the traditional method, the UAV system applied for bridge crack inspection is a better choice. Therefore, we have configured a bridge inspection UAV system with SLR camera, laser rangefinder. First, we have carried an evaluation experiment to determine the distance range of stable imaging for planning the safer bridge inspection route based on the special UAV system. Then, the crack recognition method combining neural network and support vector machine is used to locate and extract the bridge cracks, and then, the actual cracks are calculated according to the optical principle. Finally, a case study of the Xiangjiang-River bridge inspection is carried out to verify the feasibility of bridge defects recognition based on this UAV system, achieving above 90% in the crack width recognition, which provides a better platform for bridge inspection.
1. Introduction
Bridge is a critical infrastructure. However, many in-service bridges are suffering from continuous deterioration and damages, requiring their conditions to be regularly monitored and assessed. In USA, a bridge should be inspected every two years according to AASHTO requirements [1]. In UK, bridge inspection should be undertaken every 1–3 years according to the standard “Examination of Bridges and Culverts NR/SP/CIV/017” [2]. Aged bridges require periodic inspection and evaluation, which has been usually done using a detection vehicle or artificial platform. However, due to the influence by terrain, bridge type, and traffic, the traditional bridge inspection methods are limited; therefore, a new inspection platform is warranted.
In recent years, the unmanned aerial vehicle (UAV) has been extensively used in many fields, such as urban planning [3, 4], agriculture [5, 6], measurement [7, 8], and transportation [9, 10]. Especially, in the areas of structural health monitoring, the UAV has been used to obtain images of structural appearance for defect recognition and evaluation [11–16]. For example, the Aerobi EU Project has been established in Europe aimed at the development and validation of the prototype of an innovative, intelligent, and aerial robotic system [17, 18]. Salaan has designed UAV which has a carbon fiber skeleton with a spherical exoskeleton for protecting the UAV and a UAV having two wheels which can climb and run on the bridge surface [19]. Myeong has developed a different approach which is the posture-changing UAV using a tiltrotor mechanism [20]. Jiang and Zhang have configured wall-climbing unmanned aerial system to acquire detailed crack images without distortion [21]. Although UAV has been widely used in civil engineering inspection, the application of crack width recognition based on airborne images in bridge inspection is relatively few, which depends on the clarity of crack images and the sufficiency of pixels.
According to Zhong et al. [22], the UAV system equipped carrying a SLR camera with 40 million pixels can identify the crack width as small as 0.2 mm for images taken within 4 m from the object, thus requiring a close flight to the structure for recognizing the crack width. Under the combined action of self-inducing airstream, external wind effect, and near-wind effect, the UAV will be disturbed, resulting in unclear inspection image, as shown in Figure 1. Therefore, the wind resistance and obstacle avoidance of UAV have become more concerns [23]. For example, Al-Kaff et al. [24] used the monocular vision technology to determine the location of the UAV. Carrio et al. [25] used the infrared thermal imaging technology to detect obstacles, enhancing the ability of UAV of avoiding obstacles in direct sunlight and night time. Rawashdeh et al. [26] utilized the imaging frame changes in a video to estimate the pitching and rolling of a UAV. Fernández et al. [27] used the UAV to detect wind turbine blades and developed a robust feedback procedure to effectively deal with flight disturbances. The literature review reveals that some studies related to the wind resistance and obstacle avoidance of UAV have been done, but it does not consider the near-wind effect. To address the above-mentioned problems, the motion characteristics of UAV system considering different wind speeds and wind directions should be tested for planning the safer bridge inspection route before it can be applied for bridge inspection.
(a)
(b)
(c)
(d)
(e)
However, the airborne bridge inspection often generates large amounts of image data which is difficult to analyse through traditional image progress method. As a rapidly developing method, the deep learning may be adopted to recognize a large amount of airborne imaging data. For example, Li et al. [28] designed the DDLNet to recognize the concrete surface defects. Cheng and Wang [29] developed a method of sewage pipe crack detection based on the Faster R-CNN network. Tong [30] proposed a convolutional neural network to calculate mean texture depths. Wang and Zhang [7] proposed a deep architecture of convolutional neural network damage classification techniques for historic masonry structures. Bang et al. [31] proposed the deep convolutional encoder–decoder network for identifying road cracks. Nevertheless, the images taken by UAV have those characteristics. First, the images of bridge defects in real-world conditions are complex with inevitable background noise. Secondly, the crack defect is a more important indicator for the bridge safety assessment, but the pixel width of bridge crack is far less than other defects which results in unbalance in image process and recognition. Therefore, the bridge crack recognition method needs further study.
To address the above-mentioned problems, in this paper, the motion characteristics of the UAV system are measured in the wind tunnel using high-speed camera system. Then, the range of UAV flight distance for stable imaging is determined when approaching the bridge surface. Moreover, the method combining Region-based Fully Convolutional Networks (R-FCN) and support vector machine to calculate the width of bridge crack is introduced. Finally, the case study of the Xiangjiang-River bridge inspection is carried out to demonstrate the effectiveness of the proposed method.
2. Methodology
Hovering is a notable feature of UAVs rotors. In hovering, the relative rotors have the same rotational speed, giving the resultant force equal to their own gravity. In addition, since the rotor speed is equal and the front and rear rotate in the opposite direction to the left and right, the total torque of the aircraft is zero, resulting in a hovering state. According to the manufacturer’s information, the UAV can achieve a stable hovering flight against Class 6 wind in an open field with sufficient GPS signal. The manufacturer has balanced the weights of the UAV system according to the load distribution of added working equipment, and its aerodynamic stability is strictly tested in common conditions before delivery.
However, the factory parameters of this special bridge inspection UAV system do not take into account the impact of complex aerodynamic interference when the UAV is approaching the bridge, thus significantly weakening the wind resistance and hovering stability. The motion of airborne camera system in a hover state will cause a relative movement between the shooting target and the sensors (complementary metal oxide semiconductor, charge coupled device, film, etc.) during exposure, i.e., an image shift. The image shift may cause a loss of image clarity, lower contrast, or image blurring. It is necessary to test motion characteristics of UAV system to determine the inspection safety distance range and condition before it can be applied to the bridge inspection. Therefore, in this section, we proposed the motion characteristics measurement method to assess the stability when the UAV is approaching different types of bridge quantitatively.
2.1. Linear Displacement
According to [22], because the airborne camera is a coaxial spherical optical system, the direction of OX coordinate axis is the direction of camera shooting. As shown in Figure 2, the displacement of OX axis is the elongation and shortening of the optical axis, while the displacement of OY and OZ axis is the offset of the optical axis. Therefore, the displacement along OX direction is determined by equations (1) and (2), and the displacement along OY or OZ direction is determined by equations (3) and (4):where and are the distances between the image point and the center of full-frame complementary metal oxide semiconductor sensor before and after vibration, respectively; is the displacement of camera lens along OX axis; is the object distance; is the location of the target before vibration; f is the focal length of the optical system; is the relative position of the target after vibration; is the linear displacement of the airborne camera along OY, OZ axes and is the image shift. The maximum may be taken as half of the physical diagonal length of full-frame complementary metal oxide semiconductor sensor (= 21.63 mm). The values along OX, OY, and OZ axes were calculated separately.
2.2. Angular Displacement
The effect of UAV rotation on imaging quality can be quantitatively analyzed using the angular displacements of the airborne imaging system. As shown in Figure 3, OX axis is the shooting direction of the airborne camera. In the YOZ plane, the rotation is determined by (6) and the image shift around the OX axis is determined by (7). Similar to the linear displacements along OY and OZ axes, the same equation can be used to determine the rotation of the optical system around OY and OZ axes. Taking the rotation about OZ axis as an example, in the XOY plane, OY axis rotates clockwise for any two points on the axis, which is calculated by (8), while the image shift around OZ axis is determined by
In the above equations, is the image shift around axis, is taken as half of the long dimension of the full-frame complementary metal oxide semiconductor sensor ( = 18 mm), is the radius of lens ( = 33.5 mm), is the linear displacement difference between mark points and 2 in the OZ direction (Figure 2), and is the linear displacement difference between mark points 4 and 2 in the OX direction (Figure 3). The image shifts ( values) caused by the rotations of the airborne imaging system around OX, OY, and OZ axes can be obtained by substituting the test results and camera parameters into (5)–(8).
2.3. Motion Characteristics Evaluation
The optical modulation transfer function (MTF) used in the scanning method is the most common approach for evaluating the imaging quality. The modulation transfer function can expressed by [32, 33]where is the image shift during exposure and is a fixed lens parameter. The Sony ILCE-7RM2 SLR camera and FE 85 mm F1.8 lens come with a 36 × 24 mm full-frame complementary metal oxide semiconductor sensor and a pixel resolution of 7952 × 5304, giving N = 30 lp/mm. The modulation transfer function value in (9) is between 0 and 1. The closer the modulation transfer function value is to 1, the higher the image quality will be. Previous study in [22] showed that the imaging quality was affected by the displacements along or around OX, OY, and OZ axes (Figures 2 and 3). Therefore, the displacements along those axes were calculated and the imaging quality was evaluated using the modulation transfer function.
3. Motion Characteristics of Measurement Experiments
3.1. UAV System
To identify bridge crack widths quantitatively, especially the crack in bottom slab of the bridge, the UAV airborne imaging system is configured as shown in Figure 4, which is named EHM-6V. The system consists of six-rotor UAV (with diameter = 1000 mm, height = 540 mm, and load weight = 4200 g), GPS system (for locating the real-time position of UAV), wireless remote control system (for controlling the attitude of UAV), high-definition transmission system, Sony ILCE-7RM2 camera (with effective pixel 7952 × 5304 and mass = 582 g), three-point laser rangefinder (synchronized with camera shutter, used to measure the distance between the bridge appearance and the camera), and the upward three-axis stabilizing and mounting platform.
(a)
(b)
(c)
(d)
According to the information provided by the manufacturer, the UAV used in this paper has a wheelbase of 980 mm, propeller size of 18 × 5.5 inch, single arm length of 340 mm, and maximum takeoff weight of 8 kg, while the working load of this paper is only 0.7 kg. Therefore, the lift provided by UAV system has a large allocation space for working load, the vibration characteristics of UAV system hovering in open field are also tested before the wind tunnel experiment, and the test results meet the requirements of imaging quality.
3.2. Bridge Models
As shown in Figure 5 and Table 1, the wood models of arch and beam bridges are built in the tail section of the wind tunnel. The required wind is produced at the tail end of the diffusion section to simulate the aerodynamic interference on the UAV due to the bridge boundary. The type of UAV used in this experiment is six-rotor UAV (Figure 5). The size of wood bridge model used in wind tunnel test is determined to satisfy the size condition of bridge aerodynamic effect [34]. The sizes for the bridge models are 6000 mm × 2000 mm × 2000 mm for the beam bridge and 6000 mm × 2000 mm × 1000 mm for the arch bridge with the optimal rise to span ratio of 1 : 1.6, as shown in Figures 6(b) and 6(c) and Table 1, respectively.
(a)
(b)
(c)
3.3. Experimental Conditions
To secure the range of stable imaging under actual environment, there are the hovering stability and imaging quality of the UAV under different wind levels of 0 (calm), 1 (light air), and 2 (light breeze) [28], wind directions (tailwind, crosswind), positions (bottom, side), and bridge types (beam, arch). To meet the requirements of crack width (above 0.2 mm) identification and safe distance reserve, a safe flight distance of 6–2.5 m is determined based on the preliminary test flight. Also, when the UAV flew approaching the bottom bridge surface, the effect from the reflected wind from the surface is smaller and the safe flight distance could be hence at 3–1.5 m.
3.4. Experimental Process
The experiment was carried out on a sunny day with no naturally caused aerodynamic effect on the UAV. The layout of the experiment and experiment conditions is shown in Figure 5 and Table 2. Prior to the experiment, wind speeds were measured along the UAV flight path, as shown in Figure 6. The wind speed is adjusted according to the test wind speed until it meets the experimental requirement. During the experiment, four people were arranged to monitor the natural wind speeds at the monitoring points shown in Figure 6 to ensure that the experiment was carried out with negligible natural wind (0–0.2 m/s). The experiment was carried out at the wind levels 0–2. Because of the low wind speed level, the influence of viscous force on the flow field is less than that of inertia force. The wind flow is stable and the effect of turbulence need not be considered.
In order to test the imaging stability of UAV, the high-speed camera system is used to observe the real-time displacement of UAV during its motion. As shown in Figure 7, four points in the lens are marked and captured during testing. The coordinate axis PX is the shooting direction of the airborne SLR camera, where P is the origin of the coordinate system or the center of camera lens. The marker points on the SLR camera lens were selected to replace UAV system linear displacements along each coordinate axis.
3.5. Comparisons of Linear Displacement and Angular Displacement
The linear displacements along OX, OY, and OZ axes and the angular displacements around OX, OY, and OZ axes are calculated by (9). Because the airborne camera is a spherical optical system, the linear displacements in OY and OZ directions are superposed according to the Pythagorean theorem. Similarly, the angular displacements around OY and OZ axes are superposed according to the Pythagorean theorem. The influence of linear and angular displacements on the imaging quality is quite different. Table 3 shows the motion characteristics when the UAV approaches the side of beam bridge at 3 m under the level 2 wind.
Based on Table 3, the influence of linear and angular displacements on imaging quality can be determined. It is found that the effect of linear displacement on imaging quality is much greater than that of angular displacement. The linear displacements differences along X and YZ axis are smaller. The test data of all working conditions are analyzed and processed and the most disadvantageous OX and YZ values are taken as the test results of modulation transfer function.
3.6. Results of the Bridge Model Experiments
To test the imaging quality and hovering stability when the UAV system approaches the beam and arch bridge model, flight experiments are conducted according to the method described above. The MTFs are calculated for each distance. They are the average values of a set of tests with the same distance. The corresponding results are presented in Tables 4–9.
As with the beam bridge, the field test results of the dynamic characteristics of the UAV approaching the actual arch bridge are similar to those from the wind tunnel test. For wind levels 0–2, the minimum distance for motion characteristics to meet the requirements of imaging quality is 3.0 m when the UAV approaches the side of the arch bridge. In comparison, the field test results of the dynamic characteristics of the UAV approaching the arch bridge are virtually the same as the beam bridge in the wind tunnel test. For wind levels 0–2, the minimum distance for motion characteristics to meet the requirements of imaging quality for flight safety is 3.0 m when the UAV is approaching the side of the beam bridge and 2.0 m when the UAV is approaching the bottom of the bridge.
3.7. Result of the Actual Bridge Model Experiments
To verify the wind tunnel test results, field experiments are carried out at Xiangtan Xiangjiang beam bridge and Xichun arch bridge (Figure 8 and Table 10) in China. The beam bridge is 16 m long, 2 m high, and 18 m wide, while the arch bridge is 17 m long in span, 8 m high, and 28 m wide. The corresponding UAV system and airborne equipment mentioned above are used to take outdoor experiment. The laser rangefinder and the wind speed measuring instrument are used to record and taken one by one according to the time sequence. Due to the outdoor gust situation, repeated experiments are carried out until enough valid data are collected, and the data that met the requirements of experimental wind speed are selected for analysis and calculation under different wind speeds, as shown in Tables 11–14.
(a)
(b)
(c)
(d)
The test results show that the imaging quality and hovering stability of UAV are greatly affected by winds when the UAV is approaching the bridge surface. Under the action of tailwind and crosswind (wind levels = 0–2), the modulation transfer function value of UAV imaging system could reach above 0.9 at 3 m distance when the UAV is approaching the beam bridge. In comparison, the field test results of the dynamic characteristics of the UAV approaching the actual bridge are virtually the same as those in the wind tunnel test. Under the action of downward wind (wind levels = 0–2), the modulation transfer function value of the UAV system could rise above 0.9 at 2.0 m distance when the UAV is approaching the bottom of bridge. This means that the UAV could meet the imaging quality requirement at the distance of 2.0 m, while maintaining a safe flight to obtain clear images.
4. Method of Bridge Defects Recognition
To quantify the safety status of an in-service bridge, it is necessary to recognize the bridge defect. In this paper, the recognition framework of bridge defects using neural network to deal with the airborne images is proposed as shown in Figure 9, which contains two parts: one is the target recognition network for locating and classifying bridge defects, and the other is the support vector machine for extracting bridge cracks accurately. After the bridge images are taken and recorded by UAV system, the localization and classification information is generated according to the proposed method.
4.1. Architecture of the R-FCN Network
The target recognition network is composed of basic full convolution network, regional generation network, and region of interest subnetwork. The full convolution network is used to extract image features and the regional generation network generates region of interest based on these features. The recognition and location of bridge defects by region of interest subnet are made on the basis of the features extracted by full convolution network and the region of interest output by regional generation network. In the feature extraction part, the ResNet is used as the feature extraction backbone to extract the image features and avoid the information loss caused by pooling process as shown in Figure 10. To improve the ability of fractured target segmentation, different sampling rates are used to extract multiscale features from the feature map. Finally, airborne bridge crack images are segmented by the cascade output and feature decoding process.
4.2. Support Vector Machine Model
After the threshold segmentation through the maximum entropy method, a binary image of the bridge crack is obtained. The pores, holes, concrete defects, scratches, and template marks of the segmented images are white along with the cracks. These white dots and white blocks are noise. To recognize the bridge crack, the noise of “noncrack points” or “noncrack blocks” must be eliminated. However, the traditional multilevel filtering algorithm based on the connected region requires setting the threshold manually.
The support vector machine is a machine learning algorithm based on structural risk minimization, which has advantages of strong generalization ability and being suitable for small sample classification. Therefore, the support vector machine algorithm is used to identify the cracks in this paper. Then, for obtaining the image range set, randomly selected n ranges from the range set are used as the sample set . From this set, = 1, 2, 3, …, n and taking n = 1000 as the sample numbers, is the eigenvector of the image area sample of the bridge crack, and is the artificial labelling of the eigenvector, where 0 means noncrack and 1 means crack. For the crack recognition, four geometric features are used, namely, the region area S, the region rectangularity J, the minimum enclosing rectangle’s length-width ratio B, and the regional eccentricity E. The eigenvector of is constructed. From the sample set, 75% was randomly selected as the training set and the remaining 25% was selected as the validation set. The radial basis function (RBF) was selected as the kernel function of the support vector machine. The kernel function is
The corresponding optimal function is
5. Case Study
As shown in Figure 11, the proposed method and system are applied on Xiangjiang-River bridge in Hunan province to inspect the defects. The experimental result was used to guide bridge inspection. As shown in Figure 12, the inspection images of bridge defect are clear which is suited for further image processing to evaluate the bridge condition in the wind speed levels 0–2. The physical size of the complementary metal oxide semiconductor sensor in the SONY A7R2 is 36 × 24 mm and the pixel size is 7952 × 5304.
(a)
(b)
(a)
(b)
(c)
(d)
5.1. Classification Data Set Annotation
The crack inspection data sets are built from the bridge defect images acquired by the UAV system, and these are used to train the R-FCN neural network. According to above references, the data set is divided into four cases (Figure 13): water seepage, concrete spalling, reinforcing bars, and cracks. They are formed considering the following criteria related to image labelling: (1) the bridge defects need to be wrapped tightly by rectangular ground-truth boxes for precise locating; (2) the defect images with different background are selected and marked for enhancing the robustness; (3) the slender crack objects are marked with different size frames; (4) the data set should include as many defects as possible with different sizes.
(a)
(b)
(c)
(d)
There are 1600 images collected from the airborne images of bridge defects and annotated using the graphical image annotation tool “LabelImg” to form a crack target recognition data set as shown in Figure 14 and Table 15. Each labeled image has the size of 7952 pixels 5304 pixels. As shown in Table 15, the image data sets are divided into the training set and verification set with a ratio of 4 : 1.
5.2. Training and Verification
The target recognition network is trained by the random gradient descent method with the initial learning rate of 0.001, the momentum of 0.9, the weight decay of 0.0005, and the validation cycle of 5000. When the accuracy of the model reaches the required convergence, the training is stopped and the maximum number of iterations is set as 20000. The hardware system used in the training is Intel core i7-7700@ 3.60 GHz 8-core processor with 64 GBs running memory (RAM) and it is accelerated by NVIDIA GeForce GTX-1080Ti graphics card, running in TensorFlow framework under Linux Ubuntu system. The loss of the training process is shown in Figure 15. The result of the model evaluation is shown in Table 16, indicating that the R-FCN network could attain more than 91% precision.
5.3. Bridge Crack Extraction
The maximum entropy threshold segmentation is applied in Figure 16 in view of the obvious changes in the gray level of the fractured region and the background region, ensuring recognition accuracy. The artificial threshold setting is not required, which is better suited for airborne imaging; the segmentation results are shown in Figure 16. The support vector machine model is trained and validated by the collected data set. The test results show that the samples of 235 verification sets are correctly classified and the accuracy is 94%. The noncrack area is eliminated, the area is marked, and the crack image is shown in Figure 16.
5.4. Analysis and Comparison
In order to verify the crack width recognition accuracy of the proposed method based on UAV, the crack examples of the Xiangjiang-River bridge and its inspection are described and shown in Figure 17. From the recognition results of the three regions, the maximum number of pixels of the crack width is selected from Figure 17. The widths of crack pixels are calculated by [22]; the ratio of pixel width to actual width named pixel resolution is converted bywhere is the distance, is the focal length, is the pixel resolution, is the longer dimension of the image sensor, and is the number of pixels along the long side of the image sensor. The widths of crack pixels are calculated by (12), as listed in Table 17. The results show that the proposed method has high accuracy and presents broad engineering applications.
(a)
(b)
6. Conclusion
In this study, we propose a method of the minimum flight distance range test for bridge inspection based on UAV. By the wind tunnel test considering different wind speeds and wind directions, we obtain the primary approaching distance range, providing a more safe and efficient flight mode for bridge crack width recognition. Then, we verified the accuracy of bridge crack width recognition in this inspection mode through the identification method combining neural network and support vector machine. Based on this study, the following major findings are offered:
The wind tunnel and field tests have shown that the aerodynamic interference is obvious when the tested UAV approaches the surface of the bridge. The actual wind resistance performance of the tested UAV can only meet flight safety requirement under level 2 wind speed which is far less than the requirement for a stable flight against level 6 wind speed as specified by the manufacturer. This experiment provides a primary basis for future wind resistance design and use of UAV to inspect a bridge (crack width and other bridge defects).
To meet the requirements of bridge defects recognition, a novel method combining the R-FCN and support vector machine is proposed. We have built the bridge defects data set by collecting the bridge inspection images based on UAV, and the target recognition network is trained and verified by this data set, which can achieve more than 91% average precision. Especially, the proposed method has good accuracy in bridge crack width calculation, as it can achieve more than 90% precision compared with manual measurement.
In order to improve the reliability of bridge inspection based on UAV, we need to develop different UAV system with different loads equipment and improve the flight controllers in wind resistance ability based on more experimental results. And the complex situation of the environment wind, UAV wing rotation, and bridge near-wind has not been quantified by numerical simulation. In the future, it is necessary to improve the aerodynamic configuration design of bridge inspection UAV according to more experimental and simulation results of UAV approaching bridge.
Data Availability
All data, models, and codes generated or used during this study are included within the article.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
Acknowledgments
The research support provided by the National Natural Science Foundation of China (Grant no. 51678253) is greatly acknowledged.