Abstract
How to carry out fast and reliable dynamic deformation monitoring of building structures is an important issue for engineers and technicians. Digital close-range photogrammetry technology has the advantages of noncontact, multipoint, and rapid monitoring, and the application of UAVs makes the monitoring task no longer limited by the size of the field, but the current technology can not guarantee the rapid deployment of UAVs in dynamic deformation monitoring, and these methods commonly have higher requirements for image quality, and cannot ensure the stability of accuracy. To this end, we propose a dynamic displacement monitoring method for UAVs based on digital close-range photogrammetry technology and optimize the data processing process for the problem that there is still moving when the UAV hovers. A building with an overhead corridor structure with a large flow of people was selected for the experiment. The experiment used ground digital cameras and drones to monitor separately to compare the monitoring results. The results show that the proposed method is simple and easy to use. The monitoring method can be deployed quickly, which is expected to be applied to emergency monitoring scenarios.
1. Introduction
On May 5, 2020, wind vibration occurred on the Humen Bridge (Figure 1), when the natural wind speed was 10–16 m/s. The wind-induced vibration was diminished after removing the water horse on the bridge side. On May 6, the Humen Bridge vibrated again for an unknown cause. This Humen bridge passage appeared for the first time in 23 years, with nearly half a metre of wind vibration amplitude of the phenomenon. On May 12, experts published bridge vibration reasons: continuous set along the bridge across the fence water horse changed the steel box girder of aerodynamic shape. In particular, the wind environment conditions, the bridge of vortex vibration phenomenon and said the key bridge component is not an exception occurs, the bridge structure overall security. On May 16, the Humen Bridge was reopened to traffic. On May 18, 2021, an uncommon vibration occurred in the Shenzhen SEG Plaza (Figure 2) at the wind speed of about 10 m/s, and the market management department and the local government quickly decided to evacuate and suspend the operation. On July 15, an expert group organized by Shenzhen Municipal Housing and Construction Bureau came to a conclusion. The most direct cause of the sensible vibration of the SEG Plaza is the coupling of vortex-induced resonance caused by the wind of the mast and the change of dynamic characteristics of the building and the mast. Authorities removed the mast and restored the accumulated damage.
It can be seen from these two cases that the dynamic deformation of buildings can reflect the safety performance of buildings to a certain extent. Therefore, engineers and technicians need to obtain the abnormal deformation of the structure before the accident and give early warning of the danger, to protect the safety of people’s lives and property. In addition to the need for construction of the low frequency and static deformation (such as building settlement and inclination) monitoring, the need for building dynamic deformation (such as building under strong wind and earthquake deformation) monitoring, and dynamic deformation always occurs quickly, so how to a rapid and reliable method for dynamic deformation monitoring of the building, apparently it is of great significance.
Conventional deformation measurement technology mainly includes using the total station and other traditional instruments for deformation measurement, which is suitable for dynamic deformation measurement with high precision requirements, long deformation period, and low transformation rate. For buildings with the short deformation period and high deformation rate, displacement sensors and acceleration sensors can be installed. Currently, it is possible to install hundreds of sensors in a masonry structure and get data in real-time from different places performing continuous monitoring [1]. However, this is a contact measurement, and the sensor will cause irreversible damage to the monitored structure when installed. Using GNSS (Global Navigation Satellite System) to monitor the deformation of various large building structures, but the accuracy of this monitoring method and the integrity and availability of the measurement range is easily affected by the distribution of satellite systems. Monitoring cannot be implemented in places where the satellite signal is out of lock [2].
Using the digital close-range photogrammetry technology to monitor the dynamic deformation of buildings can achieve high frequency, multi-point, and contactless observation and has been successfully applied to Bridges [3, 4], shuttle steel shelves [5], Masonry Walls in Seismic Oscillation Outdoors [6], Steel structure building [7], super high rise building [8], and so on. The technologies of measuring instrument is a digital camera, digital camera in these experiments showed high-frequency measurement and monitoring of the advantages of stable performance, but due to the digital camera can only be placed on the solid ground, so in the individual monitoring scene, this method also showed the perspective and monitoring field of constrained problem. Although the monitoring results can be modified through the Small acute Angle Method [9], in practice, the monitoring flexibility is still largely subject to the field environment. In [10], we proposed The Improved Zero-centered Motion Parallax Method with high-frequency monitoring without training data. However, in practical application, it is found that UAV still has movement problem after hovering, which has the influence on monitoring results. Therefore, we propose a data processing optimization method. To verify the validity of the method, we selected a building for the experiment. The elevated corridor of the building was taken as the primary research object. A Digital camera and a UAV were used to monitor their dynamic deformation. The experimental results show that the displacement trends and laws of the UAV monitoring results after data processing optimization are the same as those of the digital camera monitoring results, indicating that the data processing optimization method proposed in this paper is effective.
2. Related Works
With the development of UAV technology, it is possible to use UAVs to monitor the deformation of buildings [10]. At present, there are abundant research studies on UAV Assisted Structural Health Monitoring (UASHM) [11]. However, when a UAV monitor the target’s dynamic displacement, it will inevitably encounter the problem that the UAV still moves after hovering. To reduce the impact of this problem on the monitoring results, relevant scholars have carried out some targeted studies on the technologies used.
Weng et al. proposed a homography-based approach to estimate the dynamic displacement of a structure from video taken by a UAV-mounted camera [12]. The transformation matrix is calculated by the coordinates of homonymous feature points in different images to weaken the influence of UAV movement on the monitoring results, but the technology is susceptible to the influence of illumination transformation. Hoskere et al. proposed the method of using a UAV to monitor the vibration of civil foundation buildings [13]. In order to improve the accuracy of UAV monitoring, they divided the monitoring area into monitoring areas one by one and used the ArUco marker to estimate the camera attitude to weaken the movement of the UAV. They use high-pass filters to remove low frequency vibrations. The study was eventually tested on the Little Golden Gate Bridge (suspension bridge) in Muhammad, Illinois and compared with the results recorded by an accelerometer mounted on the bridge, it was concluded that the method was effective in monitoring structural vibration frequencies. Catt et al. developed a UAV to carry digital image correlation cameras and showed a system that can measure deformations of structures [14]. The movement of the UAV also affects the monitoring of feature points, so this study by Catt et al. adopts the manual tracking method to reduce errors. The method proposed in this paper does not need specific marks, only the reference points and monitoring points can be identified. The monitoring method proposed in this paper can be implemented quickly, so it is expected to be used for emergency monitoring.
3. Measurements and Methods
3.1. Introduction of Measurements
The measuring instrument used in this experiment is a UAV (Figure 3) and a digital camera (Figure 4). Tables 1 and 2 show the specific parameters of this UAV and digital camera.
3.2. The Zero-Centered Motion Parallax Method
As shown in Figure 5, there are three virtual planes, which are the image plane, the object plane, and the reference plane. The camera sensor is located on the image plane, and all the monitoring points are located on the object plane, also called the monitored plane. All the reference points are located on the reference plane. The function of the reference plane is to eliminate systematic errors [10].
Suppose a monitoring point on the object plane move from position A to position B, which results in the displacement of the X-axis and of the Z-axis on the object plane. Meanwhile, the monitoring point from position to position , the X-axis, and the Z-axis also have displacements and on the reference plane.
If the monitoring point on the object plane is moved from to , its deformation and on the reference plane arewhere is the principal distance of the photo, where y is the distance of the image plane and the reference plane. and are the horizontal and vertical deformations of the monitoring point on the image plane. and are the horizontal and vertical deformations of the monitoring point on the reference plane.
Although UAV can make observation Angle more flexible, it also brings more systematic error, and the most significant systematic error is the drift error of UAV hovering. For the system error caused by UAV hovering, adjustment of kinematic observations is proposed to weaken the system error.
Strictly speaking, the photos before and after deformation are always taken under different elements of interior and exterior orientation, the interior orientation elements in a photo are the elements for restoring the shape of the photographic beam, and the exterior orientation in a photo are the elements for determining the position and orientation of the photographic beam in the space coordinate system [17]. In the experiment, we keep the UAV as stable as possible and set the digital camera in UAV mode to “manual.” Therefore, the error caused by the inconsistency of elements of interior and exterior orientation will have a slight impact on the monitoring results; we weaken the theoretical error to a certain extent by setting control points. To be exact, we weakened ; the specific process is as follows.
On the reference plane, if corresponding monitoring points in the zero image and successive image are and , compared with the ideal image which without errors of camera external and internal parameters, systematic errors of corresponding monitoring point are and , respectively. The equations can be expressed as follows:
If there are errors of camera external and internal parameters between the zero image and successive images, the control points located on the reference plane will generate parallax as follows:
The parallax and the control point must be caused by the errors of camera external and internal parameters in successive zero images, and we take parallax for example, it can be expressed as follows:where and are errors of zero image and successive images. The detailed derivation process is shown in reference [18]. From equation (4), we notice , assume the difference between the errors of zero image and successive images as follows:
Then, equation (4) can be expressed as follows:
After sorting out equation (6), it can be expressed as follows:
Because motion is caused by the change of camera external and internal parameters in the successive and zero images, we can correct . with a sufficient number of control points. Differently, motion . is caused by the interaction of . , , and camera external and internal parameters in the successive image. However, and are different at each point, control points cannot be used to correct, so we only discuss as follows:
We can express equation (8) as follows:
If there are more than five control points, each unknown coefficient can be obtained according to their. We assume the correction of is, so the error equation is
For convenience, we selected the linear part of the equation (10) for processing, as follows:
In this case, we only need three or more reference points to obtain and . Take as an example, When contains only occasional errors, equation (11) can be expressed as follows:where is the differential coefficient of . The error equation is
The equation of the composition method is
Calculate barycentric coordinates by control points on the reference plane, as follows:
Because coordinates of control points are barycentric coordinates, and the parallax coefficient in the X direction as follows:where and . Similarly, we can obtain the parallax coefficient and in the z direction. Then, we can obtain and . Finally, we figure out the value of as follows:
The displacement of the monitoring point on the reference plane after the first correction is
On the basis of the above work, we use reference points on the reference plane to further reduce the error, assume the coordinates of the 4 reference points arewhere is the photo number.
If the UAV moves slightly and passively on a plane parallel to the object plane, the positions of the 4 reference points in the image coordinate system will inevitably change, and this change can be calculated under ideal conditions as follows:
We need to subtract these two values and before we calculated the displacement of the monitoring point in the object plane.
Finally, the displacement and of the final monitoring point can be obtained through the proportional conversion relationship as Figure 6 shows.
4. The Elevated Corridor Experiment
The height difference of the elevated corridor is about 20m to 25m above the ground; the digital camera on the ground cannot observe it vertically. It can only monitor it from a low-angle as Figure 7(a) shows, which cannot meet the requirement that the monitoring surface is parallel to the control surface, but the UAV do as Figure 7(b) shows.
(a)
(b)
4.1. The Elevated Corridor Experiment Process
We have eight control points (C1 – C8) on the surface of the building, and on the ground, six monitoring points (U0–U5) were selected on the elevated corridor, as Figure 8 shows.
(a)
(b)
(c)
The UAV takes off and adjusts its altitude and attitude to ensure that all points are evenly distributed across the camera’s image; the UAV hovers and starts recording. In this process, the ground surveyor used a steel ruler to measure the relative lengths of 4 points (C5 – C8) on the ground, and measured the relative distance of 4 points (C1– C4) on the building surface with a total station as Figures 8(a) and 8(b).
4.2. Results and Discussion
Figure 9 shows monitoring result figures obtained from a digital camera for these points.
(a)
(b)
(c)
(d)
We can get the following results from Figure 9:(1)The monitoring points show an elastic trend, which we regard as vibration(2)The vibration ranges and frequency of these monitoring points along the X-axis and Z-axis are similar, as Figures 9(a) and 9(c) show(3)The displacement of the monitoring point on the X-axis is more dramatic than the displacement on the Z-axis
Figure 10 shows monitoring result obtained from UAV for these points. The upper and lower limits of the pixel deformation values of the monitored points in Figures 9 and 10 are different because the UAV is farther away from the object plane than the digital camera.
(a)
(b)
(c)
(d)
We can get some of the same results as the digital camera experiment:(1)The deformation of each monitoring point is elastic.(2)The vibration ranges and frequency of these monitoring points along the X-axis and Z-axis are similar. The deformation trends of each point on the X-axis and Z-axis are the same.(3)The deformation interval of the monitoring point on the X axis is twice that on the Z-axis.
There are few differences between the results of the two experiments; obtained the same results on the whole; that is, the monitored structure has good elasticity [19]. Based on experiments, we summarize the performance comparison of digital camera and UAV in Table 3.(1)Hover error (DJI Phantom 4 Pro): Vertical: ±0.1 m, Horizontal: ±0.3 m (Visual positioning works).(2)Maximum allowable wind speed (DJI Phantom 4 Pro): 10 m/s [15].(3)The price of sensors and the UAVs with interchangeable sensors are high, generally.
5. Discussion
5.1. Limitations
(1)Although the method improves the deployment speed of monitoring, it is still necessary to set more than four reference points in practical applications. Otherwise, it is impossible to reduce the error and calculate the true displacement.(2)The proposed data optimization method is only suitable for the slight movement of the common UAV. If the UAV has a large movement in the monitoring process, it is necessary to choose other processing methods.(3)In the experiment, because only one UAV is used for monitoring, it is impossible to measure the displacement outside the plane, and the use of two UAVs to monitor at the same time from different perspectives will be expected to achieve the displacement of the measurement monitoring point in all directions.
5.2. Future Works
For the motion problems that still exist when drones hover, most of the studies used to eliminate this effect during data processing, which has been briefly introduced in the “Related Works” section. There are currently a certain number of studies to improve the stability of drones, such as Zhang Wei et al. propose various deep learning-based methods for machinery fault diagnosis [20, 21]. Lianghao Hua et al. propose a four-rotor UAV sensor fault diagnosis and fault-tolerant control based on a genetic algorithm. It is also pointed out that the fault adjustment method can be designed in the future to accelerate the response to the fault [22]. Zhong et al. propose a robust actuator fault detection and diagnosis (FDD) scheme for a quadrotor UAV (QUAV) in the presence of external disturbances. It is also pointed out that the estimates of actuator faults and external disturbances will be conducive to improving the system performance [23]. Optimizing the data processing process on the basis of improving the stability of the rotorcraft UAV is expected to make the monitoring results more accurate.
6. Conclusions
Digital close-range photogrammetry technology has the advantages of noncontact, multipoint, and rapid monitoring, and the application of UAVs makes the monitoring task no longer limited by the size of the field. The paper proposes a dynamic displacement monitoring method for UAVs based on digital close-range photogrammetry technology and optimize the data processing process for the problem that there is still moving when the UAV hovers. The method proposed in this paper does not need a lot of preparation for monitoring targets in advance and has the characteristics of rapid deployment. The experimental results show that the monitoring results obtained by the UAV after data processing and optimization are the same as those obtained by the digital camera installed on the ground.
Data Availability
The image data used to support the findings of this study are available from the corresponding author upon request.
Disclosure
Yongquan Ge and Xianzhi Yu contributed equally to this work, they are cofirst authors. Yongquan Ge and Chengxin Yu are the first and second corresponding authors, respectively.
Conflicts of Interest
The authors declare that there are no conflicts of interest regarding the publication of this paper.
Acknowledgments
This paper was supported by the Science and Technology Project of the Shandong Province of China (Grant no. 2010GZX20125).