Abstract
The development of information technology and computer science has put forward higher requirements on the intelligence of deformation monitoring. We study a method based on image deformation analysis, which uses Scale-Invariant Feature Transform (SIFT) to extract image feature points after preprocessing the acquired images, applies All-Pixels Matching (APM) method to the sequence images to do further high-precision matching to achieve the accuracy of subpixels, and finally solves the deformation variables according to the relationship of the real size of the reference target and its pixel. Wavelet analysis and least squares are used to improve the image quality and matching accuracy. Based on this method, we design and develop a new remotely automated deformation monitoring system. In this paper, we introduce the algorithm principle of deformation analysis, the integration of the system, and the engineering application example of the monitoring system. The monitoring accuracy of the system satisfying 0.1 mm within 10 m and 0.8 mm within 60 m is verified in the simultaneous comparison observation according to the high-precision total station, which illustrates the effectiveness of the present deformation analysis method and monitoring system and also has the characteristics of low monitoring cost and high degree of automation.
1. Introduction
The modernization of cities has become increasingly evident in recent decades, and construction and public transport infrastructure have developed very rapidly. The structure of the infrastructure may be destroyed and lifetime reduced due to factors such as its own integrity, surrounding buildings, and human factors. If structural damage is not detected and maintained in time, it is likely to lead to major safety incidents, affecting people’s lives and causing significant property damage, so regular deformation monitoring of the stability of these engineering buildings is required to ensure the safety of construction and maintenance by taking measures in advance to deal with any problems that may arise. Deformation monitoring is the continuous monitoring of the dynamic changes, occurring in the monitoring target by means of instruments or certain methods, and the analysis of the changes in order to grasp or predict the trend of changes in the monitoring target. Horizontal displacement and settlement are important aspects of deformation monitoring; according to the monitoring results, we can directly determine the changes in the spatial location of the monitoring target.
Deformation monitoring has a wide range of applications and plays a key role in the operation and maintenance of various industries. Scholars at home and abroad have done substantial research on deformation monitoring [1]. There are many monitoring methods, which are generally divided into two types: contact and noncontact. Early dam deformation monitoring uses the normal and inverted plumb line method and contact monitoring method, installed at the core point of the dam’s positive inversion device and measuring the dam axis direction and vertical axis direction displacement [2]. This monitoring method is mainly used in some large hydraulic projects, but it also must be combined with manual readings; human error is inevitable.
Traditional deformation monitoring generally adopts the manual measurement method using a total station, transit instrument, and other measurement tools to directly observe and calculate the deformation value, and the observation time is long. Liao et al. [3] used an automated total station deformation monitoring scheme with the differential approach to achieve automatic deformation monitoring of the tunnel structure of the Guangzhou Metro. This method can achieve multipoint observation in a short time. However, the limitation of terrain conditions can directly affect the monitoring accuracy when establishing a high-precision monitoring network. Nickitopoulou et al. [4] used GNSS (Global Navigation Satellite System) to monitor the deformation of various large building structures, but the accuracy of this monitoring method and the integrity and availability of the measurement range are easily affected by the distribution of satellite systems, and monitoring cannot be implemented in places where the satellite signal is out of lock. The cost of using the total station and GNSS for deformation monitoring is relatively high.
The contact displacement measurement method generally requires the installation of sensors at the measured structure point, which is relatively difficult to implement. Noncontact displacement monitoring techniques include laser scanners, digital image correlation techniques, and other measurement methods. The laser scanning technique is used to acquire point cloud data of the monitoring area and calculate the position change of the monitoring target based on the coordinate information of these point cloud data. With the use of laser scanning technology for displacement monitoring, single point positioning accuracy can reach the millimeter level, but it is more difficult to achieve monitoring accuracy of the submillimeter level. Laser scanners are expensive. The amount of point cloud data is relatively large, and the data processing is rather cumbersome.
With the development of computer vision and information technology in the 21st century, digital image-related technologies are more often applied to deformation monitoring. Image deformation monitoring uses image analysis, image denoising, coarse matching, and global image matching with high accuracy to solve the deformation value of the monitored target object. The image-based noncontact monitoring method not only greatly reduces monitoring costs and improves visualization compared to other monitoring techniques but also improves monitoring accuracy. It can work in harsh environments without causing damage to the monitoring target. The improvement of the monitoring effect of this method is mainly reflected in the improvement of the image analysis algorithm. Mahajan et al. [5] used the grayscale difference method and genetic algorithm for displacement strain measurement with accuracy up to the subpixel level. Hou et al. [6] from Tsinghua University used multiple calibration methods to conduct experiments and developed a two-dimensional displacement measurement system which can achieve displacement measurement flexibly. Based on digital image processing technology, Li et al. [7] affected dynamic processing of two-dimensional displacement by comparing pixel changes at the center of an image. Luo et al. [8] used a gradient-based template matching algorithm to complete multipoint displacement monitoring of bridges. Fan et al. [9] proposed a networked automatic deformation monitoring system that can be used for scattered deformation monitoring projects. Zaminparda et al. [10] developed an integrity risk formulation based on the interaction between estimation and testing and proposed a risk assessment method for deformation monitoring systems to quantify the monitoring performance of the system after a change in the critical threshold. In recent years, many researchers have also adopted this method to monitor different monitoring targets and achieved better monitoring results, but the computational complexity of this method is high, and the real-time monitoring will be affected as the image pixels are upgraded; enhancing the hardware and software feature matching and information processing capabilities to improve monitoring efficiency is an important direction for current development [11–13]. The current noncontact methods of deformation monitoring are generally more data processing intensive and susceptible to the influence of the surrounding environment [14].
Considering the advantages of image-based deformation monitoring methods in implementation, accuracy, and visualization, this paper integrates digital image correlation technology, computer vision, and remote transmission technology; studies an image-based deformation monitoring method, and develops a remote deformation monitoring system, which can achieve image acquisition, image analysis, and deformation result display. The accuracy and practical value of the method and system are verified through experiments and engineering applications.
2. Methodology
The general idea of the image deformation monitoring displacement solution is to use the first image as the reference image and the subsequent sequence of images acquired are registered with the first image with high accuracy, respectively. The offset values of these sequence images are calculated relative to the first image. An image can be defined by a two-dimensional function , where is the image plane coordinates. Sampling and quantization of the image can be converted into digital form to obtain a pair of M rows and N columns of the image, M ∗ N, that is, the size of the image expressed in the form of a matrix in Figure 1.
This paper is based on digital image correlation technology of two-dimensional deformation monitoring, including horizontal direction and vertical direction. The reference and target image are shown in Figure 2. Points and in the reference image are corner points; after deformation, they are shown as points and in the target image. This shows the amount of change of and with respect to the original image in the x-direction , which is the horizontal displacement of the monitored target in the image, and the amount of change in the y-direction , which is the vertical displacement of the monitored target in the image.
The main process of the method in this paper includes image preprocessing, deformation analysis, and displacement calculation. The method flowchart is shown in Figure 3.
2.1. Image Preprocessing
Image preprocessing adopts filtering analysis and histogram equalization method to initially remove noise, improve image quality, and facilitate subsequent feature matching. The image is affected by the acquisition equipment and the environment; there will inevitably be some noise interference. It is necessary to filter and analyse the image to eliminate noise and remove the interference of other signals in the image. The quality of the image will deteriorate when in dark or uneven lighting environments. The histogram equalization method is chosen to enhance the image contrast and further improve the quality of the image.
2.2. Deformation Analysis
Image feature point matching is the basis of deformation analysis. This paper uses the Scale-Invariant Feature Transform (SIFT) and All-Pixels Matching (APM) algorithm for feature point matching to realize image registration [15–17]. APM is a geometric matching method to find the optimal solution from all pixels. This method carries out the global geometric matching of the image. On the basis of not limiting to feature point matching, all the pixels of the image are selected to participate in the matching, and the optimal solution is found from a large number of data, solving the required geometric uncertainty with the amount of information of the global signal. The proposed method first uses the SIFT algorithm to find the better-described feature points and further uses the APM geometric matching method on the sequence image on its basis. In order to facilitate the search for mathematical relationships between the sequence image grayscale and their geometric transformation, the following corresponding equation of the image’s gray level is listed as follows:where denotes the grayscale of the reference image at point , denotes the grayscale of the target image at point , and denotes the noise in the image’s grayscale.
The corresponding grayscale equation is expanded into a differential equation that can be solved, and is expanded into differential form as in where and denotes the geometric parameter and is the sum of the approximation and the differential components. is expanded into a fully differentiated form as in where and denote the grayscale gradient at the point , and then the geometric model is solved according to the transformed differential form equation and the corresponding grayscale value signals and of the sequence image. In this process, it is necessary to satisfy that the amount of information provided by all image elements involved in matching is greater than the uncertainty of the geometric model in order to ensure the accuracy of the results and improve the accuracy of the solution, and the image geometry matching information inequality is expressed as where is the amount of information provided by all image elements; is the uncertainty of the geometric model function to be solved, is the area composed of all image elements, and are the variance of grayscale image and image noise, respectively, and is the amount of information contained in the average image element. The geometric parameter solution is completed under the principle of satisfying the information theory image matching in equation to determine the uncertainty problem between the image and the parameter to be solved and to achieve the purpose of improving the image matching reliability and solving efficiency. Then, all pixels within the initial matching range are jointly involved using the principle of least squares decreeing the least sum of squares of image grayscale differences to achieve subpixel accuracy and obtain the best matching point location for high accuracy matching.
When calculating the offset and after image matching, as the equipment itself and the field will be affected by various vibrations, there will be light and temperature differences, and these accidental and systematic errors will be reflected in the initial calculation of and . A second of filtering of the measured offset values is required to eliminate systematic and accidental errors and obtain more accurate offset. Fourier Transform (FT) and wavelets can be used for filtering analysis. In this paper, wavelets with better effects are selected for filtering analysis. FT is the decomposition of a signal into a superposition of several continuous sinusoids of different frequencies, and for the study of local signals, the transformation is carried out by means of a small window of the signal. In practice, the acquired image signals also often have many nonsmooth signals, and FT loses the time information and is only suitable for processing some smooth signals. Wavelets are a decomposition of the signal into a superposition of multiple wavelet functions, which can clearly express the time-frequency localization characteristics of the image. Furthermore, on the basis of FT, it can also manage nonsmooth signals, so that the irregular wavelet function at a sharply varying signal performs better than a sinusoidal curve. For a one-dimensional noise-laden signal model, we can express the following equation:where is the true signal, is the noisy signal, is the noise, and σ is the noise standard deviation. Wavelets act as a microscope on image processing, with their multiresolution decomposition ability to strip out information from the image layer by layer. When using wavelets to denoise a one-dimensional signal, the first step is to perform wavelet decomposition, select a wavelet, and determine the level of decomposition. Then, we choose a threshold quantization method for the high-frequency coefficients of the wavelet decomposition. For hard threshold denoising, when the absolute value of the wavelet coefficients is less than a given threshold, the wavelet coefficients are considered to be caused by noise and thus they are discarded as 0. Wavelet coefficients greater than or equal to the threshold are caused by a useful signal and are retained, as in
For soft threshold denoising, wavelet coefficients with absolute values greater than or equal to a given threshold are retained as the result and discarded as 0 if they are less, as in where is the wavelet coefficient, is the given threshold, is the retained wavelet coefficient, and is the mathematical operation function. It returns 1 if the parameter is greater than 0, 0 if it is equal to 0, and −1 if it is less than 0. The threshold value is determined using the fixed threshold method, where the noise variance is first found, and then the threshold value is calculated by using the fixed threshold formula, as in equation (8), where is the signal length. After the threshold value is quantized, the noise can be eliminated by one-dimensional reconstruction based on the lowest low-frequency coefficients of the wavelet decomposition and the high-frequency coefficients of each layer [18]:
The wavelet transform of the image goes through low-pass and high-pass filters to separate the image information at a deeper level. The image signal is decomposed as a two-dimensional signal in the horizontal and vertical directions to achieve wavelet multiresolution decomposition, which is divided into the following subregions by the two-dimensional wavelet transform decomposition. As shown in Figure 4, L is short for “Low” frequency and H is short for “High” frequency. LL1 is the low-frequency information, which is an approximate subimage of the original image. HL1, LH1, and HH1 are detailed images, HL1 is the high- and low-frequency information, LH1 is the low- and high-frequency information, and HH1 is the high-frequency information.
The four subbands in Figure 4 are wavelet coefficients convolved by different filters. Most of the noise is concentrated in HL1, LH1, and HH1. Wavelet coefficients are applied to these regions and then the signal is reconstructed.
Due to the high-frequency dynamic monitoring method, each monitoring target matching key point in the image acquisition sequence image will have a corresponding and variation value. The offset is calculated for all the image monitoring targets, and the global signal will have a reference range value. The filtering analysis is used to first exclude the false offsets that are obviously out of range, and then the wavelet analysis is used to obtain the final accurate offset. The results from a small number of image calculations are less reliable, and it is on the basis of a large number of sequential image processing analyses that this method is also used to obtain the signal correlation patterns.
2.3. Displacement Solution
After the feature matching of the image is completed, the maximum correlation coefficient method is used to calculate the displacement, as shown in where is half the size of the image subregion; is the pixel grayscale of the distorted reference image subregion, and is the pixel grayscale of the distorted target image subregion. is the mean pixel grayscale of the reference image subregion, and . is the mean grayscale of the pixels in the subregion of the target image .
After obtaining the exact pixel offset in the image, the actual displacement change of the monitoring target can be obtained by calculating the deformation parameters based on the real length of the monitoring target and the size of the pixels occupied in the image, and then the final monitoring result is displayed in the form of a visual curve to complete the displacement solution of the image.
3. Image Deformation Monitoring System
Based on the principle of the image deformation monitoring method, this paper carries out the development of the image deformation monitoring system. The image deformation monitoring system consists of hardware system monitoring equipment and software system, which use optical imaging to monitor the target unattended and at regular intervals.
The technical flow of the image deformation monitoring system is shown in Figure 5.
The hardware of the image deformation monitoring system includes a camera, video capture card, 4G transmission module, power supply, and a vernier measuring target, which is used to obtain images of the monitoring target. It can choose a camera with the right focal length according to the monitoring distance to ensure that the captured images are clear. The specification parameters of the lens in the deformation monitoring system are shown in Table 1.
The system collects image data in real time through the monitor and transmits it to the server remotely. The monitoring software on the server finally obtains the displacement amount of the monitored target through the process of automatic identification, information extraction, and analysis of the target on the serial image and displays the displacement amount of the target with visual curves and remote query access. In practice, the monitoring equipment is installed vertically against the target or monitoring area, and the network port is connected to the computer to display the monitoring image area. The image is clearly captured by focusing, and then the monitoring equipment can automatically capture and upload to the server in real time according to the requirements after setting the parameters such as the shooting interval and working period. The monitoring frequency of the image monitoring equipment can be greater than 10 sheets/10 s, using 4G network transmission in a variety of monitoring environments for 24-hour unattended automatic monitoring, with strong controllability and high engineering applicability. Figures 6 and 7 show the hardware system diagram in the actual engineering application.
(a)
(b)
The vernier measuring target consists of an image monitoring target and a 10-degree vernier measuring scale, which can be manually controlled to move the length of the target, and the reading of the moving target can be accurate to 0.1 mm, making it easy to test. The vernier measuring target also has good results in practical monitoring.
The software system is a self-developed image processing software called Monitor. The development environment used is Microsoft Visual Studio with OpenCV. The main function of the software is to fully automate the processing and analysis of remotely transmitted images. Figure 8 is the software interface diagram, including an image display area and a result display area. The picture area of the black rectangle above the interface shows the monitoring target. The curved areas in the black rectangle below the interface show the left, right, and up-down directions of the displacement.
Not all the information of the whole image is needed in the image deformation monitoring system. In the image key point matching, we choose the target region and the specific monitoring target of the monitoring region as the region of interest (ROI) and select the feature points of the reference region for displacement solving, which significantly reduces the amount of data calculation and enhances the processing accuracy while improving the algorithm processing efficiency. The red area in Figure 9 is the ROI area selected when the software acquisition scope function is completed.
4. Accuracy Verification
4.1. Accuracy Test
The accuracy test was carried out by observing the displacement changes of the same target with the total station and deformation monitoring equipment at the same time, and the experimental verification was carried out in the hallway of the laboratory on the first floor of the teaching building of Capital Normal University in Beijing, China. According to the actual displacement of the target, the deviation comparison between the calculated displacement of the total station and the measured displacement of the deformation monitoring system is used to verify the monitoring accuracy of the deformation monitoring system. The total station model is Leica TS30, with a range accuracy of 0.6 mm + 1 ppm and angle measurement of up to 0.5″. The experiments were carried out in both horizontal and vertical directions. Two L1 and L2 targets with vernier scales were placed at 5 m and 8 m from the monitoring equipment, and the targets were matched with the image monitoring equipment and total station, respectively. The displacement of the targets in both horizontal and vertical directions was manually controlled, moving every few minutes and recording the observation time. The deformation monitoring experiment is shown in Figure 10.
The observations were made according to the changes of the target and 15 sets of data were recorded for accuracy verification. Taking the L1 target as an example, the specific results are shown in Tables 2 and 3, where the target displacement is recorded as S1, the total station displacement as S2, the image deformation monitoring system displacement as S3, the total station deviation as P1, and the image deformation monitoring system deviation as P2.
The accuracy of the image deformation monitoring system in feature matching is 0.1 pixels, and the feature extraction process can reach the subpixel level, which greatly improves the actual accuracy of deformation calculation. According to the experimental results of calculation, the Root Mean Squared Error (RMSE) of the L1 target horizontal total station was 0.17 mm with an average deviation of 0.15 mm, and the RMSE of the image monitoring system was 0.15 mm with an average deviation of 0.04 mm. The RMSE of the vertical total station was 0.06 mm with an average deviation of 0.14 mm, and the RMSE of the image was 0.09 mm with an average deviation of 0.06 mm. The RMSE of the L2 target horizontal total station was 0.17 mm, the RMSE of the image was 0.12 mm, the RMSE of the vertical total station was 0.17 mm, and the RMSE of the image was 0.13 mm. The deviation of the image deformation monitoring was basically within 0.1 mm, and the accuracy of the image deformation monitoring equipment can be determined from the calculation results of the displacement deviation of the deformation monitoring system, the total station, and the “Building Deformation Measurement Regulations” to meet the requirements [19]. The monitoring accuracy of this deformation monitoring system can reach 0.1 mm within a distance of 10 m, and the monitoring effect is better than other deformation monitoring systems of the same level.
In addition, in order to verify the monitoring accuracy of the image deformation monitoring system at a long distance, a test was conducted at a distance of 60 m on the roof of an outdoor building with reference to the indoor monitoring scheme. The results show that the monitoring error of the image deformation monitoring system is about 0.3 mm at a monitoring distance of 23 m and less than 0.8 mm at 60 m. Figure 11 shows the experimental diagram of the monitoring equipment on the roof of the building.
4.2. Engineering Applications
The deformation monitoring system has been put into a number of practical engineering project applications and plays an important role in the operation and maintenance of the project. It has been successfully applied to the inspection of the glass screen wall of Qingdao North Station in Shandong Province in China and the monitoring of the deformation of the roof’s steel frame structure and the bearing beam in Qingdao West Station in China, which are in good operating condition.
In the application of Qingdao North Station, it is necessary to monitor each pane of the glass curtain wall and obtain the changes of each pane in real time. When using the deformation monitoring system to monitor the glass curtain wall, by numbering each pane, the deformation of any pane and the historical image can be viewed after processing and analysis by the software system, which has a good monitoring effect, and the monitoring site is shown in Figure 12.
For the actual site in a separate pane, we use the installation of monitoring targets to monitor it; the monitoring system acquisition frequency is set to the same time every day, 28 days of continuous acquisition of the target area image data, listed as one of the panes shown in Figure 13(a); the horizontal coordinates indicate the monitoring date, the vertical coordinates indicate the monitoring target in the horizontal direction x and vertical direction y displacement changes, and displacement monitoring results are shown in Figure 13(b). Through the monitoring curve and the result data, we found that the monitoring target on the third day had a large deformation in the vertical direction. By comparing the image results and the site inspection, we found that this deformation may be due to a single vibration caused by the external environmental changes, and the monitoring target did not have obvious deformation in the whole monitoring range.
(a)
(b)
In another application of the high-speed railway station, it is necessary to monitor the deformation of some steel-roof steel frame structures and rail beams in the station, to obtain their change amounts and judge their stability. This monitoring also adopts the target monitoring method, through which the image deformation monitoring system continuously monitors the deformation of different target observation points fixed on the steel frame and the bearing rail beam and automatically derives the displacement change curve to observe the change in real time. In addition, the image deformation monitoring system is also applied to monitor the deformation of the indoor roof of the high-speed railway, the settlement observation of the track bed of Jing-Zhang high-speed railway, and the foundation settlement monitoring of Nanjing railroad, even including the real-time monitoring of geological slope landslide collapse. The image deformation monitoring system has played a better role and provided an important reference for the operation inspection department.
After various engineering applications proved that this monitoring method using digital image correlation technology can effectively solve the problems in engineering, this image deformation monitoring system has some advantages compared with other monitoring systems. One is to carry out automatic noncontact high-frequency monitoring, which is not affected by human factors and can significantly improve the real time. Another advantage is the displacement calculation efficiency, whose monitoring effect is good. Moreover, the equipment is lightweight with low hardware and labour costs, which makes it more convenient for popularization and mass production.
5. Conclusions and Discussions
Based on computer science and digital image correlation technology, an image deformation monitoring method has been proposed. The method uses SIFT to perform image coarse matching and APM fine matching methods to achieve high-precision feature matching to calculate pixel displacement based on image filtering and denoising. With the help of the actual length of the monitoring target and the deformation parameters calculated by the pixel size in the image, the actual displacement can be obtained. On the basis of the proposed method, an image deformation monitoring system has been developed independently to address the problem of deformations arising from engineering applications, using digital image analysis methods to achieve remote deformation monitoring through image processing, deformation analysis, and displacement resolution. The displacement measurement accuracy of the system is verified by comparison with high-precision total station displacement measurements. Based on the deformation analysis method used in this paper, the monitoring accuracy of this system can reach 0.1 mm within 10 m, the monitoring error at 23 m is about 0.3 mm and less than 0.8 mm within 60 m, and the monitoring effect of this method meets the requirements of deformation monitoring. Meanwhile, the image deformation monitoring system is more automated than other monitoring systems and can achieve high-efficiency real-time monitoring at a low cost. It has performed well in engineering applications such as the glass curtain wall of the high-speed railway station and the steel-roof frame structure deformation monitoring. The system can become an important tool for monitoring small deformations in engineering applications, has more room for development, and is suitable for promotion.
The image deformation monitoring system requires the monitoring equipment to be matched vertically with the monitoring target and can only monitor the deformation displacement in two directions in the two-dimensional plane, and the deformation in the front and rear directions cannot be monitored for the time being. When the monitoring equipment is installed unstably or the monitoring area is tilted, the monitoring accuracy will be affected to a certain extent. Future research will be based on the current application of the monitoring equipment needs and optimization to make improvements whereas, for image quality and feature matching to continue to improve the accuracy, we consider an optional infrared camera for night monitoring and add a three-dimensional solution function with two lenses, in order to achieve more relevance to the actual engineering applications.
Data Availability
The data used to support the findings of this study are included within the article.
Conflicts of Interest
The authors declare that there are no conflicts of interest regarding the publication of this paper.
Acknowledgments
The authors are grateful to the Beijing Municipal Education Commission for their support of this research project. In addition, they thank Yaocheng Ma of the Beijing Urban Construction Survey and Design Institute and Mr. Francis of Capital Normal University for their valuable advice on this paper. This study was supported by the General Scientific Research Projects of Beijing Municipal Commission of Education (KM202010028012).