Abstract
In the process of the deformation monitoring for large-scale structure, the mobile vision method is often used. However, most of the existent researches rarely consider the real-time property and the variation of the intrinsic parameters. This paper proposes a real-time deformation monitoring method for the large-scale structure based on a relay camera. First, we achieve the real-time pose-position relationship by using the relay camera and the coded mark points whose coordinates are known. The real-time extrinsic parameters of the measuring camera are then solved according to the constraint relationship between the relay camera and the measuring camera. Second, the real-time intrinsic parameters of the measuring camera are calculated based on the real-time constraint relationship among the extrinsic parameters, the intrinsic parameters, and the fundamental matrix. Finally, the coordinates of the noncoded measured mark points, which are affixed to the surface of the structure, are achieved. Experimental results show that the accuracy of the proposed method is higher than 1.8 mm. Besides, the proposed method also possesses the real-time and automation property.
1. Introduction
Deformation monitoring is vital for the safety of large-scale structure [1], and parameter calibration of instruments is essential for high-precision measurement [2]. In most mobile videometric methods, in order to achieve the global mobile measurement, the intrinsic camera parameters are calibrated in advance of the extrinsic parameters. Once the intrinsic parameters are achieved, they are assumed to be fixed. Then, the real-time varying extrinsic parameters, which reflect the pose-position relationship between the camera and the world coordinate system, are calibrated by using different methods. For instance, if the measuring camera is carried by a high-precision controllable platform, then its extrinsic parameters can be gained directly [3], or else, the extrinsic parameters should be calculated by acquiring the pose-position relationship of the camera at different positions. On the basis of the intrinsic and extrinsic parameters, the relevant measurement work is completed according to the principle of videometrics. On the one hand, although these methods can obtain extrinsic camera parameters with high precision, some of them request costly controllable platforms with high precision, and others request frequent operations with low automation. On the other hand, the environmental vibrations and other factors may influence the solution accuracy of the extrinsic parameters and change the intrinsic parameters to some extent [4]. Therefore, in actual mobile measurement works with high precision requirements, the solution error of the extrinsic parameters and the ignorance of the intrinsic parameter variation will cause large measurement error.
Many works try to overcome the influence of disturbance factors such as environmental vibration and improve the accuracy of the intrinsic and extrinsic parameters. Some works, such as Yu et al. [5], Wu et al. [6], and Wu et al. [7], compensate the camera shake referring to the known control points in the scene. Their methods effectively overcome the camera shake caused by the insufficient platform accuracy and thus ensure the accuracy of the extrinsic parameters. However, in order to ensure the stability of the error-correction result, the above methods need to perform multidirectional measurement, and it is difficult to ensure the real-time performance of the measurement. Moreover, the camera shake in the mobile measurement process will bring in significant change of intrinsic parameters. Thus, there is still a certain error existing in the measurement results. In addition, some works use auxiliary equipment, such as laser range finder [8, 9] or inclinometer [10], to overcome the interference of environmental factors. These methods achieve the real-time measurement of the extrinsic parameters. In [8, 9], laser range finder with high stability and high precision is adopted to capture the pose-position parameters of the camera. Two inclinometer sensors are used in [10] to obtain the pose change of the camera in measurement and to complete the calibration of the system parameters. However, the potential change of the intrinsic parameters of the camera is not considered in [8–10].
The self-calibration is the most commonly used method [11, 12] to overcome the variation of the intrinsic parameters. Pollefeys et al. [11] adopt the absolute quadratic surface to obtain the optimization constraint and to identify the intrinsic parameters, under the assumption that the tilt factor is zero. Wu and Wang [12] first achieve the simplified Kruppa equation by decomposing the fundamental matrix and then gain the intrinsic parameters by the constraints of two sets of equations. In contrast of the above two methods, some works [13] utilize the constraint relationship of the known coded mark points in the scene to complete the optimization of the intrinsic parameters. In [13], Batteoui et al. use parallelograms in at least five scenes to identify the constraint equations of the intrinsic camera parameters. Then, the real-time varying intrinsic parameters are obtained through optimization. In fact, this method is still a self-calibration method in essence. The methods mentioned above complete the solution of the intrinsic parameters through various constraints. However, for the mobile videometric tasks, these methods are too complicated to guarantee the real-time request.
In order to overcome shortcomings of the above works for solving real-time intrinsic and extrinsic parameters, this paper proposes a real-time calibration method with high precision based on a parameter-known relay camera. First, the real-time extrinsic parameters of the measuring camera in the mobile measurement are identified, based on the real-time pose-position relationship between the relay camera and the known control points. Second, the fundamental matrix is gained by using the two adjacent images. Referring to the constraint between the camera parameters and the fundamental matrix, the pose-position parameters of the measuring camera at the adjacent moments are captured. Then, the varying intrinsic camera parameters are real-timely calculated. At last, with the real-time intrinsic and extrinsic parameters of the measuring camera, the real-time monitoring of structural deformation is achieved.
The remainder of the paper is organized as follows. The problem formulation is given in Section 2. Section 3 presents the high-precision mobile measurement method based on relay camera. In Section 4, two experiments are shown to demonstrate the efficiency and effectiveness of the proposed method. Finally, Section 5 concisely concludes this paper.
2. Problem Formulation
In mobile videometric measurements, the measuring camera is often installed on a relatively fixed measuring platform. The camera-relevant coordinate system is built based on the relative pose-position relationship between the measuring platform and the world coordinate system. Due to limitations in production processes, experimental equipment, and existing research strengths, the accuracy of measurement results is often affected by factors such as environmental vibration and platform sway. These factors often cause solution error of the intrinsic and extrinsic camera parameters. In the following, we will carry out the error analysis from these two aspects.
2.1. Extrinsic Parameter Acquisition Error
The capture of extrinsic camera parameters in mobile videometric is a difficult problem. The extrinsic camera parameters mainly comprise the extrinsic parameters between the camera coordinate system and the world coordinate system, and the extrinsic parameters of the pose transformation between the cameras at the adjacent time. Suffering from the disturbance factors such as environmental vibration and platform sway, different degrees of interference will directly lead to different degrees of solving errors of extrinsic parameter between the camera coordinate system and the world coordinate system. The error will affect the final measurement. For example, Yu et al. [5] point out that, when the measured object distance is 20 m, a change of 1 centimeter of the camera tilt angle will cause a displacement error of 5.8 mm in the measurement object.
As shown in Figure 1, the environment vibration will cause an error between the actual camera coordinate system and the original camera coordinate system . Under the assumption that the intrinsic camera parameters are unvarying, this error will directly lead to the error of the extrinsic parameters between the camera coordinate system and the world coordinate system:
As described in the Introduction, to solve the varying extrinsic camera parameters between adjacent moments, most of the existing methods refer to a controllable measuring platform or a control camera which performs measurable pose transformation. However, the pose-position between the cameras is neither easily changed according to the controllable means set by people nor accurately measured by means of manual measurement. Therefore, the acquisition error of relevant parameters is inevitable in these methods.
As shown in Figure 2, we assume that the camera moves from the original position to the preset ideal position . However, due to the defect of the platform control, the inaccuracy of the direct measurement method, or the environment vibration, etc., the camera will be in the actual position but not the preset ideal position . We assume that the corresponding extrinsic parameters of the actual position are and , and the corresponding extrinsic parameters of the preset ideal position are and .
(a)
(b)
2.2. Intrinsic Parameter Acquisition Error
In the long-term mobile measurement process, due to the environmental vibration, rotation measurement, and other interference factors, the intrinsic parameters will change frame by frame [4]. Following the pose-position change of the measuring instrument, the image principal point will change [14], which ultimately affects the accuracy of the measurement results.
As shown in equation (2), the equivalent focal length , the scale factor, and the image principal point in the intrinsic parameter matrix are defaultly assumed unchanged in most cases. However, during the mobile measurement, the intrinsic parameter matrix will gradually change to , with the influence of environmental vibration, camera rotation, and other factors. Therefore, the invariability assumption of the intrinsic parameters during the mobile measurement process will cause obvious measurement error.
Based on the above analysis, in order to further overcome the error effects in the mobile measurement process and improve the measurement accuracy and automation of the measurement method, this paper put efforts on solving the following problems: (1)Considering the influence of factors such as measuring platform shaking, the extrinsic parameters of the moving camera are real-timely calculated with high-precision based on the relay camera. The relay camera is used to obtain the known coded mark point information in the mobile measurement process. It is assumed that the pose-position relationship between the relay camera and the measuring camera is known and unvarying, due to their relative fixed installation position in one structure. However, in practical application, much more measures such as vibration isolation are needed to guarantee this fixed pose-position relationship(2)Based on the constraint between the intrinsic and extrinsic camera parameters and the fundamental matrix, the intrinsic camera parameters are real-timely calculated after obtaining the extrinsic camera parameters with high-reliability and high-precision.
3. High-Precision Mobile Measurement Method Based on Relay Camera
The principle of the mobile measurement method based on relay camera is shown in Figure 3. First, three coordinate systems are established: the world coordinate system , the measuring camera coordinate system , and the relay camera coordinate system . Then, the motion platform is driven to measure around the outer surface of the structure, and simultaneously, the relay camera real-timely takes picture of the coded mark points on the ground whose positions are unvarying in the world coordinate system. The extrinsic parameters of the camera motion are solved firstly. At the same time, the measuring camera, which is fixed to the relay camera, can take picture of the measured mark points attached to the outer surface of the structure. After obtaining the real-time extrinsic parameters solved based on the relay camera, the real-time intrinsic parameters can be further solved by using the relevant constraints. According to the principle of the videometrics, the real-time measurement of the spatial position of the surface of the structure is completed.
3.1. Real-Time Extrinsic Camera Parameter Solution Based on Relay Camera
3.1.1. Real-Time Pose-Position Relationship between the Relay Camera and the World Coordinate System
As shown in Figure 3, the position of the target is unvarying in the world coordinate system, and the coordinate of the coded mark point on the ground is known. The intrinsic parameters of the relay camera can be precalibrated by the videometrics method [15]. Therefore, for the relay camera, the relationship between the feature image point and the corresponding coordinate in the world coordinate system can be expressed by the following homogeneous coordinate equation:
In equation (3), is the projection of the distance from the space point to the optical center in the direction of the optical axis; and are the pose rotation matrix and the translation matrix between the relay camera coordinate system and the world coordinate system, respectively. and are unknown and can be solved by the pose estimation algorithm [16–18].
3.1.2. Real-Time Pose-Position Relationship between the Measuring Camera and the World Coordinate System
The relay camera and the measuring camera are both fixed on the measuring platform, and their pose-position relationship is stable. When there is interference such as shaking in the measuring platform, the real-time pose-position relationship between the relay camera and the world coordinate system can be solved first. Then, according to the solid-state relationship between the measuring camera coordinate system and the relay camera coordinate system, the real-time extrinsic parameters of the measuring camera can be obtained, and the influence of interference is suppressed.
Let and be the positional coordinates of a spatial point in the coordinate systems and , respectively. Then, the pose-position transformation relationship between the two coordinates can be expressed as where and are the rotation matrix and the translation matrix between the two coordinate systems, respectively.
According to equation (4), the relationship between the measuring camera coordinate system and the world coordinate system can be expressed as
The relationship between the relay camera coordinate system and the world coordinate system and the measuring camera coordinate system can be, respectively, expressed as
In equation (6), and can be calibrated based on the fixed relationship between the measuring camera and the relay camera. and can be solved by the pose-position estimation algorithm described in Section 3.1.1. Therefore, according to equation (6), the relationship between measuring camera coordinate system and the world coordinate can be expressed by
The extrinsic parameters between the measuring camera coordinate system and the world coordinate system can be gained by comparing equations (5) and (7):
In equation (8), and are determined by the initial pose-position relationship of the measuring platform and can be seen as fixed constants after initial calibration. In contrast, and are varying as long as the movement of the measuring platform. At any time , they are specifically expressed as and and can be obtained by the pose-position estimation algorithm described in Section 3.1. Therefore, at an arbitrary time , the real-time pose-position relationship between the camera coordinate system and the world coordinate system can be expressed as
3.2. Intrinsic Camera Parameter Solution Method Based on Fundamental Matrix
When the measuring camera moves, it can measure the same area on the structure at different positions within a certain measuring range. Profiting from this, we firstly solve the fundamental matrix based on the polarity constraint relationship of the same feature points between the two images. Then, we use the constraint relationship between the extrinsic camera parameters and the fundamental matrix, which are solved in real time in Section 3.1, to complete the intrinsic camera parameters.
3.2.1. Solution of the Fundamental Matrix
Referring to the opposite geometry theory, the fundamental matrix can be gained by using two different images Frame and Frame of the same scene (captured from different positions). The fundamental matrix can be described by the following equation [19, 20]: where and are the homogeneous coordinates of the point in the image and image , respectively. The fundamental matrix has a dimension of , a rank of 2, and a freedom degree of 7.
For the solution of the fundamental matrix, Hartley [21] proposed the classical eight-point algorithm based on the practical application. The eight-point algorithm is simple and easy to implement. However, experiments indicate that when there is noise in the image; this method is not easy to get the accurate solution [22]. Besides, the distortion in the image may cause solution error of the fundamental matrix and affect the solution of the subsequent camera parameters.
To effectively improve the recognizing and matching accuracy of the corresponding points, we first calibrate the distortion based on the linear characteristics existing in the measurement scene by using the method in [23, 24]. Then, the robust Random Sample Consensus (RANSAC) algorithm [25] is utilized to perform the feature point matching between the two images. This strategy eliminates the mismatching in the presence of noise, thereby ensuring the solution accuracy of the fundamental matrix.
3.2.2. Solution of the Intrinsic Camera Parameters
Assuming that the corresponding homogeneous coordinates of the points on the target are and at the two adjacent moments and , respectively, then the pose-position relationship between the relay camera coordinate system and the world coordinate system can be expressed as
According to equation (12), the pose-position transformation between the camera coordinate systems at the two adjacent moments and is
Combining the pose-position relationship between the relay camera coordinate system and the measuring camera coordinate system , the pose-position transformation of the measuring camera at the two adjacent moments and is which satisfies
In the self-calibration of the camera, the following equation is commonly used to solve the fundamental matrix [26, 27]: where is the antisymmetric matrix of the translation matrix .
It can be seen from equations (10) and (14) that the fundamental matrix in equation (15) and the pose-position relationship of the measuring camera at the two adjacent moments and can be figured out. Thus, only the intrinsic camera parameters and the constant factor are unknown. Equation (15) can be transformed to
If we set in equation (17), referring to equation (14), can be real-timely solved after acquiring . Therefore, we have
The initial calibration parameter can be considered as an initial value for iterative solution. Therefore, the intrinsic parameter can be seen as known in equation (17) for an iterative method. With the fast development of the camera hardware technology, the intrinsic parameter is very close to the ideal value [22, 28] 0 in most digital camera models. Therefore, we can set . Then, we have
In equation (19), is the parameter vector to be solved. In the process of real-time monitoring using mobile cameras, it is often necessary to perform high-precision initial calibration of the camera parameters to ensure the reliability of the whole monitoring process. With equations (17) and (19), we can gain the initial approximate solution of the intrinsic parameter and the constant factor . Our goal is to find .
In order to ensure the accuracy and convergence of the solution process, nonlinear optimization method can be used for iterative solution (the LM optimization algorithm [29, 30] is used in this paper). Supposing the solution at iterative step k is , we have and
In equation (20), is identity matrix, and is the Jacobi matrix at as follows
is the residual error vector at , and is damping ratio. For the sake of simplicity, we suggest referring to [29, 30] for finding the update strategy of . After achieving , we can use to update until is less than a predetermined value such as 10-4.
3.3. The Real-Time Mobile Measurement Process of the Measuring Camera
By the solution of the extrinsic and intrinsic camera parameters in Section 3.1 and Section 3.2, we can real-timely get the camera parameters and the pose-position information of the measured structure. By detecting the same area at different time, the variation of the pose-position information of the area can be obtained, and then, the deformation monitoring of the measured structure part is completed. The specific measurement process is as follows:
Step 1. Before the measurement, the initial value of the intrinsic measuring camera parameter , the intrinsic relay camera parameter , and the constant pose-position conversion relationship and between the measuring camera and the relay camera are calibrated first.
Step 2. The coded mark points with known spatial information are measured with high-precision by the relay camera, and the real-time pose-position parameters and of the moving relay camera are obtained.
Step 3. With the parameters , , , and , the real-time pose-position of the measuring camera related to the world coordinate system can be calculated by and .
Step 4. After obtaining two different images Frame and Frame of the same scene at the adjacent moments and , the fundamental matrix can be figured out by the method in Section 3.2.1. The pose-position transfer relationship and can be gained by the method in Section 3.2.2. With the intrinsic camera parameter at the moment , the intrinsic camera parameter can be iteratively calculated, and then, the precision of the following measurement is guaranteed.
Step 5. After obtaining the intrinsic and extrinsic parameters of the measuring camera, with the bundle adjustment method in [4], the coordinate of the measured mark point in the world coordinate system can be gained with high-precision by the common coded mark points between different images acquired at the adjacent moments.
Step 6. In the measurement process, Step 3, Step 4, and Step 5 are repeated, and the coordinates of the measured mark points in the world coordinate system are gained at different moments. Then, the real-time deformation measurement of the structure is completed.
4. Experimental Results
4.1. Verification Experiment
Based on the above theoretical analysis, in order to complete the real-time deformation monitoring of large cylindrical structure, we need to set circular measure orbit around the structure. However, considering the limit of the experimental conditions, we replace the circular orbit with the parallel orbit, and this setting is adequate for the experimental validation. The images shown in Figure 4 are the measuring platform in the actual experiment and some of the pictures captured during the measuring process. The relay camera and measuring camera are fixed on the motion measuring platform. The movement of the platform is controlled by the output of the torque motor. The cameras used in the experiment are microview MVC14KSAC-GE6 with a resolution of , and the lenses are the C3516-M produced by Pentax. The intrinsic parameters of the cameras and the relative pose-position relationship between the cameras are calibrated with high-precision aforehand.
(a)
(b)
(c)
4.1.1. Extrinsic Parameter Acquisition Accuracy Experiment
In order to verify that the proposed method can effectively overcome the effects of environmental vibration and can real-timely solve the intrinsic and extrinsic parameters, the variable-control method is utilized in the comparing measurement experiments of the intrinsic and extrinsic parameters. In order to effectively verify the acquisition accuracy of the extrinsic parameters, we first use the precalibrated intrinsic parameters and ignore the influence of the intrinsic parameter variation in the comparing experiment. We, respectively, use the solution parameters of the relay camera and the rotation parameters of the torque motor to solve the extrinsic parameters, including three pose angles , , and and three translation quantities , , and . The experiment results are shown in Table 1. After obtaining the intrinsic and extrinsic parameters, we figure out the coordinates of the 20 measured mark points on the surface of the structure and then compare them with the actual coordinate information to obtain the distance deviation between the measured coordinate and the actual standard coordinate (as shown in Figure 5(a)).
(a)
(b)
We can learn from the experiment result that, when the extrinsic parameters are figured out by using the relay camera, the coordinate error of the measured mark points is not more than 2.8 mm. The minimum value is 1.5 mm, and the average error is 2.029 mm. In contrast, when the extrinsic parameters are figured out by using torque motor rotation parameters, the maximum coordinate error of the measured mark points is 3.5 mm, the minimum value is 2.2 mm, and the average error is 2.586 mm. From the experimental result, we can find that, when the intrinsic camera parameters are unified to the initial calibration value, the proposed extrinsic parameter solution method using the relay camera can effectively improve the measuring accuracy of the measured mark points.
4.1.2. Intrinsic Parameter Acquisition Accuracy Experiment
In order to verify the acquisition accuracy of the intrinsic parameters, we set the extrinsic camera parameters to the values obtained by using the relay camera in the comparing experiment. Then, we use the initial calibrated intrinsic parameters and the real-time intrinsic parameters achieved by the method in 2.2 for comparison. The comparing results are shown in Table 2. After obtaining the intrinsic parameters, we calculate the coordinates of the 20 measured mark points on the structure surface and scale the measuring accuracy by the coordinate error of the measured mark points as demonstrated in Figure 5(b).
From the experimental results we learn that, when the extrinsic parameters are consistent through the experiment, the coordinate error of the measured mark points of the proposed method is not more than 2.31 mm. The minimum value is 1.38 mm, and the average error is 1.78 mm. In contrast, when only the initial calibrated intrinsic parameters are used, the maximum coordinate error of the measured mark points is 2.74 mm, the minimum value is 1.58 mm, and the average error is 2.087 mm. It is learned from the experimental result that the measuring accuracy of the measured mark points by using the real-time intrinsic parameters is higher than that by using the initial calibrated intrinsic parameters.
4.2. Accuracy Analysis
From the verification experiment in Section 4.1, we learn that the proposed method utilizing the relay camera can not only effectively improve the accuracy of the extrinsic parameters but also achieve the real-time intrinsic parameters. To further verify the proposed method meets the request of the real-time on-line deformation monitoring for large-scale structure, we design the following experiment. We adopt the proposed method in the on-line monitoring for a metal structure surface, which contains 60 measured mark points. The measuring results of the proposed method are compared to the results of the measuring with electronic total station. The detailed comparison is shown in Figure 6.
(a)
(b)
From the coordinate error of the measured mark points shown in Figure 6(b) we can learn that the maximum, minimum, mean, and standard errors of the measured mark point coordinate are 2.38 mm, 1.35 mm, 1.797 mm, and 0.2156 mm, respectively. The mean of the coordinate measuring error is less than 1.8 mm, and the standard error is less than 0.25 mm. The coordinate accuracy of the measured mark points obtained by on-line measuring is similar to the accuracy of the verification experiment in Section 4.1. This indicates the efficiency and stability of the proposed method. The proposed method using relay camera can achieve the intrinsic and extrinsic parameters of the measuring camera with high precision and can meet the demand of the on-line deformation monitoring of the structure.
5. Conclusions
This paper focuses on the solution of the difficulties existing in the deformation monitoring for large-scale structure using mobile vision, including real-time acquirement of the extrinsic parameters, low precision of the measuring, and low level of the automation. We first achieve the real-time extrinsic parameters of the measuring camera by using a relay camera that is fixed to the measuring camera. Then, the real-time intrinsic parameters of the measuring camera are calculated with the constraint between the fundamental matrix and the intrinsic and extrinsic parameters. At last, the deformation monitoring for the large-scale structure is fulfilled with the obtained intrinsic and extrinsic parameters. The experimental results verify the validity and stability of the proposed method for solving the coordinates of the measured mark points on the structure surface. The mean of the coordinate measuring error is less than 1.8 mm, and the standard error is less than 0.25 mm, and this accuracy generally meets the needs of the deformation monitoring of large-scale structure. At the same time, the proposed method improves the automation level and the solution accuracy of the camera parameters.
Data Availability
The data used to support the findings of this study are included within the article.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
Acknowledgments
This research was funded by the National Natural Science Foundation of China under Grant 61773389 and 61833016, the China Postdoctoral Science Foundation under Grant 2019M663635, the Natural Science Foundation of Shaanxi Province under Grant 2020JQ-298, and the Shaanxi Youth Science and Technology Star Project under Grant 2021KJXX-22.