Feasibility Study of Earthquake-Induced Damage Assessment for Structures by Utilizing Images from Surveillance Cameras

Zhou, Jing; Huo, Linsheng; Huang, Chen; Yang, Zhuodong; Li, Hongnan

doi:https://doi.org/10.1155/2024/4993972

Structural Control and Health Monitoring

On this page

Abstract Introduction Conclusion Data Availability Disclosure Conflicts of Interest Acknowledgments References Copyright Related Articles

Research Article | Open Access

Volume 2024 | Article ID 4993972 | https://doi.org/10.1155/2024/4993972

Feasibility Study of Earthquake-Induced Damage Assessment for Structures by Utilizing Images from Surveillance Cameras

Jing Zhou,¹Linsheng Huo,¹Chen Huang,¹Zhuodong Yang,¹and Hongnan Li¹

Academic Editor: Jun Li

Received16 Nov 2023

Revised15 Apr 2024

Accepted29 Apr 2024

Published14 May 2024

Abstract

Rapid and accurate structural damage assessment after an earthquake is important for efficient emergency management. The widespread application of surveillance cameras provides a new possibility for improving the efficiency of assessment. However, it is still challenging to directly assess the structural seismic damage based on videos captured by indoor surveillance cameras during earthquakes. In this study, we elaborate on the concept of estimating the structural natural frequency based on the relative pixel displacement of inter-stories. Furthermore, we propose a strategy for post-earthquake structural damage assessment that integrates the computer vision and time-frequency analysis. This approach aims to navigate the difficulties inherent in earthquake damage assessment and improve emergency responses. The relative pixel displacement between the camera and the fixed features on the floor is extracted from videos by using the Harris corner detection and Kanade–Lucas–Tomasi algorithms. The structural natural frequency is estimated using the synchroextracting transform-enhanced empirical wavelet transform. The natural frequency shift-related seismic damage index is defined and calculated for damage assessment. A shake table experiment of a small-scale steel model is conducted to verify the accuracy and feasibility of the approach, and the practicality of the proposed approach is further verified by utilizing the data from a full-scale reinforced concrete benchmark model experiment. The results demonstrate that the approach can accurately and efficiently evaluate the structural damage after an earthquake based on the video captured by surveillance cameras during the earthquake. The error of the acquired damage index is less than 0.1. We will apply more advanced algorithms in the future to alleviate this problem.

1. Introduction

Seismic hazards pose a substantial risk to the safety of structures located in earthquake-prone regions [1, 2] and can cause severe social and economic losses in densely populated cities [3, 4]. After an earthquake, a rapid and accurate assessment of the true seismic damage to structures is crucial for post-disaster relief [5–7]. The seismic fragility and vulnerability models are often used to evaluate the probabilistic seismic risk of structures [8]. To improve the accuracy of prediction, an updated empirical model for structural seismic vulnerability assessment was proposed using the survey data obtained from the Yushu earthquake [9]. Additionally, as an alternative to the traditional seismic vulnerability model, a surrogate model based on an artificial neural network was also investigated [10]. To assess the damage condition of a structure after an earthquake, it is necessary to collect the on-site data and establish the relationship between the collected data and the damage status of the structure [11, 12]. Existing methods for on-site seismic damage assessment can be broadly divided into two types depending on the type of data collected: assessment based on visual inspection and assessment based on vibration monitoring [13].

Traditional visual inspection-based assessment methods primarily rely on the observable characteristics of the damage structures post-earthquake to evaluate the extent of the damage [14, 15]. The most common form is manual inspection, in which the structural damage is assessed by trained specialists on site [16, 17]. However, the manual inspection process can be inefficient, costly, and dangerous, and the assessment results can be highly subjective depending on the personal decisions of surveyors [18]. In light of these limitations, researchers have proposed acquiring images of post-earthquake structures by using imaging devices (such as handheld cameras [19], unmanned aerial vehicles [20], and satellites [21]) to reduce costs and improve the efficiency of seismic damage assessment. A robot with a vision system was applied to estimate the residual displacement of structural joints via engineered landmarks in non-line-of-sight conditions [22]. Other researchers utilized the deep learning-based computer vision technology to evaluate the structural damage [23, 24]. An improved convolutional neural network was proposed for identifying and locating the seismic damage from the images of reinforced concrete columns [25]. While techniques for visual inspection-based assessments have significantly advanced over time, the correlation between the external appearances of a structure and its actual condition remains somewhat unclear [11]. Additionally, the impact of structural surface decorations or coating on the evaluation results is a factor that cannot be overlooked. This potential influence has yet to be thoroughly researched.

In contrast to visual inspection approaches, vibration monitoring-based methods operate on a clearer physical mechanism [26, 27], primarily performing the structural damage assessment based on the damage indexes extracted from the structural vibration response [28, 29]. In general, the vibration responses of structures are obtained by a monitoring system through sensors (such as accelerometers, displacement sensors, and inclinometers) distributed along the structure [30, 31]. A time-varying system identification approach based on the adaptive unscented Kalman filtering was presented and employed to identify the time-varying stiffness of prefabricated segmental columns under earthquake excitations by using the measured accelerations and displacements [32]. A modal assurance distribution-based instantaneous dynamic parameter identification method was proposed for the structural integrity assessment [33]. However, practical applications are often limited due to the high cost of installation and maintenance of such monitoring systems [34, 35]. Thus, there are a large number of buildings that are not instrumented by monitoring systems. On the other hand, vision-based vibration measurement is expected to solve the above problems due to its low cost and high efficiency [36, 37]. The vibration responses of a structure are usually recorded by stationary or movable cameras outside the monitored structure [38, 39]. While less expensive overall, there are still costs associated with the installation and maintenance of cameras [40, 41].

In an effort to curb costs, researchers have turned their attention to the ubiquitous indoor surveillance cameras. The feasibility of using these devices to capture inter-story seismic responses has been demonstrated, indicating a promising new direction for cost-effective and efficient earthquake damage assessment [40, 42, 43]. However, it is still challenging to assess structural seismic damage based on the acquired seismic response of inter-stories only from the pixels obtained from the videos recorded by the cameras, because the seismic response of the structure is usually a nonstationary signal.

To quantify the severity of damage of a structure, the selection of an appropriate damage index is essential. Commonly used damage indices are calculated based on the inter-story drift ratio [44], vibration mode [33, 42], or natural frequency [45]. However, for structural seismic damage assessment based on surveillance cameras, the unit of structural displacement is the pixel rather than the meter. The pixel needs to be calibrated to the physical length of a given target in the image [46]. It is impractical to perform calibration for all responses after an earthquake, and thus, the inter-story drift ratio-based damage indices (such as maximum or residual inter-story drift ratio) cannot be directly available and used for damage assessment [44]. In addition, videos captured by different cameras distributed along the structure are often not precisely synchronized. The asynchronous acquisition is detrimental to the extraction of structural vibration mode, and therefore, the vibration mode-related damage indices also cannot be used for damage assessment [33, 42]. Natural frequency-based damage indices are not affected by signal amplitude and can be obtained from only one vibration signal. Consequently, damage indices based on natural frequencies might be a viable approach in structural seismic damage assessment using surveillance cameras.

The commonly used natural frequency-based seismic damage index is the natural frequency shift before and after an earthquake, and it has been utilized for structural seismic damage assessment [45]. However, the acquisition of the natural frequency shift often requires modal testing and analysis of the structure before and after the earthquake, which is time-consuming and costly. In recent years, the advancement of modern signal processing techniques and system identification theory has brought time-frequency analysis technology into the spotlight [47]. This technique, which estimates the shift in natural frequency based solely on a structural nonstationary seismic response, has been increasing usage thanks to its innovative approach and promising results [48]. These methods mainly can be divided into wavelet transform-based methods and Hilbert transform (HT)-based methods [49, 50]. For the methods based on the wavelet transform, the frequency shift is typically identified by extracting the frequency ridges from the wavelet scalogram [51, 52]. However, the selection of wavelet bases has a large impact on the analysis results. To mitigate the potential impact of suboptimal wavelet selection, adaptive time-frequency analysis methods like empirical mode decomposition (EMD) [53, 54], based on Hilbert transform (HT), were introduced. These methods first extract single-component signals from the original multicomponent signal. Subsequently, the shifts in various orders of natural frequency are obtained by using HT. However, the EMD is not sufficient or rigorous enough in mathematical proofs. Furthermore, a new method called empirical wavelet transform (EWT) [55] was proposed, and it was enhanced by the synchroextracting transform (SET) [56] to identify the natural frequency of a time-varying system by using accelerometers [57]. Drawing inspiration from the previous research [57], this study estimates the shift in structural natural frequency by using a SET-enhanced EWT method. This approach utilizes pixel-based inter-story structural responses as captured by surveillance cameras.

This study presents a novel approach to assess structural damage induced by earthquakes by combining the capabilities of surveillance cameras with computer vision and time-frequency analysis technologies. The structural seismic responses were extracted from the video by using a marker-free computer vision technique. The damage state was assessed according to the shift in structural natural frequency, which was estimated by using a time-frequency analysis technique. The accuracy and feasibility of the approach were verified through a shake table test conducted on a small-scale steel model. Then, the practicality of the approach was further confirmed using the experimental data from a full-scale reinforced concrete model.

The rest of the study is organized as follows: Section 2 presents the theoretical foundation of the proposed method. Sections 3 and 4 describe the experimental setups and results of the small-scale model and full-scale model shake table experiments. Section 5 shows the conclusion and future work.

2. Methodology

The schematic diagram of the proposed structural seismic damage assessment strategy is shown in Figure 1, where is the meter displacement of the lth story with respect to the ground; is the inter-story meter displacement of the lth story; is the inter-story pixel displacement of the jth feature point of the lth story extracted from the video; CFOV refers to the camera field of view; and ROI indicates the region of interest. First, images were captured using the existing indoor surveillance cameras during an earthquake. Next, the structural inter-story pixel displacement responses were extracted from the videos using computer vision techniques. Then, the natural frequency shift of the structure was estimated by using the time-frequency analysis techniques. Finally, the structural damage was assessed according to the calculated damage index. It is worth noting that surveillance cameras mounted on ceilings, walls, or other locations are all applicable to this method. For the sake of presentation, it is assumed that the camera is mounted on the ceiling throughout the following.

2.1. Structural Inter-Story Pixel Displacement Response Extraction

The schematic diagram of structural inter-story seismic response extraction is shown in Figure 1. First, a rectangular box was used to manually select the ROI in the first frame of the video. The ROI is defined as a region of the image that involves distinctive feature points fixed to the floor. The purpose of the ROI selection is to ensure that subsequent tracked feature points are not lost due to occlusion. Then, the feature points in the ROI are detected. Considering that there are no pre-installed marker points in practice, a typical marker-free feature detection algorithm (i.e., Harris corner detection algorithm [58]) is adopted. More specifically, the features in the ROI are automatically detected based on the magnitude of the pixel gradient in different directions. Finally, the pixel displacement of different feature points is automatically computed based on the difference between the ROI in adjacent frames using a classic optical flow algorithm (i.e., Kanade–Lucas–Tomasi algorithm [59]). The structural inter-story pixel displacement of the lth story can be obtained by averaging the motions of different feature points according to the following equation:where m is the number of detected feature points, and t is the time variable. The whole response extraction process is automated except for the selection of ROI and does not require any specific markers.

2.2. Structural Natural Frequency Shift Estimation

The flowchart for the estimation of structural natural frequency is shown in Figure 2. First, the single-mode responses are extracted from the acquired multimode inter-story response by using the SET-enhanced EWT method. Then, the natural frequency of the structure is estimated from the instantaneous frequency of the extracted single-mode response by using the HT and the Savitzky–Golay filtering (SGF) [60]. After the above signal processing, the natural frequency of the structure can be generally extracted from the seismic response of the structure. The details of the estimation process will be introduced in the following section.

2.2.1. Extraction of the Single-Mode Response

EWT is a widely used adaptive signal decomposition method. The traditional EWT method consists of two steps: (1) determining the frequency boundaries between the different single-mode signals and (2) constructing the filtering bank to extract the single-mode signals [55]. However, it is difficult to use only the traditional Fourier spectrum to determine the filtering boundary for the nonstationary signal. To tackle this issue, the SET is introduced to determine the filtering boundaries. The SET is based on the short-time Fourier transform (STFT) and can generate an energy-concentrated, time-frequency representation of the signal [56]. The EWT allows us to determine frequency boundaries between different modes based on the resulting time-frequency representation derived from the SET [57].

For an n degree of freedom shear-type building during a strong earthquake, the equation of motion can be expressed as follows:where M(t), C(t), and K(t) represent the time-variant mass, damping, and stiffness matrices of the structure during the earthquake, respectively. , , and denote the displacement, velocity, and acceleration response vectors of the structure. f(t) is the external earthquake excitation vector.

Transforming (2) into spatial modal coordinates yieldswhere and are generalized mass and generalized load of the ith mode shape, is the ith mode shape, is the ith modal coordinate, and and are the ith natural frequency and damping coefficient of the structure, respectively.

According to the modal decomposition theory, the inter-story multimode pixel displacement at the lth story extracted from the video can be expressed as follows:where l = 1, 2, …, n. The value of is constantly equal to 0. is the unit conversion factor from pixel to meter for the lth story. is the coefficient at the lth degree of freedom of the ith mode shape.

According to the analytic signal theory, the analytic signal of the can be expressed as follows:where and are the instantaneous amplitude and frequency of the ith modal response, H[ ] denotes the HT, and j is the imaginary unit.

Furthermore, the analytic signal of the can be expressed as follows:

According to the definition of the instantaneous frequency (i.e., the derivative of phase versus time), the instantaneous frequency of the ith modal response is equal to the instantaneous frequency of .

The STFT expression of the multimode signal can be expressed as follows:where represents the fast Fourier transform.

The instantaneous frequency trajectory for the can be obtained as follows:

The energy-concentrated, time-frequency representation obtained by using the SET can be expressed aswhere is defined as

According to (9), an energy-concentrated, time-frequency representation can be obtained, and the frequency boundaries between different modes can be determined. The filtering boundaries for each single-mode signal in the EWT can be defined by an interval . Additionally, a transient phase with a width of is defined for each . More details can be found in reference [57].

The corresponding empirical scaling function and the wavelet function can be determined aswhere , , and is the auxiliary function for the Meyer wavelet [55].

Then, the EWT analysis can be conducted. The detail coefficients and the approximation coefficients can be obtained as

The single-mode response extracted from the multimode response can be described as

2.2.2. Estimation of the Natural Frequency

The and of the structure can be regarded as lower-order polynomials of . Specifically, and behave as a low-pass filter, while behaves as a high-pass filter. According to the Bedrosian theorem of the HT, the following can be obtained:where .

Furthermore, (19) can be rewritten as follows by substituting the first two derivatives of .

The instantaneous frequency of the can be obtained by solving (20), (57), and (61).where the first term is a low-frequency component, while the second term is a high-frequency component. The ith natural frequency can be estimated by filtering out the high-frequency component from .

The SGF is a data smoothing technique that utilizes local least-squares polynomial approximation [60]. It has been validated in the field of digital signal processing for its excellent high-frequency noise reduction capabilities, as well as its ability to preserve the shape of the signal [62]. In this study, the SGF is applied to filter out the high-frequency components from the to get the estimated , which can be expressed as follows:where is a polynomial function, r and are the order and coefficient, M is the SGF span, and is the coefficient acquired by using the least-square fit.

After obtaining the estimated natural frequency , the natural frequency shift is approximated by the ratio of the maximum to the minimum of the acquired natural frequency.

2.3. Structural Damage Assessment

To facilitate the assessment of structural damage, given the minimal contribution of higher-order vibrations in low-rise structures, only the first modal response is extracted in the following analysis. The natural frequency shift-based structural damage index [45, 50] was defined aswhere is the first natural frequency of the structure before the earthquake; is the first natural frequency of the structure after the earthquake. The value of D is less than 1, and the larger the value of D, the more severe the structural damage.

In this study, the damage index can be approximated aswhere is the estimated first natural frequency of the structure.

A reasonable criterion regarding the correspondence between damage indices and damage states is important for structural damage assessment. To illustrate the feasibility of the proposed method, a criterion based on the principles of the performance-based earthquake design is introduced [50]. The range of the damage index for reinforced concrete structures is shown in Table 1.

3. Small-Scale Model Verification

3.1. Experimental Setup

To validate the feasibility of the proposed method, a shake table test for a small-scale model of a two-story steel frame structure was designed. The experimental setup is shown in Figure 3. To stay within the limits of the loading capacity of the shake table and the available space in the laboratory, the total height of the model is 1.1 m, and the heights of stories one and two are 0.6 m and 0.5 m, respectively. The cross-sectional dimensions of each column are 3 mm × 20 mm. Each rigid floor has a size of 500 mm × 500 mm × 5 mm and a mass of 10 kg. To simulate the real indoor situation, small-scale models of furniture were fixed on the floors, as shown in Figure 3. Two cameras were rigidly fixed on the ceilings and perpendicular to the floor below. The resolution and frame rate of the cameras are 640 × 480 pixels and 120 fps, respectively. In addition, to verify the accuracy of the measured response through computer vision techniques, the accurate dynamic response of the model during the experiment was measured at a sampling frequency of 120 Hz using the position-sensing system. The position sensor markers are fixed on the side of each floor. The shake table is a Quanser XY Shake Table III produced by the Quanser company. The cameras are RYS720P Global Shutters from Shenzhen RongyangSheng Electronic Technology Corporation. The position-sensing system is the NDI Optotrak Certus produced by the Northern Digital Incorporated.

(a)

(b)

(c)

(d)

Springs are utilized in the experiment to simulate the damage sustained by the structure during the earthquake. The springs were firstly tightened obliquely on a certain floor of the model. Then, the springs were suddenly unloaded at a certain moment when the seismic acceleration was large to simulate the stiffness degradation caused by the seismic damage. To ensure safety, protective sleeves were installed for each spring. Considering the feasibility of the operation, the springs were installed only on the first floor, and three different conditions of the structure (DS 1, DS 2, and DS 3) were tested. Different conditions of the structure were achieved by changing the number of springs. The natural frequencies of the structures before and after the seismic load were obtained by using the sweep frequency excitation (Table 2). The first natural frequency decreases after the earthquake. The seismic action applied by the shake table was the earthquake named NGA_no_1105_HIK000, which can be obtained from the website (https://ngawest2.berkeley.edu), as shown in Figure 4. Due to the limitations of the shake table, the peak acceleration of ground motion was adjusted to 0.1 g.

3.2. Verification of the Structural Inter-Story Response

The structural inter-story pixel displacements were obtained through the procedure described in Section 2.1. As shown in Figure 5, the region of the fixed small-scale model of the furniture was set as the ROI, and the feature points in the ROI were automatically detected using the Harris corner detection algorithm. Subsequently, the Kanade–Lucas–Tomasi algorithm was utilized to track the motion of feature points, and the inter-story displacement, in terms of pixels, can be obtained. To verify the accuracy of the structural response measured by computer vision techniques, the displacement was calibrated into meters by multiplying a unit conversion factor s, which can be acquired according to (24). The diagram of unit conversion is shown in Figure 6, and the P, p, and s in the schematic are 99.52 mm, 148 pixels, and 0.672 mm/pixel, respectively.where s is the unit conversion factor from pixel to meter, P is the length (meters) of the target, and p is the pixel length of the target.

The position sensor cannot measure the response of the structure during the spring unloading process due to the occlusion by personnel. Thus, the response of the structure without springs was selected to verify the accuracy. The comparison between the structural responses measured via the computer vision and as measured by the position sensor is shown in Figure 7. The relative errors of the maximum inter-story displacement of the first and second stories of the structure are 0.21% and 0.36%, respectively. Furthermore, the main peak frequencies of the inter-story displacement responses obtained by the two approaches are consistent. The consistency indicates the dynamic response of the structure can be accurately measured using the method described in this study. As observed from the spectrum of the responses, a camera with 120 fps is sufficient to capture the structural response.

(a)

(b)

(c)

(d)

3.3. Damage Index Calculation of the Small-Scale Model

After verifying the accuracy of the measured structural response, the natural frequency of the structure can be estimated from the response using the procedure described in Section 2.2. First, the structural responses in terms of inter-story pixel displacement were obtained via the cameras, as shown in Figures 8 and 9. Then, the energy-concentrated, time-frequency representation of the response was obtained by using the SET, and the bandwidth of the first modal response was determined as shown in Figures 8 and 9. The ranges of the frequency bands for the first-order modal responses are listed in Table 3. Next, the first modal response was extracted from the measured response through the EWT. The extracted first modal responses are shown in Figures 10 and 11. Lastly, the instantaneous frequency of the first modal response was calculated using the HT, and the first natural frequency was then estimated through the application of the SGF (Figures 10 and 11). The order of the polynomial r was set to 3, and the span M was set to 5999 in the SGF. It can also be seen that the error of the calculated instantaneous frequency of the response due to Gibbs’ phenomenon [63], which refers to an overshoot induced by a jump discontinuity, can be also eliminated by SGF.

(a)

(b)

(c)

(d)

(e)

(f)

(a)

(b)

(c)

(d)

(e)

(f)

(a)

(b)

(c)

(d)

(e)

(f)

(a)

(b)

(c)

(d)

(e)

(f)

To verify the feasibility of the proposed approach, the damage indices were calculated according to the estimated first natural frequency of the structure. The results of the calculated damage indices are listed in Table 4 and Table 5. The error of the damage index is defined as the difference between the damage index estimated by the proposed method and the one obtained by the modal test. The maximum error of the damage index is 0.097, and the damage indices obtained by using the structural response of the first and second stories are consistent. This indicates that the proposed approach can obtain the natural frequency-based damage index with only one camera. The error of the damage index is mainly attributed to the measurement error in the pixel displacement and the estimation error in the instantaneous frequency.

4. Full-Scale Model Verification

4.1. Experimental Setup

To further confirm the practicability and effectiveness of the proposed method, the method was applied to evaluate the damage state of a full-scale reinforced concrete building model. In contrast to the sudden reduction of the structural stiffness in Section 3, the seismic damage accumulation of the structure during the earthquake can be considered in this experiment. The model is from the building nonstructural components and systems (BNCS) project [64] and has five stories, as shown in Figure 12. The shake table test of the full-scale model consisted of two phases: base isolated phase and fixed base phase. The model was severely damaged in the second phase of the test named FB-5: DEN67. The seismic action applied by the shake table in this test followed the earthquake recorded by the TAPS Pump Station during the Denali earthquake. The peak acceleration of the ground motion was scaled to be 0.64 g. According to white noise excitation tests, the first natural frequency of the structure before and after the earthquake is 0.84 Hz and 0.62 Hz, respectively. According to Table 1, the damage index is 0.455, which corresponds to moderate damage. The camera Q6 (1280 × 720 resolution at 25 fps), which was located on the second floor, was used in this study. More detailed information can be found on the website (https://www.designsafe-ci.org/data/browser/public/nees.public/NEES-2009-0722.groups).

4.2. Damage Assessment of the Full-Scale Model

The entire assessment procedure for the full-scale model is similar to that for the small-scale model. First, the ROI was selected and the feature points were detected using the surveillance video during the earthquake, as shown in Figure 13. The structural response was obtained by tracking the feature point, which was observed to remain stationary relative to the floor, as shown in Figure 14. Then, the energy-concentrated, time-frequency representation of the response was obtained by using the SET, as shown in Figure 14. The range of the frequency band for the first modal response is 0.43 Hz to 1.00 Hz. Next, the first modal response was extracted from the measured response by using the EWT (Figure 15). Finally, the instantaneous frequency of the first modal response was calculated by using the HT, and the first natural frequency was estimated through the SGF (Figure 15). The order of the polynomial r was set to 3, and the span M was set to 1499. The maximum and minimum of the estimated first natural frequency were 0.82 Hz and 0.60 Hz, respectively. The resulting damage index was 0.465, which corresponds to moderate damage and is consistent with the result of the white noise tests [64].

(a)

(b)

(a)

(b)

5. Conclusion and Future Work

This study presents an efficient and effective strategy to accurately assess post-earthquake structural damage. The approach was developed based on computer vision and time-frequency analysis techniques and applied to data from existing surveillance cameras. The relative seismic response of inter-stories was extracted from the video by using a marker-free computer vision-based method. Subsequently, the natural frequency shift of the structure was estimated by using HT-based time-frequency analysis. The damage index related to the first natural frequency shift was then calculated and used for the damage assessment. The feasibility of the approach was verified via shake table experiments. The main conclusions are as follows:(1)The inter-story seismic response of the structure can be accurately acquired by using surveillance cameras and computer vision techniques under favorable illumination conditions. The relative error of the maximum inter-story displacement of the structure is less than 0.4%.(2)The natural frequency shift of the structure can be estimated by employing the SET-enhanced EWT method, HT, and SGF. The error of the damage index using the estimated frequency shift was less than 0.1.(3)The damage state of a structure can be directly evaluated solely by analyzing the relative pixel displacement response of the structure with natural frequency-based damage assessment theory.

The proposed strategy for assessing structural damage induced by earthquakes capitalizes on existing surveillance cameras to achieve accurate assessment. This method can serve as an alternative solution for post-earthquake structural damage assessment in situations where sensor monitoring systems are not in place. Despite the encouraging results, there are still some open research challenges in future research. These challenges include the following:(1)The quality of the images collected in the experiment is good, but the images acquired in the actual seismic scenario are adversely affected by factors such as insufficient illumination and occlusion. Thus, the computer vision-based algorithms to extract structural inter-story pixel displacement need further improvement.(2)The external images of the structure hold rich information about structural damage. Consequently, the integration of visual inspection-based assessments with vibration monitoring-based assessments represents the next crucial area of exploration.(3)Due to laboratory constraints, the feasibility of the proposed method is verified only in one-dimensional shake experiments, and the potential applicability of the proposed method in two- and three-dimensional seismic scenarios will be pursued in future studies.(4)The development of advanced algorithms for pixel displacement measurement and instantaneous frequency estimation will also be a focus of future research to further reduce the damage index error.

Data Availability

The data used to support the findings of this study are available from the corresponding author upon request.

Disclosure

Any opinions, findings, or recommendations expressed in this study do not necessarily represent the individuals involved in the project.

Conflicts of Interest

The authors declare that there are no conflicts of interest regarding the publication of this paper.

Acknowledgments

The authors would like to thank the videos from the full-size BNCS building shake table experiments. This research was supported by the General Project of the Natural Science Foundation of China (Grant 51778111).

References

W. Y. Zhang, P. Y. Chen, J. G. F. Crempien, A. Kurtulus, P. Arduino, and E. Taciroglu, “Regional-scale seismic fragility, loss, and resilience assessment using physics-based simulated ground motions: an application to Istanbul,” Earthquake Engineering & Structural Dynamics, vol. 52, no. 6, pp. 1785–1804, 2023.
View at: Publisher Site | Google Scholar
C. Li, K. M. Bi, and H. Hao, “Seismic performances of precast segmental column under bidirectional earthquake motions: shake table test and numerical evaluation,” Engineering Structures, vol. 187, pp. 314–328, 2019.
View at: Publisher Site | Google Scholar
C. Huang, L. S. Huo, H. G. Gao, and H. N. Li, “Control performance of suspended mass pendulum with the consideration of out-of-plane vibrations,” Structural Control and Health Monitoring, vol. 25, no. 9, p. e2217, 2018.
View at: Publisher Site | Google Scholar
X. Z. Lu, Y. J. Xu, Y. Tian, B. Cetiner, and E. Taciroglu, “A deep learning approach to rapid regional post-event seismic damage assessment using time-frequency distributions of ground motions,” Earthquake Engineering & Structural Dynamics, vol. 50, no. 6, pp. 1612–1627, 2021.
View at: Publisher Site | Google Scholar
H. Mouloud, A. Chaker, H. Nassim, S. Lebdioui, H. Rodrigues, and M. R. Agius, “Post-earthquake damage classification and assessment: case study of the residential buildings after the M-w=5 earthquake in Mila city, Northeast Algeria on August 7, 2020,” Bulletin of Earthquake Engineering, vol. 21, no. 2, pp. 849–891, 2023.
View at: Publisher Site | Google Scholar
Y. Idris, P. Cummins, I. Rusydy et al., “Post-earthquake damage assessment after the 6.5 Mw earthquake on december, 7th 2016 in pidie jaya, Indonesia,” Journal of Earthquake Engineering, vol. 26, no. 1, pp. 409–426, 2022.
View at: Publisher Site | Google Scholar
C. Chen, J. M. Ricles, T. L. Karavasilis, Y. Chae, and R. Sause, “Evaluation of a real-time hybrid simulation system for performance evaluation of structures with rate dependent devices subjected to seismic loading,” Engineering Structures, vol. 35, pp. 71–82, 2012.
View at: Publisher Site | Google Scholar
H. Zhang, D. H. Yu, G. Li, and Z. Q. Dong, “A real-time seismic damage prediction framework based on machine learning for earthquake early warning,” Earthquake Engineering & Structural Dynamics, vol. 53, no. 2, pp. 593–621, 2024.
View at: Publisher Site | Google Scholar
S. Q. Li and A. Formisano, “Updated empirical vulnerability model considering the seismic damage of typical structures,” Bulletin of Earthquake Engineering, vol. 22, no. 3, pp. 1147–1185, 2023.
View at: Publisher Site | Google Scholar
X. Z. Yuan, J. Zhong, Y. P. Zhu, G. D. Chen, and C. Dagli, “Post-earthquake regional structural damage evaluation based on artificial neural networks considering variant structural properties,” Structures, vol. 52, pp. 971–982, 2023.
View at: Publisher Site | Google Scholar
C. Balkaya, F. Casciati, S. Casciati, L. Faravelli, and M. Vece, “Real-time identification of disaster areas by an open-access vision-based tool,” Advances in Engineering Software, vol. 88, pp. 83–90, 2015.
View at: Publisher Site | Google Scholar
M. Tazarv, K. Won, Y. Jang, K. Hart, and E. Greeneway, “Post-earthquake serviceability assessment of standard RC bridge columns using computer vision and seismic analyses,” Engineering Structures, p. 272, 2022.
View at: Google Scholar
S. D. Shen and M. Kurata, “Rapid evaluation of structural soundness of steel frames using a coupling coefficient (CC)-based method,” Earthquake Engineering & Structural Dynamics, vol. 52, no. 4, pp. 1182–1204, 2023.
View at: Publisher Site | Google Scholar
H. V. Burton and G. G. Deierlein, “Integrating visual damage simulation, virtual inspection, and collapse capacity to evaluate post-earthquake structural safety of buildings,” Earthquake Engineering & Structural Dynamics, vol. 47, no. 2, pp. 294–310, 2018.
View at: Publisher Site | Google Scholar
X. Pan and T. Y. Yang, “Postdisaster image-based damage detection and repair cost estimation of reinforced concrete buildings using dual convolutional neural networks,” Computer-Aided Civil and Infrastructure Engineering, vol. 35, no. 5, pp. 495–510, 2020.
View at: Publisher Site | Google Scholar
M. Uros, M. S. Novak, J. Atalic, Z. Sigmund, M. Banicek, M. Demsic et al., “Post-earthquake damage assessment of buildings- procedure for conducting building inspections,” Građevinar, vol. 72, pp. 1089–1115, 2020.
View at: Google Scholar
Z. Lingxin, S. Junkai, and Z. Baijie, “A review of the research and application of deep learning-based computer vision in structural damage detection,” Earthquake Engineering and Engineering Vibration, vol. 21, pp. 1–21, 2022.
View at: Publisher Site | Google Scholar
C. Z. Dong and F. N. Catbas, “A review of computer vision-based structural health monitoring at local and global levels,” Structural Health Monitoring, vol. 20, no. 2, pp. 692–743, 2021.
View at: Publisher Site | Google Scholar
Y. Gao and K. M. Mosalam, “PEER Hub ImageNet: a large-scale multiattribute benchmark data set of structural images,” Journal of Structural Engineering, vol. 146, no. 10, 2020.
View at: Publisher Site | Google Scholar
C. Xiong, Q. S. Li, and X. Z. Lu, “Automated regional seismic damage assessment of buildings using an unmanned aerial vehicle and a convolutional neural network,” Automation in Construction, vol. 109, 2020.
View at: Publisher Site | Google Scholar
Q. G. Lin, T. Y. Ci, L. B. Wang, S. K. Mondal, H. X. Yin, and Y. Wang, “Transfer learning for improving seismic building damage assessment,” Remote Sensing, vol. 14, no. 1, p. 201, 2022.
View at: Publisher Site | Google Scholar
A. Thoms, Z. Al-Sabbag, and S. Narasimhan, “Graph-based structural joint pose estimation in non-line-of-sight conditions,” Earthquake Engineering and Engineering Vibration, vol. 22, no. 2, pp. 371–386, 2023.
View at: Publisher Site | Google Scholar
A. Ghiasi, M. K. Moghaddam, C. T. Ng, A. H. Sheikh, and J. Q. Shi, “Damage classification of in-service steel railway bridges using a novel vibration-based convolutional neural network,” Engineering Structures, vol. 264, 2022.
View at: Publisher Site | Google Scholar
Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol. 521, no. 7553, pp. 436–444, 2015.
View at: Publisher Site | Google Scholar
Y. Xu, S. Wei, Y. Bao, and H. Li, “Automatic seismic damage identification of reinforced concrete columns from images by a region-based deep convolutional neural network,” Structural Control and Health Monitoring, vol. 26, no. 3, p. e2313, 2019.
View at: Publisher Site | Google Scholar
J. Zhang, T.-H. Yi, C.-X. Qu, and H.-N. Li, “Detecting hinge joint damage in hollow slab bridges using mode shapes extracted from vehicle response,” Journal of Performance of Constructed Facilities, vol. 36, no. 1, 2022.
View at: Publisher Site | Google Scholar
Y. D. Shao, L. Li, J. Li, Q. L. Li, S. J. An, and H. Hao, “Monocular vision based 3D vibration displacement measurement for civil engineering structures,” Engineering Structures, vol. 293, 2023.
View at: Publisher Site | Google Scholar
B. Lopez-Castro, A. G. Haro-Baez, D. Arcos-Aviles, M. Barreno-Riera, and B. Landazuri-Aviles, “A systematic review of structural health monitoring systems to strengthen post-earthquake assessment procedures,” Sensors, vol. 22, no. 23, p. 9206, 2022.
View at: Publisher Site | Google Scholar
A. Tiganescu, I. G. Craifaleanu, A. Aldea et al., “Evolution, recent progress and perspectives of the seismic monitoring of building structures in Romania,” Frontiers in Earth Science, vol. 10, 2022.
View at: Publisher Site | Google Scholar
J. Y. Li and S. Y. Zhu, “Cable vibration mitigation by using an H-bridge-based electromagnetic inerter damper with energy harvesting function,” Structural Control and Health Monitoring, vol. 29, no. 12, 2022.
View at: Publisher Site | Google Scholar
Y. X. Pan, C. E. Ventura, and T. Li, “Sensor placement and seismic response reconstruction for structural health monitoring using a deep neural network,” Bulletin of Earthquake Engineering, vol. 20, no. 9, pp. 4513–4532, 2022.
View at: Publisher Site | Google Scholar
Y. Xin, J. Li, H. Hao, N. Yang, and C. Li, “Time-varying system identification of precast segmental columns subjected to seismic excitations,” Journal of Bridge Engineering, vol. 27, no. 4, 2022.
View at: Publisher Site | Google Scholar
S. Quqa, L. Landi, and P. P. Diotallevi, “Seismic structural health monitoring using the modal assurance distribution,” Earthquake Engineering & Structural Dynamics, vol. 50, no. 9, pp. 2379–2397, 2021.
View at: Publisher Site | Google Scholar
J. Zhao, Y. Q. Bao, Z. G. Guan, W. M. Zuo, J. Z. Li, and H. Li, “Video-based multiscale identification approach for tower vibration of a cable-stayed bridge model under earthquake ground motions,” Structural Control and Health Monitoring, vol. 26, no. 3, p. e2314, 2019.
View at: Publisher Site | Google Scholar
Q. Zhang, X. Fu, L. Ren, and Z. G. Jia, “Modal parameters of a transmission tower considering the coupling effects between the tower and lines,” Engineering Structures, vol. 220, 2020.
View at: Publisher Site | Google Scholar
X. W. Ye, C. Z. Dong, and T. Liu, “A review of machine vision-based structural health monitoring: methodologies and applications,” Journal of Sensors, vol. 10, 2016.
View at: Google Scholar
B. F. Spencer, V. Hoskere, and Y. Narazaki, “Advances in computer vision-based civil infrastructure inspection and monitoring,” Engineering, vol. 5, no. 2, pp. 199–222, 2019.
View at: Publisher Site | Google Scholar
Z. Y. Su, B. Wei, and J. Zhang, “Feature-constrained real-time simultaneous monitoring of monocular vision odometry for bridge bearing displacement and rotation,” Automation in Construction, vol. 154, 2023.
View at: Publisher Site | Google Scholar
E. J. Cai and Y. Zhang, “Gaussian mixture model based phase prior learning for video motion estimation,” Mechanical Systems and Signal Processing, vol. 175, 2022.
View at: Publisher Site | Google Scholar
P. S. Harvey Jr. and G. Elisha, “Vision-based vibration monitoring using existing cameras installed within a building,” Structural Control and Health Monitoring, vol. 25, no. 11, 2018.
View at: Publisher Site | Google Scholar
C. Cheng and Ki Kawaguchi, “A preliminary study on the response of steel structures using surveillance camera image with vision-based method during the Great East Japan Earthquake,” Measurement, vol. 62, pp. 142–148, 2015.
View at: Publisher Site | Google Scholar
A. Zare Hosseinzadeh and P. S. Harvey, “Pixel-based operating modes from surveillance videos for structural vibration monitoring: a preliminary experimental study,” Measurement, vol. 148, 2019.
View at: Publisher Site | Google Scholar
A. Z. Hosseinzadeh, M. H. Tehrani, and P. S. Harvey, “Modal identification of building structures using vision-based measurements from multiple interior surveillance cameras,” Engineering Structures, vol. 228, p. 13, 2021.
View at: Google Scholar
A. Ghobarah, H. Abou-Elfath, and A. Biddah, “Response-based damage assessment of structures,” Earthquake Engineering & Structural Dynamics, vol. 28, no. 1, pp. 79–104, 1999.
View at: Publisher Site | Google Scholar
J. A. Goulet, C. Michel, and A. D. Kiureghian, “Data-driven post-earthquake rapid structural safety assessment,” Earthquake Engineering & Structural Dynamics, vol. 44, no. 4, pp. 549–562, 2015.
View at: Publisher Site | Google Scholar
J. Li, B. Xie, and X. Zhao, “Measuring the interstory drift of buildings by a smartphone using a feature point matching algorithm,” Structural Control and Health Monitoring, vol. 27, no. 4, 2020.
View at: Publisher Site | Google Scholar
Z. Su, J. Yu, X. Xiao, J. J. Wang, and X. L. Wang, “Deep learning seismic damage assessment with embedded signal denoising considering three-dimensional time-frequency feature correlation,” Engineering Structures, vol. 286, 2023.
View at: Publisher Site | Google Scholar
Z. C. Wang, W. X. Ren, and G. D. Chen, “Time-varying linear and nonlinear structural identification with analytical mode decomposition and hilbert transform,” Journal of Structural Engineering, vol. 139, no. 12, p. 139, 2013.
View at: Publisher Site | Google Scholar
Z. C. Wang, W. X. Ren, and G. D. Chen, “Time-frequency analysis and applications in time-varying/nonlinear structural systems: a state-of-the-art review,” Advances in Structural Engineering, vol. 21, no. 10, pp. 1562–1584, 2018.
View at: Publisher Site | Google Scholar
H. X. He, Y. F. Chen, and B. J. Lan, “Damage assessment for structure subjected to earthquake using wavelet packet decomposition and time-varying frequency,” Structures, vol. 34, pp. 449–461, 2021.
View at: Publisher Site | Google Scholar
H. N. Li, T. H. Yi, M. Gu, and L. S. Huo, “Evaluation of earthquake-induced structural damages by wavelet transform,” Progress in Natural Science, vol. 19, no. 4, pp. 461–470, 2009.
View at: Publisher Site | Google Scholar
D. A. Aguirre, C. A. Gaviria, and L. A. Montejo, “Wavelet-based damage detection in reinforced concrete structures subjected to seismic excitations,” Journal of Earthquake Engineering, vol. 17, no. 8, pp. 1103–1125, 2013.
View at: Publisher Site | Google Scholar
Z. Y. Shi and S. S. Law, “Identification of linear time-varying dynamical systems using Hilbert transform and empirical mode decomposition method,” Journal of Applied Mechanics, vol. 74, no. 2, pp. 223–230, 2007.
View at: Publisher Site | Google Scholar
N. E. Huang, Z. Shen, S. R. Long et al., “The empirical mode decomposition and the Hilbert spectrum for nonlinear and non-stationary time series analysis,” Proceedings of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences, vol. 454, no. 1971, pp. 903–995, 1998.
View at: Publisher Site | Google Scholar
J. Gilles, “Empirical wavelet transform,” IEEE Transactions on Signal Processing, vol. 61, no. 16, pp. 3999–4010, 2013.
View at: Publisher Site | Google Scholar
G. Yu, M. J. Yu, and C. Y. Xu, “Synchroextracting transform,” IEEE Transactions on Industrial Electronics, vol. 64, no. 10, pp. 8042–8054, 2017.
View at: Publisher Site | Google Scholar
Y. Xin, H. Hao, and J. Li, “Time-varying system identification by enhanced empirical wavelet transform based on synchroextracting transform,” Engineering Structures, vol. 196, 2019.
View at: Publisher Site | Google Scholar
C. Harris and M. Stephens, “A combined corner and edge detector,” in Proceedings of the 4th Alvery Vision Conference, Manchester, UK, August 1988.
View at: Google Scholar
J. B. Shi and C. Tomasi, “Good features to track,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Seattle, CA, USA, April 1994.
View at: Google Scholar
R. W. Schafer, “What is a savitzky-golay filter?” IEEE Signal Processing Magazine, vol. 28, no. 4, pp. 111–117, 2011.
View at: Publisher Site | Google Scholar
M. Feldman, “Nonlinear-system vibration analysis using hilbert transform-II. Forced vibration analysis method 'FORCEVIB,” Mechanical Systems and Signal Processing, vol. 8, no. 3, pp. 309–318, 1994.
View at: Publisher Site | Google Scholar
H. Kordestani, C. W. Zhang, and M. Shadabfar, “Beam damage detection under a moving load using random decrement technique and Savitzky-Golay filter,” Sensors, vol. 20, no. 1, p. 243, 2019.
View at: Publisher Site | Google Scholar
P. F. Pai, “Three-point frequency tracking method,” Structural Health Monitoring, vol. 8, no. 6, pp. 425–442, 2009.
View at: Publisher Site | Google Scholar
M. C. P. E. Chen, E. Pantoli, X. Wang et al., “Full-scale structural and nonstructural building system performance during earthquakes: Part I- specimen description, test protocol, and structural response,” Earthquake Spectra, vol. 32, no. 2, pp. 737–770, 2016.
View at: Publisher Site | Google Scholar

Copyright

Copyright © 2024 Jing Zhou et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

PDF Download Citation

Download other formats

Order printed copies

Views

87

Downloads

71

Citations