A photogrammetric displacement measurement method based on machine learning was proposed to improve the robustness to environmental disturbances. (1) To reduce the target positioning error caused by environmental vibration especially atmospheric turbulence, a machine learning-based weighted location algorithm combined with an adaptive window selection strategy was developed. In an outdoor displacement table experiment, the proposed method’s root mean squared error (RMSE) is 0.04 mm when the distance is 50 m, showing better accuracy and stability. (2) To complement/correct the missing or anomalous data caused by adverse external conditions, such as severe occlusion or camera shaking, a fast data self-diagnosis using a correlation vector machine was performed to make full use of the full-field measurement results obtained from the images. The applicability of the proposed method in extreme cases was demonstrated in designed experiments and an actual displacement measurement task of a long-span bridge subjected to vortex-induced vibration.

1. Introduction

The structural properties of bridges continuously deteriorate under the action of environmental erosion, material aging, and vehicle overload. Deflection monitoring, as a key part of bridge structural health monitoring, can help clarify the working state of the bridge and identify abnormal changes promptly, thereby facilitating the prevention of sudden disasters. Photogrammetry-based deflection measurement methods can enable multipoint synchronization, real-time dynamic or static monitoring, and fine target positioning and have thus been widely used in structure displacement monitoring [1–3].

However, photogrammetry is susceptible to complex external factors such as environmental vibrations [4–6], illumination changes [7, 8], rain and fog [9], occlusion [10, 11], temperature changes [12, 13], and atmospheric disturbances [14, 15], which can lead to inaccurate image displacement extraction. Atmospheric disturbances decrease the accuracy of precision optical measurement results such as morphology, displacement, and velocity [16]. Image restoration techniques are typically applied for precision optical measurement [17, 18]. Notably, the correction effect of signal filtering [19, 20] on the displacement results is limited as it does not involve optical principles. Moreover, correction methods based on the design of camera systems [20] cannot be applied to single-camera-based measurement. Turbulent image processing methods [21], widely used in the military field, are aimed at target detection and cannot be directly applied for displacement measurement. In this context, the precise tracking of the target displacement must be realized to increase the displacement measurement accuracy based on the use of a single camera in scenarios involving atmospheric disturbances. In summary, the most effective strategy for single-camera-based displacement measurement is to enhance the subpixel accuracy of target tracking.

Center positioning algorithms such as the centroid method, fitting method, and least-squares fitting method [22] are widely used in real bridge applications. The basic principle is to perform statistical analysis or fitting of grayscale images in the effective calculation window (ECW). The existing studies on enhancing the center detection accuracy [23, 24] have focused on human-made targets with uniform gray levels, and the problem of atmospheric disturbances has not been extensively considered. Mahrt et al. [25] highlighted that atmospheric disturbances may lead to blurring and distortion of the target image, resulting in target center positioning errors. The key problems can be summarized as follows: (1) Imaging blurring and distortion lead to inaccurate determination of the ECW. When selecting the target area window, the appropriate window size must be determined, and the pixels in the ECW directly participate in the central calculation. The traditional threshold-based window determination method is susceptible to environmental interferences. To measure the infrared target radiation intensity, Yang et al. [26] proposed an adaptive ECW determination method and a real imaging region identification method [27, 28] based on the principle of normal distribution. Notably, these methods are suitable only for light targets with regular shapes, and any possible imaging distortion is ignored. Consequently, this paper proposes an adaptive ECW selection method based on energy accumulation to alleviate the influence of environmental noise. (2) Imaging blurring and distortion, especially imaging distortion, make it difficult to guarantee the positioning precision based on a single algorithm [26]. To solve this problem, several scholars have proposed weighted positioning methods [29, 30]. However, the determination of weights is challenging. In general, the internal mechanism associated with the distortion of the target center by atmospheric disturbances and its features is complex. Therefore, Wang et al. [31] attempted to determine the weights through a back propagation neural network (BPNN). However, BPNNs operate based on the empirical risk minimization criterion, which is prone to overfitting and getting trapped in local optima. In comparison, the least-squares support vector machine (LS-SVM) [28] has a higher training speed and prediction accuracy. Notably, the ECW determination in the study of Wang et al. [31] was based on a fixed threshold value. Considering this research background, in this study, an adaptive ECW selection strategy based on energy accumulation is used to establish a weighted center location method using LS-SVM.

The abovementioned environmental influences can be corrected by image preprocessing. However, traditional image processing methods may be ineffective in cases involving extreme interference problems such as dramatic illumination changes, shadows, occlusion, or unexpected camera shaking. Recently, deep learning (DL) techniques have been used to address these complex phenomena [32]. For example, Xu et al. proposed a novel distraction-free target tracking approach by integrating a DL-based Siamese tracker [33] with traditional correlation-based template matching. However, DL methods are also ineffective for extreme cases, for instance, those involving severe occlusion. In addition, bridge structures typically have few surface features and measurement points cannot always be substituted over the measured section, resulting in data loss.

Other data anomalies such as omission and loss, jump points, drifting, and trend mutation are also commonly encountered in structure health monitoring (SHM). To solve these issues, machine learning algorithms have been applied in the field of intelligent diagnosis, especially to identify abnormal data and structural damage [34–36]. Among the existing machine learning algorithms, relevance vector machine (RVM) can minimize the regression error and exhibits a high generalization and antinoise disturbance abilities. Moreover, the RVM can adapt to the characteristics of nonlinear time sequences of bridge health monitoring system data and exploit the correlation between the data and selected training samples to predict the missing data or correct the abnormal data [37]. Optical methods can be used to synchronously monitor multiple measurement points and provide data support for the model training of RVM. Therefore, in this study, such methods are used to address extreme interference problems.

The remaining paper is organized as follows: Section 2 introduces the basic principles and the validation tests of the proposed method, including the center location algorithm considering the atmospheric disturbance and the data self-diagnosis based on the RVM for adverse scenarios. Section 3 describes the field-monitoring test performed over a long-span cable-stayed bridge. Section 4 presents the concluding remarks.

2. Methodology

Figure 1 shows the framework of the proposed single-camera-based structural displacement measurement method, which includes target tracking, displacement conversion, and data correction. A center detection algorithm is used to locate the target regions in the image plane. Next, the extracted pixel displacement is converted to the physical displacement through the fitted object distance using an existing method [36]. Although several variants of target tracking methods are available, their performance is ineffective in a field-monitoring campaign involving environmental variations or other nondetectable obstructions. To overcome this limitation, a novel target tracking approach is developed in this study by integrating the distraction-free center localization algorithm and machine learning-based data diagnosis technology. The key principles are introduced in Sections 2.1 and 2.2.

2.1. Target Location Method against Atmospheric Disturbance

2.1.1. The Principles of Weighted Center Detection Algorithm Based on Machine Learning

The center location algorithm is optimized considering the effects of atmospheric turbulence on the target imaging. In general, turbulence leads to the blurring of the target boundary in the image, thereby changing the ECW. To address this problem, an adaptive window selection strategy based on energy accumulation is proposed. Furthermore, the deterioration of the image quality, especially the distortion, decreases the center location accuracy of a single algorithm. To address this problem, a weighted center location method based on two typical centroid algorithms is developed. The weight values are predicted through LS-SVM. The details of these methods are presented in the following text:

First, a rough calculation window sized pixels is determined in the initial reference image, and the energy E of the region is calculated using the following equation:where (x, y) represents the gray level at pixel coordinate (x, y). Because the edge may be fuzzy owing to the environment vibration, the pixels in this rough window cannot be directly used to calculate the center coordinates. An energy concentration area that satisfies equations (2) and (3) is determined as the ECW.

Specifically, when the gray level (x, y) is larger than the threshold , the energy of the energy concentration area must be larger than η times the total energy, where η (η = 85%) characterizes the energy concentration of the target. Otherwise, the value must be adjusted until both conditions are satisfied. This step must be implemented only at the initial moment. In the subsequent moments, only the threshold must be slightly adjusted, assuming that the concentrated energy of the target imaging remains nearly unchanged. Subsequently, the central coordinates are calculated using the gray information of this ECW. When calculating the centroid using equation (4), the gray value is the binarized gray .

Different from equation (4), to determine the squared-gray-based centroid location using equation (5), the original gray value is needed. Similarly, to reduce the influence of calculation area selection on central extraction results, this study only allows the pixels in ECW to participate in the calculation. However, the gray values of pixels in ECW cannot be directly used. Because the light intensity of both the active target and the external environment is not stable, this can also lead to fluctuations in positioning results. To alleviate this influence, this study proposes to optimize the grayscale using the threshold value . Therefore, the expression is shown as follows:

The weighted relationship between the centroid , squared gray centroid , and real center can be expressed as

The determination of weight is critical. However, owing to the complexity of atmospheric disturbances, the internal mechanism between the disturbed detection result and the real center position cannot be determined intuitively. In this study, the fitting function of the LS-SVM algorithm is used to address this problem. The nonlinear regression model of LS-SVM can be expressed aswhere is the Lagrange multiplier constituting a vector and b is the amount of deviation and is the kernel function. The radial basis kernel function with high data antinoising ability is used in this study equation as follows:

In practical applications, and can be learned from a static experiment. The experimental environment here should be consistent with the real test environment. There is no relative displacement between the target and the camera, but camera noise, environmental noise and algorithm errors lead to the displacement of the tracked target. So to obtain the displacement of higher precision, the end condition of model training is that the displacement (noise) variance equation (10) in the time domain is less than the threshold. These weights can be further used for the following practical measurement tasks.

In general, indoor or close measurement conditions easily provide sufficient conditions for the above process. However, for remote outdoor measurement, such as bridge displacement measurement concerned in this study, the piers or bearings are ideal stable points for static experiments. Although these reference points are not completely stable, this is not inconsistent with the end conditions equation (10). But there is a new problem that the measuring point of bridge displacement, such as the mid-span point, is not close to the stable reference point, which means that the measuring environment of the static experiment and the displacement test are not the same. However, the turbulence characteristics within the scope of the engineering site are uniform and will not change significantly in a short time. So in the case of long-term monitoring, the camera must be adjusted to ensure that both the reference point and measurement points are being captured.

2.1.2. Outdoor Static Validation Test

To evaluate the antiatmospheric disturbance performance of the proposed center detection algorithm, an outdoor test was performed (Figure 2(a)) in the summer. The test site is near the river. High temperatures and air humidity create conditions for atmospheric flow on the ground. To reduce the effects of daytime light, the experiment began in the evening. An infrared LED lamp (wavelength of 850 nm) controlled by a high-precision electric displacement table was used as the target (Figure 2(b)). The degraded target is shown in Figure 2(c). The range of the electric displacement table was 500 mm, and the accuracy was 0.1 mm. The object distance was 50 m.

First, the target remained stationary, and the image sequence was obtained with an acquisition rate of 2 frames/s. The gray square centroid method, binary gray centroid method, and proposed method were used to extract the center pixel coordinates, as shown in Figure 3(a). The discreteness of the location results obtained using the proposed method was smaller than those of the other two methods, corresponding to a higher resistance to disturbance. Subsequently, the target was moved in steps of 1 mm through the displacement table. The displacement measurement results are shown in Figure 3(b). To evaluate the measurement accuracy, the root mean squared error (RMSE) was calculated using the data of the displacement table as the reference values. By contrast, the displacement measured using the weighted positioning algorithm exhibited the highest accuracy and stability.

2.2. Data Self-Diagnosis Method Resistance to Adverse Interference

2.2.1. The Principles of Data Diagnosis Based on a Relevance Vector Machine

When the vision-based displacement measurement method was applied to real structures, two types of data anomalies are typically caused by environmental vibrations: (1) missing data owing to the failure of target positioning, attributable to drastic illumination changes, shadows, or severe occlusion and (2) abnormal data (data jump or shift) caused by unexpected camera shaking or other unknown factors. Considering the similarity of the vibration responses of different sections of long-span bridges, the trained RVM was proposed to be used to supplement the missing data and correct the abnormal data. Figure 4 shows the process flow of the RVM regression model. Due to limited space, detailed principles will not be introduced.

The implementation process of RVM-based abnormal data identification and correction is as follows:(a)Select the normal displacement data of different sections as the training sample(b)Initialize the kernel functions and the hyperparameter . Gaussian kernel function was adopted in this study because of its high regression accuracy and operation speed. The insensitive loss parameter is set as u = 0.02, penalty coefficient C is 10, and error accuracy is 0.0001(c)The maximum a posteriori probability (MAP) method was used to solve the weight coefficient w, and then the covariance matrix was calculated(d)Update the hyperparameter according to w and . Repeat step (c) until data residuals satisfy the accuracy requirement, where is the training sample data and is the predicted value(e)Get the predicted samples of the abnormal monitoring samples using the nonlinear model of the trained RVM(f)Locate the abnormal data using the generalized 3-delta method(g)Replace the abnormal data or missing data with predicted data

2.2.2. Verification Test of an Impacted Steel Beam

To evaluate the reliability of the abnormal data diagnosis method, the displacement measurement data of an impact test based on a high-speed camera was analyzed, as shown in Figure 5(a). The beam was 1.5 m long, and six measurement points were evenly arranged on it. Figure 5(b) shows the vibration-response time history curves for all measurement points, generated under the impact of a force hammer. The vibration displacements at P2, P3, and P4 from frames 1∼800 were used as the training samples of RVM. Then the trained model was adopted to correct the results from other frames. The case for abnormal data was established by falsifying the data of measurement point P4. Similarly, parts of the data of measurement points P3 and P4 were deliberately erased to establish the case for missing data. The measured displacement at P2 remained unchanged, and then combining the trained model, the predicted displacement at P3 and P4 was obtained. Finally, the 3-delta method was used to locate the abnormal data. The predicted value is compared with the real measured value, as shown in Figure 5(c). The missing data identification and complementation results are shown in Figure 5(d). It is found that the difference between the predicted and measured values was less than 5%.

3. Application to Vortex-Induced Bridge Vibration Response Measurement

The measured bridge is a sea-crossing cable-stayed bridge with a main span of 888 m, as shown in Figure 6(a). An unexpected vortex-induced vibration (VIV) event was observed on this bridge, potentially caused by the temporary cover placed on the bridge deck during vertical hanger replacement. In general, the use of traditional contact-type sensing technologies on this bridge when VIV occurs is dangerous and time-consuming. In contrast, the proposed camera-based displacement method can satisfy the sudden and urgent measurement requirements. During the day, the camera was set up under the bridge to track the drainage holes evenly distributed under the main girder to measure the deflection, as shown in Figure 6(b). To overcome the problems associated with poor illumination at night, the camera was placed on the shore to track the evenly distributed LED lights, as shown in Figure 6(c). The details of the field measurement at different times are described in the following sections.

3.1. Reliability Evaluation Using a Microwave Radar

As shown in Figure 7(a), to verify the measurement accuracy of the proposed optical method, a high-precision microwave radar was used to measure the deflection at 1/8 span. In this test, the pitching angle of the camera was approximately 23°. As shown in Figure 7(b), the width of the bridge bottom was used for the scale factor calibration of the section of interest, and the evenly distributed drainage holes were tracked. Figure 7(c) shows the measured displacements. Taking the measurement results of radar as the reference value, the RMSE of the two positioning algorithms were calculated, respectively. It can be seen that the results obtained by the proposed centroid tracking algorithm were closer to the data from the radar. However, it cannot be concluded that the proposed method is more accurate. Because the measurement results of the radar correspond to the average displacement for a cross-section and the cross-section position determined by the radar is not necessarily consistent with that tracked by the camera. But to a certain extent, the reliability of the proposed method was proved.

3.2. Abnormal Data Correction

As shown in Figure 8(a), when a camera was set up under a bridge, the field of view may be blocked by passing ships. Therefore, a test was performed to evaluate the abnormal data correction capabilities of the proposed method. Both the left and right drainage holes in each section were monitored. Only the 4/8 span was not disturbed by occlusion, and its complete deflection time history is shown in Figure 8(b). The other sections were disturbed, leading to missing data. Two cases of missing data were considered: (1) the left and right measurement points were not simultaneously blocked, as in the cases of 5/8 and 6/8 span, and (2) both points were simultaneously blocked, as in the case of 7/8 span.

The deflections of the other sections and different measurement points in the same section could be used to compensate for missing data. In general, a higher correlation between the measurement points can help enhance the accuracy of the prediction results. Therefore, a data correlation analysis of the 4/8 and 5/8 spans was performed using the multiple regression analysis method, as shown in Figure 8(c). The correlation of deflection data in the same section was higher than that between adjacent sections. Therefore, it was preferable to use the data from the same section for the data prediction. Nevertheless, it was preferable to use the data of adjacent measurement points when both measurement points in the same section were occluded. Since the moment of occlusion occurrence was known in this test, the displacement from frames 0∼3000 was used as the training samples of RVM. The data completion results are shown in Figure 8(d). In another implementation, the camera underwent an accidental collision, resulting in data jumps and data drifts, as shown in Figure 9. Instead of recognizing the time of mutation and subtracting a constant from the subsequent data, these two types of phenomena are uniformly processed as data anomalies. This is because some of the abnormal data is the superposition of these two phenomena before the camera comes to rest. According to the proposed method, after the 3-delta method identifies the abnormal data and RVM predicts the displacement at the corresponding time and finally replaces the abnormal value with the predicted value.

3.3. Synchronous Monitoring of the Complete Bridge Deflection

The proposed method was used to measure the vibration response of seven measurement points uniformly distributed over the complete bridge. During the nighttime, a moving load test was performed under a VIV event of the bridge. The camera was mounted by the riverside and focused on the LED targets of the bridge, as shown in Figure 10(a). The camera elevation was approximately . A lens with a focal length of 75 mm was used. The distance between adjacent lights was 36 m. For the evenly distributed target points, the object distance L could be calculated using a curve-fitting method [38] based on the center detection results of the LEDs (Figure 10(b)). The object distance L and scale factor of the control sections are summarized in Table 1. Owing to the limited space, part of the measurement results is shown in Figure 11. The vison-based measuring results were processed through the mean-shift function. Therefore, the ice-blue line indicates the coupling response of vortex-induced vibration and moving trucks, and the red curve represents the deflection influence line caused by moving trucks. Finally, the dominant frequency of the bridge during this period was 0.228 Hz, corresponding to the third vibration mode of the bridge.

4. Conclusion

A distraction-free optical structural-displacement measurement method was proposed based on a modified center detection algorithm. Two aspects of innovation were carried out to address the problems associated with environmental interferences.(1)To reduce the center detection error caused by atmospheric disturbances or environmental vibrations, an adaptive window selection strategy based on energy accumulation and a weighted location algorithm based on LS-SVM were developed. A displacement table experiment was carried out in an outdoor environment. The RMSE of the proposed method is 0.04 mm when the distance is 50 m. Compared with the gray square centroid and binary gray centroid methods, the proposed weighted positioning algorithm exhibited the highest accuracy and stability.(2)The problems of missing data (caused by target occlusion) and data jump or drift (caused by camera shaking) cannot be effectively solved through image processing. As the displacement of different stations is related to each other in both space and time, an RVM-based data self-diagnosis method for SHM was introduced. The effect of the proposed method was verified by manipulating the data of a vibration test artificially.

The proposed displacement method was applied to a long-span bridge. The measurement accuracy is verified by comparing it with a high-precision microwave radar. Especially, the environmental interference such as targets being obscured by ships and cameras being accidently shaken is solved well in the real bridge application. This study is of great significance for long-term vibration monitoring of long-span bridges.

Data Availability

Access to data is restricted due to the third-party rights.

Conflicts of Interest

The authors declare that they have no conflicts of interest.

Acknowledgments

The research presented was financially supported by the National Key Research and Development Program of China (2020YFC1511900).