Abstract
The shortage of personnel and the high cost have become a major pain point in the current safety supervision work of the inspectors. Aiming at the problem that the aircraft maintenance inspector could not visit the scene in person during the epidemic, a remote safety supervision platform was built based on intelligent glasses and 5G network, and the real-time monitoring of the aircraft skin rivet status was realized. And a method of aviation rivet classification and anomaly detection based on deep learning algorithm was proposed. Firstly, according to the appearance of rivet head, the aviation rivet is classified, the data set of aviation rivet is made, and the aviation rivet classification and anomaly detection model are constructed. Evaluate the detection results from such indicators as confidence, precision, recall rate, and mAP and compare the algorithm with the detection results of Yolox-s, Yolox-m, Yolov5-s, Yolov5-m, and Yolov4. The results show that (1) the algorithm proposed in this paper can realize the classification of aviation rivets and the detection of abnormal conditions, the confidence of the detection results is more than 90%, and the average precision, recall, and AP value are above 95%, 85%, and 88%, respectively. (2) The order of rivet classification and abnormal detection effect from good to bad is Philips screws, round head rivets, flat head rivets, countersunk head rivets, blind rivets, and abnormal condition. (3) Compared with other algorithms, the aviation rivet classification abnormal target detection based on deep learning has absolute advantages in accuracy and speed.
1. Introduction
The famous saying of the aviation industry is “strive to reduce the weight of each gram” [1]. In order to reduce the weight, engineers take great pains. Besides, the aircraft envelope is very thin, and the welding process is very complex. The welding materials will generate a lot of heat in flight, and they are not suitable for use on the aircraft. Moreover, the use of rivets can greatly reduce costs, and the small size of rivets also has a certain effect on reducing resistance in flight. Therefore, rivets are essential for aircraft. The commonly used connection methods in aircraft manufacturing include riveting, bolting, gluing, and welding. With the continuous development of the aviation industry, the flight speed and height have been continuously improved, especially after the high hardness aluminum alloy has become the main material for aircraft manufacturing, the disadvantages of aluminum alloy high temperature deformation and mechanical property reduction have become increasingly apparent, and the resonance and stress changes in flight make the welded joints very easy to break. Therefore, riveting becomes the final choice for connecting aircraft components. In aircraft manufacturing, the proportion of rivet connection is far greater than that of welding. The main reasons are as follows: (1) light materials such as composite materials are not suitable for welding; (2) the weld technology is prone to metal fatigue; (3) riveting is more reliable and stable. Rivets are more stable than welding, making the aircraft safer. Therefore, it can be seen that the efficient connection mode of aviation rivets can strengthen the stability of the structure and is closely related to flight safety.
On July 7th, 2022, during the flight of an aircraft of Air China, the incident of the engine rivet falling off and loosening caused widespread concern from all walks of life. The main material of modern large civil aviation airliner is aluminum alloy, and riveting technology is usually used to reduce the weight of the aircraft. As the lightest fastener with the best connection strength, rivets are naturally favored by major aircraft manufacturers. According to statistics, each aircraft is equipped with millions of rivets, mainly including solid rivets and special rivets [2]. The main cause of rivet looseness is flight vibration. If the rivet falls on the runway during takeoff and landing, it may be inhaled by the rear aircraft as foreign object debris on the runway, resulting in tire puncture, engine and fuselage damage, and even major aviation accidents. Therefore, it is of great significance to improve flight safety on how to quickly and accurately locate the rivet position, identify the rivet classification, and whether there is any abnormal situation.
2. Literature Review
At present, the research on inspection of aviation rivets at home and abroad is still in its infancy, and the existing achievements can be roughly divided into three categories. (1) Based on magnetooptical imaging and pulsed eddy current technology, the anomaly detection of aviation rivets is realized. Among them, Zeng et al. [1] proposed a finite element model for automatic inspection of aviation rivets, He et al. [2] proposed a nondestructive inspection method for riveting structural defects, and Gao et al. [3] and Uchanin [4], respectively, realized the inspection of rivet surface crack defects. (2) Based on optical technology, the detection of aviation rivet anomalies is realized. Among them, Overton [5] has carried out rapid detection of the three-dimensional coordinates of aircraft rivets, and Li [6] proposed an adaptive local point cloud density calculation method to achieve the detection of riveting quality; Bi et al. [7] developed a cross line structured light visual inspection system to detect the rivet hole orientation. (3) Aviation rivets recognition based on computer vision, the main application scenarios are as follows: (1) detection of rivet damage. Chen et al. [8] and Yang et al. [9], respectively, used support vector machines and noise resistant local binary mode coding algorithms to detect rivet structural defects and rivet surface defects; (2) check the rivet position. Xing et al. [10] proposed a detection method based on machine vision, which realized the detection of rivet position and rivet size. Yu et al. [11] proposed a detection method based on monocular vision, which solved the problem of rivet identification and positioning; (3) inspection of rivet holes. Tian et al. [12] proposed a rivet hole machine vision recognition algorithm based on scattered point cloud.
Target detection is a hot topic in the field of computer vision. It has a wide range of applications in robot navigation, intelligent video surveillance, and aerospace [13–15]. Deep learning, a branch of machine learning, is a frontier for artificial intelligence, aiming to be closer to its primary goal-artificial intelligence. At present, there are two widely used target detection algorithms based on deep learning: two-stage detection based on candidate regions and single-stage detection based on regression [16–20]. The former includes R-CNN and faster R-CNN. The main problem of these algorithms is that the detection time is too long to meet the real-time requirements. The single-stage detection methods based on regression include YOLO and SDD. These algorithms directly regress the candidate frame and category of the target in multiple positions of the image, which is more real-time.
In fact, magnetooptic imaging is easily disturbed by external factors, which will directly affect the effect of image detection. Meanwhile, pulsed eddy current technology cannot get rid of the impact of lift-off on the test results, the equipment requirements are very high, and the test model is not complete yet. To sum up, although the existing research has made some achievements in the detection of aviation rivets, there is still much space to improve the accuracy and speed of the detection results. In addition, the research results of rapid identification and location of aviation rivet classification and abnormal conditions have not appeared. Therefore, the author first classified the aviation rivets, proposed the aviation rivet classification and anomaly detection algorithm based on deep learning, and realized the target detection of rivet classification and anomaly in the process of walkaround inspection, in order to help the maintenance engineers to locate various rivet-related failures and ensure aviation operation safety.
3. Modeling
3.1. Classification of Aviation Rivets
According to the national standards commonly used in aerospace, aviation rivets include round head, 90° countersunk head, 120° countersunk head, pop rivet, and flat head rivet, and their codes are, respectively, GB867, GB869, GB954, B12615, and GB109. So in this paper, we will adopt the target detection method based on deep learning, and the rivets will be reclassified according to their appearance images (as shown in Table 1): (1) countersunk head rivets, marked as “CHr,” are mainly used for riveting seams on the aircraft surface that need to be smooth and lightly loaded, which can effectively reduce wind resistance; (2) flat head rivets, marked as “FHr,” are mainly used for strengthening joints; (3) round head rivets, marked as “RHr,” are mainly used for riveting seams bearing large transverse loads to improve the strength of the fuselage and maintain structural stability and are widely used in fuselage and wings; (4) blind rivets, marked as “Br,” are mainly used for single-sided riveting and are applicable to parts of the fuselage that are not convenient for double-sided riveting; (5) Phillips screws, marked as “Ps,” are mainly used in the fuselage and its interior for fastening; (6) abnormal, including loose or falling rivets, marked as “Abn.”
3.2. Target Detection Process of Aviation Rivet
In the postepidemic period, repeated regional epidemics and inconsistent standards of epidemic prevention measures of foreign airlines have brought great difficulties to the on-site supervision and review of civil aviation. How to implement the supervision and review procedures and timely grasp the safety information on the basis of ensuring the epidemic prevention safety of personnel is the key problem to be solved urgently in the civil aviation safety supervision work after the epidemic. At the same time, the continuous development of 5G-based communication technology, remote video transmission technology, intelligent glasses visual synthesis, and interaction technology has created a technical foundation for the construction of a remote and intelligent safety supervision and interaction platform for civil aviation, with a view to effectively improving the safety and efficiency of civil aviation safety supervision during the “epidemic prevention normalization” period. Therefore, based on intelligent glasses and 5G network, this paper has built a remote safety supervision platform for maintenance inspectors to achieve real-time detection of skin rivet classification and abnormal targets in the walkaround inspection, thus implementing remote real-time supervision and guidance of problems, and the flow chart of aviation rivet target detection is shown in Figure 1.
The maintenance engineer collects videos and pictures of rivets at specific positions of the fuselage with Google smart glasses and, at the same time, connects with the computer terminal through 5G network to transmit and save videos and images in real-time and then extracts the video frame by frame and save it as an image file, and carries out data enhancement processing on the image data of aviation rivet image data, including mosaic data enhancement and mixed data enhancement. And then, the rivet samples are divided and labeled, and the rivet samples are trained by using the learning method combining migration and freezing training, so as to build the classification and anomaly detection model of aviation rivets. Finally, experts can realize remote monitoring and guidance of aircraft status according to the detection results, improve the efficiency of maintenance, and ensure aviation operation safety.
3.3. Target Detection Algorithm of Aviation Rivet
As an excellent target detection algorithm at present, Yolox algorithm has the advantage that the type and speed of target detection have been greatly improved [17–20]. The aviation rivet classification and anomaly detection algorithm based on deep learning are improved on the basis of Yolox algorithm. Its network structure, loss function, and reasoning process are described as follows.
3.3.1. Network Structure
The network structure of the algorithm proposed in this paper is shallower than that of Yolox algorithm. By reducing the parameters involved in the operation, the network reasoning speed is improved. The network structure (Figure 2) still takes the DarkNet-53 structure as the baseline and consists of the backbone network, neck, and decoupling detector head, including the focus layer, cross stage partial layer, spatial pyramid pooling module, upper sampling layer, connection layer, and decoupling detector head. Among them, the focus layer plays a downsampling role and reduces the loss of feature information by slice splicing. The cross stage partial layer is designed based on the baseline of the residual block, which not only deepens the network’s ability to extract feature information better but also solves the problems of gradient disappearance and gradient explosion through layer hopping connection. The spatial pyramid pooling module is composed of two convolutional layers and parallel maximum pooling layers of different sizes. By increasing the receptive field through the parallel maximum pooling layer, the feature information of objects with different sizes can be better extracted.
3.3.2. Loss Function
As IoU and GIoU may make the training process difficult to converge when they are used as loss functions, in this paper, we will consider the distance, overlap rate, scale, and penalty term between the target box and the prediction box, that is, CIoU is used as the loss function (Formula (1)), so the prediction box can better approach the position of the target box, making the prediction results more accurate. where and represent the center point of anchor box and target box; represents the European distance between two center points; represents the diagonal distance of the minimum rectangle covering the anchor box and the target box; is a parameter used for trade off; is a parameter used to measure the consistency of aspect ratio.
3.3.3. Reasoning Process
The reasoning process of aviation rivet classification and anomaly detection algorithm based on deep learning can be summarized into three stages: feature extraction stage, feature information collection and processing stage, and prediction stage. First, the input size of the input terminal is . The focus layer is cut, spliced, and convolved to get . After layer by layer reasoning of convolution layer, cross stage partial layer, and spatial pyramid pooling module, the size of , , and feature map can be obtained in the middle, lower, and lower layers of the backbone network and then participate in network neck feature fusion. Then, the network neck adopts the structure of feature pyramid networks and path aggregation network to fuse three shallow semantic features and deeper semantic features, so that the overall network can better extract target feature information. Finally, the feature information is input to the decoupled detector head for reasoning, and then, the prediction prospect, category, and position of the target can be obtained.
3.4. Methodology
Step S1: the maintenance engineer wears smart glasses (or other video capture devices) to perform walkaround inspection, collect video and pictures of rivets at specific positions on the fuselage, and at the same time, connect with computer terminals through 5G network to transmit and save video and images in real-time. It is worth noting that the captured aerial rivet pictures should be taken from different angles, different distances, and different brightness, and the video data can also be used to extract appropriate samples by frame
Step S2: sort out the collected aviation rivet data, mark the category target frame, and divide the data set. Use the labeling tool to label various types of data. The target box categories are CHr, FHr, Br, RHr, Ps, and Abn. Divide the labeled image data into training set, verification set, and test set according to the ratio of 7 : 1 : 2. After the annotation is completed, the target data set is enhanced by mosaic data enhancement and mix up data enhancement
Step S3: the method of migration training and freezing training is used to train the aviation rivet classification and anomaly detection model. Carry out pretraining based on VOC data set, get the pretraining weight, and automatically end the training when the loss value does not fall for many times. At the same time, freeze some common pretraining weights, such as backbone network, and put more resources on the calculation of network parameters in the later part of the training, which can effectively reduce the training duration and greatly improve the resource utilization. When the loss value does not decrease for many times, the freezing training will be automatically ended, and then, the thawing training will be started, and then, all layers of the model will be trained together
Step S4: Based on the weight of aviation rivet classification obtained after training in step S3, the aviation rivet data is classified, and abnormal recognition is carried out. The inference process of convolution neural network can be summarized into three stages: feature extraction stage, feature information collection and processing stage, and prediction stage. Use the convolution neural network based on the DarkNet-53 structure as the backbone network to extract features for neural network calculation and use the focus layer, cross stage partial (CSP), spatial pyramid pooling (SPP), upper sampling layer, and connection layer as the neck network to perform feature fusion and finally use the decoupling detection head to generate detection data
Step S5: In order to verify the effectiveness of the proposed algorithm in target detection, the recall rate, accuracy rate, AP (average precision) value, and average precision value (mAP) are used to evaluate the detection results
3.5. Evaluation Criteria
In order to verify the effectiveness of the algorithm proposed in this paper in target detection, recall rate, accuracy rate, AP value, and average precision value are used for evaluation. In the following formula, TP (true positive) is the number of correctly detected targets to be detected, FN (false positive) is the number of undetected targets to be detected, and FP (false negative) is the number of incorrectly detected targets to be detected.
Among them, the recall rate refers to the probability of being detected correctly among all detected targets, and the accuracy rate refers to the probability of actually being the correct target among all detected targets.
Both the AP value and the mAP value are used to measure the overall performance of the model. The AP value does not refer to the average accuracy value of all detected targets of a single category, but the figure area value is enclosed by the P-R curve with recall as the abscissa axis and accuracy as the ordinate axis. In actual calculation, the integration method can be used to obtain this value, as shown in Formula (3). The average value of all kinds of target AP values can be used to measure the overall detection performance of the model, that is, the mAP value.
4. Example and Discussion
4.1. Construction of Aviation Rivet Data Set
In this paper, we take the fuselage rivets of Boeing series aircraft as the research object, collect rivet data, store it in the terminal equipment, label the data set with the labeling tool, and obtain 3000 corresponding PASCAL VOC format data sets in total (Figure 3). Among them, there are many samples of countersunk rivets, flat head rivets, round head rivets, and Phillips screw rivets, and the difference is very small. There are few samples of abnormal conditions and blind rivets, about 5000 and 3000, respectively. The image marked according to the appearance of rivet head is shown in the following figure.
(a) Countersunk head rivets
(b) Flat head rivets
(c) Round head rivets
(d) Blind rivets
(e) Phillips screws
(f) Abnormal
4.2. Training Process of Data Set
The algorithm proposed in this paper is implemented on the open source framework TensorFlow. The computer configuration is as follows: Intel i7 12700H processor, 16G memory, and independent graphics card NVIDIA GeForce RTX3070. Set the freezing of the first 100 layers of networks according to the actual training situation, and the score threshold is 0.5, batch size is 10, max box value is 20, the model image size is , and the initial learning rate is 0.001. Then, load the pretraining weight to start training. When the loss value does not decrease, unfreeze the training and set the batch size to 8; the learning rate is 0.0001, and the other parameters remain unchanged. Among them, migration training refers to training on a large benchmark data set to obtain the pretraining weight and then fine-tuning the pretraining weight parameters, which can significantly improve the learning efficiency of the model. Freezing training is to freeze the weight of backbone network in the pretraining weight on the basis of migration training and then adjust the remaining parameters through training. After a period of training, unfreeze all networks for fine-tuning to obtain the final weight. Before the training of input sample pictures, the data enhancement unit uses mosaic and mix up data enhancement functions to enrich the data set samples by randomly scaling, clipping, arranging pictures, and copying and pasting sample data on this basis.
The change of the loss value of the training set and the verification set with the training times in the training process is shown in Figure 4. The abscissa and ordinate are the epoch and loss values, respectively. Before the generation 120, because the learning rate is relatively high, the loss value of the training set and the verification set decreased rapidly, so that the model can quickly reach the fitting. In order to prevent the overfitting of the model, the learning rate will decrease with the advance of the times. After 120 times, the decline rate of the loss value of the model will slow down. Finally, after 300 generations of training, the loss value of the verification set gradually converges to 2, and the loss value of the training set gradually converges to 2.3. In the training process, train loss is positively correlated with val loss, which indicates that the training of the model is normal, and there are no problems such as overfitting of the network, unreasonable structure of the network, and poor quality of the data set.
4.3. Prediction Result Analysis
After the network training is completed, input the video file to be detected, and then, the aviation rivet classification and anomaly detection results are shown in Figure 5.
(a)
(b)
(c)
(d)
It can be seen that the algorithm can realize the classification of fuselage rivets and the detection of abnormal conditions and calibrate the position and confidence of the target. In Figure 5(a), five objects were detected, including countersunk rivets, round head rivets, and abnormal conditions. The confidence level of three types of objects reached about 90%. The confidence level of the detection results of countersunk rivets was the highest, and the confidence level of the detection results of abnormal conditions and round head rivets decreased in turn. The target (CHr “0.93” 684 448 741 521) indicates that the target type is countersunk rivet with confidence of 93%, the relative coordinates of the center point of the target frame are 684 and 448, and the height and width are 741 and 521, respectively, and the detection speed is 33.25 frames per second. In Figure 5(b), six objects are detected, all of which are flat head rivets. It can be seen that the confidence of the algorithm in the detection results of flat head rivets is more than 91%. Among them, the target (FHr “0.93” 253 479 307 542) indicates that the target type is flat head rivet with confidence of 93%. The relative coordinates of the center point of the target box are 253 and 479, the height and width are 307 and 542, respectively, and the detection speed is 32.15 frames per second. In Figure 5(c), five targets are detected, all of which include blind rivets. It can be seen that the confidence of the algorithm in the detection results of blind rivet targets is more than 90%. Among them, the target (Br “0.93” 399 311 454 371) indicates that the target type is blind rivets, with confidence of 93%, the relative coordinates of the center point of the target box are 399 and 311, the height and width are 454 and 371, respectively, and the detection speed is 30 frames per second. In Figure 5(d), eight objects are detected, all of which are Phillips screws. It can be seen that the confidence level of the algorithm for the detection results of cross recessed rivets is about 90%. Among them, the target (Ps “0.92” 462 531 521 591) indicates that the target type is a Phillips screw with a confidence of 92%, the relative coordinates of the center point of the target box are 462 and 531), the height and width are 521 and 591, respectively, and the detection speed is 34.33 frames per second. It can be seen that the reliability of the proposed algorithm for the classification and anomaly detection results of aviation rivets is about 90%, and the detection speed is more than 30 frames, which has met the requirements of real-time detection.
4.4. Model Evaluation
According to the evaluation indexes related to target detection mentioned above, the recall rate (), precision rate (), and average precision rate (AP) value of target category detection of aviation rivets are obtained as shown in Table 2.
It can be seen that the model can detect the normal and abnormal conditions of various rivets. Among them, the detection accuracy of targets is higher than 95%, especially the recognition accuracy of flat head rivets and round head rivets is more than 99%, and the accuracy of blind rivets is relatively low, about 95.08%. The target recall rate is more than 85%, especially the recognition accuracy of round head rivets and Phillips screws is more than 96%, while the recall rate of blind rivets and abnormal conditions is relatively low, about 86%. The recognition accuracy of target AP values is more than 88%, and the AP of other types of rivets is more than 92% except for abnormal conditions. On the whole, the order of the inspection effect of the model on all types of rivets from good to bad is Phillips screws, round head rivets, flat head rivets, countersunk head rivets, blind rivets, and abnormal conditions. The reason is that the number of samples of blind rivets collected is relatively small, and the quality is low.
Overfitting means that the gap between training error and test error is too large. In other words, the complexity of the model is higher than the actual problem. The model performs well in the training set but poorly in the test set. To solve the overfitting problem, it is necessary to significantly reduce the test error without excessively increasing the training error, so as to improve the generalization ability of the model. Acquiring and using more data are the fundamental methods to solve overfitting. Mosaic data enhancement used in this paper is specifically to read 4 pictures randomly at a time in the prepared data set and then flip (flip the original picture left and right), zoom (scale the original picture size), color gamut change (change the brightness, saturation, and hue of the original picture), and other operations on the 4 pictures, respectively. After the operation is completed, place the original image on the top left according to the first image, the second image on the bottom left, the third image on the bottom right, and the fourth image on the top right in four directions and then use the matrix method to cut down the fixed area of the four images and splice them into a new image. The new image contains a series of contents such as a label box. After this operation, the background information of the picture is enriched, and the four pictures are spliced together, which also increases the number of data set samples in disguised form. When batch normalization is performed, it is equivalent to calculating four pictures at the same time, which greatly improves its training efficiency.
4.5. Comparison of Different Algorithms
In order to verify the performance of the algorithm proposed in this paper, it is compared with the traditional Yolov4, Yolov5 algorithm, and other versions of Yolox algorithm in the same scene. The performance comparison of the detection results is shown in Table 3.
As can be seen, all algorithms can achieve the detection of classification of aviation rivets and abnormal situation, and the overall performance ranking from high to low is the algorithm proposed in this paper, Yolox-s, Yolox-m, Yolov5-s, Yolov4, and Yolov5-m. Among them, the precision, recall, and mean precision values obtained by the algorithm in this paper are 98.12%, 92.09%, and 95.07%, respectively, which are higher than those obtained by other versions of the same algorithm, the Yolox-m and Yolox-s algorithms by 4.81%, 8.06%, 6.49%, 2.71%, 5.45%, and 5.62%, respectively. The GPU processing speed of the proposed algorithm is 84.56 f/s, which is 50.97% higher than other versions of the Yolox-m and Yolox-s algorithms and 16.39% higher than the Yolov5-m, Yolov5-s, and Yolov4 algorithms, respectively. Compared with the same version algorithm, the speed of aviation rivet classification and anomaly detection based on deep learning is significantly improved, which should be more attributed to the use of lightweight network structure framework. The improvement of accuracy of aviation rivet classification and anomaly detection should be attributed to the improvement of detection head, anchorless mechanism, and label assignment method.
5. Conclusions
(1)One method of aviation rivet classification and anomaly detection based on deep learning was proposed in this paper, which can achieve classification and abnormality detection of aviation rivets with confidence level above 90% and average accuracy, recall, and AP value above 95%, 85%, and 88%, respectively(2)The algorithm presented in this paper can real-time detect the classification of aviation rivets and the abnormal situation, and the order of detection effectiveness is in the order of good to bad: Phillips screws, round-headed rivet, flat-headed rivet, countersunk head rivets, blind rivets, and abnormal situation(3)Since the algorithm uses a more lightweight network architecture, the accuracy and speed of aviation rivet classification and anomaly detection have absolute advantages compared to the other Yolox algorithms in the same version as well as the Yolov5 and Yolov4 algorithms
Data Availability
The data used to support the findings of this study are included within the supplementary information file, which is named rivets.
Conflicts of Interest
The author declares that he have no conflicts of interest.
Acknowledgments
This work is supported by the Sichuan Science and Technology Project (2022YFG0196), the Civil Aviation Safety Capacity Building Project (14001000100020J002), and the scientific research of Civil Aviation Flight University of China (J2022-060).