Abstract
Due to the repeated bearing of mechanical operations and natural factors, the container will suffer various types of damage during use. Adopting effective container damage detection methods plays a vital role in prolonging the service life and using function. This paper proposes a multitype damage detection model for containers based on transfer learning and MobileNetV2. In addition, a data set containing nine typical types of container damage is established. To ensure the validity and practicability of the model, we conducted tests and verifications in the actual port environment. The results show that the model can identify multiple types of container damage. Compared with the existing models, the damage detection model proposed in this paper can ensure the identification effect of various types of container damage, which is more suitable for the actual container detection situation. This method can provide a new idea of damage detection for container management in ports.
1. Introduction
As a protective barrier for cargo, containers are an indispensable part of modern logistics. At present, there are containers with a service life of more than 20 years in the fleet. Most containers may have a longer service life on some trade routes, and such containers are no longer in the best use condition [1]. Due to the repeated bearing of mechanical operations and natural factors, various types of damage may occur to the container during use. Damage of containers will lead to reduced container function and quality [2]. With the increase of the service life, the deterioration of containers is gradually increasing, which seriously affects the safety, applicability, and durability of the structure.
The port usually uses large-scale inspection devices and arranges security personnel to detect container damage when entering and exiting gates. The container damage detection avoids the risk of damage caused by the container in the later transportation process [3]. Security inspectors conduct inspections following the transfer requirements of containers and visually inspect the container structure and accessories. Then they release slightly damaged containers and prohibit shipping of largely damaged containers [4]. This detection method is not only time-consuming and labor-intensive but also places high demands on security personnel.
Many researchers are studying methods to assist or replace manual container inspection. The detection technologies for container damage in ports include optical characters, laser scanning, and 3D imaging [5]. Machine learning is a hotspot of image detection research. The research of container detection mainly focuses on the study of special containers (such as dangerous cargo containers and liquid natural gas containers) or a single particular type of damage (such as unclear surface container numbers, holes, and missing corner casting). Due to the diverse kinds of container damage, the actual container damage may be a single type or a combination of multitype of damage. If only detect special containers and particular damage types, it is challenging to meet the actual case of damage container inspection requirements. Therefore, we propose a multitype damage detection method for containers based on transfer learning.
Although the related research of container damage detection based on deep learning is in the development stage, there is much literature on this research direction in construction and civil engineering. Cha et al. [6] used CNN to construct a detection model for concrete cracks in building health structure detection. Also, based on the health structure detection of buildings, Han and Tang [7] used a hybrid data enhancement method combined with the Faster R-CNN recognition framework to achieve a 5.14% improvement in the average accuracy of damage detection. Maeda [8] addressed road damage identification and detection and proposed a detection model that can identify eight types of damage. Wang et al. [9] used Faster R-CNN as a recognition framework to complete road damage detection. Zhang et al. [10] used CNN to construct a detection model for road asphalt cracks [11]. Aiming at tunnel lining defect detection, Xue and Li [12] proposed a CNN-based damage detection method [13]. In addition, the computer vision method based on deep neural networks can also be used for road sewer maintenance problems [14]. This research direction is widely used in damage detection in other fields. Promising experimental results have been obtained, illustrating the research value and feasibility of the damage detection method based on deep learning.
Based on the transfer learning and MobileNetV2, this paper proposes a multitype damage detection model for containers. Moreover, we establish a data set containing nine typical types of container damage (including seven types of damage, regular container, and port environment). To ensure the validity and practicability of the model, we conducted tests and verifications in the actual port environment. The results show that the model can identify multiple types of container damage. Compared with the existing models, the damage detection model proposed in this paper can identify multiple types of container damage. The model is more suitable for the actual container detection situation at the port.
The rest of this article is structured as follows. Section 2 introduces the related work on container damage detection. Section 3 presents the classification of container damage types and the process of establishing a data set. Section 4 elaborates on the construction details of the detection model. Section 5 analyses and discusses the test and on-site experiment results. Section 6 gives the conclusion and prospects of this paper.
2. Related Works
Due to the visual limitations of manual inspection, work efficiency, and personnel safety, there are many types of research on assisting manual inspection containers. Some scholars believe that object recognition and classification technology in computer vision can solve the inspection containers problem. In contrast, other scholars believe that container damage detection aims to confirm cargo safety, so monitoring the status of the container and its internal cargo is the point. Therefore, research of container damage detection research can be divided into three aspects: focusing on container inspection research, cargo inspection research, and other auxiliary inspection research.
The current research shows the limitations and development trends of container damage detection. First of all, the automated container damage detection methods have not been applied on a large scale. Secondly, the container damage detection algorithm has a single feature to a multifeature detection method. In addition, most of the existing research is based on the container damage image data set established by each. The lack of a unified container damage image database makes it difficult to conduct objective comparison tests and performance evaluations of various existing algorithms under uniform conditions.
2.1. Research on Container Inspection
The research focusing on container inspection is concentrated in the field of computer vision. Most container damage is reflected in the appearance. Furthermore, the limitation of workforce detection can be compensated by detecting the damaged images.
As early as 1995, Nakazawa et al. [15] proposed a three-dimensional automatic detection device through the illumination difference. The device uses traditional image segmentation algorithms to segment holes and cracks based on threshold settings to achieve damage detection. Oh et al. [16] researched and proposed a container damage detection system based on the correlation coefficient method set at the port gate. Son et al. [17] proposed a function called Capsize-Gaussian-Function to detect damaged or deformed edges of containers. Based on this research, Son and Kim [18] expanded this method in 2005, using it to estimate the damage on the outer surface of containers and collecting data from the Port of Busan for verification. Kim et al. [19] proposed an automatic container identification system based on the self-organizing supervised learning algorithm of ART2 to solve unclear container identification caused by the blurred surface in the damaged container. At present, mature commercial systems for container detection have been applied to real ports, such as laser-based systems of container damage detection (Lase CDI) and container damage automatic detection systems (Visy) [11].
In recent years, machine learning technology has widely appeared in the research of container damage detection. Compared with traditional computer vision, the deep neural network model has apparent advantages in recognition accuracy and efficiency. İmamoğlu [13] proposed a method to detect container cracking based on machine learning and used the image in the port information system to verify it. Mi et al. [11] proposed an automatic detection system for container corner casting identification. Based on Faster R-CNN, MobileNet, and ResNet, Emil [5] researched a method of automatically detecting the damage of container corner castings.
Furthermore, Pambudi et al. [20] implemented a recognition algorithm for blurring codes, signs, and labels in container images. To solve complex lighting conditions and background pollution in container detection, Zhiming et al. [21] proposed a robust target detection algorithm based on a deep learning algorithm to detect and identify the container code. In addition to the fuzzy detection of the container surface, Diao et al. [22] proposed automatic identification and positioning of container keyholes based on SVM and HOG.
2.2. Research on Cargo and Auxiliary Inspection
In the research focusing on cargo inspection, Bukkapatnam et al. [23] proposed monitoring the integrity and safety of packages during transportation using wireless vibration sensors based on the Zigbee protocol. They believe that typical mechanical vibrations such as shaking, tilting, collision, and sliding of the container during transportation have caused damage to the container and the cargo. Due to the wide application of information systems, Fu et al. [24] proposed a method to control the entire transportation process and safety issues through GIS and radio frequency identification technology given the transportation characteristics of containers, to realize the management of logistics information in the process of container multimodal transportation. A large number of researches on information management systems and technologies have emerged in the field of container multimodal transportation and decision support. However, most of the literature only elaborates on its research ideas. It lacks the description of the realization and application of essential issues such as monitoring cargo transportation conditions and cargo information security.
Andziulis et al. [25] proposed a mobile control system based on RFID and sensors. The system’s primary goal is to monitor the real-time way and work of cargo conditions during transportation and evaluate the potential risks of cargo transportation, thereby providing mobile cargo security services. However, sensor-based container damage detection has data errors, mechanical damage, sensor system damage, and external environmental factors, which affect the credibility and accuracy of real-time decision-making [26]. In addition, due to differences in transportation requirements and environments for different goods, it is difficult to form a modular container detection standard based on sensor monitoring.
With more and more researches on container damage detection, many related studies have also emerged. Singh et al. [27] proposed a reasonable loading method. They believe that incorrect loading and lack of loading safety measures in the trailer may cause damage to the container and cargo and may cause further damage to the container and load in the subsequent process. Meng et al. [28] developed a real-time container panoramic generation system based on container surveillance video, aiming to solve the problem of obtaining panoramic images of container damage detection. The real-time container panorama generation system uses texture feature stitching technology, which will support container damage detection based on port monitoring equipment. Cha and Noh [3] proposed a management information system. The system can manage container damage information through RFID and image storage technology at the container entry and exit gates, reduce inspection and processing time, and ensure that the data can be traced.
2.3. Proposed Approach
The field of image recognition generally centers on convolutional neural networks (CNN). Image classification is widely used in many areas, including medical image recognition. Early image classification technology was mainly based on the artificial extraction of features and then developed into the learning of feature expression.
Initially, scale-invariant feature transform (SIFT), gradient direction histogram (histogram of oriented gradient, HOG), and other feature extraction methods encoded low-level features. Then the encoded features are gathered; finally, the support vector machine (SVM) and other classifiers are used for image classification. Although the learning method of feature expression can extract a part of image features, this method is prone to feature loss and poor generalization performance. Ultimately, it is challenging to meet the requirements of ideal image classification accuracy. In deep learning, a convolutional neural network (CNN) can complete the step-by-step expression of input information from shallow learning to deep learning and extract more accurate features.
Representative deep network models including LeNet [29], AlexNet [30], VGG [31], GoogLeNet [32]. AlexNet, and VGG to LeNet mainly aim to increase the number of channels and deepening the neural network layers for the convolutional layer module and the fully connected layer module [31]. GoogLeNet has absorbed the idea of tandem network and made great improvements on this basis [33, 34]. Given the situation that the training error tends not to drop but rises after adding too many layers, ResNet [35] and DenseNet [36] have appeared one after another.
However, in some real application scenarios, such as mobile or embedded devices, such a large and complex model is difficult to be applied. Therefore, a small and efficient CNN model is essential in these scenarios. In 2017, MobileNet proposed by Google is an efficient and lightweight network model that can be used for mobile and embedded vision applications. MobileNet is based on a streamlined architecture. The core idea is to use a deeply separable convolution operation. Under the same weight parameter amount, compared with the standard convolution operation, the calculation amount can be reduced several times to improve the network operation speed [37].
Deep learning is widely used in the research of container detection to solve the problem of container damage detection. In this direction, we propose a multitype container damage detection model. The framework diagram of the research approach is illustrated in Figure 1. This paper’s proposed approach consists of two parts: (1) classification of container damage types and construction of data sets and (2) CNN model for detecting damage of multiple types of containers.
In the first part, we classify the types of container damage according to official documents and international standards to provide a theoretical basis for later container detection. In addition, the data set is constructed through open-source databases and port monitoring data. Furthermore, we used data augmentation to expand the data set. In the second part firstly, the images are optimized by weak supervision. Then the image is input into the pretraining models of MobileNetV2 and InceptionV3, and the pros and cons of the two models are compared after training and fine-tuning. Finally, the appropriate detection model is selected to complete multiple types of container damage detection.
3. Multitype of Container Damage
3.1. Classification of Container Damage
There are usually two ways to classify container damage [38]: (1) classification according to structural or nonstructural damage and (2) classification according to the form of injury. Structural damage includes cracks, holes, depressions, deformations, and cut marks. Nonstructural damage includes rust, odor, light transmission, moisture, nails or other protrusions, damage to parts, lost, and looseness. For example, most of the leasing service standards divide container damage into structural damage (cracks, holes, dents, and deformation), functional damage (corner casting damage, component damage, lost, and looseness), and permanent damage (transparent damage, light, and wet). According to the classification of the damage form, the damage can be divided into damage type (crack, cut, hole, hole, and rust.), deformation type (bend, deformation, and depression), and environmental type (worm-eaten, wet damage, and peculiar smell). Most container inspection standards are classified according to this method.
The current container inspection is mainly based on the ISO 9897 General Container Standard [39] issued by the International Standards Organization (ISO). Therefore, based on this document, we integrate the container damage classification in Unified Container Inspection and Repair Standards [40], Container Leasing Service Standards [41], TG-01: COA Standards for Good Cargo Load [42], and Container Entry and Exit Station Inspection and Handover Requirements [43] and summarize the common types of container damage (details in Table 1). Because of the containers’ tightness, damage, falls, and collapse are frequent during transportation. Furthermore, these are different from other types of damage. These will be affected by the severity and external environment and aggravate the injury, which gives birth to other types of damage. Therefore, open and collapse are regarded as the type of container damage.
In a real scene, there may be relatively blurred boundaries between the types of damage, for example, between damage, rusty, and hole. When the container is damaged in a humid environment, the damaged part is prone to rust. When the rust is severe, the surface material of the container becomes thinner and brittle, and holes will appear. Another situation exists between bent and dent. When the degree of container depression is severe, container deformation will occur. In this study, the evolutionary relationship between the various types of damage is ignored. Only the salient characteristics of the types of container damage are considered, and the deterioration of the container is defined as visible to the human eye.
3.2. Data Set of Container Damage
In this study, an image data set of container damage was constructed according to the damage classification in Section 3.1. The data set consists of two parts: training and validation set and test set. The open-source databases are used as the training and validation set data source. Search in databases with container and damage characteristics as keywords (K1 = container and K2 = damage, hole, bent, dent, rusty, open, or collapse), and then images used as samples are obtained through manual screening. The sample images are divided into nine types (according to seven container damage characteristics, the regular container, and the port environment). Finally, a small-scale data set with 1,543 sample images was formed. Furthermore, the training set and verification set were divided according to 9:1.
The test data set comes from the monitoring video during the regular operation of the port. The container image is obtained by selecting, intercepting, and extracting key frames of the container screen contained in the video stream. Specifically, the difference method performs a difference operation on the two frames of the selected and intercepted frame images. The maximum local value of the average interframe difference intensity is selected as the container image according to the calculation result. Finally, a damage test data set with 53 sample images was formed. Figure 2 shows the sample images of different types of container damage in the two data sets.
(a)
(b)
3.3. Image Augmentation
Due to the limitations of the actual situation, this experiment cannot establish a large-scale damaged container data set. The data set is a prerequisite for obtaining better identification results. To the diversity of the data set, image augmentation is used to expand the data set to improve the generalization ability of the trained model. Image augmentation technology refers to randomly changing the image samples to expand the scale of the training data set, thereby reducing the dependence of the model on specific attributes [44]. At present, the primary data augmentation methods include horizontal/vertical flip, rotation, zoom, crop, cut, translation, contrast, and tone adjustment. The specific image augmentation strategy uses the TensorFlow interface to perform vertical flipping (equation (2)) and horizontal flipping (equation (1)). Figure 3 shows a group of damaged images of containers after data augmentation.
(a)
(b)
(c)
4. Damage Detection of Container
The experiment adopts GPU mode to implement on laptop (central processing unit (CPU): Intel(R) Core (TM)i7-9750H @2.60 GHz, RAM: 16 GB, and graphics processing unit (GPU): NVIDIA GeForce RTX 2060). The software configuration is as follows: CUDA V10.2, cuDNN 7.6.5, Python 3.8.5, TensorFlow 2.3.0, NumPy 1.19.1, Matplotlib 3.3.2, Sklearn 0.23.2, and Keras 2.2.0.
4.1. Image Optimization
This research refers to the method of weak supervision proposed by Ignatov et al. [45] to enhance the image. It optimizes and enhances the obtained container damage image to improve the quality of the data set. The WESPE algorithm enhances the damaged image of the container, and the effect is shown in Figure 4. WESPE (weakly supervised photo enhancer for digital cameras) enhances low-quality images in a weakly supervised way. Although there is still a particular gap between it and strongly supervised image enhancement, WESPE is better suited to mage collected outdoors, such as the data set of damaged containers [32]. So we use WESPE to enhance the image quality of the sample. The input and output data of the weakly supervised WESPE network model are low- and high-quality images, which use a transitive CNN-GAN structure to learn the mapping relationship. For the network structure and detailed description, please refer to [45].
(a)
(b)
4.2. Process for Experiment
The construction of the detection model based on transfer learning is divided into two parts: model pretraining and transfer. Transfer learning is widely used in machine learning, deep learning, and data mining to solve a limited number of labeled training samples. The experimental process of the multitype container detection model based on transfer learning is shown in Figure 5. In this experiment, MobileNetV2 and InceptionV3 trained on the ImageNet data set are selected as pretraining models.
MobileNet is an efficient and lightweight network model proposed by Google. MobileNet is based on a streamlined architecture. The core idea is to use the depthwise separable convolution. Under the same weight parameter amount, compared with the standard convolution operation, the calculation amount can be reduced several times, improving the network operation speed [37, 46]. The InceptionV3 network can expand the network without increasing the computational cost and extract more subtle features under the same computing power, improving the detection effect [34].
MobileNetV2 and InceptionV3 trained on ImageNet are used as pretraining models, and model parameters are transferred. After that, the network is fine-tuned through the training and verification data set of container damage. The entire model can be further adapted to the multitype of container damage detection tasks. Specific steps are as follows: Step 1. Prepare the data set: the established container damage data set, with the training and validation set as the input of the pretraining model. In addition, the quality of the data set is improved through image augmentation and image enhancement technology. Step 2. Transfer preprocessing model: select MobileNetV2 and Inception V3 pretrained models on the ImageNet data set for transfer. Step 3. Fine-tune the network structure: extract the features in the pretraining model, change the predicted category to the type of container damage, freeze some layers of the network, and adjust the parameters of the model through technologies such as Dropout. Step 4. Select the detection model: compare the pros and cons of the two models after fine-tuning and select the detection model with the best test results. Step 5. Test model effect: the test set is used as the input of the detection model. Finally, the type of container damage is output, and the predicted probability is given.
4.3. Multitype Damage Detection of Container Using CNN
In the detection of multiple types of container damage, the training set is further divided into 9:1. Part of the training set is used to train the model, and the other part is used to evaluate the model’s generalization ability initially. It is to be ensured that the parameters of the trained network model are adjusted in time. The verification set is used to verify the detection performance of the trained network model and ensure the model’s detection effect on unknown samples.
The experiment selects MobileNetV2 and InceptionV3 pretrained models trained on the ImageNet data set and prevents modification of the primary weight information in the pretraining model during the training process. Then freeze the top layer of the pre-training model, and set the learning rate to 0.0001, optimizer to Adam, Epoch to 30, and loss function to Softmax loss (equation (3)) [47]. The container damage data set is enhanced through WESPE to make the container damage characteristics prominent. Moreover, crop the picture to 224 ∗ 224 pixels as input. Then the defined basic model and feature extractor is connected to construct a container damage detection model.
Furthermore, global average pooling is performed on the acquired feature vectors. We set the Dropout to 0.3 to reduce the amount of work. Moreover, the prediction result is finally obtained through the classification layer. The expression is as follows:
where is the batch size, is the damaged category, is the prediction score of the i-th sample, and is the prediction score of the i-th sample corresponding to class j.
When the two network models complete the training set and validation set process, the MobileNetV2 and InceptionV3 pretraining models are fine-tuned. The top part of the model and the network weights are adjusted to make the model to better adapt to the classification of container damage. Specifically, the top part of the model and the added classification layer are continuously trained in the target domain data set of the damaged container sample. The pretraining is adjusted. Among them, in MobileNeV2, the first 100 layers are selected as the bottom part and frozen. The freezing starts from the 101st layer. The weight of the network is retrained with the global average pooling layer and classification layer added at the end of the network; the original is frozen in the InceptionV3 feature extraction layer (Mixed8), and the network layer is unfrozen after Mixed7. In addition, the learning rate of the fine-tuned model is set to 0.00005; the optimizer is RMSprop; and the Epoch is 50.
Table 2 shows the comparison between accuracy and loss of the two pretraining models before and after fine-tuning. It can be seen that the detection accuracy and loss of the model after fine-tuning have been optimized to a certain extent. Further generalization can be seen as MolileNetV2 better performance in the process model to the new sample.
It can be seen that by applying the transfer learning method of MolileNetV2 to container damage image processing, a detection model with better generalization performance can be constructed. Therefore, MobileNetV2 is selected as the pretraining model for multitype of container damage detection models. The building of the detection model is based on the MobileNetV2 framework and the damage characteristics of the container.
5. Results and Discussion
5.1. Test Results
The multitype container damage detection model adopts transfer learning and is based on the MobileNetV2 framework. The convolutional base of the detection model is kept in its original form, and the number of output neurons in the source code structure layer is modified. Its output result features are used as the classification of container damage.
Iteration has an essential impact on the accuracy of the experiment and training time, and it is necessary to confirm the appropriate number of iterations. Specifically, the number of iterations is small. Although the model training time is reduced, the training loss may not be minimized, resulting in lower recognition accuracy. In contrast, the number of iterations is more; the training loss is reduced to a minimum and stable, but the model training time is too long, causing unnecessary waste of resources. Therefore, this article has tried various iteration times, and the training results of the network model iteration are shown in Figure 6. It can be seen that when the number of iterative training reaches 4,000, loss is stable but has not yet tended to minimize, and accuracy is stable and tends to 1. Therefore, the accuracy, loss, and training time are fully considered, and 5,000 iterations of training times are selected for follow-up research, and the final damage detection model is obtained.
(a)
(b)
The prediction results of multitype of container damage detection models are shown in Figure 7. The detection results (damage type and probability) are given at the top of each image. According to the test results, differences in the detection effect between different types of damage can be found. On the one hand, due to the differences in the manifestations of different kinds of damage, the more pronounced the type of damage, the higher the identification accuracy of the damage detection model. On the other hand, because the number of damaged samples in the container data set is not uniform, this will also impact the identification effect.
5.2. On-Site Experiments
To verify the effectiveness and feasibility of the multitype container damage detection model in the virtual port environment, the trained detection model is deployed to the mobile phone using TensorFlow’s target detection API. The model deployed on the mobile terminal obtains images through the smartphone camera for real-time damage detection. In this study, the on-site experiment was conducted in Dalian Port, and the visibility was relatively high on the day of the trial [48]. As shown in Figure 8 and Table 3, the experiment results show that the multitype container damage detection model can give the corresponding damage types and prediction results. It achieves the expected results of the experiment. However, due to practical factors, the experiment failed to cover all types of container damage.
(a)
(b)
6. Conclusions and Prospects
Based on the transfer learning and MobileNetV2, this paper proposes a multitype damage detection model for containers. The data set containing nine typical types of container damage is established. Furthermore, to verify the validity and accuracy of the model, we tested multitype damage detection models in the virtual port environment. The results show that the model can identify multiple types of damage. The convenience and effectiveness of multitype damage detection models have excellent advantages in large-scale container inspection scenarios. In addition, security personnel can use mobile image acquisition devices (such as smartphones and drones.) to collect images. Then images are transmitted to the central processing unit, and multiple types of damage detection models are used for detection. It can provide a supplement for the existing detection methods in port.
In the future, we will be improved from three aspects. First of all, we did not consider the problem of damage detection for special type containers. Therefore, when further optimizing the data sample and increasing the multitype container damage data set, it is also considered to divide the different containers. Second, a real-time monitoring system will be developed based on port IP network cameras and integrated into the port management system. Third, it is necessary to quantify the degree of damage to the container according to the severity of the injury to the container and support the intelligent decision-making of container damage.
Data Availability
All the data in this paper are available from the author upon request.
Conflicts of Interest
The authors declare that they have no conflict of interest.
Authors’ Contributions
(a) Clarification for adding the authors. The two authors added have participated in conducting the research work for consideration duration and have significant contributions. They are from the University of South Australia in Adelaide, Australia, which is a university in another country. Due to the COVID-19 pandemic and the difficulty of international travel, many academic activities were canceled. Therefore, there is significant difficulty in effective and timely communication between the two sides. At the time of article submission, the author could not establish meaningful contact with the two authors in Australia. In revising the paper, all authors discussed and agreed that the two additional authors provided editorial support in the later revisions and made significant contributions. All authors already listed in the paper agree with adding the two authors: “Jing Gao” and “Yuhui Sun.” Therefore, the two authors need to be added to the author list. (b) Contribution of authors. Zixin Wang performed data collection, conceived and designed the study, performed analysis and interpretation, and drafted the article. Jing Gao conceived and designed the study and proposed the experimental methodology. Qingcheng Zeng provided critical guidance and support in the revision of the article. Yuhui Sun critically revised of the article.
Acknowledgments
This research was supported by the Department of Science and Technology of Liaoning Province (2020JH2/10100042) and the Key Disciplines in Dalian (2018Z0083).