Abstract
In the past, the traditional small-scale 3D reconstruction technology developed maturely and mainly relied on manual handheld devices for data acquisition. Due to multiple factors, inaccuracy, and other problems, data will be omitted, and sometimes staff will face risks directly. Nowadays, with the development of computer science and technology, three-dimensional reconstruction technology has been innovated continuously, and it has also made a breakthrough in the application of large-scale reconstruction of dangerous areas. In this paper, the expensive sensors in the past are abandoned, and the cheaper and more efficient lidar is used instead. The laser point cloud and image are combined to describe the mining heritage scene in depth, and the academic achievements are transformed into productivity. The results show that (1) comparing the performance of the algorithm, the overall error of the proposed method is 5.86, and the average time consumption is about 10.23 ms. This filtering algorithm can restore the landscape of mining heritage to a great extent and optimize the problems of noise and outliers. (2) Our designed model has a very low loss value, and the point cloud accuracy is as high as 92.9%. Compared with other model methods, the model has excellent performance. (3) After the completion of the model, the overall satisfaction effect is above 70%, which can well restore the mining heritage style. Finally, the experimental effect of 3D reconstruction is good, which is more conducive to the research of reconstruction protection. There are still some work details and performance problems that need to be optimized and solved.
1. Introduction
Due to the continuous development of the trend of the times and the transfer of industrial centers, many small- and medium-sized coal mine industries in China have gradually stopped production and closed down, leaving behind many abandoned mining waste land. Most of these waste lands do not meet the requirements of “green construction” and are left untreated. On the one hand, their existence causes the waste of natural resources and also causes great pollution to the natural environment and ecological circle. On the other hand, some mines or tunnels left in the past that were not closed in time are easy to bring life safety threats to the surrounding residents. The existence of the above problems is an important reason to promote the reconstruction and protection of mining heritage. How to “turn waste into wealth” makes the original idle and abandoned land resources glow with new vitality and brings new useable value to people, which is worth pondering and considering. Previous studies rarely fully consider the use of the geographical features of mineral tourism development methods. Therefore, in order to better maintain the fragile ecological environment caused by coal mining and restore the ecological landscape of mining wasteland, in this paper, we decided to adapt to local conditions, based on the deep learning theory, use unmanned aerial vehicle (UAV) equipped with lidar to remotely measure the location of abandoned mines in urgent need of maintenance and reconstruction, fuse the features of point clouds and images, extract the features of ground mines, and reconstruct three-dimensional models to facilitate the design of protection projects.
In view of the previous research on three-dimensional reconstruction or mining heritage protection related reference materials, we have carried out a detailed reading and reference, hoping to get valuable methods for this research. The following related literature will provide a solid theoretical basis and data support for this paper. Introducing depth learning model to improve the segmentation accuracy and robustness of lidar remote sensing image segmentation method [1], the vertical structure of cloud in Loess Plateau was studied by lidar remote sensing [2]. A 3D reconstruction simulation method based on infrared image sequence and depth learning network is proposed [3]. Three-dimensional characteristics of vegetation were obtained by UAV lidar remote sensing, and the vertical structure of subtropical evergreen broad-leaved forest community was analyzed [4]. The application of airborne lidar remote sensing technology in surveying and mapping field is discussed [5]. From the perspective of local theory, we pay attention to the regional and local relationship of mining heritage and take Wuchuan Mercury Mine in Guizhou as an example to formulate the protection and tourism development strategy [6]. Aiming at the problem of unsatisfactory reconstruction when compressing images, a lidar remote sensing image method based on orthogonal basis compressed sensing is proposed [7]. Combining UAV sequence image reconstruction method and computer vision technology, 3D reconstruction of complex terrain area is carried out [8]. Polarization lidar observes and remotely senses the three-dimensional structure information of ground objects [9]. In order to effectively map the ground buildings and terrain, remote sensing images collected by UAV lidar are used to reconstruct the ground [10]. Design and implementation of remote sensing monitoring system for resources and environment based on lidar technology [11]: a new classification method of remote sensing monitoring data is proposed by using airborne lidar and hyperspectral technology [12]. This paper discusses the 3D point cloud repair technology based on deep learning from five aspects [13]. According to the situation of WTO, the actual situation of mining heritage is analyzed to promote the two-way win-win of mining heritage protection and tourism development [14]. From “island protection” to “network development,” the mining heritage route is studied from the perspective of regional cooperation [15].
2. Overview of Theoretical Basis
2.1. Mining Heritage
Mining heritage [16]: this is the product of the gradual depletion of mineral resources after uncontrolled exploitation by human beings. To put it another way, mining heritage belongs to a part of industrial heritage, which mainly refers to the abandoned land of mining area left by mining. Mining heritage is ecologically fragile, which has been destroyed and occupied by people for many years, and it is difficult to rectify, resulting in drastic changes in the original natural landform. Human mining activities have gradually formed strange and complex landscapes such as open pit, abandoned houses, mining pits, mined-out subsidence areas, landslides, and mines. China is a populous country, and land is often in short supply. It is extremely important to reclaim every piece of land reasonably and promote the reuse and development of abandoned land in mining areas. As for how to redevelop and utilize this type of land for reasonable and long-term development, domestic research started late, and the protection thinking is single, which requires more experts and scholars to fully mobilize resources to discuss. Mining areas with good ecological restoration can be restored to agricultural and forestry reclamation land. Mining areas with high economic and cultural value and unique spirit can develop mine parks or tourist land to help the original industrialized towns transform. Areas suitable for construction land can be included in urban and rural planning areas, and the advantages of transportation and space can be used to contribute to public and commercial facilities. The area where the mining heritage is located determines its value. The first type is inclusive, the mining heritage area is within the town, the resources are concentrated, and the town depends on the mining area for development. The second kind of mining heritage is outside the town, with some intersecting areas, scattered resources around, and the field situation is more complex and changeable with the erosion and compression of nature as shown in Figure 1.
(a)
(b)
2.2. Lidar Technology
LiDAR [17] is called “Light Detection and Ranging” in English and can also be called “Laser Radar.” This is a kind of mechanical facility mainly used to detect distance, azimuth, and height. As a kind of active remote sensing equipment, this technology mainly adopts photoelectric detection means. Its structure is simple, including a receiving system and a laser, and it is also equipped with global positioning system and inertial navigation system. Its working principle is as follows: the laser emits the detection signal, and then, the received target echo is properly restored; that is, various parameters related to the target can be obtained. Compared with ordinary microwave radar, lidar is small in size, light in weight, high in resolution, and good in concealment and can resist interference well and really work at “zero altitude.” Therefore, due to the wide application range of lidar, radars suitable for various situations have been manufactured such as TLS [18], BLS, MLS [19], ALS [20], and SLS [21]. The airborne LiDAR used in this paper is equipped with ALS laser scanning system. This new type of active aviation radar, mounted on the UAV, synchronously and quickly obtains the three-dimensional coordinates and image data of the designated target on the ground, realizing the dynamic change of the target in real time and reproduce the real characteristics. The airborne radar is composed of the central control unit and the carrier of the aircraft, which records the real data on the ground as shown in Figure 2.
2.3. Laser Point Cloud Filtering
The data acquired by airborne lidar remote sensing technology is to collect point clouds generated by 3D laser scanners in a large area. Then, the collected point cloud data are classified to separate the target points from the nontarget points, which becomes “filtering.” Point cloud filtering algorithm is related to the accuracy of the results, and the robustness of the algorithm is also a difficult point to break through. Through the practical application of this algorithm, the DEM corresponding to the target can be generated, and the specific features of the ground target can be extracted. Up to now, the commonly used filtering methods can be divided into six categories: slope, mathematical morphology, surface, segmentation, deep learning, and mixing. After comprehensive consideration, this paper chooses laser point cloud based on deep learning for experimental research.(1)The generation of point cloud As shown in formula (1), after we use mathematical symbols to represent it, because each variable of laser point cloud is a six-dimensional vector, we can describe the point coordinates of each three-dimensional laser space. Namely,(2)Point Cloud Filtering [22] Because of the problem of lidar’s own structure, the collected point cloud data is doped with many noise points and points far away from the gathering of points. After denoising, in another way, it is point cloud filtering to improve the data and do a good job in the preliminary work for the subsequent registration. First of all, we should consider the location information of points to guide point cloud filtering. A point cloud neighborhood is determined based on the original three-dimensional laser point cloud of formula (3), then a k-d tree structure is constructed, and the neighborhood representation of each point is shown in formula (4). The mean number value of formula (5) is calculated from a plurality of point cloud neighbor sets, where refers to the cardinality of . Solving Output Points by Linear Transformation: Calculating function: The solution is as follows: Finally, the filtered output point is calculated:(3)Point Cloud Registration [23] Point cloud registration process: based on the point cloud, the rotation and translation transformation relationship between the point and the laser point cloud is solved. In this way, we can get laser point clouds in different positions and postures, which can express the environmental characteristics of the target. The relationship between the source point cloud and the target point cloud is shown in formula (10), where R represents the rotation matrix, and T represents the translation vector. Formula (11) represents the precise registration process of point cloud by ICP algorithm and optimizes the objective function, where m means that there are m corresponding points.(4)K-means clustering algorithm [24]. Representation of Euclidean Distance Calculate the sum of squares of errors:(5)Central iterative algorithm.
Set Point Gathering
Calculate the initial center point coordinates:
Calculate the sum of distances:
The updated dots gather
Meet the conditions:
2.4. Principle of Camera Processing
Taking the four-rotor UAV as the mobile platform, it fills the shortcomings of traditional manual handheld and can collect data for a large number of scenes. Quickly deploy machinery and equipment, with strong maneuverability and flexibility. From the perspective of UAV, multiple visual devices are mixed for aerial photography to collect high-resolution data.
In the motion of rigid body in three-dimensional space, the rotation matrix is defined:
Rotation vector:
Transformation matrix [25]:
3. 3D Reconstruction System of Mining Heritage Based on Deep Learning
3.1. Target Detection and Feature Extraction
“DL” means deep learning. As an outstanding research direction in the field of artificial intelligence, deep learning has solved many complex pattern recognition problems. At the same time, it has a large number of mature and applicable deep network architectures and related technical means. Before the formal 3D reconstruction of mining heritage by UAV in the air, we must accurately detect the target. Therefore, we introduce the deep network framework for design. As a three-dimensional point cloud, there is no regular expression form, and the particularity of point cloud data determines that we cannot directly use the typical convolution neural network model. According to the idea of projecting laser point cloud on different views, and then reducing dimensions to get images, finally, this paper chooses the fusion point cloud data and the image data processed by attention mechanism to segment the fusion point cloud and then extracts the point cloud features based on PointNet + + improved by FPN method. Finally, we can not only preserve the 3D information in the original data, but also identify the accurate 3D bounding box of the target as shown in Figure 3.
3.2. Multimethod Fusion Processing
It is found that, in order to achieve the goal, a variety of methods need to be integrated. Combine lidar with camera. The images taken by the camera can clearly see the detailed texture information of mining heritage, which is just beyond the reach of lidar technology. Complementing the advantages of the three-dimensional spatial location information of the heritage can completely realize the reconstruction of the three-dimensional scene. The above method is an idea. If you want to use both, you need to assign the RB value on the image to the point cloud and fuse the point cloud with the image. Calculate the position of the color pixel according to the parameters, and then carry out coordinate transformation between the pixel and the camera:
The depth camera provides z-axis coordinates, and the arrangement formula (22) obtains
The collection of each pixel point obtained by the transformation of coordinate system constitutes a point cloud. However, it should be noted that before the image is fused, the image data needs to be preprocessed. Because the quality of the image will be directly related to the accuracy of the final point cloud results. Motion camera uses ultra-wide-angle shooting, and the image is prone to distortion, which needs to be corrected. Introducing the distortion correction principle of fisheye camera, let P be the 3D point in the reference coordinate system:
Solving polar coordinates under fisheye hemisphere model:
Distortion mathematical model:
After distortion correction,
3.3. Three-Dimensional Reconstruction Model
After the above-mentioned target detection method based on deep learning, the target range, area, and geomorphological features are established. We set up a good mining heritage that can achieve the overall style and characteristics of the model for the framework display. The processing software is implemented based on distributed structure, which is divided into two parts, that is, UAV and ground, and the nodes are connected through 5G WIFI local area network. At the acquisition end of UAV, after the target is identified and detected, the color image and pixel depth are obtained by RGB-D camera, and the point cloud data is obtained by airborne lidar technology. After filtering algorithm and coordinate transformation, according to the embedded ARM, the fusion data of point cloud and image is collected. Then, the data is transmitted to the PC at the ground control end for final 3D point cloud reconstruction, which can effectively reduce the computing work of UAV. In PC, it can monitor the running state of the whole 3D reconstruction system and record the whole experimental data. In addition, the UAV still needs to manually control its flight through the remote controller, so the UAV will be equipped with a receiver that can receive remote control signals and a flight control device that exists as the underlying processor as shown in Figure 4.
4. Simulation Experiment Analysis
4.1. Comparison of Different Filtering Algorithms
In this section, the filtering accuracy of different filtering algorithms is tested to show the real effect of this method. SBF, SegBF, and CSF are selected to carry out experiments on 10 groups of experimental samples. After model processing, we get different error results of different algorithms. Among them, errors are divided into three categories: Class I errors, Class II errors, and total errors. Overall, Class I errors of the four filtering algorithms have achieved the best results, and the error degree is less than 10. Class II errors are the largest, and the error of SegBF algorithm can be as high as 32.38, which is in sharp contrast with the result of algorithm 13.6 in this paper. The total error value is low, which can reflect the more realistic overall situation of the target. The method in this paper has the best filtering effect, preserving the original landform as much as possible, and the error value is about 5.86 as shown in Figure 5.
After comparing the error effect of the algorithm, this section compares the time consumed by the algorithm. Time-consuming comparison is an index that can best reflect the efficiency and performance of the algorithm. 152 3D point cloud images were used in the experiment, and GTX1650 4 GB GPU and Intel i5-9300H 4-core 8-thread CPU were selected to operate on Ubuntu 18.04 system. According to the curve trend in the figure, it can be found that the average time consumed by this algorithm is about 10.23 ms, and the curve trend is flat, far lower than the other three algorithms. Compared with SBF algorithm, the efficiency is 13.2 times higher than SegBF algorithm, 6.4 times higher than SegBF algorithm, and 8.3 times higher than CSF algorithm. To sum up, the method in this paper has a good effect on dealing with noise and outliers contained in 3D point cloud as shown in Figure 6.
4.2. Model Simulation Test
Test the three-dimensional reconstruction model, analyze the training results of this detection network, and verify the feasibility and correctness of this model. In order to carry out better comparative test, we introduce the three Baseline, PointNet, and PointNet + + network models to participate in the experiment. The loss curve converges and descends quickly by analyzing the loss function during training. The proposed method converges rapidly to 0.36 and then reaches 0.178 in 100 iterations. The convergence speed of other methods is slow, and the final convergence value is about 0.2, which is slightly worse than that of this method as shown in Figure 7.
Test the accuracy of target point cloud segmentation. After the fusion of image features, this method has stronger ability to express data and learn more information. The accuracy rate is as high as 84.2% in 20 iterations and 92.9% in 100 iterations. The curve trend is much higher than the other three methods, and the setting is reasonable as shown in Figure 8.
4.3. Model Checking Effect
Taking an abandoned mine in a certain place as an experimental site, the mining heritage protection work of three-dimensional reconstruction is carried out to test the practical application effect of the model. We can survey nine different types of terrain and different mining scenarios, including 40 mining area survey sites. In this model for each different type of mining scene, the laser point cloud can be divided into ground points and nonground points, and the test results are good. Finally, the completed three-dimensional reconstruction model and the actual site data are tested for satisfactory coincidence, and the overall effect is good, both above 70%, and the highest can reach 95% as shown in Figures 9 and 10.
5. Summary and Prospect
Some idle mining relics are rich in resources and have high value in scientific research and sightseeing. Reasonable reconstruction can better protect geological and mineral resources, bring new benefits to urban development, avoid environmental pollution, and, at the same time, relieve the hidden danger of abandoned minerals. In this paper, the target detection technology is used to detect the location of the mine to be studied, and the unmanned aerial vehicle equipped with depth camera is equipped with lidar to observe the mine situation around the clock, improve the traditional point cloud filtering algorithm, and improve the processing speed of the algorithm. Based on the deep learning framework, the 3D reconstruction system is created by integrating point cloud and image, which provides an effective reference for the evaluation and protection of mining heritage. The research results of this paper show that the target detection method under deep learning network has good detection accuracy. Point cloud with image features can not only collect distance and intensity information, but also describe the color and texture of minerals in a rich way. In addition, this algorithm is compared with other traditional methods in terms of comprehensive performance. The optimized algorithm has satisfactory filtering effect for different complex terrain areas and can effectively preserve terrain details while reducing errors. Although this paper has carried out theoretical and experimental research around the research topic, it has achieved certain success. However, due to the limitations of human resources, experimental conditions, the complexity of reconstruction work, and other objective factors, this study is not comprehensive and perfect, and there are still some deficiencies that need to be reasonably improved in a series of follow-up works. For example, the filtering algorithm proposed in this paper has obvious advantages for mining scenes, but its effect is different for different scenes, and its universality is poor. The parameters used in the experiment are too complex, which increases unnecessary calculation and lacks certain robustness, so it is necessary to optimize the parameters. In the process of data fusion in experiments, it is necessary to meet a certain number of connections between images and point clouds, and it lacks training of massive experimental samples. The three-dimensional model of mining heritage created in this paper should also quantitatively evaluate mining heritage based on AHP and construct systematic indicators for value evaluation, so that the protection work is more in line with the local actual situation.
Data Availability
The experimental data used to support the findings of this study are available from the corresponding author upon request.
Conflicts of Interest
The authors declared that they have no conflicts of interest regarding this work.
Acknowledgments
This study was funded by 2020 Shanxi Province Philosophy and Social Science Planning Project, Study on the Protection and Development of Shanxi Mining Heritage from the Perspective of Ecological Civilization Construction (2020YJ210).