Abstract
With the continuous development of multimedia technology and electronic information technology, the application of 3D images has become more and more extensive, and 3D reconstruction and interactive technology have also received more and more attention from researchers. Digital twin (DT for short) is a technology that makes full use of digital twin models and real-time monitoring data to complete real-time mapping of the real world in the digital world. Visual communication (CV for short) design is a design that is expressed and communicated to the audience through a visual medium. 3D reconstruction refers to the mathematical process and computer technology that uses 2D projection to recover 3D information of an object. This essay aims to study an image 3D reconstruction and interaction method based on DT and CV and analyze the practical feasibility and practical effect of this method. This essay proposes a model for image 3D reconstruction and interaction, which combines DT and CV effects and conducts a system simulation test for the model. The simulation is carried out on the three-dimensional reconstruction of the craniomaxillofacial area in medicine, and the reconstruction process is relatively smooth. And the final simulation test shows that the maximum CPU usage rate of the system during the rebuilding process is about 50%, which is relatively stable. The average CPU usage is about 30%, the overall system energy consumption is low, the lower limit of the overall SNR of the image is 57, and the upper limit is 62. The image quality of the reconstruction process is good, and the overall system reliability is high, which is feasible.
1. Introduction
After years of in-depth research on digital twins by many scholars, digital twins have made rapid progress in both theory and application. The term digital twin first appeared in a course given by a professor at the University of Michigan in the United States in 2003. He proposed the concept of “virtual digital representation of physical products,” which can be understood as digitally constructing a digital representation of a summary that has the same shape as the physical entity and can be monitored remotely. It was later defined as including physical and virtual products and a digital system that connects the two.
Digital twins can use data-driven models to simulate the physical world. Through the digital modeling of physical entities and the collection, storage, and integration of real-time data of the whole life cycle, whole process, and all elements, the functions of three-dimensional operation, monitoring, and control are realized.
A design that is conveyed and transmitted to an audience through a visual medium is known as visual communication design, which reflects the graphic design and rich connotation of the design of the times.
The three-dimensional reconstruction and interaction of digital twins and visual communication effects are studied in this essay. Applying the oblique image-based modeling method and the point cloud data-based 3D modeling method to the construction of the digital twin can obtain high-quality image 3D visual effects with more efficiency and low power consumption.
2. Related Work
With the rapid development of multimedia technology, the application of digital twin technology and visual communication technology has become more and more extensive, and the processing of images has become more and more refined. At present, many scholars have discussed the three-dimensional reconstruction and interaction of images. Blum A summarizes the medical applications of new postprocessing tools available. Global illumination rendering (GIR), unfolded rib reconstruction, subtraction CT angiography for bone investigation, dynamic research, temporal subtraction, and image fusion are among the novel uses. GIR creates realistic volume rendering by simulating the whole interaction of photons with scanned objects. Expanding the thoracic cavity allows for a more precise and quicker identification of bone fractures. These technologies may be utilized in conjunction to offer morphological, topographic, and functional information, as well as to expand the capabilities of CT [1]. The main purpose of Dimen is to provide a nondestructive GIS-based method for the analysis of murals and icons of cultural heritage. Dimen proposes a system that combines topographic surveying, digital photogrammetry, and image processing methods. Physical property of the mural surface can be quantified, as well as variations in the mural surface that cause deviations in flatness. Dimen applies bundle adjustment to the 2D reconstruction algorithm in Ramet Monastery instead of the 3D reconstruction algorithm to control the murals [2]. There are many reasons and motivations for 2D digital modeling of real-world objects, which includes virtual reconstruction of icons and murals of historical artifacts that no longer exist or only partially exist. Combined resection and immediate cranial reconstruction of invasive tumors of the skull rely primarily on freehand solutions. Dodier developed a cadaveric [3] study using a new software tool for single-stage reconstruction using prefabricated implants following resection of infiltrating lesions in the skull. Bone flaws are used to create polyetheretherketone implants. To test navigation accuracy, image-guided craniotomy was repaired using polyetheretherketone (PEEK) and compared to polymethyl methacrylate (PMMA) [4]. Chatterjee shows a noncontact 3D fingerprint identification system that uses only one edge forecast and biospeckle analysis to prevent spoofing. Frequency filtering using the Fast Fourier transform: a function that meets specific requirements may be written as a fractional derivative (sine and/or cos curve) or a fixed number of their equation [5]. A new technique based on neural recurrent neural networks was developed by Zhang D. Prior domain information regarding optical is little encoded by GPR and is used in the technique. This approach can monitor microrobots with complicated forms while also estimating optical microrobots’ posture and depth values. Based on position and depth estimate findings, Zhang delivers understandable visualizations to promote successful social contact during genetic manipulation [6].
3. Visual Communication Model Based on Digital Twin
3.1. Digital Twin
The real-time visualization for digital twin is to use the multidimensional data collected from the physical world to establish a real-time connection with the digital twin to realize the dynamic mapping between the physical space and the digital twin. These data are used to drive various models in the virtual scene to make them consistent with the dynamic behavior of physical entities in the physical world. It allows users to grasp all the trends of the physical world factory in real time in the digital twin system without having to visit the site in person, which facilitates management and maintenance, improves management efficiency, and saves manpower. Various equipment parameters in the physical world are collected by sensors, and these data are stored in the corresponding database in real time. Figure 1 shows the data acquisition method of the digital twin database [7, 8].
The basic architecture of digital twin is roughly divided into user layer, digital twin layer, virtual entity interaction layer, and physical entity layer. The main function of the user layer is to realize the operation and use of the user terminal, and the digital twin layer is used to model and simulate the simulation process. The virtual entity interaction is to monitor data and send commands to the physical entity, while the physical entity layer runs under the instruction control of the upper layer and is monitored by the upper layer.
The database is used as the link between the physical world data and virtual digital twin-oriented 3D rapid modeling and real-time visualization of applied research quasi-digital twins, by continuously reading the real-time data in the database and mapping these real-time data to the digital twin scene and by classifying various information in the physical real world and storing them in various tables. When the digital twin is displayed in real time, it connects to the database according to the corresponding mapping relationship to find the corresponding data table and reads the data in the table. According to the corresponding mapping relationship, the data is mapped with the virtual digital twin to achieve the real-time visualization of the dynamic behavior of the dynamic model.
Table 1 shows the literature growth of the digital twin of the domestic literature search website HowNet in recent years. It can be seen from Table 1 that the number of documents basically shows an exponential growth trend. As the research on digital twins has become more and more extensive in recent years, there are more and more levels of use, because models built based on digital twins can indeed greatly improve work efficiency [9].
3.2. Visual Communication
Logo design, advertising design, package design, internal and exterior environment design, corporate image design, and so on are all examples of visual communication design. At present, visual communication mainly includes five dimensions, as indicated in Figure 2, which are content, channel, receiver, feedback, and transmitter [10].
In terms of the breadth of their concepts, there is no distinction between visual communication design and graphic design in a restricted sense; both are concerned with utilizing vision to communicate with people. After induction, organization, and refining, the identifying symbol in Figure 3 is the communication’s content. These symbols have an artistic and design sensibility in order to generate mutual resonance [11, 12].
(a)
(b)
(c)
A design that is conveyed and transmitted to an audience through a visual medium is known as visual communication design.
Nowadays, with the rapid development of domestic social economy and science and technology, the interaction of pictures and texts, and the high-speed and efficient multimedia communication methods, the concept of visual communication design has been extended and expanded again. This has brought unlimited business opportunities to the field of modern visual communication design and also provided it with a broad space for development. Figure 4 shows a smart interactive billboard designed by a company to achieve the effect of visual communication [13, 14].
(a)
(b)
3.3. Image 3D Reconstruction and Interaction
There are two kinds of 3D modeling technology, one is the 3D modeling based on oblique photography, which is suitable for the construction of large area environment model. The other is based on 3D laser point cloud data-assisted reverse modeling, which is suitable for reverse modeling of single buildings and equipment. The difference between the two is indicated in Table 2. In the following, we will introduce them, respectively [15].
3.3.1. 3D Modeling Based on Oblique Photography
The modeling can collect not only the image information of the vertical angle but also the image information of the side of the object through the camera with the oblique angle. This solves the problem of missing image information on the side of objects caused by only collecting image information from a vertical angle [16]. The core steps of the 3D reconstruction method based on oblique photography can be summarized as four steps: image acquisition, multiview image adjustment, multiview image dense matching, and texture mapping. Its modeling process is indicated in Figure 5.
(1) Image acquisition. The image data collection work is mainly divided into survey area observation, survey area route planning, flight plan formulation and real-time data export, and synthesis.
Observation of the survey area: the main purpose of the survey area observation is to observe the size of the area to be measured and the maximum height of the buildings in the area to be measured. Based on this information, the aircraft’s altitude route and charging plan are formulated at the time of data collection. When determining the area to be measured, it can carry an aircraft and use the aircraft to do a simple flight to determine the height of the highest object in the measured area and the approximate area of the measured area [17].
Survey area route planning: the planning of the survey area route needs to fully consider image factors such as the terrain of the survey area and the generation accuracy of the model. Therefore, the most suitable course overlap, side overlap, and flight height are selected to ensure that high-quality image data can be collected. According to experience, the flight altitude is generally set to be about 30% higher than the tallest building in the area to be tested [18]. Side overlap and heading overlap are set between 65% and 85%. The shooting mode of the camera is generally set to shoot at equal intervals.
Development and implementation of flight plans: the flight time should be 3 hours before 12 : 00 noon and 3 hours after 2 : 00 p.m. on a sunny day as much as possible to ensure sufficient light and prevent the impact of excessive strong light on image data collection. Checking the flight status, before uploading the flight plan to the aircraft, focus on checking whether the flight altitude exceeds the highest building in the tested area, whether the return altitude can ensure flight safety, and whether the estimated flight time is within a reasonable range. After it is confirmed that it is correct, it can be uploaded to the aircraft for the collection of impact data [19].
Data export and synthesis: after the aircraft has landed smoothly and completed the flight, it is necessary to stop the operation of the camera, copy the image data in the 5 folders, and copy the POS data of the aircraft together. Due to the influence of high air flow and air pressure, the flight platform will be unstable, resulting in a large change in the attitude angle of the collected photos. And due to the influence of factors such as changes in air quality and lighting conditions, the oblique images obtained at different times will have differences in brightness, contrast, and other attributes. In order to eliminate the influence of the above unfavorable factors, it is necessary to perform color correction and light and color uniform processing on the obtained image photos and delete the images with serious defects. And after the processing is completed, the corresponding POS data needs to be deleted together to ensure that the image information corresponds to the POS data one by one. After checking the number of pictures and the number of POS data, assign the POS data information to one of the images captured by the five cameras and generally select the image captured by the camera perpendicular to the ground, which has been prepared for subsequent use [20].
(2) Multiview image adjustment. Using the beam method to adjust the multiview image, the basic principle of photogrammetry to build a model is the collinear formula (the image square point, the photographic center, and the object square point are on a straight line).
(3) Intensive matching of multiview images. The classical least squares matching algorithm is extended and applied to multiview images to form a multiview image matching algorithm with collinear conditions. This method is based on the geometric constraints of epipolar lines, one-dimensional space search is performed on epipolar images, and the object space and pixel coordinates are calculated by combining the gray scale observation formulas. Assuming that there are n cameras A1, A2, An that collect n pieces of image data of the same object on the ground, the corresponding image point coordinates are A1′(x1, y1), A2′(x2, y2)An′(xn,yn). Among them, A1′(x1, y1) is selected as the pixel coordinates of the reference image, and the remaining n-1 images are used as the search image [21]. Select a point m0 on the reference image, and obtain the initial image points m’2...m’n with the same name on the search image through the epipolar constrained multiview least squares matching algorithm. At the same time, the object coordinates (X′, Y′, Z′) of the ground point m’ corresponding to the initial point with the same name are obtained. The multiview image matching algorithm with collinear condition determines the exact same-name point image point coordinates m2, m3...mn and the object coordinate (X, Y, Z) corresponding to m0 on the search image.
The multiview image least squares dense matching algorithm is also constrained by the collinear formula. Therefore, the image point coordinates m1, m2...mn corresponding to the object-side ground point P and the photography center Ai satisfy the collinear formula, so the object-side coordinates of m can be calculated by substituting them into it.
(4) Texture mapping. Model texture information is one of the key indicators that affect the later display effect. The traditional method is to use the photos taken by the camera to do uniform lighting and face-up processing and then use the texture method to paste the processed texture photos on the surface of the 3D model. But oblique photography itself has texture information. Different from traditional methods, when using oblique image modeling to give real texture information, it is only necessary to find the mapping function between the two-dimensional texture image and the surface of the three-dimensional model, that is, to establish the corresponding texture mapping relationship. Its essence is to assign the color value or gray value of the point on the two-dimensional texture space to the corresponding point on the surface of the three-dimensional object according to the corresponding functional relationship. Finally, a 3D model with texture information [22] that meets real color vision is obtained, and the mapping principle is indicated in Figure 6.
3.3.2. 3D Laser Point Cloud Data-Assisted Modeling
The modeling can accurately record the three-dimensional data of the object surface according to the distance from the scanner to the object surface measured by the scanner and the angle information. Finally, the high-precision point cloud data can be obtained by computing these data, and finally the processed point cloud data can be used to reconstruct the inverse 3D model.
Compared with traditional data acquisition, 3D laser scanning can significantly reduce the work intensity of field personnel, reduce the risk factor of work, and improve work efficiency due to its unique acquisition method. And the collected data is more refined and the information is more comprehensive and rich, which provides an accurate and reliable data basis for the reconstruction of the 3D model of the target object. The use of point cloud-assisted 3D reconstruction methods can be summarized into four steps: 3D scanner point cloud data acquisition, multisite point cloud data registration, point cloud data denoising and lightweighting, and point cloud data-assisted modeling [23–25].
(1) 3D scanner point cloud data collection. The collection of 3D scanner point cloud data is mainly divided into simulated model area inspection and exploration, 3D scanner configuration, and point cloud data collection. Exploring the simulated modeling area: before using the 3D scanner for data collection, it is necessary to conduct on-the-spot 3D rapid modeling and real-time visualization application research and exploration for digital twins in multiple acquisition areas, so as to determine the erection site of the 3D scanner [26, 27]. Due to the huge amount of data scanned by the 3D scanner each time, it is necessary to set a reasonable scanning scheme to reduce the number of stations and the amount of data collected as much as possible while ensuring the integrity of the simulated model area.
(2) Registration of multisite cloud data. When using a 3D scanner to collect data, in order to collect data comprehensively and completely for the target area, it is often necessary to collect data from different angles and multiple sites. At present, the most popular method for stitching point cloud data obtained from different sites of a 3D laser scanner is target-based registration [28, 29]. It can be subdivided into registration based on target and registration based on the characteristics of the point cloud itself, but the principle is the same, all of which are to find the feature points in the same area in the point cloud data. In the registration process, firstly, each target in the two adjacent sites is fitted and modeled, and the center point coordinates of each target are calculated. Then, the targets are numbered, and the same target numbers in different stations are the same to form a target pair. Then the target pair is used to solve the seven transformation parameters required for the registration of two adjacent sites, and finally the point cloud data of the two adjacent sites is rotated, translated, and zoomed to complete the point cloud registration using the obtained transformation parameters.
(3) Noise reduction and lightweight of point cloud data. During the modeling process, the unavoidable moving objects such as pedestrians and vehicles will bring a lot of noise when collecting data, and the existence of a lot of noise will bring huge pressure to the later data processing [30]. Therefore, after the point cloud splicing is completed, the point cloud data needs to be denoised. The noise reduction processing of point cloud data mostly adopts bilateral filtering algorithm and robust noise reduction algorithm. After noise reduction, those outlier flying spots and small-scale spots will be greatly reduced. Finally, the point cloud of the part to be reconstructed is manually selected, which can greatly reduce the amount of data of the point cloud, which is convenient for importing other software for auxiliary and rapid modeling in the later stage.
(4) Point cloud data-assisted modeling. Finally, the processed point cloud data is exported in.rcp format, and then the point cloud data in.rcp format is imported into Revit software. In Revit, the level function is used to cut the point cloud data into layers to determine the specific height and shape of each object in the model, which is convenient for the establishment of the model. And some common building structures can be used to quickly turn over the model according to the point cloud data and finally get a refined model to meet the later interaction needs. After the model is built, it needs to be imported into 3Dmax for material and texture map processing to visually meet the requirements of interaction.
3.4. Interaction of 3D Images
The interaction design of three-dimensional images is very important, which is directly related to the user’s experience and reflects the user’s experience effect. The principles of interaction design are simplicity, ease of use, safety, and fun. Interaction design generally includes data layer, visual layer, and interaction layer. The interaction steps are indicated in Figure 7.
4. 3D Reconstruction Model Test Based on DT and VC
4.1. 3D Reconstruction Model Algorithm
For light in vision, the absorption model is
Among them: is the intensity of the light entering the eye, S is the projection length, and is the attenuation coefficient.
The light reflection model is
is the light source item.
Light absorption and reflection model:
Shorthand it aswhere represents the intensity of the light observation point, and the sum of the light intensity contributions at the observation point.
In 3D space, a voxel is constructed. When the voxel edge is parallel to the X axis, the two ends of the edge are set to and , and the intersection point is , and
Similarly, when it is parallel to the Y axis, we can get
When it is parallel to the Z axis, we get
Connect these intersections into triangular patches to get the isosurface within that voxel.
The isosurface is often the interface between two substances with different densities, so its gradient vector is not zero, that is:
The central difference formula for the intersection of voxels is
Among them, , , and are the lengths of each side of the voxel.
For a dataset in a 3D field it can be expressed as
When formula (13) is satisfied, they are classified into the same class.
Let C represent the color value of the voxel, that is:where p is the percentage of a pigment.
After knowing the three-dimensional coordinates of the space point, the distance between the two points can be calculated:
The angle between two points can be calculated using the law of cosines:
For a point (, , ) in space, its normal vector is represented by (, , ); then the point normal formula is
This gives the cosine between the space vector and the normal vector:
And they satisfy + + = 1; therefore, in order to determine the normal vector, just give the size of any two direction angles and the quadrant where the third direction angle is located.
4.2. Simulate the Experimental Process of Craniomaxillofacial 3D Reconstruction
The object selected for the simulation experiment in this essay is medical-related craniomaxillofacial three-dimensional reconstruction, the database is MySQL (MySQL is a RDBMS, relational databases hold information in different charts rather than retaining all data for one big warehouse), and the system test environment is indicated in Table 3.
Among them, during the rebuilding of a picture, the control of vision is more important, and its control is mainly carried out through image interaction. The three-dimensional vision control process is indicated in Figure 8.
The 3D reconstruction process of the craniomaxillofacial area is indicated in Figure 9, in which Figure 9(a) is the craniomaxillofacial to be reconstructed, Figure 9(b) is during the 3D reconstruction, and Figure 9(c) is the craniomaxillofacial after the 3D reconstruction. Judging from the reconstruction effect, the craniomaxillofacial region was complete and the reconstruction process was smooth.
(a)
(b)
(c)
In addition, this essay conducts data statistics on the CPU of the digital twin system in the reconstruction process and obtains the statistical graph indicated in Figure 10. Figure 10 demonstrates that the CPU usage of the system is relatively high at the beginning and at the end of the rendering process. However, the highest CPU usage in the middle reconstruction process is about 50%, which is relatively stable. The average CPU is only about 30%, and the overall energy consumption of the system is low.
In this essay, in the case of different three-dimensional image sizes and different data amounts, the relevant statistics of the operation time of the system are carried out, and the findings indicate in Table 4. Table 4 illustrated that the system is processing more than 100 M data per minute on average, and the calculation speed is acceptable.
Finally, in order to understand the real effect of the image, this essay summarizes the image SNR of the system reconstruction process, as indicated in Figure 11. As can be seen from Figure 11, the overall SNR of the image is as low as 57 and as high as 62, indicating that the quality of the images in the reconstruction process is good.
5. Discussion
3D reconstruction technology has always been a difficult problem to overcome in CV. In the field of CV, many predecessors have developed many 3D reconstruction algorithms. This essay selects some excellent algorithms to create a digital doppelganger 3D reconstruction method based on digital twin and visual communication and conducts experiments on the whole system, and the expected results are obtained. However, due to its own capabilities, there are still shortcomings in this overall design, which can be roughly summarized as follows:(1)Although some optimizations have been made for camera calibration, the process is too cumbersome, and the subsequent calibration process needs to be simplified to further improve the calibration accuracy.(2)The algorithm provided in this essay is constantly updated and improved, and the matching effect is constantly getting better, but the slow matching speed of this algorithm is a big disadvantage, and it cannot perform real-time matching. If some optimization is performed on the preprocessing of the image, the efficiency of the algorithm matching will be greatly increased.(3)Although the matching rate is greatly increased when removing the inaccurate matching spots, the algorithm is utilized, and the overall matching points are reduced. This necessitates more study into the statistical method in order to reduce the mismatch rate and increase the system’s accuracy.
6. Conclusion
This article initially explains the digital twin technology, including its idea and database, in the theoretical research section, and then introduces the visual communication, including its concept and its related content. Finally, the 3D reconstruction and interaction of images are introduced, and the two modeling methods are introduced in detail, that is, 3D modeling based on oblique photography and 3D laser point cloud data-assisted modeling, and the specific process of modeling is explained.
In this essay, the craniofacial three-dimensional reconstruction is carried out in the experimental test, the database is MySQL, and the three-dimensional visual control process is introduced. The experimental findings suggest that the system associated with digital twins is effective and visual communication effect for 3D reconstruction and interaction of images has low overall energy consumption, high SNR of the obtained images, and good image quality, and the system has certain feasibility.
Data Availability
Data sharing is not applicable to this article as no new data were created or analyzed in this study.
Conflicts of Interest
The author states that this study has no conflicts of interest.