Abstract
The traditional art education model often cannot achieve good teaching results due to the development of digital technology. Art is integrated with technology based on three-dimensional (3D) panoramic vision sensing technology, so as to study the application of the combination of visual sensing technology and digital image art in the field of art education. With the application of projection virtual game “Whac-A-Mole” as an example, the integration of panoramic color volume structured light generation technology with single emission point and omnidirectional visual imaging technology with single view is proposed; the mathematical model of the active vision system is established, and the camera model, projector model, object surface illumination model, and their relationship are studied. On this basis, the mathematical relationship between the color of the projected light source and the color of the corresponding imaging point is proposed; a light source color correction algorithm based on the two-color reflection model is proposed. The object surface color is corrected by three-channel reflectance after the light source color correction algorithm is used for correction. The results suggest that the 3D panoramic vision sensing technology makes the current active stereo vision develop from the visual perception of surface structure to the visual perception of volume structure and reduces the error rate of recognition from 44% to 11%. Moreover, the algorithm does not need to know the coding method of the projected light source and the surface material of the object in advance. The scene teaching combined with technology overcomes the limitations of the teaching site and brings students into a completely realistic teaching situation. The presentation of digital image art using visual sensing technology can inspire children’s imagination, combine education and games, and perform edutainment. Thereby, the application research of digital image art based on 3D panoramic visual sensing technology has an irreplaceable development prospect in the field of art education.
1. Introduction
Today, digitalization is reflected in many fields in life. Human beings have once again entered the era of picture reading under the impact of massive information [1]. Great changes have taken place in the way of human communication with the continuous improvement of socialization. Human beings urgently need an art form that can cater to this fast-paced life. Digital image art is born in response to the needs of the times and plays an increasingly crucial role in life. The research on digital image art enables people to better grasp the characteristics of digital image art and make use of its unique form of expression to make digital image play a better role in the field of art [2].
Visual sensing technology needs digital image art to provide an external display window, and digital image art needs the intervention of visual sensing technology to better complete the interactivity and virtuality, so that more people and more fields can enjoy the achievements brought by technology. The research on sensing technology abroad is relatively mature. Yamaguchi et al. (2019) [3] showed that the director of the college specializing in VR in New York invented an interactive system using the Internet to realize touch, which is named “remote pulse taking” and “remote palpation.” Domestic research on visual sensing technology and digital image art mostly stays at the technical level. Xie (2017) [4] showed that active stereo vision mostly belongs to the visual perception mode of surface structure, which cannot directly perceive, express, and reconstruct 3D stereo panorama with one image. The research on digital image art is mostly limited to the art field itself. Some achievements about the application of visual sensing technology in other fields have been made, but the application research in the field of art education is far from enough. Roa’a et al. (2021) [5] proposed a method to enhance the panoramic image environment by calculating the high dynamic range. Panorama is constructed by combining several photos taken by traditional cameras at different exposure times. Unlike the high dynamic range in real panoramic environment, the dynamic range of traditional cameras is usually much lower. In a real panoramic environment, images taken with a traditional camera will have areas that are too bright or too dark. Intel real R200 sensor is favored by massive researchers, but its integrated commercial stereo matching algorithm has small detection range, short measurement distance, and low depth map resolution, which seriously restricts its use scene and service life. To solve these problems, Zhong et al. (2020) [6] proposed an infrared stereo matching algorithm combining semi global method and sliding window idea based on existing research. It reveals that the research on only one direction is very one-sided.
The method of establishing the mathematical model of the active vision system is adopted, and the algorithm is applied to a case for verification. The innovation is to integrate the panoramic color volume structured light generation technology of single emission point and the omnidirectional visual imaging technology of single view, so as to realize a 3D active stereo vision perception centered on the observer, which makes the current active stereo vision develop from the visual perception of surface structure to the visual perception of volume structure. It is hoped that this exploration can be applied to more fields and make more people feel the charm of the integration of technology and art.
2. Research Method
2.1. Theoretical Basis
2.1.1. 3D Vision Sensing Technology
Machine vision [7] is mainly to use the computer to simulate human visual function, extract information from the image of objective things, and process, understand, and finally use it for actual detection, measurement, and control. 3D vision sensing technology refers to seeing stereoscopic images without wearing stereoscopic glasses [8]. The acquisition of 3D image information is based on image sensor acquisition and quantitative image information. The image information includes visible light images obtained directly, and other forms of image information obtained through the detection of radiation, infrared, and ultrasonic. Different sensor technologies have different resolutions, accuracy, and noise. People can extract useful signals from image sensors for simultaneous interpreting and application. 3D vision sensor has a wide range of applications, such as multimedia mobile phone, network camera, digital camera, automobile safety system, biomedical pixel analysis, human-computer interface, and virtual reality [9]. These different applications are based on 3D vision image sensor technology. Figure 1 shows the principle.
2.1.2. 3D Panoramic Vision Sensing Technology
This is a virtual reality technology based on static images. It is to put a group of photos taken by the camera at 360 degrees into a panoramic image [10] and display the image on the internet with special playback software. In this way, users can control the direction of looking around with the mouse and can view objects or scenes from left to right and near to far. The virtual environment is established by using real photos, and the creation of virtual reality is completed according to the mode of , as shown in Figure 2. Regarding traditional 3D technology and 3D network technology represented by VRML (virtual reality modeling language) [11], computer-generated images are employed to build a 3D model, which is simpler and more practical.
2.1.3. Digital Image Art
Digital image art refers to the image art made by the creation platform based on digital information processing technology, which is constructed by various digital technology information processing devices with the computer as the core. It is a comprehensive art involving computer software, fine arts, engineering art design, film and television, advertising, and other industries through digital processing technology [12].
Digital image art using visual sensing technology is a modern art integrating technology and art, which is a perfect combination of the two (Figure 3). Digital image art plays a crucial role in today’s art world and brings people an all-round artistic experience in an omnidirectional way. Today, the use of various advanced visual sensing technologies makes the presentation of digital image art very rich. It is a kind of art involving film, television, image, photography, internet, and other media fields. It also includes a virtual image from the technical level [13], commonly known as a CG image [14].
The intervention of new visual sensing technology has changed the previous keyboard and mouse input mode, liberated the user’s hands, and brought a new user experience. This way can not only liberate people’s body but also interact in real-time and accurately, so that people can get a more relaxed and pleasant user experience. It can be said that digital image art using visual sensing technology extends people’s visual, auditory, tactile and other experiences, and also extends people’s spiritual thinking [15]. The image effect presented by digital images can be comparable with real things and even more aesthetic and real than reality.
2.1.4. Application of Digital Image Art Based on 3D Panoramic Vision Sensing Technology
The interactive game based on a lighting projection device consists of three parts: action acquisition device, data processing device, and image display device. With the ground interactive projection game represented by the new version of “Whac-A-Mole” as an example, this is a cognitive game for young children, similar to the game device of “Whac-A-Mole.” The omnidirectional immersive interactive game interface is a clever combination of game and aesthetic education. Figure 4 shows the working principle. The game is to use the ground interactive projection to put the interactive image on the ground to construct a real interaction, such as grasping, beating, running, jumping, sliding, and a series of actions. 3D panoramic vision sensing technology is applied to integrate omnidirectional vision sensing technology of single view with panoramic color volume structured light generation technology of single emission point, which makes the current active stereo vision develop from the visual perception of surface structure to the visual perception of volume structure [16].
2.1.5. Architecture of Active 3D Panoramic Vision Sensor
Omnidirectional vision sensor (ODVS) [17] is a crucial part of active 3D panoramic vision sensor, which is responsible for the collection of panoramic video information. The ODVS used has a 360° horizontal field of view and 139.8° vertical field of view, which has a wide field of view; the single view imaging constraint can ensure that the whole imaging process conforms to the principle of perspective imaging [18], and the imaging of any point in the scene in the image is unique. The incident angle information of pixels can be obtained by inverse operation of the panoramic image as long as the parameters of ODVS are known. Figure 5 shows the vision sensor model.
In Figure 5, Ov is the single view point of omnidirectional vision sensor, and Op is the single emission point of the panoramic color structured light generator (PCSLG). The distance between single view Ov and single emission point Op is called baseline distance. is the midpoint of the baseline distance, which is called the central eye. is a point in space. Because of the time shared control technology in ASODVS, the color information of object point in the image includes the information of projection light source (Figure 5(a)) when the film of color panoramic projection light source is ON; the color information of the object point in the image collected is the color information of the object point itself (Figure 5(b)) when the film of the color panoramic projection light source is OFF.
ODVS configures the camera behind the hyperbolic mirror, and the camera lens is placed at the virtual focus of the refracted mirror, which can sense all the objects above the hemispherical surface in real-time. Equations (1) and (5) display the optical imaging process.
, , and are the coordinates of the points on the hyperboloid. and are the coordinates of the imaging points. is the distance between the focus of the hyperboloid mirror and the origin, and 2c is the distance between the two focuses. and are the length of the real axis and the imaginary axis of the hyperboloid mirror. β represents the angle of the incident light on the projection plane. γ donates the angle between the incident light and the horizontal plane of the focus of the hyperboloid. α is the angle between the refraction and reflection light of the space point and the principal axis of the hyperboloid.
2.1.6. Stereo Imaging Principle of 3D Panoramic Vision Sensing Technology
The principle of an active 3D panoramic vision sensor is as follows. The projection angle αp and incident angle αo of point at the point of on the imaging plane are determined by the combined action of panoramic color volume structured light generator and ODVS. In other words, the depth information of point can be determined on point on the imaging plane. In theory, a pixel color of ODVS contains the color of the projection light source (projected by panoramic color volume structure light source). By identifying the color of the projected light, the projection angle αp of the projected light can be determined; when the calibration parameters of the ODVS are known, the incident angle αo of the reflected light corresponding to the projection light source of the pixel is also known. For example, the PCSLG emits red light with a wavelength of 650 nm at 30° northern latitude. After the refraction and reflection of the object point in space, it is imaged on the imaging plane of the ODVS, and the red imaging pixel ) as shown in Figure 5(a) is obtained. After recognition by the color recognition algorithm of color structured light, the projection angle [19] of object point is 30° according to the corresponding relationship between the color of PCSLG and the projection angle. The incident angle o of object is 45° according to the calibration parameters of ODVS. The distance between object point and central eye can be calculated by equation (6) when the baseline distance is known.
is the baseline distance, is the incident angle, and is the emission angle. Besides, the azimuth angle β of point relative to the central eye of ASODVS can be obtained. The equations used are (7) and (8).
and are the coordinate value of the pixel on the panorama. and represent the pixel coordinate value of the center point.
2.1.7. Modeling of the Active Vision System
Ideally, the color of every pixel in the ODVS is consistent with the color of the projection light source, but the actual situation is not so simple. The spectral component [20] may change greatly after the reflected light passes through the surface of the object. Besides, the spectral sensitivity of ODVS is also a crucial aspect affecting the color recognition of projected light. The factors affecting the color of the object surface are evaluated. On this basis, the mathematical model of the active vision system is established to explain the relationship between the color of the projection light source and the color of the corresponding imaging point. Figure 6 displays the active vision system model.
Figure 6 reveals that the relationship between the projection light source (, , ) and the three-channel response values (, , ) of the corresponding imaging point is established, which reflects the coupling difference between the projection device and the digital camera. If the values are approximately equal, it means that the spectral curve emitted by the three-color channels of the projection device is consistent with the spectral sensitivity curve of the three-color channels of the digital camera; the interference between the three-color channels of the digital camera is relatively small; the digital camera has good color reproduction performance for the projection device. The transformation between the projection instruction and the real projection value can be expressed by the monotone nonlinear transformation function [21].
2.1.8. Color Recognition of Light Source Based on Colorimetry
The illumination model of the object surface suggests that the reflected light of the object surface is mainly composed of diffuse light and specular light; the diffuse light [22] reflects the proper color of the object, and the specular light reflects the light source color. Thereby, a light source color estimation algorithm based on CIE chromaticity diagram is adopted from the perspective of chromaticity.
The images collected by ODVS represent the color of the object in the form of RBG three-color channel component values, including the brightness, tone, and saturation information of a certain object point on the surface of the object. Tone and brightness are determined by the light intensity of the light source. When the light starts from the light source and propagates in space, its intensity will decay with the increase of distance. The reciprocal of the quadratic decay function can express this attenuation.
It is hoped that the color of the light source projected by the panoramic color structured light at a certain projection angle will not change with the increase of distance; that is, the color of the light source will not change with the change of light intensity. Therefore, it is hoped that the brightness information can be separated from the color information and one or two other stable features can represent a light source color, which is called color representation. There are many kinds of color representations suitable for active panoramic vision, such as wavelength, tone, CIE chromaticity [23], and diagram.
2.1.9. Color Estimation Algorithm of the Light Source Based on CIE Chromaticity Diagram
According to the two-color reflection model, the reflected light of the projected light source on the object’s surface is mainly composed of diffuse light and specular light [24]. The diffuse light reflects the proper color of the object, and the specular light reflects the light source color. According to the law of intermediary colors in Grsassmann color law, any two kinds of noncomplementary color light mixing will produce intermediate color, whose color depends on the relative energy of the two kinds of color light. The above rule is reflected in CIE chromaticity diagram, and its manifestation is as follows. If the chromaticity coordinates of two kinds of color light are known, the chromaticity coordinates of the mixed light are on the line between the two chromaticity coordinates. It can be inferred that if the chromaticity coordinates of one color light and mixed color light are known, the chromaticity coordinates of the other color light must be on the extension line of the former two. Hence, in theory, the chromaticity of the mixed color of the light source color and the proper color of the object can be obtained as long as the area where the specular reflection and diffuse reflection coexist on the surface of the object is known; if the chromaticity of the proper color of the object is known, it can be judged that the chromaticity coordinates of the light source are on the extension line of the chromaticity coordinates of the mixed color and the proper color of the object. Similarly, the same trend can be obtained for different object colors. Figure 7 presents the principle of light source chromaticity estimation.
In Figure 7, point represents the chromaticity coordinates of the object color, point represents the chromaticity coordinates of the mixed color of the light source and the object proper color, and point represents the chromaticity coordinates of the light source. Point may be different due to the degree of mixing of proper color and light color, but it is always on the line of . Point must be on the extension line of according to the Grassmann color law. Light sources have similar effects on different object colors, as shown by multiple pairs of arrows in Figure 8. Similarly, it can be deduced that the chromaticity coordinates of the light source should be on the intersection of the above lines. Figure 8 is a flow chart of the light source chromaticity estimation algorithm.
The specific implement steps are as follows: (1)All the pixels of the collected image before illumination are converted to HIS color space, and the color regions of different object colors in the image are segmented(2)Each pixel in the same color partition of two images before and after illumination is projected into CIE chromaticity space, and the average proper chromaticity and mixed chromaticity are calculated(3)The linear equation of each color partition is obtained by using the chromaticity of the object’s proper color and the chromaticity of the mixed light(4)The point with the least sum of distance squares of all lines is calculated as the chromaticity coordinates of the estimated light source. Due to the existence of image noise [25], the line direction from the object chromaticity to the mixed light chromaticity does not necessarily point to the light source chromaticity correctly, which will cause the intersection of all lines is not unique. Therefore, the chromaticity of the light source can be obtained by finding the point with the least sum of the squares of the distances of all the lines
3. Results and Discussion
3.1. New Version of “Whac-A-Mole” Ground Interactive Projection Game Case
The small animals will make lovely sounds, such as English and Chinese when the children tread and hit correctly. When children fail to tread and hit the small animals, it will prompt the voice of failure, so that the children can enhance their cognitive ability in the process of the game. Figures 9 and 10 are application examples.
3.2. Color Separation Algorithm and Correction of Coupling Difference between Digital Camera and Projector in HIS Space
The coupling difference between digital camera and projector will affect the recognition of light source color. Therefore, the first step is to correct the color of the digital camera and projector. Table 1 is the experimental data.
PCSLG with single emission point is adopted as the projection device. It is hoped that there is a certain linear relationship between the color characteristics of the projected light and the corresponding projection angle, as shown in Figure 11.
The experimental results of the modified algorithm are compared with those of the direct color separation algorithm in HIS space. Figure 12 presents the comparison results.
The experimental results show that the correct recognition rate of light source color for different color object surfaces is greatly improved after the correction of the light source color correction algorithm, and it has a significant effect on improving the recognition rate of projected light color. The color reproducibility of the projection light source is determined by the following aspects: whether the spectral reflectance of the object surface is accurately estimated, whether the separation of specular reflection and diffuse reflection is accurate, and whether the estimation of coupling difference between camera and projector is accurate. Moreover, the algorithm does not need to know the coding mode of the light source and the material of the object’s surface in advance. The error rate of recognition is reduced from 44% to 11% by using three-channel reflectance to correct the surface color. However, there will be a slight difference due to the difference of light intensity in the calculation of three-channel reflectance under natural light or standard white light sources. This is reflected in the actual case application of Whac-A-Mole.
4. Conclusion
Based on previous studies, with “Whac-A-Mole” as an example, the 3D panoramic visual sensing technology is applied to digital image art to study the application of the combination of visual sensing technology and digital image art in the field of art education, so as to provide a new stereo panoramic vision acquisition method; the camera model, projector model, object surface illumination model, and their relationship are studied, and the mathematical model is established. A light source color correction algorithm based on two-color reflection model is proposed, which overcomes the influence of object color, specular reflection, and coupling difference between camera and projector equipment on structured light color recognition to a certain extent, so as to improve the accuracy of color structured light color recognition algorithm. The three-channel reflectance is used to correct the object surface color to reduce the recognition rate from 44% to 11%.
Some basic research on the application of active 3D panoramic vision sensing technology in digital image art is made. The disadvantage is that there is still a certain distance from the practical application stage, and the establishment of the active vision system model is not perfect. Due to the different spectral absorptivities of different light sources, the assumption itself introduces some errors, which need to be confirmed by further research in the future. The specific reasons for the slightly different diffuse reflection components of the same object point under the two light sources are worth exploring.
Data Availability
The data used to support the findings of this study are available from the corresponding author upon request.
Conflicts of Interest
The authors declare no conflicts of interest.