Abstract
In order to improve the effect of film space performance, this paper applies the combination of virtual and reality to the film space performance processing. Moreover, this paper analyzes the light field distribution of the three-dimensional object in space, and modulates the projected image projected on it with the spectral characteristics of the optical screen to finally restore the light field of the original object. In addition, this paper uses the multi-projection method to study the recovery process of the cylindrical microlens array screen to the light field, and divides and reorganizes the multi-view image of the scene shot by the camera array to obtain the correct projected image. Finally, this paper applies the space processing algorithm based on the combination of virtual and reality to the film space performance processing, and uses experimental research to verify the reliability of the method proposed in this paper.
1. Introduction
When filming the film, the most important thing is how to use perspective to enhance and capture the sense of space in the picture. “Perspective is described as drawing various objects on a plane through the convergence of lines and giving people a sense of relative distance between objects in real space.” The term perspective originated from Greece and refers to the method or technique of depicting the spatial relationship of objects on a plane or curved surface [1]. Perspective first appeared in painting, and at the beginning, the painter could only outline the outline of the scene with lines. What the painting showed was a pure two-dimensional form, which could not show the three-dimensional and spatial depth of the actual scene. Later in the Middle Ages, grid forms appeared in paintings. When the painters observe, the scene is gridded through a grid, and then the grid is reduced in proportion to the drawing paper, and then the actual scene is drawn using the grid on the drawing paper. The scenery drawn by this method, on a two-dimensional drawing paper, presents a three-dimensional sense of distance and a distinct depth of space, which is actually perspective [2].
The human eye is born with a perspective effect. When observing, all visible objects in the field of vision automatically enter the perspective state. Just as the painter observes the scene through the mesh grid, in the eyes of humans, the objective scene seems to be automatically included in a virtual grid plane. Moreover, the scenery lines and the air present a perspective effect, and show the perspective effects such as the density of virtual reality and the density of light and darkness [3].
Surrealist film creators usually apply the elements of surrealist dreams to movies. The fusion of surrealist space and reality is worth exploring in terms of technique and function. It forms a unique film narrative structure while also highlighting the emotional expression of the characters and showing the inner state of the characters. Digital technology helps the fusion of surrealist space and reality in a variety of ways, and on the basis of ensuring the transition of natural fusion, realizes the rich narrative features of surrealist space.
Based on the combination of virtual and real technology, this paper conducts research on film space performance, explores the detailed application process of film space expression techniques, and proposes the intelligent system of this paper to provide a theoretical reference for subsequent film space performance.
2. Related Work
Digital film and television” is a brand-new type of film and television produced under a new film and television production method that combines certain traditional film production technologies through computer and digital, audio and video, processing disk recording, and network technologies. The complete process from the early stage to the later stage to the release [4]. For example, in the early stage of creation, the computer-aided system is used to design, draw, test and simulate the scenes, plots, pictures, etc. of future films in order to find the best narrative Techniques and solutions for creating visual impact; another example is the use of computer control technology in actual shooting to complete certain shooting that cannot be done with traditional methods; another example is the use of computers to process and process images and sounds in post-production, The real shot material and computer images are synthesized, and the final synthesis processing is performed on the computer [5]. The emergence of digitization not only allows us to see the hope of revival of image creation through computers; but also makes us feel all kinds of In the growing market, computers and software have been used in all aspects of film and television, which has improved production efficiency [6]. “The practice of using computers to change, enhance and reshape the original pictures of films has developed steadily. Computers use digital methods and methods of enhancing film images to create magical images: using data and movies as digital outdoor scenes, digital color correction, filter effects and painting, digital image synthesis, digital animation production, Digital film repair, digital line mark removal, digital image enhancement, digital defect elimination, the integration of computer-generated material and film shooting material [7]. “Digital technology has caused us a strong and dazzling visual impact, and the fake process of its modified images is so simple and clever. With these, we have displayed the power of computer imaging technology at a glance. No wonder some people would say, “Today’s movies are no longer “made”, but “made”. “The digitization of film and television has fundamentally changed the fate of film and television: digital film and television create a new era of entertainment facilities [8]. When film and television audiences are attracted by other media, film and television can only survive by reforming again. Digital computing creates The multi-layer synthesis of the breathtaking scenes, the shocking sound effects, and the extra large screen without the frame, the presence and reality of the cinema projection will be unmatched by any media [9]. Various special effects Entertainment programs, dynamic movies, virtual scenes, and various new types of programs will inject fresh blood into the production of motion video programs. The production and broadcasting of film and television programs has become more diversified, randomized, globalized and accessible due to the addition of digital methods. Pursuing. The development of film and television technology is closely linked to the continuous progress of science and technology. In the early stage of film and television development, the film recorded life at that time, and the development of digital technology has recorded our lives in a rich and colorful [10]. From the early silent black and white movies to the current multi-channel surround sound digital movies, from the original reception of TV signals to the current online video on-demand, the continuous evolution and development of technology and equipment has promoted the change and innovation in the thinking of creators [11]. “At the same time, digital technology has had a revolutionary impact on film and television production methods since its birth. It has incredible expansion of film and television expression space and realizability, creating unprecedented audiovisual wonders and virtual reality that people have never heard of, seen, or even imagined. .Therefore, a new narrative method is produced, and a set of new rules is proposed for us to understand, use, and break through. The new digital technology not only produces new video works, but also cultivates a new generation of video audiences [12].
The production process of digital special effects movies is roughly divided into three main parts: pre-planning, mid-term production, and post-production. The pre-planning is mainly composed of directors, special effects artists, screenwriters and other key creative personnel to jointly determine the story outline, that is, the visualized script. It is necessary to complete a unified planning, design, and determination of all involved image elements, such as characters, environments, and props. Special effect production plan and test, determine personnel’s responsibilities and coordination work [13]. Mid-term production is carried out around the production of animation, including model establishment, material and lighting settings, special effects generation, image rendering, etc., and in conjunction with these core productions, some peripheral cooperation work should be carried out, such as providing three-dimensional scanning. Data model production, texture mapping, motion data capture, etc. [14]. The post-compositing work is mainly divided into three aspects: one is to complete the shots that are inconvenient in the animation software at one time by layered synthesis; the second is to add special effects; the third is to improve the overall artistic style and picture quality of the film Process [15]. Faced with the lack of real images in digital images, the anxiety of filmmakers has blurred the content of the ideological film to convey to the audience, because it blurs the difference between the essence of the film and the entertainment of the film. This sense of crisis is expressed in a “loss” argument: digital images used to be thought that the film may present a poor version of the effect. Literature [16] even regards digital special effects as a representative of industrial revolution automation. Digital media is only a representation of symbols. Living in such a formal and diversified world, the degree of separation from the material of the real world is unprecedented. This has an important trend, which is to liberate the engrossed audience from the so-called real world.
3. Space Image Processing Algorithm Based on the Combination of Virtual and Reality
The plenoptic function is based on the observer’s description of light in space and time. The mathematical form of the plenoptic function is [17]:
It is a 7-dimensional plenoptic function. Among them, (x, y, z) represents the coordinates of the light in space, and t are the wavelength of light and the time when the light is observed, respectively. Because the wavelength of light changes very little in free space, for a single observer, the plenoptic function of an object observed at a specific position at a certain moment can be expressed in a 5-dimensional form [18]:
In the light field three-dimensional display, the three-dimensional object to be displayed can be regarded as composed of a large number of three-dimensional points, and each three-dimensional point actively or passively emits light to the surroundings. The light that enters the human eye causes us to see the corresponding three-dimensional object. Therefore, the plenoptic function can be used to represent the light field of a three-dimensional object:
Among them, i = 1,2, …, N is the number of three-dimensional points of the object, as shown in Figure 1.
Due to the huge amount of information of the five-dimensional light field, usually in the process of three-dimensional display of the light field, the light field information in the vertical direction is compressed in combination with the optical screen, and only the light field information in the horizontal direction is processed. The light field at this time is expressed as [19]:
In Figure 1, the holographic directional scattering screen is located at , the light emitted by the three-dimensional point intersects the screen with point M, the direction vector is , and the straight line OM can be expressed as:
From formula (5), the x and y coordinates of point M can be calculated as: and respectively represent the spatial angle information in the x and y directions, and the formula (6) can be written as [20]:
At this time, the light field of the light emitted by the three-dimensional point on the holographic directional scattering screen can represent .
Next, in order to better understand the reproduction of the spatial light field by the three-dimensional display of the light field, the light field distribution of the three-dimensional object in the space is analyzed below. We assume two point elements A and B of the object in space, and analyze their light field distribution on the plane P. It can be seen from formulas (5)–(7) that x and y are related to α and β respectively, but they are independent of each other. Therefore, this section selectively analyzes only the x direction of light.
The three-dimensional coordinates of the point elements A and B are and respectively. They emit light in all directions, and the plane P is located at , as shown in Figure 2. Generally, for any piece of light, according to formula (7), the light field of point element A and B at plane P can be expressed as:and
The corresponding light fields are shown in the right figure of Figure 2, which are two oblique straight lines. In the figure on the right, the abscissa represents the spatial position of the light field, and the ordinate represents the angle information of the light in that direction. Since the point element A is far away from the plane Р, the light field it produces has a smaller slope.
By doing the above analysis on all the point elements of the object in space, the entire light field of the object at the plane Р can be obtained. Therefore, the purpose of the three-dimensional display of the light field is to use the optical screen to restore the light field of the object so that it can be seen by the observer, just like seeing a real object.
Considering that the projection distance of the projector is usually much larger than the projection aperture, it can be approximately considered that the light reaching the surface of the cylindrical microlens array screen is parallel light. The modulation mechanism of the cylindrical microlens array screen to the projection light is shown in Figure 3. The red, green, and yellow lights in the figure represent the collection of light projected from three different directions. When the light in the horizontal direction is converged, due to the scattering of the scattering film, part of the light will propagate back to the original projection direction according to the incident light path. Therefore, observers at different positions in the observation area will see the image projected by the corresponding projector, as shown in Figure 4.
Next, this paper analyzes the recovery process of the cylindrical microlens array screen to the light field by means of multi-projection. In Figure 4, the projector array is located in front of the screen, and the distance from the screen is d. The modulation function of the cylindrical microlens array on the light field can be expressed as:
The light to be projected travels in the opposite direction. The pupil coordinate of projector is , which is regarded as a point light source. The light field of the light projected by on the cylindrical microlens array screen is expressed as:
The information of each ray in the light field is determined by the image projected by P. By analogy, the light field corresponding to the projected image of the entire projector array can be obtained. The light field distribution corresponding to multiple projected images is shown in Figure 5(a), which shows the light field distribution in the x direction. It can be seen from Figure 5(a) that the light field of the image projected by the projector array is a series of straight lines with the same slope, and the slope is negative. Therefore, by combining the real light field distribution of the object point element in formula (8) and formula (9), the algorithm uses formula (11) to sample it to calculate the projected image corresponding to each projector in the projector array, as shown in Figure 5(b).
(a)
(b)
In particular, it can be found from Figure 4 that when the observer is under the projector array, the viewpoint image observed by the human eye is the same as the projected image of the projector. When the human eyes at different positions at this time are imagined as a camera array, it can be inferred that the multi-view image of the scene shot by the camera array is the projected image of the projector array.
We assume that the holographic directional scattering screen is located at zi in Figure 2, and the projector array is located in front of the screen, and the distance from the screen is . The observation area is located behind the screen with a distance of , as shown in Figure 6. The modulation function of the horizontal holographic directional scattering screen can be expressed as:
That is, the horizontal direction does not change the spread of light. The pupil coordinate of the projector in Figure 6 is , and the light field of the light projected by on the holographic directional scattering screen is expressed as:
The x-direction distribution of the light field corresponding to multiple projection images is shown in Figure 7(a), and the light field is a series of straight lines with the same slope and positive values.
(a)
(b)
Similarly, by combining the real light field distribution of the object point elements in formula (8) and formula (9), when the algorithm uses formula (13) to sample it, the projected image corresponding to each projector in the projector array in the case of using the holographic directional scattering screen can be calculated, as shown in Figure 7(b). However, unlike the light field recovery under the cylindrical microlens array screen, the viewpoint image content of the observation area in Figure 6 is not a projection image of a certain projector, but a collection of multiple image blocks projected by the projector array. For example, the viewpoint image at in the figure is a mosaic of all the projected images after being modulated. Therefore, when performing multi-projection display, the multi-view image of the scene shot by the camera array needs to be segmented and reorganized to obtain the correct projected image.
In this paper, a front projection light field three-dimensional display system based on a cylindrical microlens array screen (projection array and the observer are on the same side of the screen) and a rear projection light field three-dimensional display system based on a holographic directional scattering screen are built (the projection array and the observer are on both sides of the screen). The system mainly includes two main parts: light field acquisition and light field display. The hardware structure of the system is shown in Figure 8.
The camera array (GigE interface) of the acquisition part collects the light field of the real scene. The camera array is arranged in an arc and compact, and is connected to the computer through a switch. The display part is mainly composed of a projection control computer, a projector array (HDMI interface) and a display screen. The projector array is staggered and closely arranged, and is connected to the computer through a screen splitter and multiple multi-channel graphics cards. The light field data captured by the collecting terminal is transmitted to the display terminal for display through the UDP protocol. The working resolution of the camera array and the projector array are both set to 1280 × 800.
The software module of the system is designed according to the structure of the hardware. The functions realized by the collection terminal include the collection of the light field by the camera array, the correction of the collected image, the generation of the projected image and the synchronous control of the collection. The display terminal mainly includes projection correction and synchronization control. The software block diagram of the system is shown as in Figure 9.
Assuming that there are four landmark points A, B, C, and D in the target light field, their coordinates are , and the coordinates of target points A′, B, C′, and D′ are , as shown in Figure 10. After extracting the coordinates of the marker point, map it to the target point through perspective projection to obtain the perspective transformation matrix P. The perspective transformation formulas are:
In the formula, Р is a 3 × 3 perspective transformation matrix, and is a constant value coefficient. The form of P is:
To simplify, we normalize , that is, . From formula (14) and formula (15), we can get:
There are 8 unknowns in the above formula, which can be solved by 8 equations formed by four pairs of coordinate points to obtain the perspective transformation matrix P. By multiplying the light field multi-view image collected by the camera array with the perspective transformation matrix Р, the light field image of the same field of view can be obtained. The proposed camera array light field acquisition image correction method will reduce the spatial resolution of part of the light field, and finally the light field data within the range of the mark point will be retained.
When the three-dimensional model of the scene is known, a virtual camera can be used to render the viewpoint image of the scene, and then segmentation and reorganization are performed to obtain the projected image. However, the efficiency of selecting this method for three-dimensional display is low. When the 3D model of the scene is known, the projection image is usually rendered directly from the 3D model. We establish a three-dimensional coordinate system with the center of the holographic directional scattering screen as the origin, and assume that the array of projectors in the system are arranged in arc rows, the arc radius is , and the angular interval between adjacent projectors is .
The viewpoint distribution of the observation area is a circular arc with a radius of and a height of , and is any point on the surface of the three-dimensional model. The following analyzes the process of using the model rendering method to obtain the projection image corresponding to the projector in the projector array.
As shown in Figure 11, first, the optical center of the projector and the three-dimensional point S are connected and extended to the cylindrical surface where the arc of the viewpoint distribution of the observation area is located, and the intersection point is denoted as . The equation of light can be expressed as
Among them, is the proportional coefficient. The cylinder where the arc 1 of the viewpoint distribution is located can be written as:
Through simultaneous formula (17) and formula (18), the proportional coefficient can be calculated as:
Thus, the three-dimensional coordinates of can be expressed as:
Therefore, it can be known that the coordinate of the viewpoint corresponding to is . It can be seen from Figure 11 that the three-dimensional point S observed at the viewpoint.
Since the holographic directional scattering screen is parallel to the y = z plane, the normal vector at this time is , and its surface equation can be written as:
In the same way, through simultaneous formula (21) and formula (22), the three-dimensional coordinates of S’are calculated as
Among them, . Then, through the decoy projection transformation, the two-dimensional pixels in the projection image corresponding to the three-dimensional point S′ are calculated.
We assume that the horizontal and vertical field of view of the projector is o. Since the projection plane of the projector is in the yz plane, the image coordinates in the projector P corresponding to the three-dimensional point S′ after transmission projection can be expressed as:
In the formula, the round symbol means to take the closest integer, and M and N are the horizontal and vertical resolutions of the projected image, respectively. Repeatedly, by using the transmission projection method to traverse the entire three-dimensional model, the light field of the model corresponding to the complete projection image in the projector P can be obtained. After that, the same processing is performed on the entire projector array to generate projection images corresponding to all the projectors.
When a camera array is used to collect light field data of a real scene for three-dimensional display, the collected light field viewpoint image is the viewpoint image in the observation area. In this case, the viewpoint image needs to be segmented and reorganized to generate the projected image.
In order to facilitate understanding, the three-dimensional display of the light field of the three projection images is used as a column for analysis. As shown in Figure 12, since the holographic directional scattering screen does not change the angle of the projected light in the horizontal direction, viewpoints located at different positions in the observation area on the right side of the screen will see different projected image content. And each viewpoint image is composed of different parts in different projection images. The specific performance is as follows. We assume that the left, center, and right three projection images are denoted as A, B, and C respectively, and they are all divided into three columns of sub-image blocks. When the projected images A, B, C are projected to the same area of the holographic directional scattering screen, the viewpoint images seen by the observer at the left, middle, and right positions of the observation area will be B, C, and A, B, C, and A, B respectively, which are composed of three different sub-image blocks of the projected image.
We assume that the camera array captures N viewpoint images of the light field of the real scene. Each viewpoint image is divided equally into columns, and the number of equally divided columns is N, which is represented by the symbol V[i,j]. Among them, i is the serial number of the viewpoint image, and j represents the jth column in the image i. In the process of generating the projected image, in order to better explain the method of projected image generation, all the projected images are regarded as a large image that is horizontally spliced by N columns of equally divided projected images, which is represented by the symbol P[n]. The n in is the column number in the projected large image, and satisfies 1 ≤ n ≤ N’. For example, P [6] represents the sixth column in the projected large image. It is concluded through experiments that the corresponding relationship between the viewpoint image and the projected large image can be expressed as:
This paper proposes a projection correction method based on solving the homography matrix. The purpose of projection correction is to correct all projected images to a designated area of the display screen, and its essence is to map the projected image plane coordinates to the world coordinates where the display screen is located. Therefore, the mapping relationship between the planes can be achieved through homography mapping. In particular, in order to improve the efficiency of projection correction, a camera is used as an auxiliary tool for correction.
The correction method involves three plane coordinate systems: the projection coordinate system , the camera coordinate system , and the display screen coordinate system . The homography matrix from the projection plane to the camera plane is denoted as , and the homography matrix from the camera plane to the display plane is denoted as . The homography transformation matrix for projection correction can be expressed as:
The implementation steps of the projection correction method are as follows. First, each projector projects a standard chessboard image on the display screen, uses the camera to record the deformed chessboard image, uses feature extraction algorithms to extract the corner coordinates of the recorded chessboard image, and calculates the homography matrix . Then, the coordinates of the corner points of the deformed chessboard corresponding to each projector recorded by the camera are mapped to the specified screen coordinates, and the homography matrix Hes is calculated. Finally, the homography transformation matrix H is obtained by formula (26). The flow of the projection correction algorithm is shown in Figure 13.
4. Evaluation of the Application Effect of the Combination of Virtual and Reality in the Film Space Performance
On the basis of the above research on the virtual and real combination algorithm, this paper analyzes the application of virtual and real combination technology in film space performance. This article uses expert evaluation method to evaluate the effect, and obtains multiple sets of resource data through the Internet as the research sample of this paper. First, this paper evaluates the spatial image correction and projection process of the system proposed in this paper, and the results are shown in Table 1 and Figure 14 below.
It can be seen from the above research that the virtual-real combination technology proposed in this paper has good spatial image correction and projection effects in the design of film space. Afterwards, the film space performance effect of this system is evaluated, and the results shown in Table 2 and Figure 15 below are obtained.
From the above research, it can be seen that the virtual and real combination technology proposed in this paper performs well in the design of film space, which verifies that the method proposed in this paper has a certain effect.
5. Conclusion
The perspective effect produced on the film screen is similar to the principle of the perspective effect produced by painting. The objective scene in the form of three-dimensional space can produce a perspective effect through the optical lens of the camera due to the difference of the spatial position in the actual space. Then, three-dimensional images of different sizes, shapes, and colors are formed on a flat photosensitive medium to present images with a sense of space and depth. In film shooting, the lens replaces the human eye. As far as perspective is concerned, there is no essential difference between the perspective of a film picture and the perspective of a conventional painting, but the viewing range is limited by the focal length of the lens. From the perspective law, it can be found that when the human eye puts the scene into the field of view, it can automatically see through, thereby judging the actual size and distance of the object. Based on the combination of virtual and real technology, this paper conducts research on film space performance, explores the detailed application process of film space expression techniques, and proposes the intelligent system of this paper to provide a theoretical reference for subsequent film space performance.
Data Availability
The labeled dataset used to support the findings of this study are available from the corresponding author upon request.
Conflicts of Interest
The authors declare no competing interests.
Acknowledgments
This study is sponsored by Sejong University.