Abstract
Stereoscopic display is the means of showing scenes in Virtual Reality (VR). As a type of stereo images, anaglyphs can be displayed not only on the screen, but are currently the only solution of stereo images that can be displayed on paper. However, its deficiencies, like retinal rivalry and color distortion, could cause visual fatigue. To address this issue, an algorithm is proposed for anaglyph generation. Unlike previous studies only considering one aspect, it considers both retinal rivalry and color distortion at the same time. The algorithm works in the CIE color space and focuses on matching the perceptual color attributes especially the hue, rather than directly minimizes the sum of the distances between the perceived anaglyph color and the stereo image pair. In addition, the paper builds a relatively complete framework to generate anaglyphs so that it is more controllable to adjust the parameters and choose the appropriate process. The subjective tests are conducted to compare the results with several techniques which generate anaglyphs including empirical methods and computing methods. Results show that the proposed algorithm has a good performance.
1. Introduction
Through the stereoscopic display technology, VR can show various scenes by vivid realistic visual effects and provide more information. The technology can be used in education, training, and many different application fields. The mechanism of stereo imaging is so-called the binocular vision. The image received by the left eye is slightly different from that received by the right eye of the same scene. Based on these two images, the human brain calculates the depth information of the scene [1]. A binocular vision system includes two cameras placed in parallel using different views to capture two images in order to obtain the depth information [2, 3]. There are three kinds of stereoscopic display technology based on the principle of binocular parallax. They are autostereoscopic display, helmet-mounted display (HMD), and stereo glass display. Autostereoscopic display can realize the stereoscopic effect viewed by naked eyes. It is currently very limited and needs much further researches. HMD has developed rapidly in recent years, and the device becomes much lighter than before. It completely blocks people’s vision from the outside world, so that people are immersed in the stereoscopic virtual world. HMD uses two channel videos of stereo image pairs as input signals of each small screen which is close in front of each eye. Stereo glasses are the most used stereo display devices in public where the stereo image pairs or two channel videos are projected onto a large curtain, displayed on computer screens or prints. Colored-filter glasses are a type of stereo glasses with anaglyphs as input stereo image pairs, by which the stereoscopic visual effect can be perceived.
Anaglyphs have some unique advantages which are not available from other stereoscopic display technology: (1)Compared with full-color stereo image pairs, the image compression ratio of anaglyphs reaches more than 79% for the same scene. This is an outstanding advantage for data transmission(2)Users can perceive the stereo effect not only on screen, but also even on prints [4, 5]. Anaglyphs are the only way that can display stereo objects on prints. It just needs ordinary paper without using any special ink or special printer. This obviously provides an attractive way of conveying stereo information to users, because the stereo objects in papers can be taken anywhere to provide a vivid visual experience when people communicate(3)Compared with the polarized glasses which need to use expensive displaying devices and compared with the shutter glasses which need to strictly control the high refresh rate, anaglyphs just need an inexpensive colored-filter glasses instead to form an specific stereoscopic display system. The stereo effect is achieved by allowing different light waves of anaglyph images to reach each eye using different colored-filter glasses. The colored-filter glasses are the most inexpensive hardware among the stereoscopic display devices. This characteristic removes an economic obstacle, especially for underdeveloped areas(4)Only anaglyphs can be used as the input signals for all the other display devices. That means anaglyphs have the strongest commonality
However, anaglyphs are also riddled with problems. A problem needs to be considered for generating anaglyphs is retinal or binocular rivalry. If the difference of color or luminance between the perceived left and right images exceeds a certain level, it will cause one of the two images to dominate to be seen at some point in turn [6, 7]. Thus, the eyes will perceive two slightly different images alternatively, instead of fusing them into a single and stable stereoscopic perception. Retinal rivalry is of great concern in anaglyphs because it often leads to visual fatigue and other side effects. Fortunately, the adverse consequences of retinal rivalry can be effectively reduced by manipulating the luminance.
When generating anaglyphs, color distortion is also an intractable problem which will cause degradation in the stereoscopic effect. Color distortion can also be understood as a limited color reproduction. It may lead to the perceived color through the colored-filter glasses to be very different from the original view and directly determines the color fidelity of the perceived stereo scene.
In this paper, a novel method is proposed for anaglyph generation, which considering the retinal rivalry and color distortion together instead of some studies which only consider one aspect in a one-sided way. We choose LCD as the displaying device and colored-filter glasses as viewing device. For colored-filter glasses, red-cyan glasses are most common nowadays. It can be bought for just a few cents, and most books with anaglyphs are also sold with red-cyan glasses together. So we choose red-cyan glasses to do this study. The principle may also be applied to the other colored-filter glasses but some parameters may need to be manipulated renewedly, which are beyond the scope of this paper.
Then, we will introduce the outline of this paper. Section 2 gives the brief review of existing studies on anaglyph generation. Section 3 gives the mathematical framework of the proposed algorithm and its motivation and novelty. Section 4 introduces a detailed implementation of the proposed method. Section 5 compares the results of our method with other methods and gives the research results of the experiment. At last, conclusions are given in Section 6.
2. Related Work
The anaglyph method was patented in 1891 by Louis Ducos du Hauron, but similar methods had been demonstrated previously by W. Rollmann in 1853 and J.C.D’Almeida in 1858. Since the conception of anaglyphs in the 1850s, not many anaglyph generation techniques have been developed and most of them are empirically designed until a mathematical analysis by Dubois [8] in 2001. The remarkable advantages of anaglyphs are low-budget viewing devices, efficiency transmission, and being the only solution that can be printed on paper and perceived stereoscopic effects for arbitrary angles without reducing brightness of images. The improved algorithms based on Eric Dubois’ method are constantly emerging, which make anaglyphs viewed more comfortable to some extent. They can be summarized as computed method.
Computed method takes into account the spectral parameters of the display device and the absorption curve function of colored-filter glasses for different bands. Dubois’ method of generating anaglyphs begins with an understanding of the visual pathway from the digital image values until it is perceived by the viewer. The core idea of McAllister’s method [9] is transforming the calculation of the color distance between the perceived anaglyph and the stereo image pairs into CIE color space which is a uniform color space. The whole processing is similar to the procedures of Dubois’s method. Midpoint algorithm [10] is also worked in CIE color space. It directly use the midpoint of the perceived left image and perceived right image in CIE color space as the information of the anaglyph. In contrast to the above methods, Songnan Li’s method [11] is trying to match the perceptual color appearance attributes instead of minimizing the error matrix of color distance in CIE color space. It presents a new train of thought. The method is aimed at minimizing the perceptual differences between the original and perceived colors based on matching the meaningful color appearance attributes instead of different color spaces, which is also the inspiration of our proposed algorithm.
From Figure 1 of gate, stereo image pairs were taken by the stereoscopic camera, and the anaglyphs of four computed methods are shown in Figure 2. The anaglyphs in upper line are generated, respectively, by Dubois’ method and McAllister’s method, and the anaglyphs in lower line are generated, respectively, by Midpoint algorithm and Songnan Li’s method.
3. Framework of Anaglyph Designing
3.1. Systematic Framework
For the convenience of presentation, we denote the left image of the stereo pairs as , the right image as , and the final anaglyph as in RGB color space. The systematic framework of our method for generating anaglyphs is shown in Figure 3.
As the frame figure shows, we need to convert the stereo pairs into CIE color space firstly and then take different operations for the left and right images. Our objective is to obtain the and channels from the processed right image and the channel from the processed left image. Instead of using the channel of left image to be the channel of the final anaglyph and the and channels of right image to be the final anaglyph’s and channels directly, our algorithm will do a series of operations for the stereo pairs in CIE color space before this procedure. CIE color space is based on the physiological characteristics of human beings. It is a digital way of describing persons’ visual perception. In the CIE color space, the change value of the color is consistent with the change amount of the eye perception of the color. So in the second step, a matching process based on the color attributes for the right image is implemented in CIE color space. In addition to the matching principles in Songnan Li’s method, our method is more focused on the matching of hue as well so that we can get a better color reproduction. Next, we will give a delightness operation for the left and right images simultaneously to reduce the effect of retinal rivalry caused by the nonuniform brightness between them, which is also conducted in CIE color space. At last, we need to calculate the , , and component values of the processed right image and processed left image and then convert them into RGB color space. Combining channel derived from the processed left image and and channels derived from the processed right image, the anaglyph is completed by this time.
This systematic framework just shows the simple process of generating anaglyphs which is used to help the readers to form a preliminary understanding. The specific implementations will be shown in Section 4.
3.2. Inspiration and Advantages
This method is inspired by Professor McAllister’s method and Songnan Li’s method. The anaglyph generated by McAllister’s method has a great improvement in color reproduction but with a heavy retinal rivalry especially in the red-like region, and the anaglyph generated by Songnan Li’s method does have a better visual comfort but it has a heavy color distortion. Through the analysis of this phenomenon, we make an improvement on Songnan Li’s method to gain a better color reproduction by adding another matching principle which is the matching process of hue in CIE color space as well. In order to ensure the hue of matched image as close to the hue of original image as possible during the matching processing, we can get a more accurate color reproduction.
In terms of computation complexity and time complexity, our algorithm has obvious advantages compared to McAllister’s method which needs iterative optimization. At the same time, because the matching processing based on color attributes need to be conducted in CIE color space, we can adjust the saturation and lightness more flexible according to different stereo pairs.
In summary, our method can dynamically adjust the balance between the retinal rivalry and color reproduction.
4. Detailed Information of Generating Anaglyphs
As the systematic framework in Section 3 described above, we need to do different treatments for the left and right images. Detailed information on each processing step of the systematic framework will be given below.
4.1. Converting the Stereo Pairs to CIE Color Space
For the convenience of description, we denote the left image in CIE XYZ color space as and the right image in CIE XYZ color space as . Accordingly, can represent the left image, and can represent the right image in CIE color space. In the first step, we need to chip the elements of the stereo pairs in the range 0-1 and then convert the stereo pairs to CIE color space, which requires the CIE XYZ color space as the intermedium. So we can separate this conversion into two steps, and the first step is to map the stereo pairs from RGB color space to CIE XYZ color space:
where , , and represent, respectively, the , , and component values of left image in CIE XYZ color space, and , , and represent, respectively, the , , and component values of right image in CIE XYZ color space. The values of conversion matrix are given in formula (2) which refers to [9]. At the same time, we give the values of and by formula (3). These values were calculated by Professor McAllister, who made an improvement based on Dubois’ computing method so that these values are also used for LCD displaying which is exactly what we need.
In the second step, we need convert the left image and right image to CIE color space, respectively, by the formulas below:
where , , and represent, respectively, the , , and component values of left image in CIE color space, and , , and represent, respectively, the , , and component values of right image in CIE color space. is , , and values of white point, and we can get its values by
The specific expression of function is
4.2. Matching Color Attributes in CIE Color Space
According to [11], color attributes are deemed to be the color representation in the late stage of HVS. Therefore, similar to Songnan Li’s method, our designing method is also matching the meaningful color attributes. But in addition to its matching principles which require the matched right image can generate lightness, hue, and saturation values as close to its original perceived right image and require the matched image can be represented by with , our method is more focused on the matching of hue as well. Thus, we can get a better color reproduction. For the stereo pairs which have been converted to CIE color space, we need to deal with them, respectively, as follows.
4.2.1. Preprocessing for the Right Image
In CIE color space, the meaning of saturation is the same as the chroma under the condition of describing the psychological attributes of one color. So, in terms of right image, we can get its corresponding saturation and hue by
4.2.2. Matching the Right Image
For the matching principles we have mentioned above, two of the most important are as follows: (1)In CIE color space, , , and are corresponding to , , and component values of the matched right image which should be close to the perceived right image filtered by the colored glasses as possible. We can regard this principle as a projection, i.e., we need to ensure that the distribution of the matched image in CIE color space is in the scope of the distribution of the perceived right image in CIE color space(2)Utilize the immutability of hue to assure the hue of the matched right image is equal to the hue of the original right image. We can regard this principle as the rule of projection, i.e., we must ensure the hue of matched right image as close to the hue of original right image as possible during the projection
Therefore, according to the first principle, we can use a matrix which could represent every color in RGB color space as the right image to calculate its corresponding perceived , , and component values which can be denoted by , , and in CIE color space and draw the 3D figure. Thus, the distribution range in the 3D figure is the range of any perceived color filtered by the colored glasses in CIE color space, which is called the perceived field. However, the computational complexity and time complexity of drawing the target 3D figure are considerable, which is a big burden for the computer’s memory. Instead of drawing the perceived filed directly using the above means, we choose to utilize the relationship between RGB color space and HSV color space to convert the matrix which has elements that can represent every color in RGB color space into 100 HS figures whose is an integral ranges from 1 to 100. Here, is the brightness of HSV color space, HS figure is a 2D plane whose horizontal axis is the hue, and vertical axis is the saturation. Figure 4 shows a HS figure with .
Based on the above analysis, we can get down to the 100 HS figures in turn. If , , and channels of HS figure in RGB color space can be denoted by , , and , and , , and can represent its corresponding , , and values, thus they can be calculated by
Then the perceived , , and can be calculated by
where the values of conversion matrix have been given above, which is calculated by taking the spectral absorption curve of cyan glass and the primaries of the display devices together into consideration. is the reference white point in CIE XYZ color space, which can be computed by
After calculating , , and of these 100 HS figures in CIE color space, we can draw the 3D figure. The distribution range of this figure is a series of point cluster, which means the range of most perceived colors filtered by cyan glass and can be regarded equivalent to the perceived field. In fact, the purpose of the matching processing is aimed at mapping the distribution range of the original right image into this perceived filed, so it is necessary to solve the function of the surface of the point cluster. Fortunately, we have conducted a number of experiments based on the relationship between HSV color space and RGB color space to find that there is a lot of overlap in the distribution range of the 100 HS figures converted into the CIE color space. In order to further reduce the calculation amount, we decide to choose convert 10 HS figures with , 20… 100 to CIE color space to draw the 3D figure. As for the other HS figures, they must also be mapped into this distribution range based on the transformational rules. Figure 5 shows the 3D figure drawn by the 10 HS figures, which is proved to be approximate to the perceived field as well.
From Figure 5, we can see that the original surface is nearly perpendicular to 2D plane. Therefore, according to the first principle of matching, we can set the component of aside temporarily and do the matching procedure based on the color attributes in 2D plane. Figure 6 shows the projection of the 3D point cluster in 2D plane, and 10 different colors represent, respectively, the different distribution range of 10 different HS figures when they converted into CIE color space.
To satisfy the first matching rule, we need to make a nice curve fitting for the boundary of the distribution range in 2D plane. Defined the fitting curve composed by the boundary points in the top-right corner as the curve ① and the fitting curve composed by the boundary points in the below-left corner as the curve ②, which have signed in Figure 6. Considering the distribution of point cluster and the efficiency of curve fitting, we adopt the quadratic polynomial curve fitting to calculate the curves ① and ②, and their functions of the fitting curve are as follows:
where and are the and components of curve ① and and are the and components of curve ②. After the fitting processing, we can get the perceived filed shown in Figure 7 which is composed by the curves ① and ② and their internal zone. For the right image of any stereo pairs, the distribution range of the perceived right image filtered by the cyan glass in CIE color space will be on the inside of the perceived filed.
According to the second principle of matching, we need to assure the hue of the matched right image is equal to the hue of the original right image, which can be described by the following formula:
Thus, what we need to do next is map the points outside the perceived filed into the boundary of perceived filed with minimum distortion and the points inside the perceived filed remain the same in CIE color space to make the , , and of matched right image as close to , , and of the perceived right image as possible which also is the first principle of matching rules. At the same time, keeping the hue of the right image unchanged after the matching processing, i.e., for the points located outside the top-right of the perceived filed, we need to use formula (16) to project them into the curve ①.
Simplifying this equation set, we can get the following:
Similarly, for the points located outside the below-left of the perceived filed, we need to use formula (18) which has been simplified to project them into the curve ②.
With this, the matching processing based on the color attributes for the right image has been completed, and we can get the components of and in CIE color space by above calculations.
4.3. Lightness Adjustment for the Stereo Pairs
During the processes of generating anaglyphs, the phenomenon of retinal rivalry will be quite obvious if saturation and lightness of the red-like color are very bright simultaneously, which will lead to visual fatigue easily. Therefore, it is necessary to adjust the lightness properly to reduce the effect of the retinal rivalry.
In RGB color space, the , , and values of red point can be denoted as . We can utilize the conversion procedure to convert this point into CIE color space to calculate its corresponding hue, i.e., and then adjust the hue which is in this range ( and ). According to a number of experiments (refer to Songnan Li’s method), we determine the threshold . So that the processed lightness of the right image is
where is a weight parameter; its values can be changed by its corresponding saturation, and its specific expression is
where , , and , which refer to the Songnan Li’ s method. It means that will be increased if the saturation is higher, which will lead to the values of which has to be reduced to avoid the effect of retinal rivalry. On the contrary, if the only changes a little, which means that the effect of retinal rivalry cause by its own lightness is not too much, so the matched lightness has not a big change.
Similarly, the lightness adjustment of the left image will be implemented by
where is the processed lightness of left image. also is a weight coefficient, which plays a role similar to the . Its specific expression is
and are the hue and saturation of the left image, which can be calculated by
With this, we have completed the lightness adjustment for the stereo pairs. On one hand, this empirical operation is helpful to improve the chrominance accuracy of red-like colors to avoid the heavy color distortion. On the other hand, this operation can keep the balance of the lightness between the processed left image and processed right image to reduce the retinal rivalry caused by the different perceived lightness.
4.4. Converting the Processed Stereo Pairs into Color Space
Through the above processes, we can calculate the , , and of the processed right image and of the processed left image. Firstly, for the processed right image, we need to convert it into the color space. Undoubtedly, this conversion needs the CIE XYZ color space as the intermedium. We denote the , , and as the , , and components of the processed right image in CIE XYZ color space, which can be calculated by
where is the inverse function of the function , and its specific expression is
Then, utilize the matrix which is the inverse matrix of to convert the , , and to RGB color space by the following formula:
For the processed left image, we have already known the lightness component . Here, we keep its and components unchanged, i.e., and . Similarly to the inverse conversion of the processed right image, we denote the , , and as the , , and components of the processed right image in CIE XYZ color space, which can be calculated by
where is the reference white point relative to in CIE XYZ color space, which can be computed by
Then, utilize the matrix which is the inverse matrix of to convert the , , and to RGB color space by the following formula:
At last, we need combine the processed left and right values together, using the values of and as the and channels of the final anaglyph and the as the channel. Denote the , , and as the , , and values of the final anaglyph; then, this procedure can be represented by
Then, an anaglyph is performed when they scaled by multiplying 255. The above operations are performed on each pixel of the anaglyph, and finally, the anaglyphs are obtained which is shown in Figure 8.
5. Evaluation
In this section, we conduct subjective tests to evaluate the performance of our algorithm by comparing it to the other computed methods mentioned in Section 2, i.e., Dubois’ method, McAllister’s method, and Songnan Li’s method. We do not put the Midpoint method into the comparison because it destroys the 3D perception heavily and makes no sense to compare.
5.1. The Scheme of Subjective Test
We use 15 stereo pairs with diversified colors as the input and then downscale them to a proper size while keeping their aspect ratios unchanged. For every stereo pair, we need to generate 4 anaglyphs by 4 different methods and their relative positions on the screen are shown in Figure 9. We put the input stereo pairs on the left of the screen is to make convenience for comparing the color reproduction of 4 anaglyphs simultaneously while the color of the stereo pairs is the reference standard. On the right of the screen, we set anaglyphs side-by-side to speed up the comparison processing to alleviate the visual fatigue of subjects instead of presenting each anaglyph sequentially, and this layout can achieve more intuitive comparison about the retinal rivalry of 4 anaglyphs at the same time. In addition, both the positions of the anaglyphs which are denoted as ①-④ and the orders of the 15 experimental images are set casually to avoid the subjects’ inertial choice and evaluation.
This experiment is conducted in a room with fluorescent lights. We choose the 20 HP LCD display as the displaying devices and Bertha red-cyan glass as the colored-filter glasses in the subjective test. Viewing distance is about 3 times the screen height, and the time of the subjective tests with 15 stereo pairs is controlled in 10 minutes to avoid fatigue test. For the 10 subjects who have normal vision and have passed the color blindness test, a training process can be implemented to familiarize the subjects with the anaglyph 3D viewing. Then, they need to complete two tasks when the 15 experimental images displayed in sequence on the screen. One task is to give the assessment of color fidelity which is divided into three grades, i.e., 3: no distortion, 2: a little distortion, and 1: severe distortion. Another task is to give the visual quality which can assess the anaglyph fully, and it also can be divided into 4 grades, i.e., 4: comfortable (with no visual fatigue), 3: fair, 2: a little uncomfortable, and 1: uncomfortable (with heavy visual fatigue). Of course, in order to get a better feedback from subjects, we can encourage these subjects to express their feelings about the anaglyphs and point out the reasons of their assessment, and another person will record the corresponding grades and evaluations meanwhile.
5.2. Evaluating Result
Through the subjective tests described above, we can get the evaluating result which is shown in Table 1. The data in this table represents how many subjects choose the corresponding grade by conducting the subjective test. The data indicates that our method does have a better color reproduction compared to Dubois’ method and Songnan Li’s method. And in terms of the visual quality, our method is no less than the other three methods. Of course, the result may be a little different by different subjects, but it also is able to explain that our method is effective especially in the excellent balance between the color distortion and retinal rivalry to some extent.
6. Conclusion
With the development of stereoscopic display technology, VR is becoming more and more mature. Anaglyphs have some irreplaceable values including high data compressibility, low cost of use, and input sources for all display devices. In this paper, we have concluded the predecessors’ methods for generating anaglyphs at first. Then, we proposed a new algorithm for generating anaglyphs based on matching color attributes especially the hue in CIE color space, which can achieve a better color reproduction and can dynamically adjust the balance between retinal rivalry and color distortion. In order to compare the results of different methods, we built a scheme for subjective test, which turns out that our method does demonstrate a good performance to some extent. It is also worth mentioning that the process of matching color attributes is not the only solution to improve the stereoscopic visual effect of anaglyphs. Some literatures have been pointed out that gamma correction and ghosting phenomenon elimination also can be used to improve the stereoscopic effect [12, 13], but the specific reasons and parameters need to be researched further.
Data Availability
Data sharing is not applicable to this article as no datasets were generated or analysed during the current study.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
Acknowledgments
The authors acknowledge the funding of the Key Project of Shaanxi Province Innovation Program, China (2017ZDCXL-GY-11-02-02).