Abstract
Under foggy and other severe weather conditions, image acquisition equipment is not effective. It often produces an image with low contrast and low scene brightness, which is difficult to use in other image-based applications. The dark channel prior dehazing algorithm will cause the brightness of the image to decrease and sometimes introduce halos in the sky area. To solve this problem, we proposed a region similarity optimisation algorithm based on a dark channel prior. First, a vector comprising RGB layer dark channel value was obtained as the original atmospheric ambient light, and then, the proposed regional similarity linear function was used to adjust the atmospheric ambient light matrix. Next, the transmittance of different colour channels was derived and the multichannel soft matting algorithm was employed to produce more effective transmittance. Finally, the atmospheric ambient light and transmittance were substituted into the atmospheric scattering model to calculate clean images. Experimental results show that the proposed algorithm outperformed the existing mainstream dehazing algorithms in terms of both visual judgement and quality analysis with nonhomogeneous haze datasets. The algorithm not only improves the image details but also improves the brightness and saturation of the dehazing result; therefore, the proposed algorithm is effective in the restoration of the hazy image.
1. Introduction
Fog is a near-surface atmospheric weather phenomenon caused by the desublimation of suspended water droplets in the air. Foggy conditions cause considerable inconvenience to human life and production, especially in imaging. Because of the light scattering caused by various particles in a foggy atmosphere, such as PM2.5, image acquisition equipment tends to produce images with fewer details, low contrast, low scene brightness, and whitish hue, which not only renders a poor intuitive visual experience but also affects the normal operation of the work system in image-related fields, such as image recognition. Thus, research on haze removal technology is valuable and significant.
At present, image dehazing technology is mainly divided into three categories [1–3]: image enhancement methods, physical model-based methods, and deep learning-based methods using neural networks. Among them, representative image enhancement algorithms [4–9] include histogram equalisation, homomorphic filtering, wavelet transform, and MSR algorithm based on the Retinex (SSR) theory.
The histogram equalisation algorithm nonlinearly stretches and rearranges the dense part of the whole grey histogram to achieve approximately uniform distribution, thus enhancing the local image contrast. The Retinex model proposed by Edwin Land was based on the colour constancy theory [4]. Based on the theory of the single-scale SSR algorithm, Ma and Wen [5] combined edge detection within a Gaussian filtering algorithm to amend the estimated value of the reflection component and mitigate the loss of image edge information. Zheng et al. [6] proposed an adaptive structure decomposition-integrated multiexposure image fusion dehazing method. It linearly adjusts the underexposed images through gamma correction to enhance the visual effect after dehazing, but the calculation complexity is high. Zhu et al. [7] proposed an image fusion-based algorithm that reduces haze effects via gamma correction. This algorithm can enhance the saturation of images and can be applied to geographic remote sensing and underwater images. Shu et al. [8] proposed a hybrid regularised variational framework to improve the scene depth of images along with dehazing. In addition, a two-step correction mechanism is employed to address the problem of colour distortion in the sky area.
Physical model-based methods use the traditional atmospheric scattering model [10] as the main research object and then estimate atmospheric ambient light and transmittance via a priori hypothesis to make them similar to the real scene, thus realising image defogging processing [11–24]. Kaiming He et al. [13] proposed a single-image dehazing algorithm using the dark channel prior theory. This method adopts soft matting method to refine the transmittance and has a remarkable effect. Later, He et al. [14] substituted guided filtering for soft matting in transmittance refinement to improve the operating efficiency, but phenomena such as overall bluish and caliginous hue and colour distortion in the sky area still exist. Zhu et al.[15] obtained the mapping relation by training a linear model built from a colour attenuation prior. This algorithm achieves less colour distortion and outperforms most traditional algorithms. Meng et al. [16] employed a regularisation algorithm to effectively constrain the boundary of the sky region to obtain a restored image. This method causes image detail loss and artifacts in the image boundary. Sulami et al.[17] assumed that the scene chrominance was uncorrelated with the transmittance and calculated the transmittance using statistical laws, resulting in a poor effect in images with relatively single colour features. Berman et al. [18] proposed a nonlocal prior within each type of pixel cluster; the pixel points will have a linear relationship according to the difference in fog concentration. They estimated the scene depth based on this prior and accordingly restored the fog-free image. Ju et al. [19] introduced a novel light absorption coefficient parameter to attain an enhanced ASM (EASM). This model increases the visibility of hazy images while dehazing outdoor hazy images. Later, Ju et al. [20] proposed an image dehazing method using blended priors. This algorithm combines multiple modules, including the atmospheric light estimation module to increase brightness and multiple prior constraint modules to create a nonlocal prior, local prior, and global prior, which improve the brightness and saturation of images. Wang et al. [21] proposed an additional channel method based on quad-tree subdivision and refined the transmittance using Gaussian blurring. These measures prevent the phenomena of oversaturation and halo effects from happening and tackling the problem of distortion in the brightest areas.
In recent years, machine learning has become an important topic for its wide application and methods based on deep learning and neural networks have also made progress in image defogging [25–34]. Chen et al. [25] proposed a single-image dehazing method based on CNN, obtained the detailed features of foggy images via convolution layer operation and multiscale convolution kernel operation, and then fitted the transmittance map using nonlinear regression. In contrast to CNN, Cai et al. [26] proposed a trainable end-to-end transmission medium estimation model called DehazeNet, which directly learns and estimates the mapping relationship between foggy images and transmittance. However, this algorithm still uses atmospheric light as a global constant, leading to a loss in the precision of the image to a certain degree. Li et al. [27] improved the traditional atmospheric scattering model and used a convolutional neural network to estimate the residual image containing atmospheric light and transmittance. Qin et al. [28] proposed an end-to-end feature fusion dehazing network. This network structure can adaptively learn different weights of different-level feature information. The results show significant dominance in image detail restoration and colour fidelity. Liu et al. [29] designed and enhanced the CNN learning method for ship detection, which incorporated nonmaximum suppression to accurately obtain the results and construct a mixed loss function balance class during the training phase. Like the relative fine dehazing results as those aforementioned networks have, the deep learning-related algorithm relies considerably on big data; this is difficult to implement under sparse samples.
Algorithms, such as the dark channel a priori based on the traditional atmospheric scattering model, ignore most of the image detail information when estimating the atmospheric ambient light. To solve this problem, we made the following modifications based on the dark channel a priori theory: (1) The dark channel values were obtained for each of the three RGB colour channels of the hazy image, and the regional mean value of each pixel in each dark channel was considered as the corresponding atmospheric ambient light value, after which the entire atmospheric ambient light was linearly transformed with its top 1% value to obtain the final atmospheric ambient light value. This reduces the influence of image noise to a certain extent. (2) In this study, we proposed to refine the transmittance through guided filtering and selected the V channel in the HSV with the best effect as the guided map for transmittance optimisation through experiments to obtain a transmittance closer to the real scene. In this study, we ran experiments on the O-Haze dataset, NTIRE 2021 dataset, and real images and analysed the results from both subjective and objective aspects. The results show that the proposed algorithm effectively improves the brightness and saturation of the image and the haze-free image after dehazing has high colour contrast and good visual effect and also improves the problem of blue and dark images after the dark channel a priori algorithm processing.
2. Background
Under foggy and other severe weather conditions, the light will scatter when it meets suspended small particles in the air during propagation, resulting in degraded images produced by image acquisition devices. Nayer and Narasimhan improved the atmospheric scattering model [10] in 1999, and this model (Figure 1) is widely used in computer vision and computer graphics.
It can be formulated aswhere refers to the hazy image received by imaging equipment, is the real-scene radiation, is the atmospheric ambient light value, and is the transmittance, which can be expressed aswhere is the atmospheric scattering coefficient and is the distance between the actual scene and the photographed point, that is, the depth of the field.
As per the formula, can be obtained via formation simplification after estimating atmospheric ambient light and transmittance of input .
Kaiming He et al. [13] conducted experiments on large quantities of outdoor haze-free images and found that in the vast majority of nonsky regions in haze-free images, some pixels in a certain region have at least one colour channel with a rather low value. In dark channels, they pointed out that dark channel values tend to zero in nonsky areas. The dark channel value in a local patch , with the centre pixel x, is denoted by as follows:
In practice, small particles inevitably exist in the atmosphere. The presence of haze is fundamental for observers to perceive scene depth. Therefore, after minimisation on both sides of equation (3), the parameter (0 < ≤ 1) is introduced to maintain a certain amount of haze, thus enhancing the visual experience of observers, as shown inwhere represents the RGB colour channels, , is the hazy image of each colour channel layer, and is the atmospheric ambient light of each colour channel layer.
The dark channel prior is subject to limitations in that it does not hold in the sky area and white object area; therefore, results with images containing such areas will display local image distortion and a widespread halo in the sky area.
3. Proposed Algorithm
Due to the unrobustness of the dark channel prior, its dehazing results are subject to problems such as bluish hue and colour distortion in the sky area. The proposed algorithm refines the estimation of the atmospheric ambient light and transmittance based on the original dark channel prior algorithm through the region similarity correction, thus obtaining more effective estimate values, which significantly improves the dehazing effect. The flowchart of the proposed algorithm is shown in Figure 2.
As shown in Figure 2, (a) is the input hazy image, we first select the low brightness pixels in a moving windows from (a) as the dark channel image (b). Then, we use the region similarity correction method to estimate the enhanced atmospheric light image (c). By using the dark channel prior, the rough transmittance map can be calculated as (d). In addition, the input image are transferred to HSV domain (e) to get the intensify map (f). The detailed transmittance map (g) is generated by a guide filter which employed the intensity map (f) as the guide map. Finally, the enhanced atmospheric light (e) and detailed transmittance (g) are submitted to the atmospheric scattering model to get the dehazing output (h).
3.1. Estimating the Dark Channel Value
For a hazy image with a size of , we designed a window centered around it and chose the average value of the lowest grey value for each pixel in the R, G, and B colour channels, respectively. Furthermore, within the range of , because of the dark channel value of this pixel, we obtain an RGB dark channel matrix (equation (6)):
The size of has an impact on the dark channel image; the larger its value, the darker the pixel points it contains and the darker the generated image; the experimental effect will not be suitable. The smaller the value, the less the information of the dark channel included, and the difference with the original image is not significant. Therefore, s = 31 was chosen in this study to achieve the best balance.
We further compute the atmospheric ambient light using it.
3.2. Estimating the Atmospheric Ambient Light
To better realise region selection, for each pixel in the obtained as the starting point, we determined a square window from four different directions, including the upper left, upper right, lower left, and lower right of the pixel independently to calculate the mean dark channel value from within as the atmospheric light of this pixel and experiment with their dehazing effect. The results show that the lower right direction achieves the best; therefore, we chose this direction as the final method, as shown inwhere represents the dark channel value of the pixel with coordinates , , is the length of the neighbourhood window, and are the abscissa and ordinate, that is, . Provided that the size of the square window size at the image edge is less than , then we select the window with side length .
Larger atmospheric ambient light values will make the image brighter, while too small values will result in a darker image overall and affect the accuracy of the estimated transmittance . This is particularly obvious in the dense haze region. Therefore, in this study, we proposed an improvement scheme, in which the obtained values are first arranged in a descending order and the first 1% of them are considered as the ideal values of the atmospheric ambient light to ensure that the atmospheric ambient light is more consistent with the real scene as shown inwhere the positive integer refers to the corresponding value of the pixel of the whole image (rounded down).
Refactoring the vector rearranged from as , then the pixel with coordinates can be formulated as follows:where is the subscript position of the value with the original coordinates . Subsequently, we rearrange in descending order and denote the new column vector as .
Next, we rearrange the entire atmospheric light matrix according to their linear proportion. Then, we mark the minimum atmospheric light value as and the maximum value of the optimised value as . The minimum value of the optimised atmospheric light is equal to . The equation is
By simplifying (11), we can obtain the rearranged value of each pixel from the original , which leads to
The calculation and rearrangement steps of the atmospheric light are shown in Figure 3.
As shown in Figure 3, a dark channel image (a) was input to obtain the refined atmospheric ambient light image. Considering the pixel point at coordinate (1, 1) as an example, Ω(X) is selected down to the right (as shown in the red box in (a)) and the mean value of the region Ω(X) (i.e., (b)) is calculated as the rough atmospheric ambient light value at coordinate (1, 1) (as shown in the red box in the upper left of (c)). Thereafter, we arrange the atmospheric ambient light value of (c) into the right matrix of (d) in descending order and linearly stretch the first 1% of the pixel points into the new atmospheric ambient light values (as shown in the left matrix of (d)), which is later mapped back to the original coordinate positions for subsequent dehazing (as shown in the change process of the red pixel points from (c) to (e)).
3.3. Estimating the Transmittance
Based on the principle that visible light of different wavelengths has different transmittances, transmittances of different colour channels are obtained in this study, as shown inwhere represents the three-layer colour channel transmittance matrix.
Experimental analysis indicates that V channel overperforms other choices as the guiding map to employ guided filtering to transmittance , and we obtain a more effective transmittance , as the following formulas show:where is the guiding image, that is, the V channel of the hazy image represented in the HSV image system, is the covariance of the window , and and are the variance and mean value of in window , respectively.
The parameter also plays a decisive role in the transmittance; the smaller its value, the less obvious the defogging effect. After extensive experiments, we fixed to 0.9 and the minimum transmittance was set as 0.1 to avoid excessive enhancement of the image after fog removal.
A comparison of the transmittance with and without the guided filter is shown in Figure 4.
(a)
(b)
(c)
Thereafter, we submit the improved atmospheric ambient light and transmittance into equation (1) to obtain clean images.
4. Experimental Results
To evaluate the feasibility and effectiveness of our algorithm more intuitively, we performed comprehensive experiments on a large dataset containing the public O-Haze image dataset, NTIRE 2021 dataset, and real-world images. The O-Haze dataset contains 45 outdoor scene images with the same visual content recorded under fog-free and foggy conditions, from which we stochastically selected six images as the testing set for neural network algorithms and used the others as the training set. The real-world image dataset comprises approximately 150 images, including a public dataset that is available online and self-collected images. The NTIRE 2021 dataset includes 35 pairs of real and outdoor nonhomogeneous hazy images. All operations were performed under the Windows 10 operating system, MATLAB R2020a operating environment, and Intel(R) Core (TM) i7-7500U CPU @2.70 GHz 2.90 GHz processor, with 8 GB RAM hardware configuration. We compared our performance to the state-of-the-art algorithms including DCP [13], Meng et al.’s [16], Sulami et al.’s [17], Berman et al.’s [18], IDE [19], DehazeNet [26], and AOD [30] from both subjective and objective aspects. According to the experimental analysis, we set the window size as in this study.
Image evaluation has extremely important application value in image dehazing, image compression, video compression, and other fields. Image evaluation methods are mainly divided into subjective evaluation and objective evaluation. Among them, subjective evaluation refers to the perceptual judgement based on observers’ subjective feelings and is widely used in neural networks, image recognition, and other fields; however, it is susceptible to many capricious factors, such as personal preference, and is relatively ex parte.
To solve the problem of differentiation caused by subjective factors, it is necessary to formulate a standard for image quality evaluation (QA), that is, to use a specific mathematical model to quantify the difference between each image, to objectively evaluate the image quality.
We adopted the image visibility evaluation criterion and image structure similarity evaluation criterion using the peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and natural image quality evaluator (NIQE) evaluation indicators to conduct a comprehensive evaluation and analysis of the aforementioned eight main dehazing algorithms. The PSNR uses the mean square error (MSE) to calculate the different values of the image pixels. The larger the value, the better the quality of the restored image. The calculation formula is shown in
Its MSE is calculated usingwhere and refer to a hazy image of size and a processed image, respectively. The pixel value is represented by a B-bit binary number. In this study, the image pixel domain is [0, 255]; therefore, = 8. The MSE represents the comparison result of the pixel-by-pixel difference between two images.
The structural similarity formula is implemented via brightness comparison formula (18), contrast comparison formula (19), and structure comparison formula (20):where and represent the mean values of and are the variances of , respectively. is the covariance of ; and are constants with default values to avoid the case where the denominator is zero during the calculation. Equation (21) shows how the SSIM operates:
Considering , the calculation formula of SSIM can be obtained as
has symmetry, which implies that is equivalent to . The domain of is [0, 1]. The larger the value, the smaller the difference between the dehaze image and the original image. Images with large values show better integrity and a more reasonable structure. The evaluation criterion can also be used as an evaluation standard for the SR technology.
The NIQE algorithm evaluates the image quality by calculating the distance between the multivariate Gaussian (MGV) feature parameters extracted from the dehazing images and the pretrained model parameters, as shown inwhere and are the mean vectors of the natural MVG model and are the covariance matrices of the distorted image’s MVG model.
The characteristic of NIQE is that no haze-free image is required as a reference. The smaller the value is, the smaller the gap between the haze-free image and the pretrained model is and the higher the image quality is. Therefore, this paper takes it as the evaluation index of real-world images.
4.1. Experiment on the O-Haze Dataset and NTIRE Dataset
Substantial experiments were performed on the entire O-Haze and NTIRE datasets. The O-Haze dataset contains 45 paired hazy and haze-free images, whose hazy conditions are manufactured by a professional haze-making machine. The NTIRE dataset includes 35 pairs of outdoor nonhomogeneous hazy images. Part of the defogging results (with a size of 512 × 512) is shown in Figure 5 to facilitate comparison and analysis.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)
(j)
For the first image in the O-Haze dataset, the results of the DCP, Fattal, and AOD methods present an overall bluish hue; the Berman algorithm has a better foreground effect, but the distant details are fade, while our method, IDE, and the Meng algorithm retain clear distant details and show the best restoration effect; the IDE algorithm significantly increases the brightness. For the 11th image in the O-Haze dataset, all the algorithms achieved an insignificant dehazing effect on distant trees but our method outperformed the others on colour restoration in close-range processing. For the 12th image in the O-Haze dataset, our method achieved the most significant dehazing effect and the result is close to the real scene, whereas the results of the Fattal and AOD methods are overall bluish and DehazeNet has a poor dehazing effect. For the 31st image in the O-Haze dataset, our algorithm has no obvious effect on the processing of the branches in the upper left corner, the overall colour of the Fattal algorithm is yellow, and the image of the Berman algorithm is distorted. For the 33rd image in the O-Haze dataset, all traditional algorithms have bright strips on the left side of the image and the effects of the two neural network algorithms are not obvious. Our algorithm and IDE algorithm achieved the best effect; for the 41st picture in the O-Haze dataset, our algorithm and Berman’s algorithm are better than other algorithms in processing the sky area. For the 2nd image in the NTIRE dataset, our algorithm has the most thorough dehazing effect, the Berman algorithm has severe colour distortion, and the results of all other algorithms are bluish. For the 7th image in the NTIRE dataset, all algorithms have a poor dehazing effect owing to the heavy fog concentration, especially in the middle area above the image, but our algorithm and IDE algorithm have a better dehazing effect than other algorithms in the near field. For the 9th image in the NTIRE dataset, the DehazeNet algorithm does not have a significant dehazing effect. Our algorithm has the best defogging, but the hue is darker, and the other algorithms have different degrees of bluishness. In summary, our algorithm has certain advantages in terms of the subjective dehazing effect.
The results of the objective evaluation indexes PSNR and SSIM are listed in Table 1. As can be observed from the table, our algorithm has certain advantages over the other seven methods in both evaluation indexes. Most SSIM of our algorithm is optimal in most images in the O-Haze dataset and suboptimal in the NTIRE dataset, indicating that the gap between the defogging image and the original image is small and the structural integrity is strong. The DCP algorithm has a higher value when processing the 33rd image in the O-Haze dataset; however, from the perspective of the subjective defogging effect of the image, the brightness of the defogging image of DCP and the other four algorithms is low, whereas our algorithm significantly improves the overall brightness of the image, solves the problem of the bluish hue of dark channel dehazing, and is more in line with the characteristics of the real fog-free image. Although in the seventh image picture in the NTIRE dataset, the PSNR and SSIM of our algorithm are not the highest, the subjective feeling effect is better than that of other algorithms, which also shows that there is still room for improvement where the haze is particularly thick and efforts are required.
4.2. Experiment on Real-World Images
To further verify the effectiveness of the algorithm in this study, real-scene images were selected for the experiment. The results are shown in Figure 6. The size of the foggy image was 512 × 512. As shown in Figure 6, when processing the first and third images, the Meng, Fattal, and Berman algorithms exhibit serious colour distortion problems in the sky area, while the algorithm in this study has an ideal processing effect. For the second image, the detailed processing of the Fattal algorithm in the building part is not obvious, and the sky area shows a dark hue. All the algorithms have no obvious effect on the sky region of the third image, and our algorithm and Fattal algorithm have an ideal effect in close view. For the fourth image, the colour of the Berman algorithm turns green, while AOD’s processing and DehazeNet’s processing of distant buildings lose details. The IDE algorithm can significantly improve brightness when processing real-world images. Compared with the DCP algorithm, the algorithm in this study is more natural in image defogging and has a higher colour contrast, which can realise image defogging technology in a variety of different scenes.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)
The NIQE values are shown in Table 2. It can be seen that in the images of road, urban area, and urban rail, IDE algorithm obtains low parameter values due to overexposure of the images, but obviously, it cannot be applied to all the images. As for image parameters of urban area images, IDE obtains the highest parameter values among the 8 algorithms. The proposed algorithm is more consistent with people’s subjective feelings in the road image, and the detail processing of the distant houses in the city rail image are better than the IDE algorithm. The proposed algorithm obtains relatively low values in the processing of urban images and is second only to the DehazeNet algorithm in the housing images. The reason is that there exits colour distortion in the upper left corner of the image. But, the DehazeNet algorithm has a poor haze-removing effect in terms of subjective perception. Comprehensively, we believe that our algorithm preserves more original colour information in the image dehazing process and the effect is better.
From the above experimental comparison, it can be observed that the proposed algorithm can achieve a finer effect in the processing of image details and the resulting images are more realistic, providing people with a better visual experience. The parameters PSNR and SSIM are higher than those of the other algorithms in most cases. However, the algorithm proposed in this study has a poor effect in thick haze areas. This is because, under the influence of large white areas, the significance of the regional similarity value is lost, which leads to excessive brightness exposure in such regions after dehazing, resulting in regional white blocks.
5. Conclusion
In this paper, a dehazing algorithm for heterogeneous images based on the optimisation of regional atmospheric ambient light was proposed to solve the problem of image colour distortion and the overall bluish and somber hue, which exists in the dark channel prior algorithm. In this method, the regional mean value of the RGB three-layer dark channel in a foggy image was solved and the corresponding regional atmospheric ambient light value was obtained by linear stretch correction. Subsequently, according to the transmittance obtained by the dark channel prior theory, the V channel of the foggy image is considered as the guiding graph for guiding filtering according to different channels to approach the real image transmittance and the atmospheric scattering model is used for defogging processing. The experimental results show that the image dehazing optimised by the proposed algorithm is superior to most classical dehazing algorithms in both subjective vision and objective quantification indexes and the problem of the dark hue is well solved. However, the algorithm in this study should traverse the entire image when calculating the atmospheric ambient light value, leading to a long execution time of the algorithm. To achieve better defogging efficiency, this study will continue to explore a better solution.
Data Availability
The research data used to support the findings of this study are available from the corresponding author only under license and so cannot be made freely available. Requests for access to these data should be made to the corresponding author.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
Acknowledgments
This work was mainly supported by the Beijing Municipal Education Commission Research Foundation under Grant KM202111417008, the Academic Research Projects of Beijing Union University under Nos. ZK90202104 and ZB10202004, and the Beijing Municipal Natural Science Foundation under Grant L191006 and partly supported by Beijing Union University Students’ Science and Technology Innovation Project (no. 20213065).