Abstract
An undecimated dual tree complex wavelet transform (UDTCWT) based fusion scheme for astronomical visible/IR images is developed. The UDTCWT reduces noise effects and improves object classification due to its inherited shift invariance property. Local standard deviation and distance transforms are used to extract useful information (especially small objects). Simulation results compared with the state-of-the-art fusion techniques illustrate the superiority of proposed scheme in terms of accuracy for most of the cases.
1. Introduction
The visible light astronomy due to reflection, refraction, interference, and diffraction enables scientists to unearth many of nature’s secrets; however, brightness of stars creates a haze in the sky. On the other hand, the infrared (IR) astronomy enables us to peer through the veil of interstellar dust and see objects at extreme cosmological distances. IR images have good radiometric resolution whereas visible images provide detailed information. In this regard, various image fusion techniques have been developed to combine the complementary information present in both images. These techniques can be grouped into wavelet, statistical decomposition, and compressive sensing.
The wavelet transform based fusion schemes generally decompose the visible and IR images into different base and detail layers, to combine the useful information. In [1], contourlet transform fusion is used to separate foreground and background information; however, the separation is not always accurate, which causes loss in target information. In [2], nonsubsampled contourlet transform, local energy, and fuzzy logic based fusion claims better subjective visual effects; however, merger and description of necessary components of IR and visible images in fusion model require improvements especially in case of noisy images. In [3], wavelet transform and fuzzy logic based scheme utilizes dissimilarity measure to assign weights; however, some artifacts are also introduced in the fused image. Contrast enhancement (using ratio of local and global divergence of IR image) based fusion lacks color consistency [4]. In adaptive intensity hue saturation method [5], the amount of spatial details injected into each band of multispectral image is appropriately determined by the weighting matrix, which is defined on the basis of the edges present in panchromatic and multispectral bands. The scheme preserves the spatial details; however, it is unable to control the spectral distortion sufficiently [6]. In [7], gradient-domain approach based on mapping contrast defines the structure tensor matrix onto a low-dimensional gradient field. However, the scheme effects the natural output colours. In [8], wavelet transform and segmentation based fusion scheme is developed to enhance targets in low contrast. However, the fusion performance is dependent on segmentation quality and large segmentation errors can occur for cosmological images (especially when one feature is split into multiple regions).
Statistical fusion schemes split the images into multiple subspaces using different matrix decomposition techniques. -means and singular value decomposition based scheme suffers from computational complexity [9]. In [10], spatial and spectral fusion model uses sparse matrix factorization to fuse images with different spatial and spectral properties. The scheme combines the spectral information from sensors having low spatial but high spectral resolution with the spatial information from sensors having high spatial but low spectral resolution. Although the scheme produces better fused results with well preserved spectral and spatial properties, its issues include spectral dictionary learning process and computational complexity. In [11], an internal generative mechanism based fusion algorithm first decomposes source image into a coarse layer and a detail layer by simulating the mechanism of human visual system for perceiving images. Then the detail layer is fused using pulse coupled neural network, and the coarse layer is fused by using the spectral residual based saliency method. The scheme is time inefficient and yields weak fusion performance. In [12], independent components analysis based IR and visible image fusion scheme uses kurtosis information of the independent components analysis based coefficients. However, further work is required for determining fusion rules of primary features.
Compressive sensing based fusion schemes exploit the sparsity of data using different dictionaries. Adjustable compressive measurement based fusion scheme suffers from empirical adjustment of different parameters [13]. In [14], a compressive sensing approach preserves data (such as edges, lines, and contours); however, design of appropriate sparse transform and optimal deterministic measurement matrix is an issue. In [15], a compressive sensing based image fusion scheme (for infrared and visible images) first compresses the sensing data by random projection and then obtains sparse coefficients on compressed samples by sparse representation. The fusion coefficients are finally combined with the fusion impact factor and the fused image is reconstructed from the combined sparse coefficients. However, the scheme is inefficient and prone to noise effects. In [16], a nonnegative sparse representation based scheme is used to extract the features of source images. Some methods are developed to detect the salient features (which include the target and contours) in the IR image and texture features in visible image. Although the scheme performs better for noisy images, the sparseness of the image is controlled implicitly.
In a nutshell, the above-mentioned state-of-the-art fusion techniques suffer from limited accuracy, high computational complexity, or nonrobustness. To overcome these issues, a UDTCWT based visible/IR image fusion scheme for astronomical images is developed. The UDTCWT reduces noise effects and improves object classification due to its inherited shift invariance property. Local standard deviation along with distance transforms is used to extract useful information (especially small objects). Simulation results illustrate the superiority of proposed scheme in terms of accuracy, for most of the cases.
2. Proposed Method
Let be the input source IR () and visible () registered images (with dimensions ). The local standard deviation for estimating local variations of is where is local mean image computed as The local standard deviation measures the randomness of pixels in a local area where the high values indicate presence of astrobodies and the low value values correspond to smooth/blank space (without any object or astrobody).
The image is obtained by thresholding to remove the pixel containing large variations: that is, where is a controlling parameter and and are mean and variance of , respectively. The gray distance image (to classify different points which are present inside/outside any shape/object) is computed using and mask as The distance transform (used to eliminate oversegmentation and short sightedness) measures the overall distance of the pixel from other bright pixels. For instance, a pixel closer to a cluster of stars (objects) tends to be part of the segmented mask and vice versa.
Let be the binary image obtained from the distance image : that is, where denotes mean image and is a positive constant. The image segments the foreground from background regions. The connected components image (to segment different binary patterns) with structure element (a matrix of all ones) is Let and represent area and perimeter of the th connected component placed at th location, respectively; a binary segmented image is constructed as where and are thresholding parameters. UDTCWT is applied on the source images to obtain coefficient matrix of dimensions (where represents wavelet coefficients). The decomposition obtained using UDTCWT not only eliminates noise/unwanted artifacts, but also is effective in preserving the useful information present in the input images (due to its undecimated property). The binary coefficient matrix is obtained by assigning nonzero values at pixel locations where visible image provides more information than the IR image. This binary thresholding ensures that the fused image contains the significant/important information of both source images (as the higher value of UDTCWT corresponds to presence of significant/important information): A binary fuse map is computed as where represents operation. LetThe final fused image is obtained by computing the inverse UDTCWT of fused coefficients . Figure 1 shows the flow diagram of proposed technique.
3. Results and Discussion
To verify the significance of the proposed technique, simulations are performed on various visible/IR datasets. Quantitative analysis is performed using (luminance/contrast distortion), (mutual information), (weighted quality index), (edge dependent quality index), (structural similarity index measure), (human perception inspired metric), (edge transform metric), and (image feature metric) [17–22].
The metric [17, 18] is designed through modality image distortion as combination of loss of correlation, luminance distortion, and contrast distortion. The metric [17] represents the orientation preservation and edge strength values. It models the perceptual loss of information in fused results in terms of how well the strength and orientation values of pixels in source images are represented in fused image. It deals with the problem of objective evaluation of dynamic, multisensor image fusion, based on gradient information preservation between the inputs and the fused images. It also takes into account additional scene and object motion information present in multisensor sequences. The metric [17] is defined by assigning more weight to those windows, where saliency of the input image is high. It corresponds to the areas that are likely to be perceptually important parts of the underlying scene. The index [17] takes into account aspects of the human visual system, where it expresses the contribution of the edge information of the source images to the fused images. The measure [19] is the similarity between two images and is designed to improve traditional measures of mean square error and peak signal to noise ratio, which are inconsistent with human eye perception. The metric [20] evaluates the performance of image fusion for night vision applications, using a perceptual quality evaluation method based on human visual system models. Image quality of fused image is assessed by contrast sensitivity function and contrast preservation map. The metric [21] assesses the pixel-level fusion performance and reflects the quality of visual information obtained from the fusion of input images. The metric [22] evaluates the performance of the combinative pixel-level image fusion, based on an image feature measurement (i.e., phase congruency and its moments), and provides an absolute measurement of image features. By comparing the local cross-correlation of corresponding feature maps of input images and fused output, the quality of the fused result is assessed without a reference image.
These quality metrics [17–22] work well for noisy, blurred, and distorted images, using multiscale transformation, arithmetic, statistical, and compressive sensing based schemes for multiexposure, multiresolution, and multimodal environments. These are also useful for remote and airborne sensing, military, and industrial engineering related applications. The normalized range of these measures is between 0 and 1 where high values imply better fusion metric for each quality measure.
Figure 2(a) shows Andromeda galaxy (M31) JPEG IR image taken by Spitzer space telescope [23] while Figure 2(b) shows the corresponding visible image taken using 12.5′′ Ritchey Chretien Cassegrain (at F6) and ST10XME [24]. Figures 2(c)–2(f) are the outputs of local variance, distance transform, and segmentation steps.
(a)
(b)
(c)
(d)
(e)
(f)
Figure 3 shows the fusion results obtained by ratio pyramid (RP) [25], dual tree complex wavelet transform (DTCWT) [26], nonsubsampled contourlet transform (NSCT) [27], multiresolution singular value decomposition (MSVD) [28], Ellmauthaler et al. [8], and proposed schemes. By visual comparison, it can be noticed that the proposed scheme provides better fusion results, especially the preservation of background intensity value as compared to existing state-of-the-art schemes.
(a)
(b)
(c)
(d)
(e)
(f)
Figures 4(a) and 4(b) show visible and IR Jupiter’s moon JPEG images taken by new Horizons spacecraft using multispectral visible imaging camera and linear Etalon imaging spectral array [29]. The fusion results obtained by RP [25], DTCWT [26], NSCT [27], MSVD [28], Ellmauthaler et al. [8], and proposed schemes are shown in Figures 4(c)–4(h), respectively. Note that only the proposed scheme is able to accurately preserve both the moon texture (from IR image) and other stars (from visible image) in the fused image.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
Figures 5(a) and 5(b) show visible and IR Nabula (M16) JPEG images taken by Hubble space telescope [30]. The fusion results obtained by RP [25], DTCWT [26], NSCT [27], MSVD [28], Ellmauthaler et al. [8], and proposed schemes are shown in Figures 5(c)–5(h), respectively. The fused image using proposed scheme highlights the IR information more accurately as compared to existing state-of-the-art schemes.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
Table 1 shows the quantitative comparison of existing and proposed schemes (where the bold values indicate best results). It can be observed that the results obtained using proposed schemes are significantly better in most of the cases/measures as compared to existing state-of-the-art schemes.
4. Conclusion
A fusion scheme for astronomical visible/IR images based on UDTCWT, local standard deviation, and distance transform is proposed. The use of UDTCWT is helpful in retaining useful details of the image. The local standard deviation variation measures presence or absence of small objects. The distance transform activates the effects of proximity in the segmentation process and eliminates effects of oversegmentation in addition to short sightedness. The scheme reduces noise artifacts and efficiently extracts the useful information (especially small objects). Simulation results on different visible/IR images verify the effectiveness of proposed scheme.
Conflict of Interests
The authors declare that there is no conflict of interests regarding the publication of this paper.