Abstract
The blind image inpainting problem need to be handle when faced with a large number of images, especially medical images in medical health. For the proposed nonconvex sparse optimization model, a proximal based alternating direction method of multipliers (PADMM) method is designed to solve the problem. Firstly, sparse regularization is imposed to the binary mask since the missing pixels are sparse in our experiments. Secondly, the total variation term is utilized to describe the underlying clean image. Finally, regularization of the fidelity term is used to solve the given blind inpainting problem. Experiments show that this method has better performance than traditional method, and could deal with the blind image inpainting problem.
1. Introduction
Medical images can directly reflect the function and health status of human tissues, and have become one of the standards of diagnosis and medical intervention. With the increasing availability and utilization of modern medical imaging such as disease database, X-ray film and magnetic resonance imaging, the demand for automatic processing of medical image data is increasing. With the application of medical images, automatic medical image analysis has become one of the hot directions of contemporary medical imaging research [1, 2]. In the process of image capture, imaging sensors broken or the error in information transmission may cause some pixels missing or corrupted by impulse noise [3–8].
In this work, the image is defined as a vector with n pixels in lexicographic ordering, . It can be represented mathematically as follows: where is the observed image, is the clean image and is an identity matrix. If the clean image x is corrupted by the additive Gaussian noise , the model will be updated as follows:
The image is corrupted by impulse noise, and the pixels in y is corrupted by the impulse noise . The blind image inpainting model with the mixture noise (Gaussian and impulse noise) is finally described as follows [9]:
How to solve the E.q. (2) efficiently and effectively is the most important issue. It is clear that E.q. (2) is quite challenging. It has three unknown term A, and . The goal of this paper is to estimate the clean image x from the partial observation y without unknow mask A, noise and .
In the earlier researches, the inpainting problem can be solved by many approaches related image reconstruction from the aspect of sparse modeling [10–13]. More common strategies for the removal of impulse noise for blind inpainting problems are to estimate an approximated A by computing the support set of the noisy pixels with some outlier detection methods, and apply the reconstruction methods for a known mask A.
The difficulty of reconstructing an image with missing data and mixture noise is basically to detect the locations of outliers. Some filtering based methods estimate the missing values, such as the adaptive median filter [14], and adaptive center weighted median filter [15].
Two phase based methods for blind inpainting problems involve the estimation of mask A, which is calculated by some outlier detection approaches. After the mask estimation, the inpainted image is generated by the image reconstruction step, which is implemented by a standard convex optimization. Some convex methods were ulitized to estimate the inpainted images [16]. Used the total variation (TV) regularizer [17] estimates the inpainted images. Xiao et al. [18] proposed a combination of -norm and -norm regularizers for simultaneously removing impulse noise and computing learning dictionary after the mask. In addition, the authors [19] presented an approach for mixed impulse and Gaussian noise removal. In the approach, a logarithmic transformation strategy is applied to convert the multiplication between the image and binary mask. Then, the image and mask terms are estimated iteratively with TV regularization applied on the image. Especially, the method can also be extended to the removal of impulse noise by relaxing the regularizer from the norm to the norm.
Some approaches could estimate the mask and impulse noise field by an iterative process instead of involving a separated mask detection step, such as a low-rank matrix recovery method [20]. The proposed approach belongs to the category of simultaneously estimating the mask and impulse noise.
To address the challenging blind inpainting task with mixture noise, a novel model is proposed based on imposing a sparse regularization to the binary mask. The proposed model can be efficiently solved by a designed proximal based alternating direction method of multipliers (PADMM) method. The main contribution of this work is given as follows: 1) A new model that fits in the practical situation of blind inpainting problem is proposed. 2) The new model solves the challenging blind inpainting task with mixture noise. 3) An efficient algorithm is given to effectively solve the proposed model.
The outline of this paper is given as follows. The proposed method including the new model and the designed algorithm is exhibited in Section 2. The solution of the proposed method is described in Section 3. Section 4 shows the experimental results and analysis. Finally, we draw some conclusions in Section 5.
2. Model Building
The paper proposes a minimization model to solve the image inpainting problem on the basic blind image with mixture noise. we denote a as the verctor form of mask A and the E.q. (3) can be expressed as follows: where stands for a dot product between vectors, and is a vector with all the value .
is the vector form of one matrix with in (2), and a, y are the vectors with the same dimensions as x. As a is the position of pixels missing, the -a represents the locations of impulse noise in the image.
we will give more explanations for E.q.(4) to present the proposed model for the blind image inpainting with mixture noise: (1)Since mainly represents Gaussian noise, we use the norm to construct the fidelity term3(2)2. only represents impulse noise (e.g., salt-peppers, random value), thus we may use sparse regularization to describe (3)Especially, if the impulse noise is relatively dense, it may impose £q regularization to a in the new model, i.e., . Otherwise, if the impulse noise is sparse and the condition of minimizing may not hold, we could easily set a small parameter to control it(4)Finally, we employ the (anisotropic) total variation (TV) regularization to the underlying clean image x, i.e., , the TV regularization is quite popular and useful in the applications of image processing
As presented above, we formulate the final proposed model for the inpainting task as follows:
The minimization can be equality described as such that and based on [8]. Thus, E.q.(2) can be expressed as follows:
As discussed above, we may get the following augmented Lagrangian problem instead of the constrained minimization E.q. (6) with variable substitution.
Where , and are Lagrange multipliers, and , and are three positive parameters. The Lagrangian problem can be solved alternatively and iteratively by the following minimization subproblems in Section 3.
3. Model Solution Method
We add the proximal term to the x subproblem from E.q.(7) and denote to get the following proximal
Where
The solution of E.q.(8) is given as follows:
Where
The a-subproblem is shown as the following:
We need to discuss the solution by the following two cases:
When a >0,
When a <0,
Therefore, the reformulation is:
Where
The z-subproblem can be written as the following minimization problem: which has a closed-form solution by soft-thresholding [7].
Where
Similarly, the w-subproblem is given as the follows: which holds the closed-form solution by soft-thresholding:
The v-subproblem is given as the follows: which could also hold the closed-form solution [9]:
Where
We finally update the Lagrange multipliers by:
We may effectively obtain the solution of the constrained model (5) with the initial guesses . We summarize the above steps to get the following Algorithm 1:
| ||||||||||||||||||||||||||||||||||
Although Algorithm 1 involves some parameters, these parameters are actually not sensitive and easy to select. We also compute the energy of each iteration. If the energy is below a given tolerance, the iteration will stop and output the final result.
In the next section, we will exhibit the experiment results to demonstrate the effectiveness of the proposed method.
4. Experimental Results and Analysis
The numerical experiments in this section are implemented with MATLAB (R2016a) for both simulated and real images. The experimental computer has 2G RAM and Intel(R) Core(TM)i3-2370 M CPU: @2.40GHz 2.40GHz. Since the literatures for blind image inpainting with mixture noise are limited, we here only compare the proposed method with one recent state-of-the-art blind inpainting approach [6], denoted as “ASInpaint” 4.
In Figure 1, we present the whole process of image, which is degraded jointly by Gaussian noise and impulse noise. The goal of this work is to recover the clean image X from the degraded image Y. To evaluate the quantitative performance of the compared approaches, we employ two kinds of metrics to estimate the performance of different methods: peak signal-noise ration (PSNR) and structural similarity (SSIM)5 [21].
In the experiments, we assume that the pixel values are within the interval [0, 255]. The added salt&pepper type of impulse noise can have a value of either 0 or 255. For Gaussian noise, the values are also uniformly distributed within the interval [0, 255]. For the parameters in Algorithm 1, we empirically set =0.9, =0.08, =0.08, and for experiments. Note that we could tune the paramters to get better results, and we fix them in the experiments to illustrate the stability of the given method. For the parameters of “ASInpaint”, we keep the default settings of the provided code.
In Figure 2, we illustrate the visual performance of the two compared methods by four different simulated images with mixture noise (see Figure 2(a)) named “Lena”, “Cameraman”, “Phantom”, and “Satellite”. We added the Gaussian and impulse mixture noise on the images (see Figure 2(b)). In particular, we evaluate the effectiveness of the proposed method by the same image with different levels mixture noise (see Figure 2 the 1st and 3rd rows). Although the ASInpaint approach also obtained competitive results (see Figure 2(d)), the proposed method could obtain better visual performance, especially on the shape profile of the images (see Figure 2(e)). we have to emphasize that the visual performance of both methods seem to be not better than the recovered image by other literatures. The problem addressed is quite challenging that there are four unknown variables in the problem, such as the underlying clean image X, the mask A, the impulse noise NI, and the Gaussian noise NG. Although we may reduce to only two unknown variables X and A, it is still very difficult to recover the underlying clean image X. However, the given model has also recovered the relatively good visual results by the given algorithm.
(a)
(b)
(c)
(d)
(e)
Meanwhile, denoising results on a real color image by all competing methods (NLH method, NSNR method, WNNM method) is shown as Figure 3. The 15 test images used in image denoising experiments are shown as Figure 4. Inpainting results on images Starfish by different methods (Random mask with 75% missing values) are shown as Figure 5. Inpainting results on images Monarch by different methods (Text mask) are shown as Figure 6. These results show that PADMM algorithm has high performance of image denoising.
(a) Noisy image
(b) NLH
(c) NSNR
(d) WNNM
(a) Input image
(b) Ground truth
(c) WNNM (PSNR: 27.11 dB)
(d) McS: (PSNR: 27.57 dB)
(a) Input image
(b) Ground truth
(c) WNNM (PSNR: 32.85 dB)
(d) McS: (PSNR: 33.85 dB)
The quantitative comparisons of all methods are reported in Table 1, which indicates that the PADMM algorithm can improve the performance and yield the best quantitative results. Meanwhile, the paper tests the performance of NLH, KSVD, BM3D and WNNM algorithms on PSNR and SSIM of 15 pictures in the different value of σn. The experimental results are shown in Tables 2 and 3. The results show that with the increase of σn value, the PSNR and SSIM of each picture gradually decrease, but the performance of these algorithms is significantly worse than PADMM algorithm.
5. Conclusions
In this paper, we present a novel optimization model and design the corresponding algorithm to address the challenging blind inpainting task with mixture noise. There are three main contributions in this work: 1) The model intetrates a sparse regularization to the binary mask, the total variation term to the underlying clean image and a regularization to describe the fidelity term; 2), Theproximal based alternating direction method of multipliers (PADMM) method was utilized and implemented to solve the optimization problem; 3) Experiments on some simulated examples with complex mixture noise are implemented, and the visual and quantitative results demonstrate the proposed method outperforms the other method.
Data Availability
The data that support the findings of this study are available from the corresponding author, [author initials], upon reasonable request.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
Acknowledgments
This study was supported by Huaian Natural Science Research Project (HABZ201919).