Deblurring a Camera-Shake image using a Thinning Kernel

.


Introduction
Image deblurring has been widely discussed over the past decades (1)(2)(3) .As far as handheld devices, the most common types of blur are defocus blur and motion blur.The latter is the result of camera-shake and object motion.Camera-shake, which we focus on this paper happens because it is difficult to stabilize a camera for a long exposure time.This often leads to camera-shake blurred images.
Generally, a motion blur is modeled as where B is a blur image, L is a latent image, K is a blur kernel, n is a noise,  is a convolution operator.Here, if the blur kernel K is known, we just estimate the latent image L from B and K.We refer to this as non-blind deconvolution.
Otherwise, if the blur kernel K is unknown, we estimate both L and K from B. This we call blind deconvolution.However, blind deconvolution is an ill-posed problem.It is difficult to estimate an accurate combination of both latent image L and blur kernel K with the only information provided by blurred image B, which has almost infinite combinations.In order to solve this ill-posed problem, we usually impose some extra restrictions (through the statistics or the features of the natural image).This process introduces additional information wherefore the problem is solvable.The general form to describe such a restricted problem is given by an objective function as in (2), where the first term is the likelihood term, the second term is the image prior term, which restricts the latent image, and the third term is the kernel prior term, which restricts the blur kernel.Most deblurring methods emphasize the estimation of interim images (the second term).However, the resulting kernel is often wider than it should be and degrades the quality of deconvolved images.To overcome this, we present a thinning scheme to better estimate a kernel.Unlike other methods, our approach focuses on the estimation of a good kernel.In this way, a clear image can be recovered from a camera-shake blurred one.The outline of the paper is as follows.Section 2 introduces the existing research on blind methods and non-blind methods.Section 3 discusses the proposed method followed by the experimental results.Section 5 concludes our paper.

Related work
First, we discuss previous research on non-blind deconvolution, where the blur kernel is known, and only the latent image is unknown.The simple and most common methods are the Richardson-Lucy and Wiener filters (4)(5)(6) .The drawback of these two methods is that when the blur kernel is not accurately estimated, severe ringing artifacts occur, and results are easily affected by noise.Alternatively, the heavy-tailed distribution of gradients in natural scenes has proven to be effective when using priors for deblurring.Krishnan and Fergus used Hyper-Laplacian priors to model the gradient distribution of natural images, and further reconstruct latent images (7) .This method has been adopted by many studies to estimate latent images, when the blur kernel bas been already determined (8)(9)(10) .
Blind deconvolution, where both the blur kernel and latent image are unknown, is closer to a real situation.Existing methods of blind deconvolution typically use an alternative approach.By iterating the estimation of the interim image and blur kernel, finally, the latent image is reconstructed by using the blur kernel.As mentioned above, due to the limited information of kernels, most blind deblurring methods focus on how to estimate the interim image.These methods usually have different priors.The different properties of images can be observed to better reconstruct latent images.Fergus et al. used a mixture of the Gaussians model with a heavy-tailed natural image prior (11) .Shan et al. introduced a probabilistic model with a new local smoothness prior, which can suppress ringing artifacts (12) .Cho and Lee predicted salient edges by using a shock filter and bilateral filter, and used a threshold to remove small structures (13) .The prediction gradient map is effective for estimating blur kernels.Krishnan et al. proposed a normalized sparsity measure which gives the lowest cost for a true sharp image (8) .The normalized sparsity measure is used to replace l 0 measure, because l 0 measure is difficult to optimize.Komodakis and Paragios showed an MRF-based prior which utilizes the concept that the blurred images generated from a latent image and the quantized versions constituted by using a MRF prior are visually quite similar (14) .Xu et al. proposed an l 0 gradient minimization method which can effectively smooth images and optimize the objective function (15) ; therefore, based on this method, Pan et al. and Xu et al. showed a prior by using l 0 sparsity (9)(10) .All of the above methods focus on how to estimate the interim image, and provide an easy prior to limit the blur kernel, since the kernel information is completely unknown.

Proposed Method
In the proposed method, we explore the model of a blur kernel and make some assumptions.Our approach includes three main parts.The first part is just like other existing methods, i.e., we use a prior to estimate the interim image, and find the shape of the blur kernel in a simple way.In the second part, we use a thinning kernel algorithm to obtain the skeleton of blur kernel.In the third part, we reweight the blur kernel by the Gaussian smoothing function.Finally, we can reconstruct a high quality deblurred image by using the blur kernel.

Assumptions of blur kernel
It is unknown that the blur kernel looks like, so we make a simple inference.Fig. 1 illustrates the influence of kernel convolution on images.The images on the top show that a sharp image can be expressed as the sharp image convolves with a delta kernel which has a center value of one and zero otherwise.Other words, the delta kernel can be seen as a light which the exposure focuses on.However, as shown on the bottom, when the shutter speed is slow compared to the speed of unintended camera shaking, we get blurry images.The shaking trajectory with light generates a blurring kernel, thus producing a blurred image.Based on this inference, we assume that (a) the shape of kernel is thin and continuous since it is generated by the shaking trajectory, (b) the pixel values of the blur kernel are between 0 and 1, and sum to 1, and (c) the blur kernel is sparse.

Estimation of interim image
Next, we want to find an interim image, which only has strong edges.Because the strong edges are likely to exist in the latent image, the edge information can be used to estimate the correct blur kernel.Here, we use an l 0 sparse method like Xu's (10,15) , in which the objective function is where x is the interim image, k is the blur kernel, y is the blurred image,  is the convolution operator, and λ is the weight.∇ denotes the gradient operation.The first term is the likelihood term such that the interim image convolved by the blur kernel should be similar to the blurred image.
The second term is the prior term of the image to encourage the interim image to be smooth and only have strong edges.By the loss function in Xu's (10) , ( 3) can be rewritten as where l denotes an auxiliary variable, h and v are the horizontal and vertical directions, |l *i | 0 =1 if l *i  0 and zero otherwise, and  is the weight which controls the optimization of l 0 .The detailed derivations are referred to the supplementary document (16) .
To solve this half-quadratic function, we alternate between computing l and x by where ( 5) is a fast pixel-wise closed form solution for the l 0 sparse loss function (10,16) .Equation ( 6) is solved by using a Fast Fourier Transform (FFT) and working directly in the frequency domain.This saves a lot of time but sometimes introduces ringing artifacts.

Estimation of the primitive blur kernel
We use the interim image obtained above to estimate the blur kernel.However, not all the strong edges are suitable to estimate a blur kernel.Xu and Jia claimed that if the scale of an object is smaller than that of the blur kernel, the edge information could mislead kernel estimation (17) .To avoid this ambiguity, we calculate a confident mask to select the informative edges for kernel estimation (17) .Firstly, we calculate the confidence values of each pixel by where x p is the index of each pixel in the interim image, N(x p ) is a window centered at x p with the size similar to the kernel's, and 0.5 is to prevent producing a large C(x p ) in flat regions.After obtaining the confidence values, we use a threshold  C to generate a mask with , where H(k) is the Heaviside step function, which is zero if k is negative, and one otherwise.Finally, the selected edges for kernel estimation are determined as where  is an element wise multiplication operator.Contrary to the interim image that has some information from the blurred image, the shape of the blur kernel is completely unknown.We impose a sparse prior in estimating the kernel.We define the objective function as min , where  is a weight.Using the quadratic function, we can quickly get the preliminary shape of the blur kernel.
Equation ( 9) can also be solved very efficiently by using FFT as follows (10)   The estimation of the blur kernel is very sensitive to noise.To relax the sensitivity, based on our assumption given before, we use the following four steps as a post-process to modify the blur kernel: (1) set a pixel value to be zero if its pixel value is negative, let it equal zero; (2) if the pixel value is less than 1/20 of the maximum value of the blur kernel, let it equal zero; (3) clustering pixels via connected components, discard a group if the sum of the pixel values is smaller than 0.1, let the pixels of the group equal zero; and (4) normalize the blur kernel so that the sum to be one.
Figure 2 describes the algorithm of preliminary estimation of the blur kernel and interim image.The Algorithm is carried out in a coarse-to-fine manner using an image pyramid (9)(10)(11) .The goal is to estimate the overall shape of the kernel by using loose restrictions.

Thin kernel estimation
Due to the lack of information of the blur kernel, we use quite loose conditions to estimate the blur kernel in the first part of our system.From the assumption that the trajectory of camera shake is thin and continuous, the estimated preliminary kernel should be refined accordingly.We propose a method consisting of thinning and Gaussian reweighting to refine the estimated kernel.

Thinning step
We use a simple thinning algorithm to find the skeleton of the kernel (18) .This algorithm applies on the kernel obtained in Algorithm 1.The resulted skeleton of the kernel serves as the information to estimate a weight mask in the next step.
Firstly, a binary kernel map is created such that it would be one if it were non-zero in the estimated kernel and zero otherwise.Then, for those pixel P(i, j) = 1, named it as P 1 and its eight neighbors starting from the one right above it as P 2 , followed by P 3 to P 9 clockwise, we use the following four rules to determine whether P 1 should be thinning (deleted).If P 1 fulfills all four rules (a)-(d), it will be deleted.Otherwise, P 1 is a skeleton pixel and will be preserved

Reweighting step
The weight mask is designed to have large weight for a pixel if it is close to a skeleton pixel.For every skeleton pixel, we take the gradient direction and reweight it using a Gaussian distribution.The weight may be rapidly solved with the lookup table (LUT) shown in Table 1.In this way, we can efficiently obtain a weight mask W and using this weight mask we can reweight the blur kernel as where  is an element wise multiplication operator.

Estimating step
Finally, using the reweight kernel of (11) to estimate a more accurate blur kernel, our objective function is where  is a weight.We apply the well-known Alternating Direction Method of Multipliers (ADMM) method (14) , which is essentially a variant of the Augmented Lagrangian method and allows us to efficiently optimize the half-quadratic problem through an auxiliary variable K'.By using the auxiliary variable K', (12) can be rewritten as where a is a transformed vector of Lagrange multipliers, and b is a penalty weight.We use FFT to solve K ' i +1 then a soft shrinkage operator to solve K i +1 : Algorithm 1 Input: blurred image B Output: blur kernel K, interim image x 1. Initial: k 1 = delta function, x 1 = y 1 = B2 Scales 2. for S = 1 : Scales 3.
solve for x S+1 by ( 6 where .Therefore, we can achieve optimization via an iterative way.An example of the thin kernel estimation is shown in Fig. 3.

Latent image estimation
We use the blur kernel obtained via the thinning kernel step to estimate the latent image.Here, if we use the l 0 regularization method as in Subsection 3.2, it will lead to a lack of texture and detail in the image.Due to the property of sparseness, it only has strong edges and other details are ignored.Instead, we use the method adopted in Krishnan et al. with the objective function of (17) where  is a weight (7) .
Likewise, it is a half-quadratic function, so we solve it by using a variable w, where the equation is rewritten as (18) To solve variable w, we also use a soft shrinkage operator, which is efficient and simple.To solve latent image L, we use FFT as follows in (19).Algorithm 2 summaries the steps for image deblurring via a thinning kernel process.The goal is to estimate a more accurate kernel by adding new restrictions and reconstruct a high quality deblurred image by using the blur kernel.

Experimental Results
The proposed method is tested by a public database, provided by Levin et al. (3) , of 32 blurred images produced by 4 images and 8 different blur kernels shown in Fig. 5, where the size of the images is 255 by 255, and the various sizes of the blur kernel range from 13 by 13 to 27 by 27.
To demonstrate the effectiveness of the estimated blur kernels, we use error ratios to measure it, ( where I with subscripts g, r and k g are the ground truth image, the reconstructed image by the proposed method and by the ground truth kernel, respectively.The smaller the error ratios the better.Empirically, error ratios above 2 are already visually implausible (3) .The following results, Fig. 6, are obtained from the average of 32 blurred images.
Shown in Fig. 6(a), 68.75% of the results our method takes fall under the error ratio of 1.5.Compared to other state-of-the-art methods, shown in Fig. 6(b), our method is the best, with 93.75% of the results falling under the error ratio of 2. Due to incomplete accessibility of the source codes, we take the error ratios of other methods of Xu's (10) .
Next, we compare our results with Pan et al. (9) and Xu et al. (10) , who used a similar l 0 regularization method to estimate the interim image.However, they only used the    (3) .relaxed restrictions for blur kernel estimation.In Fig. 7, we can see that because Pan et al. (9) and Xu et al. (10) used a similar method for estimation of both the interim image and blur kernel, their reconstructed images and blur kernels are very similar.Comparing the reconstructed kernels in (b) and (c) to the ground truth in (e), the estimated blur kernels are wider than the ground truth kernel.The proposed method, in contrast, can recover the blur kernel correctly using the thinning algorithm as indicated in (d).Examining the second row of Fig. 7, although the blur kernel estimation of the proposed method is more similar to the ground truth kernel, more serious ringing artifacts appear in the boundary of the image.The main point is that when using FFT to estimate the latent image, it is necessary to extend and fill the image in order to meet the period in the frequency domain.The proposed method uses a simple mirroring method to pad it, while other approaches use optimization methods.In the third row, we show a failure example.The ground truth kernel in (e) indicates that the main light of exposure is at the lower-left corner.However, we see an unnatural circular trajectory on the upper-right side.In the proposed algorithm, under the assumption of natural shake trajectory, we take the delta kernel as the initial kernel.However, with the unnatural shake trajectory, the delta function is not an appropriate initial kernel, because it produces error estimation for the overall structure of the kernel.When the recovered kernel is not correct, the reconstructed image is doomed to fail.

Conclusions
Blind deblurring methods usually consist of estimation of interim images and estimation of blur kernels.Since there is little information in kernels compared to that in interim images, when only a blurred image is available, most deblurring methods emphasize the estimation of interim images.Sometimes, these methods obtain a wider blur kernel, thus affecting the quality of the reconstructed deblurred image.In this study, to deblur camera-shake blurring images, we focus on how to estimate a good kernel.
We make simple inferences and assumptions for blur kernels based the trajectory of the camera shake and propose a thinning kernel method for better estimation.The method has three main steps.Firstly, we relax the condition to find the overall shape of the blur kernel.Next, we use a thinning algorithm to obtain a skeleton of the blur kernel.Thirdly, we reweight the blur kernel through Gaussian distribution.By repeating these steps a few times we can get a more accurate blur kernel.Finally, we can reconstruct a high quality deblurred image by using the blur kernel.In addition, our method can be used for post-processing in the kernel estimation of other existing methods.Comparison with other famous methods (modified from Xu's (10) ).The horizontal axis shows error ratios and the vertical axis shows the percentage of testing images falling under the indicated error ratio.
(a) Input (b) Pan [9] (c) Xu [10] (d) Ours (e) G. T. Fig. 7. Comparison with others using l 0 regularization to estimate the interim image.The insets on the lower left of the images indicate the blur kernels.Compare estimated kernels of (b), (c), (d), and the results indicate that our method can resolve the wide kernel problem and get better estimation of blur kernels.However, the proposed method may cause ringing artifacts appear in the image border, as in (d) of the second row when implementing FFT.However, our method is not suitable when the shake trajectory is unnatural.The boundary problem also influences the reconstruction quality of latent images.In the future, we will study the problem of kernel size estimation.This is an important that has received little attention.

Fig. 1 .
Fig. 1.The influence of the blur kernel on images.

Fig. 4 .Fig. 3 .
Image deblurring by thinning kernel.(a) The kernel shape from Alg.1.(b) Thinning algorithm is applied.(c) The weight mask obtained by Gaussian distribution.(d) The reweight kernel obtained by (a) and (c).(e) The final kernel obtained by the proposed method.(f) The ground truth kernel.
Fig. 6.(a) Error ratios of the proposed method.(b)

Table 1 .
Lookup table of the Gaussian mask W (d: the distance and std is the standard deviation)