Improvement of Patch Selection in Exemplar-based Image Inpainting

In the existing exemplar based image inpainting algorithms, the most similar match patches are used to inpaint the destroyed region, and are searched in the whole source region in a fixed size. However, sometimes such an approach would decrease the connectivity of structure and clearness of texture while increases the time complexity. To solve these problems, firstly we propose an adaptive sample algorithm based on patch sparsity, by which divide the patch into three types ( smooth type, transition type and edge type ). Then the size of the sample patch can be adaptively changed according to the type. Secondly a candidate patch method is also proposed to improve the patch matching performance. Simulation results show that the proposed method obtains a more reasonable patches than the traditional one. Furthermore, it gives a better texture inpainting effect, especially when the processed image either has the complex and regular textures or has a large destroyed region.


Introduction
The concept of digital image inpainting is firstly mentioned by Bertalmio [1] in 2000, for many reasons, an image may be destroyed by human, nature and other factors, some regions in the image may lost its information. Digital image inpainting technology is by using the remaining part of the image to maintain the integrity of the original image and now it is becoming a very important topic and comes out a lot of applications for representing the ancient painting, movie processing, image coding and transmission.
The most fundamental in-painting approaches are inpainting the image at pixel level, they diffuse the pixel information from the remaining regions into the missing regions [1]∼ [4]. These algorithms are well founded on the theories of partial differential equation (PDE) and variational method. Bertalmio et al. [1] filled in holes by continuously propagating the isophote (i.e., lines of equal gray values) into the missing region. To improve the result, they further introduced the Navier-Strokes equation in fluid dynamics and applied it into the inpainting processing [2]. Afonso proposed a new fast algorithm firstly solving the constrained optimization formulation of regularized image restoration.. The approach can be used with any type of convex regularizers and it is based upon a VS technique [5]. Zhu improved the classical PDE model and proposed a novel PDE framework for the striped texture image inpainting, it employed the orientation field to diffuse the gray information along a significant direction [6]. [7] overcome the disadvantage of high calculation load made by the traditional PDE method and improve the inpainting performance by proposing an enhanced Curvature-Driven Diffusions (ECDD) model and a fast local non-texture in painting scheme. FMM algorithm is also one research hotspot of pixel level algorithms. It firstly measures the direction, the distance, and the level set influences made by the neighbor pixels, then through them calculating the pixel's gray value. Telea firstly introduced this algorithm and gave the basic assumption of it. It inpaints the image very quickly and has a similar result in the visual effect [8]. Yang tries to employ the anisotropic diffusion theory into the FMM algorithm and have a good result in large region in-painting [9]. In the improved method [10], the author changed the traditional direction index and gave a new direction selection method, which can select more significant pixels and enhance the edge effect. The diffusionbased in-painting algorithms have successfully achieved to maintain the integrity of the image, and can generate excellent results for inpainting the missing region which has nontextured or smaller relationship with the image. However, this category of approaches tends to produce the smooth effect in the textured region and larger missing region.
The second category approaches are the exemplar-based inpainting algorithm. These approaches propagate the image information from the known region into the missing region at the patch level. This idea is firstly proposed by Criminisi in 2006 [11]. But it stems from the texture synthesis technique proposed in [12]. It samples the most suitable patch from the known region to synthesize the texture. However, natural images are composed of structures and textures, in which the structures always constitute one image's primal sketches (e.g., the edges, corners, etc.) while the textures are always represented the image regions with homogenous patterns or feature statistics (including the flat patterns). So only by texture synthesis technique it cannot clearly repair the missing region with composite textures and structures. To take the advantages of both diffusionbased method and texture synthesis technique, Bertalmio et al. [13] firstly divided the image into structure and texture layers, then using diffusion-based method to in-paint the structure layer and texture synthesis technique to in-paint texture layer. By this processing, the smooth effect disadvantage brought from the diffusion-based in-painting algorithm is well overcome, but it is still very hard to recover the larger missing structures. Criminisi et al. [11] designed an exemplar-based in-painting algorithm by propagating the known patches (i.e., exemplars) into the missing patches gradually. To handle the missing region with composite textures and structures, patch priority is defined to encourage the filling-in of patches on the structure. In [14], Zoran performed a better result by learning a better patch priors. It firstly presented a generic, simple Gaussian Mixture prior, and then applying it on a generic framework for the whole image restoration. [15] introduced three basic techniques: copy-and-paste texture synthesis, geometric partial differential equations (PDEs), and coherence among neighboring pixels. It combined these three building blocks in a variational model and completed to approximate the minimum of the proposed energy functional.To improve the result in a more natural quality with high performance, Chung introduced the minimum error boundary of cut technique into the exemplar-based method [16]. Li [17] designed an adaptively size-changed patch according to the location, which can generate a better result in inpainting the region with complex texture. In [18], the fast priority and the coherent nearest neighbor search method were employed to produces better completeness of linear edges and reduces the error propagation problem. Compared with diffusion-based in-painting algorithms, the exemplar-based in-painting algorithms has an improvement in inpainting the image which has complex texture, especially has a good result in inpainting the large missing region. This paper learns from the A. Criminisi et al. [11] and try to inpaint the image on patch level. And it tries to overcome the most important disadvantage of the traditional method: the fixed size of the sample and the non-optimal selection of exemplars. It is organized as follows. In section 2, it introduced the theory of the traditional exemplar-based method. In section 3, it proposed two improvements to give a better result. Contains the detail of the adaptive sample size algorithm and the candidate patch system. And the experiments and comparisons with the traditional methods are performed in the section 4. At last, we conclude the work in the section 5.

Theory of the exemplar-based method
The main idea of the exemplar-based method is an isophotedriven image sampling process. Figure 1 illustrates the procedure of the exemplar-based inpainting method.
The exemplar-based method separate the destroyed image into three compontents indicated in Fig. 1: the Φ is the source region(non-destroyed part), the region Ω is destroyed by some reasons, and the ∂Ω is the edge between source region and destroyed region shown in Fig.1(a).
∈ Ω, and     Firstly we need to decide the location to start the inpainting process, we called this decision procedure "patch priorities" which shown in Fig.1(b), the details are described in the following section. Secondly, by evaluating the texture and structure information between the source part and the destroyed part, the most similar patch can be find as shown in Fig.1(c). Thirdly, update the destroyed patch with the most similar patch as shown in Fig.1(d), then recalculate patch priorities for the updated area and loop these steps until the whole image was inpainted.

Patch priorities
The proposed algorithm defines the inpainting sequence by the patch priorities, and this parameter has a higher priorities when the below conditions are satisfied: (i) if the center pixel of the patch is located on a strong edge of the image.
(ii) if the patch has more pixels that the gray values are known.
The Fig.2 shows the theory.
∈ Ω, and The prioritie P(p) is defined as the product of two terms: where C(p) is called the confidence term which means if the patch has more known pixels, it gives a large influence to the inpainting processing. It is defined as: Where C(q) is set to C(q) = 0: ∀q ∈ Ω, and C(q) = 1: ∀q ∈Ω, |Ψ p | means the total number of the pixels in the patch.
On the other hand, where the D(p) is called the data term which means if the patch is located over on the edge of the image, then it contributes a large influence to the inpainting processing. And it is defined as: where ∇I ⊥ p means the gradient of the pixel p, the n p means the unit vector orthogonal to the edge ∂Ω in the pixel p, the α means the gray scale of the image.

Similarity Evaluation
We evaluate the image texture and structure information of a select patch which randomly get form the source region, then decide the most similar patch q to the destroyed patch p. This processing can be evaluated by the following equation: Where the d(Ψp, Ψ q ) means the distance function of two patches. Here it is the sum of squared differences (SSD) of the known pixels in the two patches. And this time d(Ψp, Ψ q ) we just use the Euclidean distance.
Having found the source exemplar patch q, each absent pixel in destroyed patch p is copied from its corresponding position of patch q.

Update the confidence values
After the patch p has been completed inpainted by the patch q, we need to update the confidence C(p) in the previously inpainted area so that we can go into the further inpainting processing.

Adaptive sample size algorithm
From the lots of experiment results, we found that the size of the sample patch always plays an important role in the inpainting processing. While in the inpainting process, if the method uses a large sample size, it would remain the detail texture but cause the false matching, because the matching patches may be have a small difference value but the texture features are quite different. On the opposite, a smaller sample can greatly spread the edge information into the destroyed region but lose the texture, and it will makes the inpainting blurring. So our aim is to balance the advantages and disadvantages made by the large sample and small sample.
Proposed method employs an adaptive function to decide the size of the patch.
Firstly we introduce a concept of patch sparsity S (p) which is define as: where N(p) is number of the pixels in the destroyed patch, N s (p) is the number of the known pixels in it. ω p,q means the similarity between the patch Ψ p and patch Ψ q , and it is defined as: where The Z(p) is a normalized constant defined as and w 0 is the standard patch size, it depends on the destroyed region size, here we use the fortieth of the destroyed region as it.
The value of the S (p) represents the location of the patch. The patch locates in the edge region when the S (p) closes to 1, oppositely, the patch locates in the flat region when the value closes to 0, and the texture region obtains a value around the middle of 0 to 1. In this research, the boundary of these values are decided empirically.
Therefore, in this research the sample size w is defined as: T 1 = λ 1 · (p max − p min ) + p min T 2 = λ 2 · (p max − p min ) + p min p max and p min are the maximum and minimum of all the patch sparsity. λ 1 and λ 2 threshold set by the experience, here we set λ 1 equals 0.55 and λ 2 equals 0.15.

Candidate patch method
Before introducing the second improvement, It is necessary to explain the concept of the pixel difference and the texture difference precisely. These two types of differences are both generated between the sample patch and the destroyed patch. For the patch pair under matching processing, even if the textures are almost the same, but the distribution of the pixel value is complex, therefore it may generate a large difference, this difference is called the pixel difference. On the other hand, the texture difference means the difference generated by the different texture. The pixel difference is always generated in several regions and each one has a small value while the texture difference is always generated in few regions but has a large value.
The template matching algorithm generally selects the most similar patch to inpaint the destroyed region. But it sometimes obtain a mismatch result because the accumulation error of pixel difference is larger than the texture difference. This happening bother the final inpaint result very much because it is expected that the texture difference should guide us to the correct result but not the pixels difference in the matching process. Therefore, we proposed a candidate method to reduce the mismatches. Candidate method is a matching method not only using the most similar patch but also using a group of similar patches. Fig. 3 shows an example of the candidate patch method, in Fig. 3(a), the right three patches are most similar to the destroyed patch. The traditional method may selects the top patch because it contributes the minimal cost of the difference. But in the proposed method, by expanding the destroyed patch and the candidate patches, it can be found that the third patch is most reasonable one to match the destroyed patch.

Experiment and result
In order to prove the performance of proposed algorithm, experiments are performed to different kinds of images, ranging from simple graphic to nature images that including complex textures. In all of the experiments, the patch size is set as the fortieth of the destroyed region, and the remaining of the destroyed image is set to be the source region. All experiments are carried out by using the Matlab 8.6.0, and run on a Intel Core i5 2.5GHz CPU(Macmini6,1) with 4GB of RAM.
Firstly, we apply the proposed method to inpaint a simple graphic pattern image like Fig. 4.
Assuming by some reasons, the part of origin image Fig.4(a) was disappeared as shown in Fig. 4(b). Compared with the traditional exemplar-method result shown in Fig. 4(c), Fig. 4(d) shows the proposed method adaptively changed the sample size to inpaint the absent part. Also we can find that the proposed inpainting process searches the more significant patches until to the top of the image pattern. Because we don't know the true answer just from the Fig. 4(b), it is impossible to select a optimal patch to repair the top part, and at last the program just select a reasonable patch to finish inpaint process. Due to the disadvantage of exemplar-based method, it can not inpaint the corner perfectly without an extra supervise, but the proposed method tried its best to reduce the error region.
Secondly, Fig. 5 shows the inpainting experiment of a  Fig. 5(a), to inpaint the destroyed one ( Fig. 5(b)), the traditional method ( Fig. 5(c)) selects a similar but not correct patch(like the chapter 3.2 shown) after the inpainting process entering into the center of destroyed region. But the proposed method ( Fig. 5(d)) can overcome the disadvantage by using the candidate patch system. Even though in some connection part of the image the result has one pixel offset, but globally to see it is hard to be found visually.
We also try the proposed method to inpaint a scenery image and get the result shown in Fig. 6. For the origin image Fig. 6(a), to restore the destroyed image shown in Fig. 6(b), the resulting image of Fig. 6(c) inpainted the image on pixel level so the result is very fuzzy around the edge, and when inpainting entering into the inside, it even gives none of useful information only maintain the integrity of the original image. Traditional method shown in Fig. 6(c) inpainted the image on patch level so it can remain some texture information inside, but the result is not very good. The proposed method shown in Fig. 6(e) obtains a better result, especially in the connection part between destroyed part and the source part. It gives a quite different visual effect. The detail result is shown in Fig. 4 (the red line shows the destroyed edge.), from these result we can see the proposed method has two advantages. Firstly, it has a better connection to the remaining region. Secondly, compared with the others two results, the proposed method also has a better effect on the connection of different patches. As I think, the differences are contributed by the improvement of the patch selection. The improvement enhances the edge effect of the image and has a better connection effect between different patches, so at last the result has a strong edge and a smooth inpainting content after the processing.

Conclusion and future work
In this paper, a new method was proposed to reduce the mismatching happened in the traditional exemplar-based method. The proposed method contains two parts. The first one is using an adaptive patch, it uses the patch sparsity to adaptively change the size of the patch, which can enrich more useful texture information in to the patch area. The second one is employing a candidate patch method. Different from the traditional method, the candidate patch method selects a group of similar patches, not only the most similar patch. This method can reduce the influence made by the pixel difference and amplify the guide made by texture difference. Therefore, proposed approach can selects the most reasonable patch. From the result, the proposed method not only reduced the mismatch but also gave a better visual result.
In this work, both the traditional method and the proposed method were using the SSD to calculate the similarities between the sample patch and destroyed patch. SSD is a distance function only using the grey value information without any space transformation. In the future, we are planning to design an efficient cost function to calculate the difference target for the inpainting task. Also we are planing to adopt the feature extraction method to solve the problem happened in Fig. 4 which is the better but not perfect result.