Image inpainting based FMM algorithm by a direction selection method

In this paper, we propose an improved image inpainting method based on FMM algorithm. The traditional approach gives a fast processing speed but it cannot contribute an optimal edge result. Using gradient matrix to detect the texture direction, the significant pixels for the inpainting process are selected. Furthermore, this procedure also reduced the noise pixels’ influence. Therefore, it can yield a better edge effect for inpainting procedure than the traditional approach.


1.Introduction
For various reasons, an image may be destroyed and several image region may be lost, then it is necessary to fill the missing region by any method.This is called the image inpainting, which is becoming an important topic in the field of image processing now.Image inpainting has been widely used in the applications of digital effect, image restoration, image coding and transmission.
The most fundamental inpainting approach is the diffusion based approach [1]- [3], in which filling the missing region by diffusing the image information from the known region into the missing region at the pixel level.These algorithms are well founded on the theories of partial differential equation (PDE) and variational method.Bertalmio et al. [1] filled in holes by continuously propagating the isophote (i.e., lines of equal gray values) into the missing region.They further introduced the Navier-Strokes equation in fluid dynamics into the task of inpainting [2].In [3], Chan and Shen recover the missing information by using a variational framework based on total variation (TV) model .Then a curvature-driven diffusion equation was proposed to realize the connectivity principle which does not hold well in the TV model [4].A joint interpolation of isophote directions and gray-levels was also designed to incorporate the principle of continuity in a variational framework [5].Recently, image statistics learned from the natural images are applied to the task of image inpainting [6]- [8].The diffusion-based inpainting algorithms have achieved convincingly excellent results for filling the non-textured or relatively smaller missing region.However, they tend to introduce the smooth effect in the textured region and larger missing region.
The second category approaches are the examplar-based inpainting algorithm.These approaches propagate the image information from the known region into the missing region at the patch level.This idea stems from the texture synthesis technique proposed in [9], in which the texture is synthesized by sampling the most suitable patch from the known region.However, natural images are composed of structures and textures, the structures always constitute one image's primal sketches (e.g., the edges, corners, etc.) and the textures are always represented the image regions with homogenous patterns or feature statistics (including the flat patterns).Due to Pure texture synthesis technique cannot clearly handle the missing region with composite textures and structures, Bertalmio et al. [10] proposed to firstly divided the image into structure and texture layers and then in-paint them by different method, using diffusion-based method to in-paint the structure layer and using texture synthesis technique to in-paint texture layer.It overcomes the smooth effect disadvantage brought from the diffusion-based inpainting algorithm; however, recovering the larger missing structures also is very hard to achieve.Criminisi et al. [11] designed an examplar-based inpainting algorithm by propagating the known patches (i.e., examplars) into the missing patches gradually.To handle the missing region with composite textures and structures, patch priority is defined to encourage the filling-in of patches on the structure.Wu [12] introduced the DOI: 10.12792/iciae2014.005concept of cross-isophotes to the examplar-based inpainting algorithm, in which he designed a cross-isophotes patch priority term based on the analysis of anisotropic diffusion.Wong [13] improved the examplar-based inpainting algorithm by proposing a nonlocal means approach.The image patch is inpainted by not the most suitable patch but a set of candidate patches in the known region.Compared with diffusion-based inpainting algorithms, the examplar-based inpainting algorithms are more suitable for inpainting the large missing region.
FMM algorithm focuses on the diffusion-based inpainting approach.This algorithm firstly proposed by Telea, it has a high speed in processing and has a similar result with other method [14].
Yang add the Anisotropic diffusion theory to it and have a good result in large region inpainting [15].This paper is organized as follows In section 2, it tell the development of FMM algorithm in the recent years, In section 3, it tells the improvement of the new method.The experiment and result is shown in section 4 and finally we conclude this work in section 5.

.The current development of the FMM algorithm
Fig .1.image in-paint theory In the theory of the FMM algorithm, we regard the Ω as the region need to be in-painted, regard as the edge of the in-paint region, divide the pixels into 3 sets: Band(the pixels on the edge of region), Known(the grey value we've already known), Inside( grey value unknown pixels).By using a time function T(x,y) to determine the sequence of the in-paint, then in-paint the region from the edge to the inside.
In-paint process: the p is the pixel needed to be in-painted, we select a window B  ( p), use all the known pixels(q) and equation  to compute the grey value of p, I (q)means the gradient of q, for different q ,we use a weight function w(p, q)to weight the influence between p and q, and finally weighted average it, so the equation changed to be.
(2-2) for the weight function w(p, q),it has three elements w(p,q)=dir(p,q)*dst(p,q)*lev(p,q) (2-3) In it, the three elements are dir(p,q)= dst(p,q)= (2-4) lev(p,q)= These three elements represent the 3 points of the known pixels have influence to the inpainting.dir element shows the direction influence for the in-paint, it means if the pixel q's direction is close to the pixel p's direction, the more influence it will have.dst element shows the distance influence for the inpaint, it means if the pixel q is near the pixel p, the more influence it will have.lev element shows the time influence for the inpaint, if the pixel q is close to the known information, the more influence it will have.
From the sets of equation, we can see that at last the three elements have the same importance in it, but actually in the inpainting, the pixels which have the same texture direction with the destroyed pixel will do more contribute to the inpainting.So, if we compute under this method, the pixels which are near the destroyed pixel but with different texture direction will regard have much useful information, in fact, this kind pixels always are noise pixels.So it will make the edge unclear.
The most important advantage is this method can spend very little time and give a good result.But when in-paint one pixel, this method use all the neighbor information, so it may add any noise into the in-paint pixel.And finally we can see that if the width of the in-paint area is larger than 20 pixels, the result will be unclear, especially in the isolux line and edge tangent place.

.Use gradient matrix to improve the FMM algorithm
Because of the disadvantage of the old FMM method, I consider a new method to remain the fast advantage and give a good edge remaining result at the same time.For the natural image, the pixel on the texture direction may make more influence to the in-paint pixel.So if we just select the pixels on the texture direction of the in-paint pixel, we can reduce the influence made by the noise pixels.

extract gradient matrix from the image
First we compute the gradient value and direction of the all the Known pixels, for Known pixels: For Band pixels, they are indispensable in the computing, but one hand of band pixels may be the inside pixels, so the grey value we don't know and we can not get the Band pixels' gradient, then we need to change the equation, use the band pixel itself replace the inside pixel.
Finally we get two matrix, one matrix gv(x, y) represent the gradient value of every pixel, and the other matrix g(x, y) represent the gradient direction of every pixel.
3.2 select the texture direction by the gradient matrix Now we've get the two gradient matrix, the value are continuous, but we hope the gradient direction to be a discrete value to carry on the next work .So we should define several gradient direction to replace the gradient direction value.
According to the number of the defined direction, it will influence the in-paint result, here we just for demonstrating the feasibility of our method ,so we use a easier condition to work, set the direction number to 4.so the gradient direction will be divided into 4 situations.
Then add another situation represent the situation that the gradient value of all the pixels in the window B(p) are very small .weset a threshold, if the sum of the gradient value is less than the threshold, regard it gradient direction as this situation.
Count the gradient value on every direction of all the known pixels in the window ,select the direction which has the biggest value as the in-paint pixel's gradient direction .thenaccording to the gradient direction ,only select the pixels on its texture direction(from gradient direction we can get texture direction because they are vertical) to compute the grey value.

compute the weight
At last, using the FMM algorithm to compute grey value of the pixels, due to the new method, here the old dir element can be canceled, so only use the dst element and the lev element to compute now the w (p, q) is changed to w(p,q)=dst(p,q)*lev(p,q) (3-2) In this equation, the dst element and lev element are the same as the old method.

.Experiment result
In this section, we test the proposed improved FMM algorithm These are the result of second example, the image has a little larger destroyed region , from them we can see ,compared with the old method, the new one maintains a good edge ,the road part and the lake wave part both have a clear edge result .Because the old method select all the neighbor pixels to compute.For the destroy region the outside part just use the real information to in-paint, but the inside part is different, they use the in-painted pixel to in-paint, so the noise influence gained to make the inside part unclear.But the new method is different, it only select the same texture direction pixels, so the noise influence reduced.

Conclusion
In this work, the gradient matrix was adopted to the FMM inpainting algorithm, it overcame the disadvantage of traditional FMM inpainting method, which cannot contribute an optimal edge result.By using the gradient matrix to detect the texture direction, and selecting the significant pixels for the inpainting process, the resulting images seem more reasonable and more similar to the natural scenes.In fact, the proposed approach not only remains the advantage of fast processing speed, but also contributes a better estimation result of edges compared to the traditional method.

Fig. 2
Fig.2 the gradient direction situation (a)-(d) are the 4 gradient direction, (e) is the adding situation.Red pixels are the inpainting pixels, blue pixels are the selected pixels in each situation.
on a variety of natural images.And then compare the result with the old method.In the following examples, they show the advantages of our method and the feasibility in theory.The window B ( p) is set to 9 by 9 (according to a lot experiment result, too small window will not have enough pixels, too large will bring more noise ),the direction number is set to 4,and the threshold is set to 25. is complete image,(b) is destroyed image,(c) is old method inpainting result, (d) is the new method result.Here is the first example, the new one is a little better, especially the edge of the building.is complete image,(b) is destroyed image,(c) is old method inpainting result, (d) is the new method result.