Super-Resolution from a Single Image to Improve Sharpness of Texture Based on Optimum Parameter of Total Variation Regularization

The single image super-resolution method can improve blur edges in a magnified image. However, it is difficult to improve the sharpness of a texture region in a magnified image since high frequency components of texture are different from others. In this paper, we propose super-resolution from a single image based on total variation (TV) regularization to improve sharpness of texture in magnified image. We use diverged components of total variation regularization to generate high frequency components. We show experimentally that the diverged components can improved the sharpness of texture in the magnified image. However, the noise as visual distortion occurs except for texture region by using the diverged components. Therefore, we propose a determination method for optimum λ that is a TV parameter to generate an appropriate high quality magnified image meaning an image with sharp texture and low noise in the edges and background. Our proposed method can generate sharper texture in a magnified image than Glasner’s method which is representation method of single image super-resolution.


Introduction
The super-resolution method can generate high-resolution image from low-resolution images by adding high frequency components.In order to generate high frequency components, Freeman et al. (1) use many additional high-resolution images.The super-resolution based on Freeman's approach is called Example-Based super-resolution.However, a problem with Example-Based super-resolution is that the image quality of a magnified image is not improved if the characteristics of additional high-resolution images are different from those of the input image.Therefore, a different super-resolution method, known as single image super-resolution, that uses only an input image, is needed to solve this problem.
Glasner et al. (2) developed single image super-resolution, which combines the Example-based super-resolution and classical super-resolution (3,4,5) approaches.Their method is based on self-similarity, which refers to the original image to generate high frequency components.It is based on an assumption that similar edge components are included in the original image.Therefore, the obtained magnified image has improved the edge blur than traditional interpolation methods or other super-resolution method.However, texture in the magnified image is deteriorated since it is difficult to find similar parts within an original image.
In this paper, we propose a new single image super-resolution method to improve the sharpness of texture in a magnified image.Our proposed method uses total variation (TV) regularization (6) to generate high frequency components.The super-resolution based on total variation regularization (SRTV) has been proposed (7,8,9) .Although SRTV can improve the blur of edges in a magnified image by separating texture and other components by TV regularization, it has a similar problem as the other single image super-resolution.Our proposed method uses an only input image to generate high frequency components.Specifically, it applies TV regularization with a negative parameter for the input image to generate diverging components.We show experimentally that high frequency components by diverging TV regularization can improve sharpness of texture of the magnified image.However, if the parameter of the TV regularization to the input image is fixed, visual distortion is observed except for the texture region in the magnified image.Therefore, we propose a determination method for optimum parameter to each pixel.

Super-resolution based on TV regularization
The super-resolution based on total variation regularization (SRTV) extends ROF (6) to generate called up-sampled skeleton image U as follows: Here,  , ↓ which is obtained by down-sampling  , , has the same dimension as  , , the original low-resolution image. and  represents the 5x5 Gaussian impulse signal and equilibrium parameter, respectively.Also, * is the convolution operator. and  are the numbers of rows and columns, respectively.Eq. ( 1) can be solved using an iterative calculation (10) similar to the process of ROF, known as TV up-sampling.
Figure 1 shows the framework of the simplest SRTV, which uses an input image only.TV up-sampling and TV regularization are applied to  , to generate  , and  , respectively.Here,  , is a texture image is obtained by subtracting  , from  , .The Lanczos3 interpolation is applied  , to generate  , which has the same dimension as  , .A magnified image is obtained by adding  , and  ,

Our proposed approach
The framework of our proposed method is similar to that shown in Figure 1.The key to our proposed method is the negative value of , the TV regularization parameter, which is not generally applied to an input image.For example, Figure 2 shows images magnified by a non-negative and negative  .Figure 2(b) shows an improved sharpness of texture, whereas visual distortion can be observed in other regions.On the other hand, the noise cannot be observed from Figure 2(a), but the sharpness of texture of the magnified image is lower than in Figure 2(b).Therefore,  should be optimized in the spatial region in order to generate high quality magnified images with sharpness texture and low noise in the edges and background.

Determination method for 𝝀
Our proposed method uses the variance of each pixel of the input image to assess complexity.The variance of each pixel of the input image is calculated by ( Here,  ̅ , is the average in a 5x5 window around  , . , are considered to be comprehended features of the input image, including the background, texture and edge region.These regions need different optimum  for an improved magnified image.In Figure 2(a), if  set to 0 at background and edge region, noise cannot be observed the image.On the other hand, setting  to a negative value at the texture region will be sharper this region (Figure 2(b)).Figure 3 shows the optimum value of  according to Here, _ and  ̅ are the variance and the average of pixels, respectively.Eq. ( 3) represents the complexity of the input image.The higher value of  corresponds to the more complex texture.Figure 4 shows distribution values of  , ′ for the Lenna image.The colors of blue, green, white, purple and red denote 0 ≤  , ′ ≤ 5, 5 <  , ′ ≤ ,  <  , ′ ≤ 64, 64 <  , ′ ≤ 128 and 128 <  , ′ , respectively.In

Results and discussion
The calculation of TV regularization in this study will not converge since our proposed method uses negative values of .Therefore, we finish this calculation after 15 iterations.This value was determined experimentally.For a color image, our proposed method applies only to the luminance components while the rest components are simply magnified using bicubic interpolation.The results of magnified images by Glasner et al. (2) are referred to their web site (11) .
Figure 5 shows an original image,  distribution map and comparison of 3x magnified images by SRTV using positive , Glasner et al. and our proposed method.Glasner et al. and our proposed method have a sharper texture of the magnified image than SRTV using positive .In Figure 5, Glasner el at.has little sharper edges in the magnified image than our proposed method, whereas the texture of the feature of stone in the magnified image by our proposed method is better than the rough deposits of stone by Glasner et al. since in Figure 5(b), the value of  at the stone is determined to have a lower value than the background region.
Figure 6 shows an original image and comparison of 3x magnified images by SRTV using positive , Glasner et al. and our proposed method.Our proposed method improves the sharpness of the koala's fur and the tree trunk in the magnified image compared with both Glasner et al. and SRTV.
According to our experimental results, Glasner's method improves the blur edges.However, textures such as stones or animal furs are less improved, since Glasner's method is based on self-similarity, and it is difficult to find a similar part within a given input image to generate a magnified image.On the other hand, our proposed method can improve sharpness of texture in a magnified image, since our proposed method generates high frequency components properly by diverging the TV regularization.

Conclusion
In this paper, we have proposed super-resolution from a single image based on TV regularization to improve sharpness of texture in a magnified image.Setting negative  for the TV regularization generates high frequency components can improve the sharpness of texture in a magnified image.However, may cause the visual distortion at regions of the edges and background.For this reason, we defined the optimum value of  in accordance with the complexity of the input image.
Our proposed method is better than SRTV and Glasner's on producing sharper texture in the magnified images.For our future works, we will improve the image quality in edges using other image features to optimize  for the TV regularization.

Figure 4 ,
Figure 4, the background and texture region contain low and middle  , ′ , 0 ≤  , ′ ≤ 5 and 5 <  , ′ ≤  , respectively.Therefore, in 0 ≤  , ′ ≤ , the  function uses a downward convex quadratic function since low value of  is calculated at the texture region.On the other hand, in  <  , ′ shown in Figure 4, texture and edge region contain middle and high  , ′ ,  <  , ′ ≤ 64 and 64 <  , ′ , respectively.Therefore, in  <  , ′ , the  function uses a convex quadratic function since high value of  is calculated at the edge region.Accordingly, the determination function for the optimum  is defined as