Comparing Interpolations in Frequency Domain and Improvement of Interpolated Images by Using Cube of Pixel Difference

We propose a method for improving image resolution. The method consists of an interpolation and a sharpening process. We research about some interpolations for producing high resolution images and show the properties of the interpolations in the frequency domain. Based on the research, we selected Lanczos function for the interpolation and applied the sharpening method which uses weighted sum of cubic operations of the pixel differences to the interpolated image. Experimental results also show that the sharpening with Lanczos interpolation gives clear high resolution images.


Introduction
Improving image resolution is a process for making a high resolution image from a low resolution image. For improving image resolution, some interpolation has to be applied and qualities of resultant images are dependent on the interpolation.
In order to produce a high resolution image which has extra-high frequency components, a method using cube of high pass filter has been proposed [1]. In the paper, the cube of Laplacian filter is applied to video images to produce high resolution images of them and the validity of the method are shown for producing high frequency components.
For sharpening interpolated images, we proposed a sharpening method which used weighted sum of cubic operations of the differences between a value of pixel and values of the surrounding pixels [2]. In the paper, we showed that the method produced a sharp high resolution image which was more similar to a target image than that by any other sharpening method. In the process, the bilinear interpolation was used for enlarging original small size images and the sharpening method was applied the interpolated images.
In this paper, we use another interpolation and apply the sharpening method. Needless to say, since the resultant images are dependent on the interpolation, we research about some interpolation methods used in the field of image processing and show the properties of the interpolation methods in the frequency domain. Based on the research, we select and use an interpolation and then apply the sharpening method to the interpolated images.
Experimental results are also shown that the sharpening with the selected interpolation gives clear high resolution * Corresponding: ono@design.kyushu-u.ac.jp † Faculty of Design, Kyushu University, images which preserve the information in the original images adequately.

Properties of Interpolations 2.1 Interpolation of Sampled Points and Convolution
An interpolation of sampled points is accounted to be a convolution with a function for the interpolation. In this section, we describe the properties of some well-known interpolating functions in the frequency domain.
For the convenience, we assume a 1 dimensional sampled data set { f (i); i = 0, · · · , N − 1} , such as shown in Fig. 1. This data set is assumed to be obtained by sampling an original analog function with sampling distance 1. According to the sampling theorem, this sampling is equivalent to a filtering by an ideal low pass filter with band width −1/2 < ω < 1/2. In other words, the sampling data set has information on −1/2 < ω < 1/2 in the frequency domain.

The Bilinear Interpolation
As shown in Fig.2  (a), an interpolated value f L (x) at x between points i and i + 1 by the bilinear interpolation is calculated by where h L (x) is described by The trigonometric function (2) is an interpolating function for the bilinear interpolation and is shown in Fig.2(b). Figure 2(a) illustrates a convolution process by eq.(2). The Fourier transform of eq. (2) is given by (3) Figure 3 shows power spectrum of H l (ω), i.e., a transfer function for the bilinear interpolation. For the comparison, the ideal low pass filter concerned with the sampling of original data set is also illustrated in Fig.3.

The Bicubic Interpolation
As shown in Fig.4(a), an interpolated value f B (x) at x between points i and i + 1 by the bicubic interpolation is calculated by where weight coefficients c 1 , c 2 , c 3 , c 4 are calculated by respectively and   The function (6) is an interpolating function for the bicubic interpolation and is shown in Fig.4(b). Fig.4(a) illustrates a convolution process by eq.(4), eq(5) and eq.(6). The power spectrum of Fourier transform of eq.(6) is shown in Fig.5 with the ideal low pass filter function.

The Nearest Neighbor Interpolation
An interpolated value f N (x) at x by the nearest neighbor interpolation is calculated by where [·] designates to count fractions of 5 and over as a unit and disregard the rest. As shown in Fig.6(a), this procedure is equivalent to the convolution with a rectangle function The interpolating function for the nearest neighbor interpolation is shown in Fig.6(b). The Fourier transform of eq.(8) is Sinc function given by which is shown in Fig. 7.

The Lanczos Interpolation [3]
Lanczos function is often used for image interpolation and generates better results than those by the bilinear interpolation. Lanczos  function is an approximation function of Sinc function and described by If we set n = ∞ , eq. (10) is equivalent to Sinc function Lanczos functions with n = 2, 3, 5 are shown in Fig.8 and the power spectrum of those functions are shown in Fig.9 respectively. The larger n is, the more similar Lanczos interpolation and the ideal low pass filter are.

Interpolation by the Ideal Low Pass Filter
The input sampled data set has spectrum on −1/2 < ω < 1/2  in the frequency domain. Expanding spectrum in the frequency domain with preserving the spectrum on −1/2 < ω < 1/2 and applying invers Fourier transformation to the expanded spectrum, we have an interpolated values of the input data set. However, the ideal low pass filter yields oscillation which is called Gibbs oscillation [3] shown in Fig.10. For a set of sampled function shown by blue points, an interpolated function oscillates around a left side edge . This interpolation is accomplished by a convolution of the input sampled data with eq.(11) in the real space domain. This interpolating function for the ideal low pass filter is Sinc function shown in Fig.10. The convolution by a Sinc function tends to yield some oscillations around step changes in the interpolated result.

Comparison of Interpolations
In order to preserve the information of sampled data set, whole spectrum should be held. The sampled data set has spectrum on −1/2 < ω < 1/2 in the frequency domain. Only the interpolation by the ideal low pass filter holds whole spectrum. However in general, as we mentioned previous section, the ideal low pass filter yields oscillation around abrupt changes of data.
On the other hand, other interpolations lose some information since the transfer functions of the interpolating functions fall gradually near the boundary ω = ±1/2 of   Since the attenuation of the bilinear interpolating function starts near the origin in the frequency domain, the degradation of result is the largest. The bicubic interpolation has better property in the frequency domain and gives better result than that by the bilinear interpolation. However, a remarkable change around x = 1 in the bicubic interpolating function tends to yield overshoots in the results.
In Lanczos function, the attenuation in the frequency domain and the oscillation can be controlled by the parameter n. Since the oscillation is difficult to remove afterword, we apply an interpolation with suppressing the oscillation and then improve the blur afterword for producing high resolution images. Based on these consideration, we select Lanczos function with n = 2 for the interpolation.

Sharpening for interpolated image 3.1 Unsharp masking by liner operation
The fundamental idea of unsharp masking is to add high frequency components to the input image. In other words, the input image is modified by its own high frequency component.
Given an interpolated image {d i j (i = 1, · · · , M; j = 1, · · · , N)}, the linear unsharp masking (LUM) produces an enhanced image f Li j expressed by where w lm is Gaussian filter In eq.(12) and eq.(13), α(> 0) and p are parameters which control smoothness and size of mask respectively. Furthermore we control the sharpness by a parameter δ(> 0) .

Sharpening by cube of liner masking
In the process of unsharp masking, if we adopt the cube of the difference as the high frequency components, slight changes in the image can be enhanced and detail information becomes more visible effectively. According to this idea, an enhanced image f Ci j is produced by This method is, so to speak, the cube of LUM. Using the cube of Laplacian as the high frequency component instead of second term in eq.(14), the sharpening procedure is equivalent to that in the method proposed by S.Ghoshi [1].

Sharpening by using cube of pixel differences[2]
We proposed a sharpening method which was not cube of linear filter but uses weighted sum of cubic operations of the differences between a value of pixel and those of the surrounding pixels. By the method an enhanced image f Di j is produced by This method also enhanced slight changes effectively. However, since the value of improvement often becomes extremely large, we suppress the value of improvement by setting a limit value L. The limitation should be applied to the cube of LUM with eq. (14).

Experiments of Making High Resolution Images 4.1 Comparison of Interpolated Images
An original image, shown in Fig.12, consists of 256×256 pixels with 256 gray levels. Firstly, we interpolated the original image into enlarged images which consists of 512 × 512 pixels by the bilinear interpolation, the ideal low pass filter interpolation, the bicubic interpolation and Lanczos interpolation. Figure 13, Fig.14, Fig.15 and Fig.16 show the interpolated images by the interpolations respectively. In Lanczos interpolation, we set n = 2 to suppress the oscillation and the processing times.
The image in Fig.13 by the bilinear interpolation is blurred. In Fig.14 given by the ideal low pass filter, some oscillations near around image frame can be observed. Interpolated images in Fig.15 and Fig.16 look better than those in Fig.13 and Fig.14.
After interpolating image, it is desired to preserve the frequency components included in an input image before interpolation in order not to change the original image information. On the other hand, the high frequency components which are not contained before interpolation should be created effectively to give a better impression as a high precision image. Figure 17 shows power spectrum of the FFT result for the original low resolution image. Although the frequency domain of the original image is 256 × 256, the components are superimposed in the domain of 512 × 512 to compare with the interpolated images. Figure 17 also shows that the original image does not have high frequency components in the outside of 256×256 low frequency domain. In the graph the values over 0.01 are truncated in order to observe small values of components. Table 1 shows the mean squared differences of power spectrum between the interpolated images and the original       Although an image by the ideal low pass filter interpolation gives no error in this evaluation theoretically, the image may have oscillations. Except for the image by the ideal low pass filter interpolation, an interpolated image by Lanczos interpolation gives smallest value of the difference than those by other interpolations. In addition, differences in high frequency domain mean high frequency components generated by the interpolations. The high frequency com-ponents should be create effectively in order to give a better impression as a high precision image.

Comparison of Improved Interpolated Images
We compared the improved interpolated images by Lanczos interpolation and the bilinear interpolation. For improving the interpolated images, we applied the conventional LUM, the cube of LUM and the cube of pixel differences experimentally.
The images enhanced from Lanczos interpolation are clearer than those from the bilinear interpolation. Although the LUM gave clear texture details, conspicuous edges looks to be enhanced too much with overshoot and undershoot. The cube of LUM with bilinear interpolation (Fig.18(b)) lost the texture details especially in the area of a hat. The image obtained by the cube of pixel difference with Lanczos interpolation (Fig.19(c)) shows clearer details of textures with suppressing excessive edge enhancement such as overshoot and undershoot than those by other improvements.
By improving the resolution of image, the components in higher frequency domain are newly created. We evaluate the components in the low frequency domain included in the original low resolution image and those in the high frequency domain generated by the interpolations respectively. Table 2 and Table 3 show the mean squared differences of power spectrum between the improved images and the original image in Fig.12. For all improved images in table 2 and 3, the high frequency components increase from those of a bilinear interpolated image and a Lanczos interpolated image in Table 1 respectively. This shows that the sharpening is effective for generating high frequency components.
The difference in low frequency domain of a result by the cube of pixel differences is smallest in Table 3. In other words, the sharpening by the cube of pixel differences with Lanczos interpolation is superior in point of keeping information of an original image.
From this experiment, the sharpening by the cube of pixel differences with Lanczos interpolation yields high frequency components with suppressing change of information of original image effectively.

Conclusions
In this paper, we proposed a method for improving image resolution. The method consists of an interpolation and a sharpening process.  We researched about some interpolation methods and showed the properties of the interpolations in the frequency domain. Some interpolations yield oscillation or remarkable overshoot. Since the oscillation is difficult to remove afterword, it is better to select an interpolation with suppressing the oscillation and then improve the blur afterword for producing high resolution images. Based on the research, we selected Lanczos function with n = 2 for the interpolation.
We evaluated the interpolated images in the frequency domain experimentally. An interpolated image by Lanczos interpolation showed better quality than any other interpolations in the experiment. For the sharpening method, we recommend weighted sum of cubic operations of the differences between a value of pixel and those of the surrounding pixels.
In experiments, we applied the sharpening methods, i.e., the LUM, the cube of LUM and the cube of pixel differences, to interpolated images by the bilinear interpolation and Lanczos interpolation. Interpolated images are improved by the sharpening methods. Especially the sharpening by the cube of pixel differences yields high frequency components with suppressing change of information of the original image reasonably.
Experimental results show that the combination of the sharpening by the cube of pixel differences with Lanczos interpolation gives clear high resolution images effectively.