Generation of Mottled Images Using Smoothing Filtering and Extension / Return Processing

We propose a non-photorealistic rendering method for generating mottled images from photographic images. Mottled patterns are mixed with different colors and shades of the same color, and are irregular in how to change colors and shapes of the patterns. Mottled images are expressed by overlapping the mottled patterns on photographic images. Our method is executed by an iterative calculation using smoothing filtering and extension / return processing. Our method has an advantage that mottled patterns can be automatically generated according to the luminance of photographic images. In addition, our method can change the size of mottled images by changing the value of the parameters. In order to verify the effectiveness of our method, we conducted experiments using various photographic images. As a result of the experiments, we clarified that the two advantages can be reproduced.


Introduction
Many non-photorealistic rendering (NPR) methods have been proposed that imitate patterns formed in human society and nature and generate images of new expressions [1]∼ [5]. Specifically, there are non-photorealistic images that formulated antecedent terrain rendering techniques including inclined contours and hachures [1]. Then, there are maze images generated using minimum spanning trees [2], moire images generated using staircasing effect bilateral filter [3], reaction-diffusion-pattern images including papercut, stylized halftone and motion illusion [4], and bubble images generated using additive and multiplication averages in different window sizes [5]. These non-photorealistic images are impressive.
With the spread of smartphones and social networking service in recent years, many applications incorporating NPR have appeared, and various NPRs that have never existed are required. Therefore, in order to develop one of NPRs that has never existed, we propose an NPR method for generating mottled images from photographic images in this paper. Mottled patterns are mixed with different colors and shades of the same color, and are irregular in how to change colors and shapes of the patterns. Mottled images are expressed by overlapping the mottled patterns on photographic images. According to our search of the literature, no previous research has studied the generation of mottled images in NPR.
Our method is executed by an iterative calculation using smoothing filtering and extension / return processing. Our * Corresponding: hiraoka@sun.ac.jp † Faculty of Information System, University of Nagasaki Nagasaki-ken, 851-2195 Japan method has two features. The first feature is that mottled patterns can be automatically generated according to the luminance of photographic images. The second feature is that the size of mottled patterns can be changed according to the values of the parameters in our method. In order to verify the effectiveness of our method, we conduct an experiment that examines how to form mottled patterns when the values of the parameters are changed with Lenna image. In addition, we conduct an experiment using various photographic images other than Lenna image, and examine whether mottled patterns can be automatically generated with all photographic images. As a result of the experiments, we clarify that our method can realize two features.

Our Method
Our method generates mottled images from photographic images. Our method is executed in three steps. In the first step, the images are processed by smoothing filtering. In the second step, the images are transformed by extension / return processing. The first and second steps are repeated.
In the third step, it is converted into mottled images using the images obtained in the first and second steps. A flow chart of our method is shown in Figure 1. Details of the steps in Figure 1 are explained below.
Step 0 The input pixel values (R, G, B) for spatial coordi- The pixel values f i, j are calculated using the pixel values f R,i, j , f G,i, j and f B,i, j as follows.
Then, the pixel values of the image at the t-th iteration number are defined as Step 1 The image with the pixel values f (t−1) i, j is processed by smoothing filtering. In smoothing filtering, a moving average filter that calculates the average of the pixel values in the window of W pixels as follows.
where k and l are the positions in the window.
Step 2 The pixel values s (t) i, j are converted by extension processing. In extension processing, the pixel values s (t) i, j are stretched by a constant multiple around 127.5, which is an intermediate value between 0 and 255. The extended pixel values g (t) i, j are calculated as follows.
where a is a constant greater than 1. The pixel values g (t) i, j are converted to values from 0 to 255 by return processing. The returned pixel values f (t) i, j are calculated until they range from 0 to 255 as follows.
The processing of Steps 1 and 2 is repeated T times.
Step 3 The pixel values m R,i, j , m G,i, j and m B,i, j of a mottled image are calculated using the pixel values f (T ) i, j as follows.
In case m R,i, j , m G,i, j and m B,i, j are less than 0, then m R,i, j , m G,i, j and m B,i, j must be set to 0, respectively. In case m R,i, j , m G,i, j and m B,i, j are greater than 255, then m R,i, j , m G,i, j and m B,i, j must be set to 255, respectively. An image composed of the pixel values m R,i, j , m G,i, j and m B,i, j is the mottled image.

Experiments
We mainly conduct two experiments. First, we conduct the experiment with changing the values of the parameters in the proposed method using Lenna image shown in Figure  2. Second, we conduct the experiment using various photographic images. All photographic images used in the experiments are 512 * 512 pixels and 256 gradation. And, the iteration number T is set to 20 that mottled patterns were generated on the whole images.
For reference, images obtained by applying smoothing filter of Equation (2) (W = 2, 3, 4 and 5) to grayscale Lenna image converted by Equation (1) are shown in Figure 3, and the appearance frequency of the pixels in smoothed images  is shown in Figure 4. In Figure 4, the horizontal and vertical axes respectively represent the pixel value and the number of pixels, and the appearance frequencies of the pixels in grayscale image and smoothed images are represented by broken lines. Observing Figure 3, as the value of the window size W is larger, image is more blurred and noise is more reduced. Observing Figure 4, as the value of the window size W is larger, the range of the pixel values constituting smoothed images becomes narrower. First, mottled metal images by changing the window size W are visually confirmed using Lenna image. The window size W is set to 2, 3, 4 and 5. The parameter a is set to 6. The results of the experiment are shown in Figure 5. As the value of the window size W is larger, the size of mottled patterns become larger.
Next, mottled images by changing the threshold a are visually confirmed using Lenna image. The parameter a is set to 3, 6, 9 and 12. The window size W is set to 3. The results  of the experiment are shown in Figure 6. As the value of the parameter a is smaller, the size of mottled patterns become larger and it becomes difficult to visually recognize Lenna image. In the case of adjusting the size of mottled patterns, in order to facilitate visual recognition of photographic images, it is better to adjust the value of the window size W to be smaller and the value of the parameter a to larger.
For reference, Figure 7 shows the enlarged view of mottled image with W = 3 and a = 6.    Tables 1  and 2, respectively. Mottled images are more saturated than Lenna images. The number of the pixels with 0 and 255 does not change greatly depending on the values of W and a. Thus, the fineness of mottled patterns changes without greatly changing the saturation and lightness of entire mottled images.
Finally, our method is applied to four photographic images shown in Figure 8. The parameters W and a are set to 3 and 6, respectively. The results of the experiment are shown in Figure 9. In all cases, mottled patterns can be automatically generated on the whole image.

Conclusion
We proposed a novel NPR method for generating mottled images from photographic images by an iterative calculation using smoothing filtering and extension / return processing. In order to verify the effectiveness of our method, we conducted experiments using various photographic images. As a result of the experiments, we revealed that mottled patterns can be automatically generated according to the luminance of photographic images. In addition, we revealed that the size of mottled patterns can be changed according to the values of the parameters in our method.
A subject for future study is to expand for applications to videos and three-dimensional data.
Toru Hiraoka (Member) He received B.Des., M.Des. and D.Eng. degrees from Kyushu Institute of Design in 1995, 1997 and 2005, respectively. He is currently a Professor in University of Nagasaki. His research interests include nonphotorealistic rendering and disaster prevention.