Development of Finger Touch Position Detection System Methods Using Single Camera

We propose a new image processing-based finger touch position detection sensing method using a reflected image reflected on the screen. Using single camera by image processing to detect touch position has several significant advantages: (1) Installation of camera becomes easily (2) This system can reduce the failure rate to realize maintenance free operation. (2) This approach enables easy attachment and low-cost touch sensing. The problem when using single camera is impossible to detect the depth of the finger from capture image. In order to solve this problem, we use the reflected image of the finger appears on the screen. Detecting the fingertip and the reflected image on the screen effectively, we enable to detect touch position only using single camera. In this paper, we provide concrete method to extract fingertip and reflected image, implementation details, results of evaluation experiment, and outline future works.


Introduction
This work has focused on the finger touch position detection methods for Image processing-based touch sensing systems. The image processing-based touch sensing system is the new method that recognizes the action of the finger by using cameras. Image processing-based touch sensing system is used to large touch screen for TV weather forecast, Digital Signage. Image processing-based touch sensing systems has advantages: (1) the surface or bottom sides of touch screen need not be instrumented like resistive membrane-based systems and capacitive-based sensing systems. (2) this approach can convert non-touch screen into touch screen. (3) this approach enables various operations not only by touch but also by gesture operations.
However, most of the researches of image processing-based touch sensing system are using a plural camera to detect the position of the finger by using the triangular surveying.
Therefore, this method of touch screen has some weak points: (1) installation location of camera is limited because distance is required between the cameras (2) it is necessary to adjust the optical axis precisely of multiple cameras. Moreover, it is not possible to determine the depth (distance between finger and the screen) from the triangular surveying. From this reason, strictly speaking, the triangular surveying based image processing based system cannot recognize the touch operation.
Therefore, we propose finger touch position detection methods using single camera. The problem when using single camera is impossible to detect the depth of the finger from capture image. In order to solve this problem, we use the reflected image of the finger appears on the screen. Recognizing the fingertip and the reflected image on the screen effectively, we enable to detect touch position only using single camera. Using single camera by image processing to detect touch position has several significant advantages: (1) installation of camera becomes easily (2) this system can reduce the failure rate to realize maintenance free operation. (2) this approach enables easy attachment and low-cost touch sensing.
In this paper, we provide concrete method to extract fingertip and reflected image, implementation details, results of evaluation experiment, and outline future works. Figure.1 shows the concept of proposed system. A camera establishes in the range of 20~70 degrees, so that the reflection picture of the finger is reflected to the screen. From the camera image, recognize the fingertip to get x-coordinate and y-coordinate by using projective transformation matrix and recognize fingertip and reflected image to get z-coordinate. If coordinates of fingertip and reflected image overlapping, determine finger has touched the screen. Capture a single view image includes fingertip and reflected image from the camera fixed at the top of the screen.

2.
From the capture image, recognize fingertip and reflected image.

3.
Get fingertip and reflected image coordinates.

4.
From two obtained coordinates, determine overlap of two coordinates. 5.
If coordinates of fingertip and reflected image coordinates overlapping, determine finger has touched the screen. 6.
After detecting fingertip, convert the coordinate of fingertip on the camera into the coordinate on the screen. 7.
Output the result.

Fingertip and the reflected image recognition methods
The most important part of our system is how to recognize the fingertip and the reflected image appear on the screen in high precision. In this paper, the method of recognizing the fingertip and the reflected image based on Dynamic background subtraction method and Skin color filter. Figure 3 shows the flow of the flow of extraction of fingertip and the reflected image. If the brightness value I in ′ − σ − φ ≤ I ≤ ′ + σ + φ determine this pixel as the background pixel. In background region, update the average brightness value 2 ′ and amplitude σ ′ as equation (2).
n is a update rate parameter. Moreover, brightness average value 2 ′ retain the original value in the pixel of object region, Then, updated only the amplitude of brightness σ ′ as equation (3).
m is a update rate parameter in the object region. Skin color filter is a filter using the HSV color space, and sets the threshold H, S, V to extracts skin color regions. HSV color space is composed of three components: Hue, Saturation, and Value.

Evaluation experiment
In order to confirm the effectiveness of the proposed method, we conducted an evaluation experiment.
In the evaluation experiment, we tested three kinds of methods to extract the fingertip and reflected image. (1) dynamic background subtraction method, (2) skin color filter (3) dynamic background subtraction method + skin color filter. Figure 4 shows one scene of the camera image.

Consideration
From the experimental result of evaluation experiment 1, Shadow is also extracted at the same time when only the dynamic background subtraction. From this result, we cannot to determine the finger only by the dynamic background subtraction.
From the experimental result of evaluation experiment 2, the fingertip is extracted, but the color close to skin color or other regions are also extracted by using only skin color filter.
From the experimental result of evaluation experiment 3, both the fingertip and reflected image are extracted very clear.

Conclusion
In this study, we propose a new image processing-based touch sensing method using a reflected image reflected on the screen. We carried out experiments with a combination of the dynamic background subtraction method and skin color filter. As a result, it was possible to extract clean fingertip and reflection image on the screen.
However, the threshold value of the skin color filter is adjusted by manual operation now. We need to re-adjust the threshold value by the environment. Therefore, we would like to create like the dynamic color filter to change the threshold automatically.