Study of the Finger Detection Technology for the One Camera Touch Panel

Nowadays, touch panel has been widely use in various fields. Most of the daily use touch panel equipped devices such as a smart phone, car navigation systems using resistance film method and capacitance method. However, the sensor covered touch panel surface has a limited size and the sensor might be easily damaged. Especially for a hard environment tolerate device, instead of the traditional approach, the image processing method touch panel has been studied in recent years. By recognizing the touch event and detecting the fingertip coordinates when the touch event happened, most of image processing methods use multiple cameras. But such devices have some weak points such as the big size and the low reliability comes from complex construction of the device. This study propose a one camera touch panel system which can detect the finger position and touch event at the same time, furthermore it supposed has a high reliability obtains by simple structure compact size of one camera.


Introduction 1.1 Background
The touch panel is a device that can perform the operation of the computer by touching directly on the display.By integrating sensors for touch operation detection on the display unit, it is possible to sense the position of the finger or the pen touching the screen and submit an instruction to the computer.Since you can touch by directly touching the icons and buttons displayed on the screen, the operation method is easy to understand, and it is easy to use even for those persons who unfamiliar with computer operation.By integrating the display and the input panel into one device, it is possible to downsize or simplify the whole device.Moreover, since there are no hardware buttons, there is a feature that operability can be flexibly changed by software.
In recent years, the touch panel of the image processing system has been studied.Most of the approach are recognizing the fingertip coordinates in three dimensions by using multiple cameras, it can detect whether a fingertip is in any position of the panel.However, the touch panel of the conventional image processing method has identified the position of the fingertip in the manner of stereo vision.Since stereo vision requires two or more cameras, the setting conditions of the camera are severe, and the burden of image processing is larger than when one camera is used.Therefore, the touch panel recognition system by one camera have been important.

Purpose
The purpose of this study is to consider a novel finger detection technology for the touch panel with the image processing system using one camera.This time, we use the detection techniques of the template matching and optical flow.

One camera touch panel
One camera touch panel is a touch panel using one camera and one panel.A camera is attached to the top of the panel, and fingers of real images and fingers of reflected images appearing on the panel are detected using image information obtained from the camera.Fig. 1 shows overview of one camera touch panel.
There are three merits by using one camera touch panel.1, Simplification of equipment 2, Loss reduction in failure 3, Huge panel area

Template matching
One of image processing applications is object detection.Template matching is often used to find a specific pattern from images.In this method, the position of the template present in the image is found by comparing the region of interest while moving the template on the image.

Optical flow
The optical flow is a technique for predicting to which pixel each pixel has moved between adjacent frames in a moving image.Since the optical flow varies with the movement of the object, information on speed can be obtained.Optical flow is mainly used for detection of moving objects and analysis of their movements.Fig. 3 shows overview of optical flow.

Coordinate transformation
Figure 4 shows the image of coordinate transformation.In the case of the 1 camera touch panel used in this research, it is necessary to attach the camera to the top of the panel.Therefore, when viewed from the camera, the panel appears like a trapezoid, and the actual coordinates on the panel and the coordinates on the camera are different.In order to correct this state, a coordinate transformation algorithm is used.This method is based on the idea of projective transformation.The coordinate transformation algorithm is shown in equation ( 2 The transformation parameters a, b, c, d, e, f, g, h can be represented by determinants using the eight previously known coordinates.(2.4.2)

Experiment environment
This study used raspberry pie 3. Similarly, this study used a raspberry pie camera module for the camera.Fig. 5 shows one camera touch panel used in this study.

Finger detection using template matching
First, using a template matching as a finger detection technology.In this method, the fingertip portion in the real image and the reflected image was taken as a template image.We made matching between these template images and the input images from the camera.When the frame of the real image part exceeds the frame of the reflection image part, we set it to judge as touch.Fig. 6 shows the template image used in this method.

Finger detection using optical flow
Second, using optical flow as a finger detection technology.Optical flow can obtain the motion of an object as a vector.In this method, fingertip detection is performed by using the motion vector of the finger in the input image from the camera.When the finger approaches the panel, the reflected image moves in the opposite direction.By using this fact, it is presumed that there is symmetry in the motion vector.We thought that fingertips could be detected by extracting the tip of the vector of each direction.Moreover, in this method, pixels with the largest vector size towards the panel were detected as the tip.

Finger detection using template matching
Firstly, we verified "Method of comparing template and image area" prepared by OpenCV's template matching function.We decided to find an optimal method from the following three.

1, CV_TM_SQDIFF_NORMED 2, CV_TM_CCORR_NORMED 3, CV_TM_CCOEFF_NORMED
In each method, the relationship between the distance from the camera to the finger and the degree of matching is summarized in Table 1.SQDIFF_NORMED indicates that the value is closer to 0, and CCORR_NORMED and CCOEFF_NORMED indicate that the degree of matching is higher when the value is closer to 1.
In all three methods, the part other than the finger was often erroneously detected as a finger.Therefore, we set a threshold to judge whether it is a finger or not.In this method, we decided to use CCOEFF_NORMED with the greatest change.We set the threshold as 0.78 for both real image and reflected image.As a result, it was specified as a touch only in the 5 x 5 cm square area of the panel.From now on, we decided to conduct study under these conditions.Fig. 7 shows the matching result when touching, and Fig. 8 shows the matching result before touch.Both of them can detect two fingertips of real image and reflected image.
We obtain the coordinates at the time of the touch, and converted using a conversion algorithm from the coordinates on the camera to panel coordinates and outputs the result.Fig. 9 shows the output result of drawing a line at the touch.
In the four patterns shown in Fig. 10, fingers could not be detected.

Finger detection using optical flow
Fig. 11 shows the output result of the vector.White is a vector in the direction in which the finger of the real image moves toward the panel, and red is a vector in the direction of the finger of the reflected image toward the panel.
We thought that the motion vector of the finger became symmetrical between the real image and the reflected image, but in reality, vectors in both directions were seen toward the real image.Therefore, we limited the detection range of the vector to the vicinity of the panel.Fig. 12 shows the result.As a result, only vectors near the fingertip can be detected.Finally, we detected fingertip coordinates from this output vector.Fig. 13 shows the detection results.

Conclusions
In this study, we combined template matching and optical flow as the finger detection technology.Template matching was able to be detected stably under the condition of 5 x 5 cm square at the center of the panel.However, the detection range was limited by the finger size of the template image.In the failed area, the optical flow, symmetric vectors could not be obtained between the real image and the reflected image.Therefore, it was not able to detect the fingertip area and touch event stably.Both of the methods are susceptible to the influence of light and need improvement in order to be practical.In the future, it is necessary to study optimal detection techniques using other methods such as object tracking.
Figure4shows the image of coordinate transformation.In the case of the 1 camera touch panel used in this research, it is necessary to attach the camera to the top of the panel.Therefore, when viewed from the camera, the panel appears like a trapezoid, and the actual coordinates on the panel and the coordinates on the camera are different.In order to correct this state, a coordinate transformation algorithm is used.This method is based on the idea of projective transformation.The coordinate transformation algorithm is shown in equation (2.4.1).The parameters N x and N y operate as shift parameters, and the parameter k operates as a scale parameter.The converted fingertip coordinates X′ and Y′ can be calculated using the original coordinates x and y, the transformation parameters a, b, c, d, e, f, g, h, and the correction parameters k, N x and N y .(2.4.1)

Fig. 5 .
Fig. 5.One camera touch panel used in this study.

Table 1 .
Distance from camera and matching degree.