Contact Position Estimation Algorithm using Image-based Areal Touch Sensor based on Artiﬁcial Neural Network Prediction

: In this paper, we propose an artiﬁcial neural network ﬁtting model to estimate contact position an image-based areal touch sensor (IATS) with soft physical contact. First, the principle of the proposed artiﬁcial neural network ﬁtting model for contact position estimation is described. Then, the structure of the IATS for verifying the algorithm is described. Second, an experiment was conducted to verify the model. Experimental data was obtained and analyzed for accuracy. Accuracy is estimated based on the relative error rate to show the accuracy between actual and estimated contact positions. As a result, the accuracy for each axis is as follows. The accuracy for the X-axis is 86.7% on average and the accuracy for the Y-axis is 96.5% on average. The depth accuracy is 94.9%. It is analyzed to solve various problems and it is expected that it will be possible to develop a sensor with accuracy similar to the actual contact position in the future.


Introduction
In the fourth industrial revolution, many companies and researchers are interested in an automation system such as an artificial neural network system or a next generation IoT. Apart from electronic devices, such as smartphones and personal computers (PC), human-friendly intelligent devices, such as wearable devices and humanoid robots have become essential products around us [1]. A wearable device, which is a small and lightweight new generation intelligent device worn on human body or clothes for portable use during movements, can be beneficial to human health by learning variable human biometric data and delivering appropriate information [2]∼ [4]. In addition, there are humanoid robots that perform difficult tasks in place of humans or intelligent robots that are used in industrial fields [5] [6]. These devices often collaborate with humans to support them in a variety of fields. Thus, it is necessary to exchange information quickly, accurately, and reliably between humans and intelligent devices; for this reason, sensors are required as an interface.
Human have five senses: sight, smell, taste, hearing, and tactile sense. All senses can be recognized as body organs such as eyes, nose, mouth, ears, and skin. All these physical signals can be converted to electrical signals through sensors. Sensors are employed to convert human's five senses into mechanical or electronic signals [7]. The sensor is composed of various types such as an image sensor, a contact sensor, a pressure sensor, and a position sensor. It is an essential element in all fields. Sensors are used as essential elements for coexistence and collaboration between humans and intelligent systems. It is a medium for efficient information exchange. Of the human senses, the senses most associated with physical signaling are tactile senses. All tactile senses also begin with contact. Recent researches on sensors related to contact have been proceeding rapidly [8]. In this paper, we use sensors related to new concepts of contact which are different from existing ones.

Research trend of contact sensor
Most sensors are made by chemical or electronic processing methods based on specific semi-conductor materials. A typical touch sensor is a touch screen. When the user presses or touches the screen with a finger or a pen, the position is recognized and transmitted to the system. The touch screen panel consists of a touch panel, a controller IC, and a driver software. The touch panel is composed of a top plate and a bottom plate (film or glass) on which a transparent electrode (Indium Tin Oxide, ITO) is deposited. Detects the signal generation position according to the capacitance change, and transmits the signal to the controller IC. The controller IC converts the analog signal transmitted from the touch panel into a digital signal and converts it into a coordinate form that can be displayed on the screen. The driver software receives the digital signal coming from the controller IC. The touch panel is structured to be implemented according to A resistive touch sensor is a plate-shaped structure of conductive elastic rubber mixed with carbon powder or metal on silicone rubber [9]. It extracts contact position from changes in resistance due to applied force.
A piezoelectric touch sensor is a structure using piezoelectric film, such as polyvinylidene fluoride [10]. It extracts contact position based on the level of flexibility of pressure film according to the deformation shape of the piezoelectric film due to applied force.
Apart from the above sensors, other types of touch sensors, such as capacitive touch sensor can be found [11]. These touch sensors require capacitive coupling to accurately detect touch position.

Purpose of the research
As previously mentioned, the existing contact sensor has an advantage that it can accurately acquire information related to the contact. Nevertheless, they require the design of complex circuits or fabrication processes. Furthermore, it has several weaknesses due to inherent characteristics of semi -conductors, such as rigid physical property, weak durability, and degraded compatibility [12]. This study proposes an imagebased areal touch sensor (IATS) [13] [14] using soft material and image sensor. We propose a contact position estimation model based on artificial neural network. In Chapter 2, the structure of the IATS and a process to create the artificial neural network model are explained. In Chapter 3, a learning process for the artificial neural network and results of contact position estimation of the developed model, as well as performance evaluation through data analysis are described. Finally, in Chapter 4, conclusions are presented.

IATS and ANN Model 2.1 Structure of IATS
The IATS has a touch area made from soft materials, such as artificial leather or urethane rubber. It also has a circle marker to recognize when a change in shape occurs due to physical touch, and an image sensor to capture the change in shape. Therefore, this touch sensor is a new concept with various features that are different from existing touch sensors. Figure 1 shows the IATS used to validate the proposed model in this study. The IATS consists of soft fabric [3] [4], to which several circle markers are attached at regular intervals to function as the touch area, LED [2] that supplies light within the sensor, and a camera module [1] to capture images of the marker. Once a user presses the touch area of the sensor, a camera attached to the lower end receives real-time images of the circle marker, which is moved by an external force. The IATS is thought to have various ad- vantages if the size of hardware can be improved, that is, become thinner by adopting small-size image sensor and wide-range angle lens. It can be applicable to soft 3D input device, soft pressure distribution sensor as well as soft sensor for human-robot coexistence or collaboration. The received images are pre-processed and digitized to estimate contact position through the artificial neural network model created through learning. Table 1 compares the features of an existing touch sensor with that of the ATS.

Extraction of learning data set
It is necessary to perform a data extraction process for learning in order to develop the artificial neural network model. Once images obtained through the camera are inputted, features are extracted using an algorithm for preprocessing to acquire accurate contact position, as well as shortening the processing time. Figure 2 shows a flow chart for the preprocessing algorithm.
To shorten the processing time, a three-channel RGB image is first converted into a single channel image. In general, a single channel image is implemented with the mean of RGB values. However, since the color of the circle marker attached to the IATS is green, a single channel image is created with only green channel among the RGB channels to obtain a sharp contrast between the background and the marker. Next, morphology processing is performed to remove noise. Noise removal and images with single channel should have binarization transform to separate the background from the maker. To do this, a threshold value in terms of brightness is determined using the Otsu algorithm [15] [16]. The Otsu algorithm searches a threshold value in terms of brightness of inputted images. It finds a threshold value between two classes that minimizes dispersion inside a class or maximizes dispersion between classes when image pixels are classified into two types of classes based on histogram distribution. Since the inner structure of the IATS is closed, it is significantly affected by light. When a fixed threshold is used, it is difficult to accurately detect a marker moved by an external force. Thus, if a threshold value that is changed according to circumstance using the Otsu algorithm is used to perform binarization transform, the background and marker can be accurately separated. Next, features (area, x/y-axis coordinate) of each marker can be detected by individualizing all markers through labeling [17]∼ [19]. Once the above processes are performed, a single learning data set can be produced.

Fitting by artificial neural network
An artificial neural network, such as deep learning has been widely-used in a structure where machine learning is required for complex pattern recognition or prediction that shows a correlation and fitting between input and output data, as well as having complex functional relationships while depending on a large amount of learning data [20]∼ [22]. The IATS also aims to estimate contact position using images, and its midprocess is quite complex. When a touch occurs in a specific position, contact position should be through inputted images. To do this, the correlation between input and output data can be established by means of fitting using artificial neural network. Figure 3 shows a structure of the artificial neural network used in this study. It consists of five hidden layers and one output layer to increase fitting performance, and the learning time was shortened using the Levenberg -Marquardt(LM) back-propagation algorithm, which was optimized for artificial neural network learning [23]. The artificial neural network model created through learning can estimate contact position and depth using input data. Figure 4 shows the experimental setup used in this study. The setup consists of 3-axis linear motor stage to extract learning data through a touch. A touch can be applied to an accurate position using experimental equipment. Figure 5 shows a marker region attached to the inside of the IATS. The number of markers was 117. The first marker located in the left upper end was set to a relative coordinate of 0, 0 (mm), and the last marker located in the right lower end was set to a relative coordinate of 60, 45 (mm). A total of 256 learning data sets can be created if touches were made in the X-axis with 4 mm increment 16 times, and in the Y-axis with 3 mm increment 16 times. Furthermore, learning data sets were created by differentiating the depth since the area and coordinate of all markers were different if the same position is touched according to the depth. Through the above procedure, learning datasets were created, and the number of input data neurons was set to 351, and the number of target data neurons was set to 3, thereby creating the artificial neural network model through learning. Figures 6 and 7 show fitting results of training and test- ing data. In the figures, R refers to a relationship between output and target data. If R is close to 1, the relationship between the two data sets is close to a linear relationship. Thus, the figures show that fitting was properly implemented through learning. The learning time to create the artificial neural network model was approximately 13 h and 20 min. However, the processing time of input and output in the model created through learning was 0.0058 sec. Next, the accuracy of the artificial neural network model was verified. While touching with constant movements in the horizontal or vertical direction based on an arbitrary reference position, the accuracy in terms of the coordinate and the depth was analyzed. Figures 8 and 9 show the actual contact positions, and contact positions estimated using the model. Based on the relative coordinate 4, 30 (mm), the results of the contact position that was moved by 4 mm in the right direction in the X-axis are shown. Table 2 presents numerical data and errors between the two data sets. The  Table 2 means the gap between the two positions. The maximum gap is 3.24mm. Also, the mean error in terms of the depth was 0.2 mm.  Figure 10 and Table 3 show the actual contact position and the estimated contact position when random contact at any position. In addition, it can be seen that the size of the circle indicating the contact position is changed in size of the depth. Therefore, it is easy to visually ascertain the difference in depth.

Conclusions
This study proposed an artificial neural network learning and model for contact position estimation using IATS. Con-  tact position was estimated by substituting feature data with an artificial neural network model after preprocessing of input images.
Based on the experimental results, the accuracy of the proposed algorithm is analyzed. The relative error rate is analyzed based on the error for each axis. As a result, the accuracy of the X-axis is 86.7% on average. The accuracy on the Y-axis is 96.5% on average. Also, the accuracy of depth is 94.9% on average. The error rate for determining the accuracy is as follows. The maximum error rate on the X-axis is 67.5% and the maximum error rate of Y-axis is 10%. Finally, the maximum error rate for depth is 10%. The accuracy of the X-axis is more than 85% on average, but the maximum error rate is 67.5%. Analysis of these problems has led to such problems when the data generated for learning is very similar to other learning data. IATS is a structure that obtains contact information with minute movements of markers. Therefore, if the artificial neural network model can't make an accurate judgment, I think that there is a large error in a specific area. To solve these problems, it is necessary to develop an algorithm that can classify similar data according to cases. If the following problems are solved, the overall accuracy and the error rate will decrease compared to the existing models.
The artificial neural network fitting model using the proposed IATS can perform contact position estimation only within the marker region. Therefore, it is planned to further improve contact position estimation in a wider area by increasing the number of markers and the region by means of miniaturization of the IATS in the future. Furthermore, this study found that the functional relationship was relatively simple and the processing speed was fast with high accuracy compared with touch position estimation using image processing in previous studies. Thus, if the error in the touch position can be reduced to as low as possible with improvements in the training algorithm and appropriate parameter setup, the proposed contact position estimation can be combined with other intelligent systems. To verify the accuracy of the proposed method, the comparison with a conventional sensor such as capacitance type is also one of our future work.