Development of handwritten telecommunication system for the Deaf-blind based on the multi-lateral control

A Deaf-Blind (DB) person has impairment of both vision and hearing. Visual and hearing impairments increase with age. The rate at which persons with vision or hearing impairment become DB is very high. Therefore, it is thought that the DB population will increase as society ages. Currently, DB people are expected to communicate via the Internet in order to participate in society. Research and development of new communication technologies for this population are ongoing. However, most require use of braille, which is difficult to learn after becoming DB. Here, we propose a method for simulating handwritten characters that conveys words to a correspondent by writing characters on the palm with a finger. In using handwritten characters that can be read, it is considered that communication would be possible without special training. The purpose of this study was to develop a handwritten communication system that DB people can use to communicate via the Internet. Haptic two-axis devices capable of force detection and position control currently exist. The telecommunication system proposed here was designed based on multilateral remote control in order to enable haptic communication by many people. Each DB person can operate the two-axis haptic device. Each device is moved to the common position in accordance with the total operating force measured by the force sensor. In experiments, the recognition rate of characters was confirmed for healthy subjects wearing earmuffs and an eye mask. Results confirmed that the recognition rate of characters was more than 80% for simple sentences. Consequently, it was possible to recognize characters with this system.


Introduction
Deaf-Blind (DB) person has impairment of both vision and hearing.They are difficulty to gather information visually and audibly, and different tactile and haptic communication (e.g., tentacle talk and braille) are used because of degree of disability, living environment and individual background.But the diversity of the communication method makes it difficult for the DB to communicate with each other.Also, Visual and hearing impairments increase with age.The rate at which persons with impairment of vision or hearing become DB is very high.Therefore, it is thought that the DB population will increase as society ages.Today, DB people are expected to communicate via the Internet in order to participate in society.Research and development of new communication technology for DB people is ongoing (1)(2)(3)(4) .However, most systems require braille, which is difficult to learn after becoming DB.Here, we propose a method for simulating handwritten characters to communicate by writing characters on the palm with a finger.With handwritten characters, it is considered that communication would be possible without special training.Our image of telecommunication system is shown in Fig. 1.

Fig.1 Image of Telecommunication system
This system can be realized by multilateral telecontrol.Multilateral control that exchanges information bidirectionally on n to m is an extension of bilateral control that exchanges information bidirectionally, one to one.As an example of multilateral telecontrol, a virtual tug-of-war experiment was conducted at eight locations in Japan, and was executed stably despite a time delay of 71 -169 [ms] (5) .The purpose of this study is to develop a handwritten communication system that DB people can use to communicate via the Internet.

Telecontrol system model 2.1 Multilateral telecontrol system
The proposed multilateral telecontrol system is shown in Fig. 2. In the figure, the description of the y-axis is omitted because the difference between x-and y-axis is only the sign.

Fig. 2 Block diagram of shared haptic mouse with multilateral control system
In this system, n clients are connected to the motion server through the network and the scattering matrix.They then interact dynamically with each other.The multilateral telecontrol system is a star type, with the server/client system normally used in a network, and the system as a whole is not affected by the success or failure of a connection between a specific client and server.Therefore, it is easy to expand the number of connections.
In Fig. 2,   () is the client model,   () is the server model,   () and   () are a phase control filter,   () is a phase lead compensator,   () is a phase delay compensator,   and   are communication time delay,   is the force added by the operator,   is the force received at the server,   is the position of the device calculated at the server, and   is the position received at the master.The scattering matrix is the cross section of Fig. 2. The scattering matrix was first introduced into telecontrol by Spong (6) , It was developed by Miyoshi as a method to stabilize the bilateral control system of non-passive linear systems against   ()，  () (7) .Stability of a server/client multilateral control system using a scattering matrix was also proposed by Ogawa (8) .

Client model
The models of the clients   () are admittance-control systems.As shown in Fig. 4(a), the haptic device of clients assumed that the xy sliders could be positioned by the servomotor, and the force sensor was equipped to detect the operating force.The operator can transmit operating force  ℎ to the haptic device and feel the position   by the xy slider, which is moved by position control and sensor.When the transfer functions of the servo motor positioning is   (), the transfer function   () is given by expression (4) in Fig. 4(b).
ℎ [N/m] is the spring constant of the reaction force generated by the rigidity of the human hand when the xy table moves, and  ℎ [N/(m/s)] is a damper coefficient between hands and the xy slider.These are determined by the dynamic characteristics of the human arm.The operator's forces  ℎ are excluded from consideration of stability, because they are extremely difficult to model, and it is considered a disturbance.  () is approximated to 1 because it is assumed that   () has a quicker response than the whole system and will rarely overshoot.Also, on the master side of Fig. 2, the sign of the scattering matrix is positive.This does not indicate positive feedback.Generally, when the xy slider moves in the positive direction, the resistance force caused by the human hand is in the negative direction.Therefore, when   are positive,   are negative signals.

Structure of experimental system
Fig. 5 shows the configuration of the multilateral telecontrol system.The operator side is constructed of the haptic device shown in Fig. 4(a), which includes a control PC, and an AD/DA converter.The operator uses the haptic device and the force added to the device is sent to the server through the network.On the server side, a control PC calculates the dynamical interference and the position of each client is calculated from the sent-force information.The server sends the calculated position information to each client via the Internet.Each client controls the position of the haptic device based on the received position information.In other words, each device is moved to a common position in accordance with the total operating force measured by the force sensor.In this research, we sought to achieve communication by controlling the position of a haptic device with a two-axis drive and presenting the form of characters.However, with only two axes, it was difficult to distinguish between the trajectory of the character shown in Fig. 6 and the moving parts.Therefore, a vibratory device was used to distinguish between the trajectory of characters and the moving parts.Also, a web page was created to visualize the written characters for DB people with low vision, caregivers, and interpreters.PID control was used to control motors with the positions obtained by the position sensor.It was assumed that a human finger was on the force sensor, and this acted as a disturbance.Therefore, a speed loop was added in the position loop in order to improve follow-up.The control block is shown in Fig. 7.

Character perception experiment
Experiments were conducted to check whether a person can recognize (tactilely recognize) characters by presenting characters with the designed device.In this experiment, the trajectories of characters generated in advance by the computer are reproduced by the haptic device and presented to the subjects.At this time, the subject closed his eyes.This experiment is not connected to the Internet.In experimental method, a character is presented to subjects and the presented character is answered.This process is repeated 15 times.This experiment has 3 times per a person.In addition, because of the designed device is only plane drive, the trajectory is set to 50 [mm/sec] and the moving parts are set to 150 [mm/sec] for distinguishing between trajectory and moving parts of characters.There were five subjects.The character used was 24 katakana characters shown in Fig. 8.The character recognition rate of five subjects is shown in Fig. 9.The total result of the five people was 64 [%].However, the third time is 76 [%].It has risen by 20 [%] in the first and third times.This shows that the recognition rate improves by practice.

Tele-communication experiment
It is considered that the recognition rate could be improved by using meaningful small strings of words and sentences; that way, if a word is missed it can be guessed from characters before and after.Therefore, the experiment was conducted to compare recognition rate of characters and sentences.In this experiment, three strings of five characters, and three sentences consisting of 10-15 characters were presented to subjects by tactile perception.In the case of sentences, the recognition rate was obtained by dividing the number of characters in the sentence by the number recognized.The katakana characters used don't have small characters.Fig. 10 shows examples of the sentences used.The average recognition rate of characters and text is shown in Fig. 13.This graph indicates that ANCT and TNCT tended to have a higher recognition rate for sentences than characters; no such difference was observed at NNCT.It is considered that recognition rate is greatly affected by the communication environment.NNCT did not have a significant difference in character recognition rate despite a poor communication environment.So it is additionally thought that the variation in data is greatly affected by individual differences.The relationship between the recognition rate and the frequency of the result of the recognition rate is shown in Fig. 14.From this graph it may be seen that in simple sentences, more than half of the people had a recognition rate of 80% or more.

Conclusions
In this study, we developed a telecommunication system to simulate handwritten characters using multilateral telecontrol for the DB.We then conducted experiments to verify the system.The recognition rate of characters and sentences were evaluated and compared.The recognition rate of text was about 80%, indicating that it is possible to perceive tactile characters by force.The current haptic device has two degrees of freedom (DOF), and it was impossible to distinguish the trajectory of the character and the moving part.For that reason, discrimination was achieved by a vibrating device.In future, we will attempt to improve recognition rate by using a haptic device with 3 DOF, which is closer to actual handwritten characters.Also, special equipment was used in this system.Simplification of system configuration will be an important task toward practical application.
has  inputs and  outputs for each axis.Passive systems   () ( = 1, 2, . . .) are connected to the individual terminals of passive systems   (), which have multiple inputs and outputs, respectively.Specifically, Fig. 3 (b) shows how the operator's forces   [N] are input to the mass point with  [kg] mass through the spring   [N/m] as inputs.The position   of the spring end is the output;   and   belong to the local coordinates   that make angles   of   = cos   from the global coordinate   .It is also assumed that global coordinate   and   are orthogonal.  is assigned to client  in advance, and  1 , = 1;  2 = −1 means that those are arranged opposite to each other.Furthermore, it is assumed that the mass point receives the resistance of the viscosity coefficient  [N/(m/s)] for the   directions in proportion to the velocity;   are determined by the influence of all inputs ( 1 ,  2 , . . .,   ) of each axis.A physical image depends on the relationship between pushing force and sinking quantity when a ball with moderate elasticity is pushed from its surroundings.The model is expressed by equation (1).

Fig. 3
Fig.3 Block diagram and physical model of multilateral

Fig. 4
Fig. 4 Photograph and block diagram of the client system

Fig. 5 Fig. 6
Fig. 5 Configuration of the server client system

Fig. 8
Fig. 8 Katakana used in the experiment

Fig. 10
Fig. 10 Examples of the sentences

Fig. 12
Fig. 12 Connection status of experiment

Fig. 13
Fig.13 Recognition rate of each NCT

Fig. 14
Fig. 14 Relationship of frequency and recognition rate