Development of Learning Support System for Fingerspelling by Augmented Reality

Our purpose of this study is to develop a learning support system for hearing-impaired people who learn sign language and fingerspelling alone. Fingerspelling is one type of the visual language which is used among hearingimpaired people as a communication tool. One fingerspell letter is represented by the shape of communicator’s hands, and it corresponds to one letter of a writing system. Because of the simple structure of fingerspelling, it is used for proper nouns and some words which are difficult to express as a native word of sign language. In general, learners of fingerspelling refer to photographs or illustrations. These learning materials are described from the opposite point of view of learners in most cases. Therefore, the best way to learn fingerspelling is to take lessons conducted by a teacher who has already mastered it. It is desirable for learners to learn anytime they want. However, it is difficult to find the best schedules for both teacher and learners. We propose a new learning support system using augmented reality. Our system is implemented using smart glasses. In this system, finger shapes are displayed on the internal display of the smart glasses so that user can be given a sense that the user is looking at the fingerspelling shape in the real world, and thus, the user can grasp the finger shapes intuitively. In this paper, we show how the system works, and evaluate the efficiency of learning fingerspelling by using this system.


Introduction
We describe a new learning system using augmented reality.This paper aims to develop a learning support system for hearing-impaired people who learn sign language and fingerspelling.Hearing-impaired people can learn fingerspelling alone with this system even if they are at home.As a result, we believe that they can acquire the skills needed for sign language and fingerspelling in a short period of time.
We first provide a model for the learning support system for Japanese fingerspelling in this study.Fig. 1 shows an example of Japanese fingerspelling.In general, the instructor teaches fingerspelling to the learner while showing a real example by using the flat photographs or illustrations as shown in Fig. 1.In a situation where instructors and learners are near each other, the possibility that they learn fingerspelling incorrectly is low.However, when a hearingimpaired person learns fingerspelling alone, it is difficult to form the correct shape.A reason for this difficulty is that the pictures are drawn from just one viewpoint.When there is no instructor nearby, it is currently difficult for hearingimpaired people to learn sign language and fingerspelling correctly by themselves.In order to solve such a problem, it is necessary to develop teaching materials that can be used without any instructor.Therefore, we have made a prototype learning support system using augmented reality as a new learning method.This system has the following features.
1.The learner can observe fingerspelling from various viewpoints.2. The learner can watch hand movement used in fingerspelling through animation.With these features described above, independent fingerspelling learning can be realized by hearing-impaired people even if there is no instructor.

Method
In this section, we describe augmented reality and this system.

Augmented reality
Augmented reality (AR) is a technology that adds digital information to real images and makes them appear as if they were actually there.There are several types of AR.In this research, we use marker type AR.Marker type AR detects characteristics of the image photographed by a camera and adds the information to it at the moment the feature value of the previously registered marker is detected.

How the system works
We made markers as shown in Fig. 2 as a prototype of our system.It draws one character to be learned at the center of the card.Consider, for example, one card of "i" rotates by 90 degrees in Fig. 2. They almost look like the same shape, so our system sometimes misrecognized these two characters.Therefore, we added a geometric frame around the character to increase the number of features as shown in Fig. 3 and resolved this misrecognition.We made new markers for the 46 kinds of fingerspelling in Fig. 1 in a similar fashion.
Our system recognizes these markers by using the camera, and displays a 3D model of the target fingerspelling above the marker.Fig. 4 shows an example image displayed by our system.
We have implemented the AR processing, the camera and the display part separately to date (1,2) .However, in recent years, the performance of the display has improved, so we use smart glasses (3) embedded with these functions.The reason is that learners can learn with both hands free.Also, it is assumed to be used as a dedicated device for fingerspelling learning.

Results and Discussions
In order to verify the superiority of this system, we performed an experiment on memory, comparing the two cases: learning by using ordinary flat educational materials and learning by using our new system.
Research participants are 10 healthy 19 year old men and women.We conducted the following experiment with them and gathered the number of memorized fingerspelling as data.Five out of ten people used the equipment and the rest of them learned only with illustrations.Neither teaching material had a supplementary explanation to make the Fig. 4. The state of learning fingerspelling by using our system Fig. 3.A marker example of "a" Fig. 2. Marker examples of "i" and "ko" learning conditions as equal as possible.
1. Remember the fingerspelling shown in Fig. 1 for 30 minutes.2. Test how long the participants remember immediately after 30 minutes of memory time.3.After 2 hours, test again.In the meantime let's do your usual tasks so as not to think about fingerspelling.4.After 1 day, test again in the same way.The experiment was carried out in a quiet environment, where the participants are not allowed to hear surrounding sound.The experimenter hands the participant the 46 kinds of markers, which are randomly aligned.They learn fingerspelling for these markers in 30 minutes.30 minutes later, the experimenter pronounces 46 kinds of characters randomly, and the participants show it by fingerspelling.We record the correctness as a result.
The participants then lived normally so as not to think about fingerspelling.The experimenter subsequently tested them again after 2 hours and 1 day to ascertain whether or not they remembered the fingerspelling.These results are shown in Fig. 5 and Fig. 6.Fig. 5 shows the result of learning with our system.It should be noted that there is no result for 2 persons after 1 day because of their absence.Fig. 6 shows the result of learning with illustrations only.In Fig. 5, there is a tendency that the number of fingerspelling after 2 hours tends to be somewhat larger than the number of memories immediately after, but the number tends to decrease in Fig. 6.This is because fingerspelling is easily remembered as images by the AR of our system.However, although the number of memorized fingerspelling after 1 day tends to decrease, it can be seen that by using this system, the number doesn't decrease from immediately after the experiment.
Meanwhile, table 1 shows the average of number of memorized fingerspelling with our system or illustrations.In the cases learning by using this system, the number of memorized fingerspelling immediately after the experiment is 36.5% lower than learning by using illustrations.We believe this is due to the fact that the time length to grasp one fingerspelling using this system is longer than using illustrations.Since this system plays back the process of making the fingerspelling with animation, it will take time to finish watching.From this, it is expected that this system is not time efficient for one letter, but can be effective in terms of easiness of understanding.
Let us now consider memory rate of each fingerspelling.Fig. 7 shows the memory rate of each fingerspelling from the experimental results of 5 persons who carried out experiments with our system.From this graph, it can be judged that the memory rate of "a, i, u, ka, ki, se, ta, te, to, ya, yo" after 2 hours is 80% or more.These characters are relatively easy to memorize.In contrast, the memory rate of "ti, na, ni ne, hu, ma, mu, mo, ya, ru, re, ro, wa, wo, nn" after 2 hours was 20% or less.These characters are relatively difficult to memorize.
In addition, some participants did not learn all of the 46 types of finger letters in 30 minutes by using this equipment.This is a problem in learning fingerspelling by animation  Fig. 5.The result using our system which is a feature of this system.Further, the memory rate of the fingerspelling tends to be low in the left part of Fig. 7.This is considered to be due to the fact that the participants learned after sorting randomly aligned markers in alphabetical order.

Conclusions
We developed a learning support system using AR to facilitate fingerspelling learning.We also conducted a learning experiment of fingerspelling using this system and examined this system.Experimental results showed that in the learning using this system, the number of memories after 2 hours tends to slightly exceed the number of memories immediately after learning.From this, it is thought that this system gives the impression of a fingerspelling strongly to the learner's brain, making it difficult to forget it.Even in the number of memories after 1 day, it did not fall below the number immediately after learning.
We will review the experimental method and increase the data in future research.More specifically, we will discuss the presence or absence of animation after we examine the degree of learning effect by animation.Moreover, with respect to the memory rate of each fingerspelling, it is necessary to make conditions such as learning the randomly aligned markers because it is natural for Japanese to learn Hiragana in alphabetical order.
Furthermore, we consider making the judgement of fingerspelling strict and recording how learners make mistakes.
Healthy adults were the subjects in our experiment, so we will also conduct experiments with young people or hearing impaired people as subjects and verify the usefulness of this system.

Table 1 .
The average of number of memorized fingerspelling