Evaluation of the Posture Recognition Method for the Hand Pose Rally System

The "stamp rally" is an event that participants go the round with predetermined points for the purpose of collecting stamps. They bring a stamp card to these points. They, however, sometimes leave or lose their card. In this case, they do not reach the final destination of the stamp rally. The purpose of this research is the construction of the stamp rally system which distinguishes each participant with his or her hand instead of the stamp card. We have realized our method recognizing a hand posture by the image processing. We have made a simple experiment to confirm the accuracy of our recognizing method by 30 examinees. We have also evaluated accuracy by 7 examinees with the prototype system developed based on the system design.


Introduction
The "stamp rally" is an event that participants go the round with predetermined attractions for the purpose of collecting stamp.The participants have a card distributed by a rally organizer.They bring the card to some check points such as shops which the organizer decides.When they visit a checkpoint, the organizer presses the special stamp, which is installed in each point, on the card.Figure 1 shows an example of the stamp rally.A participant has to bring his or her card to 6 checkpoints.He or she has already visited A, B and C checkpoint.Therefore, 3 stamps have been put on his or her card.Since he or she brings it to D checkpoint, the special stamp installed in D point will be put on the card.In this way, the participants collect all stamps installed in the checkpoints.
In recent years, stamp rally sometimes use a FeliCa card instead of the stamp card (1) .There is, however, a problem in the stamp card and FeliCa card.The problem is that it is likely to lose the card and it is troublesome to take out the card.
The purpose of this research is the construction of the new stamp rally system distinguishing with participant's hand shapes, which we call "hand pose rally system."One of the advantages of this system is that participants do not carry around any card.Another is they present the hand when passing through the authentication checkpoint.We have realized recognizing method with the hand posture by image processing program using OpenCV library.We have, furthermore, evaluated our method recognizing the hand posture.

Hand posture 2.1 The posture number
There are 32 patterns of posture made by bending and stretching of fingers of a single hand.We assigned a posture number for each hand's posture based on the finger Fig. 1.A stamp rally.binary, which is a system for counting and displaying binary numbers with the fingers of a hand.The posture is distinguished by treating the five fingers as a binary number.The stretched finger is treated as "1," and the bended finger is treated as "0." Figure 2 shows some examples of the number indicated by each posture.They are the images of the right back of the hand.The thumb means the place by 2 4 , and the little finger means the place by 1.In the figure, the posture at the upper leftmost indicates "0" because no fingers stretch.The next posture indicates "8" because only the index finger stretches.In the figure, the bit pattern, whose rightmost figure is the lowest rank, is shown on each posture.

Types of posture in our system
Our system captures an image of the back of the participant's right hand.It handles three types of hand posture such as following.
(a) Registered Posture The registered posture (RP) is the posture which the participant presents to our system when the rally starts.If the participants register the different posture each other, our system can identify them.The system requires the participants to present four RPs and stored them in the presenting order.Since a participant selects a RP from 32 types of the hand posture, the system can ideally distinguish 1,048,576 (32 4 ) participants.
(b) Presented Posture The presented posture (PP) means the posture which the participant presents to our system when he or she passes through the checkpoint.The system requires him or her to present four PPs.If the participant indicates the same posture as his or her RPs in the registered presenting order at the checkpoint, it is equal to passing through this checkpoint.
(c) Typical Posture The typical posture (TP) means the posture of the samples that are previously provided on the system.It is used to recognize the PP.We have collected some images for each posture.We initially, therefore, prepared 232 images of hand of some friends for each posture.

Handling the posture in our whole system
We have designed our system with the client-server technique.The client subsystem is assigned at each checkpoint and compares the PP and each TP.
We have decided to store the TP data at the server-side and to deliver them to each client whenever it needs the TP data in order to recognize the participant's hand posture.The quality of the TP data influences the accuracy of our recognizing method.
We thought, therefore, that to include the posture of the participant's hand in the TP data is advantageous to improve the accuracy.In our system, the posture of the participants is also registered as TP when it stored as RP.The participant registers his or her own posture at any check point.The latest TP data, therefore, is in the one of the client PCs.We designed that the new TP data was transmitted to the server PC and was stored on the server.In this way, the latest TP data is held on the server.

Vectors generated from the feature points
Our system recognizes hand posture with the vectors connecting the inter-digital space and the fingertips in the image of their right hand (2,3) .There are some researches which distinguish hand posture based on estimating the fingertips and inter-digital spaces (4)(5)(6) .We have also designed our method to recognize hand posture by detecting inter-digit spaces and fingertips.Our method, firstly, detects the contour of the hand in an image.It generates the convex hull (7) and hollow of contour.It regards the hollow as the inter-digital space.It generates the two vectors connecting from the inter-digital space to fingertip.These vectors are the green arrows and the blue arrows from each inter-digit space in Fig. 3.It also generates the vector connecting the inter-digit space and the center of the wrist.This vector is shown the red arrows in Fig. 3.The three vectors (red one, green one and blue one) is handled as one set.Our system, consequently, identifies participants by recognizing their hand posture.Figure 4 shows an example of our recognition.The top image is a PP.The set of vectors of the rightmost Fig. 2. The number indicated by each posture.image is most similar to one of the PP if our system has only 3 images as TP.Our system, therefore, estimates the number indicated by the PP to be 31.
Our prototype system is shown in Fig. 5.The system consists of an ordinary PC, which has Intel core i7 CPU.A USB camera is connected.The camera is mounted over a table on which a black cloth is laid.An examinee presents his or her hand on the black cloth.The system captures the hand, extracts the sets of the vector, and recognizes the posture of the hand.

Method comparing the vectors
Our system has to recognize the posture of the hand which a participant presents to it.We decided that the system compares a PP with each TP by using the vectors described in above subsection.Two vectors are compared by the cosine of the angle between those as following: where  ⃗ and  ⃗⃗ denotes the vectors in the PP and the TP, and  is the angle between two vectors.This equation is very famous.The direction of vector A is similar to one of vector B when the value of the cosine is close to 1.
Our comparing method selects a set of vectors one by one from the PP and each TP in order from the rightmost set.The selected sets are not just directly compared because the direction of the arm in the PP is different from one in the TP such as Fig. 6.This is reason why our method aligns the direction of the red two vectors as shown in the photo at the bottom right.
Our system compares the blue (green) vector in the selected sets in the PP and the TP after aligning the red vectors.The system calculates the values of cosine in the angle between the blue vectors (green vectors) respectively.It calculates the similarity between the PP and the i-th TP with following: where N denotes the number of sets of vectors in the PP, cos   (cos   ) denote the cosine in the angle between the blue (green) vectors of k-th sets in the PP and the i-th TP respectively.The system, consequently, extracts the TP whose similarity S i is maximum and is higher than a predetermined threshold (0.95).

Experimental evaluation
We evaluated the posture identification performance.We actually recognized each posture of the hand that 30 examinees presented.We calculated the accuracy for each posture.We evaluated our method with 23 types of unconstrained posture.Our system identified a posture of the examinee's hand five times for each posture.We requested him or her that his or her hand was presented from different direction or was tilted, as shown in Fig. 7.
We prepared 232 images for TP in total to recognize 23 types of unconstrained posture.Both images in Fig. 8 indicate "12."The number of the sets of vectors in the right image, however, is different from one in the left image.Our system sometimes misrecognizes the outside hollow of the thumb as inter-digital space such as the right image of Fig. 8.This is reason why we registered both of the images as TP.Table 1 shows the registration of the TP database.This database is categorized TP based on the number of sets of vectors.The posture of no. 12 is actually registered in both category 2 and 3.
Table 2 shows the accuracy for each posture.The posture # means the number indicated by the posture.The total average of accuracy is 57.0%.There are some postures whose accuracy is relatively high.The posture which is recognized most accurately is no.31 posture.This posture has four sets of vectors because all fingers stretch.Since there are few postures that have four sets of vectors, no.31 posture is easy to recognize.There are many postures whose accuracy is not so good.No.5 posture and no.21 posture are very inaccurate ones.Figure 9 shows the example of the error in our identifying.The main reason of this error is that the directions of both blue vectors and green vectors are similar.Our system compares only the direction of each vector.To solve this problem, we have considered that our system must also compare not only the direction but also magnitude of the vector.Figure 10 shows another example of the error.The posture presented by examinee is no.21.The TP of no.21 posture is shown the top image in the figure.The TP has two sets of vectors but the PP has three sets in this case.Another TP, accordingly, is most similar to the PP.

Evaluation based on the system design
We measured the accuracy of our recognizing method with a prototype system, which simulates the TP data handling described at section 2.3.We prepared, firstly, 232 initial images for TP.Eight examinees present 23 kinds of the posture as the TP, and our system stored them.As a result, the number of the TP images became 416 in total.This simulates that the posture which the examinees present as the PP was presented previously as the RP and was registered as TP.
We measured the accuracy of our recognizing method of the posture.Seven of the examinees presented 23 kinds of the posture as the PP.Our system identified a posture of the examinee's hand five times for each posture.We investigated whether the judge of our system is correct or incorrect.
Increase of the TP data contributed to improve the accuracy of recognizing the examinee's hand posture.Table 3 shows the accuracy for each posture in this experiment.The total average of accuracy had improved to 70.8%.Accuracy is worse only in no.7 and no.29 postures, but is rising in all the other posture.The postures which are not misrecognized at all evaluation data also exist.
To use the TP data containing the examinee's posture is not working always advantageous.The examinee's hand shape is subtly different individually.We used the TP data which included the posture of his or her hand.We considered initially that the hand posture presented by them was matched the image of their own hands.However, correctly recognized PP images by the examinee's own posture image were 18.9% of the correctly recognized ones.That is because the PP vectors are not exactly the same as the RP vectors, which were calculated by our system.

Conclusion
The purpose of this research is to develop a hand pose rally system.We have realized by the image processing to recognize the postures of a hand.We have also evaluated accuracy of our recognition method based on the system design.
One of future works is to improve the accuracy of posture recognition, because our experimental results are not so good.The major reason is that our similarity between two vectors uses only the angle between them.We adopted this similarity because the size of the image of a hand varies depending on the distance between the hand and the camera.However, the problem of the distance can be evaded when the system uses the size ratio of the blue (green) vector to the red vector as the size factor.Therefore, we are planning to add the size factor to our

Fig. 7 .
Fig. 7.An example of the different direction and the different tilt.

Fig. 9 .
Fig. 9.The example of the error in our identifying.

Table 1 .
The categorization of TP database.

Table 2 .
The accuracy of each posture.

Table 3 .
The accuracy of each posture.Another future work, finally, is to implement the entire system.