Proposal of pointing device for handicapped person using ﬁnger position detection with web camera

The internet usage rate of a handicapped person is increasing year by year. When they use personal computers, they use assistive technologies according to their disabilities. In this study, we aimed to develop assistive technology for people whose hands cannot move freely due to accidents or illness. The general input device such as a mouse is not suitable for the handicapped people, because the movement of the cursor and the movement of the device are linked and the movement distance of the hand is large. Assistive technologies include button mouse and stick mouse. Although these devices have succeeded in reducing the movement distance of the hand, there is still a problem that it is incompatible with the high-speed movement and the small movement of the cursor because the movement speed of the cursor can not be changed. Therefore, This research proposes a mouse operation assistive technology using ﬁnger position detection technology with a web camera and machine learning that resolve the problem of existing support technologies.


Introduction 1.1 background
In recent years, the penetration rate of the Internet has increased year by year and reached 80% This number shows that various people are using the Internet.So I this research focused on the Internet usage rate of disabled people.The internet usage rate for people with disabilities was 53%.When classified by type of disability, each Internet usage rate is 82.7% for the physically disabled, 91.7% for the visually impaired, and 93.4% for the hearing impaired.Generally, a disabled person operates a PC using operation support technology corresponding to his / her disability.The pc operation support technology includes keyboard operation support and mouse operation support.Above all, we focused on mouse operation support technology for people with physical disabilities who have the lowest internet usage rate.Symptoms of the physically handicapped include deficits in limbs, paralysis in the body, and weakness in muscle strength.The general input device such as a mouse is not suitable for the handicapped person, because the movement of the cursor and the movement of the device are linked and the movement distance of the hand is large.As existing mouse operation support techniques, there are a button mouse and a stick mouse.Although these devices have succeeded in reducing the movement distance of the hand, there is still a problem that it is incompatible with the high-speed movement and the small movement of the cursor because the movement speed of the cursor can not be changed.

Purpose
The purpose of this research is to resolve the problem of existing assistive technology that it is incompatible with the high-speed movement and the small movement of the cursor because the movement speed of the cursor can not be changed.Therefore, this research proposes a mouse operation assistive technology using finger position detection technology with a web camera and machine learning.

AdaBoost
In learning by AdaBoost, if learning samples of patterns belonging to multiple classes to be identified are given, functions for identifying input patterns whose classes belong to Fig. 1.Classifier obtained by AdaBoost unknown are obtained.We make different classifiers while changing the weight of learning sample sequentially, and give the final classification function by the weighted majority of these classifiers.Individual classifiers are called "weak classifiers" or " weak hypotheses", and combinations of them are called "strong classifiers" or "final hypothesis".A method that learns simple and weak classifiers sequentially and boosts classifier accuracy is generically called Boosting.AdaBoost is a Boosting algorithm that succeeds in using samples by updating sample weights to "adaptive".Fig. 1 shows the classifier obtained by AdaBoost.For an input pattern x, output the label y of the class to which the pattern belongs.A strong classifier is constructed by combing T weak classifiers h t (x) with a confidence of α t When learning, first select h 1 (x), and the next h 2 (x) is selected sequentially so that h 1 (x) can identify poor samples.
The learning algorithm of AdaBoost is shown below.

step1. Preprocessing
Prepare N learning samples ((x 1 , y 1 ), ... (x N , y N )).x 1 is an image, y 1 is a class label., It is an index to determine whether the image is correct or incorrect, and the detection target is labeled +1, and others are labeled -1.

step2. Initialize sample weights
Initialize the weight of each sample using the following formula.
(ii) Calculation of reliability α t from error rate Calculate the reliability α t from the error rate.The smaller the error rate, the higher the reliability.
(iii) Update sample weights Update sample weights.
If the sample was correctly recognized by the weak classifier, In other words, the weight decreases.
If the sample was not correctly recognized by the weak classifier, In other words, the weight increases.If ϵ t = 0.5, then α t = 0, so the weights are not updated and there is no point in continuing learning.

(iv) Sample Weight Normalization
This normalizes the sum of the sample weights to 1.

step4. Creation of final classifier
The final classifier weights all weak classifiers with confidence.It can be expressed by the following equation.

Proposed Method
Fig. 2 shows the simple processing flow of the proposed system.The proposed system performs finger detection using a classifier created using AdaBoost.In the proposed system, a new mouse operation approach linking the movement of the cursor and the action of the finger.It is a method of determining a home position as a reference of operation, stopping cursor movement when the finger is in the area, and moving the cursor when the finger is in the area.Besides, the moving speed of the mouse cursor changes according to the distance between the finger and the area.This method improves the disadvantage that it is incompatible with the high-speed movement of the cursor and small-movement of the cursor of the conventional mouse operation support technology.By using the position of the finger for the operation, do not need force for operation.Moreover, in the proposed system, two kinds of operation methods were implemented.There is a method of operating at one home position and a method of operating at two home positions.The proposed system is capable of cursor movement, multiple click and drag, scroll, and page forward / backward functions.

Experiments
Fig. 3 is the experiment environment created with the webcam and the stand used in this experiment.

Experiment 1
Experiment 1 evaluates the accuracy of various operations for each degree of disability in the proposed system.When operating the proposed system, The degree of disability is reproduced by setting the three-stage limit shown in Tab. 1. Performing various operations of mode1 and mode2 ten times, the accuracy of each operation in each mode of the proposed system is evaluated from the number of successes.For cursor movement operation and scroll operation, success is only counted when user can move the cursor to the specified position.

Experiment 2
Experiment 2 evaluates the operation accuracy of the proposed system by different persons.Perform each operation of the two modes of the proposed system 10 times and measure the number of successes.The success criteria for cursor movement and scrolling are the same as in Experiment 1.The experiment is evaluated from the number of successes of various operations of person A contained in the data for learning and person B and person C not contained in the data for learning.

Experiment 1
In Experiment 1, we evaluated the accuracy of each degree of failure of various operations of the proposed system.By setting three levels of restriction in the operation of the proposed system, it was reproduced as (i) normal or mildly impaired, (ii) moderate impaired, (iii) severely disabled.The results of mode 1 are summarized in Table,2, and the results of mode 2 are summarized in Tab. 3.

Experiment 2
Experiment 2 evaluated the operation accuracy of the proposed system by different persons.Person A who is included in the data for learning and person B and person C who are not included in the data for learning perform various operations in two modes ten times.The success numbers are summarized in the following Tab.4, Tab. 5.

Experiment 1
From the Tab.2, the number of successes according to the degree of failure in mode 1 was not significantly changed in the number of successes in (i) healthy and mildly impaired people and (ii) moderate impaired people.However, (iii) In severely disabled people, operations other than the page movement operation in the browser resulted in a low number of successes.The cause of the low number of successful click operations of (iii) is considered as follows.In (iii), the change in finger height was limited to 1 cm.Thereby, it is conceivable that the change in the moving average of the area used for the success determination did not exceed the threshold.As the cause of the low number of successes of the mouse cursor movement and scroll operation in (iii), it is conceivable that movement in the vertical direction is difficult because the wrist is fixed in (iii).From the Tab.3, the number of successes for each degree of failure in mode 2 is high for all of (i), (ii) and (iii) except for the scroll operation and the cursor operation.The cause is that the wrist is fixed in the same way as mode 1, so it may be difficult to move vertically.

Experiment 2
Tab. 4 shows that the number of successes by a person in mode 1 is generally lower for person C than for person A and person B.As a cause, the recognition rate of the finger detection of the person C is low.It can be seen from 5 that the number of successes for each person in mode 2 was high for all three.Although the recognition accuracy of the finger of the person C is low, a high success number is obtained.This is considered to be because Mode 2 has less change in the number of fingers and change in position compared to Mode 1.Therefore, it was found that mode 2 is less affected by the finger recognition rate than mode 1.

Conclusions
This study proposed a new computer cursor operation assistive technology.By using a web camera, image processing, and machine learning algorithm, a high-speed movement and precise position control of the mouse cursor has been realized while suppressing the movement distance of the hand.Future issues include the improvement of the detection rate and the reduction of the false detection rate by adopting some deep learning algorithms, also planning to make a prototype device to prove the availability of this proposed.

Table 1 .
Three levels of limitations in experiments

Table 2 .
Number of successes by limitations of mode 1 operations

Table 3 .
Number of successes by limitations of mode 2 operations

Table 4 .
Number of successes of various operations in mode 1 by person