Meal Assistance Robot with Blink Operation Interface

The number of people with disabilities is growing in Japan. They are facing many kinds of difficulties in their daily lives. Especially, for the people with upper limb disabilities, it is even difficult in having meals by themselves. The systems which are able to help them in having meals are expected. There are already several types of assistant robots which are operated with joysticks or touch sensors. However, people with upper limb disabilities can barely operate the robots with full functions. Therefore, a maneuverable meal assistance robot which is easy to operate with an eye-interface is proposed for people with upper limb disabilities. The proposed robot with orthogonal structure, that offers simple and stable motions, is able to provide foods to users smoothly. The robot is operated by user’s eye blink. The blink is designed to be detected by a web camera connected to control unit based on Raspberry Pi. Thus, the system in compact size can complete the food providing effectively.


Introduction
In Japan, the number of people with disabilities is growing in recent years.Owing to the background of aging society, the growth is considered to continue in near future (1,2) .The people with disabilities are facing severe situation with many kinds of difficulties in their daily lives.Especially, for the people with upper limb disabilities, it is even difficult to have meals by themselves.They have to get assistance from their family or helpers in eating.It is a frustrated situation for the care receivers who cannot complete basic daily activities.Meanwhile, it is also a heavy work for their families or care centers.To help the care receivers in eating, it is important to match the pace and amount of help giver and receiver.It is such a careful work that the meal assistance requires at least one helper in a meal in home nursing or at care centers.Considering the aging society in Japan with shortage of labors, it is not an easy problem to solve.According to the situation stated above, some robots are introduced to help the people with disabilities in eating, relieving the severe situation.
Handy1 (3)(4)(5) , iARM (6) and My-spoon (7)(8)(9) are well known as robots which are able to assist in eating.Handy1 is developed by Rehab-Robotics Company Ltd in United Kingdom.It is a robot system to provide support in daily life of users.This system helps in eating by spoon up food to users.By that way, the robot is suitable for helping to have liquid food particularly.However, it has difficulty in assistance to have food in solid form which is not easy to spoon up.
iARM is developed by Exact Dynamics in Netherlands.It is a robot arm that is designed to help in all living activities.The robot arm is operated by using a joystick.For this reason, it is difficult to operate the iARM for people with upper limb disabilities.
Meanwhile, in Japan, there is a dedicated robot system developed to assist eating.It is named as My-spoon developed by Secom Co., Ltd.The system specializes in assistance in eating than the two robots introduced above.

2
The meal assistance robot employs a fork and a spoon as manipulators.The manipulators help in eating by griping food.The manipulation is considered effective in some cases.However, owing to the structure of the manipulators, the system has difficulty in griping soft and slippery food.There is also a risk to make fragile food into pieces.Based on the robots introduced above (10,11) , some researches related to meal assistance robots are carried out.It is not difficult to summary the meal assistance robot that the articulated robots introduced are not easy to avoid vibration owing to the structure.Since in taking and passing food procedures, vibration generated on one axis will be transferred to other axis.Another reason why conventional systems are not easy to be applied is that the robots are designed to be with interfaces such as joystick which is supposed to be operated with normal hands.In most cases,  the interfaces are not suitable for people with upper limb disabilities.
Because of the shortcomings introduced above, we propose the meal assistance robot to solve the existent problems with the following features.
(i) help eating by passing food to user smoothly (ii) orthogonal coordinate structure (iii) user friendly eye-interface The details of the proposed will be introduced in the following sections.In Sect.2, the structure and functions of the robot is introduced.The eye-interface which is applied to make the robot easy to use to people with upper limb disabilities is introduced in Sect.3. The conclusions of the paper is given in Sect. 4.

Meal assistance robot 2.1 Structure of proposed meal assistance robot
Different from previous meal assistance robots our proposed one is structured in an orthogonal coordinates.Fig. 1 (a) shows configuration of proposed robot.The main components of the robot are including the pusher which applied to push food from food plate in y-axis direction as shown in the figure; the shutter which can be adjusted in vertical direction of z-axis; the food plate with tracks, and the LEDs which are used for marking the tracks; a spoon which is suitable to carry most kinds of food to the user.Operation direction of each axis is defined as follows.Plate moves in only x-axis direction, shutter moves in only z-axis direction, pusher and spoon move in only y-axis direction.Functions and specifications of each axis are summarized in Table 1.The forces from actuators are transmitted in axis with rack-and-pinion.The DC motors are employed to the plate, pusher and spoon as actuators.For the shutter a servo motor is used.

Proposed robot action
Each axis at proposed robot moves as shown in procedure of Fig. 3 based on proposed robot structure described at preceding section.
(i) Foods are set up at the predetermined location on the plate tracks.As we see in Fig. 2, the food are set in five tracks.The pusher stops at the 3 rd track.(ii) The user can chose food using the eye-interface.
The x-axis movement of plate will make the selected track moving to the pusher position.(iii) The pusher moves to the position of the selected food.(iv) The shutter comes down to the track.(v) The pusher pushes out the food to the spoon in standby status.(vi) The spoon extends and passes the food to user's mouth.The location shutter can be modified according to setting based on different food.The food is supposed to be offered following their locations on the tracks.(b) Elevation view.Fig. 2. External view of plate.

Choice food and provide
There are LEDs applied for marking the tracks on the plate.The user can choose the food according to LED on the tracks.In choosing procedure, LEDs sequentially light up.The user chooses the food when LED of the track which contains the objective food lights up.The choice of user will be recognized by eye-interface.
We have developed operable interface which helps people giving information to the computer by blink.By this way people with upper limb disabilities can operate the proposed robot easily.

Available provided foods
The proposed robot is able to help user with having solid food.Experiment with marketed lunchbox has been implemented to confirm that the proposed robot can provide all of solid foods perfectly.
Foods from seven kinds of lunchboxes were employed in experiments as subjects.There are 47 kinds of foods in the lunchboxes: rice, noodles, fried foods, simmered foods, grilled foods, salads, fruits and others.The food offering experiments have been repeated for 232 times totally to verify the effectiveness for passing the food introduced above to user.The proposed robot provided all of the foods except four times on non-solid foods.

Algorism for deliberate blink detection
We have developed user friendly eye-interface, by which people with upper limb disabilities can operate the proposed meal assistance robot easily.Here are approached in the use of the interface.
(i) LEDs in the proposed robot light up sequentially.
(ii) The user chooses a food during LED of food tracks light up sequentially.(iii) The robot implements assistance manipulating in eating automatically.The user can operate the robot smoothly because food can been chosen by only blink.Raspberry Pi with web camera (Logicool HD Pro Webcam C920t) is used in the proposed eye-interface.
Firstly, user's face image will be fetched by the web camera.Next, eyes will be recognized and blink will be detected.These algorithms are shown in Fig. 3. Eye search is implemented by the following order.
(i) Facial recognition is performed using Haar-like feature on the acquired image.Only, Haar-like feature is use to object detection obtaining feature quantity from contrast in the image.(ii) Eye recognition is performed similar to obtain as the above way.On this occasion, eye is searched in the region of Fig. 4 (vertical direction: h/6≦y≦h/2, horizontal direction: 0≦x≦w/2).It is assumed that eye exists.(iii) Eye position relative to face position (rate_x rate_y) and its scale (width, height) are memorized after eye position (x, y) is recognized (Fig. 5).rate_x and rate_y are expressed in the following equations.(2) (iv) After (i) ~ (iii) achieved, facial recognition is performed, and eye region is calculated from the relative position and the scale.The interface detects blink according to the detected eye position.The blink is determined by detecting iris.Percentile method is applied to detect iris.Percentile method decides threshold value t as ratio of lower pixel value which in binarized image is more than p.This method is effective in case which p is basically known.Proportion of iris in extracted eye image is about 5%.Black pixels more than 5% of eye area are leave.Extracted opening eye image and closing eye image are shown in Fig. 6 and Fig. 7. Extracted object get near circular form because iris comes out at opening eye.Extracted object get near narrow form because eyelashes come out at closing eye.It is possible to determine opening eye or closing eye from horizontal to vertical ratio of the object.

Experiment for surveying specs of proposed interface
Frame rate of the proposed interface was surveyed.If the frame rate is slow, blink detection would not be performed enough.In the results of the survey, the frame rate is 6.0 fps, thus inter-frame circle is 0.17 second.Average natural blink is 0.1 second.Closing eye during 5 frames (about 1 second) and over is determined as deliberate blink in view of consecutive blink.Next, the rules in the proposed interface to recognize the deliberate blink from natural blink were established.Recognition times of deliberate blink and false positive recognition times of natural blink were measured.An examinee performed total of fifteen natural blinks in every 12 seconds in the experiment.The proposed robot was required for passing food.The experiment is performed on ten examinees.The examinees performed natural blink during interval of consecutive blink.
The experimental results are shown in Table 2.A ~ F are examinee without glasses, G ~ J are examinee with glasses.As we see in the results, some false recognition exists.False positive recognition was not found.There is a small number of false recognition found.Generally the   proposed interface recognized the deliberate blink in spite of no blink.Therefore, the interface has a high degree of usability.

Conclusions
In this paper, a meal assistance robot system operated by eye-interface is the proposed.The robot was built up with simple structure and stable motion.Meanwhile, the robot can be manipulated in an easy mean owing to the user friendly eye-interface proposed in this research.We confirmed that the proposed system can provide solid foods in lunchboxes.Meanwhile experiments of eye blink recognition using the proposed interface were implemented.As a result, the proposed system has adequate operability.By using it, people with upper limb disabilities can operate the proposed robot easily.
(a) Default position.(b) Moving plate left or right.(c) Moving pusher forward.(d) Lower shutter.(e) Pushing food and guiding onto spoon.(f) Moving spoon forward.

Table 2
Experimental results.Fig.4. Search region on user's face for eye detection.