Suggestion of the 3 D User Interface Using the Hand Gesture

We suggest the 3D user interface of the non-contact using the hand gesture in the field of the human computer interaction. Move a hand and measure information of the movement with various sensors and make a model on a computer and really reproduce it. Recognize the form of the finger using a bending sensor in this study. Set a hand gesture by the above. Perform image processing and acquire the information of the X-Y coordinate with using an infrared camera and infrared LED. The z-coordinate measures the distance between the screen and sets it with using a supersonic wave sensor.


Introduction
The operation with a mouse and the keyboard and operation by the touch panel is the mainstream of the user interface now.The operation with a mouse and the keyboard can process making and the image of the document and program.Actually, it cannot do intuitive operation.Development of the user interface that works intuitively like touch panel and VR advances now.We can just operate it by touching it by operation by the touch panel.A viewpoint in VR moves to the direction that we turned in real world.As noted above a virtual space approaches real world.And demand for operation in the 3D space increases.⁽¹⁾ I can just operate it by touching it by operation by the touch panel.Though, Input information has only the coordinate of the touch point.Therefore, there are few variations of the operation.Show the variation of the touch gesture in Fig. 1.
We think about HCI next.HCI is a study to make a computer a better one for a human being.When we give an example develop new interaction.And, improvement of the interaction that there is now.And, human characteristic analysis.The mouse is one of HCI, too.One express the information that changed in reality on a computer.One relate it to reality and became easy to do operation.Develop a new interaction in HCI in this article.We suggest the user interface that we can operate sensuously.
We raise LeapMotion and a SmartGlass as precedent.LeapMotion is comprised of two infrared cameras and infrared LED.One photograph a hand and a finger lighted up by the infrared LED by two infrared cameras.And, calculate the position of a hand and the finger in the 3D space by image analysis.Then, SmartGlass can display a screen put it on real scenery.It is the user interface that we can operate by a sound command by speech recognition.Movement for the depth is not implemented very much in LeapMotion.SmartGlass has few variations of the operation.
We develop the interaction that can be compatible of various operation by the operation with a mouse and the keyboard and the intuitive operation that becomes possible in touch display and VR.It is necessary to receive information changing in reality on a computer to perform intuitive operation.Thus, a limit occurs by the operation using the touch.So we think about a device of the non-contact.A hand gesture is suitable to perform various operation with a device of the non-contact. (2)The hand gesture does not only make a short cut in form of the hand.
We can obtain the dynamic information.Thus, there is more information than a touch display.And it approaches more actually.
( 3 ) Therefore, Study 3D user interface of the non-contact using the hand gesture in this study. ( 4 )A glove-shaped device is suitable to come true. (5)I attach a bending sensor to each finger of the glove type device and recognize the form of each finger.One do the movement corresponding to the hand gesture that set.This is because it performs the above.Then, Acquire the information using an infrared camera and infrared LED for the acquisition of the information of the X-Y coordinate.Attach infrared LED to a finger-tip and photograph it with an infrared camera and acquire the information of the X-Y coordinate.Use a supersonic wave sensor for the acquisition of the information of the z-coordinate.Measure distance and acquire the information of the z-coordinate.It can perform movement more intuitive than the above.It is the interactions that can operate colorful. (6)Acquire information of the movement from various sensors and images.It reappears on a computer.Then a frame of the user interface grows more. (7)

Principle 2.1 Structure
It consist of glove-shaped input device and screen, a projector, an infrared camera.Show the details of this device in Fig. 2. Attach infrared LED and supersonic wave sensor to the finger-tip of the forefinger to a glove-shaped input device and attach a bending sensor to each finger.Show the details of the glove-shaped input device in Fig. 3. Install an infrared camera to receive infrared LED in the screen side.Use Arduino to control a sensor.Make the model in Unity.

The acquisition of the X-Y coordinate
We must acquire information of the movement of the hand if we reproduce the hand which moved in reality in a model.Use an infrared camera and infrared LED as a method to acquire the information of the X-Y coordinate with the device of the non-contact.Information except the infrared light is intercepted by receiving light of infrared LED attached to a finger-tip with an infrared camera.Therefore, information except necessary information is intercepted and can come true by simple image processing.Expose screen to light of the infrared rays LED which Installed in the glove.Recognize where gets light by infrared camera.Show the comparison that photographed an image with a normal Web camera and an infrared camera in Fig. 4. Intercept unnecessary information such as the infrared light in the daily life to distinguish only light of the infrared LED with the image by set the threshold value and become a 2 level.We can acquire X-Y coordinate by giving a center of gravity of the light of the infrared LED of the image which became the 2 level.

The acquisition of the Z coordinate
We must acquire information of the movement of the hand.Measure distance with a supersonic wave sensor attached to the finger-tip of the glove-shaped input device to acquire the information of the z-coordinate.Reproduce movement of the hand in a model by coordinating the information.Measure distance with a supersonic wave sensor and send the value to the computer side by serial communication.Operation is difficult when we can always move it to the three-dimensional direction.Set to set a gesture and can move it only in the two dimensions direction.Thus, operation becomes easy.Show the neighboring circuits of this supersonic wave sensor in Fig. 5.

Gesture
You must recognize the bending of the finger of the hand to set a gesture.This installs a bending sensor in the inside of the glove-shaped input device and performs it.We can acquire the movement of the finger of the hand with an image.But, know how long a finger bends when use a sensor.Therefore, we can operate it at degree of the bending of the finger.Attach a bending sensor to all fingers except the forefinger to make various gestures.Show the neighboring circuits of the bending sensor in Fig. 6.A short cut is made if take action that it is a certain gesture on a model.Recognize the bending of the finger.Set it with numerical value.Therefore, information decreases on doing serial communication.Show to Fig. 7 how you install a bending sensor.Install a sensor in the hand in Fig. 7. But, actually install a sensor inside of a glove.

The transmission of the information
The X-Y coordinate makes an image a 2 level on a computer and demand a center of gravity.Therefore, we can handle it without the transmission of the information.However, acquire a z-coordinate by using a supersonic wave sensor.Acquire the gesture by using a bending sensor.Therefore, we must send a z-coordinate and the information of the gesture to the computer by serial communication.It is necessary to always communicate because reproduce what move in reality by a computer.Send information as little as possible to quicken transaction speed and want to communicate.Transmit the information with one character string.Set it to the first digit and the second digit when temporarily assume the information of the gesture 100 kinds.We multiply 100 in the numerical value of the supersonic wave sensor.Set it in values more than the three-digit.Therefore, we can read distance and the information of the gesture from one character string.

The making of the movement model
Made the model which be moving it in reality, and moves by using Unity.Unity is the game engine has IDE and It corresponding to plural platforms.Make the space of the square to make a model to do set it to operate movement by an input device.Set the movement of the X-Y coordinate with a central focus on a camera.Set the movement of the z-coordinate when moved a hand.Screen is as the deepest part.And, when do a certain gesture, set it to be movable In addition, set it to make plural gestures, and to change the size of the object by a gesture.Change a color so that the wall understands a difference.Show this model in Fig. 8.

The acquisition of the X-Y coordinate
Tested it to acquire X-Y coordinate definitely.Irradiated infrared LED on screen and acquired an image with a camera.Set the threshold to distinguish only infrared LED and made an image a 2 level.Calculate a center of gravity and acquired X-Y coordinate.Show a result in Fig. 9.We were able to recognize a center of gravity than Fig. 9 definitely.Perform this in a model on Unity.Photograph the infrared LED which attached with an infrared camera.Set it to work in the same way as movement of the hand.Show this in Fig. 10.nx

The acquisition of the Z coordinate
We did an experiment to acquire a z-coordinate definitely.Changed distance and measured the output of the supersonic wave sensor.Measured it every 5cm from 10cm to 50cm.Show this result in Fig. 11.We were able to measure distance definitely than Fig.

Input of the gesture
Confirmed the output of the bending sensor.Attached a sensor and measured the output which is in condition to have bent a finger and be in condition to have stretched out a finger.Show a result in table 1.The recognition of the finger is enabled if make the threshold 2.5V than table 1. Recognize a gesture by a sensor on Arduino.Sent information by serial communication.Used it as a command of the permission of the movement of the Z-axis in the model that made on Unity and implemented it.Show a gesture in Fig. 13.

Fig
Fig. 5. Neighboring circuits of the supersonic wave sensor 33kΩ 曲げセンサ 11. Set it to operate a model, and serial communicates the value.Set it to work by real distance in the model that made on Unity.Show this in Fig. 12.This Fig. shows a plan view.

Table 1 .
Bending of the sensor