Development of the 3D User Interface Using the Hand Gesture

We suggest the 3D user interface of the non-contact using the hand gesture in the field of the human computer interaction. Move a hand and measure information of the movement with various sensors and make a model on a computer and really reproduce it. Recognize the form of the finger using a bending sensor in this study. Set a hand gesture by the above. Perform image processing and acquire the information of the X-Y coordinate with using an infrared camera and infrared LED. The z-coordinate measures the distance between the screen and sets it with using a supersonic wave sensor.


Introduction
The operation with a mouse and the keyboard and operation by the touch panel is the mainstream of the user interface now.The operation with a mouse and the keyboard can process making and the image of the document and program.Actually, it cannot do intuitive operation.Development of the user interface that works intuitively like touch panel and VR advances now.We can just operate it by touching it by operation by the touch panel.A viewpoint in VR moves to the direction that we turned in real world.As noted above a virtual space approaches real world.And demand for operation in the 3D space increases.⁽¹⁾I can just operate it by touching it by operation by the touch panel.Though, Input information has only the coordinate of the touch point.Therefore, there are few variations of the operation.Show the variation of the touch gesture in Fig. 1.
We think about HCI next.HCI is a study to make a computer a better one for a human being.When we give an Fig. 1.Gesture example develop new interaction.Improvement of the interaction that there is now.Human characteristic analysis.The mouse is one of HCI, too.One express the information that changed in reality on a computer.One relate it to reality and became easy to do operation.Develop a new interaction in HCI in this article.We suggest the user interface that we can operate sensuously.
We raise LeapMotion and a SmartGlass as precedent.LeapMotion is comprised of two infrared cameras and infrared LED.One photograph a hand and a finger lighted up by the infrared LED by two infrared cameras.And, calculate the position of a hand and the finger in the 3D space by image analysis.Then, SmartGlass can display a screen put it on real scenery.It is the user interface that we can operate by a sound command by speech recognition.Movement for the depth is not implemented very much in LeapMotion.SmartGlass has few variations of the operation.
We develop the interaction that can be compatible of various operation by the operation with a mouse and the keyboard and the intuitive operation that becomes possible in touch display and VR.It is necessary to receive information changing in reality on a computer to perform intuitive operation.Thus, a limit occurs by the operation using the touch.So we think about a device of the noncontact.A hand gesture is suitable to perform various operation with a device of the non-contact. ( 2 )The hand gesture does not only make a short cut in form of the hand.
We can obtain the dynamic information.Thus, there is more information than a touch display.And it approaches more actually. (3)Therefore, Study 3D user interface of the noncontact using the hand gesture in this study. ( 4)A gloveshaped device is suitable to come true. (5)I attach a bending sensor to each finger of the glove type device and recognize the form of each finger.One do the movement corresponding to the hand gesture that set.This is because it performs the above.Then, Acquire the information using an infrared camera and infrared LED for the acquisition of the information of the X-Y coordinate.Attach infrared LED to a finger-tip and photograph it with an infrared camera and acquire the information of the X-Y coordinate.Use a supersonic wave sensor for the acquisition of the information of the z-coordinate.Measure distance and acquire the information of the z-coordinate.It can perform movement more intuitive than the above.It is the interactions that can operate colorful. (6)Acquire information of the movement from various sensors and images.It reappears on a computer.Then a frame of the user interface grows more. (7)

Principle 2.1 Structure
It consist of glove-shaped input device and screen, a projector, an infrared camera.Show the details of this device in Fig. 2. Attach infrared LED and supersonic wave sensor to the finger-tip of the forefinger to a glove-shaped input device and attach a bending sensor to each finger.Show the details of the glove-shaped input device in Fig. 3. Install an infrared camera to receive infrared LED in the screen side.Use Arduino to control a sensor.Make the model in Unity.

The acquisition of the X-Y coordinate
We must acquire information of the movement of the hand if we reproduce the hand which moved in reality in a model.Use an infrared camera and infrared LED as a method to acquire the information of the X-Y coordinate with the device of the non-contact.Information except the infrared light is intercepted by receiving light of infrared LED attached to a finger-tip with an infrared camera.Therefore, information except necessary information is intercepted and can come true by simple image processing.Expose screen to light of the infrared rays LED which Installed in the glove.Recognize where gets light by infrared camera.Show the comparison that photographed an image with a normal Web camera and an infrared camera in Fig. 4. Intercept unnecessary information such as the infrared light in the daily life to distinguish only light of the infrared LED with the image by set the threshold value and become a 2 level.We can acquire X-Y coordinate by giving a center of gravity of the light of the infrared LED of the image which became the 2 level.

The acquisition of the Z coordinate
We must acquire information of the movement of the hand.Measure distance with a supersonic wave sensor attached to the finger-tip of the glove-shaped input device to acquire the information of the z-coordinate.Reproduce movement of the hand in a model by coordinating the information.Measure distance with a supersonic wave sensor and send the value to the computer side by serial communication.Operation is difficult when we can always move it to the three-dimensional direction.Set to set a gesture and can move it only in the two dimensions direction.Thus, operation becomes easy.Show the neighboring circuits of this supersonic wave sensor in Fig. 5.

Gesture
You must recognize the bending of the finger of the hand to set a gesture.This installs a bending sensor in the inside of the glove-shaped input device and performs it.We can acquire the movement of the finger of the hand with an image.But, know how long a finger bends when use a sensor.Therefore, we can operate it at degree of the bending of the finger.Attach a bending sensor to all fingers except the forefinger to make various gestures.Show the neighboring circuits of the bending sensor in Fig. 6.A short cut is made if take action that it is a certain gesture on a model.Recognize the bending of the finger.Set it with numerical value.Therefore, information decreases on doing serial communication.Show to Fig. 7 how you install a bending sensor.Install a sensor in the hand in Fig. 7.However, actually install a sensor inside of a glove.

The transmission of the information
The X-Y coordinate makes an image a 2 level on a computer and demand a center of gravity.Therefore, we can handle it without the transmission of the information.However, acquire a z-coordinate by using a supersonic wave sensor.Acquire the gesture by using a bending sensor.Therefore, we must send a z-coordinate and the information of the gesture to the computer by serial communication.It is necessary to always communicate because reproduce what move in reality by a computer.Send information as little as possible to quicken transaction speed and want to communicate.Transmit the information with one character string.Set it to the first digit and the second digit when temporarily assume the information of the gesture 100 kinds.We multiply 100 in the numerical value of the supersonic wave sensor.Set it in values more than the three-digit.Therefore, we can read distance and the information of the gesture from one character string.

The making of the movement model
Made the model which be moving it in reality, and moves by using Unity.Unity is the game engine has IDE and It corresponding to plural platforms.Make the space of the square to make a model to do set it to operate movement by an input device.Set the movement of the X-Y coordinate with a central focus on a camera.Set the movement of the zcoordinate when moved a hand.Screen is as the deepest part.And, when do a certain gesture, set it to be movable In addition, set it to make plural gestures, and to change the size of the object by a gesture.Change a color so that the wall understands a difference.Show this model in Fig. 8.

Recognition of X and Y coordinates of both hands
To operate with both hands, you need to recognize each LED of both hands.Rather than finding the center of gravity of the light it is necessary to recognize the two lights and find the center of gravity of the two lights.Just binarizing and finding the center of gravity is not enough when you want to find more than one center of gravity.By recognizing the outline, it is possible to detect two or more points.And, by attaching a label you can find the center of gravity of those points However, recognize multiple unnecessary points or overlap two points in the case where light is captured by a camera and binarized as shown in the Fig 9 .This problem can be solved by repeating shrinkage.If the point near 8 is black, set that point to black.Also, even if the reduction process is performed, noise such as living light may occur.We recognize 2 points in descending order of area of each point and recognize it as light of LED attached to both hands.

Wireless communication
Communicate wirelessly using the Bluetooth module.When communicating with wires, it is inconvenient to operate with both hands.Therefore, do communication wirelessly.The created device is shown in Fig. 10.

The acquisition of the X-Y coordinate
Tested it to acquire X-Y coordinate definitely.Irradiated infrared LED on screen and acquired an image with a camera.Set the threshold to distinguish only infrared LED and made an image a 2 level.Calculate a center of gravity and acquired X-Y coordinate.Show a result in Fig. 11.We were able to recognize a center of gravity than Fig. 11 definitely.Perform this in a model on Unity.Photograph the infrared LED which attached with an infrared camera.Set it to work in the same way as movement of the hand.Show this in Fig. 12. nx

The acquisition of the Z coordinate
We did an experiment to acquire a z-coordinate definitely.Changed distance and measured the output of the supersonic wave sensor.Measured it every 5cm from 10cm to 50cm.Show this result in Fig. 13.We were able to measure distance definitely than Fig. 13.Set it to operate a model, and serial communicates the value.Set it to work by real distance in the model that made on Unity.Show this in Fig. 14.This Fig. 14 shows a plan view.

Input of the gesture
Confirmed the output of the bending sensor.Attached a sensor and measured the output which is in condition to have bent a finger and be in condition to have stretched out a finger.Show a result in table 1.The recognition of the finger is enabled if make the threshold 2.5V than table 1. Recognize a gesture by a sensor on Arduino.Sent information by serial communication.Used it as a command of the permission of the movement of the Z-axis in the model that made on Unity and implemented it.Show a gesture in Fig. 15.

Confirmation of the movement
Perform coordinate movement and input a gesture.Confirmed whether it could perform a series of movement in succession.Show this in Fig. 16.We were able to confirm that worked definitely from a Fig. 16.

Recognition of X and Y coordinates of both hands
We experimented whether XY coordinates of both hands can be obtained correctly.The original image is shown in Fig. 17(a).The binary image is shown in Fig. 17(b).Label and color randomly.This is shown in Fig. 17(c).If the point near 8 is black, set that point to black.Repeat it.This makes it possible to obtain the center of gravity of the two lights.This is shown in Fig. 17

Conclusions
We suggested the 3D user interface using the hand gesture by the above-mentioned method.Because intuitive operation that cannot perform with a mouse and a keyboard and cannot do it by the operation in the touch panel gained variations of the operation, we were enabled.The product of various touch displays comes to be imminent now.However, there are many problems so that touch panel is replaced with an existing keyboard and mouse.Performed suggestion to solve this problem in this study.In addition, technical development of 3DCG advances now.Thus, operation in the three-dimensional space is necessary.When the operation with such a touch display is suitable, a mouse, the operation with the keyboard become mainstream.This is caused by many problems in the operation with the touch display.Suggestion of the user interface in this study is suggestion to solve the problem mentioned above.Think that this is suitable for the operation of the three-dimensional direction.Perform the implementation with both hands because go only with one hand now as a future problem.Make the model that We add the physical phenomenon.

Fig. 13 .
Fig. 13.The measurement of the distance
(a) The top, expansion (b) Bottom, expansion (c) Bottom, normal (d) Front, normal (e) Back, normal (f) Back, expansion Fig. 16.A series of movement (a) original image (b) binary image (c) colored image (d)result Fig.17.Detection of 2 points

Table 1 .
Bending of the sensor[V]