Trial Development of Immersive Virtual Reality Supporting Content for Training Engineers Using Oculus Rift and Leap Motion

In Japan, accidents in the factory are constantly becoming serious problems as a result of lack of technology transfer due to mass retirement of skilled technicians in machine tools. As a solution, this study proposes immersive VR content for work support using Head Mounted Display (HMD) and haptic device. This is a system to reproduce machine tools with dangerous handling such as milling machines and lathes in the VR space and present arbitrary feedback to the fingertips with a haptic device when performing erroneous work or fine work, thereby safely carrying out work training support. In this study, HMD and haptic device were used in combination as a method to feedback to users, so that more realistic work can be done in virtual space. VR environment was created with Unity that can manipulate machines in Virtual Space. Leap Motion was used to track and display subjects own hand in VR environment. Haptic device was made using Arduino and vibration motor to present feedback on fingertips. As a result, the tracking of the hand using Leap Motion when gloves are used together shows that the detection rate greatly differs depending on the material of the glove. With VR environment and vibration feedback, we showed the effectiveness of intuitive operation similar to reality.


Introduction
According to the Goldman Sachs Global Investment Research Division in the global AR/VR market, the AR/VR market will expand to about US $95 billion in 2025 in the future, and it is said that there is a possibility of forming a market as the third platform following the PC and smartphone.However, the 2025 market breakdown of VR related software, it is predicted that entertainment related business will close about 75% of the market as a whole, and it can be said to be a challenge in how VR will be applied in the industrial field in the future spreading (1).
Also, Domestic AR/VR market has begun to grow rapidly.Especially in 2016 it is said that this year was "VR Gannen", the first year of VR due to the fact that low-cost head mounted displays, essential for VR, came out on the market (2).In the past, a system using VR and HMD so far required a special system at a high cost and was only accessible for large companies, but in recent years it was commercialized at a reachable price range and high quality for general consumers due to popularization of HMD, and it became an environment where consumers can enjoy VR comfortably.In addition, since it can introduce VR at low cost, it attracts attention as a sales tool for commercial use such as customer service and tourism.Although the VR market attracts such attention, in fact the domestic VR market is behind the rest of the world.In the world VR industry landscape, not only major IT companies such as Facebook and Google, but also famous companies such as Amazon and SAMSUNG stand out.On the other hand, Japanese companies listed there are limited industries such as SONY, Nikon, RICOH and few in number.
By the way, in Japan, accidents in the factory are constantly abandoned due to misoperation and dangerous operation by ignoring safety manual.There are many basic human errors such as manual violation, accidental mistakes, and misjudgment as reasons to be conceived.In 2013, at the steel manufacturing company in Hyogo Prefecture, a man was working on metal turning lathes and caught between blades, eventually his arm was cut and death was confirmed due to his dangerous operation (3).However, it can be inferred from successive accidents that not enough knowledge and experience has been passed on to young workers successfully due to a mass retirement of veteran workers born in baby-boom generation (4).
Also, as the automation of machines progresses and the work up to now has been left to the machine, the "work force" of the young workers corresponding to the trouble tends to decline.The term "work force" here means an ability to resolve problems in all kinds of situations that might happen.There is a strong sense of crisis for veterans to retire as young people are immature, and more companies are entering into countermeasures.For example, Kao Corporation intends to enhance training to simulate driving and trouble scenes using mini plants and simulators to cultivate basic knowledge on plant operation and trouble responding ability of young workers (5).Also, Hitachi Zosen creates a skill instructor system in 2011 that veterans and others who are retirement ages can train young engineers with one on one hand (6).High quality, safety, and high efficiency which are the strength of Japanese manufacturing is coming from this "work force", but how to train young workers who support its technique is now a big task.
For the purpose of this study, we will utilize VR technology, which is state-of-the-art technology, to support the training of young engineers without having to think about getting injuries, and develop a system to support operation training of dangerous industrial machines or alert misoperations to reduce plant accidents that are frequently occurred in Japan.Moreover, by using haptic feedback together, usability can be improved by giving more information to the user and obtaining accurate information.Since haptic feedback is still in the process of development and has the high potential, we present the possibility of being able to play an active part in fields other than entertainment.In addition, since haptic feedback can present various kinds of feelings such as soft, sharp, hard, dull, etc., reproducing the unique feeling possessed by experienced technicians and aim to improve the efficiency of acquiring skills by having young engineers experience themselves can lead to the transfer of technology.

Concept
The goal of this study is to simulate industrial machine in virtual environment and use haptic glove to give arbitrary haptic feedback to user's fingers during operations, developing the training system to prevent accidents caused by misoperation, and efficiently pass on skilled skill to young engineers.Also it will be more quick and efficient than skilled veterans train young engineers one to one by using HMD, since its immersive feeling is so effective that it can obtain realistic feeling as if user is trained by real veterans.The industrial machine to be simulated in this system is a lathe.It is one of the most common industrial machines operated in many factories and its operation method is very analog, meaning to operate a lever, handle or a button, not a touch panel or other digital interfaces.Also, developing dedicated haptic device to enhance affinity with this system.The feedback users get from dedicated haptic device is a slight vibration when the button was pushed or lever and handle was turned and strong constant pattern of vibration in case of dangerous behavior.

Advantages of VR with HMD and Haptic Device
Virtual reality is a sense that the space equivalent to the actual real world is reproduced, and it is the technology for that.Its immersive feeling is tremendous, and by further using HMD (Head Mounted Display), a visual presentation device for space reproduction of the first person, it reinforces the sense of presence.Using the technology of virtual reality, you can feel as if you are at that place in the original place by looking at the space reproduced on the virtual space.This is called ultra-realistic feeling.The more accurate this reproducibility is, the more the sense of being on the spot increases, in other words, obtaining the ultra-realistic feeling.Moreover, Humans have five senses of "visual", "hearing", "olfaction", "tactile", and "taste".A sense of daily experience is said to be information obtained from vision by more than 80%, it is said that it becomes 90% or more when combined with hearing sense (7).Therefore, by using head-mounted display that dominates "visual" and "hearing" and can control more than 90% of the human sense, an overwhelming immersive feeling can be obtained and more of realistic feeling can be obtained than planned view.
In addition to these, it is possible to select the information that can be spontaneously obtained by subject by looking at the space with HMD rather than a virtual reality shown to the TV monitor unilaterally.By preparing information of the surroundings so as to correspond to the reality, information can be obtained no matter where you look at the intentions of the subject, or by touching the object.It is the greatest advantage of VR when HMD is used together that experiencers move themselves and select information and acquire information, which further enhances the realistic feeling.
In this research, I emphasize this ultra-realistic feeling.By simulating dangerous work in virtual space and train while experiencing the realistic feeling will lead to prevention of accidents or misoperation in work.

Method to Receive Tactile Feedback
In this study, haptic device is used to obtain skin sensation feedback by giving force, vibration, movement etc.When grasping objects or manipulating machines in virtual space, vibration motor attached to glove's fingertip will vibrate to obtain feedback.It is also used to obtain feedback by vibrating the fingertip as a warning when stepping on dangerous operations or misoperation.

System Block Diagram
This diagram shows how each device is interacting with other devices.Output defines that Unity is sending video signal to Oculus Rift CV1 and Oculus Software is controlling CV1's behavior.Unity also outputs serial signal to Arduino when object collides with user's finger in VR space.User's hands are tracked by Leap Motion mounted in front of HMD, inputs hands motion data to Unity and projects hands in virtual space.Haptic Glove represents a golf glove with motors attached and by wearing it user can obtain haptic feedback when user's hand collides with object.Its motor is controlled via Arduino.3 tracking sensors were used in this system to obtain accurate position and behavior of HMD.System block diagram of this developed system is shown in Fig. 1 3. Hardware

Arduino UNO
Arduino UNO has an 8-bit AVR microcomputer with low processing power.Although it is inferior to 32-bit Arduino with a high-performance processor such as Cortex message -and turn it into an output -turning on an processor, Arduino UNO is a representative Arduino board, so there are many related information such as example projects, circuits, and programs on books and online.In this study, Arduino UNO is used to communicate with Unity via USB and controls the vibration motor according to the Fig. 1 System block diagram of developed system.value that came from Unity.Both Arduino UNO and vibration motors gets power from USB.

Motor Circuit
The current obtained from each pin of the Arduino board is small.Although it is possible to drive one LED, if you try to move a device with high current flowing such as a motor or LCD constantly, a great load will be applied to the Arduino side and there is a chance of damaging the Arduino.In this study, it is necessary to avoid this because five motors are controlled by one Arduino.As a solution, a motor circuit was fabricated using a MOSFET which can flow a large current with a small current.Since the MOSFET can switch current at ultra-high speed, there is no worry of latency whatsoever.
Regarding the response delay, we determined that the serial communication speed between Arduino and PC is the default value of 9,600 bps, so it takes about 1.04 ms per byte in time.In general, this value is low as the data communication speed, but this time only simple data transmission / reception, setting the communication speed above this value may cause data to be lost, so the serial communication speed was set to 9600 bps.
An LED was sandwiched between each MOSFET and the vibration motor.As a result, it is possible to confirm the operation simply and it is possible to confirm at a glance which finger is touching from a third party as well.Fig. 2 shows developed motor circuit using MOSFET.

Haptic Device: Gloves
Various glove type haptic feedback devices have been developed so far but it cannot be used for this study.Existing glove-type haptic feedback devices use their own tracking system and are not designed to be used in conjunction with Leap Motion.Since Leap Motion is used for hand tracking in this system, they are difficult to use in combination with Leap Motion.Also, their affinity with game engines such as Unity and Unreal Engine is unclear.Therefore, it is necessary to independently develop a glove type haptic feedback device that can be used with Leap Motion.
According to a Team of Leap Motion Developers, Leap Motion's tracking performance depends on lighting and environment, but that is only when tracking in bare hands.When it changes to gloves, its material and color also effects tracking performance.Also, the controller is optimized for human skin.Therefore, a material that resembles human skin in IR needs to be selected to achieve better tracking performance.Since Leap Motion controller contains a stereo IR camera, it can easily be tested which material performs best by just looking at the glove though the cameras.The brighter the hand, the better it will track, especially at far distance.Also, from their experience, cotton and nylon gloves showed quite bright in infrared and worked well.However, materials that are reflective didn't track well and should avoid these.It is also best to keep the glove as skin-tight as possible so that the hand proportions look correct in the virtual environment.Based on this, 4 different types of gloves shown in Table 1 were chosen to test which one performs the best in tracking (Table 1).
The test was done to each glove by looking at the glove through the Leap Motion controller mounted in front of Oculus Rift as shown in Fig. 3. VR Visualizer from Leap Motion Software Development Kit (SDK) was used during this test.The first image is a bare hand to obtain reference point to select.Yellow line is an actual hand and the bone is a data Leap Motion tracking (Fig. 4).
As a result, the golf glove performed the best, and the work glove performed the least.Although, at a glance, the disposable glove seems to be performing the best but since it is meant to be disposed, it was not comfort to wear long period of time.Therefore, the golf glove was chosen to use in this research.In addition, the golf glove was the most skin-tight of all four, and its white color showed brightest of all four.By looking at these figures, the lighter the material color is, the more it shows bright in VR Visualizer.This is because Infrared sensor can react to the color close to the white color rather than black color.This is because those having visible wavelengths such as infrared rays, ultraviolet rays and other which has vicinity wave to this absorbs black and easily reflected to white.Good example is a Line tracing car, as it judges the course with white and black using infrared sensor.From this result, it can tell that the infrared sensor used in Leap Motion controller is a reflection type, not a light shielding type which is detected by blocking infrared rays.

Haptic Devices: Types of Vibration Motor
For the vibration motor, three types of motors were selected (Fig. 5).Disk type brushless vibration motor, cylindrical eccentric type vibration motor and linear vibration motor.Selection method was to fix each motor to golf glove at the position corresponding to the antinode of the finger and each motor was vibrated several times.
To control each motor, Adafruit's haptic motor controller DRV2605L was used.This is a module for controlling vibration motor to present the tactile sense, and can give a click feeling or make a smooth vibration change etc.Moreover, 123 kinds of vibration patterns are stored in the memory, and by using I2C communication, it can freely control from a microcomputer such as Arduino.In this module, both the type of Eccentric Rotating Mass motor, ERM and the Linear Resonant Actuator, LRA can be o p e r a t e d , w h i c h s u i t s b e s t f o r t h i s t e s t .
The selection criterion is whether it is possible to obtain a sharp pseudo click feel or not, like the Taptic Engine.Taptic Engine is a name Apple created for its technology from a combination of "tap" and "haptic feedback" and provides sharp tactile sensations in the form of vibrations to users of Apple devices like the Apple Watch, iPhones and MacBook laptops.
As a result, the cylindrical vibration motor was able to present the vibration closest to Apple's Taptic Engine.The response of the disk type brushless vibration motor was slightly slow, and the vibration was weak.Regarding the cylindrical vibration motor, because the weight of eccentricity was small, the response was quick and the vibration was sharp.

Haptic Devices: Vibration Motor Mounting Position
Adopting the same method as above, position of the vibration motor was also selected.As a criterion for selection, it was decided whether to obtain a more realistic sense of touch without disturbing the operation.The position to install is verified four different positions.The finger top (nail), the right side of the finger, the bottom (ventral) side, and the left side of the finger was in the picklist (Fig. 6).
As a result, there was no significant difference in feeling when attached to the belly and nail side of the finger.However, when attached to the side it was not actually something hitting your fingers, it was just a finger being vibrated.In addition, when attached to the side fingers interfere with each other, and it was impossible to move the finger properly.When attached to the ventral side of finger, there was a feel of something not working right when making a fist, and since that neither belly side nor nail side felt much of a difference in feeling, it was decided on the

Software 4.1 About Unity
Without recreating the machine tool in 3D model from the beginning, we created a system capable of simple haptic feedback, and confirmed the usefulness of feedback by vibration and haptic device fabricated in this study.
This system uses Unity 5.6.2f.For version, when using Oculus Software and Leap Motion SDK, this environment works best at this present stage.When the blue button is pressed, user's finger vibrates and produces haptic feedback.Each finger has its own script to vibrate when something collided.Specifically, when index finger collided with button, the motor on index finger vibrates.

About Interaction Engine 1.0.1
Interaction Engine 1.0.1 was used for hand's collision behavior.The objective underlying the Interaction Engine is to properly handle the interaction with digital objects correctly.With some simple contacts, there are also interactions that require more complex processing.For example, when a user manipulates an object by hand, the following relationship is considered: "Contact", "Grab", and "Approach".The act of "Contacting" the object with hand by users is one.Whether the object roll, break, or an object through hands passes through is mentioned.The user "grasps" the object by hand.What kind of treatment is necessary for grabbing naturally or releasing it naturally?
What users are seeking is a consistent experience that is common across applications.If the feeling of operation differs depending on the application, the user will be confused and feel stress.To avoid this phenomenon, Interaction Engine does important roll.

Interaction with Arduino UNO
There are some items that need to be set for serial communication between Unity and Arduino.In Player Settings, "Virtual Reality Supported" in rendering column has to be checked so that Oculus Rift CV1 can be used as HMD.In addition to that, "Api Compatibility Level*" in Configuration column has to be changed .Net 2.0 subset to .NET 2.0.This is to use the System.IO.Ports.SerialPort class which performs serial communication from Unity, and the .NET 2.0 Subset does not include the System.IO.Ports.SerialPort class.
Secondly, a specific script called "SerialHandler.cs"has to be created.This is the part that Unity reads and writes to messages that it has done by serial communication.This is applied by attaching Serial Handler to the object you want to use.Since our Arduino was connected to port 9 this time, we set Port Name as COM9.Since the port number varies widely for each PC, this value has to be changed each time the environment changes.The baud rate is set to the value set for Arduino and the default is 9600.
Arduino is programmed when arbitrary data was received via serial communication and vibration is presented to the user with the vibration pattern associated with arbitrary data.More specifically, if user pushed a button in virtual area with an index finger, the tip of index finger vibrates a moment.

Developed user interface
Based on above software, we developed a user interface which can control via haptic device.If user push the button on VR space, the push feeling is transmitted to user by vibration motor attached to nail side of gloves which is user are wearing (Fig. 8).

5.
This system was verified by 15 people to measure the usefulness and usability of this system.We did the verification of how users felt when they pushed buttons,  First of all, it turned out that 63.3% of users actually feel "pushed" against the item "Do you feel that you are actually pushing the button?"At the stage immediately after the system was completed, the delay on the system side was remarkable and it was requested to improve the delay, but thanks to the Interaction Engine 1.0.1, it was possible to almost eliminate the delay, and in actual verification most people not feel almost any uncomfortable delays.However in the other hand, few users still felt some minor delay and I would like to set as a future task.
It seems that there is not much discomfort to the nail side, but there seems to be some people who also felt a sense of incongruity at all.User who felt discomfort said that their finger was just vibrating rather than tactile feeling.However, this opinion is under assumption, we think that this method is still useful because there is a study that actually presents vibration to the nail side and reproduces tactile sense to ventral side.However, the point that the tactile sense could not be presented sufficiently with our tactile feedback device needs to be improved.
Since Leap Motion sometimes occasionally misrecognized the haptic glove and collision judgment showed strange behavior, the stress level at the time of system operation was also verified at the same time.The stress meaning whether you could do what you wanted to do.The result shows 30%of users felt the stress during operation.As the claim which was the most frequented was, "Although I pointed my index finger, the sensor misrecognized other fingers, I felt stress." There is a certain feeling when vibration is presented to the fingertip with the head mounted display attached rather than just looking at a flat screen, but it is slightly different from the feeling of operating like the reality.Pursuing further human tactile sense is more required.However, it is not an exaggeration to say that the VR environment using the head mounted display is perfect for training of dangerous work because users can experience realistic first person situation without concern for injury.In addition, by presenting sensory feedback, it is possible to reproduce the same feeling as in real feeling of operation.

Conclusion
Since this system does not depend on the environment and device, it can also support many various HMDs, such as HTC Vive, Gear VR and others.Indeed Leap Motion is now developing an SDK for Android device so Android can also be able to handle this system in the future.Doing so makes it possible to use this system with common Android devices without using any large-scale spaces or expensive equipment.However, since other glove type haptic devices that are compatible with Leap Motion have not been developed yet, there is still a room for improvement as a system including haptic feedback.From the point that it is easy to introduce at a low cost, we think that we can also cope even when someone wishes to learn correct usage of machine safely even for ordinary people, or someone who wants to try new machine with actual sense close to reality before purchasing new instruments.
Although this time it has been verified only the operability and usability of VR UI used in conjunction with haptic devices, I would like to set as the future task so that the same machine tool can be operated in VR space.In addition to that, increase the types of machines to operate in VR space is another future task in order to increase flexibility and young engineers can master machine tool technologies other than lathes.In addition, since we had few subjects this time, we would like to further increase the number of subjects and to verify the usefulness of this system.

Fig. 6
Fig. 6 Mounting positions of vibration motor

Fig. 9
Fig. 9 Result of system and haptic device test

Table 1 4
different types of gloves considered in this development.