Study on Hand tracking with IMU

Many devices that capture hand movements and body movements have limited tracking range. For example, Kinect which is a motion capture device using a stereo camera can perform motion tracking only in the range captured by a stereo camera. Therefore, in order not to be trapped within the capture range, it is necessary to perform motion capture using a sensor that does not limit the measurement range. Therefore, we will capture the posture and position of the hand using IMU. However, in order to detect movement in the IMU, it is necessary to integrate the acceleration twice, and the movement diverges due to the influence of sensor noise. Therefore, in this paper, we propose a method to estimate hand position from arm posture and perform hand capture.


Introduction
Many devices that capture hand movements and body movements have limited tracking range.For example, Kinect, which is a motion capture device using a stereo camera, can perform motion tracking only within the range captured by a stereo camera, and Leap Motion, which can track the motion of the hand, also becomes a tracking operation only in the range that can be captured by the infrared camera ing.
Therefore, in order not to be trapped within the tracking range, it is necessary to perform motion tracking using a sensor that does not limit the measurement range.In this research, focusing on tracking movement of hands, we aimed to develop a hand tracking system that is not limited in tracking range using IMU.
The absence of limit in the hand tracking area means that the user interface by hand tracking which was originally only in the PC screen using Leap Motion etc. can be performed anywhere without restricting the place, You can use the user interface by hand tracking in various places.This system is a new input device that replaces smartphones that are commonly used as interfaces to computers that are increasingly used around the world due to IoT, which is becoming more active in recent years.

Method
The mechanism of the tracking system developed this time is to detect the angle of the user's arm and estimate the position of the hand by the IMU worn on the arm.

Estimation of arm attitude in IMU using EKF
Obtain attitude angle of user's arm from IMU sensor data.It is assumed that the IMU is attached to the arm of the user as follows.

Fig. 1. Attaching the IMU to arms
The axis of the IMU is defined as the x axis parallel to the direction of extension of the forearm from the elbow to the hand and is defined as the z axis in the normal direction of the skin of the back of he hand, the y axis in the outer product direction of the z axis and the x axis Was defined.The origin of the IMU is near the base of the hand.The attitude of the IMU was defined by the Euler angles of the z-y-x system (angles around the xyz axis: roll, pitch, yaw).
Knowing the attitude of the IMU, you can determine the attitude angle of the arm.Estimation of the attitude of IMU used EKF (extended Kalman filter) [ The prediction step is The update step is However, is Jacobian In the prediction step, the state of the current system is predicted from the control vector and the state of the system one step before.In the updating step, the Kalman gain is calculated from the error covariance matrix, and the current system state is estimated again from the measured sensor value (measurement vector).
The posture angle of the IMU was estimated using the above method.Knowing the attitude angle of the IMU, you can know the attitude angle of the arm, so you can estimate the position of the hand

Estimation of hand position
In this example, the positions of the hands in the up and down direction and the left and right direction, which decided to measure the position of the hand in two dimensions, were defined as shown in FIG.
Let the length of the forearm be [m], the coordinates of the hand are [m] is the posture angle of the arm estimated by the IMU.
Estimated by this equation.Regarding the accuracy of hand tracking, it is difficult to accurately measure the coordinates of the hands in the first place, so it has not been verified to date.

Conclusions
In this research, we developed a hand tracking system with no limit in tracking area using IMU.As an experimental result, although the approximate position of the hand could be estimated, it was a result that the accuracy could not be verified.
In this experiment, we performed experiments as an input device to the PC in front of the PC in the experiment, but since this device hands tracking using only the IMU, treat it as an input device anywhere outdoors You can do.For example, you can manipulate the drones remotely using the movement of the hand, release the smart lock using the hand gesture, and various ways of using it.
As a future prospect, it is a goal to combine it with a device that can track the shape of the fingertip [2] and make it a more user-friendly system.
1].The state equation and the output equation of the state space model in EKF are shown below.However, the state vector of the system is with the attitude angle [deg] of the IMU, the control vector is the IMU 3 axis angular velocity [deg / s], , the measurement vector is the IMU 3 axis acceleration [g ], , system noise is , measurement noise is , and filter execution period is [s].The state equation of the system is The output equation is However, The algorithm of the extended Kalman filter is shown below.By repeating the prediction step and the updating step each time the sensor value of the IMU is updated, it is possible to estimate the attitude angle of the IMU.Where is the error covariance matrix of the state estimate at time t.