Research on Fatigue Driving State based on Multi-source Information Fusion

To improve the accuracy and acceptance of driver fatigue recognition in practical applications and vehicle safety, by comparing the driver's face expression features, car driving characteristics and other multi-source fatigue information collection and driver acceptance, a fatigue driving state recognition scheme based on multi-source information fusion is proposed. The scheme includes two cameras and a steering angle sensor. The front-up camera collects eye position, frequency of blinks, frequency of glancing, staring time, and other indicators; The side camera collects the information of driver's head position; the steering angle sensor collects the steering wheel angle data. The fatigue driving state of the driver is classified and defined by a multi-source information detection method such as pupil characteristics, head tilt angle and steering wheel angle data. Through a comparison of Test Schemes for fatigue driving, it is concluded that the simulation driving experiment scheme has the best comprehensive index in terms of safety, economics, driving state fit and so on. Based on the work above, an experimental platform for fatigue driving recognition was established.


Introduction
Traffic accidents are one of the serious social problems in the world.According to the World Health Organization (WHO), casualties caused by road traffic accidents by 2030 will be one of the main causes of human death and injury.According to statistics on traffic accidents in many countries, fatigue driv ing is one of the most important causes of traffic accidents (1) .There are at least 100,000 traffic accidents caused by fatigue driving in the United States each year, and 10% o f accidents caused by fatigue driving in traffic accidents in the Un ited Kingdom.
Accidents caused by fatigue driving account for 20.6% of casualties in France.Therefore, the research on drivers fatigue detection is of great significance.
Many countries have begun research and development of fatigue driving detection and Driver Fat igue Monitor System.S.A.M (Steering Attention Monitor) is developed by Digital Installations of the United States that monitors abnormal movement of the steering wheel (2) .The AWAKE project of European Union realizes driver fatigue classification by information fusion technology and develops fatigue driving early warning system by sound, light flashing and seat belt vibration (3) .
Chinese research on fatigue driving started late.The JLUVA-DFWS system developed by Jilin Un iversity infers the driver's mental state based on eye and lip opening, and give warn ing message if the driver appears fatigue state during driv ing.BAWS of BYD infers the driver's fatigue status based on the driver's physiological signals, alerts the driver and takes action.
The acceptance of the testing equipment by the driver in practical applications is not taken into account in the current fatigue detection research.And the results of fatigue detection are highly correlated with the environ ment.However, some experimental platforms of fatigue driving in Chinese universities simulat ing the interio r environment are different from the real driving environment.Therefore, this paper studies the fatigue driving state based on mu lti-source Information fusion.The d river drives in the mod ified abandoned vehicle in front of the screen showing traffic information.Based on the characteristics of driver's pupil fro m the front-up camera, the in formation of driver's head position fro m the side camera, and the steering wheel angle data fro m the steering angle sensor, the driving state of the driver is discriminated.

Analysis of Schemes for Collection of Fatigue Information
Nowadays, the method used to detect driver's fatigue mainly includes three kinds.The first one is a method based on examin ing driver's physiological info rmation; the second is a method based on the driver's behavior; the third one is a method based on monitoring the situation of the motor vehicle.In the first method, the mechanism used to detect  (4) .When fall into fatigue,δ wave and θ wave increase, while α wave and β wave decrease (5)(6)  Where, the heart rate and the variation of heart rate are significant physiological criteria for gauging the fatigue state (7)(8) .As the driver's fatigue increases, the ratio of low frequency and high frequency of the ECG is gradually reduced (9)(10) .But all censors define biological index must stick electrodes on the body, with high dependency and low acceptance.With high credibility and few changes to the car therefore it is suitable as a reference parameter.The degree of eyelid's closure expresses clearly human's fatigue.Therefo re, driver's fatigue recognition methods relied on this characteristic have fast result and high accuracy.With low dependency, high acceptance and accuracy, based on the characteristics of driver's pupil fro m the front-up camera, the driver's fatigue can be gauged.
There is few changes in the body, such as the installation of cameras and control systems without changing the core components that affect safe driving.
With this method through driver's head position in driving process to determine the driver whether is fatigued or not.The head position relates to the state of fatigue.
When driver is fatigued, his head position will change.Nodding or shaking will appear or even drowse.This informat ion can be used as the basis to determine the fatigue.Based on the information of driver's head angle fro m the earplug, fat igue driving can be identified exactly.The driver can accept it within the constraints and the installation of the control system makes small changes to the car.
When the driver starts to feel drowsy due to his attention was distracted, slow reaction movement is not flexib le, etc., and possibility leads to lane departure.Base on the information fro m the front-up camera about the movement of vehicle such as the variation of mot ional orb it, position, etc., the driver's fatigue can be gauged without change of the core co mponents that affect driv ing .It has minimal interference with the driver with low dependency and high acceptance.However, it is difficult in the evening because of lo w definit ion of wh ite lines.Affected by the environment, the accuracy is low.
When the driver is fatigued, the reaction is not flexible, the movement is slow or in dangerous situations, the driver adjusts the steer with big amp litude.Therefore we can use this feature to determine the driver's fatigue.Based on the steering wheel angle data easily to be ext racted fro m the steering angle sensor at the steering column, the driver's fatigue can be gauged.It has minimal interference with the driver with low dependency and high acceptance.However, it needs to change the steering wheel, which is the core component of driving.It is greatly affected by factors such as vehicle characteristics, driving habits and road environment.Other fatigue driving detection methods need to be combined because of low accuracy.
The paper analyzes and compares the schemes and characteristics of six kinds of collection of fatigue information.The results are shown in Table 1.

Table 1. Comparison of fatigue information collection
Schemes.T he head position (12) ★★★★★ ★☆☆☆☆ T he driver can accept it within the constraints

6
T he movement of vehicle (15) ★☆☆☆☆ ★☆☆☆☆ High acceptance Based on the above Analysis and Comparison of fatigue informat ion collection Schemes, the following conclusions are drawn: (1) The higher the dependence of fatigue information collection scheme on people, the less acceptable the user is; (2) The fatigue information collection scheme is more difficult to change the vehicle, and the lesser the opportunity to put into practical use; (3) Whether the fatigue informat ion collection scheme is applicable o r not is related to the accuracy and the total cost; (4) The lower the dependence on people, the lower the total cost, the higher the accuracy of the scheme, and the more promising it is in practice.
Therefore, co mb ined with the above conclusions, a fatigue driving state recognition scheme based on mu lti-source information fusion is proposed.The front-up camera collects eye position, frequency of blinks, frequency of glancing, staring time, and other indicators.The side camera collects the information of driver's head position and the steering angle sensor collects the steering wheel angle data.This scheme has low dependence on the driver, less modification of the car, reasonable cost, and improved fatigue recognition.

Analysis of fatigue driving state based on multi-source information fusion
Referring to the existing research on fatigue driving, the driver's driving mental state is classified and defined separately.The driver's driving state is divided into five levels, wh ich are the driver concentration, the driver distraction, the driver mild fatigue, the driver moderate fatigue and the driver deep fatigue.
When the driver's spirit is highly concentrated, the average eye blinks about 10-20 times per minute, the head moves within the normal range, and the percentage of nonsteering is 25%.When the driver is distracted, the attention is dispersed by other things.His eyes are faint, his head swings slightly back and forth, left and right and the percentage of non-steering is 31%.
PERCLOS refers to the proportion of time taken by the eye closure time.The P80 (16) with high correlat ion with fatigue is selected to define the driver fat igue level.When the driver is mild mental fatigued, blinking 15 times per minute [21][28], PERCLOS is 5.4%-14.7%,head tilt is 15 degrees, the percentage of non-steering is 38% and steering wheel angle standard deviation is 1.20.When the driver was moderately fat igued, the PERCLOS is 14.7% -23.3%, the head tilts 17.5 degrees back and forth, and the standard deviation of the steering wheel angle is 1.50.When the driver is in deep fatigue, eyes close for a long time ,the PERCLOS is greater than 23.3%, the pupil's brightness is significantly reduced, the head is tilted 20 degrees, and the standard deviation of the steering wheel angle is 1.81.
In order to facilitate the research on active intervention of fatigue in the future, the driver's driving state is classified into five states: driver concentration, driver distraction, driver mild fatigue, driver moderate fat igue, driver deep fatigue.The parameters such as pupil characteristics, head tilt angle, and steering wheel characteristics of each state are also divided, as shown in Table 2.

Analysis of Test Schemes
There are three kinds of driv ing fatigue experimental schemes, namely, the on-road experiment scheme, the simu lation driv ing experiment scheme and the Virtual Reality (VR) experimental scheme.
In the on-road experimental scheme, the driver drives on the selected experimental road.Based on the characteristics of driver's pupil fro m the front-up camera, the information of driver's head position fro m the side camera, and the steering wheel angle data fro m the steering angle sensor, the driving state of the driver is discriminated.
It is of high similarity of driver driv ing status while it is difficult to detect deep fatigue state with low security and high cost.
In the simu lation driv ing experiment scheme the driver drives in the modified abandoned vehicle in front of the screen showing traffic informat ion.Based on the characteristics of driver's pupil fro m the front-up camera, the information of driver's head position fro m the side camera, and the steering wheel angle data fro m the steering angle sensor, the driving state of the driver is discriminated.
In the simu lation driving experiment scheme, the driver does not travel on the road, so there is a slight gap between the driver's driving state and the driving state on the road.
However, the cost is low and the deep fatigue state can be simulated.
In VR experimental scheme, the driver wears VR glasses with analog road condition informat ion and sits in the car to simu late driving.Based on the characteristics of driver's pupil fro m the front-up camera, the in formation of driver's head position fro m the side camera, and the steering wheel angle data fro m the steering angle sensor, the driving state of the driver is discriminated.The cost is low and the deep fatigue state can be simulated.Ho wever, the driver's driving state has low similarity co mpared to the simu lation driving experiment scheme.

Experimental platform for fatigue driving recognition
(1) In the scrapped commercial vehicle, the steering angle sensor is installed in the steering column.(2) Camera A is installed in the range of about 30 degrees in front of the driver's seat, and camera B is mounted on the side.(3) The camera transmits the driver's driving status to the Advantech IPC-900 (Industrial Personal Co mputer) v ia 3.0 interface, 1.3 megapixels, 210 frames per second USB.
(4) IPC is installed outside the vehicle, and the C language is used for modeling and corresponding software programming to achieve driver fatigue recognition.(5) The screen is placed 1 meter in front of the car to simulate driving scene and road conditions.(6) The experimenter simu lates the driver, sits in the driver's seat, and simulates the driver's fatigue state through changes in the pupil, head tilt, and steering wheel angle.

summary
First, through the analysis and comparison of six fatigue informat ion collect ion schemes, it is concluded that the lower the dependence on people, the lower the total cost, the higher the accuracy of the scheme, and the mo re promising it is in pract ice.And a fatigue driving state recognition scheme based on mult i-source information fusion is proposed.The front-up camera collects eye position, frequency of blin ks, frequency of glancing, staring time, and other indicators; The side camera collects the informat ion of driver's head position; the steering angle sensor collects the steering wheel angle data.
Then the driver's driving state is classified into 5 levels, and each level of driving state is defined by mu lti-source informat ion such as face expression features and steering wheel characteristics.
Finally, this paper co mpares the three experimental schemes of on-road experiment scheme, simu lation driving experiment scheme and VR experimental scheme, and selects the simulat ion driv ing experiment scheme wh ich has low cost, high driver's driving state fitting degree, and can simu late deep fatigue state.And the scheme of the experimental platform was proposed.

Fig. 2 .Fig. 3 .
Fig. 2. Experimental platform B.The front-up camera collects the characteristics of driver's pupil, the side camera collects the information of driver's head position and the steering angle sensor collects the steering wheel angle data.The camera and the sensor transmit the driv ing state to the IPC via USB in real t ime, and the IPC determines the driving state of the driver through the software.

Table 3 .
Comparison of Test Schemes