A Study on Tiredness Measurement using Computer Vision

: In this paper, we studied tiredness measurement based on several di ﬀ erent detection methods in real time. We know that the driver tiredness is one of the major causes of tra ﬃ c accidents. So tiredness detection can play a vital role for preventing road accidents. By developing an automatic solution for alerting drivers of tiredness before an accident occurs, this could reduce the number of tra ﬃ c accidents. The Haar-cascade classiﬁer is exploited based on Haar-like features to ﬁnd the eyes. The main purpose of the Haar-cascade classiﬁer is to classify closed or open state of the eyes. If we can notice that the eyes are closed for a predeﬁned span of time, we consider the state of the eyes can be closed. Based on this closed-state of the eye, a notiﬁcation (like alarm) is initiated to alert. We have detected only right eye for saving processing load on the system. The reason is that when a person closes his eyes he usually does not close one eye, but both eyes at the same time. Several steps are taken into account for this system; we ﬁrst capture the frame from the webcam. Then we need to detect face as well as eye. To detect blinks, we process ROI (region of image) of pupil area. Our result is found to be satisfactory.


Introduction
Computer vision-based activity recognition, behavior understanding, gait analysis, emotion deciphering, fall detection assessment, eye-blink study, face recognition, etc. are major application areas for the last decade in different fields [3] [4] [7] [9] [12] [13] [16] [23] [24]. Studies related to tiredness evaluation or eye-blinking rate becomes necessary in this arena for some important applications, especially in driver's alert system from micro-sleep. Driver alert system is very much required if possible [1] [2] [5] [35] [36]. This field is very challenging due to multiple factors related to the position of the eyes, head movements, background lights and vicinity, distance, etc. In this paper, we attempt to deal this important research area. A large number of road accidents are occurred due to the driver tiredness. So in the field of active safety research, developing a system for assessing driver's alertness level is becoming a central issue [1] [2] [5]. Eye detection and eye blink detection are very informative that has versatile applications. Eye blinking has a wide range of applications in human-computer interaction (HCI) like the eye typing applications as in [8], for disabled persons to operate all mouse operations, driving safety for general people like Google glass and so on.
The primary goal of this paper is to produce a system to detect tiredness using three different methods based on thresholding, distance from the eyelid and angle of the head.
Thorough these three methods, the human eyes are the main aspect as eyes are one of the most important salient. The problem of human eye detection and blink detection has received significant attention during the past several years because of human-computer interaction and surveillance applications [14]. In this system high resolution cameras were used, for example in [10] with a focus on high resolution images of the eye-region which simplifies eye-features detection. In the past ten or fifteen years, the driver safety problem was the main concern for many countries all over the world and begun to pay attention to that problem. In accordance to do that they started to investigate the driver's mental states relating to driving safety. That is why they proposed some driver tiredness detection methods which can detect whether the driver is tired, such as drowsiness, and for generating some warning alarms to alert the driver [1] [2]. Using computer vision technology of the existing approaches systems for extracting characteristics of a driver is very successful though but it is a challenging issue due to a variety of factors. External illumination interference, lighting conditions and the variety of eyes moving speed is the main reason [11].
The paper is organized as follows: after introduction we present some related works in Section 2. In Section 3, we presented the methodology of this work. Then we presented the experimental results and analysis in Section 5. Finally, we concluded the paper in Section 6 with a few future work guidelines. A real time method based on some captured video and image processing algorithms for eye blink detection is proposed by Mohammed et al. [21]. But it works on an offline captured video not online video. USB camera is used by Chau and Betke [14] as well as Magee et al. [26]. A blink detection system based on real time eye tracking with USB camera is proposed by Chau and Betke [14]. They used template matching approach to detect eyes and analyse eyeblinks. They achieved 95.3% blink detection accuracy. It can distinguish between two eye-states, open and closed. On the other hand Magee et al. [26] also implemented in USB camera. In their system, they used multi-scale template correlation for tracking the face, left and right eyes was exploited to detect if the computer user is looking on camera. To control computer applications they used the detected eye direction.
Gruman et al. [25] proposed two video based HCI tools called Blink Link and Eyebrow Clicker. The first tool can automatically detect eye blinks of a user's and it can measure their duration. A long blinks can trigger mouse clicks, while short blinks are ignored. Blink patterns are used for this system. The other tool can automatically detect trigger a mouse click when a user raises his or her eyebrows. Webcam-based system is also available, e.g., in [8] [20]. Galab et al. [8] presented a webcam-based system for detecting eye blinks accurately without any restriction on the background. It automatically classifies the eye. The proposed system is tested with the users who wear glasses. Elahi et al. [20] proposed a webcam-based eye-tracker, especially for low power devices. They used mean of gradient vector algorithm to detect eye-ball center position. Tobii eyeX device [28] is one of the most fundamental things which can now be used to control PC by using eye blink. Uzma et al. [7] explored Tobii eye-tracker to visual face scanning and emotion perception analysis. Ref. [6] proposed a system to analyze the performance of eye-blink detection using android mobile phone.
Eye Tribe tracker [30], the first affordable eye tracker that precisely determines the on-screen gaze position. It also enables interact devices with human being by blinking. We can summarize this related field in the Table 1.

Methology
This paper is to produce a system to detect tiredness measurement using three different methods based on (A) thresholding, (B) distance from the eyelid, and (C) angle of the head in a real-time approach. In accordance to do this detection, first we need to detect eyes with Haar-cascade classifier using webcam and try to detect the presence of tiredness with these methods. The whole process is illustrated in the flowchart (see Fig. 1).
The real time application can be divided into six steps: capture image from live video webcam, face detection, eye detection, processing of ROI, blink detection and alert. The first step of the total process is the capturing frame from webcam. Afterwards, the colored frames will be converted to gray scale frames by extracting the luminance component [18]. When the program starts, at first, it searches for faces in front of the camera. If there is none then it starts monitoring changes of the image. If the change is significant then it checks for face in the image. The Haar classifier is used for face detection. Haar-cascade is feature-based face detection algorithm in cascaded form. It detects face features one after one in cascaded form. If one fails to detect it doesn't advance to the next feature. Feature detection also contains a value for relative percentage of matching. If the total percentage of matching of all key features is less than threshold then there is no face in the region. Haar classifier rapidly detects any object, based on detected feature not pixels [6] [15] [17]. The main advantage of Haar features is their computational efficiency [15]. The third step is to detect the eyes. Eye detection can be done by various approaches. We consider the Haar cascade classifier. We do not exploit any feature based eye detection algorithm. We just use the fact that, if the face is not out of the ratio eyes are placed in a certain position of face while detecting the eye (in Fig. 2).
ROI means region of image. We have taken only one fourth portion of the face and only a single eye for further analysis (Shown in Fig. 3).
Blinking is rapid closing of the eyelid. The detection of blinking is based solely on observation of the previous step using the template of the user's eye. In this approach, open eyelid is taken as a template for detecting eye blink. As the user's eye opens during the process of a blink we try to match by frame. From the flow chart (in Fig. 4) we can de-  duce some steps, are describing below in short form: Step 1: In the first step, we read data from webcam. Read data file containing feature details of face. Search for face feature match in every frame. Crop the face region.
Step 2: Crop the eye-region from the face region. Eye region also contain eyebrows.
Step 3: Reduce eye region, take ROI of pupil area and track the pupil.
Step 4: Check the conditions for tiredness detection Step 5: If the assigned condition is true then the tiredness is detected and the algorithm will notify otherwise the system will find another sample frame from the webcam.

Experimental Setup
We developed a real-time webcam based eye-blink detection system using python programming language. The resolution of the webcam is 1280 × 720 pixels. Python is an interpreted, object oriented high level programming language with dynamic semantics. And OpenCv is the most popular and portable open source computer vision library available for developers. OpenCV-Python makes use of Numpy, which is a highly optimized library for numerical operations with MATLAB-style syntax. All the OpenCV array structures are converted to and from Numpy arrays. This also makes it easier to integrate with other libraries that use Numpy such as SciPy and Matplotlib [32]. We used a personal computer with core i7 processor and 640 GB hard drive to run the python program. This system is very easy to configure and use as well as works with webcam and runs at a frame rate of 30 frames per second [14].

Tiredness Detection Based on Thresholding
If E(x, y) is a threshold version of f (x, y) at some threshold T.
Here f (x, y) is a function of black pixels.When the black pixel is greater or equal than the threshold value then the eyes are open and when the black pixel is smaller than the threshold value then the eyes are closed. This system provides the necessary analysis: Step 1: In the first step, we read data from webcam. Read data file containing feature details of face. Search for face feature match in every frame.
Crop the face region.
Step 2: Crop the eye-region from the face region. Eye region also contain eyebrows.
Step 3: Re- Step 5: If the minute limit is smaller than 'how long the eyes are closed' the system will show alert, and if the minute limit is greater than 'how long the eyes are closed' the system will find another sample frame from the webcam.

Tiredness Detection Based on Angle of head
Tiredness detection based on Angle of Head is a real time detection approach. As the user's eye opens during the process of measure the angle of head we try to match the open eye template in the eye region rectangle area frame by frame. We can also mention a theorem of angle measurement.
Here is the relationship between the radius, and the central angle in radians (in Fig. 5). The formula is S = rθ where s represents the arc length, S = rθ represents the central angle in radians and r is the length of the radius. Firstly, we need to convert the value of θ to degrees because the whole system works only in degrees. A fixed value of θ is taken to detect tiredness of the driver. We compare the tiredness of the driver at different angle level of the head (in Fig. 6). We assume a right angle where θ = 90 • as a straight head. There is 'tired' and 'heavy tired' notification when the value of θ is 3 • above the right angle (θ = 93 • or θ = 93 • + +) and further increases.
This system also provides the necessary analysis using previously discussed method of face detection and eye blink detection.
Step 1: In the first step, we read data from webcam. Read data file containing feature details of face. Search for face feature match in every frame. Crop the face region.
Step 2: Crop the eye-region from the face region. Eye region also contain eyebrows.
Step 3: Reduce eye re-gion, take ROI of pupil area and take pixel data from ROI pupil.
Step 4: Check the number of black pixel with predefined black pixel data, if the black pixel exceeds the predefined black pixel value then the eyes are closed and if not then the eyes are opened.
Step 5: If the minute limit is smaller than 'how long the eyes are closed' the system will show alert, and if the minute limit is greater than 'how long the eyes are closed' the system will find another sample frame from the webcam.

Tiredness Detection Based on Distance from Eyelid
In this approach, open eyelid is taken as a template for detecting distance. As the user's eye opens during the process of measure the distance we try to match the open eye. Though we used a fixed balck pixel value (for this case minimum black pixel value is 80) to measure the tiredness. We can use the 'Euclidean distance' formula while measuring distance from eyelid. The Euclidean distance is a distance measure between two points based on Pythagoras' theorem. According to the Euclidean distance formula, the distance between two points in the plane with coordinates (x, y) and (a, b) is given by This system also provides the necessary analysis using previously discussed method of face detection and eye blink detection.
Step 1: In the first step, we read data from webcam. Read data file containing feature details of face. Search for face feature match in every frame. Crop the face region.
Step 2: Crop the eye-region from the face region. Eye region also contain eyebrows.
Step 3: Reduce eye region, take ROI of pupil area and take pixel data from ROI pupil.
Step 4: Check the number of black pixel with predefined black pixel data, if the black pixel become smaller than the predefined black pixel value then the eyes are closed and if not then the eyes are opened.
Step 5: If the minute limit is smaller than 'how long the eyes are closed' the system will show alert, and if the minute limit is greater than 'how long the eyes are closed' the system will find another sample frame from the webcam.

Experimental Results and Analysis
For the analysis of the result, we divided this section into three categories (according to the methodology in Section 3): Tiredness detection based on (i) Thresholding, (ii) Angle of Head, and (iii) Distance from Eyelid We took 50 samples in lighting condition for detecting tiredness for every setup based on pixel computation.

Thresholding
The overall performance was very satisfactory; we have not seen any error in face detection. However, after face detection, depending on the camera quality and lighting conditions sometimes the eye position cannot be found so getting centre position gives error. When the light source is behind the face the performance may degrade to some extends and may also detects multiple faces in the frame which creates some trouble. But most of the time it detects biggest face and work with it. In poor lighting condition the whole eye region may go dark. Here is  the analysis of detected eye in different light conditions. We have taken some samples in night/indoor and some samples have taken in daylight conditions (see Fig. 7). We also have taken closed and open eye state samples. The accuracy is much better in lighting condition.

Experimenting in outdoor environment
In our experiment, we took the camera in different lighting conditions. At first we illuminated face of the person with light from the front, which is the most ideal case. We have got 5 errors out of 50 samples, which gave us 10 percentage of errors. Then we took the illumination from the back and side of the person. In this case, we have got 9 errors and 7 errors. Both gave us 18 and 14 percentage errors respectively. Our system is based on image threshold method so skin complexion is vital challenge. Hence we have tested our system for two different skin complexions. Dark skin complexion gave us 12 errors out of 50 which are 24% errors. For bright skinned persons we have got only 12 percents of errors which is 6 errors out of 50 samples. We also tested based on gender. The whole result is shown in the table (Table 2) and for comparative understanding chart diagram is provided (shown in Fig. 8).

Experimenting in indoor environment
For night simulation, we tested the system in indoor. As illumination level was very low, we had to use a table lamp for better reading. We took the camera in different lightening conditions. At first we illuminated face with light from the front, which is the most ideal case. We have got 4 errors out of 50 samples, which gave us 8 percentage of error. Then we took the illumination form the back and side of the person. In the first case, we have got 12 errors and for the side case we have got 10 errors. Both gave us 24 and 20 percentage of errors respectively. Persons with dark skin complexion gave us 16 errors out of 50 which is 32% error. For bright skinned persons, we have got only 10 percent of errors which is 5 errors out of 50. For both Boy and Girl, we have got 10 percent of errors. The whole result is shown in the table (in Table 3) and for comparative understanding chart diagram is provided (Fig. 9).

Experiment on Angle of Head
For this approach, first we need to track eyes because we are dealing with only eyes, not head. When we move head we also move our eyes so if we track our eyes we can detect head angle. If our head angle became wider than predefined value  then the alert notification will notify 'tired' (when θ > 3 • ) or 'heavy tired' (when θ ≫ 3 • ) notification according to the angle of the head. Let's define a predefined value (θ = 3 • ) for the angle of the head of the driver. We need to divide the value into two parts if the angle of head exceeds the first part (θ > 3 • ) then the alert notification will show only 'tired' notification. And if the angle of head exceeds second part (θ ≫ 3 • ) then the alert notification will show only 'heavy tired' notification (see Fig. 10).
We took 50 samples in Daylight as well as Indoor condition for detecting tiredness. The result was enough satisfactory. This approach can detect (among 50 samples) almost 41 samples in daylight and 37 samples in indoor at the (θ > 3 • ) condition and 43 samples in daylight as well as 38 in indoor at (θ ≫ 3 • ) condition. The errors and percentage of errors are listed in (Table 4).

Experiment on Distance from Eyelid
The main concern of this approach is the black pixel of the eye if the black pixel become smaller to smaller, the distance will be longer and longer, that is why the alert notification will notify 'tired' or 'heavy tired' notification. Let a predefined (a) Tired state (b) Tired state (c) Heavy Tired state (d) Heavy Tired state Figure 11: Tiredness detection based on distance (with 'tired' and 'heavy tired' notification). value for the black pixel of the driver. We need to divide the value into two parts if the black pixel (in dpi) exceeds the first part (> 80) then the alert notification will show only 'tired' notification and if the black pixel (in dpi) exceeds second part (≫ 80) from the eyelid then the alert notification will show only 'heavy tired' notification (see Fig. 11). We took 50 samples in Daylight as well as Indoor condition for detecting tiredness. The result was enough satisfactory. This approach can detect (among 50 samples) almost 41 samples in daylight and 35 samples in indoor at the (> 80) condition and 42 samples in daylight as well as 39 in indoor at (≫ 80). The errors and percentage of errors are listed in (Table 5).
In this paper, we have implemented alarm alert system in our application by software, not on any hardware implementation. It works well though it has some error due to light conditions and bit execution processing time. It can be solved by using high configured processor on the computer while processing frame by frame. And we need to do more research work for getting better accuracy. We have developed a real time application for the purpose of safetydriving. One of the major reasons for vehicle accident is micro-sleep of a driver on highway. So we are trying to solve this problem by implementing microsleep alarm, alert system through this real time application. We took 50 sam-ples in lighting condition for detecting tiredness based on pixel computation. The result can be improved in a better lighting condition with a better perform-able processing system. This kind of research can be applicable in future to understand the quality of workers in industries and offices. If some workers are found to be sleepy or tired at the end of the day, they may not get overtime. Countries like Japan, Korea as well as some European countries population are increasing very rapidly and hence, less young people to work for huge people. So selection of quality employees and extraction issues can be assessed through automatic analysis of tiredness of workers.

Conclusion and Future Work
This paper presents a real time application on face detection and tiredness detection, which can be developed for a simple human-computer interaction system. It only requires a webcam with abundant light on face. So the objective of this paper is to develop a system to detect face and measure tiredness. In this paper, we produced a system to detect tiredness using three different methods based on thresholding, distance from the eyelid and angle of the head. The results were satisfactory in every method. In thresholding method the errors were quite reasonable and it was almost 6% to 8% while the error was almost 14% to 18% in the angle of head and distance from the eyelid respectively. We have done this experiment with 50 samples. Our algorithm performs well with little error, which can be acceptable considering the level of difficulties in computer vision. In future, we look forward to enhance performance by testing on different number of people in different lighting conditions. We may use cursor movement in future for general and disabled person to operate all mouse operation by eye movement and blink, driving safety for driver from micro-sleep and making life easier.