A Gathered Images Analysis Method to Evaluate Sound Sleep

This paper proposes a method to evaluate a sound sleep using an image gathering technique and its analysis techniques. The proposed method consists of three phases; gathered images generation, gathered images analysis and sound sleep evaluation. The gathered images designed to gather sleep postures and their changes are generated at 1 second, 10 seconds, 1 minute, 10 minutes, 1 hour intervals and all times, respectively. In the gathered image analysis, the gathered images are analyzed by calculating difference values among the gathered images of 10-minute and all times. Then, the sound sleep conditions are evaluated by visual inspection and analysis results. In order to show the effectiveness of the proposed method, we conduct experiments using real movies and their images. In experimental results, we confirm that there were sound sleep conditions, bad sleep conditions and borderline cases by checking subjective evaluation using questionnaire and generated gathered images visually. Moreover, we confirm that the calculated difference values among the gathered images of 10minute and all times are different between sound sleep and other cases. Furthermore, the analyzed results show that the proposed method was successful in the sleep conditions classifications on four of five subjects. These results suggest that the gathered images analysis method is effective for evaluating whether sleep condition is sound sleep or not. In particular, it is important to calculate the difference values among the gathered images of 10-minute and all times to evaluate sleeping conditions.


Introduction
Sleep has an essential role to play in a quality of life for human. A quality of sleep takes on a major significance. If the quality of sleep becomes low, human may become less healthy compared to the high quality sleep like sound sleep. It is therefore important to have a good sleep. However, it is not easy to have the good sleep and know whether he/she has high and/or low quality of sleep. If he/she understand the sleep conditions by his/her self, it may be easy to have good sleep. To confirm the sleep conditions, he/she may watch videos recorded during sleep. However, it is difficult to watch the videos on daily basis because the videos are longer than 6 hours. If the sleep conditions are recorded, measured and confirmed easily, he/she understands his/her sleep conditions him/herself and feels relief. This paper attempts to record, confirm and evaluate the sleep conditions which are the sound sleep and otherwise.
There are lots of methods to record, evaluate and confirm the sleep conditions by analyzing bio-signals, which are electrocardiogram (ECG), electro-oculogram (EOG), electroencephalogram (EEG), etc., and videos recorded while sleep. In previous studies, polysomnography (PSG) is used to evaluate the quality of sleep conditions in sleep overnight generally. The PSG evaluates sleep conditions using the ECG, the EOG, the EEG, electromyogram (EMG) of the jaw and leg. Also, the airflow and blood oxygen concentration are monitored. Liang et al. evaluated the 5 stages of sleep conditions with an accuracy of 88.1% using C3 EEG signals in EEG sensing positions based on the international 10-20 system [1]. Zhu et al. evaluated the 6 stages with an accuracy of 87.5% using Pz EEG signals [2]. Singh et al. proposed a method to discriminate rapid eye movement (REM) from non-REM sleep conditions using the low frequency (LF) : high frequency (HR) ratio [3]. Boudreau et al. and Penzel et al. confirmed that the LF:HF ratio is different in each sleep conditions (arousal, REM, shallow sleep, moderate sleep, deep sleep) [4] [5]. When using the PSG, he/she has to wear the any sensors. Although the sleep conditions can be evaluated, it is difficult to use the PSG on daily basis because he/she cannot feel sound sleep while wearing the devices for the PSG. In non-contact system for monitoring and evaluating the sleep conditions, Murthy et al. proposed a method to measure the breathing function using infrared imaging with an accuracy of 96.4% [6]. Takano et al. proposed a method to measure the heat rate based on a time-lapse image with a correlation coefficient of 0.90 and 0.93 for the measurement of heart rate and respiratory, respectively [7]. Liao et al. proposed a method to classify the sleep and wake conditions using near-infrared video with an accuracy of 90% or more [8]. Michael et al. proposed a method to evaluate the sleep breathing using IR-sensitive camera with an error ratio of 3.4 and 5.0% for respiratory rate and heart rate, respectively [9]. Hossain et al. proposed a method to monitor and classify the sleep conditions for monitoring a sound sleep using wearable devices. In the experimental results, they achieved an accuracy of 78% to classify the sleep stages (sleep, awake, movement, getting up and sitting, and getting up from bed) [10]. Although the sleep stages and a part of sleep conditions like apnea syndrome are detected, classified and evaluated, there is no one directly evaluating the quality of sleep like sound sleep. Even if a part of the sleep conditions and the sleep stages have been evaluated quantitatively and detected respectively, the previous studies have not attached overall assessment of the sleep conditions like sound sleep, bad sleep and so on, and have not confirmed the sleep conditions directly.
To solve the issues of the previous studies, this paper proposes the method to confirm the sleep conditions directly based on toss and turn in his/her sleep and classify the sound sleep and otherwise comprehensively. There are information visualization techniques for observing long-term movement. These techniques are used to check the video of the surveillance camera. One of these techniques creates gathered images by gathering images within a certain amount of time into one image. Finkelstein et al. proposed a method to create gathered images hierarchically by smoothing the time axis direction [11]. Irani et al. proposed a method to track the moving objects by placing the captured images in a mosaic pattern on a single still image [12]. Akutsu et al. proposed a method to collect parts where changes occur within a certain amount of time when creating the gathered image [13]. We are able to check and confirm changes within a certain amount of time by viewing a gathered image. Since it spends a long time for sleep, we employ the method to create the gathered images and confirm the sleep conditions easily to check and confirm the sleep conditions. Because the gathered image is a technique for visualization, the gathered image has not been analyzed. This paper analyzes the gathered images and classify whether he/she feels the sound sleep based on analyzed results of the gathered images.

Proposed Method
This paper proposes a method to confirm the sleep conditions directly and evaluate the sound sleep based on gathered image generation technique. The proposed method consists of three phases; gathered images generation, gathered images analysis and sound sleep evaluation. In the gathered images generation, the gathered images are generated by conflating any images. The gathered images designed to gather sleep postures and their changes are generated at one-second, one-minute, 10-minute, one-hour interval and all times, respectively. In the gathered images analysis, the gathered images are analyzed by calculating difference values among the gathered images of 10-minute interval and all times. Then, the sleep conditions are evaluated based on the results of the gathered images analysis. When the changes of position are constant, steady and not moving more active during sleep, this paper regards sleep condition as sound sleep. Therefore, when the calculated complexity of the gathered images by calculating the difference values become small, the proposed method regards sleep condition as sound sleep.

Gathered Image generation
A gathered image is an image obtained by gathering information of any captured images in the movie that is recorded while sleeping into one image. The gathered images are generated by conflating any captured images within the specified time. This paper generates the gathered images of one-second, one-minute, 10minute, one-hour interval, and all times, respectively. The gathered image of one-second interval is generated using 30 images because the frame rate is 30 fps. The gathered image of one-minute interval is generated using 60 gathered images of one-second interval. As well as, the gathered images of 10-minute, one-hour interval and all times are generated using 10 gathered images of one-minute interval, 6 gathered images of 10-minute interval and all gathered images of one-hour interval, respectively.
Here, the gathered image is generated by embedding information. The procedure of the gathered image generation follows below steps.
[step1] A movie is converted to still images.
[step2] Their images are transformed into gray-scaled images as follows; where gray , r , g and b are the values of gray-scaled, red, green and blue on a pixel, respectively. Fig. 1 shows original RGB image and the transformed image.  [step3] The gray-scaled image is divided into blocks, each of which has N × M pixels. The average of the pixel values in the block is calculated on each block, respectively. Fig. 2 shows example blocks of the image. Then, difference values (DVs) of the average of each blocks between an image and the next image are computed, respectively, as follows; where Qi, x, y and n are attention block, x-coordinate, ycoordinate and an index of frame, respectively. The above calculation (equation 2) is carried out on all blocks of all image combinations. Information for embedding to an image is extracted by calculating the maximum values of DVs on each block in the all image combinations, respectively. The proposed method regards the pixel values of the image that has the maximum value of DVs as information for embedded to an image.
[step4] The information of all blocks is embedded to an image. The proposed method regards the image embedded information as the gathered image. Fig. 3 shows example of the gathered image.

Gathered Image Analysis
The gathered image of all times becomes the most complex image because much information while sleeping is included. The sleeping conditions and sleeping postures are not changed frequently. The interval of their changes is at several 10 minutes. In particular, the sleeping posture during REM sleep is unchanged. At first, this paper visually confirms the results of the generated gathered images of 10-minute and one-hour intervals are calculated to evaluate the sound sleep. The difference values (Di f Val) are calculated as follows: where P , x and y are a pixel value, x-coordinate and ycoordinate, respectively. all − time and 10-min mean the gathered image of all times and 10-minute interval, respectively.
Then, the proposed method judges whether the sleep condition is sound sleep or not based on the calculated difference values as follows: where Eva , Di f Jud and Di f Val are an evaluation value to judge whether sleep condition is sound sleep or not, a judgement value to judge whether a gathered image is complex and a mean value of the difference values among the gathered images of all times and 10-minute interval, respectively. The sleep condition is regarded as the sound sleep when Eva is equal to or greater than 0.

Sound Sleep Evaluation
The sound sleep is evaluated by checking the gathered images of 10-minute, onehour interval and all times visually, by judging and calculating the evaluation value (equation 4) based on the difference values among the gathered images on all times and 10-minute interval. Also, the sound sleep is evaluated by matching the subjective evaluation, which is questionnaire, and the judging results. In gathered images checking, the gathered images of all time, one-hour and 10-minute interval are arranged hierarchically. They are divided into a complex gathered image, which contains noise and overlaps any information, and a normal image. The number of the complex gathered images is counted. In sound sleep judging based on the difference values, the sound sleep is specified by calculating the evaluation value (equation 4) based on the mean value of difference values. In comparative verification, the proposed method matches the subjective evaluation results based on questionnaire to the results of sound sleep judging. If the matching result matches, we regard it as a successful specification of sound sleep.

Experiments
In order to show the effectiveness of the proposed method, we conducted experiments. The subjects were 2 healthy males and 3 healthy females (average: 22.3 years). Fig. 4 shows the experimental condition. They slept in the bed. The video camera was set 155 cm away from the bed. The brightness of the room was 1 to 3 lux during sleep. Then, he/she completed the easy questionnaire that included whether you feel sound sleep or not after waking up, which you feel that wake up condition is very good, good, bad, so bad and otherwise, and which you feel that falling sleep condition is very good, good, bad, so bad and otherwise, respectively. N and M pixels of block size for step 3 of gathered image generation were 4 and 3, respectively.
Figs. 5 to 9 show the gathered images of subjects A to E, respectively. The top image represents the gathered image of all times. The images of the second row from the top mean the gathered images on each hour. The images of third and subsequence stages is the gathered images on each time at 10-minute interval. Red square mean the complex gathered images clearly. The complex gathered images were specified manually. Fig. 10 shows example of counted image and uncounted image. The counted image means complex gathered image clearly. The uncounted image means simple gathered image. Then, the sleeping times on the subjects A to E were 23:40-6.18, 0:10-5:46, 23:50-6:35, 0:20-6:18 and 0:00-5:00, respectively. Fig. 11 shows the difference values among the gathered image of all times and the gathered images of 10-minute interval. Horizontal and vertical axises are clock time and difference values, respectively. (a) to (e) are the subjects A to E, respectively. Table 1 shows the results of experiments for sound sleep evaluation on each subject. SubA to SubE are the subjects A to E, respectively. Falling sleep, Awaking and Sound sleep are the questionnaire results of whether you feel sound sleep after waking up, which you feel that wake up condition is very good, good, bad, so bad and otherwise, and which you feel that falling sleep condition is very good, good, bad, so bad and otherwise, respectively. The appearance rate of complex gathered images are clearly calculated by dividing the total number of the gathered images to the number of the gathered images of red square. Pro. Method means the results judged whether the sleep condition is sound sleep or not based on the calculated difference values. Eval. Pro. Method means the results of matching the subjective evaluation results to the results of sound sleep judging.
In Figs. 5 to 9, we confirmed the gathered images of each subject. Then, the gathered images of all time were the complex gathered images clearly as though the images were blurred images and/or noise images. In the gathered images of one-hour interval, all gathered images with the exception of two images were the complex gathered images. In the gathered images of 10-minute interval, the numbers of the complex gathered images were small in the subjects A, B and D. In the gathered images of the subjects C and E, the number of the complex gathered images were large. Also,  the number of the complex gathered images were continuously small in the subjects A, B and D. In the gathered images of 10-minute interval of the subjects C and E, the number of the complex gathered images were continuously large.
In Fig. 10, we confirmed that the complexity was different between the complex and simple gathered images.
In Fig. 11, we confirm that the differences between the gathered images of all times and 10-minute interval are different on each subject. In the results of subjects A, B and   In Table 1, we confirm that the results of subjective evaluations using questionnaire and computer simulations are different on each subject. In questionnaire results of "Falling sleep", all subjects chose "Very good" or "Good". In answers of "Awaking", the subject D chose "Bad" and other subjects chose "Good". In answer of "Sound Sleep", the subjects A and B chose "Yes", the subjects C and D chose "No", and the subject E chose "Borderline". In computer simulations results, we confirm that the results of the subjects A, B, D and E were success. Also the result of the subject C was failure. In the results of the subjects A and B, their results were success because the answers of "Sound Sleep" and evaluation value (equation 4) were Yes and equal to or greater than 0, respectively. In the results of the subjects D and E, their results were success because the answers of "Sound Sleep" and evaluation value were No and less than 0, respectively. In the results of the subject C, her result was failure because the answers of "Sound Sleep" and evaluation value were not Yes and equal to or greater than 0, respectively.

Discussions
In Figs. 5 to 9, all gathered images of all time were the complex gathered images clearly as though the images were blurred images and/or noise images. These results seem that all subjects tossed and turned a few and/or a lot in their bed while sleeping. In the gathered images of one-hour interval, all gathered images with exception of images on the subjects A and D were the complex gathered images. In general, human changes the sleeping position to prevent physical burden caused by keeping their body still and bedsore. Also, the number of turns is about 20 times in a night. Therefore, the gathered images of one-hour interval were the complex gathered images. In the gathered images of 10-minute interval, the gathered images of the subjects A and B had a small number of the complex gathered images clearly as though the images were blurred images and/or noise images. These results suggest that it is possible to spend a pleasant night because the number of turning over in bed increased and the number of the complex gathered images (red square) were supposed to increase when they spent an unpleasant night. Then, the gathered images of the subjects C and E had a large number of the complex gathered images. These results suggest that it is possible to spend an unpleased night because the complex gathered images mean that the subject got physical activity clearly, and those numbers were large. Moreover, the gathered images of the subject D had a small number of the complex gathered images and her body and face hided in bed. Although the number of the complex gathered images were small, it is unknown that she spent a pleasant night because her body and face hided in bed and it is difficult to confirm her sleeping conditions visually. In Fig. 11, the difference values in the subjects A, B and E until two or three a.m. were smaller than the mean values. The difference value becomes high when the movements of the subjects are strong, and becomes low when the movements are weak because the gathered image of all times is the most disturbed and composed of the most information. These results suggest a possibility that any gathered images to two-to-three a.m. became the main factor making up the gathered image of all times, and that they moved strongly until two to three a.m. In the results of the subjects C and D, the variance was small. These results suggest that the subjects were extremely tossed and turned a lot or a few in his/her sleep during sleep. Because if the sleep conditions tossed and turned a lot in his/her sleep, lots of gathered images became complex gathered image and the differences between the gathered images of all times and 10-minute interval became small. Also if the sleep conditions tossed and turned a few in his/her sleep, lots of gathered images became not complex gathered image and the differences between the gathered images of all times and 10-minute interval became small. We confirmed that the results of the subjects C and D in Figs. 7 and 8 had lots of the complex gathered images and a few of the complex gathered images, respectively.
In Table 1, we could obtain the successful results of four out of five subjects to classify the sound sleep and otherwise. These results suggest that the sleep conditions are able to be evaluated by analyzing the gathered images of 10-minute interval compared with the gathered image of all times although the gathered images were the technique for visualization and were not analyzed. Also, these results mean that the gathered image generation and its evaluation methods are the effective for the sound sleep detection and evaluation. However, the result of the subject C was failure to classify the sleep conditions. These result suggests that the evaluation method using equation 4 may be incorrect when the sleep conditions tossed and turned a lot in his/her sleep, because lots of the gathered images of 10-minute interval become complex gathered images and the differences between the gathered images of all times and 10-minute interval became small.

Conclusion
This paper proposed the method to evaluate the sound sleep using images gathering technique and its analysis techniques. The proposed method consisted of three phases; gathered images generation, gathered images analysis and sound sleep evaluation. The gathered images were designed to gather sleep postures and their changes were generated at 10 minutes, 1 hour intervals and all times for analyzing the gathered images, respectively. In the gathered image analysis, the gathered images were analyzed by calculating difference values among the gathered images of 10-minute and all times. Then, the sound sleep conditions were evaluated by visually inspection and the analysis results. In order to show the effectiveness of the proposed method, we conducted experiments using real movies and their images. In experimental results, we confirmed that there was sound sleep conditions, bad sleep conditions and borderline cases, respectively, by checking subjective evaluation using questionnaire and generated gathered images visually. Moreover, we confirmed that the calculated difference values among the gathered images of 10-minute and all times were different between sound sleep conditions and other conditions. Furthermore, the analyzed results showed that it was successful in the sleep conditions classifications on four (subjects A, B, D and E) of five subjects. These results suggested that the gathered images analysis method is effective when evaluating whether sleep condition is sound sleep or not. In particular, it is important to calculate the difference values among the gathered image of 10-minute and all times. In the proposed method, it was possible to visually check and confirm the information of tossing, turning and body movement, and evaluate the sleep conditions quantitatively. However, the result of the subject C suggested that the proposed evaluation method could be incorrect when the sleep conditions tossed and turned a lot in his/her sleep.
Our future works will involve efforts to improve the evaluation technique for subjects whose sleep conditions tossed and turned a lot in his/her sleep. We will try to evaluate the sleep stages using the gathered images generation and its evaluation methods.