Recognition System of Foggy Windshield Side for Semi-automatic Run

This system detects fog on car’s windshield. This study aims to help semi-automatic run of the car. We put up “Maker A” on the outside of the windshield and set “Marker B“ by the windshield. The system gets an image of the windshield taken from inside of the car. Both the markers are in the image. In case of the inside of the windshield is foggy, both markers are seen hazily. In case of the outside of the windshield is foggy, only “Marker B” is seen hazily. Making use of this features, we recognize whether the windshield is foggy or not and which side of the windshield surface is foggy.


Introduction
Nowadays, approximately 1,100 million cars spread throughout the world (1) and the car cannot be separated from our life.However, traffic accidents such as drink-driving, inattentive driving and dozing driving occur.The number of recognized accidents come up in 600,000 cases a year in Japan (2) .The development of the safety support system of the car is pushed forward to reduce such traffic accidents by Cabinet Office.According to Cabinet Office (3) , the aim of the project is a safely automatic run that the system performs all of the car control; the acceleration, the steerage, and the control (level 4).The driver does not participate in at all.Its process is classified in three levels.In level 3, the system performs all the acceleration, the steerage, the control, other than emergency.In level 2, the system performs plural operations among the acceleration, the steerage, the control at the same time.In level 1, the system performs one of the acceleration, the steerage, the control.It remains in realization of level 1 now, and as for level 2, the realization is expected in 2017.The driver takes assistance to driving after level 2, and the system comes to perform the main driving.In this case, the driver does not always have to look at the front, but he must operate it in the emergency.However, he may not cope because he does not always look at the front when the windshield is foggy in emergency.Therefore it is necessary for view to always make the windshield a good state.However, as for the windshield, the outside of the car may be foggy or the inside of the car may be foggy.When the inside of car is foggy, the system operates air-conditioner and excepts fog.And when the outside of the car is foggy, it operates the windshield wiper and excepts fog.Therefore it is necessary to determine whether the inside of windshield is foggy, or the outside of the windshield is foggy.The system does not need to make this decision if it performs both operations at the same time, but uses useless energy.Today, there are multiple patents to except the fog of the windshield (4) , and also there is the thing which can distinguish the fog of both of the inside and the outside of the windshield.This system needs the in-vehicle camera and the temperature sensors at both sides of the car.
In this study, we developed the system which could distinguish the fog of the inside and the outside of the windshield with an in-vehicle camera.Because the apparatus to use is only a camera, the temperature sensors and those wirings are unnecessary.Moreover, the in-vehicle camera is cheap and can be installed easily.In addition, because the in-vehicle camera is established in the inside of the car, it is unlikely to come out the deficiency and the trouble caused by the wind and the rain.As one of the function of the semi-automatic run system, we suggest recognition system of foggy windshield side for semi-automatic run.

Structure of this System
We show the schematic diagram of the system in Fig. 1 and the view from the camera in Fig. 2. The marker A is stuck on the outside of the windshield, and the marker B is installed in the slightly remote position by the windshield.When we fog up the inside of the windshield, it becomes Fig. 3.And when we do the outside of the windshield, it becomes Fig. 4. When we fog up the inside of the windshield, both markers are seen hazily, and when we fog up the outside of the windshield, only marker B is seen hazily.From this difference how fog up, we distinguish it whish side of the windshield is foggy.Practically, we carry out gray scale imaging of the image, extraction of the gray scale level every pixel, and making of the histogram of the value we extracted, and distinguish based on the rough shape of the histogram.

About marker A
In this section, we explain the fog recognition method about the marker A. We show the image when glass is not foggy and its histogram in Fig. 5, and the image when the inside of the windshield is foggy and its histogram in Fig. 6, and the image when the outside of the windshield is foggy and its histogram in Fig. 7.
About the concrete distinction method of the marker A, we illustrate using Fig. 8.At first, we follow the rough shape of the histogram from the right-side end and find the first local maximum.At this time, when the value of the vartical axis (the number of the pixels) of the local maximum is more than 4,000, we assume it the temporary maximum, and when the value is less than 4,000, we search  the local maximum to appear next.If we find a temporary after that, we search a local minimum following the rough shape of the histogram.When the difference between this local minimum and the temporary maximum is more than 4,000, we consider the temporary maximum to be the true maximum, and when the difference is less than 4,000, we do not consider it to be the true maximum and follow the rough shape again.When we assume the maximum that we found in this way the top of a heap, we distinguish it whether the windshield is foggy or not watching the width of the heap (difference of the gray scale levels) at the value of 20% of the maximum.

About marker B
In this section, we explain the fog recognition method about the marker B. We show the image when the windshield is not foggy its histogram in Fig. 9, and the image when the inside of the windshield is foggy and its histogram in Fig. 10, and the image when the outside of the windshield is foggy and its histogram in Fig. 11.
About the fog distinction method of marker B, it makes a little difference except that we follow the histogram the left.The others are the same as the method of the marker A.

Experiments
We installed the markers as explained in Chapter 2 and tested it.And then we made histograms, changing the background and the condition.We show images and its histograms in Fig. 12 and Fig. 13 when the background is blue sky.And we show them in Fig. 14 and Fig. 15 when the background is sunset.And we show them in Fig. 16 and Fig. 17

Results
We show the averages of the width of the histograms which we required in bar graphs of Fig. 18 and Fig. 19.From the two figures, it led to that when the windshield is not foggy, the averages of both markers became low, and when the inside of the windshield is foggy, the averages of both markers became high, and when the outside of the windshield is foggy, the average of the marker A became low and the average of the marker B became high.
From these results, when we assume the threshold of the average 40, if the averages of both markers are less than 40, we can distinguish that the windshield is not foggy, and if they are more than 40, we can distinguish that the inside of the windshield is foggy, and if the average of the marker A is less than 40 and of the marker B is more than 40, we distinguish that the outside of the windshield is foggy.

Conclusions
In this study, we suggested a recognition system of foggy windshield side for semi-automatic run.As a results of experiments, we were able to detect it whether the windshield is foggy or not, and which side of the windshield is foggy when the windshield is foggy.
However, in this system, because the distance and the angle of the camera with both markers were not fixed and the size and the position were not constant, it is necessary to make the experimental device which fixed these.In addition, it is necessary to produce fog in various situations and to test it because there were little number of the experiments.
when the background is cloudy sky.

Fig. 12 .
Fig. 12.About marker A in the state of the blue sky.

Fig. 14 .
Fig. 14.About marker A in the state of the sunset.Fig. 16.About marker A in the state of the cloudy sky.
Outside windshield is foggy.

Fig. 17 .
Fig. 17.About marker B in the state of the cloudy sky.