Recognition of Road Mirror with Vehicle Camera

Driving a motor vehicle, if road mirror is hard to see, there is a risk of leading to an accident. To avoid such a risk, we attempt to realize a driving assistance system presenting an enlarged image of the road mirror to the driver automatically. This paper shows an algorithm to recognize a road mirror captured by car driving camera in real time. By utilizing the edge information and the color information of a road mirror, the road mirror can be recognized efficiently. In addition, we propose an algorithm for detecting moving objects on the image of the road mirror. Through the preliminary experiment, we demonstrate that practically the road mirror is enlarged and moving object (car) is presented to the user.


Introduction
Recently, the number of traffic accidents in Japan has declined but fatal traffic accidents of 5000 are still occur every year [1].Thus, many safe driving support systems have been researched and developed.
In Japan, secure road traffic societies are working on some safe driving support systems by road-vehicle cooperation including the intelligent transport system (ITS) with public-private partnership.For example, the national police agency promotes driving safety support system (DSSS) [2].The ministry of land, infrastructure, transport and tourism promotes advanced cruise-assist high-way system (AHS) [3] and advanced safety vehicle (ASV) [4].They assume a system that alerts the presence of the invisible things and forgettable things to the driver by using variable sensors and a communication technology.
In the AHS, support levels are classified into three stages as follows; level-i is reporting service, level-c is operation support service and level-a is automatic driving service.
The goal of our research is to develop real-time safe driving support system using the camera footage that is mounted on an automobile in support level-i.Specifically, in the operation of an automobile, if the road mirror is hard to see, there is a risk of leading to an accident more.In order to avoid such a risk, we attempt to realize a driving assistance system presenting an enlarged image of the road mirror to the driver automatically.
This paper shows an algorithm to recognize a road mirror captured by car driving camera in real time.By utilizing the edge information and the color information of a road mirror, the road mirror can be recognized efficiently.In addition, we propose an algorithm for detecting moving objects on the image of the road mirror.Through the preliminary experiment, we demonstrate that practically the road mirror is enlarged and moving object (car) is presented to the user.

System configuration
In order to offer exact information to a driver needing safety support, it is necessary to fill followings.

Target road mirror
There are various kinds of road mirror which is installed on the road.For example, Shapes are a round and a quadrangle.The colors are an orange and green.In this research, it is assumed that a target mirror on a curved road has the following features for a first implementation.
(1).It is set up in the intersection.
(2).Shape is a circle (3).It is set up by orange pole (4).The number of mirrors is one

Video image processing flow
Figure 1 shows the processing flow of the targets system.Through a car driving camera, this flow recognizes a road mirror in real time and detects a moving body in the mirror.In addition, the flow expands the captured and processed image, and provides the driver the expanded image to avoid a road accident caused by the invisibility of the road mirror.
The image processing flow consists of five major sections.The input part takes the image from the camera.The recognition processing part does the recognition processing of the road mirror.The expansion processing part does the expansion processing of the recognized road mirror.The object detection part does the object detection processing from the recognized road mirror.Finally the output part displays the expansion image and the detection result.

Cutting out around road mirror
If processing whole image captured, the computational time may increase significantly.In addition, the large image may lead to an inaccuracy for detecting the road mirror by various noises.Therefore, we first extract a region including the road mirror from the captured image and then process the extracted region to find the road mirror.
As a method of region extraction, we utilize the precondition mentioned above; the color of pole with a mirror is orange.The input image is converted from RGB value into the HSV value, and we estimate a region including the road mirror by extracting a region colored by orange.To extract an orange region, we assume the range of each HSV values to be H: 0-180, S: 0-255, and V: 0-255, the pixel of H: 0-30, S: 70-255, and V: 70-255.
In order to identify the location of the pole, first, the number of pixels for the orange is horizontally counted on the image every ten pixels in a raster scan fashion.Second, on the range with most large number of counts, the orange pixels are counted to every one pixel in the range of 5 horizontal pixels.Third, the 100 pixels on the right and left edges of the extracted region are cut out.Finally, the height of the cut image is made a half.Figure 2 shows an example of which the road mirror region is actually cut out from the captured image.

Recognition of road mirror
In the following, we recognize the road mirror from the image obtained by cutting around the road mirror, and cut the area closer to the road mirror.
First, as prerequisite, we noted that road mirror is circular, and the recognition of road mirror used circle detection by Hough transformation.
As a result of this recognition process, we obtain the parameters of circle's center and radius in the mirror section, and we cut the mirror section using these parameters.
In addition, as a pretreatment, we perform smoothing and edge detection for the purpose of improving the recognition rate.

Smoothing
We use a Gaussian filter for smoothing.When calculating the average of luminosity value around target pixel, more closer to the target pixel, to increase the weight when calculating the average value, and more farther to it, to decrease the weight, for that the Gaussian distribution function It is a Gaussian filter that calculate the rate of each pixel by using formula (1).The effect of smoothing becomes small, so that the value of σ is small, and the effect of smoothing becomes big, so that the value is small.

Edge detection
We performs edge detection on the image was smoothed.As a method of edge detection, we use the Canny method.Algorithm of this approach, carried out in the following four steps.

Circle detection by Hough transformation
As shown in Figure 3, all circles passing through the point (x, y) of the Cartesian coordinates are represented by a radius R and the center point (X center , Y center ) of the circle, it is represented by the following relationship.
If this is mapped also about the point (x0, y0) on other rectangular coordinates, all the coordinates on the same circle intersect at a point of three-dimensional space (R， X center ，Y center ).It is a parameter of the intersection circle to be extracted.We use the hough gradient method in actual circle detection.We consider the local gradient to all points corresponding to the edge from the image that is edge detection, and (1).All the points are recorded along the line that this inclination shows (2).All the points more than a certain threshold (minimum value of the radius) are made the candidate of a center point in the recorded point.(3).Sort the distance to the point of all applicable to the edge for each center points.The above process, we are recognized the Space to which (R，X center ，Y center ) is the most corresponding.

Object detection
Here, we detect the moving body in the image of the detected road mirror.As a detection technique of the moving body, we use the method of detecting the object that considers a time brightness change in the background image.
The procedure is shown below.

[1] Modeling of background
After the recognition image is acquired, an initial value of the amplitude σ of brightness I is requested from the mean value from the image for the average of the radiance value of the each pixel between 20 frames by the following expressions. 2

[2] Separation of background
The brightness amplitude σ of each pixel is used is calculated, it is compared whether radiance value I is ranges of the value, and preserves the result as a mask image.

[3] Processing of background area
A luminosity value is re-calculated like (1) and only a background region updates the value of a luminosity average and brightness amplitude using the mask picture of (2).

[4] Processing of object area
The mask image of ( 2) is reversed to update the brightness amplitude of the object area, and only the brightness amplitude is updated.To display only the object area that was able to be detected in the output image, pixel value is copied according to the mask image.

Still image
First, in order to ascertain whether the road mirror really recognizable, The image which used for an experiment photographed a road mirror installed in the vicinity of Kyushu Institute of Technology and used in the verification.The targeted image is 640×480 pixel, and 24 bit color image.
We show the results for the still image in Figure 4. We have to display the rectangle with a red frame in the mirror portion of the road mirror in the input image.The processing time is 200-300 ms.From the figure, it turns out that the mirror part can be recognized.
Moreover, Figure 5 shows the input image that was not able to be detected.As a cause that cannot be detected in these images, it is mentioned that this algorithm cannot correspond to the detection of the oval and paul's orange cannot be detected by the environment.

Real-time video image
Next, the same algorithm was applied to animation.The experiment applied this algorithm to all frames by using the dynamic scene of which it took a picture beforehand with the digital video camera on PC.Moreover, the moving body in the mirror confirmed to the image that detected the curve mirror by the expansion processing image and the moving body detection processing, and whether it was possible to detect it was verified.Figure 6 shows the appearance.

Summary
From the above experiment, we were able to show the algorithm that recognize the road mirror from color information and shape with the image and detect the object in the mirror.As development in the future, we would like to recognize of road mirror in the ellipse, simultaneous recognition of a road mirror installed two or more, and recognition of the road mirror while driving because it is video processing of conditions that the car is stopping.

Figure 2 .
Figure 2. Region extraction for road mirror