A Study on Extraction and Recognition of Speed Limit Signs to Assist Drivers of Vehicles

Road signs provide important information that guides and regulates the behavior of drivers and pedestrians to make their travel safer and more comfortable. However, safety might be compromised by disregarding the speed limit signs. The rate of accidental death due to speed limit violations is extremely high at night. We can address this issue by implementing road sign recognition using videos. This can contribute to the reduction of traffic accidents. In this study, we propose a novel extraction method that focuses on the outlines of signs and a recognition method that focuses on number symbols in signs in videos at night.


Introduction
Road signs provide important information to guide and regulate actions of pedestrians and drivers of vehicles alike to make their travel much safer and pleasant.However, safety will be impaired if we cannot recognize and read a road sign.Detection and recognition methods to read various traffic signs that can prevent traffic accidents have been developed.
The authors have proposed a method to detect and recognize ring road signs based on their color and accurate information obtained from images taken in the daytime (1)(2)(3) .As a result, one of the methods that effectively utilized light, hue, and saturation demonstrated that highly accurate circular road signs can also be detected under backlighting and sun shading conditions (3) .On the other hand, numerous studies on recognition of road signs have been conducted everywhere.For example, a method using a support vector machine (4) and a neural network (5) has been proposed.However, the acquired data information varied a lot from day to night.To prevent accidents while driving a vehicle, it is necessary to make use of robust processing against sunshine conditions and time zones.The problem is that the mortality rate due to maximum speed violation is very high.Furthermore, it is more dangerous during night than in the daytime.Fig. 1 shows the number of death accidents and mortality rate (6) in 2017.In order to deal with this problem, we can help reduce number of traffic accidents by using automatic video extraction and road sign recognition.We developed processes to recognize and read the speed limit sign at night (7,8) .It was proved that focusing on the numerical symbols on the signs was useful in recognizing the speed limit sign.However, these approaches are not practical, as they misrecognized speed limit signs and acknowledged the fact that the correct result could not be obtained.In this study, we propose a recognition method with high robustness of extracted images of speed limit signs for the purpose of extracting and recognizing the highest speed limit signs during practical applications.

Death Accidents
Accident rate of death Fig. 1.The number of death accidents and the mortality rate in 2017 (6) .

Data Used
A high-sensitivity camera manufactured by SONY (α 7 S) was installed in the passenger seat of the car.We drove within the legal speed limit and got a night view video from the vehicle.Experiments were conducted under appropriate weather conditions and data was collected at 6:30 pm and 10:00 pm on 2 September 2017 in Akita, Japan.
A total of about 3 h and 7 min of video (RGB color images, 640×480 pixels, 30 FPS) was captured.Various characteristics were noticed, like: (ⅰ) the signs do not exhibit marked discoloration or deformation (ⅱ) trees and buildings do not overlap (ⅲ) signs for lanes where we were traveling were recognized as targets.The vehicle turned after reading the signal in the front, but it did not acknowledge the sign on the other side of the intersection.
312 instances were used as the experimental dataset (i.e. 120 cases of 30 km / h speed limit signs, 103 cases of 40 km / h speed limit signs, and 89 cases of 50 km / h speed limit signs).Fig. 2 shows examples of the speed limit signs.

Proposed Method
The proposed method mainly comprises of two processes for extracting and recognizing speed limit signs.Specifically, we propose an extraction process focusing on the shape of the signs and a recognition process focusing on the numbers on the signs.Fig. 3 shows the flow of the proposed method.

Extracting the speed limit signs.
We paid attention to where the sign appeared in the video, set the target range to the upper 75% and rejected the remaining 25%.After that, we reduced every 2×2 pixel area to 1 pixel and reduced the pixel processing speed.In the visual scene recorded at night, the gamma value could be adjusted since the color information is minimum.
Since the outline of Japan's speed limit indicator is red, the 8 direction Laplacian operator (9) focusing on the red value of the RGB color system was applied.Furthermore, it was combined with the edge image by the Canny edge detector (9) by logical OR, and the outline was extracted as an edge.
Because the speed limit signs are circular, we have estimated the circular region in the edge image using the generalized Hough transform (9) .However, the circular area assumed here contains not only the speed limit sign but also several noises.Therefore, we focused on the features of the edge in the speed limit sign that appear in the video and removed the noise.To be specific, (ⅰ) the circle protrudes from the image (ⅱ) the circle passes through the lower third of the image and has a radius of 10 pixels or more (ⅲ) the circle passes through the upper third of the image and has a radius of 10 pixels or less (ⅳ) those with a voting count less than 60% of the circumference pixels (ⅴ) those with 70% or more white pixels in the circle were removed as noise.

Recognizing the speed limit signs.
First, adjust the gamma value for the extracted circular area and change the size using the bilinear interpolation method (9) .Thereafter, it is divided this image into a 5×5 and the threshold value of the binarization process (9) using the equivalence analysis method is obtained in the central region.Further, it synthesizes with the mask image and focuses inside the circular area.
In this study, we split vertically at the part where the black pixel concentration is the maximum, divide the numbers, and obtain the common and feature part.In other words, the number 0 in the area to the right of the speed limit sign is defined as the common part of the speed limit sign.Numbers 3, 4 and 5 located to the left of the symbol were defined as feature part of speed limit of 30 km / h, 40 km / h and 50 km / h, respectively.The signs in Japan are regulated by the Ministry of Land, Infrastructure, and Transport and the shape of the signs is consistent (10) all through the country.
Recognition of signs was done by matching templates.First, template matching (9) is performed for all extracted images while changing the image size of the common part using the bilinear interpolation method.Here, the size of the common part having the highest similarity is demarcated as a representative of the extracted image.An image having the highest degree of similarity among all the extracted images is classified as the speed limit sign and the coordinates and size of the common part at that time are acquired.Next, using the coordinates and size that are obtained, roughly define the region in which the feature part exists among speed limit signs.In this region, template matching is performed on feature quantities (3, 4, and 5).As a result, the feature number having the highest similarity is presented as the recognition result.

Comparison with Conventional Study
The proposed method improves the recognition interval for the purpose of improving robustness and the recognition rate of the conventional study (8) .In this chapter, we describe the conventional recognition method and improvement points of the proposed method over it.
First, binarization (9) processing is performed on the extracted image using the discriminant analysis method.Next, labeling processing using OpenCvBlobs (11) is applied to extract regions where the area of white pixels is the maximum.This area is split into the left 50% and the right 50% to obtain a common part and a feature part, which include a numeric character for each.Using the bilinear interpolation method, template matching is performed by matching the size of the template image to that of the extracted image in the common part.When the degree of similarity exceeds the threshold value, the template matching is similarly performed for the feature part, and similarity is calculated.When the similarity exceeds the threshold value, the result is output.
Specific changes in the proposed method when compared with the conventional method are listed below.
In the binarization (9) process, the conventional method focuses on an entire image, whereas the proposed method focuses on the central region of an image.This is to ignore the noise contained outside of speed limit signs.
In the proposed method, when dividing a sign into left and right, it divides at the part where the number of black pixels is the maximum.This is because the feature part and the common part can be properly divided even when an extracted sign is inclined or misaligned.
In the conventional method, template images were created.In the proposed method, figures extracted from speed limit signs that actually appear in the video are used as the template.This is because it considered practicality.Fig. 4 shows the template images and threshold values used.
In the conventional method, template matching is performed for an entire image, whereas in the proposed method, the matching is performed focusing on characters.This is because even if the extracted sign contains noise, it can be ignored and recognized.

Extracting and Recognizing Results for Dataset
In order to verify the usefulness of the proposed method, experiments were conducted on the acquired videos.Figs.5-8 show examples of the extraction and recognition results of the speed limit signs by the proposed method.Table 1 shows the experimental result of extraction and recognition of speed limit signs by the proposed method.The proposed method achieved 98.4% extraction success rate and 97.1% recognition success rate.In addition, the recognition success rate for the number of the successful extraction is calculated to be at 98.7%.
On the other hand, experiments were conducted using the conventional recognition method.The obtained result is shown in Table 2. Compared with this results, it is revealed that the recognition rate of the proposed method improved by 14.7% in total.Furthermore, when a speed limit sign was erroneously recognized in one frame or more (e.g. a case a speed limit sign of 30km / h was recognized as 40km / h.), this was counted.As a result, in the conventional method, the total of the misrecognition was 192 signs.On the other hand, there was only one misrecognition in the proposed method.Compared with the conventional method, the proposed method was proved to accurately recognize speed limit signs.
In order to investigate the practicality of the proposed method, the processing time of the peripheral frame of the example shown in Fig. 5 to 8 was calculated.As a result, the processing time was calculated to be about 0.18 s, 0.09 s, 0.23 s, and 0.09 s in the peripheral frame of Figs.5-8.

Conclusions
In this study, we proposed a novel method for extracting speed limit signs using the Hough Transform and a recognition method using template matching.Experimental results show that the proposed method is useful for recognizing the speed limit on signs.The proposed method also showed high sensitivity to changes in speed limit sign state (color information, size, etc.) at night.With respect to the processing time, this result was faster than the recognition method of the speed limit sign in the night videos data developed prior to this study (7,8) .In the proposed method, multiple frames can be recognized from when the speed limit sign appears in the image until it passes through.However, in order to improve the accuracy, it is desirable to process every frame of the acquired videos.
For future work, we will work to shorten the processing time for practical application.In addition to this, we plan to develop a tracking process that analyzes recognition results of speed limit indicators by preceding and succeeding frames.

Fig. 7 .
Fig. 7.An example of the result under the bright environment.
Fig. 6.An example of the result for the roadside type.
Fig. 8.An example of the result under the dark environment.

Table 1 .
Result of extraction and recognition with proposed method

Table 2 .
Result of recognition with conventional method