Sifting Method of Defect Candidate on Coated Automobile Roofs based on Binarization and Brightness Difference

Defects in automotive coated surfaces have a significant impact on consumers' purchase decisions. At present, most of the global automotive companies still rely on visual inspection to detect defects. With the development of industry4.0, in order to reduce the burden on inspectors, an inspection device is needed to help inspectors work more effectively. A defect detection system using a single camera, which filters the defect candidates using the tracking trajectories of the defect candidates on multiple frames has already proposed. However, this method has many noises for metallic color coated surfaces. This paper presents a new method to sift the defect candidates based on binarization and brightness difference. The experimental results demonstrate that this method can more effectively suppress the negative effects of sifting defect candidates. In the experiment, the F-measure are 100% for the coated surface.


Introduction
In recent years, with the development of industry4.0, competition in the automotive industry has become more and more fierce, and consumer requirements for automobiles have become more and more stringent. The appearance of an automobile has a significant impact on consumer buying decisions. The automotive industry attaches great importance to the quality of paintwork as the appearance of automobiles is mainly influenced by the paint finish. However, the detection of surface defects after car painting depends to a great extent on human vision and experience. Many automotive companies will arrange professional inspectors to inspect cars visually and repair unqualified automotive surface defects (more than 0.1 mm). The inspectors usually rely on the deformation of light projected on the surface when it encounters anomalies to determine whether it is defective. This is known in the field of optical and computer vision as the deflectometry principle (1) , and the phase measuring deflectometry (PMD) (1), (3) , a commonly used method to measure specular free-form surfaces, also utilizes similar lighting environments. However, the visual acuity and attention of the inspector will be affected by long-term work in this light environment. Therefore, it is necessary to have a device to assist inspectors in detecting defects in the coated surface.
The purpose of car painting is not only to have the aesthetic effect of surface coloring, but also to have anti-corrosion, anti-oxidation, and other functions. Therefore, the coating is generally multi-layer structure. This multi-layer structure gives the coating complex optical properties. For example, the incoming light reflects specularly on a smooth surface, diffuses on a rough surface, directionally on a smooth material, and diffuses and absorbs on the paint surface of the inner coating. On the other hand, the causes of the defects are also varied. Although modern paint shops have strict control of temperature, air, and humidity, many of them use static electricity for automatic spraying. However, it is still difficult to maintain a completely constant level of all the parameters in the application process, and the presence of dirt, dust particles, and fibers in the air, all of which cause defects in the coated surface.
Automatic defect detection of automotive coated surfaces is usually based on the simulation of visual detection steps. There are three main steps: firstly, it is necessary to create an environment that is easy to detect the defect. Then, collect the image. Finally, the defect is detected and identified by image processing. In the taken image, there are many interference phenomena, cause difficult to detect, on the coated surface. For example, during the drying process of the coating, uneven fine lines will be formed due to the drying speed and uniformity of the coating. These uneven fine lines cause the painted surface in the image to produce a dot that looks like a cloud or orange peel. This is known as the cloud or orange peel effect and is not considered as a defect. In this study, the ground truth of defect is verified by experts.
In order to eliminate the effect of these phenomena on the defect detection, Toyota motor east Japan is developing a low-cost defect detection system. This system uses binarization and differential image method to select defect candidates and uses the motion track of the defect on multiple frames to classify the defect and noise. However, when detecting a white color surface, a lot of noise is generated during the screening of defect candidates for this method. In this paper, the brightness difference of the defect and noise are analyzed. And a new sifting method of defect candidate based on binarization, and brightness difference is proposed, which can effectively reduce the negative impact of the defect candidate sifting.

Related Work
There have been a lot of studies on defect detection. However, the subjects of these studies are not limited to coated surfaces, such as steel (4), (5) , magnetic tile (6), (7) , road (8) , fruit (9) , textiles (10) , etc. The detection methods of these studies can be divided into traditional image processing technology and deep learning technology.
Traditional image processing techniques describe and detect defects by extracting features, such as: the gray symbiosis moment proposed by Haralick et al. (11) , which is a widely used method to describe textures using statistical features; the 14 texture features based on GLCM in literature (12), (13) , which provide more effective texture feature quantities; and the template matching method proposed by Jian et al. (14) ; Li et al. (15) propose a novel and robust fabric defect detection method based on the low-rank representation (LRR) technique; local binary patterns (LBP) (16) and a histogram of oriented gradient (HOG) (17) are also common features. However, due to the difference of illumination and background clutter, the above detection methods cannot be applied directly to the coated surface.
In deep learning, defect detection can be viewed as a problem of target detection. For example: Tao et al. (18) uses a compact convolution neural network (CNN) to accurately locate and classify metal defects; Li et al. (19) presents a surface defect detection method based on MobileNet-SSD; Azizah et al. (20) uses a CNN to detect surface defects of mangosteen fruit. However, the accuracy of deep learning is proportional to the number of data and cannot be completely detected. It is difficult to apply when you cannot get a lot of data in advance.
With the improvement of automobile painting technology, the defects after coating have been greatly reduced. Most of the defects appear as dots. Also, noises appear as dots, produced by random reflection. To improve the accuracy of distinguish between these, many practical methods have combined reducing the noise techniques with the binarization. For example: in a patent (21) , images taken by multiple cameras are synthesized as reference images and compared with the original image, then the result is obtained by binarization. A patent (22) uses only one camera and firstly binarizes the image to get defect candidates, then separates noise by using the time-axis information and moving distance of the defect candidates between two frames. Research (23) proposes defect motion trajectory detection with higher robustness to separate noise. Also, this research uses a sifting method, which combines binarization and using differential image to get defect candidates at first. However, the sifting method of defect candidates produces a lot of noise, which reduces the accuracy of defect motion track recognition.
Although image binarization has been studied for many years, image thresholding is still a problem, because there is no standard and it is difficult to apply one method to different types of images, such as uneven illumination, image contrast variation, etc. In coating surface detection, classical image segmentation methods, such as OTSU (24) binarization, are ineffective due to the non-uniformity of illumination. In order to solve the problem of binarization of images with non-uniformity illumination, there are two types of methods. One is to preprocess the image first and then use the global threshold. For example, Phong modeling is used to model the character of non-uniform illumination and then used to adjust the OSTU binarization threshold (25) . Research (26) proposed an improved method following recursive method. The global approach is usually simple and effective, however this approach is not suitable for our problem, because the preprocess will change the image information. Another method is the local threshold method. For example, this commonly used thresholding methods (Niblack (27) , Bernsen (28) , and Sauvola (29) ) and estimated local thresholds of images to produce better binarization results. However, these methods are easy to produce a lot of pseudo noise and are not suitable for defect detection. Research (30) presents an improved particle swarm optimization (IPSO) to solve the local binarization threshold and classify the defects of the steel billet. However, the IPSO method is not fast as binarization. In this paper, a new binarization method is presented, which is combined with the brightness difference method to detect the defect candidate.

Proposed Method
In this method, the detection methods are divided into three main parts. The detection of the region of interest (ROI) in the image, the defect candidates are sifted based on a single frame, and the verification of defect candidate based on the movement trajectory from multiple consecutive frames.

Detection of ROI
With the above shooting method, the reflected light image formed by the light source through the coated surface can be obtained. In the image, the intensity of the reflected light on the defective part is different from one on the flat part. As shown in Fig. 1 when the defect is in the bright field, its pixel value is lower than the surrounding pixels, and when the defect is in the dark field, its pixel value is higher than the surrounding pixels. Since this phenomenon mainly occurs in or near the LED area in the image, the first thing we need to do is to select the ROI.
ROI detection is mainly divided into three steps: First, the input color image is converted to a gray-scale image, and the noise is reduced by a median filter. This is to remove the effects of uneven lighting and color in the image.
Secondly, the gray-scale image is binarized. Given the global binarization threshold, the bright area is identified as the reflective area of the LED. The global binarization threshold is calculated by Eq.(1) , where ℎ is the threshold for binarization, and std are the average intensity and standard deviation of gray-scale images, respectively, and and are constants. ℎ = + + (1) Finally, based on the reflected area of the LED, its bounding rectangle is obtained and expanded to obtain the ROI in this study. The up and down expanding sizes are 1/12 of the image resolution, and the middle area is the middle value of the reflected area of the two adjacent LEDs.

Sifting the defect candidate
LED Camera

LED Camera
From Fig. 1, we can see that there are two cases of defect detection, one is to find the dark pixel group in the bright field, the other is to find the bright pixel group in the dark field. In the first case, the brightness value of the LED reflection (the reflective area of the LED tube) area approaches 255, and environmental changes have little effect on it. We just need to perform local binarization in the white areas and detect the black blocks that meet the size criteria. The second case is more complex and produces more noise than the first. For example: due to different colors of paint and uneven lighting problems, will cause different gray levels in areas outside the LED reflection area; due to the proximity of the LED tube area, the halo and distortion will be generated. In this study, the method shown in Fig. 2 is used.
First, the ROI of each LED reflection is divided horizontally.
Next, the best binarization thresholds are explored for each region after segmentation, and the results are used for binarization. The exploration of the optimal threshold for binarization uses a threshold-decreasing approach. The threshold decreases from the maximum until the number of white blocks in the binary image increases substantially.
Finally, the white blocks found in the binary image are validated. There are two main types of validation, existence area and brightness difference. To remove the effect of orange peel and clouding around the LED reflection, this study divides the detection area into 6 parts and determines whether the defect exists in 3 to 6 areas. The division of the area is shown in Fig. 3. Area 1 is the LED reflection, then expanding 10 pixels outward will result in areas 2 to 5, and the remaining areas will be area 6. Then, the brightness difference of all white blocks in the regions is calculated, and a difference of more than 10 is saved as the defect candidate. The calculation method of brightness difference in this paper refers to Eq.

Verification of defect candidate
Due to the multi-layer structure of the coating, it is difficult to detect the defects by the binarization method alone, because it cannot be determined precisely whether the change of reflected light is caused by in homogeneous or outer defects. At the same time, in the video data collected from the shooting environment, the identical defects will occur continuously in multiple video frames and produce noise randomly due to orange and halo. As a result, defect candidates are tracked and those that are continuously detected in multiple frames have higher confidence. Conversely, candidate defects that cannot be detected continuously are considered to be noise. In this study, the pre-prepared trajectory of the marker was used as a reference to verify whether the defects detected by different frames are the same. The specific validation method is shown in Fig. 5.
Then, the detected candidate defects are grouped according to the conditions. In Fig. 4, A, B, and C, three defects are detected. When A, B, and C appear in three different frames and frame of A is earlier one of B, frame of B is earlier than one of C, there are three main judgment conditions: the displacement spacing between the two defects matches Eq.(4); the straight inclination angle of the two defects is similar, the inclination angle can be obtained by Eq. (3); A, B, and C must have a defect with brightness difference greater than BD. This BD is determined by experiment as shown in Section 4. Settings can be obtained from marker data.
Here, is the interval between two frames, and are the maximum and minimum moving distances along the x-axis, similarly and .
Finally, we compare it with the marker track image. When the straight line conforms to Eq.(5), it is determined that the defect candidate is a defect that exists on the coated surface.
Where is the intersection of the line and the horizontal line, is the angle between the line and the horizontal line, and and are the intersection and angle between the nth marked line and the horizontal line.

Experiment
In this study, since no public video data was found for the defective paint surface, we used a camera with anti-flicker function to take pictures of 23 defective roofs to obtain experimental image data. Each group of experimental data is 30 frames and 16 seconds video, and the image size is 1600 × 1200 pixels. In particular, the actual width of the roof portion of the image is about 1000mm, and our method detects a range of 1 × 1 pixel to 40 × 40 pixels. Therefore, the defect detection size of this method is within [0.6mm, 25mm]. To evaluate the detection performance of this method, the 23 roofs come in nine different colors from two different SUV models. A ground truth is marked by experts around defects that appear in the sequence data based on visual inspection. When the gravity point of the detection defect overlaps those of the marker, it is assessed as a true positive detection. If not, it is a false positive detection. As shown in the Fig. 5.
Our experiment is mainly divided into three steps. Firstly, the difficulty of defect detection is evaluated. Secondly, the proposed binarization method is verified by experiments. Finally, the verification method is added to verify the final detection accuracy. As can be seen from Fig. 1, defect detection is based on the difference in brightness between the pixels of the defective part in the image and the surrounding pixels. Therefore, the brightness difference between the defect and the noise is investigated using Eq.(2). The following table is a typical example. Table 1 and 2 refers to the investigation results of white paint and metallic paint. According to the results, the pixel value of white coating in the image is higher, closer to the pixel value of defective part, smaller brightness difference and difficult to observe. Regarding noise, all colors have the most noise in the area 2 near the LED area. According to the above survey results, our experimental parameters are set as shown in Sect.3.2.

Verification of the proposed binarization method
For the validity of the proposed binarization method, we compare it with other binarization methods. The comparison methods are global binarization (Otsu, Triangle) and adaptive threshold (Mean, Gaussian). Local binarization (Niblack (27) , Bernsen (28) ) produces a lot of pseudo-noise, so it is not included in the comparison experiment. The results are shown in Table 3. In particular, the F-measure score of candidates is calculated by the harmonic mean of precision and recall. The recall is the number of defect candidates actually detected divided by the total number. The precision is the number of true candidate detections divided by the total number of times it was detected. Processing time is the average time consumed per frame. The best results are marked in red in the table. As shown in the results, compared with other binarization, this method is slow, but its detection accuracy is higher.

Verification of detection candidate
Finally, we add the multi-frame verification method to get the final detection results. The results are shown in Table 3. In particular, the F-measure score of defects is calculated similarly to the F-measure score of candidates, and the value used changes from the candidate correlation to the defect correlation. The experimental results show that this is an effective defect detection technique, and excellent results can still be obtained for the difficult-to-detect white coatings.   Triangle (Tri), Adaptive Threshold Mean (ATM), Adaptive Threshold Gaussian (ATG).

Conclusions
In this study, we propose a defect candidate detection method based on binarization and brightness difference, which effectively reduces the occurrence of false detection. The experimental results show that this method is especially effective in the detection of metal-colored coated surfaces, with 100% F-measure. However, the optimal parameters of this method depend on the brightness difference between the defects and noise in the photographic environment, which is difficult to detect when the brightness difference of the defects is low. In the correction of detection accuracy, the optimization of photographic mode is especially important. In addition, changes in photography may lead to changes in motion tracks. Therefore, other forms of motion tracks need to be reviewed to propose a more robust classification method.