Proposal of a method to detect obstacle using projector and camera

In recent years, the introduction of business and service robots has progressed in Japan, and the number of unmanned transfer robots is increasing. Therefore, I will focus on obstacle detection, which is an indispensable function for automated guided vehicles. Conventional methods for detecting obstacles include ultrasonic sensors, PSD sensors, and 2D-LIDAR, but they have disadvantages such as a narrow measurement range, being easily affected by disturbances, and having a long mechanical life. Therefore, as a previous research, our laboratory has been developing an obstacle detection system using a line laser and a camera in order to improve these disadvantages. In this method, a line laser is used to irradiate the floor surface in front of the robot with a horizontal line, and obstacles are detected from changes in the horizontal line. This method improves the disadvantages of the conventional method and enables the detection of obstacles with a long life that is not easily affected by disturbances over a wide area. However, there are disadvantages such as being able to detect only obstacles on the straight line of the laser and being able to detect only part of the obstacles. Therefore, I propose an obstacle detection system using a projector and a camera for the purpose of improving the disadvantages of the conventional method and the disadvantages of previous research. This method is a system in which a projector irradiates the floor surface in front of the robot with multiple vertical and horizontal lines and detects obstacles from the characteristic changes in the lines. In this study, we conducted three experiments to verify the superiority of this study when compared with the previous studies. As a result of the experiment, it was confirmed that this study improved the disadvantages of the conventional method and the previous study.


Introduction
In recent years, the introduction of business and service robots has been progressing in Japan, and the number of autonomous mobile robots is increasing. Since automatic guided vehicle robots are highly expected in terms of unmanned work in factories, it is predicted that the number will continue to increase. Most of the unmanned transfer robots currently in use have a predetermined travel route, and the layout of factories and warehouses cannot be easily changed. In addition, humans and robots often work together in factories and warehouses, and irregular situations occur due to the movement of people, so even if it is restricted that luggage is not placed on the robot's travel path, it is an obstacle. Since the possibility that an object does not exist is not zero, there is a risk of collision with the robot, and the robot will stop, resulting in poor transportation efficiency. Therefore, it is necessary to accurately detect obstacles and avoid them appropriately.
In our laboratory, we have been developing a method using a line laser and a camera as a system that can be implemented at low cost and can detect a wide range of obstacles. In this research, we improved the detection process of the system and developed a new method using a projector and a camera. transmitting ultrasonic waves from the transmitter and receiving the ultrasonic waves reflected by the object at the receiver. Then, the distance from the sensor to the obstacle is calculated by calculating the relationship between the time required from the transmission to the reception of ultrasonic waves and the speed of sound. B) Obstacle detection method using PSD sensors The light emitted from the light source is focused by the lens and irradiates the obstacle. The reflected light from the obstacle is collected on the one-dimensional position light receiving element (PSD) by the light receiving lens. The imaging position on the PSD differs depending on the presence of obstacles and the position of the obstacle, and the two output balances of the PSD change. The distance between the sensor and the obstacle is calculated by calculating the change.
C) Obstacle detection method using image processing Prepare a background image in advance and compare the input image with the background image. Then, the presence or absence of obstacles is detected by extracting objects that do not exist in the background image.

D) Obstacle detection method using Lidar
This is a technology that irradiates a target with a laser beam that emits pulsed light, detects the scattered light that is reflected and returns, and calculates the distance from the reflection time. A wide range of scans is performed by transmitting and receiving laser pulsed light using a reflecting mirror and a condenser lens.

Disadvantages of the conventional method
The disadvantages of the conventional obstacle detection method shown in 2.1 are described below. (a) The measurement range is narrow and it is not possible to measure in a wide range. Since ultrasonic sensors have strong directivity and can measure only in a narrow range of about 15 °, it is necessary to increase the number of sensors or measure multiple times at different angles in order to measure in a wide range. The PSD sensor has a narrower measurement range than the ultrasonic sensor and can measure only within a range of about 5 °. In order to measure in a wide range, the same measures as ultrasonic sensors are required. (b) Susceptible to disturbance Ultrasonic sensors are prone to errors due to noise. If there is a sound source with the same frequency around the sensor, the sensor will receive it and erroneously detect the presence or absence of an obstacle. In addition, the speed of sound changes depending on the temperature, so temperature correction is required. With PSD sensors, errors are likely to occur due to reflected light from the floor. Strong ambient light may enter the light receiving part directly around the sensor, resulting in erroneous detection. (c) It takes time to capture the background image In obstacle detection by image processing, it is necessary to prepare a background image in advance, and it takes time and effort to prepare an image in a factory or office. Therefore, the travel path of the robot cannot be easily changed. (d) The distance to the obstacle cannot be measured In the detection of obstacles by image processing using background subtraction, it is difficult to measure the distance to the obstacles, although the presence and size of the obstacles can be known. (e) There is a drive range, so there is a mechanical life Obstacle detection using LiDAR requires scanning with a mirror to measure a wide range. Therefore, there is a mechanical life and it leads to failure.

Proposal of a method to detect obstacle using line laser and camera
In order to solve the disadvantages of the conventional method described in 2.2, our laboratory studied obstacle detection using a line laser and a camera. The system is described below. Fig. 2 shows an image of the obstacle detection system using a line laser and a camera as seen from the side and top. Fig. 3 shows the input image from the camera installed in the robot. As shown in Fig. 2, a green line laser is radiated from the top of the robot toward the floor, and when the laser hits an obstacle, the difference between the line laser radiated to the ground and the line laser radiated to the obstacle. Obstacles are detected from.

Advantages and disadvantages of obstacle detection method using line laser and camera
Obstacle detection using a line laser and a camera has improved the disadvantages of the conventional method, and has made it possible to detect obstacles with a long life that can detect distance, height, and width without being affected by disturbance over a wide area. However, there are the following two disadvantages of the method of the previous research.
A) Only obstacles on the straight line of the laser can be detected Obstacles that are not on the straight line of the line laser cannot be detected. In particular, if an obstacle appears inside the line laser when viewed from the camera, it may collide and work may be interrupted.

B) Only part of the obstacle can be detected.
Previous studies can only irradiate obstacles with one horizontal line. Therefore, only the shape of that part can be detected, and there is a risk that the width and height of the obstacle itself will be detected incorrectly.

Proposal of a method to detect obstacle using projector and camera
Fig. 5 shows the system image and the input image from the camera when there is an obstacle in front of the robot in this research. In this way, a projector is installed in the upper rear part of the camera, and four vertical lines and three horizontal lines are irradiated forward to acquire the state with the camera. Then, image processing is performed on the image extracted from the camera to detect each type of line. For the detected line, the coordinates of all the points of the line hitting the obstacle are calculated by the calculation methods of the vertical line and the horizontal line. In this study, obstacles are detected using still images taken by a camera.

Line extraction principle
In order to read the change of the line illuminated by the projector, it is necessary to extract only that line. Therefore, the pretreatment up to the line extraction irradiated by the projector will be explained below. (2) A) HSV conversion The image input from the camera is usually an RGB system expressed from the ratio of the three primary colors of red, green, and blue. By converting this expression method to the HSV system of hue, saturation, and value, it becomes possible to easily make sensory adjustments such as color brightness and darkness. B) Mask processing Mask processing is image processing that extracts only a specific part and hides the other parts. This time, a quadrangle or an image extracted with a specific color is used as a mask image, and by matching it with the original image, an image with only a specific part and color is generated.

C) Binarization
The process of generating a binarized image from a color image or grayscale image, which is an image of only two kinds of colors based on an appropriate threshold, is called a binarization process. The binarization process is used to extract the region of interest from an image and to remove unnecessary parts. By performing binarization processing, objects above a certain threshold can be recognized as one group.

D) Morphology transformation
Morphology transformation is an image process that is mainly performed on a binary image, and is a process that contracts or expands the original image from the input image and the structural element (kernel) that determines the nature of the process. This time, noise is removed by performing two types of processing, erosion and dilation. E) Progressive Probabilistic Hough Transform The Hough Transform is a general method that can detect a shape that can be expressed by a mathematical formula, and even if a part of the shape is damaged or deteriorated, it can be detected. The Progressive Probabilistic Hough Transform was optimized by using two thresholds, the minimum length of the straight line detected from this Hough transform and the maximum length allowed when two line segments are regarded as one straight line. It is a thing.
In this research, after acquiring the original image and performing HSV conversion, only the part of each color is separated by the threshold value to create a mask, and the line (each color) irradiated by performing AND processing (bit operation) with the original image. In addition, the lineextracted image obtained there is binarized using an appropriate threshold value, and fine line breaks are corrected and noise is removed by morphology conversion. Then, by applying a Progressive Probabilistic Hough Transform, the pixel position that is the characteristic of the obstacle is detected. Each line is discriminated by mask processing using a quadrangle and mask processing by color extraction.

Relationship between the number of pixels in the extracted image and the actual length
First, find the shooting range of the camera. (1) In addition, Fig. 7 is an extracted image image of a camera whose size is × . The actual distance ( * , * ) at the distance * from the camera of the distance ( + , + ) between two arbitrary points in this figure is calculated by the following formula by the ratio. [mm] (4)

Coordinate calculation principle
The principle of calculating the coordinates of all points irradiated with a vertical and a horizontal line from the image after line extraction is described. Fig. 8(a) shows an image with a vertical line extracted when there is an obstacle, and Fig. 8(b) shows an image with a horizontal line extracted. Using the relationship shown in 4.2 based on ( *+ , *+ ) obtained by the coordinates of any point R 1 on the extracted line, the actual distance ( * , * ) at the camera reference distance * in Fig. 9 is calculated. It is shown in (5)-(6). The coordinates of the point R 23 at the camera reference distance * are shown in (7)-(8).  [mm] (5) [mm] (6) 4* = * (7) y 23 = !56 + * (8) Fig. 11 (a) shows a plan view when only the vertical line is irradiated, and Fig. 11    [mm] 但し ≠ (16)

Coordinate calculation principle for vertical lines
A, b, c, d represent the slope of a straight line and the xintercept. In this study, it is necessary to shift the positions of the projector and camera on the plane. Fig. 12 (a) shows a side view when only the horizontal line is irradiated, and Fig. 12 (b) shows a plan view when only the horizontal line is irradiated. The formula for calculating the coordinates is shown below as in 4.4.  [mm] 但し ≠ (25)

Coordinate calculation principle for horizontal lines
A, b, c, d represent the slope of a line and the y-intercept. In addition, there is no case where a = c in this study.

Experiment
Regarding the system configuration used in this study, Fig. 13 shows the parameter settings of the position constants used when using the obstacle detection system this time, and Fig. 14 shows the constant parameter coordinates of the camera. Fig. 15 shows the image projected by the projector in this study   The results are shown in Table 1. In any case, it was confirmed that the measurement was possible in the double range when there were three horizontal lines. Therefore, it can be said that the detection range is wider than in the previous research by having multiple lines.

Obstacle detection accuracy verification when the distance to the obstacle is reduced
By shortening the distance to the obstacle, we verified the measurement accuracy of this system and the change in coordinates with respect to the distance.
The image of this experiment is shown in Fig. 16 Fig. 16(b).  The obtained image is shown in Fig. 17. Among the obtained results, the changes seen from the top and the sides of points A, B, and C are shown in Fig. 18-19. From the results, it was confirmed that the error of all coordinate values was within 50 [mm]. In this experiment, obstacles pass through all horizontal lines, so it can be said that obstacles can be detected at each horizontal line. As can be seen from the graph in Fig. 18, it was confirmed that the width of the obstacle, which cannot be detected only by the vertical line, can be detected by the horizontal line at all points. It was also confirmed that the height of the obstacle itself, which cannot be detected only by the horizontal line, can be detected at most points by the vertical line, as can be seen from the graph in Fig. 19. Therefore, by having multiple vertical lines and horizontal lines as in this experiment, it is possible to compensate for the parts that cannot be detected by each.

Obstacle detection with complex shapes
There are obstacles that cannot be sufficiently detected by obstacle detection using a line laser and a camera, which is a previous study. Therefore, in this study, we detect complex obstacles that cannot be detected in previous studies. The photographs and theoretical values of the obstacles detected this time are shown below.  Fig. 22 shows an x-y graph as seen from the camera, and Fig. 23 shows a z-x graph as seen from above. For the horizontal line, the maximum error was 11 [mm], which was an accurate result. In addition, a large error occurred in the z coordinate for the vertical line. This is because the camera and projector are tilted and the image projected by the projector. If there is only one line for obstacles with different widths at the top and bottom, the width of the obstacle will be mistakenly recognized as 100 [mm], and the robot will collide when avoiding the obstacle. There is a risk that it will end up. Therefore, it was confirmed that the risk of such a collision can be reduced because the detailed coordinates of the obstacle can be detected when there are multiple horizontal lines or vertical lines as in this system.

Conclusions
In this study, in order to allow the robot to avoid obstacles and run freely, we detected obstacles by using a projector and camera that are different from the conventional obstacle detection method. In this research, the points that could be improved compared to the obstacle detection method using a line laser and a camera, which was the previous research, are described below.
l In the previous study, only obstacles on the line laser can be detected, but in this study, the measurement range can be expanded by irradiating multiple horizontal lines, as can be seen from the experiment in 5.1. l In previous studies, only a part of the shape of the obstacle irradiated with the line laser can be detected. However, in this study, as can be seen from the experiment in 5.2, the height of the obstacle itself can be measured by having a vertical line. It also reduces the risk of erroneously detecting the width of obstacles with different widths at the top and bottom, as in the experiment in 5.3.
It can be said that practical obstacle detection is difficult because the current accuracy of this system causes a large error as in the experiment of 5.3. However, by securely fixing the projector and camera to eliminate tilt and adjusting the image projected by the projector, the reference value can be measured accurately, and by using a laser projector, a line with high luminous intensity can be displayed regardless of the focal length. If the system like the experiment in 5. 2 can be maintained at all times, practical obstacle detection will be possible.