Using real and artificial images for testing vision based protective devices

In the paper the vision based system is presented. On the basis of images from two cameras the distance between man and moving parts of a machine is calculated and compared to the minimum acceptable. Experiments conducted using real and artificial images indicate that our system can work in a real time and detect successively dangerous situations. Results of performed experiments using real images show that the safety system works properly in different conditions typical for industrial environment, eg. for a wide range of illumination and rapid changes of illumination level. Tests based on artificial images of real assembly line show that application of vision based system can significantly decrease downtime.


Introduction (12pt,Bold)
In human-machine cooperation systems monitoring device using camera can be applicable to safety (1,2) .On the basis of images from camera the distance between man and machine is calculated and compared to the minimum acceptable.In the simplest case a machine is stopped by the system while people intrude the dangerous area of the machine monitored by video camera (eg.panoramic cameras (3) ).Such approach can simplify safety issues for a robot arm moving in close proximity to human beings, since they can simultaneously observe a 360° field of view (4) .In more sophisticated case on the basis of information about the environment situation the machines can accordingly take some corresponding actions automatically to avoid hazard (5) .
On one hand the algorithm used for image analysis has to be as less time demanding as possible.On the other hand very high accuracy in detection of location of worker and moving parts of machine is not required.Note that we have to stop the movement of machine, when the distance between the worker and the machine is relatively large (note that the process of stopping the robot's arm can take a much more than 100 ms (5) ).More than high accuracy is important the ability to cope with the interferences in images, like noise and rapid changes in illumination, which can appear on automatized work stand in the factory.Suitable for such applications are stereo systems (6,7,8) and stereo analysis based on artificial cellular neural network (CNN) (7) was used in our system.
Real and artificial images of were used to test stereovision based protective device.In order to collect real images for tests two Basler Scout scA780-54gc color cameras with 54fps were used.Those images were analyzed in real time.The stereovision based system was tested in many different conditions, eg. to check if the system will work properly during rapid changes of illumination level, in the presence of fast moving objects or if it is able to ensure safe cooperation of worker with an industrial robot.
To prepare artificial images a computer model of real assembly line was used.Such approach allows comparing effectiveness of different types of safety system (in this case stereovision based safety system and light curtain).

Methods
Real and artificial images of were used to test stereovision based protective device.In order to collect real images for tests two Basler Scout scA780-54gc color cameras with 54fps were used.Those images were analyzed in real time.The stereovision based system was tested in many different conditions, eg. to check if the system will work properly during rapid changes of illumination level, in the presence of fast moving objects or if it is able to ensure safe cooperation of worker with an industrial robot.
To prepare artificial images a computer model of real assembly line was used.Such approach allows comparing effectiveness of different types of safety system (in this case stereovision based safety system and light curtain).

Real Images
Because human arm can move very fast one of the most important features of safety system is detection of fast moving objects.During tests objects with different geometry (sphere and cube and cylinder) as well as with different textures were used (solid colors and the texture of the floor) were used.The size of all objects were comparable with the size of human palm and in the case of the cube the length of an edge equals 6 cm.
To move the object step motors and very thin rope (the rope was invisible for the cameras because its radius was much smaller than pixel's size) were used.Each object (attached to the rope) was floating approximately 1m above the ground.The step motor was working with constant angular velocity in order to move the object with linear velocity 2 meters per second.In each time step the position of the object calculated by the stereovision based system and the number of steps of the step motor were written down to a log file.Those data allow comparing distance covered by the object from the start position calculated by stereovision system and on the basis of the number of motor's steps (the step correspond approximately to 1.2 cm of displacement of the object)the results are presented in Fig. 1.Distances calculated by two different methods agree very well and this result show that the stereovision based safety systems can detect presence of fast moving objects and properly calculate its position.
In order to investigate influence of the rapid changes in illumination level two strong sources of flashing light were used (each 2700 W).Dashed line in Fig. 2 shows changes in illumination level observed by cameras defined as average intensity of all pixels (for <V> equals 0 or 255 all pixels were black or white, respectively).As results from Fig. 2 the average intensity of all pixels increases approximately four times during flashes.
During experiment the object did not move.However the position r of the object calculated by the systems has changed (solid line in Fig. 2, the position of the object is defined as a distance from the origin r 2 = x 2 + y 2 + z 2 ).Obtained results that the system works properly during flashes and the changes in calculated position are relatively small (during the 10 4 measurements changes in r were smaller than 1 cm).
In the case of safety systems the position and size of detection zone must be defined (to avoid some accidents caused by improper functioning of machine control systems (9) ).Results of analysis performed by safety system are often unpredictable if the object is near the border of detection zone.Thus, it is very important do determine experimentally length of tolerance zone.Inside the tolerance zone object detection probability is smaller than 1-10 -3 but greater than 10 -3 .To perform the experiment test object was moved near the border of the detection zone 1m above the floor.To move object step motor was used.For each motor's step the position of the object changed by 0.9 mm and for each such position ten thousands measurements were performed.Obtained results (Fig. 3) show that the length of toleration zone (approximately 2 cm) is relatively small in comparison to typical size of detection zone.One of the main motivation of using vision systems is to ensure safe work in industrial robot's proximity.It is crucial in the case of robots cooperating with human.To make test simple functional model of industrial robot was developed (10) .Stereovision based safety system observes area around the robot and if human presence is detected inside detection zone it sends signal to slow dawn or stop robot's arm movement.In other words if distance between human body and moving parts of machine is lower than acceptable minimum safety system has to change machine movement to avoid injuries.To distinguish moving parts of the machine from other objects a bicolor marker was used (7) (in the case of the robot it is placed near arm's axissee Fig. 4).Results of 10 3 performed test (in all cases robot's arm were stopped and collision was avoided) indicate that may be used at work stands equipped with a robot to detect dangerous situations and prevent accidents.

Artificial Images
In this sections an interesting example of application of our system is presented.This is an assembly line from a factory in Poland.The computer model of the assembly line is shown in Fig. 5.The assembly line consists of nine machines (production cells) and fast moving trolley.The line is operated by two workers.
If the production in one of the production cells is finished, the trolley is automatically called to take away the new product from the machine.The trolley transports the product and leaves it for further processing.After each production process the production cell has to be prepared by an operator for next one (the production cycle takes approximately few minutes).In order to do so the operator sometimes has to access to dangerous zone, where the trolley moves.In such cases the movement of the trolley is stopped, because the worker crosses detection zone of a light curtains.Because of length of the assembly line (approximately 30 meters), such stops can cause unnecessary downtimes, hence it can decrease the productivity.The movement of the trolley is stopped if it is far away from a worker (or the trolley is approaching to a production cell, which is between trolley and worker).Moreover, the trolley is stopped even if it is carrying away from a worker (the probability of collision equals zero).
In order to compare effectiveness of light curtain and vision based safety system we performed simulations in virtual environment (the visualization of assembly line is shown in Fig. 5).We assumed that the trolley moves with v = 2 m/s and in every 5 minutes an operator hast to enter to the area where the trolley moves.The simulation lasted in both cases 24 hours (3 shifts).In the case of light curtain the movement of trolley was stopped 290 times and the total downtime was 93 minutes.In the case of vision system the movement of trolley was stopped 90 times and the total downtime was 15 minutes.However if the relative velocity of trolley is taken into account (i.e. the trolley is not stopped if it is carrying away from the worker) the number of stops and the total downtime decrease approximately two times.

Conclusions
Vision Based Protective Devices (VBPD) supports flexibility in a workplace.This is useful because modern Fig. 2. Solid line shows the position of the object (the curve was vertically shifted), dashed line shows illumination level calculated as the average value of all pixels.
Fig. 3.The probability that calculated position of detected object is inside detection zone (near the border of detection zone).For each object's position 10 4 measurements were performed.production processes becomes more and more flexible.The flexibility of a vision system is especially useful in the case of cooperation between human and the robot, e.g. the operator can insert the component directly into the robot's gripper.Note that at the same time, the operator has free access and a clear view of the production process, because the fences and barriers can be removed.Experiments conducted with a robot show that developed safety system detects the human presence of and properly identifies the position of moving robot's arm.Therefore it can be used in the case of cooperation between human and machine.As results from the experiments, conducted using real images, the safety system is able to work efficiently in real time for a wide range of illumination, in the presence of flashing lights and is robust against shadows.Our system calculates the position of fast moving objects (v = 2 m/s) with high accuracy.The size of tolerance zone is very small and equals approximately 2 cm.The results of computer simulation using artificial images show that vision based protective devices can decrease downtimes in comparison to light curtains.

Fig. 1 .
Fig. 1.The distance covered by test object by vision system (d v ) in function of distance calculated obtained from the number of motor's steps (d s ).The results are averaged over 100 tests.

Fig. 4 .
Fig. 4. Detection of human presence inside detection zone causes movement of robot's arm to stop.