Wearable Extraction System of Region Suited to Safe Ambulation for Low Vision

The objects have potentially dangerousness for pedestrians, such as illegally-parked bicycles, power poles, stopping vehicles, etc., could be found in a street. Those objects are threats to the safety of elderly and disabled people with impaired vision, because it is difficult for them to find those objects. In this paper, to enhance the elderly and disabled people's mobility, we propose the method to estimate the region which is feasible for people's mobility in life space. In this method, the images are obtained from a head-mounted wearable sensor device, and the suited region for walking is estimated using a weak assumption with respect to an installation requirement of the device and the geometric relationship between the device and the target region. The performance and the validity are shown by applying the proposed method to a number of scenes in life space.


Introduction
The objects have potentially dangerousness for pedestrians, such as illegally-parked bicycles, power poles, stopping vehicles, etc., could be found in a street as shown in Fig. 1 (a).Those objects are threats to the safety of elderly and disabled people with impaired vision, because it is difficult for them to find those objects and get around.``Low vision'' is a case of impaired vision, and people with ``low vision'' have low visual acuity even when they use the best possible spectacle or contact lens correction available.
There are individual differences in appearance of low vision depending on the symptoms.For example, Fig. 1 (b) shows appearance image of defocus, such as myopia (shortsightedness), hyperopia (farsightedness) or presbyopia.As seen in the example of Fig. 1 (b), bicycles and stairs could not be recognized for those people.Hence, those people have a high risk of sudden accident in life space because it is hard for them to recognize the potential dangerous objects.It is important to ensure environments for walking.If we indicate regions of walkable road for low vision people, they go out at ease and accidents are prevented before happen as well.
To aid the mobility of people who suffer from visually disabilities, a number of vision assistance system are proposed.For example, Fujiyoshi et al. [1] proposed a road observation and information providing system for pedestrian.In this system, the information of moving objects and road situation is detected using environment embedded terminal, such as monitoring cameras, placed in a street, and is provided for users through a mobile terminal.Therefore, user's activities could be cut down, because the place where users go to is needed to be the detectable area of environment embedded terminal to use this system.Bryant et al. [2] developed the wearable system for weak eyesight.The system detects obstacles adjacent to user with infrared radiation.And location information is presented to the fiber retinal scanning display.In this system, the analysis of visual information, such as image segmentation, is not performed, although an imminent collision hazard is detected and pointed.Aoki et al. [3] proposed a barrier detection system using wearable single camera.The researchers in [3] improve a conventional barrier detection algorithm on in-vehicle cameras for wearable camera.However, the detection of an artifact have poor texture could be unstable, because the method is based on feature point analysis.
In this paper, we propose a system to detect the region which is feasible for people's mobility in life space.In proposed system, a user fit the device consisting of a camera, a laser range scanner and a gyroscope.And the sensors sense surrounding condition of the user.The user recognizes the suited region for walking is estimated from the sensor-acquired information with head-mounted display.To stabilize the estimation, we employ a weak assumption with respect to an installation requirement of the device and the geometric relationship between the device and the target region.This approach aims to detect a poor texture object stably.

Proposed Method
In this section, we describe extraction algorithm of region suited to safe ambulation.Figure 2 shows the overview of proposed system.In proposed system, a user's visual information is obtained as camera image, and depth information of measuring line on the camera image is obtained by laser range sensor.Figure 3 shows sensing devices of our system.And the pose of the sensor device is measured by gyroscope.Therefore, if the brief distance between the device and plane of walkway could be calculated, the zone suited to safe ambulation is estimated by thresholding of depth of measuring line as shown Fig. 4.
The following flow is the procedure to extract the region suited to safe ambulation in proposed system.
1) obtain a camera image, depth information of measuring line on a camera image, and a pose information of the device 2) synchronize the sensor data 3) estimate the brief distance between the device and plane of walkway based on its geometric relationship 4) extract the zone suited to safe ambulation by thresholding of the estimated distance of measuring line 5) extend extracted region based on color similarity of pixels in the image 6) mark areas except for extracted regions, as a barrier in walkway  In proposed system, the sensors (i.e.camera, laser range sensor and gyroscope) are fixed together, and measure user's surrounding condition independently.Thus, the sensor data (i.e.images captured by camera, distance information obtained by laser range sensor and pose information obtained by gyroscope) is synchronized.Next, the distance of measuring line on the image between the device and plane of walkway is estimated from the pose information and its installation requirements, as mentioned later.The zones that is suited to safe ambulation are extracted by thresholding of the estimated distance of measuring line is extracted as shown in Fig. 4. The extracted zones on measuring line extend vertically based on color similarity of pixels in the image.Finally, the region with the exception of the extended region as a barrier on a walkway is marked to attract attention.
In the above procedure, it is important to estimate the distance between the device and plane of walkway in an orderly fashion.The brief distance between the device and plane of walkway can be calculated by the pose information of the device as follows.

‖𝑂𝑃 ⃗⃗⃗⃗⃗ ‖ = m + sin(𝑎) cos(𝑏 + 𝑐)
(1) ‖ ⃗⃗⃗⃗⃗ ‖ = m − sin() cos( − ) Here,m = ‖ ⃗⃗⃗⃗⃗ ‖ = ‖ ⃗⃗⃗⃗⃗ ‖, when b = 0. Let  be an inclined angle of the device, let  be a rotation angle of the device, and let  be half a view angle of a camera, as shown in Fig. 5. Let ℎ be a height of the mounting position of the sensor device. ⃗⃗⃗⃗⃗⃗ is an optical axis of a camera. ⃗⃗⃗⃗⃗ and  ⃗⃗⃗⃗⃗ are straight lines through right end and left end of measuring line in an image.The distance between a camera and a plane is obtained as the depth information on measuring line, i.e. the depth information on  ⃗⃗⃗⃗⃗ .

Experimental Results
To verify the validity and effectiveness, we apply the proposed method to a number of scenes in life space in this section.

Evaluation of Accuracy of Distance Estimation
In this section, we evaluate the accuracy of distance estimation with respect to a setting of installation of the device.Table 1 shows a comparison of measurement value and estimated value with respect to height, inclined angle and rotation angle.The error rate to an actual measurement value is at around 3% irrespective of installation height.Therefore, it is considered that a margin of thresholding process is more than 3 cm when the depth of measuring line (i.e. the brief distance between the device and plane of  walkway) is around 1 m.

Experiments for Real Scenes
In this section, the performances are shown by applying the proposed method to a number of scenes in life space.In the Experiments, the device consisting of a camera, a laser range scanner and a gyroscope is moved by user's hand in 100 centimeters high and estimate regions which is feasible for people's mobility.In result images, the regions are unmasked and a barrier in walking way is filled with ``blue''.Figure 6 shows a result image of the region extraction in an office.Despite the camera's movement less like that of a surveillance camera and more like the adorable shakes and nods of a person's head, obstacles in a walkway are detected.In Fig. 7, a nearby signboard is extracted but a back trash box is unmasked.This is because the trash box is out of range of detection.If user walks into the back, it will be detected.
Fig. 8 and Fig. 9 are result image of the region extraction out of doors.In Fig. 9, despite it is difficult to recognize a barrier object in image visually; a region of barrier is extracted by integration of image data and depth information by laser range sensor.

Conclusions
In this paper, we have proposed a system to detect the region which is feasible for people's mobility in life space.In proposed system, a user fit a head-mounted display and the device consisting of a camera, a laser range scanner and a gyroscope.The suited region for walking estimated from the sensor-acquired information and the user recognize the suited region with head-mounted display.In experiment, it is confirmed a suited region for walking are extracted and obstacles in a walkway are detected in situations that camera move less like that of a surveillance camera and more like the adorable shakes and nods of a person's head.
In future, we plan to extend proposed region extraction method to image segmentation, and tackle a development of presentation method of the visual information for visual impairment.

Fig. 1
An example of the differences in vision.(a) a scene image, (b) the simulated image of the vision of the people with low vision.

Fig. 4
Fig.4 Extraction of region suited to safe ambulation.

Fig. 5
Fig.5 Estimation of a brief distance between the device and a plane.

Table 1 .
The estimation of a brief distance between the device and a plane.