Construction of Multi Viewpoints Database by GA for AR

Recently, as a useful application, the Augmented Reality (AR) has been widely adopted in mobile devices. In order to make the AR working smoothly on a low calculation ability smartphone, the high robustness and low calculation cost method is required. To realize such requirements, the Representative Image (RI) has been proposed. But, it does not guarantee the robustness in every case. In this study, an improved RI method using genetic algorithm is proposed. The numerous simulation result shows the proposed method is more robustness than the traditional one.


Introduction
Augmented Reality (AR) is an information adding technology, which can obtain the real environment information virtually to user by watching a two-dimension marker pattern through a camera. People can get various information that associated with the real content through a audio/visual presentation system. For example, the pictures, videos and music, those associated with the AR respectively.
Recently, AR technique has been adopted in many fields and most of the application software was implemented into the mobile device like a smart phone. In particular, it is a very attractive item in case of campaigns or event announcements. The specification of the traditional AR applications use monochrome pattern image as an AR marker. However, with advances in AR technology, a new AR system that allows using the natural color image as AR marker appeared. Such a "color nature image" can be a person or a landscape image, and we call the selected image a color AR marker. Color AR marker is a much more complex depiction object than the monochrome AR markers, because it contains various colors and shapes. Because the detection of color AR marker is a sophisticated processing, the processing load of smart phone must be considered. Operating speed of smart phone may slow down when the color AR system performs. This leads stress to the user during operation. Therefore, a low processing load and high accuracy algorithm is required.
RI method (1) has been proposed to reduce the processing load of AR applications that employ color AR marker. It gives a good the robustness when detecting the shooting distance and angle of the camera towards to the AR marker in a limited condition. If the limitations do not be abided, the detection performance drops significantly.
In this study, Using Genetic Algorithm (GA) (2) to select the optimal feature point of a color AR marker, an improved RI method is proposed to increase the robustness of the traditional one under a loose constraint.

RI method
In order to detect an AR marker in a real environment scene, RI method detects the feature points and calculates its feature values.
The feature point is a selected pixel point. Its brightness is higher or lower than the surrounding pixel. The feature value is a collection of brightness. The collection is pixel values surrounding feature point. In order to detect the point and extract the value, RI method utilizes the Oriented Fast and Rotated BRIEF (ORB) (3) .
The ORB is an algorithm, which obtains both feature point and feature value. Its feature point detector is improved by using FAST (4) . Furthermore, its descriptor is improved by using BRIEF (5) . Its advantage is that the calculating speed is faster than other algorithm. Therefore, it is suitable for processing on a smart phone.
In order to guarantee the robustness, RI method obtains 80 features information (feature point and its value) from an AR marker. These are extracted information from 80 perspectives of AR marker. Figure 1 shows the 16 perspective points. In order to create 16 representative images, a reference image is transformed through parameter P. The parameter P follows the formula 1.
The e z is unit vector along the z axis, and the e p is unit vector along the perspective direction. The rod (…) is Rodriguez conversion. The A is internal parameters of a camera. Fig 2 shows 5 images of a reference image. There are projected images with different distance of shooting (200mm, 400 mm, 600 mm, 800 mm, and 1000 mm). As a result, 80 images are created by transform the 5 images along the P. And, RI method can obtain 80 information points and features from these images. Then record the information to a database.
RI method allows numerous images as an AR marker. But, some images give a large error during the AR marker detection process. Because the traditional IR method simply detects the feature point using likelihood score. Therefore, the distribution of selected feature points occasionally gathered in a small local area. Practically, the feature points gathered in a small area cause a high marker detection error than the case that the feature points are selected in a wide range within the marker image.

Automatic selection of feature points
A GA improved AR (6) has been proposed in this research. Using this method, feature points are auto selected in a wide area with a good spread, furthermore, it can reduce the number of feature points. Although feature points are usually required large number for AR, it allows smaller number. GA is an algorithm that mimics the process of evolution of organisms. It is possible to obtain a good approximation solution for a calculation or a combinatorial optimization problem. In order to solve a problem, GA generates many individuals that have genes. In fact, these genes are candidate of approximate solution. GA can obtain an appropriate approximate solution by iterative updating the genes.
For example, to the target image Fig. 6b, the traditional IR method contributes feature points shown in Fig. 3. However, the result using a GA improved IR algorithm is shown Fig. 4, in which the feature points are widely spread, it is more suitable than Fig. 3.
Authors conclude that feature points should be distributed to wide range. Therefore, to keep the robustness with several feature points, it is necessary that feature points be widely distributed. In this research, GA algorithm is thought to be an effective way to select a optimal feature point-set.

Overview
Proposed method incorporates the GA to the RI method in order to create a more optimal database than the traditional one.  (a) approach.

Database construction
It is comprised of six processes to construct an optimal database.

Creation of multi viewpoint images
This process creates multi viewpoint images of a target image in the same way of the RI method. By shooting 16 view angles to 5 different distance, 80 images are created in this study.

Feature points selection in each viewpoint
This process detects the feature points. Premising that N points should be detected for each image finally, 5N feature points are detected from each image first. Then, N points are selected randomly within the 5N points as a gene.

Feature values obtaining
This process extracts the feature values from each image using the feature points selected in previous step.

Features registration to database
This process registers the feature points and its values as database. Because of registering the 80 image features, there are 79 sections in one database record.
For the processing of GA which will be described later, 100 candidate database records are created by repeating Section from 3.2.2 to 3.2.4..

Database operation
First, this process calculates fitness of the database. In order to obtain the fitness, 550 test patterns, which based on the reference image, are created. The test patterns are created by referring that shooting distances are 200mm, 400 mm, 600 mm, 800 mm and 1000 mm, rotation is between 0 and 70 degree (Increased per 7 degrees), angle is between 0 and 359 degree (Increased per 36 degrees). The fitness is defined as the successful number of marker detection. The successful is defined that re-projection error is within 20 pixels.
Second, 100 database records are sorted with respect to the fitness. After that, top 20 databases records are kept and discarded otherwise. Further, two databases records are selected randomly within the 20 databases and performed genetic operation. This genetic operation has three kinds of routines which are Mutation, Crossover, and Duplicate. Mutation randomly renews one of the 79 sections included in a database record. This section is created along the Section from 3.2.3 to 3.2.5. Crossover means mutually exchange half sections between two different database records. And Duplicate clones a new one.
Third, the 80 discard databases are newly created along the Section from 3.2.3 to 3.2.5.
Finally, before performing the subsection 3.2.6, this cycle of GA (subsection from 3.2.3 to 3.2.5.) operation continues 100 times.

Database adoption
After updating the databases by subsection from 3.2.1 to 3.2.6, the database that obtains the highest fitness is constructed. This is adopted as the database of reference (adapted from bibliography (6)

Performance evaluation
Evaluation experiment was conducted to confirm the robustness of the proposed method. This experiment compares the successful number of test marker detection. The successful is defined that re-projection error is within 20 pixels. Test marker is created by referring that shooting distances are 200mm, 400 mm, 600 mm, 800 mm and 1000 mm, rotation is between 0 and 70 degree (Increased per 1 degrees), angle is between 0 and 359 degree (Increased per 1 degrees), therefore test image has 128700 patterns. This experiment uses five types of images (Fig. 6). Table 1 shows the experimental results of the RI method. Table 2 shows the experimental results of the proposed method. Successful number of table 2 is larger than all of table 1.

Conclusions
In this study, in order to increase the robustness of the detection of the actual image RA markers, improved RI method using GA algorithm was proposed.
It was confirmed that proposed method is more robustness than traditional method from the result of evaluation experiment.
Although the feature point decision procedure is complicated, and takes times to construct the database, the AR marker detection procedure is as same as the traditional one. Therefore the user side calculation time is not changed if the number of feature points is same.
Moreover, due to the feature points were selected optimally, it is possible to reduce the feature point. This contribute a short calculation time to user side, and make the actual image AR marker detection application more possible to be implemented on a mobile phone.