A method for 3D environment model construction using the AR marker

This paper proposes a method for constructing a 3D environment model using the AR marker. In this method, we assume that the 3D models of the structures such as rooms or buildings and the objects such as desks or chairs are prepared in advance. A mark called AR maker is put on the each object. The position and orientation of the AR maker are measured from the captured image by the camera using ARToolkit. Then, the position and orientation of the objects are obtained and the 3D models of the objects are placed in the 3D models of the structures. Experimental results are shown and future issues are discussed.


Introduction
We are developing a simulation system of movement by wheelchairs for wheelchair users.In this simulation system, the 3D models of real world are inputted in this system in advance.Then, we move the wheelchairs in the virtual world.As a result, we can find the dangerous places for wheelchair users.To realize such a system, it is necessary to input 3D environment model of real world easily.
3D models of buildings might be easily constructed from such blueprint at the time of construction of the buildings.Furthermore, it is not necessary to modify the 3D models of the buildings itself unless renovations.
On the other hand, furniture and various other things such as desks and chairs have been placed on the inside of the building.Because we can move easily to them, it is not easy to build 3D models of the building including their placement.For example, each time the placements of chairs or desks are changed, it is necessary to modify the 3D models.
We can easily identify the objects such as chairs and desks, and can understand the placements of them based on visual information.In the same way, there are various attempts to acquire position and orientation of objects by image recognition and understanding technology (1) .However, it is not easy to accurately identify the objects which have various colors and shapes.Another way is to create 3D models from distance images acquired by a laser rangefinder or Kinect camera (2,3) .However, it requires a special device which can obtain a distance image.This paper proposes a method for easily constructing a 3D environment model which includes objects such as chairs and desks.In this method, AR markers are used to detect the objects.In this method, first, we put an AR marker on a predetermined position of the object.The position and orientation of the AR marker can be measured by taking an image by a camera.We can get the placement of the objects from the position and orientation of the AR marker, and can place the object into the 3D environment model.As a result, we can obtain the 3D environment model which include the objects such as chairs and desks.

Overview of the proposed method
The basic idea of the proposed method is as follows.As mentioned above, the purpose of the proposed method is to place 3D models of movable objects into the 3D environment model.3D model of the building floors and walls is assumed to be built in advance in a different way.
[Processing flow of the proposed method] (1) Creating a 3D model of the object such as chair or desk.
(2) Creating an AR marker, and pasting the AR marker on a predetermined position of the object.
(3) Taking a picture of the object with AR marker by camera, and measuring the position and orientation of the AR marker.(4) Calculating the position and orientation of the object from the position and orientation of the AR marker.
(5) Placing the 3D model of the object into the 3D environment model based on the position and orientation of the object.
In the above description, the AR marker is a marker used in the software library called ARToolkit (4,5) which is used to development the augmented reality applications.

Creating a 3D model of the object
Since the proposed method only determine the position and orientation of an object, it is necessary to prepare a 3D model of the object in advance to create a 3D environment model containing the object.The 3D model of an object can be created by 3D modeling tools.Figure 1 is an example of a 3D model of a rectangular box.The coordinate system of an object's 3D model is called the model coordinate system.

Creating an AR marker and pasting on an object
Next, an AR marker to be put on the object is created, and is pasted on a predetermined position of the object.Figure 2 shows an example of the AR marker.Figure 3 shows an example of an object which is pasted the AR marker.
When the AR marker was pasted on the object, it should be measuring the position and orientation of the AR marker in the model coordinate system.These are necessary to calculate transformation matrix which transform the model coordinate system to the coordinate system of the AR marker.The coordinate system of an AR marker is called the marker coordinate system.
For example, in case of Figure 3, the relationship of the marker coordinate system and the model coordinate system is shown in Figure 4.If the position of the origin of the 3D model is ሺ‫ݔ‬ , ‫ݕ‬ , ‫ݖ‬ ሻ in the marker coordinate system, the transformation matrix ‫ܯ‬ is shown in equation (1).By using the matrix ‫ܯ‬ , it is possible to transform the model coordinate system to the marker coordinate system as shown in Equation ( 2).In Equation ( 2), ሺ‫ݔ‬ ଵ , ‫ݕ‬ ଵ , ‫ݖ‬ ଵ ሻ is a coordinate in the model coordinate system, and ሺ‫ݔ‬ ଵ , ‫ݕ‬ ଵ , ‫ݖ‬ ଵ ሻ is a coordinate in the marker coordinate system.

Measurement of the position and orientation of the AR marker
As shown in Figure 3, an image is taken by a camera so that the entire AR marker is visible.Then, the position and rotation of the AR marker is measured by the ARToolkit.In the ARToolkit, the position and orientation of the AR marker is expressed in the camera coordinate system, and is expressed in the form of a transformation matrix.This transformation matrix transforms the marker coordinate system to the camera coordinate system.The camera coordinate system is the coordinate system in which the position of the camera is the origin of coordinate, as shown in Figure 4.
The transformation matrix ‫ܯ‬ ଷ obtained from ARToolkit is a 3×4 matrix which is omited the fourth row from the common 4×4 transform matrix, as shown in the equation (3).Therefore, we make a 4×4 transformation matrix ‫ܯ‬ ସ by adding a fourth row to ‫ܯ‬ ଷ , as shown in the equation ( 4).In the equation ( 3) and ( 4 Figure 4 shows the camera coordinate system ሺ‫ݔ‬ , ‫ݕ‬ , ‫ݖ‬ ሻ and the marker coordinate system ሺ‫ݔ‬ , ‫ݕ‬ , ‫ݖ‬ ሻ in the ARToolkit.As shown in Figure 4, in the camera coordinate system, the bottom of the camera is +y, and the shooting direction of the camera is +z.However, we often decide the +y direction as the upward direction.Therefore, we convert from the ARToolkit camera coordinate system shown in Figure 4 to the coordinate system shown in Figure 5.In this conversion, a point ሺ‫,ݔ‬ ‫,ݕ‬ ‫ݖ‬ሻ in the ARToolkit camera coordinate system is converted to a point ሺെ‫,ݔ‬ െ‫,ݕ‬ െ‫ݖ‬ሻ.This conversion can be done by rotate 180 degree around x-axis.In other words, this can be done by multiplying the matrix ‫ܯ‬ ௫ଵ଼ shown in equation (5).
After all, as shown in equation ( 6), the matrix obtained by multiplying the matrix ‫ܯ‬ ସ and ‫ܯ‬ ௫ଵ଼ is the transformation matrix ‫ܯ‬ which transforms the marker coordinate system to the camera coordinate system.By using this matrix ‫ܯ‬ , it is possible to transform the marker coordinate system to the camera coordinate system, as shown in equation ( 7).In equation ( 7), ሺ‫ݔ‬ ଵ , ‫ݕ‬ ଵ , ‫ݖ‬ ଵ ሻ is a coordinate in the marker coordinate system, and ሺ‫ݔ‬ ଵ , ‫ݕ‬ ଵ , ‫ݖ‬ ଵ ሻ is a coordinate in the camera coordinate system.

Placing the object into the 3D environment model
By the processing described above, we can obtain the position and orientation of the object in the camera coordinate system.However, we want the position and orientation of the object in the coordinate system of a room or a building.In this paper, it is assumed that the world coordinate system to the coordinate system of a room or a building.
For this purpose, it is sufficient a transformation matrix ‫ܯ‬ ௪ for converting the camera coordinate system to the world coordinate system.By using the transformation matrix ‫ܯ‬ ௪ , we can place the 3D model of the object into the world coordinate system as shown in equation (8).In equation (8), ሺ‫ݔ‬ ଵ , ‫ݕ‬ ଵ , ‫ݖ‬ ଵ ሻ is a coordinate of the 3D model of the object in the model coordinate system, and ሺ‫ݔ‬ ௪ଵ , ‫ݕ‬ ௪ଵ , ‫ݖ‬ ௪ଵ ሻ is a coordinate in the world coordinate system.The transformation matrix ‫ܯ‬ ௪ can be calculated from the position and orientation of the camera in the room or the building.

Experimental condition
We made an experiment by using a PC desk as a target object as shown in Figure 6. Figure 7 shows the 3D model of the PC desk.The width of the PC desk is 800mm, height is 650mm, and depth is 700mm.The AR marker was pasted on the front left side of the upper surface of the PC desk as shown in Figure 7.The size of the AR marker is 100×100mm.The center position of the AR marker is (65, 650, 635) in the model coordinate system shown in Figure 7.
This PC desk was placed in a room.The floor size of the room is 2770mm × 5440mm, height is 2500mm as shown in Figure 8.The position of the PC desk in the room was (735, 0, 30) in the coordinate system of the room shown in Figure 8.The position of the camera was (1150, 1080, 2000) in the coordinate system of the room.The shooting direction of the camera was set to -z direction.

Transformation matrix
As can be seen from Figure 6 and Figure 7, it is possible to match the marker coordinate system in the model coordinate system by the following coordinate transformations.( 1) Translating (-65, -650, -635), and (2) rotating 90 degree around x-axis in the model coordinate system.Therefore, the transformation matrix ‫ܯ‬ becomes Equation (9).The position and orientation of the AR marker on the PC desk was measured by taking an image by a camera as shown in Figure 6. Figure 9 shows the image taken by the camera.The transformation matrix ‫ܯ‬ derived from the position and orientation of the AR marker became as shown in equation (10).‫ܯ‬ ൌ 0.995 0.100 0.030 െ0.121 െ0.018 െ130.9 0.992 െ459.4 0.097 0 െ0.988 0 െ0.123 0 െ1105.9 1 ሺ10ሻ

Transformation matrix
As described above, the position of the camera in this experiment is (1150, 1080, 2000) in the coordinate system of the room, and the shooting direction of the camera is -z direction.Therefore, If we assume that the world coordinate system to the coordinate system of the room, the transformation matrix ‫ܯ‬ ௪ , which transform the camera coordinate to the world coordinate system, became as shown in Equation (11).

Conclusions
We have presented an easily construction method of a 3D environment model including the placement of moveable objects by using a general camera.An experiment using a PC desk as a target object was made, and we obtained the 3D environment model in which a 3D model of the PC desk was placed at roughly the correct position.
It is necessary to accurately measure the position and orientation of the camera in this method.However, we have not been described this point in this paper.It is a future issue.
This approach may not be in the new technology.It is not easy to paste AR marker on all of the target objects.However, bar codes have been attached to most of the products that are sold in retail stores today.We believe that attaching machine readable marker to all of the target objects is not unrealistic.

Figure 4 .Figure 5 .
Figure 4. Camera coordinate system and marker coordinate system in the ARToolkit

Figure 6 .
Figure 6.Target object(PC desk) and camera

Figure 9 .
Figure 9. Image taken by the camera 4.3 Measuring of the position and orientation of the AR marker on the PC desk By using the transformation matrices shown in Equation (9), (10) and (11), the 3D model of the PC desk can be placed in the 3D model of the room.The result is shown in Figure11.The position of the PC desk in Figure11was generally aligned with the actual position.

Figure 11 .
Figure 11.PC desk placed in the 3D model of the room