An Efficient Method of Parallax Adjustment for Stereo Camera Systems Based on Image Processing

In recent years, stereoscopic technology has developed rapidly and it has gained a significant importance in the entertainment industry. However, to produce a visually pleasing 3D content with high quality, parallax adjustment which requires mastering the camera parameters (interaxial distance and convergence), lens parameters and the content composition, is still a challenging work in 3D shooting. In this paper, we propose a parallax adjustment method for stereo camera systems based on image processing. With the specified parallax parameters of the scene, the optimal position of interaxial distance and convergence can be provided by feedback calculation. Also, we have implemented the method on an active stereo camera system. With this system, the interaxial distance and convergence can be controlled automatically to achieve the 3D effect, which users’ want to obtain. The experimental evaluation of the proposed system in the natural environment reveals that the specified 3D effect can be obtained easier, faster and more accurate than the traditional manual method.


Introduction
In recent years, stereoscopic 3D technology has grown rapidly and it has employed in many different fields.After the great success of the 3D movie "Avatar", there has been a significant increase on the demand of 3D contents in all over the world (1) .
Unlike the 2D shooting, use of two identical cameras is the common method to produce a 3D content in the stereo camera system.In the 3D shooting process, all parameters (such as frame rate, focal length, shutter time, etc.) of the two cameras should be set exactly the same.Furthermore, the position of the two cameras also needs to be controlled with high accuracy (2) .
The camera control can be separated into two processes.First process is the optical axis alignment.It means the optical axis of the two cameras should always keep in a horizontal plane.Second is parallax adjustment which means the interaxial distance and convergence should be set properly to create a pleasant 3D effect.The common way in stereo camera system for camera control is still using manual techniques.We had developed an auto-calibration algorithm to make the optical axis alignment can be adjusted automatically in our previous research (3) .However, to make high quality 3D contents, parallax adjustment is still a challenging in 3D production.
Proper parallax adjustment can provide a pleasant 3D experience, but improper adjustment can cause wrong depth perception and adverse effects such as eye strain, fatigue, even motion sickness (4) .Parallax adjustment requires an operator's high level technical skill and time consuming process because of the complicated interplay between camera parameters, display properties, human perception and content composition (5) .The operator needs to set the interaxial distance and convergence manually and correctly while considering the composition, movements of cameras and objects, lens parameters and current parallax distribution to create the intended 3D effect that prepared in DOI: 10.12792/icisip2015.042preproduction.Therefore, it can be mastered by only a few experts who called stereographers.Furthermore, the process becomes more challenging and complicate when all parameters are dynamically changing in each scene, especially in the 3D live broadcasting.Most stereographers adjust the interaxial distance and convergence rely on their years of combat experiences, which is often included individual differences and is difficult for the less experience stereographers, it extremely affects the quality of the 3D contents (5) .Therefore, a faster, easier and accurate parallax adjustment method is required in 3D producing process (2) .Some researchers have proposed their methods to make the process become easier, but they need some parameters such as objects' distances from the camera.The measurement of those parameters is hard to make and time consuming that cause it is difficult to use in practical situation (6)(7) .
In this paper, we propose an efficient parallax adjustment method for stereo camera systems based on image processing.Users need to assign the target parallax bound and the target parallax of the awareness object if any.The interaxial distance and the convergence can be set properly and automatically by the feedback calculation to achieve the intended 3D effect.Furthermore, we evaluate the method on a stereo camera system, and the results shown that the proposed method is more effective in parallax adjustment.

Stereoscopic Principle
Shooting stereoscopic images is an attempt to replicate what we see with our two eyes.In 3D shooting process, two cameras are used to capture separate images of an object from different directions at the same time.The distance between the left and right camera is called interaxial distance and the intersection of the optical axis is called convergence point.Also, the distance between convergence point and stereo camera is called convergence distance, and the angle between optical axis is called convergence angle.
When overlap the left and right image, the horizontal separation of an object is called parallax of the object.The parallax of an object can be calculated as Equation (1).X L and X R are the abscissa values in pixels of the object in the left and right image.S is the width pixel number of the image.Thus, the unit of parallax is a percentage.Parallax = (X R -X L ) / S * 100% (1) When we watching 3D images through 3D display, objects with zero parallax appear on the screen plane, objects with negative parallax jump out from the screen, and objects with positive parallax appear behind the screen.

Geometric Relations of Parallax
Parallax is the most important factor for stereoscopic depth perception.The relationship between Parallax and perceptual distance y e of an object can be calculated as Equation (2).Where i means the pupil distance of the audience, d means the distance from screen to the audience, and w is the width of the screen.From Equation ( 2 In 3D shooting, Parallax is related to the interaxial distance, convergence distance, focal length of the lens, camera film size and the distance between objects and the stereo camera.The relationship is complicated, but we can approximate it as in Equation ( 3).Where b is the interaxial distance, f is the focal length of two lenses, e is the size of camera film, g is the convergence distance, and y c is the distance from the object to stereo camera.
Therefore, we can conclude that Parallax is a linear function of interaxial distance b.This means that if we change the interaxial distance b and keep other parameters fixed, the Parallax of all objects in the image will change linearly, which means the 3D volume for perception will change as well (Fig. 1(a)).Also, if we change the convergence distance g and keep the other parameters fixed, the Parallax of all objects in the image will change in the same direction with the same amount, which means the objects' distance for perception will change at the same time (Fig. 1(b)).

Parallax Adjustment Method
According to the theories about geometric relations of Parallax, we proposed a parallax adjustment method based on image processing, which can set the interaxial distance and convergence properly by feedback calculation of the parallax of the current frame.
It is difficult to measure and control the convergence distance accurately using a stereo camera system.Usually, we choose to control the convergence angle instead.For interaxial distance b and the convergence distance g, the convergence angle c can be described as Equation (4).
A depth script including the desired 3D effect for each scene will be designed in preproduction.Most of the time, it contains the parallax bounds value (which related to the 3D volume for perception) of the scene.And if there is an awareness object on focus in this scene, the target parallax for it (which related to the object distance for perception) will also be assigned (5) .So, we propose our methods according to if there is an awareness object or not.

Parallax Range Adjustment Method
For a scene there is no special awareness objects, such as a landscape scene, make parallax range of the image coincide with the desired parallax bounds can provide a better 3D experience.As shown in Fig. 2, we define the upper limit of the specified parallax range as p_upper, lower limit as p_lower.Moreover, we defined the maximum parallax value on the image as p_max, minimum parallax value as p_min.With our proposed parallax range adjustment method, interaxial distance and convergence will be controlled to make the p_max coincide with p_upper while p_min coincide with p_lower.
To implement the parallax range control, the control block diagram of interaxial distance and convergence angle is shown in Fig. 3. p_upper and p_lower as the input parameters, b(s) and c(s) are the output to control interaxial distance and convergence.G 1 (s) is the transfer function of interaxial distance, G 2 (s) is the transfer function of convergence, and H is the function about the relationship between parallax and other parameters (such as camera position, lens parameters, object position), which is complicated and time-independent.The p_max and p_min value can be obtained by image processing algorithm, therefore function H doesn't need to consider in the control loop.Because the interaxial distance can change the 3D volume of the image, for the current parallax range e_c and the specified range e_t, interaxial distance can be controlled to make e_c coincide with e_t, which means the 3D volume of perception is the same with specified value.Also, convergence should be controlled to make the median value of specified range p_mid such that it coincides with the median value of current parallax range p_cen, which means Fig. 2. Perceptual space changing with parallax range adjustment method (Top View).
the parallax range will fill the specified range at the end by the feedback interaxial distance and convergence control.

Target Parallax Adjustment Method
For some close-ups of characters, the target parallax of the awareness character is assigned to provide a proper distance for perception.And the perceptual distortion of the object in stereoscopic (8)(9) is also being greatly considered in 3D making.To make a high quality effect, the character's parallax should coincide with the target parallax and the perceptual shape (ratio of perceptual depth to perceptual width) of the object should as close to the same object's shape in the real world as possible, which means the perceptual distortion should be prevented.
We defined the target and current parallax value of the focused object as p_tar and p_cur.As shown in Fig. 4, interaxial distance and convergence will be controlled to make the p_tar coincide with p_cur while the perceptual shape of the focused object is as close to the same object's shape in the real world as possible.From Equation 2, we can tell the perceptual shape will change in different viewing conditions.Therefore, the optimal screen width w, viewing distance d and the average pupil distance i will also be considered in 3D shooting (5) .We defined the coordinate of the real world and the perceptual space as Fig. 5.  p_max and p_min can be obtained by image processing therefore H doesn't need to consider in this control loop.For an object (x c ,y c ) in the real world, the object's position in the perceptual space shows as (x e ,y e ).The relationship of (x c ,y c ) and (x e ,y e ) is shown as Equation ( 5).
We defined the ratio of the real distance y c to the perceptual distance of the object y e as E, the ratio of the real depth to the real width as T, the ratio of the perceptual depth to the perceptual width of the object as S, and the ratio of T ans S as Q.Therefore, the perceptual shape of the object is the same with the shape of the real object when Q equals 1.The equation of E and Q is shown as Equation (7), and the relationship of E and Q is shown as Equation (8).
Thus, at time t, to keep Q as 1, the target position of convergence and interaxial distance can be calculated as Equation ( 10 For interaxial distance control, we can read the focus parameter from the lens to get the distance between focused object and camera (Function F).The control block diagram is shown as Fig. 6.Interaxial distance will be controlled to make the perceptual shape of the focused object to be the same with the shape in the real world, while convergence will be controlled to keep p_cur(t) coincide with p_tar.

Experiments and Results
We evaluated the method on the programmable stereo camera system called BinoQ-PSeries made by Bi2-Vision Co., Ltd.Interaxial distance and convergence angle are motorized, and can be real-time controlled by program.Two SONY HDC-P1 cameras with Fujinon lens HA14BEZE-T5DD are used to provide the stereo full HD images in 30fps.The maximum speed and acceleration of interaxial distance is set to 9.98mm/s and 9.98mm/s 2 , the maximum speed and acceleration of convergence angle is set to 13.8degree/s and 37.0degree/s 2 .The frequency of motor control program is 23KHz.We use watershed algorithm to extract feature points from left and right image, and use block matching to match the feature points into pairs use regression method to calculate the parallax of the feature points.The frequency of the image processing loop is 10Hz.

Experiment for Parallax Range Adjustment Method
We set the target parallax bounds (p_lower, p_upper) as [-3.0%, 3.0%], [-2.0%, 2.0%], [-1.0%, 1.0%], [-0.5%, 0.5%], [-0.3%, 0.3%], [-0.2%, 0.2%] respectively, and performed 20 times for each pattern.The result about p_max and p_min are shown in Table 1.The Average and Std Dev columns show the average value and standard deviation value of p_max and p_min, and the Max and Min columns show the maximum and minimum value of p_max and p_min for the 20 times.The difference between target value and result value is within 0.02%, which means the method is effective for parallax range control.Fig. 7 shows an example of parallax changing with proposed parallax range adjustment method.The parallax range at beginning is [-0.08%,0.20%], and after about 7 seconds the range is adjusted to specified range [-1.0%, 1.0%] by control the interaxial distance and convergence.

Experiment for Target Parallax Adjustment Method
In this experiment, we chose a cube (ratio of width and depth is 1) as the focused object.We set it at 0.3m, 0.6m, 1.0m away from the camera respectively.For the watch condition parameters, we set the pupil distance i as 65mm which is the average pupil distance of human.We set the screen size w and watching distance d as [w=0.5m,d=1.0m], [w=1.0m,d=2.5m], [w=2.0m,d=3.75m] respectively, the target parallax (p_tar) of the object as 0.0%.We performed the experiment 5 times for each pattern.The initial interaxial distance is set to 0.0mm and the convergence angle is set to 0.0 degree, which means the perceptual depth is close to 0mm at the beginning.After the control is finished, we extract the coordinates of the 4 corners from the stereoscopic image, and the perceptual position and the perceptual shape of the object can be calculated according the geometric equations.
The result of the target parallax adjustment experiment is shown in   average error (absolute value) of the object's parallax.The Width/Depth column shows the maximum, minimum and average value of the ratio of perceptual width to depth of the cube.From Table 2, the difference between target parallax value and adjusted value is under 0.02% which means the object will appear in the specified perceptual position when watching under the specified viewing condition.And the ratio of perceptual width to depth is close to 1.0, which means the perceptual shape is close to the same object's shape in the real world.Thus, the result shows that the method is effective for parallax control of the target object.

Compare Experiment with Manual Parallax Adjustment Method
As mentioned in section 1, the common way of parallax adjustment method is setting interaxial distance and convergence manually while considering camera parameters, objects movement, lens parameters, parallax information of current image and so on.We performed a comparative experiment on adjustment speed and result accuracy between proposed method and conventional method.For the camera operation device, we use the controller called C-Motion which is the most popular device in stereoscopic photography and it can control the interaxial distance and convergence in a wireless environment.Three testers familiar with the system with stereoscopic experience adjust the parallax while watching the 3D image, camera parameters, lens parameters and the parallax information (p_max, p_min, p_cur, ratio of perceptual depth to perceptual width) on screen.We prepared 6 different patterns, and performed 10 times for each pattern.Fig. 8 shows the comparison result about the average value of cost time and residual error.The result shows that the proposed method takes about half of the time cost in traditional manual method, and with more accuracy than human work.

Conclusions
In this paper, we describe the geometric relationship between parallax and shooting or watching parameters, and summarize the theories about parallax adjustment.We propose our parallax adjustment method by image processing, which can control the interaxial distance and convergence automatically to make a specified 3D effect.Also, we evaluated our method on the stereo camera system.The result shows that the method is faster, easier, and more accurate than traditional manual method.
However, the camera movement or focal length changing also makes the 3D contents looks uncomfortable.We want to solve these problems as the further work, and make our effort on the improvement about the quality of 3D contents.

Acknowledgment
This work was supported in part by the Science and Technology Commission Foundation of Shanghai (No.13511507501)

Fig. 3 .
Fig. 3.Control block diagram of Parallax Range Adjustment Method.κ b , λ b and κ c , λ c are the control parameters for interaxial distance and convergence.G 1 (s) and G 2 (s) are the transfer functions for interaxial distance and convergence.H is the relationship function about parallax and other parameters.p_maxand p_min can be obtained by image processing therefore H doesn't need to consider in this control loop.

Fig. 5 .
Fig. 5. Object Position in the Real World and in the Perceptual Space (Top View).

Fig. 6 .
Fig. 6.Control block diagram of Target Parallax Adjustment Method.κ b , λ b and κ c , λ c are the control gains for interaxial distance and convergence.G 1 (s) and G 2 (s) are the transfer functions for interaxial distance and convergence.F is the relationship function about focus value and distance of focused object.B(s) is the control function (Equation (11)) for interaxial distance.H is the relationship function about parallax and other parameters.p_cur can be obtained by image processing therefore H doesn't need to consider in this control loop.
(a) Parallax Range Changing.(b) Interaxial distance and Convergence angle Changing.
(a) Comparison of Cost Time.(b) Comparison of Residual Error.

Table 2 .
The Parallax Error column shows the

Table 2 .
Result of Target Parallax Adjustment Experiment.
Each Experiment has been repeated 5 times for the same conditions.