Three-dimensional fish body contour measurement and posture estimation from stereo images in controlled conditions

Measurement of fish locomotion is an essential issue for robotics researchers, biological studies and ecological scientists. This paper focuses on three dimensional fish posture measurement using stereo camera. Three dimensional fish contour is reconstructed using two dimensional fish contours obtained from left and right views. Corresponding contour points from each view were paired for obtaining 3D coordinates of each contour point. Subsequently, RANSAC algorithm was utilized to estimate the 3D fish posture from the 3D contour. Fish posture was estimated by fitting a piecewise line to the 3D fish contour. Experiments were conducted with the presented fish observations system. In the experimental results, estimated posture was accurately fit to original fish pose.


Introduction
Fish has remarkable swimming ability in hovering, turning in intricate water currents and dexterous manipulation under floating conditions [1].The perfect body mechanisms and swimming modes of fish could inspire innovative designs to improve the ability of locomotion of aquatic robots [2].Biomimetic robotics has attracted an increasing number of attentions from the researchers.Recently, the fish-like propulsion mechanism, the fin material, and the mechanical structures have been focused on in research works.
Most of the previous works studied the fish locomotion patterns and swim modes by the mathematical models and computer simulations.The artificial fish model was studied for producing the animation of fish school for computer graphic [3].As the demand of bio-inspired robotics the swimming pattern of fish was studied and a number of fish model has been proposed and applied to control locomotion of robotic fish [4,5,7].
In the last decade, modeling of fish swimming had a great progress and was successfully applied to robotic fish control.Fish locomotion should not only be studied by the mathematical models but also need to be verified by the actual fish locomotion data.
As the achievement of the computational capability of personal computers and the reliable image processing techniques has been implemented, visual monitoring of fish behaviors becomes a commonly tool for the scientific studies and environmental monitoring systems.The fish swimming modes were measured by video camera and colored markers attached to the fish body [6].Five markers were manually attached to fish body from tail to head.The body motion data was obtained by the 5-point data.In order to track the body motion, markers should have high contrast with the fish body in gray scale.Attaching markers to fish body needs a lot of manual operations and it requires skilled operator to minimize the positioning error of markers.Consequently, the marker less automatic observation method is necessary to track and measure the fish locomotion, such as model based posture analysis [8].Recently, A deformable fish model is constructed based on the active shape model (ASM).Shape variations of fish body were modeled in two-dimension space.Precise fish posture was measured from top view of fish body by using the deformable fish model.And the fish posture was represented as a skeleton line for further analysis.
In this paper, a three-dimensional fish posture measurement is studied based on stereo vision.The 3D contour of fish is reconstructed based on 2D edges from fish images.And the 3D posture of the fish is represented by a straight line estimated from the 3D fish contour.
The fish motion data was captured and recorded by a digital stereo camera in this work.The structure of fish observation system is presented in Fig. 1.This observation system consisted of a small aquarium, a digital camera, a light source and a desktop computer.Top view observation was chosen to measure and analyze the fish locomotion.The digital camera was placed over the aquarium that capturing top view images of the aquarium.Since the shape of fish body would not change much in the top view image, it is stable and reliable to analyze the posture of fish and track fish movements.We constructed a stereo camera using two webcam produced by Logitech® company (Logitech® C525) with 2 Megapixel resolution.The cameras were fixed on a plastic board that ensuring optical axes of the two cameras were in parallel.
The field of view of the stereo camera was required to cover the whole movement area of the fish.Only the overlapping area of the left and right camera is available to calculate the 3D information.The field of view of the camera was optimized by adjusting the distance from camera to the water surface in the arena.According to the experimental conditions, the light source was placed below the aquarium in order to acquire the maximal contrast between the fish and the background.Locomotion of a single fish was observed in this study.Three-dimensional fish observation system

Stereo fish measurement
A binocular camera system was adopted in our work.To obtain accurate measurement by the binocular camera system cameras should be calibrated to rectify the optical distortion of the pinhole camera structure.Cameras were calibrated with chessboard image according to Zhang's method [10].Camera parameters were obtained in the calibration process and original images were subsequently rectified.The rectified fish images are presented in Fig. 2. Image distortion was rectified and the two images were aligned to the epipolar line in Y axis.Therefore, three dimensional coordinates could be calculated by pairing the corresponding pixels from left and right view along the epipolar line.Since the observation was conducted under stabilized light condition and background of aquarium was clean which providing the optimized contrast for extracting fish images.The fish showed a strong contrast to the white background which could be easily extracted by the thresholding method.Comparing with the widely used background subtraction method, adaptive thresholding is more robust for fish segmentation against the illumination changes.Usually, background subtraction requires calculating mean image of background, such as MOG (Mixture of Gaussians), to deal with the light changes.However, fish could not continuously move all the time when a fish stay for a certain period in the image it would be counted as the background in the process of calculating mean background over time.Therefore, the adaptive local thresholding method was utilized to extract the individual from the background for enhance the robustness of fish segmentation.The fish body was presented in a white shape in the binary image after segmentation.Further analysis was conducted based on the binary images which minimized the computational complexity and processing time.As shown in Fig. 3, observation images were binarized by using the adaptive thresholding method.Fishes were accurately extracted from the background.Although noise blobs were detected with the fishes they could be effectively removed by examining the geometric parameters (e.g., area size, length and etc.).Subsequently, 2D fish contours were obtained by analyzing 2D blobs of fish images.
The 2D fish contours are presented in Fig. 4. Fish contours were aligned according to the epipolar line since the fish images from stereo camera were rectified.The yellow line indicates the epipolar line that fish contours were exactly aligned.Therefore, we could find the corresponding pairs of contour points from left and right view to reconstruct 3D fish contour.With the binocular vision system, the depth of an object can be calculated by using the triangulation method.Usually the triangulation returns the depth information described in camera coordinate, this depth is the distance from the camera optical center to the object.Camera coordinates provide the 3D information in space.The relation of image coordinate and camera coordinate is given as the following equations.
where f is the focus length and   is the depth in Z axis of the camera coordinate which calculated in Eq. (2).
This equation shows the triangulation on Z axis, b is the baseline between two cameras and U L and U R are the projections of a scene point on the two image planes.Thus, from Eq. ( 1) and Eq. ( 2) we have the equations that represent the transformation from image coordinate to camera coordinate.

Experimental Results
Experiments were carried out in laboratory condition with the proposed 3D fish observation system.A single fish was tested for 3D posture measurement.The proposed measurement scheme was implemented using OpenCV and C++.
An example of 3D posture measurement result is given in Fig. 5. Black dots are the 3D coordinates calculated from sampled contours points.And a piecewise of straight line was estimated from these 3D coordinates.The estimated line was accurate to fit the fish pose.To reduce the computational complexity a proportion of the contour points were selected to reconstruct the 3D fish contour.And the RANSAC algorithm was adopted to estimate fish pose by fitting to the 3D fish contour.Outliers of the 3D fish data were successfully removed in the fitting process of RANSAC.Consequently, best fitting to the fish contour was obtained as presented in Fig. 5.

Conclusions
In this paper, a 3D fish measurement study is developed based on stereo vision.Two-dimensional fish contours were obtain from left and right view in stereo camera.Three dimensional fish contour was obtained according to the epiploar geometric.Subsequently, fish posture was estimated by fitting a piecewise line to the 3D fish contour.The experimental results showed estimated posture could accurately represent the actual fish posture.Measuring fish posture in 3D is important for fish behavioral study, environmental monitoring and even for robotic fish study.This work presented a preliminary study on 3D fish measurement.Numerical evaluation and stability of the proposed measurement scheme should be carried out through more experiments and optimization should be conducted in further works.

Fig. 4 .
Fig. 4. 2D fish contours from left and right view aligned according to epipolar line.
where x and y are the offset to the image center on X axis and Y axis of the image plane.Consequently the 3D points of the camera coordinate are achieved by Eq. (3).