Real-Time Automatic Target Selection and Tracking Algorithm Using Keypoint-Based Template Matching

.


Introduction
Among a variety of different approaches for robotic bin picking, computer vision has been widely used in development of vision-based object detection and tracking systems, which is an important technique to bring intelligent robots into industrial applications and daily life of human beings.Although many vision-based object detection and recognition algorithms are well studied in literature, only a few attentions have been devoted to develop detection and tracking of an object-of-interest (OOI) from multiple stacking objects in a box, which is a fundamental operation in applications of robotic random bin picking.Therefore, this paper presents a novel target selection and tracking algorithm for multiple object separation, target selection, and target tracking.Moreover, the proposed method is computational efficiency and achieves about 30 frames per second (fps) for a video stream with size 640x480 pixels.This advantage improves the applicability of the proposed method.Experimental results validate the performance of the proposed method.

Problem Statement
Keypoints are image features that can be repeatedly detected in images robust to translation, scaling, and rotation transformations.To detect an OOI appeared in the image, various keypoint description approaches, such as SIFT (1) , PCA-SIFT (2) and SURF (3) algorithms, have been proposed to detect the OOI in a query image.Given a reference image of the OOI, the keypoint-based object detection methods first extract a reference keypoint set from the reference image.As a new image captured from the visual sensing system, an object detection module extracts a query keypoint set of the captured image and finds the keypoint matches between the query and reference keypoint sets.To improve the object detection accuracy, a RANdom Sample Consensus (RANSAC) algorithm (4) is usually applied to remove outliers in the keypoint matches.The traditional keypoint-based object detection methods are useful to detect a single OOI in a complex environment; however, they may not work well to detect multiple objects randomly placed in a box, as shown in Fig. 1.This problem motivates us to develop a real-time keypoint-based target selection and tracking system to detect the top OOI from the multiple stacking objects randomly placed in the box.

Proposed Method
Figure 2 shows flowchart of the proposed target selection and tracking system, which consists of object detection, object separation, and object tracking steps.In the object detection step, the SURF algorithm is employed to extract keypoints in the current input image.In the object separation step, the image coordinates of the center point of each object are first estimated by a center-point voting process (5) .Then, the number of objects is determined by a data clustering algorithm, such as K-means (6) or mean shift (7) techniques.Here, we use mean shift algorithm for data clustering as it is suitable to classify multiple classes without an initial guess of the number of classes.After separating each OOI, a template-based visual tracking method (8) is used to track the top OOI in real-time.

Object Detection
To detect the OOI in the image, the keypoint-based object detection method is used, which detects all possible objects based on keypoint matches between a given object reference image and an input query image.Figure 3 shows the SURF keypoints extracted from the reference and query images.Next, a keypoint matching set is obtained from the reference and query keypoint sets by using a fast keypoint descriptor matching algorithm (9) .

Object Separation
To separate the detected objects in the image, a center-point voting algorithm (5) is first used to estimate the position of center point of each object.Let m q and m r denote the matched SURF keypoints in the query and reference images, respectively.Then, the center point c q of a corresponding object in the query image is estimated by where r is a scale ratio defined as where σ q and σ r denote the scale of the keypoints m q and m r , respectively.v r is a reference central vector from the center point of the reference object to the matched reference keypoint m r such that where c r is the center point of the reference object.M ori (ϕ) denotes an orientation rotation matrix associated with the difference between the orientation angle of the two matched keypoints m q and m r such that ( ) where θ q and θ r are the orientation angle of the keypoints m q and m r , respectively.Figure 4 shows the center-point voting result obtained from Eq. ( 1).It is clear that Eq. ( 1)  efficiently maps each matched query keypoint to the location near the center point of a corresponding OOI.
To estimate the number of objects and a more accurate center point of each object, the mean shift clustering algorithm is then used to separate the center-point voting result into multiple object clusters by iteratively searching the densest places of the voted center points.Figure 5 shows the object clustering result obtained from the mean shift clustering algorithm.One can see from Fig. 5 that the voted center points are separated into three groups, which is the number of objects appeared in the query image.Moreover, the red points indicate the estimated center point of each detected object, which is close to the real center point of each object.

Object Tracking
Object tracking plays an important role for the object picking task as it is able to efficiently locate the position of OOI in the image with real-time performance.Thus, after separating and locating each object in the query image, a visual tracking algorithm is used to track the top OOI, which is the object cluster having maximum voted center points.Here, we employ a template-based visual tracking algorithm because a part of the reference image around the center point can be used as the template patch (Fig. 6) for template matching.The performance of the proposed target selection and tracking algorithm is validated in the next section.

Experimental Results
The proposed algorithm was implemented in C++ with OpenCV 2.4.3 library running on a Windows 7 platform with 3.1GHz Intel Core i5-4440 processor.A box containing three randomly placed objects was used to test the performance and reliability of the proposed algorithm in the application of real-time target selection and tracking.Figure 7 presents the experimental results, in which the proposed algorithm aims to consecutively detect and track one OOI in the box.In Fig. 7(a), the three objects in the box are detected successfully, and the top object is selected as the OOI to be tracked and picked up from the box, as shown in Fig. 7(b).In Figs.7(c) and 7(d), the second largest object cluster is then chosen as the OOI to be removed from the box.Finally, the last object is detected as the OOI to be picked up, as shown in Fig. 7(e).Therefore, these experimental results validate the performance of the proposed algorithm.
Note that, the proposed algorithm is also computationally efficient as it achieves about 30 fps when processing a standard VGA (640x480 image resolution) video stream in the experiments.This advantage improves the usability of the proposed algorithm in practical applications.

Conclusions
In this paper, a novel keypoint-based automatic target selection and tracking algorithm is proposed.The proposed algorithm consists of object detection, object separation, and object tracking steps.Any SIFT-like keypoint detector can be used in the object detection step of the proposed algorithm.In the object separation step, a center-point voting process combined with the mean shift clustering algorithm is proposed to estimate the number of objects and the center point position of each object.Moreover, the top object also can be found according to the number of voted center points in the classified object clusters.Finally, a template-based visual tracking algorithm is used to track the top OOI in real-time.Experimental results validate the performance of the proposed algorithm, in terms of detection accuracy, usability, and computational efficiency.

Fig. 1 .
Fig. 1.The target selection problem considered in this study: multiple stacking objects are randomly placed in a box, and we have to decide which one should be taken from the box first.

Fig. 2 .
Fig. 2. Flowchart of the proposed target selection and tracking system for the application of random bin picking.

Fig. 3 .
Fig. 3. Keypoints extracted from an object reference image (left) and an input query image (right).

Fig. 5 .
Fig. 5. Estimation of the center point of each OOI by using mean shift clustering algorithm.

Fig. 6 .
Fig. 6.Template patch used in the object tracking step.

Fig. 7 .
Fig. 7. Experimental results of the proposed target selection and tracking algorithm to consecutively detect and track one OOI randomly placed in a box.