Scale Invariant Segment Detection and Tracking

Transcription

1 Scale Invariant Segment Detection and Tracking Amaury Nègre 1, James L. Crowley 1, and Christian Laugier 1 INRIA, Grenoble, France firstname.lastname@inrialpes.fr Abstract. This paper presents a new feature detector that fits correctly anisotropic elongated shapes. This detector that mixes ridges and lines detection appears to be very robust to affine and particularly scale transformation. A tracking process, that uses the same properties than the segment detector has been studied in the aim to estimate time-to-contact from detected obstacles in an urban environment. Experimental results show that the method gives good results even in complex scenes. 1 Introduction Visual feature detection and tracking is often the root of many visual robotics tasks like localization and mapping [2,4], object recognition[9], etc. Local interest point detection for image matching began with the Moravec corner detector [11]. This detector was improved by Harris and Stephens [6], who made it more repeatable under image transformations. Nevertheless the scale invariance was not achieved as the detection was made only at a fixed scale level. The scale representation of image has been studied in [7] and used in [1] to detect peak and ridge in a scale space. Linderberg [8] also worked on automatic selection of the best scale for feature detection. Lowe [9] proposed an object recognition method based on local 3D extrema in a difference-of-gaussian pyramid. In [10] an other scale invariant interest point has been developed by mixing Harris and Laplacian detector. Two problems remain with these interest point detection algorithms : they are not adapted to anisotropic structures and they are not centered with the "physical" object they represent. For an elongated object like pedestrian or pole, one would obtain a feature point centered in the middle of this object and some parameters describing the object s shape. In this article, we propose an algorithm based on ridge lines [5] to detect scale invariant segments that fit correctly elongated shapes. In a second part, we will see how to track such segments using a particle filter to estimate object s motion in the 3D scale-space and we ll see how to use this tracking to measure the time-to-contact. Next, we ll experimentally evaluate the quality of our algorithm by detecting and tracking objects in a very simple case and comparing with a ground-truth. And then we ll apply our algorithm in a complex urban environment with a moving camera video sequence.

2 2 Amaury Nègre, James L. Crowley, Christian Laugier 2 Scale invariant segment detector 2.1 Algorithm description Extremal points in a Laplacian scale space have long known to provide scale invariant feature points. However, if we consider only this criteria we ll need to deal with two problems : 1. The Laplacian exhibits local extrema in the center of object but also on the edge, where characteristic scale is nonsense, we thus need to eliminate such response; 2. In the case of elongated object like poles, pedestrians, etc., detected points can move all along the object, only the two extremities can give stable points, but the center of object is often preferable. To solve those problems, our method consists in detecting segments instead of 2D points in a Laplacian scale-space (approximated by a Laplacian pyramid) : 1. For each pixel of the Laplacian pyramid we first remove the edge response by eliminating pixels where the ratio between the Laplacian and gradient values is greater than a threshold. 2. We compute for each pixel the best segment direction which corresponds to the ridge direction. As shown in [5], this direction is normal to the principal curvature direction given by the eigen vector ) associated to the biggest eigen value of the Hessian matrix H = ( 2 f 2 f x 2 xy 2 f 2 f xy y 2 where 2 f x 2, 2 f xy, 2 f y 2 are the second derivatives of the Image. 3. For each pixel we search the length l that maximizes the following score function: S(X, l, u) = ( L(X + k u X ) + L(X k u X ) k=0..l 2 L(X + k u X ) L(X k u X ) 2 L min ) Where L(X) is the Laplacian value at pixel X and u is the previously computed direction This score is high when many pixels along the line have a high Laplacian ( L(X + k u X ) + L(X k u X ) ), and is maximal at the center of the object ( L(X + k u X ) L(X k u X ) increases near the border). The term L min makes the score decrease when k is greater than the object s characteristic length. It should be noted that the cost of the score function is only proportional to the maximal searched length. 4. We then search local maxima of the previous score in the pyramid to obtain best segments. When the feature is not a perfect line, the local direction of the principal curvature is not strictly aligned with the feature. The score can then be optimized by scanning some neighboring directions. (1)

3 Scale Invariant Segment Detection and Tracking 3 A typical example of segments detection in an urban environment is shown in Figure 2.1. The detected segments are represented by red ellipses, the main axis of the ellipses represents the segment s half-edges, and the width of the ellipses represents the object scales. The interesting point is that all detected segments are centered on the object they represent and the detected scales fit very well object s size. Fig. 1. Scale invariant segment detector in an urban environment, the detected segments are represented by red ellipses. 2.2 Performance evaluation With the aim of studying the stability of segments detection to various transformations, we used the repeatability criterion as described in [13]. It consists in detecting all the segments in an original image, applying a known image transformation and counting the number of new segments that can be associated with the first set. The association between segments is done considering the center s position and scale. To take into account the fact we work with a Scale Space Pyramid, the maximal association distance is proportional to the scale. We compared the segments repeatability with a Harris detector [6] computed at different scales and a SIFT detector [9]. The results are shown in Figure 2, we can see that the segments detector is not as much robust as we thought to rotation change, but actually the low performance comes from the localization error that can be high along the ridge direction. As expected, the change of scale doesn t affect the detector s performances.

4 4 Amaury Nègre, James L. Crowley, Christian Laugier (a) Original test image and detected segments Ridge Segments SIFT Harris Repeatability Angle (degree) (b) Repeatability with respect to rotation angle Ridge Segments SIFT Harris Repeatability Scale factor (c) Repeatability with respect to scale factor. Fig. 2. Repeatability of Harris, SIFT and Ridge-Segment detector with respect to rotation and scale change. We can see that the segment detector obtain moderate score for rotation changes but gives similar results than the SIFT detector with change of scale. 3 Segment tracking and estimation of time-to-contact 3.1 Introduction of the Time-to-conctact The time to contact τ (also called time-to-collision or time-to-crash) is a crucial information in the context of obstacle avoidance in dynamic environment. This

5 Scale Invariant Segment Detection and Tracking 5 measure represents the distance between two actors in the temporal space. It has been shown in [12] that the time to contact between the camera and a visible obstacle can be computed in the image space by measuring the variation of the intrinsic scale : if s is the intrinsic scale of the obstacle in the image, then τ can be approximated by : τ = s s t (2) The difficult point with this method is to identify and measuring the change of scale of an obstacle in the video sequence. This can be achieved by detecting obstacle s ridge segments as explained in the previous section and track these segments in the 3D scale-space in order to evaluate the object s motion and change of scale. 3.2 Segment tracking To perform the segment tracking, we propose to use the score function described in 1 as a measurement function to the tracking process. As this function is non-gaussian and non linear, we were motivated to use a particle filtering method [3]. Such a particle filter uses a set of samples (particles) to represent a probability distribution resulting for a Bayesian filtering process. Formally, if we let X t be the state of the target at time t, Z t be the observation of the target at time t, then the goal of the particle filter is to estimate : P (X t Z t1...z tn ) To this end, the filter relies on two models : (1) the observation model P (Z t X t ) that predicts a target observation, and (2) a displacement model P (X tk X tk 1 ) that predicts the motion of the particles. The target tracking is implemented as follows ; we use a set of particles to characterize the target s position and the target s speed in the image coded by three 3-dimensions vectors : - c : the segment s center position in the 3D Scale-Space; - v : the segment s center speed ; - r : the 3D vector between the center and an extremity (called "half-edge" in the following). c X = v r When a target is identified in an image for the first time, all particles are initialized around the detected position and with a zero speed. Next, each camera s image (Z t ) is used as an observation to update the particle filter. The observation model takes into account the score function described in Equation 1 :

6 6 Amaury Nègre, James L. Crowley, Christian Laugier r P (Z t X t )) S(c t, r t, r t ) In between two images, we consider that the target s center is subjected to a Gaussian acceleration a and that the half-edge can also be affected by a Gaussian noise n. Each particle is then updated using the following model : X t+ t = c t+ t v t+ t = c t + (v t + a t) t v t + a t r t+ t r t + n t The output of the particle filter is a probabilistic estimation of the target state. In practice, to obtain a single state of the target, we compute the average pose of all particles weighted by their probabilities. It can be noted that the tracking is automatically ended either when the target get out the image frame or when the observation probability fall down. As we track many segment at same time, we also add a mechanism to fuse differents segments that converge. The global schema of the detection and tracking of scale invariant segments is shown on Figure 3. Fig. 3. Global schema of scale invariant segment detection and tracking. 4 Experimental results 4.1 Simple tracking To demonstrate the ability of the tracking, we consider a simple video sequence with a black rectangle printed on a white sheet. The tracking result is shown on

7 Scale Invariant Segment Detection and Tracking 7 Figure 4(a). We can see that the estimated segment (red ellipse) is centered on the black rectangle, but the length doesn t fit very well the rectangle s length, however, the rectangle scale (height) is precisely approximated. To evaluate this precision, the height of the rectangle has been measured by hand in each image and used to estimate the actual time-to-contact. The results are plotted in Figure 4(b)(c). As expected the two curves fit very well. (a) Image sequences Detected size Real size 40 Estimated TTC Real TTC 90 Segment size (pixels) Time (b) Scale of the tracked segment TTC Time (c) Time-To-Contact of the tracked segment Fig. 4. Tracking of a simple black rectangle on a white sheet. In (a) we can see the set of particles particles (green lines) and the estimated state (red ellipse). The plot (b) represents the estimated scale of the tracked segment and a ground truth obtained by measuring by hand the rectangle height for each image. We can see in (c) that the estimated time-to-contact fits very well the ground truth. 4.2 Tracking in urban context To demonstrate the efficiency of the scale invariant segment detector and tracking, we have tested our algorithm on real world sequences. To obtain real-time performance, we implemented the algorithm on a Graphic Processing Unit (GPU), which made it possible to detect and track simultaneously 64 segments in 640x480 images at 17 frames per second.

8 8 Amaury Nègre, James L. Crowley, Christian Laugier Some results are shown in Figure 5, tracked segments are represented by red ellipse and each segment s trajectory in the image is drawn with a black curve. In this sequence, we noted that the segment s tracker locks well on to road lines, car s elements, trees and poles. For a more detailed analysis, we plotted the evolution of scale of the pole located in the center right part of the image (Figure 6(a)). As the camera moves forward and backward, the object s size increases and decreases which is approved by the tracker. The approximated time-to-contact (Figure 6(b)) is also appreciated as it begins positive and decreases (which mean the obstacle is approaching) and next it swaps negative, which mean the obstacle is getting away). 5 Conclusion In this paper, we developed a new feature detector that can be used for many visual tasks. Based on ridge and line detector, this detector is scale invariant and well adapted to elongated objects that are few textured. A tracking algorithm based on a particle filter has been elaborated for such segment s features. The same ridge and linearity criteria than the segment s detector has been used for the filter s importance function. This tracking has prove effective in complex urban video-sequences and has been used for time-to-contact estimation based on change of scale. One particularity of the detector is that it is localized in the center of corresponding object, this property is interesting as it can improve object s localization and data association, as a appearance descriptor would only describe all the object and not only a part (like on object s edge or corner). Such a descriptor is necessary to perform long term tracking and for the SLAM problem, this would be approached in future research. References 1. J. L. Crowley and A. C. Parker. A representation for shape based on peaks and ridges in the difference of low-pass transform. IEEE Transactions on Pattern Analysis and Machine Intelligence, 6(2): , Andrew J. Davison, Ian D. Reid, Nicholas D. Molton, and Olivier Stasse. Monoslam: Real-time single camera slam. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(6): , Arnaud Doucet, Nando De Freitas, and Neil Gordon, editors. Sequential Monte Carlo methods in practice. Springer, E. Eade and T. Drummond. Scalable monocular slam. Computer Vision and Pattern Recognition, 2006 IEEE Computer Society Conference on, 1: , June TRAN Thi Thanh Hai and Augustin Lux. Extraction de caractéristiques locales : Crêtes et pics. In RIVF, pages , C. Harris and M. Stephens. A combined corner and edge detector. pages , Manchester, Jan J. Koenderink. The structure of images. Biological Cybernetics, 50: , Tony Lindeberg. Detecting salient blob-like image structures and their scales with a scale-space primal sketch: a method for focus-of-attention. volume 11, pages , Hingham, MA, USA, Kluwer Academic Publishers.

9 Scale Invariant Segment Detection and Tracking 9 Fig. 5. Segment detection (right) and tracking (left) in an urban context. Red ellipses with blue axes represent the tracked segments, black curves represents segment s trajectories in the image.

10 10 Amaury Nègre, James L. Crowley, Christian Laugier Estimated Size (pixels) Estimated TTC Time (a) Evolution of scale for the segments localized in the white pole Time (b) Estimated Time-To-Contact of the white pole. Fig. 6. (a) Evolution of the scale of the tracked segments localized on the white pole (center right of images). As the car moves forward and backward, the scale increases and decreased. The estimated Time-To-Contact (b) is well approximated : at the beginning it is positive and it decreases (which mean the obstacle is approaching) and next the TTC swaps negative (which means the obstacle is getting away. 9. D. G. Lowe. Object recognition from local scale-invariant feature. In International Conference on Computer Vision, pages , K. Mikolajczyk and C. Schmid. Indexing based on scale invariant interest points. In Proceedind of the International Conference on Computer Vision, pages , H. Moravec. Rover visual avoidance. In International Joint Conference on Artificial Intelligence, pages , Vancouver, Canada, Amaury Negre, Christophe Braillon, Jim Crowley, and Christian Laugier. Real-time Time-To-Collision from variation of Intrinsic Scale. In Proc. of the Int. Symp. on Experimental Robotics, Rio de Janeiro, Brazil, Schmid, R. Mohr, and C. Bauckhage. Comparing and evaluating interest points. pages , 1998.