2 Algorithm Description Active contours are initialized using the output of the SUSAN edge detector [10]. Edge runs that contain a reasonable number (

Transcription

1 Motion-Based Object Segmentation Using Active Contours Ben Galvin, Kevin Novins, and Brendan McCane Computer Science Department, University of Otago, Dunedin, New Zealand. Abstract: The segmentation of an image sequence into meaningful regions is an important early step in computer vision. We describe a motion segmentation algorithm that tracks contours through an image sequence using active contours. This approach eliminates the problems of feature correspondence and feature drop-outs encountered with other feature-based tracking techniques. It also allows accurate recovery of the positions of segment boundaries and features. We present a simple texture-based energy equation that makes the active contours more stable and addresses the problem of `sliding' along edges. The method is capable of processing a frame per second without specialized hardware. The algorithm performs well, although active contours can be \captured" by occluding objects. Results are presented for real image sequences. Keywords: Motion segmentation, snakes, active contours 1 Introduction The segmentation of an image sequence into meaningful regions is an important early step in computer vision. Many cues for segmentation exist, however image motion has the advantage that discontinuities in the motion eld almost certainly correspond to segment boundaries or signicant structural features of the object (exceptions include shadows and highly reective surfaces). This is not the case with discontinuities in the image gradient, which respond to object texture. Motion-based segmentation algorithms fall into two categories: optical ow based and featured based. There have been some encouraging results with optical ow techniques, however most algorithms are slow and fail to locate segment boundaries accurately [7, 3, 1]. The accuracy of the extracted boundaries is important in many applications, for example view-based object recognition and pose estimation. Feature based techniques attempt to overcome some of these problems by focusing attention on a small number of features in the image sequence. Smith and Brady describe a real-time, feature-based motionsegmentation system called ASSET-2 [9]. Corners are extracted from the frames, then correspondences are sought using motion prediction. Object hypotheses are then extracted by clustering the corners displacements over the last 10 frames. Potential problems with this type of approach includes the consistent detection of features, diculties in establishing reliable feature correspondences, and the tracking of more complex shapes (for example, people). These problems can be partially addressed by utilizing contours, rather than corner points, as the features to be tracked. Active contours provide an attractive framework for contour tracking [5]. Estimated velocity is used to move the active contours into their expected positions in the current frame. The active contours are then allowed to re-attach themselves to image features. This eliminates the problems of feature drop-outs and feature correspondence encountered with other feature-based methods. It also allows accurate recovery of the position of segment boundaries and features. The resulting motion information is integrated across several frames to form robust object hypotheses. The main deciency with the method is the inability to handle object occlusion, although we hope to address this in the future. Active contours have been used for tracking contours previously, however many of these algorithms still use the original equations proposed by Kass et al [5]. We have developed a new energy equation which includes a texture correlation term. In constrast to previous implementations of texture based snakes, our technique still works well when the snake is aligned with an occlusion boundary. A notable exception is the work of Rowe and Blake [8], however their algorithm is signicantly slower, more complex, and requires training on the scene.

2 2 Algorithm Description Active contours are initialized using the output of the SUSAN edge detector [10]. Edge runs that contain a reasonable number (10 in the examples presented here) of pixels are approximated by a piecewise linear curve [4]. Initially the curve is approximated with a single linear segment. Next the point on the curve that is farthest from the line segment is found. This point becomes the next node in the linear segment. This process is repeated recursively on the newly created segments until the greatest distance is below some threshold. Once the positions have been initialized, the processing of each subsequent frame proceeds as follows: 1. Using a combination of the active contour's momentum and the cluster's momentum, move each active contour from its position in the previous frame to its expected position in the current frame. 2. Relax the active contours on the current frame. 3. Remove any active contours that are too small, or move outside of the image. 4. Identify segments by clustering on the active contours centroid velocities. The following sections discuss these steps in more detail. 2.1 Move Active Contours to Expected Position Each active contour is translated into its expected position using a weighted average of its previous velocity and the average velocity of this contour's cluster. We considered using optical ow to move the contours, but found it did not signicantly increase accuracy. This can be attributed to the fact that contours are frequently placed on occlusion boundaries where ow is dicult to compute accurately [6]. 2.2 Relaxation The original active contour algorithm described by Kass et al. was an energy-minimizing spline that was attracted to discontinuities in the image [5]. Their energy equation is Z 1 E total = 0 E int(v(s)) + E ext (v(s))ds; (1) where v(s) is a parametric description of the active contour's position, E int is the internal energy, and E ext is the external energy. This energy equation is not appropriate for our application, since the resulting contour tends to straighten out and reduce in size. This is not consistent with the behavior you would expect from a 3D contour undergoing 3D motion. Active contours also have a tendency to slide along edges or jump onto other nearby edges. We have attempted to address these problems by dening a new energy equation. First, we dene the internal energy as: N?1 X E int = w int jv i? v i?1? p i j (2) i=1 where N is the number of control points, v i is the position of the ith control point, p i is the time-decaying, average value of the quantity v i? v i?1. The set of vectors p i describe the active contour's \average" shape over the last few frames. E int then measures the dierence between the current shape and this \average" shape. This ensures that the shape doesn't change rapidly. Our external energy equation has two terms: E ext (v) = w edge E edge (v) + w texture E texture (v) (3)

3 The rst term, E edge, is the usual?ri(x; y) term which attracts snakes to edges. The weight for this term is typicaly very small. The second term, E texture, in the external energy equation allows the active contour to attach itself to texture in the image. If the active contour is situated on an occlusion boundary, it will also learn to attach itself to the texture on the occluding object side, and ignore the unreliable texture on the occluded side. We dene E texture as a weighted average of the left texture correlation, C left, and the right texture correlation, C right. The weighting is controlled by the variable. A of 0:0 indiciates the active contour is attached to the texture on the left-hand side only, a of 1:0 indicates it is attached to the texture on the right-hand side. More formally, N?1 X E texture (v) = C left + C right (1? ) (4) i=1 Texture correlation is measured by summing the dierences of corresponding pixel values. A small sum indicates a high correlation. C left and C right are dened as M?1 X C left = ji(v i?1? n l + d j)? left j j (5) j=0 M?1 X C right = ji(v i?1 + n l + d j)? right j j (6) j=0 where I(u) is the interpolated gray value at u, M is the number of samples taken along each segment (typically 20), d = (v i? v i?1)=(m? 1), n is a unit length vector perpendicular to v i? v i?1, l is a small constant (typically 3 or 4), left j and right j are time-decaying averages of the gray value samples along each side of the active contour. These variables are illustrated in Figure 1. Initially is assigned the value 0:5. After each iteration it is updated using a weighted average of the previous value and the current left-right correlation: 0 = 0:7 + C left =(C left + C right ) 0:3 (7) We've found that using E texture makes the active contours signicantly more stable, and if sucient texture is present, prevents the familiar problem of active contours sliding along edges. Rowe and Blake [8] have proposed a similar, although signicantly more complex texture-based active contour model. The gray-level intensity proles along line segments normal to the contour are modeled with probability density functions. These are extracted from a training phase using a simple bootstrap contour tracker. The tracking algorithm uses dynamic programming to nd the warping function which best maps the actual intensity prole to the PDF along each line segment. The mappings for each line segment are combined to nd the new control points for the contour. They have demonstrated that this approach performs well in the presence of background clutter, however it comes at considerable computational cost: the system is able to track a single contour with 8 PDFs at a rate of 0.2Hz on a Sun IPX. The algorithm presented in this paper can track and cluster at least 100 active contours at 0.2Hz on a Pentium 233, and does not require a seperate training phase. Because the energy equations do not require second derivatives, a simple version of the Viterbi relaxation algorithm can be used, which reduces computation time by a factor of 9 [2].

4 Figure 1: The variables used to calculate E texture. 2.3 Clustering The active contours are clustered on their centroid velocity over the last n frames (7 in our examples). More complex measures, including consistency with 3D rigid motion were considered, however this simple technique gives adequate performance without the additional computational overhead. The clustering procedure is an adaptive k-means algorithm, as described by Wang and Adelson [11]. The distance metric is dened as d(a; b) = ja? bj jaj + jbj + (8) where a and b are displacement vectors. The position of the cluster centers is then recalculated as the average position of its member data vectors. If at any stage a cluster becomes empty, or becomes too close to another cluster, it is removed. This process iterates until cluster membership is stable. where is a constant used to compensate for noise. After the clustering is complete, it is necessary to establish a correspondence between the clusters in this frame and clusters in the previous frame. We have developed the following simple technique for establishing the correspondence. Suppose M i is the set of active contours in the ith cluster of the previous frame, and N j is the set of active contours in the jth cluster contains in the current frame. We begin by creating a matrix A with dimension max(jm j; jn j) by jn j, where jm j denotes the cardinality of the set M. Each element of A is calculated as a ij = jm i \ N j j jm i j + jm j j ; (9) where jm i \ N j j is the number of contours that are in both cluster i of the previous frame and cluster j in the current frame. Cluster correspondences are then extracted by the following greedy algorithm: 1. Find (p; q) such that 8ij; a pq a ij. 2. If a pq =?1 then nish. 3. Assign a correspondence between cluster q in the current frame to cluster p in the previous frame. 4. Update the matrix A such that 8j; a pj =?1, and 8i; a iq =?1. 5. Goto step 1

5 (a) (b) (c) (d) (e) (f ) Figure 2: Results for the Hamburg taxi sequence. The rst frame on the sequence is displayed in (a). The snakes extracted from this frame are displayed in (b). The nal frame in the sequence is displayed in (c). The 3 clusters extracted from the sequence are illustrated in (d), (e) and (f). 3 Results The results obtained from applying this algorithm to the Hamburg taxi sequence are displayed in Figure 2. All of the unoccluded active contours successfully track through the 20 frame sequence. The background and taxi have been successfully segmented from the sequence. An error has occured in the third cluster where the snake originally associated with a road marking has been `captured' by the occluding vehicle. We hope to resolve this problem by detecting the occlusion and preventing the occluded active contour from relaxing for the duration of the occlusion. The vehicles on the lower left and right of the frame have few snakes attached because the edges weren't found by the edge detector. The results for the ambulance sequence are shown in Figure 3. Most of the unoccluded active contours track successfully over the 30 frame sequence. The ambulance and jeep successfully segmented after 4 frames. A number of active contours have again become trapped by occlusion, particularly around the jeep on the left, and to the immediate left of the ambulance. 4 Conclusion We have described a method for motion-based segmentation by tracking contours through the sequence. A modied energy equation was presented that increases stability by allowing each active contour to attach to nearby texture. This energy function also performs well when the snake is situated on an occlusion boundary. Object hypotheses are extracted by clustering on contour displacement. A simple algorithm for establishing correspondences between objects found in adjacent frames was also presented. The algorithm was applied to real image sequences, and the resulting segmentation was stable, robust to image noise, and usually recovered object boundaries to within pixel accuracy. Over 100 snakes can be tracked at a rate of 0.2Hz without special hardware. We intend to extend the algorithm by compensating for object occlusion, and extracting 3D structure. Other possible avenues for research include motion based object recognition.

6 (a) (b) (c) (d) (e) (f ) Figure 3: Results for the ambulance sequence. The rst frame on the sequence is displayed in (a). The snakes extracted from this frame are displayed in (b). The nal frame in the sequence is displayed in (c). The 3 clusters extracted from the sequence are illustrated in (d), (e) and (f). References [1] G. Adiv. Determining three-dimensional motion and structure from optical ow generated by several moving objects. Proceedings of the IEEE on Pattern Matching and Machine Intelligence, 7(4):384{ 401, [2] A. Amini, T. Weymouth, and R. Jain. Using dynamic programming for solving variational problems in vision. IEEE Transactions on Pattern Analysis and Machine Intelligence, 12:855{867, [3] M.M. Chang, A.M. Tekalp, and M.I. Sezan. Simultaneous motion estimation and segmentation. IEEE transactions on image processing, 6(9):1326, [4] R. Jain, R. Kasturi, and B. Schunck. Machine Vision, page 196. McGraw-Hill Inc., [5] Michael Kass, Andrew Witkin, and Demetri Terzopoulos. Snakes: Active contour models. International Journal of Computer Vision, pages 321{331, [6] B. McCane, B. Galvin, and K. Novins. On the evaluation of optical ow algorithms. In International Conference on Control, Automation, Robotics and Vision, Singapore, [7] D.W. Murray and B.F. Buxton. Scene segmentation from visual motion using global optimization. Proceedings of the IEEE on Pattern Matching and Machine Intelligence, 9(2):220{228, [8] S. Rowe and A. Blake. Statistical feature modelling for active contours. In European Conference on Computer Vision B '96, pages 561{569, [9] S.M. Smith and J.M. Brady. Asset-2: Real-time motion segmentation and shape tracking. Proceedings of the IEEE on Pattern Matching and Machine Intelligence, 17(8):814{820, [10] S.M. Smith and J.M. Brady. Susan - a new approach to low level image processing. International Journal of Computer Vision, 23(1):45{78, [11] J. Wang and E. Adelson. Representing moving images with layers. IEEE Transaction on Image Processing, 3(5):625{638, 1994.