Geodesic Active Regions for Tracking I.N.R.I.A Sophia Antipolis CEDEX, France.

Transcription

1 Geodesic Active Regions for Tracking Nikos Paragios? Rachid Deriche I.N.R.I.A BP. 93, 24 Route des Lucioles 692 Sophia Antipolis CEDEX, France Abstract. In this paper we propose some new ideas for tracking multiple moving objects by the propagation of curves. We assume a static observer, as well as the existence of a background reference frame. The tracking is performed using an improved Geodesic Active Contour model that incorporates boundary-based and region-based motion information. This model is called a Geodesic Active Region model. Initially, a statistical analysis is performed which provides a measurement that distinguishes between the static and mobile regions in terms of conditional probabilities. Based on this analysis, the boundary-based information is expressed via discontinuities in the associated probability space, while the region-based motion information is expressed directly from the conditional probabilities. An objective function is dened that integrates boundary-based and region-based tracking modules by seeking curves that provide high boundary-based information and maximum a posteriori segmentation probability with respect to the mobile hypothesis on the interior curve regions. This function is minimized using a gradient descent method, where boundarybased and region-based motion forces move the initial curves towards their global minima. Additionally, a regularity constraint is imposed for the propagation of the curves which is manifested as an intrinsic term of the partial dierential equation that deforms these curves. This PDE is implemented using a Level-Set approach, where a very fast front propagation algorithm evolves the initials curve toward their global minima. Very promising experimental results are provided using real video sequences. 1 Introduction The tracking problem has a wide variety of applications in computer vision and motion analysis, including coding, video surveillance and robotics. Additionally, it provides a good basis for high level tasks of computer vision like 3-D reconstruction and 3-D representation. This paper addresses the problem of tracking several non-rigid objects over a sequence of frames acquired from a static observer using boundary-based and regionbased motion detection/segmentation information. During the last years, a great variety of tracking algorithms have been proposed. These may be classied in two distinct categories: { Motion-based approaches rely on a robust method for grouping visual motion consistencies over time [1]. These methods are quite fast but they cannot ensure that the tracked regions have any meaning. { Model-based approaches impose high-level semantic representation and knowledge [8] and are more reliable compared with the motion-based approaches. On the other hand they suer from being computationally expensive due to the need to cope with scaling, translation, rotation, and deformation. In both cases the tracking is performed using measurements provided by geometrical or region-based properties of the tracked object. In this direction there are two main approaches: the boundary-based (they usually referred to as edge-based approaches) rely on the information provided by the object boundaries (shape properties) [16] while the region-based approaches rely on information provided by the entire region [2, 11] (texture and motion-based properties). The idea of using boundary-based features to provide tracking has been widely adopted. The boundarybased features (edges) are well adapted to the tracking problem since they provide reliable information which is not dependent on motion type, as well as on the object shape. Usually the boundary-based tracking? This work was funded in part under the VIRGO research network (EC Contract No ERBFMRX-CT96-49) of the TMR Programme

2 algorithms employ active contour models, like snakes [7] or geodesic active contours [4]. Both models are energy-minimized approaches that evolve from an initial curve under the inuence of external potentials, while being constrained by internal energies. These models are quite exible due to the fact that no restriction is imposed on the shape of the target nor on the type of motion: rigid or deformable [3, 9, 16]. These methods require a good initialization step, since the initial contour converges iteratively toward the solution of a partial dierential equation. Region-based methods use a motion estimation/segmentation technique. In this case the estimation of the velocity of the target is based on a correspondence between the associated target regions at dierent time instants [11]. This operation is time consuming. Usually, parametric models are used to describe the target motion with a limited set of parameters. As a consequence, these models have diculty in tracking the target boundaries successfully, especially in the case of non-rigid targets. On the other hand these methods are robust due to the fact that they make use of information provided by the whole region and do not depend (strongly) on the initialization step. The main goal of the proposed work is to provide a general framework that incorporates boundary-based and region-based tracking modules. The use of geodesic active contours is well adapted to our problem due to the fact that it demands less computational cost than the region-based approaches, and it provides an accurate extraction of the target boundaries. We extend these contours to incorporate region-based features that will increase robustness. The region-based features are estimated from a motion detection/segmentation module. To avoid the optical ow estimation, the motion detection/segmentation is based on statistical analysis. We assume that the dierence frame (between the current and the background reference frame) is composed of two populations, one that corresponds to the static part (static background) and one that corresponds to the mobile part (moving targets). Based on this hypothesis we can model the observed probability density function of the dierence frame using a two-component mixture distribution of conditional probability density functions. The rst corresponds to the static part and is zero-mean, while the second corresponds to the mobile part and is a multi-component density function due to the fact that dierent moving objects provide dierent statistical behavior (in terms of the dierence frame). The boundary-based information is expressed via discontinuities on the motion detection features space, while the region-based is expressed via conditional probabilities. We incorporate the two information sources to a coupled geodesic active contour model which leads to a system where the two modules operate simultaneously, namely a Geodesic Active Region model [15]. The objective function is minimized using a gradient descend method where smoothing, boundary-based and region-based forces move curve toward the moving object. The partial dierential equation that deforms the initial curve is implemented using a Level-Set approach where topological changes are naturally handled [12], thereby allowing multiple moving objects to be pursued simultaneously. Finally, the level-set propagation is performed using a very fast front propagation algorithm [14]. Various experimental results demonstrate the performance of our approach. The remainder of this paper is organized as follows. In section 2 we introduce the geodesic Active Region model for tracking. This section is divided in two parts. First, we provide the statistical analysis that is required for the motion detection/segmentation module. Additionally, in this sub-section we propose a new method to determine the object boundaries, as discontinuities of the motion detection/segmentation space. In the second part of this sections we propose the geodesic active region objective function that incorporates boundary-based and region-based features in an active contour model. The minimization of the objective function is described in section 3 where a Level-Set approach and a fast front propagation algorithm (Hermes) are employed. Finally, section 4 contains some experimental results, followed by conclusions and discussion. 2 Geodesic Active Regions 2.1 Motion Detection/Segmentation Module Let I(x; y; t) be the current frame, R(x; y) be the background reference frame, and D(x; y; t) = I(x; y; t)? R(x; y) the current dierence frame. If we assume that this frame is a selection of independent points, then it is composed of two populations. The Static one contains the background points while the Mobile one contains

3 the points that belong to moving objects and usually preserves dierent illumination properties with respect to the corresponding background illumination properties. Additionally, we could assume that the Mobile population can be decomposed into many dierent sub-populations with respect to the dierent intensity properties preserved by the moving objects. These assumptions can be easily projected to a statistical model, where the observed density function with respect to the dierence values can be decomposed into two main statistical components, the static one and the mobile one p D (d) = P Static p(djstatic) + P Mobile p(djmobile) where P static (resp. P Mobile ) is the a priori probability for the Static (resp. Mobile) case. Additionally, we can consider that the conditional probability density function with respect to the Mobile component is a collection of sub-components that express the dierent illumination properties of the observed objects (in terms of the dierence frame). Thus we can assume that a mixture density can be used to model the statistical behavior of the Mobile component, which is given by p(djmobile) = C N X i= P i p(dji; Mobile) where P i is the a priori probability of the i component. Finally, we assume that these probability density functions follow Gaussian law, 1 p(dja) = p e? (x? A )2 2 2 A 2A As for the unknown parameters of this model, some constraints are imposed by the problem. It is quite obvious that dierences between background values appear because of the presence of white noise, and as a consequence, the conditional probability density function with respect to the static case is zero-mean. Additionally, we can assume that the mobile mixture model contains a zero-mean Gaussian density function due to the fact that some moving objects may preserve similar intensity properties with respect to the background. The estimation of the unknown parameters of this model (P static ; P mobile ; Static ; ; fp i ; i ; i g : i = 1; :::; C N ) is done using a the maximum likelihood principle [6]. 2.2 Geodesic Active Regions for Tracking { Geodesic Active Regions and boundary-based information An obvious way to provide the boundary-based information is to seek high gradient values in the current or in the dierence frame. The use of an edge-based detector on the input frame could provide reliable boundary-based information for the moving objects, but it will provide similar information for the high contrast parts of the static background. On the other hand the use of the dierence frame will eliminate the static background edges but it will provide high boundary information inside the moving objects due to the fact that their intensities are not homogeneous. In order to deal with these problems we propose an alternative method for the extraction of boundarybased information. Let (x; y) be a point of the image, and let N L (x; y); N R (x; y) be a partition of (x; y) neighborhood. If (x; y) is a boundary point (of a moving object), then there is a partition of its neighborhood that creates two regions with quite dierent statistical properties. One is composed of points which come from the moving object while the second is composed of background points (we consider the boundaries between the moving targets and the static background only). These regions should have dierent behaviors in the probability feature space. The rst should present high probability with respect to the static hypothesis, while the second should present high probability with respect to the mobile hypothesis. Let h(x; y) be the function that expresses the boundary-based information. This function can be dened as a normalized dissimilarity measure between the statistical behaviors of the dierent neighborhood regions with respect to dierent hypothesis, thus we could write h(x; y) = p(d(n L(x; y))jstatic)p(d(n R (x; y))jmobile) + p(d(n L (x; y))jstatic)p(d(n R (x; y))jmobile) p(d(n L (x; y))jstatic)p(d(n R (x; y))jstatic) + p(d(n L (x; y))jmobile)p(d(n R (x; y))jmobile) where D(N L (x; y)) (resp. D(N R (x; y))) is the mean value of observed dierence data over the region N L (x; y) (resp. N R (x; y)). The interpretation of the above function is quite obvious. If (x; y) is a boundary point, then

4 there is a partition of its neighborhood that provides high static probability for the \left" region and high mobile probability for the \right" region. For the possible neighborhood partitions we consider the for main directions (vertical, horizontal, and the two diagonals). Since the statistical analysis has been performed, we try to reformulate the tracking problem as a frame partition problem, since we would like to incorporate region-based features which can be easily expressed within this framework. In other words we would like to create a partition of R into N subregions fr i g, i = ; 1; :::; N, i.e R = [ N R i= i; R i \ R j = ;, if i 6= j, where R corresponds to the static part of the frame (background points), while fr i g, i = 1; :::; N corresponds to dierent moving objects. i be the boundary of region R i and = [ N i be the boundaries of the dierent moving objects. Based on work developed in [4] we reformulate the problem of tracking within the framework of curve evolution theory. Let E(@R) be the objective function to be minimized. Based on the classic denition of the geodesic active contours and taking into account the fact that we are looking for the boundaries of moving objects we can rewrite the snake equation as E(@R) = NX i=1 Z 1 j@ R _ i (p i )j 2 dp i + internal Z 1 g 2 (h(@r i (p i )))dp i external i (p i ) : [; 1]! R 2 is a parameterization of the region boundary R i in a planar form, and f; g are real positive constants. The internal energy component accounts for the expected spatial properties (i.e. smoothness) of the contour while the external energy component stands for the attraction energy term of the curve towards the objects contour (high gradient values). Finally, g is a monotonically decreasing function such that g(r)! as r! inf and g() = 1 (in our case g(r) = e?r ). Following the work of [1, 4], it can be proved that the minimization of (1) leads to a geodesic curve with a new metric, where is a positive constant. E(@R) = { Geodesic Active Regions and region-based information NX i=1 Z 1 (1) g(h(@r i (p i )))j@ _R i (p i )jdp i (2) The main goal of region-based tracking methods is to create a correspondence between the same region at dierent instants with respect to the region properties. In our case these properties are expressed via conditional probability density functions with respect to motion detection/segmentation measurements. A moving object is well tracked if the corresponding curve at dierent time instants refers to the same region. Additionally, since such a region is a mobile region it should provide high a posteriori segmentation probability with respect to the mobile case. On the other hand, the background region should provide high a posteriori segmentation probability with respect to the static case. Guided by the way of dening similar objective functions for Markov Random Fields, our objective function is dened as: E(R) = NX Z Z where is a negative weight constant and p Ri (:) is given by: i= R i log [p Ri (D(x; y))] dxdy (3) p Ri (D(x; y)) = p (D(x; y)jstatic), if i = p (D(x; y)jmobile), if i 6= (4) Motivated by the excellent work proposed in [5, 17], and following our previous work on tracking using geodesic active contours [13, 16], tracking using classical snakes and region information [2], and the incorporation of boundary-based and region-based features in a common framework [15], we fuse the two dierent models, by dening a Geodesic Active Region Tracking model: E(@R) =

5 NX Z 1 g(h(@r i (p i )))j@ _R i (p i )jdp i + i=1 Boundary?based information NX Z Z i= Region?based motion information R i log [p Ri (D(x; y))] dxdy The above equation has a simple interpretation. The tracking is obtained by minimizing two kind of \energy terms" for each moving object. The rst, is boundary-based and gives the best geodesic minimal-length smoothed curve, while the second is a region-based and maximizes the a posteriori segmentation probability with respect to the mobile case inside the region dened by the curve. Additionally, we would like to maximize the a posteriori segmentation probability with respect to the static case for the static region (R ). (5) 3 Geodesic Active Regions: Implementation Issues The minimization of the objective function is obtained using a gradient descend method. Let u = (x; y) be a point of the initial curve. We compute the Euler-Lagrange equations [4, 18] and deform each point u of the initial curve towards the local minima of the objective function using the following equation: du dt = flog[p R (u) + log[p Rr (u)g + (u) + rg(h(u)) (u) + rg(h(u)) (u))g (u) (6) where r is the index of the region in which the point u belongs, is the Euclidean curvature with respect to the r and is the unit inward normal r. Since the r have inverse inward normals at each common point u, we have N = N Rr =?N R. Taking this into account and replacing by K, the motion equation for u can be rewritten as: K Rr du dt = prr (D(u)) [g(h(u))k(u) + rg(h(u)) N(u))] + log p R (D(u)) Boundary?based Forces 1 C A Region?based Forces N(u) (7) The obtained motion equation has two kind of \forces" acting on the contour, both in the direction of the inward normal. The rst one, is a boundary-based force that contains information regarding the boundaries of the dierent moving objects. At the same time it is a smoothing force, since its value depends on the curvature term. This force shrinks or expands the contour towards the object boundaries. The second term is a region-based motion detection/segmentation force. Suppose that u is a background pixel. In such a case the dierence frame value must support the static hypothesis R (p R (D(u)) > p Rr (D(u))). The inuence of region-based force is quite evident. In the case where the observed dierence frame at u ts better to h log h prr (Diff(u)) i i the static case p R > the tracking region is going to be compressed (the contour is (Diff(u)) located in the background), otherwise it will expanded (the contour is located inside the tracked object). This is a very nice property, since the contour can evolve either inwards, or outwards at the same time for dierent points. As a consequence, the contour initialization doesn't pose signicant problems. The contour initialization from frame to frame is performed based on the displacement of the gravity center of the object, which is estimated using a block matching gradient descent method. We suppose that this displacement is valid for the whole contour points set. This initialization is possible thanks to the region-based motion detection/segmentation forces that move the contour either inwards, or outwards according to the observed data set. 3.1 Level Set Implementation The key idea of the level-set framework [12] is to represent the moving as the zero-level set f = g of a function. This representation is implicit, parameter-free and intrinsic. Additionally, it is topology-free since dierent topologies of the zero level-set do not imply dierent topologies of. It easy to show, that if the moving front evolves according to t) = F (p)n dt

6 for a given function F, then the embedding function deforms according to d (p; t) = F (p) jr(p; t)j dt For this level-set representation, it has been proved that the solution is independent of the embedding function, and in our case is initialized as a signed distance function. Based on (5) and embedding (7) in we can say that the minimization of dened objective function can be done by searching for a steady-state solution of the following equation: d dt = [g(h(u))k(u) + rg(h(u)) r(u))] + log prr (D(u)) jrj (8) p R (D(u)) where t) is represented by a level-set of and the value of K is estimated on (K = div(r=jrj)). A direct implementation of equation (8) involves the re-estimation of the characteristic image of all the level set pixels (not simply the zero level set corresponding to the front itself). This front evolution method is computationally very expensive, due to many useless operations (especially in pixels which are out of interest). In order to overcome this drawback the \Hermes" method is used [14]. The main characteristic of Hermes algorithm is that it proposes a fast way to deform the initial curve towards the global minimum of the objective function. In our case the equation which deforms the initial curve (8) can be rewritten in a more general form as: t+1 = (x;y) t (x;y) + V(x; y; )dt (9) where V(x; y; ) is the speed function, depending on geometric-based features (curvature) and image-based features (\boundary-based" and \region-based" forces). Since the evolving speed V(x; y; ) is estimated according to image characteristics, there are some points for which the front evolves faster compared to the others. The key idea on which Hermes approach is based is to evolve the contour with respect to the speed values (in terms of absolute value) of a point at that point. The algorithm at each step selects the point with the highest speed from a set of actual contour points, and deforms the level-set frame locally. 4 Conclusions Very promising experimental results have been obtained using the proposed framework for video-surveillance sequences [g. (1,2)]. The main drawback of the model is the cases of occlusion. As a possible extension we could incorporate dierent region-based tracking modules. In that direction texture-based features, motionbased features (optical ow) as well geometric-based features (object representations) could be used to deal successfully with these situations. A coupled Geodesic Active Region model that incorporates dierent region-based features with the classical boundary-based features to provide tracking under various conditions is the future direction of our work. In this paper, we presented some new ideas concerning the integration of boundary-based and regionbased motion/detection segmentation approaches for tracking. The main contribution of this work consists of creating a powerful global tracking model, where dierent sources of information could be used under a common framework which provides a connection between the minimization of an objective function and a curve evolution approach. The contour propagation is obtained by integrating two dierent modules, a boundary-based module that causes the contour to shrink or expand towards the object boundaries and a region-based module that causes the contour to move towards the direction that maximizes the a posteriori segmentation probability for the interior (mobile hypothesis) and the exterior (static hypothesis) region. This leads to a system where the two modules operate simultaneously, where the contour propagation is guided by smoothing, boundary-based, and region-based forces. The changes of topology can be easily achieved using a Level-Set approach. Thus several moving objects can be tracked simultaneously. Finally the tracking results are obtained quite fast thanks to a \smart" front propagation algorithm. Various experimental results (in MPEG format), including the ones shown in this article, can be found at: Acknowledgments: The rst author would like to thank Tom O'Donnell from Siemens Corporate Research for reviewing the nal draft of this paper.

7 References Fig. 1.: Tracking for Highway Sequence (white line, left to right and top to down) 1. G. Aubert and L. Blanc-Feraud. An elementary proof of the equivalence between 2d and 3d classical snakes and geodesic active contours. Technical Report 334, INRIA, France, B. Bascle and R. Deriche. Region tracking through image sequences. In proc. of International Conference on Computer Vision, pages 32{37, Puerto Rico, USA, V. Caselles and B. Coll. Snakes in Movement. SIAM Journal on Numerical Analysis, 33:2445{2456, V. Caselles, R. Kimmel, and G. Sapiro. Geodesic active contours. International Journal of Computer Vision, 22:61{79, A. Chakraborty, H. Staib, and J. Duncan. Deformable Boundary Finding in Medical Images by Integrating Gradient and Region Information. IEEE trans.medical Imaging, 15(6):859{87, R. Duda and P. Hart. Pattern Classication and Scene Analysis. John Wiley & Sons, Inc., M. Kass, A. Witkin, and D. Terzopoulos. SNAKES: Active contour models. International Journal of Computer Vision, 1:321{332, D. Koller, K. Daniilidis, and H-H Nagel. Model-based object tracking in monocular image sequences of road trac scenes. International Journal of Computer Vision, 1:257{281, F. Leymarie and M. Levine. Tracking deformable objects in the plane using an active contour model. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33:617{634, F. Meyer and P. Bouthemy. Region-based tracking using ane motion models in long image sequences. CVGIP: Image Understanding, 6:119{14, J-M. Odobez and P. Bouthemy. Robust multiresolution estimation of parametric motion models. Journal of Visual Communication and Image Representation, 6:348{365, S. Osher and J. Sethian. Fronts propagating with curvature dependent speed : algorithms based on the Hamilton- Jacobi formulation. Journal of Computational Physics, 79:12{49, 1988.

8 Fig. 2.: Tracking for Pierre-Imed Sequence (black line, left to right and top to down) 13. N. Paragios and R. Deriche. Detecting multiple moving targets using deformable contours. In IEEE International Concerence on Image Processing, volume II of III, pages 183{186, Santa-Barbara,California, N. Paragios and R. Deriche. A PDE-based Level Set Approach for Detection and Tracking of Moving Objects. Technical Report 3173, INRIA, France, N. Paragios and R. Deriche. Geodesic Active Regions for Texture Segmentation. Technical Report 344, INRIA, France, N. Paragios and R. Deriche. A PDE-based Level Set approach for Detection and Tracking of moving objects. In proc. of International Conference on Computer Vision, pages xxx{xxx, Bombai, India, S. Zhu, T. Lee, and A. Yuille. Region Competition: Unifying Snakes, Region Growing Energy/Bayes/MDL for Multi-band Image Segmentation. In proc. of International Conference on Computer Vision, Boston, USA, S. Zhu and A. Yuille. Region Competition: Unifying Snakes, Region Growing, and Bayes/MDL for Multiband Image Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 18:884{9, 1996.