Tracking Deforming Objects Using Particle Filtering for Geometric Active Contours

Transcription

1 1470 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 29, NO. 8, AUGUST 2007 Tracking Deforming Objecs Using Paricle Filering for Geomeric Acive Conours Yogesh Rahi, Member, IEEE, NamraaVaswani, Allen Tannenbaum, and Anhony Yezzi Absrac Tracking deforming objecs involves esimaing he global moion of he objec and is local deformaions as a funcion of ime. Tracking algorihms using Kalman filers or paricle filers have been proposed for finie dimensional represenaions of shape, bu hese are dependen on he chosen paramerizaion and canno handle changes in curve opology. Geomeric acive conours provide a framework which is paramerizaion independen and allow for changes in opology. In he presen work, we formulae a paricle filering algorihm in he geomeric acive conour framework ha can be used for racking moving and deforming objecs. To he bes of our knowledge, his is he firs aemp o implemen an approximae paricle filering algorihm for racking on a (heoreically) infinie dimensional sae space. Index Terms Tracking, paricle filers, geomeric acive conours. 1 INTRODUCTION Ç THE problem of racking moving and deforming objecs has been a opic of subsanial research in he field of acive vision; see [1], [2], [3] and he references herein. In his paper, we propose a scheme which combines he advanages of paricle filering and geomeric acive conours realized via level se models for racking deformable objecs. The possible parameerizaions of shape are of course very imporan. Various finie dimensional parameerizaions of coninuous curves have been proposed, perhaps mos prominenly he B- spline represenaion used for a snake model as in [2]. Isard and Blake (see [1] and he references herein) use he B-spline represenaion for conours of objecs and propose he CONDEN- SATION algorihm [4] which reas he affine group parameers as he sae vecor, learns a prior dynamical model for hem and uses a paricle filer [5] o esimae hem from he noisy observaions. Since his approach only racks he affine parameers, i canno handle local deformaions of he deforming objec. The approach in [2], [6], [7] uses a Kalman filer in conjuncion wih acive conours (using marker paricle represenaion of curves) o rack nonrigid objecs. Anoher approach for represening conours is via he level se mehod [8], [9] where he conour is represened as he zero level se of a higher dimensional funcion, usually he signed disance funcion [8], [9]. For segmening an objec, an iniial guess of he conour (represened using he level se funcion) is deformed unil i minimizes an image-based energy funcional. Mos level se mehods rack by segmening he objec a each frame and do no uilize he emporal coherency of he deforming objec. As a resul, such mehods fail o rack large changes in he spaial locaion (rigid moion) of he objec. Some previous work on racking using level. Y. Rahi and A. Tannenbaum are wih he School of Elecrical and Compuer Engineering, Georgia Insiue of Technology, VL E392B, Alana, GA {Yogesh.rahi, annenba}@gaech.edu.. N. Vaswani is wih he Deparmen of Elecrical and Compuer Engineering, Iowa Sae Universiy, Ames, IA Namraa@iasae.edu.. A. Yezzi is wih he School of Elecrical and Compuer Engineering, Georgia Insiue of Technology, VL E370, Alana, GA ayezzi@ece.gaech.edu. Manuscrip received 18 Aug. 2005; revised 7 Apr. 2006; acceped 28 Nov. 2006; published online 18 Jan Recommended for accepance by D. Comaniciu. For informaion on obaining reprins of his aricle, please send o: pami@compuer.org, and reference IEEECS Log Number TPAMI Digial Objec Idenifier no /TPAMI se mehods is given in [10], [11], [12], [13], [14], [15], [16], [17], [18]. Mos of hese works formulae conour racking as he problem of compuing he MAP esimae of he conour using a Bayesian formulaion (wih an image likelihood energy and a prior erm). In [16], [14], he prior is only a smoohness prior while in [10], i is a disance from a finie se of possible conour exemplars. The work of [15] uses a shape energy erm only when occlusion is deeced. In [11], [17], he objec deecion sep a each ime is separaed from he racking sep. There is, of course, a huge lieraure devoed o visual racking and, hus, he work sampled above is by no means exhausive. The work in his paper addresses he limiaions of he CONDENSATION algorihm [1] and level se based mehods and exends on he ideas presened in [12], [13]. More precisely, in [12], he auhors rack by performing a join minimizaion over a group acion (euclidean or affine) and he conour a each ime sep, which is compuaionally very inensive. Also, for nonlinear sysems such as he one used in [13], here is no sysemaic way o choose he observer marix o guaranee sabiliy. The presen paper addresses he above limiaions. We formalize he incorporaion of a prior sysem model along wih an observaion model. A paricle filer is used o esimae he condiional probabiliy disribuion of he group acion and he conour a ime, condiioned on all observaions up o ime. Thus, his work presens a novel mehod o perform filering on an infinie dimensional space of curves for he purpose of racking deforming objecs. Finally, a conference version of his paper has appeared in [19]. Our conribuion in his work is he following hree modificaions o he sandard paricle filer (PF) [5], [20]: 1) We propose o use an imporance sampling (IS) densiy [20] which can be undersood as an approximaion o he opimal IS densiy when he opimal densiy is mulimodal. 2) We replace IS by deerminisic assignmen when he variance of he IS densiy is very small (happens when local deformaion is small). Because of his sep, we are acually only sampling on he six-dimensional space of affine deformaions, while approximaing local deformaion by he mode of is poserior. This is wha makes he proposed PF algorihm pracically implemenable in real ime. The full space of conour deformaions is heoreically infinie. In pracice, is dimension is beween , even for he small sized images shown in he resuls. 3) In addiion, we also discuss an efficien way o compue an approximaion o he mode of he poserior of local deformaion. As explained in [21], hese modificaions are useful o reduce compuaional complexiy of any large dimensional sae racking problem. This paper is organized as follows: In Secion 2, we provide a brief overview of he proposed algorihm and in Secion 3 we provide all he relevan deails. Experimenal resuls are given in Secion 4, while we conclude he paper wih a summary and limiaions in Secion 5. 2 THE PROPOSED ALGORITHM This secion describes he overall framework of he proposed mehod wih deails given in he remainder of he paper. Le C denoe he conour a ime (C is represened as he zero level se of a signed disance funcion, ðxþ, i.e., C ¼fx 2R 2 : ðxþ ¼0g [8]) and A denoe a six-dimensional affine parameer vecor wih he firs four parameers represening roaion, skew and scale, respecively, and he las wo parameers represening ranslaion. We propose o use he affine parameers ða Þ and he conour ðc Þ as he sae, i.e., X ¼½A ;C Š and rea he image a ime as he observaion, i.e., Y ¼ ImageðÞ. Denoe by Y 1: all he observaions unil ime. Paricle filering [5] allows for recursively esimaing pðx jy 1: Þ, he poserior disribuion of he sae given he prior pðx 1 jy 1: 1 Þ. We will employ he basic heory of paricle filering here as described in [5]. The general idea behind he proposed algorihm is as follows:. Imporance Sampling. Predic he affine parameers A (parameers governing he rigid moion of he objec) and /07/$25.00 ß 2007 IEEE Published by he IEEE Compuer Sociey

2 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 29, NO. 8, AUGUST perform imporance sampling for C o obain local deformaion in shape, i.e., - Generae samples fa ðiþ A ðiþ ¼ f p A ðiþ 1 ;uðiþ ; ðiþ g N i¼1 using ; ðiþ ¼ A ðiþ C ðiþ 1 : - Perform L seps of curve evoluion on each ðiþ 1 C ðiþ ¼ f CE ðiþ ;Y ;u ðiþ ;def ; u ðiþ ;def Nð0; defþ:. Weighing and Resampling. Calculae he imporance weighs and normalize [5], i.e., ~ p Y jx ðiþ p X ðiþ jx ðiþ E image ðy ;C ðiþ Þ d 2 ðc ðiþ ; ðiþ Þ 1 ¼ / e 2 obs e 2 d ; q X ðiþ jx ðiþ N f CE ðiþ ;Y ; def ¼ ~ P N j¼1 ~wðjþ ; 1 ;Y where d 2 is any disance meric beween shapes (see Secion 3.5) and E image is any image based energy funcional (see Secion 3.3). Resample o generae N paricles fa ðiþ ;C ðiþ g disribued according o pða ;C jy 1: Þ. The resampling sep improves sampling efficiency by eliminaing paricles wih very low weighs. We now explain in deail each of he seps above. 3 THE SYSTEM AND OBSERVATION MODEL The problem of racking deforming objecs can be separaed ino wo pars [13]: 1) Tracking he global rigid moion of he objec and 2) Tracking local deformaions in he shape of he objec, which can be defined as any deparure from rigidiy (nonaffine deformaions). The global moion (affine ransformaion) can be modeled by he six parameers of an affine ransformaion, A, using a firs order Markov process. We assume ha he local deformaion from one frame o he nex is small and can be modeled by deformaion in he shape of he conour C. Thus, he sae vecor is given by X ¼½A C Š. The sysem dynamics based on he above assumpion can be wrien as A ¼ f p A 1 þ u ; u Nð0; A Þ; ^x ¼ A ;1 A ;2 x þ A ;5 ; 8x 2 C 1 ; ^x 2 ; i:e:; ¼ 4 A ðc 1 Þ; A ;3 A ;4 A ;6 ð1þ C ¼ f def ð ;u ;def Þ; u ;def simnð0; def Þ; where f p models global rigid moion of he objec while f def is a funcion ha models he local shape deformaion of he conour. We furher assume ha he likelihood probabiliy, i.e., probabiliy of he observaion Y ¼ ImageðÞ given sae X, is defined by E image ðc ;Y Þ pðy jx Þ¼pðY jc Þ/e ; where E image is any image dependen energy funcional and is a parameer ha deermines he shape of he pdf (probabiliy densiy funcion). The normalizaion consan in he above definiion has been ignored since i only affecs he scale and no he shape of he resuling pdf. In general, i is no easy o predic he shape of he conour a ime (unless he shape deformaions are learned a priori) given he previous sae of he conour a ime 1, i.e., i is no easy o 1. One can also perform L seps of sochasic curve evoluion as in [22]. find a good funcion f def ha can model he shape deformaions and allows o sample from an infinie (heoreically) dimensional space of curves. Thus, i is very difficul o draw samples for C from he prior disribuion. This problem can be solved by doing imporance sampling [23] and is one of he main moivaions for doing curve evoluion as explained in he following secions. Thus, samples for A can be obained by sampling from Nðf p A 1 ; A Þ while samples for C are obained using imporance sampling, i.e., we perform imporance sampling only on par of he sae space. This echnique of using imporance sampling allows for obaining samples for C using he laes observaion (image) a ime [24]. The cenral idea behind imporance sampling [23] is as follows: Suppose pðxþ /qðxþ is a probabiliy densiy from which i is difficul o draw samples and qðxþ is a densiy (proposal densiy or imporance densiy) which is easy o sample from, hen, an approximaion o pðþ is given by pðxþ P N i¼1 wi ðx x i Þ, where is he normalized weigh of he ih paricle. So, if he samples, X ðiþ, were drawn from an imporance densiy, qðx jx 1: 1 ;Y 1: Þ, and weighed by p X ðiþ jy 1: / ; q X ðiþ jx ðiþ w i / pðxi Þ qðx i Þ 1: 1 ;Y 1: hen P N i¼1 wðiþ X Þ approximaes pðx jy 1: Þ. In his work, he sae is assumed o be a hidden Markov process, i.e., pðx jx 1: 1 Þ¼pðX jx 1 Þ; and we furher assume ha he observaions are condiionally independen given he curren sae, i.e., pðy jx 1: Þ¼pðY jx Þ. Furhermore, if he imporance sampling densiy is assumed o depend only on he previous sae X 1 and curren observaion Y, we ge qðx jx 1: 1 ;Y 1: Þ¼qðX jx 1 ;Y Þ. This gives he following recursion for he weighs [23]: p Y jx ðiþ ¼ p X ðiþ jx ðiþ 1 1 : q X ðiþ jx ðiþ ðx ðiþ 1 ;Y The imporance densiy qð:þ and he prior densiy pð:þ can now be wrien as 2 qðx jx 1 ;Y Þ¼pðA ja 1 Þ qðc j ;Y Þ; ð2þ pðx jx 1 Þ¼pðA ja 1 Þ pðc j Þ; where qða ja 1 Þ¼pðA ja 1 Þ, since A is sampled from pða ja 1 Þ¼ Nðf p A 1 ; A Þ. Thus, he weighs can be calculaed from: p Y jx ðiþ ¼ p C ðiþ j ðiþ 1 : ð3þ q C ðiþ j ðiþ ;Y The probabiliy pðc j Þ can be calculaed using any suiable measure of similariy beween shapes (modulo a rigid ransformaion). One such measure is o ake pðc j Þ/e d 2 ðc ; Þ 2 d ; where d is assumed o be very small such ha i saisfies he consrain of (10) in [21] and d 2 is any meric on he space of closed curves. In his work, we have used he disance measure given in Secion Noe ha he curve obained afer doing curve evoluion is denoed by C, while he curve obained by applying he affine ransformaion is denoed by, i.e., ¼ A ðc 1 Þ.

3 1472 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 29, NO. 8, AUGUST Approximaing he Opimal Imporance Densiy The choice of he imporance densiy is a criical design issue for implemening a successful paricle filer. As described in [25], he proposal disribuion qðþ should be such ha paricles generaed by i, lie in he regions of high observaion likelihood. One way of doing his is o use a proposal densiy which depends on he curren observaion [24]. In [25], he opimal imporance densiy (one ha minimizes he variance of he weighs condiioned on X 1 and Y ) has been shown o be pðx jx 1 ;Y Þ. Bu, in many cases, i canno be compued in closed form. For unimodal poseriors, i can be approximaed by a Gaussian wih mean given by is mode [25], which is also equal o he mode of pðy jx Þ pðx jx 1 Þ. In our case, he disribuion pða ja 1 Þ can be mulimodal, hus, he formulaion of [25] canno be direcly used. Hence, we propose o use he following: Sample A from he prior sae ransiion kernel, pða ja 1 Þ, and find he mode of pðy jx Þ pðc j Þ o obain samples for C. Noice ha, for small deformaions, pðy jx Þ pðc j Þ is indeed unimodal [21]. Using (2) and he likelihood probabiliy pðy jx Þ defined before, finding he mode of pðy jx Þ pðc j Þ is equivalen o finding he minimizer of E o ðc ; ;Y Þ¼ E imageðc ;Y Þ þ d2 ðc ; Þ 2 : d Noice ha, from his energy poin of view, i is clear why we can ignore he pariion consans (in he definiion of pðy jc Þ and pðc j Þ) which are needed o normalize he various densiies so ha hey define proper probabiliy measures. Indeed, all we are ineresed in is he minimizer of E o. This observaion has also been made in various oher works including [26], [27]. Finding he exac minimizer of E o for each paricle a each is compuaionally expensive and hence we use he following approximaion: Assuming a small deformaion beween 1 and, boh he erms in his summaion will be locally convex (in he neighborhood of he minimizers of boh erms) and, so, he minimizer of he sum will lie beween he individual minimizers of each erm. Thus, an approximae soluion o find he minimum of E o will be o sar from he minimizer of one erm and go a cerain disance (i.e., a cerain number of ieraions of gradien descen) oward he minimizer of he second. I is easy o see ha C ¼ minimizes he second erm and, hence, saring wih as he iniial guess for C, and performing L ieraions of gradien descen will move C a given disance oward he minimizer of E image, where L is chosen experimenally. We would like o reierae here ha he opimal choice of L will be one ha finds a curve C o minimize E o, bu o avoid performing he complee minimizaion of E o, we are doing his approximaion, and have found ha i works well in pracice. Using he above echnique, we are acually only sampling on he sixdimensional space of affine deformaions, while approximaing local deformaion by he mode of is poserior. The full space of conour deformaions has dimension around even for he size of images shown in he resuls. Sampling on such a high-dimensional space for each paricle canno be done in anyhing close o real ime. However, he mode racker mehod described above reduces he compuaions significanly. 3.2 Curve Evoluion for Compuing C We now describe how o obain samples for C by doing gradien descen on he energy funcional E image. In wha follows, his operaion is represened by he funcion f CE. The non-linear funcion f CE ð; Y ; u def Þ is evaluaed as follows (for k ¼ 1; 2;...;L): 0 ¼ ; ð4þ k ¼ k 1 k r E image k 1 ;Y;u def ;fce ; Y ; u def ¼ L : The above equaion is basically a PDE which moves an iniial guess of he conour so ha E image is minimized. u def Nð0; def Þ is a noise vecor ha is added o he velociy of he deforming Fig. 1. Likelihood probabiliy disribuion (a) wih (b) wihou using imporance densiy qð:þ for frame 2 of car sequence (200 paricles). conour a each poin x 2 (see [8], [9], [22] for deails on how o evolve a conour using level se represenaion). For pracical examples wih small deformaions, def is very small and, in fac, even when one does no add any noise o f CE, here is no noiceable change in performance. In numerical experimens, we have no added any noise o he curve evoluion process. Thus, he imporance sampling densiy for A is pða ja 1 Þ, while ha for C is qðc j ;Y Þ¼Nðf CE ð ;Y Þ; def! 0Þ. The curve C, hus obained incorporaes he predicion for global moion and local shape deformaion An Alernaive Inerpreaion for L-Ieraion Gradien Descen We perform only L ieraions of gradien descen since we do no wan o evolve he curve unil i reaches a minimum of he energy, E image. Evolving o he local minimizer is no desirable since he minimizer would be independen of all saring conours in is domain of aracion and would only depend on he observaion, Y. Thus, he sae a ime would loose is dependence on he sae a ime 1 and his may cause loss of rack in cases where he observaion is bad. In effec, choosing L o be oo large (aking he curve very close o he minimizer) can move all he samples oo close o he curren observaion and, hus, resul in reducion of he variance of he samples leading o sample degeneracy. A he same ime, if L is chosen o be oo small, he paricles will no be moved o he region of high observaion likelihood and his can lead o sample impoverishmen. The choice of L depends on how much one russ he sysem model versus he obained measuremens. Noe ha, L will of course also depend on he sep-size of he gradien descen algorihm as well as he ype of PDE used in he curve evoluion equaion. Fig. 1 shows he hisogram of he likelihood probabiliy of he paricles wih and wihou using he imporance densiy. As can be seen, more paricles are moved o he region of high likelihood if he imporance disribuion qðþ is used. Based on he above discussion, he imporance weighs in (3) can be calculaed as follows: ¼ 1 p Y jx ðiþ p C ðiþ j ðiþ / q C ðiþ j ðiþ 1 ;Y E image C ðiþ / 1 exp ;Y d A exp@ ð E image C ðiþ ;Y e N f CE ðiþ C ðiþ 2 d Þ d 2 ðc ðiþ ; ðiþ Þ e 2 d ;Y ; def 1 A; ; ðiþ ð5þ where we have used he fac ha C ðiþ is he mean and def is very close o zero, implying ha NðC ðiþ ; def! 0Þ can be approximaed by a consan for all paricles. 3.3 Curve Evoluion Using he Chan-Vese Model Many mehods (see, e.g., [28], [10], [29], [30]) have been proposed which incorporae geomeric and/or phoomeric (color, exure, inensiy) informaion in order o segmen images robusly in presence of noise and cluer. In our case, in he predicion sep

4 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 29, NO. 8, AUGUST E ðiþ image p Y jx ðiþ e! obs / 1 B E A þ 2 1 d2 ðsþ ; ðiþ P N ; ð7þ P N j¼1 e j¼1 d2 ð ðsþ ; ðjþ Þ above, f CE can be any edge-based or region-based curve evoluion equaion (one can use [10] or [16] o rack exured objecs). In his work, he Mumford-Shah funcional [31] as modeled by Chan and Vese is used [32] o obain he curve evoluion equaion as follows: One applies he calculus of variaions o minimize he following energy E image : E image ¼ ði c 1 Þ 2 HðÞdx dy þ ði c 2 Þ 2 ð1 HðÞÞ dx dy ð6þ þ jrhðþjdx dy; where c 1, c 2 and he Heaviside funcion HðÞ are defined as R Iðx; yþhðþdx dy c 1 ¼ R ; HðÞdx dy R Iðx; yþð1 HðÞÞdx dy c 2 ¼ R ; HðÞ ¼ 1 0; ð1 HðÞÞdx dy 0 else; and, finally, Iðx; yþ is he image and is he level se funcion. The energy E image can be minimized by doing gradien descen via he following PDE ¼ r ðþ div ði c 1 Þ 2 þði c 2 Þ 2 ; where jrj ðsþ ¼ ð 2 þ s 2 Þ ; where is he evoluion ime parameer and he conour C is he zero level se of (see [8], [9] for deails). We should specify ha we have chosen he Chan-Vese funcional because of ease of implemenaion, and because i gave nice resuls on he image sequences o which i was applied. However, any geomeric curve evoluion procedure for segmenaion may be pu ino our paricle filer framework. 3.4 Dealing wih Muliple Objecs In principle, he CONDENSATION filer [1] could be used for racking muliple objecs. The poserior disribuion will be mulimodal wih each mode corresponding o one objec. However, in pracice i is very likely ha a peak corresponding o he dominan likelihood value will increasingly dominae over all oher peaks when he esimaion progresses over ime. In oher words, a dominan peak is esablished if some objecs obain larger likelihood values more frequenly. So, if he poserior is propagaed wih fixed number of samples, evenually, all samples will be around he dominan peak. This problem becomes more pronounced in cases where he objecs being racked do no have similar phoomeric or geomeric properies. We deal wih his issue as given in [33] by firs finding he clusers wihin he sae densiy o consruc a Voronoi essalaion [34] and hen resampling wihin each Voronoi cell separaely. Oher soluions proposed by [35], [36] could also be used for muliple objec racking. 3.5 Coping wih Occlusions A number of acive conour models [30], [29], [37] which use shape informaion have been described in he lieraure. Prior shape knowledge is necessary when dealing wih occlusions. In paricular, in [10], he auhors incorporae shape energy in he curve evoluion equaion o deal wih occlusions. Any such energy erm can be used in he proposed model o deal wih occlusions. In numerical experimens, we have deal wih his issue in a slighly differen way by incorporaing he shape informaion in he weighing sep insead of he curve evoluion sep, i.e., we calculae he likelihood probabiliy for each paricle i using he corresponding image energy E ðiþ image (6) and a shape dissimilariy measure d 2 as follows: ðjþ image where 1 þ 2 ¼ 1 and d 2 ð ðsþ ; ðiþ Þ is he dissimilariy measure (modulo a rigid ransformaion) as given in [37] by d 2 ð ðsþ ; ðiþ Þ¼ ð ðsþ ðiþ Þ 2 hð ðsþ Þþhð ðiþ Þ dx dy 2 wih HðÞ hðþ ¼ ; R HðÞ dx dy where ðsþ and ðiþ are he level se funcions of a emplae shape and he ih conour shape, respecively. The dissimilariy measure gives an esimae of how differen wo given shapes (in paricular, heir corresponding level ses) may be. So, higher values of d 2 indicaes more dissimilariy in shape. We use his sraegy for he following reason: In case of occlusion, E image will be higher for a conour ha encloses he desired region compared o a conour ha excludes he occlusion (see he car example, Fig. 3). Since paricle weighs are a funcion of E image, he MAP esimae will be a paricle ha is no he desired shape. However, using he weighing scheme proposed above, paricles which are closer o he emplae shape are more likely o be chosen han paricles wih occluded shapes (i.e., shapes which include he occlusion). Of course, his formulaion will only work if he objec being racked does no undergo large deformaions as is he case wih oher saic shape based echniques [10], [29], [37]. 4 EXPERIMENTS In his secion, we describe some experimens performed o es he proposed racking algorihm. We cerainly do no claim ha he mehod proposed in his paper is he bes one for every image sequence on which i was esed, bu i did give very good resuls wih a small number of paricles on all of he image sequences. We should add ha o he bes of our knowledge his is he firs ime geomeric acive conours in a level se framework have been used in conjuncion wih he paricle filer [5] for racking such deforming objecs. Resuls of applying he proposed mehod on four image sequences are given below. The model of Chan and Vese [32], as described earlier, was used for curve evoluion. In paricular, choosing L (he number of ieraions of curve evoluion) beween 3 and 6 gave accepable resuls. The level se implemenaion was done using narrow band evoluion [8]. Learning [1] was performed on images wihou he background cluer, i.e., on he oulines of he objec. 1. Van Sequence. In his video, we rack a van moving amid cluer in he background. There is sudden and large moion of he van (in some cases, he van moves more han 20 pixels beween consecuive frames) due o jier in he camera moion. Furhermore, i ges largely occluded (only a small fracion of he van is visible) many imes by a building or a ree. Tracking such a sequence using acive conours [32], [10] alone is bound o fail since he van may lie ouside he basin of aracion of he saring conour. The sandard CONDENSATION algorihm [1] may also ge suck on he srong edges of he building or on oher objecs in he background, especially when he van ges occluded. As shown in Fig. 2, he proposed mehod racks he van successfully despie large moion and occlusion. For his es sequence, no moion model was learned, i.e., he sae ransiion was given by A ¼ A 1 þ Bu, where u is whie

5 1474 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 29, NO. 8, AUGUST 2007 Fig. 2. Tracking he van sequence. Fig. 3. (a) Tracking resuls using Chan-Vese [32]. (b) Tracking using he proposed mehod. Fig. 4. Couple sequence: Demonsraes muliple objec racking. Fig. 5. Plane sequence: Tracking wih 30 paricles. Images have been cropped for beer visualizaion. Fig. 6. Plane sequence: Tracking wih condensaion filer using 1,200 paricles. Images have been cropped for beer visualizaion. Gaussian noise and B is a known covariance marix which is assumed o be consan hrough he sae evoluion process. Fig. 2 shows racking resuls wih 50 paricles. 2. Car Sequence. In his sequence, he car is parially occluded as i passes behind he lamp pos. I is unclear if he sandard CONDENSATION algorihm would be able o rack he car hrough he enire video since he shape of he car (including he shadow) undergoes a change which is no affine. Noice ha he shadow of he car moves in a nonlinear way from he side o he fron of he car. On he oher hand, rying o rack such a sequence using geomeric acive conours (for example, (7)) wihou any shape energy gives very poor resuls, as shown in Fig. 3. However, using he proposed mehod and a weighing sraegy as described in Secion 3.5, he car can be successfully racked (Fig. 3). The emplae shape ðsþ was obained from he firs frame of he sequence. Noe ha we used (7) for he curve evoluion which does no conain any shape erm. A second-order auoregressive model was used for f p. Resuls shown in his paper were obained wih 50 paricles. 3. Couple Sequence. The walking couple sequence demonsraes muliple objec racking. In general, racking such a sequence by he sandard CONDENSATION mehod [1] can give erroneous resuls when he wo pedesrians come very close o each oher or ouch each oher since he measuremens made for he person on he righ can be inerpreed by he algorihm as coming from he lef. Our mehod naurally avoids his problem since i uses region based energy E image (6) and weighing as given in Secion 3.5 o find he observaion probabiliies. To rack muliple objecs, we used he mehod described in Secion 3.4. Since he number of frames in he video is very small (only 22), no dynamical moion model was needed o be learned. This video demonsraes he fac ha he proposed algorihm can rack robusly (see Fig. 4) even when he learn model is compleely absen. The number of paricles required in his case was 100. Anoher soluion o racking his sequence has been proposed in [35]. 4. Plane Sequence. This sequence has a very low conras and in general, i is very difficul o locae he boundary of he plane. The moion of he plane from one frame o he oher is also quie large, hence radiional acive conour based mehods fail o rack he plane. In his experimen, only ranslaional moion was assumed for he moving plane. No moion model was learned and, hence, he sae ransiion equaion was as described in he previous example. Fig. 5 shows a few frames of he racking resuls. Even hough, no scale parameer was included in he moion model, he conour deformaion par of he algorihm adjuss for his change in size of he plane (see he firs and las frame). Oher ypes of affine changes in he shape are also aken care of wihin he proposed framework wihou having o explicily model hem. Tracking resuls were obained wih jus 30 paricles. Fig. 6 shows he resuls using he sandard CONDENSATION filer (wih 1,200 paricles) assuming a euclidean moion model. As is eviden, he filer fails o

6 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 29, NO. 8, AUGUST rack in many frames, especially when he edges are weak. I also fails o adjus for changes in scale. Our experimens show ha increasing he number of paricles o 2,000 or more does no change he resuls significanly. Tracking wih 30 paricles gives exremely bad resuls and he racker failed o rack in roughly 60 percen of he frames. 5 CONCLUSION AND LIMITATIONS In his paper, we proposed a paricle filering algorihm for geomeric acive conours which can be used for racking moving and deforming objecs. The proposed mehod can deal wih parial occlusions and can rack robusly even in he absence of a learn model. I also requires significanly fewer paricles han oher racking mehods based on paricle filers. Fas level se implemenaions [14] can be used o achieve near real-ime speeds. The above framework has several limiaions which we inend o overcome in our fuure work. Firs, we have o include some kind of shape informaion when we rack objecs which undergo major occlusions. This resrics our abiliy o rack highly deformable objecs in such siuaions. Second, he algorihm migh perform poorly if he objec being racked is compleely occluded for many frames. ACKNOWLEDGMENTS This research was suppored by grans from he US Naional Science Foundaion, he US Naional Insiues of Healh (NAC P41 RR hrough Brigham and Women s Hospial), he US Air Force Office of Scienific Research, ARO, MURI, MRI-HEL, and Technion-Israel Insiue of Technology. This work was done under he auspices of he Naional Alliance for Medical Image Compuing (NAMIC), funded by he US Naional Insiues of Healh hrough he NIH Roadmap for Medical Research, Gran U54 EB REFERENCES [1] Acive Conours, A. Blake and M. Isard, eds. Springer, [2] D. Terzopoulos and R. Szeliski, Tracking wih Kalman Snakes, Acive Vision, MIT Press, pp. 3-20, [3] D. Comaniciu, V. Ramesh, and P. Meer, Real-Time Tracking of Non-Rigid Objecs Using Mean Shif, Proc. Conf. Compuer Vision and Paern Recogniion, vol. 2, pp , [4] M. Isard and A. Blake, Condensaion Condiional Densiy Propagaion for Visual Tracking, In l J. Compuer Vision, vol. 29, no. 1, pp. 5-28, [5] N. Gordon, D. Salmond, and A. Smih, Novel Approach o Nonlinear/ Nongaussian Bayesian Sae Esimaion, IEE Proc. (Radar and Signal Processing), vol. 140, no. 2, pp , [6] N. Peerfreund, Robus Tracking of Posiion and Velociy wih Kalman Snakes, IEEE Trans. Paern Analysis and Machine Inelligence, vol. 21, no. 6, pp , June [7] N. Peerfreund, The Velociy Snake: Deformable Conour for Tracking in Spaio-Velociy Space, Compuer Vision and Image Undersanding, vol. 73, no. 3, pp , [8] J.A. Sehian, Level Se Mehods and Fas Marching Mehods. second ed. Cambridge Univ. Press, [9] S. Osher and R. Fedkiw, Level Se Mehods and Dynamic Implici Surfaces. Springer Verlag, [10] T. hang and D. Freedman, Tracking Objecs Using Densiy Maching and Shape Priors, Proc. Ninh IEEE In l Conf. Compuer Vision, pp , [11] N. Paragois and R. Deriche, Geodesic Acive Conorus and Level Ses for he Deecion and Tracking of Moving Objecs, IEEE Trans. Paern Analysis and Machine Inelligence, vol. 22, no. 3, pp , Mar [12] A. Yezzi and S. Soao, Deformoion: Deforming Moion, Shape Average and he Join Regisraion and Approximaion of Srucures in Images, In l J. Compuer Vision, vol. 53, no. 2, pp , [13] J. Jackson, A. Yezzi, and S. Soao, Tracking Deformable Moving Objecs under Severe Occlusions, Proc. IEEE Conf. Decision and Conrol, [14] Y. Shi and W. Karl, Real-Time Tracking Using Level Ses, Proc. Conf. Compuer Vision and Paern Recogniion, [15] A. Yilmaz, X. Li, and M. Shah, Conour-Based Objec Tracking wih Occlusion Handling in Video Acquired Using Mobile Cameras, IEEE Trans. Paern Analysis and Machine Inelligence, vol. 26, no. 11, pp , Nov [16] A. Mansouri, Region Tracking via Level Se PDES wihou Moion Compuaion, IEEE Trans. Paern Analysis and Machine Inelligence, vol. 24, pp , [17] N. Paragios and R. Deriche, Geodesic Acive Regions and Level Se Mehods FPR Moion Esimaion and Tracking, Proc. Conf. Compuer Vision and Image Undersanding, [18] M. Niehammer and A. Tannenbaum, Dynamic Geodesic Snakes for Visual Tracking, Proc. Compuer Vision and Paern Recogniion, vol. 1, pp , [19] Y. Rahi, N. Vaswani, A. Tannenbaum, and A. Yezzi, Paricle Filering for Geomeric Acive Conours wih Applicaion o Tracking Moving and Deforming Objecs, Proc. Conf. Compuer Vision and Paern Recogniion, [20] S. Arulampalam, S. Maskell, N.J. Gordon, and T. Clapp, A Tuorial on Paricle Filers for On-Line Non-Linear/Non-Gaussian Bayesian Tracking, IEEE Trans. Signal Processing, vol. 50, pp , Feb [21] N. Vaswani, A. Yezzi, Y. Rahi, and A. Tannenbaum, Paricle Filers for Infinie (or Large) Dimensional Sae Spaces Par I, Proc. In l Conf. Acousic, Speech, and Signal Processing, [22] M. Kasoulakis and A.T. Kho, Sochasic Curvaure Flows: Asympoic Derivaion, Level Se Formulaion and Numerical Experimens, J. Inerfaces and Free Boundaries, vol. 3, pp , [23] A. Douce, N. defreias, and N. Gordon, Sequenial Mone Carlo Mehods in Pracice. Springer, [24] R. van der Merwe, N. de Freias, A. Douce, and E. Wan, The Unscened Paricle Filer, Advances in Neural Informaion Processing Sysems 13, cieseer.is.psu.edu/aricle/vandermerwe00unscened.hml, Nov [25] A. Douce, On Sequenial Mone Carlo Sampling Mehods for Bayesian Filering, Technical Repor CUED/F-INFENG/TR. 310, Dep. of Eng., Cambridge Univ., cieseer.is.psu.edu/aricle/douce00sequenial.hml, [26] K. Okuma, A. Taleghani, N. de Freias, J. Lile, and D. Lowe, A Boosed Paricle Filer: Muliarge Deecion and Tracking, cieseer.is.psu.edu/ okuma04boosed.hml, [27] P. Perez, C. Hue, J. Vermaak, and M. Gangne, Color-Based Probabilisic Tracking, Proc. Sevenh European Conf. Compuer Vision-Par I, pp , [28] D. Cremers, T. Kohlberger, and C. Schnörr, Nonlinear Shape Saisics in Mumford-Shah Based Segmenaion, Proc. Sevenh European Conf. Compuer Vision, vol. 2351, pp , [29] M. Rousson and N. Paragios, Shape Priors for Level Se Represenaions, Proc. Sevenh European Conf. Compuer Vision, pp , [30] M. Levenon, W.L. Grimson, and O. Faugeras, Saisical Shape Influence in Geodesic Acive Conours, Proc. IEEE Conf. Compuer Vision and Paern Recogniion, pp , [31] D. Mumford and J. Shah, Opimal Approximaion by Piecewise Smooh Funcions and Associaed Variaional Problems, Comm. Pure Applied Mah., vol. 42, pp , [32] T. Chan and L. Vese, Acive Conours wihou Edges, IEEE Trans. Image Processing, vol. 10, no. 2, pp , [33] D. Tweed and A. Calway, Tracking Many Objecs Using Subordinaed Condensaion, Proc. Briish Machine Vision Conf., pp , [34] R. Sedgewick, Algorihms. Addison-Wesley, [35] H. Tao, H. Sawhney, and R. Kumar, A Sampling Algorihm for Tracking Muliple Objecs, Proc. In l Conf. Compuer Vision Algorihms, [36] J. MacCormick and A. Blake, A Probabilisic Exclusion Principle for Tracking Muliple Objecs, In l J. Compuer Vision, vol. 39, pp , [37] D. Cremers and S. Soao, A Pseudo-Disance for Shape Priors in Level Se Segmenaion, Proc. IEEE Workshop Variaional, Geomeric, and Level Se Mehods in Compuer Vision, For more informaion on his or any oher compuing opic, please visi our Digial Library a