Track and Cut: simultaneous tracking and segmentation of multiple objects with graph cuts

Transcription

1 INSTITUT NATIONAL DE RECHERCHE EN INFORMATIQUE ET EN AUTOMATIQUE Track and Cu: simulaneous racking and segmenaion of muliple objecs wih graph cus Aurélie Bugeau Parick Pérez N 6337 Ocober 2007 Thèmes COM e COG appor de recherche ISSN ISRN INRIA/RR FR+ENG

2

3 Track and Cu: simulaneous racking and segmenaion of muliple objecs wih graph cus Aurélie Bugeau, Parick Pérez Thèmes COM e COG Sysèmes communicans e Sysèmes cogniifs Proje Visa Rappor de recherche n 6337 Ocober pages Absrac: This paper presens a new mehod o boh rack and segmen muliple objecs in videos using min-cu/max-flow opimizaions. We inroduce objecive funcions ha combine low-level pixelwise measures (color, moion), high-level observaions obained via an independen deecion module, moion predicion and conras-sensiive conexual regularizaion. One novely is ha exernal observaions are used wihou adding any associaion sep. The observaions are image regions (pixel ses) ha can be oupu by any kind of deecor. The minimizaion of hese cos funcions simulaneously allows "deecion-before-rack" racking (rack-o-observaion assignmen and auomaic iniializaion of new racks) and segmenaion of racked objecs. When several racked objecs ge mixed up by he deecion module (e.g., single foreground deecion mask for objecs close o each oher), a second sage of minimizaion allows he proper racking and segmenaion of hese individual eniies despie he observaion confusion. Experimens on differen ype of sequences demonsrae he abiliy of he mehod o deec, rack and precisely segmen persons as hey ener and raverse he field of view, even in cases of parial occlusions, emporary grouping and frame dropping. Key-words: racking, segmenaion, graph cus Unié de recherche INRIA Rennes IRISA, Campus universiaire de Beaulieu, Rennes Cedex (France) Téléphone : Télécopie :

4 Suivi e segmenaion d objes par graph cus Résumé : Ce papier présene une nouvelle méhode de suivi e de segmenaion de plusieurs objes dans une vidéo, à l aide d un echnique de coupe minimale dans un graphe. Nous inroduisons une foncion d énergie qui combine des mesures calculées sur l image (couleur, mouvemen) en chaque pixel, des observaions obenues par un module exerne de déecion, la prédicion par le mouvemen de l obje e une régularisaion spaiale reposan sur les gradiens d inensié de l image. L uilisaion des observaions ne requier pas l ajou d une éape d associaion enre les objes e les observaions. Ces observaions son des régions d image (masques de pixels) qui peuven êre le résula de n impore quel déeceur. Quand plusieurs objes suivis se rerouven fusionnés (e.g., un seul masque de déecion pour plusieurs objes d apparence proche), une deuxième minimisaion d énergie perme le suivi e la segmenaion indépendane de ces eniés individuelles. Des résulas sur différens ypes de séquences monren la capacié de la méhode à bien déecer, suivre e segmener des objes présens dans le champ de la caméra, même en cas d occulaions parielles, de regroupemen emporaire d objes ou d absence d observaions. Mos-clés : suivi, segmenaion, coupe dans un graphe

5 Track and Cu: simulaneous racking and segmenaion of muliple objecs wih graph cus 3 Conens 1 Inroducion Exising mehods Overview of he paper Descripion of he objecs and of he observaions Descripion of he objecs Descripion of he observaions Background subracion Moving objecs deecion in complex scenes Principle of he mehod Tracking each objec independenly Principle of he racking mehod Principle of he segmenaion of merged objecs Energy funcions Graph Energy Daa erm Binary erm Energy minimizaion Creaion of new objecs Segmening merged objecs 13 6 Experimenal Resuls Tracking objecs deeced wih background subracion Tracking objecs in complex scenes Conclusion 16 1 Inroducion Visual racking is an imporan and challenging problem in compuer vision. Depending on applicaive conex under concern, i comes ino various forms (auomaic or manual iniializaion, single or muliple objecs, sill or moving camera, ec.), each of which being associaed wih an abundan lieraure. 1.1 Exising mehods In a recen review on visual racking [37], racking mehods are divided ino hree caegories: poin racking, silhouee racking and kernel racking. These hree caegories can be recas as "deecbefore-rack" racking, dynamic segmenaion and racking based on disribuions (color in paricular). RR n 6337

6 4 Bugeau & Pérez "Deec-before-rack" mehods The principle of "deec-before-rack" mehods is o mach he racked objecs wih observaions provided by an independen deecion module. Such a racking can be performed wih eiher deerminisic or probabilisic mehods. Deerminisic mehods amoun o maching by minimizing a disance beween he objec and he observaions based on cerain descripors (posiion and/or appearance) of he objec. The appearance (which can be for example he shape, he phoomery or he moion of he objec) is usually aken ino accoun wih hisograms : he hisograms of he objec and an observaion are compared using a disance measure, such as correlaion, Bhaacharya coefficien or Kullback-Leibler divergence. The observaions provided by a deecion algorihm are ofen corruped by noise. Moreover, he appearance (moion, phoomery, shape) of an objec can vary a lile beween wo consecuive frames. Probabilisic mehods provide means o ake measuremen uncerainies ino accoun. The are ofen based on a sae space model of he objec properies and he racking of one objec is performed using a filering mehod (Kalman filering [19], paricle filering [16]). Muliple objecs racking can also be done wih a filering echnique bu a sep of associaion beween he objecs and he observaions mus be added. The mos popular mehods for muliple objecs racking, in a deec-before-rack framework, are he MHT (Muliple Hypoheses Tracking) [28, 12] and he JPDAF (Join Probabiliy Daa Associaion Filering)[1, 2]. Dynamic segmenaion Dynamic segmenaion aims a exracing successive segmenaions over ime. A deailed silhouee of he arge objec is hus sough in each frame. This is ofen done by making evolve he silhouee obained in he previous frame owards a new configuraion in curren frame. The silhouee can eiher be represened by a se of parameers or by an energy funcion. In he firs case, he se of parameers represens a sae space model ha permis o rack he conour wih a filering mehod. For example, in [33], some conrol poins are posiioned all along he conour and heir dynamics define he sae model. The correcion of he poins posiion is obained using he image gradiens. In [17], he auhors proposed o model he sae wih a se of splines and some moion parameers. The racking is hen achieved wih a paricle filer. This echnique was exended o muliple objecs racking in [24]. Previous mehods do no deal wih he opology changes of an objec (fusion and/or spli). By minimizing an energy funcion, he changes can be handled. The objec is defined as a mask of pixels [26, 14] or by he zero level se of a coninuous funcion [27, 31]. In boh cases, he conour energy includes some emporal informaion in he form of eiher emporal gradiens (opical flow) [3, 13, 25] or appearance saisics originaed from he objec and is surroundings in previous images [29, 36]. In [35] he auhors use graph cus o minimize such an energy funcional. The advanages of min-cu/max-flow opimizaion are is low compuaional cos, he fac ha i converges o he global minimum wihou geing suck in local minima and ha no a priori on he global shape model is needed. They have also been used in [14] in order o successively segmen an objec hrough ime using a moion informaion. Kernel racking This las group of mehods aims a racking a small and simple porion of he image (ofen a recangle or an ellipse) based on he appearance. The bes locaion of he region in he curren frame is he one for which some feaure disribuions (e.g., color) are he closes o he reference one for he racked objec. Two approaches can be disinguished : he ones ha assume a local conservaion of he appearance of he objec and he ones ha assume his conservaion o be global. The mos popular INRIA

7 Track and Cu: simulaneous racking and segmenaion of muliple objecs wih graph cus 5 mehod for local conservaion is probably he KLT approach [30]. For he global conservaion, he mos ofen used echnique is he one of Comaniciu e al. [10, 11], where approximae mean shif ieraions are used o conduc he ieraive search. Graph cus have also been used for illuminaion invarian kernel racking in [15]. Advanages and limis of previous approaches These hree ypes of racking echniques have differen advanages and limiaions, and can serve differen purposes. The "deec-before-rack" approaches can deal wih he enrance of new objecs in he scene or he exis of exising ones. They use exernal observaions ha, if hey are of good qualiy, migh allow robus racking. On he conrary if hey are of low qualiy he racking can be deerioraed. Therefore, "deec-before-rack" mehods highly depend on he qualiy of he observaions. Furhermore he resriced assumpion ha one objec can only be associaed o one observaion is ofen made. Finally, his kind of racking usually oupus bounding boxes only. By conras, silhouee racking has he advanage of direcly providing he segmenaion of he racked objec. Represening he conour by a se of parameers allows he racking of an objec wih a relaively small compuaional ime. On he oher hand hese approaches do no deal wih opology changes. Tracking by minimizing an energy funcional allows he handling of opology changes bu no always of occlusions (i depends on he dynamics used). I can also be compuaionally inefficien and he minimizaion can converge o local minima of he energy. Wih he use of recen graph cus echniques, convergence o he global minima is obained for modes compuaional cos. However, a limi of mos silhouee racking approaches is ha hey do no deal wih he enrance of new objecs in he scene or he exis of exising ones. Finally kernel racking mehods, by capuring global color disribuion of a racked objec, allow robus racking a low cos in a wide range of color videos. They also do no deal wih he enrance of new objecs in he scene or he exis of exising ones, and he do no give he complee segmenaion of he objecs. Furhermore hey are no well adaped o he racking of small objecs. 1.2 Overview of he paper In his paper, we address he problem of muliple objecs racking and segmenaion by combining he advanages of he hree classes of approaches. We suppose ha, a each insan, he moving objecs are approximaely known hanks o some preprocessing algorihm. These moving objecs form he observaions (as explained in secion 2). Here, we will firs use a simple background subracion (he conneced componens of he deeced foreground mask serve as high-level observaions) and hen a more complex approach [8] dedicaed o moving objecs deecion in complex scenes. An imporan novely of our mehod is ha he use of exernal observaions does no require he addiion of a preliminary associaion sep. The associaion beween he racked objecs and he observaions is joinly conduced wih he segmenaion and he racking wihin he proposed minimizaion mehod. A each ime insan, racked objec masks are propagaed using heir associaed opical flow, which provides predicions. Color and moion disribuions are compued on he objecs in previous frame and used o evaluae individual pixel likelihood in he curren frame. We inroduce for each objec a binary labeling objecive funcion ha combines all hese ingrediens (low-level pixel-wise feaures, high-level observaions obained via an independen deecion module and moion predicions) wih a conras-sensiive conexual regularizaion. The minimizaion of each of hese energy funcions wih min-cu/max-flow provides he segmenaion of one of he racked objecs in he new frame. Our algorihm also deals wih he inroducion of new objecs and heir associaed racker. When muliple objecs rigger a single deecion due o heir spaial viciniy, he proposed mehod, as mos deec-before-rack approaches, can ge confused. To circumven his problem, we propose o minimize a secondary muli-label energy funcion which allows he individual segmenaion of RR n 6337

8 6 Bugeau & Pérez concerned objecs. The paper is organized as follows. Firs, in secion 2, he noaions are inroduced and he objecs and he observaions are described. In secion 3, an overview of he mehod is given. The primary energy funcion associaed o each racked objec is inroduced in secion 4. The inroducion of new objecs is also explained in his secion. The secondary energy funcion permiing he separaion of objecs wrongly merged in he firs sage is inroduced in secion 5. Experimenal resuls are repored in secion 6, where we demonsrae he abiliy of he mehod o deec, rack and precisely segmen objecs, possibly wih parial occlusions and missing observaions. The experimens also demonsrae ha he second sage of minimizaion allows he segmenaion of individual objecs when spaial proximiy makes hem merge a he foreground deecion level. 2 Descripion of he objecs and of he observaions For he clariy of he paper, we sar by explaining wha are he objecs and he observaions we are manipulaing and how hey are obained. 2.1 Descripion of he objecs In all his paper, P will denoe he se of N pixels of a frame from an inpu image sequence. To each pixel s of he image a ime is associaed a feaure vecor where z (C) z (s) = (z (C) (s),z (M) (s)), (s) is a 3-dimensional vecor in he color space and z (M) (s) is a 2-dimensional vecor of opical flow values. We consider a chrominance color space (here we use he YUV space, where Y is luminance and U and V he chrominances) because he objecs ha we will rack ofen conain skin, which is beer characerized in such a space [20, 32]. Furhermore, a chrominance space has he advanage of having he hree channels, Y, U and V, uncorrelaed. The opical flow vecors are compued using an incremenal muliscale implemenaion of Lucas and Kanade algorihm [23]. This mehod does no hold for pixels wih insufficienly conrased surroundings. For hese pixels, he moion is no compued and color consiues he only low-level feaure. Therefore, alhough no always explici in he noaion for he sake of conciseness, one should bear in mind ha we only consider a sparse moion field. We assume ha, a ime, k objecs are racked. The i h objec a ime is denoed as O (i) and is defined as a mask of pixels, O (i) P. The pixels of a frame no belonging o he objec O (i) belong o he background of his objec. Boh he objecs and he backgrounds will be represened by a disribuion ha combines moion and color informaion. Each disribuion is a mixure of Gaussians 1. For objec i a insan, his disribuion, denoed as p (i), is fied o he se of values {z (s)} (i) s O. We consider ha moion and color informaion are independen. Hence, he disribuion p (i) of a moion disribuion p (i,m) p (i,c) (fied o he se of values {z (M) is he produc ) and a color disribuion, (s)} (i) s O ). Under his independency assumpion for color and (fied o he se of values {z (C) (s)} (i) s O moion, he likelihood of individual pixel feaure z (s) according o previous join model is: p (i) (z (s)) = p (i,c) (z (C) (s)) p (i,m) (z (M) (s)). (1) 1 All mixures of Gaussians evoked in his paper are fied using he Expecaion-Maximizaion (EM) algorihm. INRIA

9 Track and Cu: simulaneous racking and segmenaion of muliple objecs wih graph cus 7 As we consider only a sparse moion field, only he color disribuion is aken ino accoun for he pixels no having an associaed moion vecor: p (i) (z (s)) = p (i,c) (z (C) (s)). The background disribuions are compued in he same way. The disribuion of he background of objec i a ime, denoed as q (i), is a mixure of Gaussians fied o he se of values {z (s)} (i) s P\O. I also combines moion and color informaion: q (i) (z (s)) = q (i,c) 2.2 Descripion of he observaions (z (C) (s)) q (i,m) (z (M) (s)). (2) The goal of his paper is o perform boh segmenaion and racking o ge he objec O (i) corresponding o he objec O (i) 1 of previous frame. Conrary o sequenial segmenaion echniques [18, 21, 26], we bring in objec-level observaions. We assume ha, a each ime, here are m observaions. The j h observaion a ime is denoed as M (j) and is defined as a mask of pixels, M (j) P. As objecs and backgrounds, an observaion j a ime is represened by a disribuion, denoed as ρ (j), which is a mixure of Gaussians combining color and moion informaions. The mixure is fied o he se {z (s)} (j) s M and is defined as: ρ (j) (z (s)) = ρ (j,c) (z (C) (s)) ρ (j,m) (z (M) (s)). (3) The observaions may be of various kinds (e.g., obained by a class-specific objec deecor, or moion/color deecors). Here we will consider wo differen ypes of observaions Background subracion The firs ype of observaions comes from a preprocessing sep of background subracion. Each observaion amouns o a conneced componen of he foreground map afer subracing a reference frame from he curren frame (figure 1). The conneced componens are obained using he (a) (b) (c) Figure 1: Observaions obained wih background subracion. (a) Reference frame. (b) Curren frame. (c) Resul of background subracion (pixels in black are labeled as foreground) and derived objec deecions (indicaed wih red bounding boxes). "gap/mounain" mehod described in [34] and ignoring small objecs. For he firs frame, he racked objecs will be iniialized as he observaions hemselves Moving objecs deecion in complex scenes In order o be able o rack objecs in more complex sequences, we will use a second ype of objecs deecor. The mehod considered is he one from [8] ha can be decomposed in hree main seps. RR n 6337

10 8 Bugeau & Pérez Firs, a grid of moving pixels having valid flow vecors is seleced. Each poin is described by is posiion, is color and is moion. Then hese poins are pariioned based on a mean shif algorihm [9], leading o several moving clusers, and finally segmenaion of he objecs are obained from he moving clusers by performing a graph cus based segmenaion. This las sep can be avoided here. Indeed, since in his paper we will propose a mehod ha simulaneously rack and segmen objecs, he observaions do no need o be a segmened objec. Therefore, he observaions will direcly be he deeced moving clusers (figure 2). The las sep of he deecion mehod will only be used when (a) (b) Figure 2: Observaions obained wih [8] on a waer skier sequence sho by a moving camera. (a) Deeced moving clusers superposed on he curren frame. (b) Mask of pixels characerizing he observaion. iniializing new objecs o rack. When our algorihm oupus ha a new racker should be creaed from a given observaion, he racker is iniialized wih he corresponding segmened deeced objec. In he deecion mehod, flow vecors are only compued on he poins of he grid. Therefore, in our racking algorihm, when using his ype of observaions, we will keep considering ha only he poins of he grid are characerized by a moion and a color vecor. All he oher poins will only be characerized by heir color. The moion field is hen really sparse here. 3 Principle of he mehod Before presening our approach ino deail, we sar by presening is main principle. In paricular, we explain why i is decomposed ino wo seps (firs a segmenaion/racking mehod and hen, when necessary, a furher segmenaion sep) and why each objec is racked independenly. 3.1 Tracking each objec independenly We propose in his paper a racking mehod based on energy minimizaions. Minimizing an energy wih min-cu/max-flow [7] (also known as Graph Cus) permis o assign a label o each pixel of an image. As in [5], he labeling of one pixel will here depend on he closeness beween he appearance a a pixel and he objecs appearances and also on he similariy beween his pixel and is neighbor. Indeed, a smoohness binary erm ha encourages wo neighboring having close appearance o ge he same label is added o he energy funcion. In our racking scheme, we wish o assign a label corresponding o one of he racked objecs o each pixel of he image. By using a muli-label energy funcion (each label corresponding o one objec), all objecs would be direcly racked simulaneously by minimizing a single energy funcion. However, in our algorihm, we do no use such an energy and each objec will be racked independenly. Such a choice comes from he will o disinguish he merging of several objecs from he INRIA

11 Track and Cu: simulaneous racking and segmenaion of muliple objecs wih graph cus 9 occlusions of some objecs by anoher one, which can no be done using a muli-label energy funcion. Le us illusrae his problem on an example. Assume ha wo objecs having similar appearance are racked. We are going o analyze and compare he wo following scenarios (described on figure 3). On he one hand, we suppose ha he wo objecs become conneced in he image plane a ime Figure 3: Merge of several objecs or occlusion? and, on he oher hand, ha one of he objecs occludes he second one a ime. Firs, suppose ha hese wo objecs are racked using a muli-label energy funcion. Since he appearance of he objecs is similar, when hey ge side by side (firs case), he minimizaion will end o label all he pixels in he same way (due o he smoohness erm). Hence, each pixel will probably be assigned he same label, corresponding o only one of he racked objecs. In he second case, when one objec occludes he oher one, he energy minimizaion leads o he same resul: all he pixels have he same label. Therefore, i is possible for hese wo scenarios o be mixed up. Assume now ha each objec is racked independenly by defining one energy funcion per objec (each objec is hen associaed o k 1 labels). For each objec he final label is eiher "objec" or "background". For he firs case, each pixel of he wo objecs will be, a he end of he wo minimizaions, labeled as "objec". For he second case, he pixels will be labeled as "objec" when he minimizaion is done for he occluding objec and as "background" for he occluded one. Therefore, by defining one energy funcion per objec, we are able o differeniae he wo cases. Of course, for he firs case, he obained resul is no he waned one: he pixels ge he same label which means ha he wo objecs have merged. In order o keep differeniaing he wo objecs, we will add o our racking mehod a sep of separaion of he merged objecs. The principles of he racking and he separaion of merged objecs are explained in nex subsecions. 3.2 Principle of he racking mehod The principle of our algorihm is as follows. A predicion O (i) 1 P is made for each objec i of ime 1. We denoe as d (i) 1 he mean, over all pixels of he objec a ime 1, of opical flow values: d (i) 1 = s O (i) 1 O (i) z (M) 1 (s) 1. (4) The predicion is obained by ranslaing each pixel belonging o O (i) 1 by his average opical flow: O (i) 1 = {s + d(i) 1, s O(i) 1 }. (5) RR n 6337

12 10 Bugeau & Pérez Using his predicion, he new observaions, as well as he disribuion p (i) of O 1, (i) an energy funcion is buil. The energy is minimized using min-cu/max-flow algorihm [7], which gives he new segmened objec a ime, O (i). The minimizaion also provides he correspondences of he objec wih all he available observaions, which direcly leads o he creaion of new objecs o rack. Our racking algorihm is summed up in figure 4. O (i) 1 Disribuions compuaion Predicion O (i) 1 Consrucion of he graph Observaions Energy minimizaion (Graph Cus) O (i) Correspondances beween O (i) 1 and he observaions Creaion of new objecs Figure 4: Principle of he algorihm 3.3 Principle of he segmenaion of merged objecs A he end of he racking sep, several objecs can have merged, i.e. he resuls of he segmenaions for differen objecs overlap, ha is i=1...k O (i). In order o keep racking each objec separaely, he merged objecs mus be separaed. This will be done by adding a muli-label energy minimizaion. 4 Energy funcions We define one racker for each objec. To each racker corresponds, for each frame, one graph and one energy funcion ha is minimized using he min-cu/max-flow algorihm [7]. Nodes and edges of he graph can be seen in figure 5. In all his paper, we consider a 8-neighborhood sysem. However, for clariy, on all he figures represening a graph, only a 4-neighborhood is represened. 4.1 Graph The undireced graph G = (V, E ) is defined as a se of nodes V and a se of edges E. The se of nodes is composed of wo subses. The firs subse is he se of N pixels of he image grid P. The second subse corresponds o he observaions: o each observaion mask M (j) is associaed a node n (j). We call hese nodes "observaion nodes". The se of nodes hus reads V = P {n (j), j = 1... m }. The se of edges is decomposed as follows: E = E m P. The se E P represens all unordered pairs {s, r} of neighboring elemens of P, and E M (j) wih s M (j). j=1 E M (j) is he se of unordered pairs {s, n (j) }, Segmening he objec O (i) amouns o assigning a label l s,, (i) eiher background, bg, or objec, fg, o each pixel node s of he graph. Associaing observaions o racked objecs amouns o INRIA

13 Track and Cu: simulaneous racking and segmenaion of muliple objecs wih graph cus 11 Objec i a ime -1 n (1) Graph for objec i a ime O (i) 1 n (2) Figure 5: Descripion of he graph. The lef figure is he resul of he energy minimizaion a ime 1. Whie nodes are labeled as objec and black nodes as background. The opical flow vecors for he objec are shown in blue. The righ figure shows he graph a ime. Two observaions are available, each of which giving rise o a special observaion node. The pixel nodes circled in red correspond o he masks of hese wo observaions. Dashed box indicaes prediced mask. assigning a binary label l (i) j, ( bg or fg ) o each observaion node n (j). The se of all he node labels forms L (i). 4.2 Energy An energy funcion is defined for each objec i a each insan. I is composed of unary daa erms R (i) s, and smoohness binary erms B s,r,: (i) E (i) (L (i) ) = R s,(l (i) s,) (i) + B (i) {s,r},(1 δ(l(i) s,, l r,)). (i) (6) s V {s,r} E In order o simplify he noaions, we omi in he res of his secion he index i. Previous equaion is hen rewrien as: E (L ) = R s, (l s, ) + B {s,r}, (1 δ(l s,, l r, )). (7) s V {s,r} E Daa erm The daa erm only concerns he pixel nodes lying in he prediced regions and he observaion nodes. For all he oher pixel nodes, labeling will only be conrolled by he neighbors via binary erms. More precisely, he firs par of energy in (7) reads: m R s, (l s, ) = ln(p 1 (s, l s, )) + α d 2 (j, l j, ). (8) s V s O 1 j=1 Segmened objec a ime should be similar, in erms of moion and color, o he preceding insance of his objec a ime 1. To exploi his consisency assumpion, he disribuion of he objec, p (i) 1 (equaion 1), and of he background, q (i) 1 (equaion 2), from previous image, are used. Remember ha we chose o omi he index of he objec. Previous disribuions are hen denoed as p 1 and q 1. The likelihood p 1, wihin prediced region, is finally defined as: p 1 (s, l) = { p 1 (z (s)) if l = fg, q 1 (z (s)) if l = bg. (9) In he same way, an observaion should be used only if i is likely o correspond o he racked objec. To evaluae he similariy of observaion j a ime and objec i a previous ime, a comparison RR n 6337

14 12 Bugeau & Pérez beween he disribuions p (i) 1 and ρ (j) (equaion 3) and beween q (i) 1 and ρ (j) mus be performed hrough he compuaion of a disance measure. A classical disance o compare wo mixures of Gaussians, G 1 and G 2, is he Kullback-leibler disance [22], defined as: The likelihood p 1, is finally: KL(G 1, G 2 ) = G 1 (x)log G 1(x) dx. (10) G 2 (x) d 2 (s, l) = { (j) KL(ρ, p 1 ) if l = fg, KL(ρ (j), q 1 ) if l = bg. (11) A consan α is included in he daa erm in equaion (8) o give more or less influence o he observaions. As only one node is used o represen he whole mask of pixels of an observaion, we have chosen o fix α equal o he number of pixels belonging o he observaion, ha is α = M (j) Binary erm Following [5], he binary erm beween neighboring pairs of pixels {s, r} of P is based on color gradiens and has he form B {s,r}, = λ 1 1 dis(s, r) e As in [4], he parameer σ T is se o σ T = 4 (z (C) z (C) (s) z (C) (r) 2 σ 2 T. (12) (s) z (C) (r)) 2, where. denoes expecaion over a box surrounding he objec. For edges beween one pixel node and one observaion node, he binary erm depends on he disance beween he color of he observaion and he pixel color. More precisely, i is compued as B {s,n (j) Parameers λ 1 and λ 2 are discussed in he experimens Energy minimizaion }, = λ 2 ρ (j) (z (C) (s)). (13) The final labeling of pixels is obained by minimizing, wih he Expansion Move algorihm [7], he energy defined above: ˆL (i) = argmin E (i) L (i) (L (i) ). (14) This labeling gives he segmenaion of he i-h objec a ime as: 4.3 Creaion of new objecs O (i) = {s P : (i) ˆl s, = fg }. (15) One advanage of our approach lies in is abiliy o joinly manipulae pixel labels and rack-odeecion assignmen labels. This allows he sysem o rack and segmen he objecs a ime while esablishing he correspondence beween an objec currenly racked and all he approximaive objec candidaes obained by deecion in curren frame. If, afer he energy minimizaion for an objec i, an observaion node n (j) is labeled as fg (ˆl (i),j = fg ) i means ha here is a correspondence beween INRIA

15 Track and Cu: simulaneous racking and segmenaion of muliple objecs wih graph cus 13 he i-h objec and he j-h observaion. Conversely, if he node is labeled as bg, he objec and he observaion are no associaed. (i) If for all he objecs (i = 1,..., k 1), an observaion node is labeled as bg ( i, ˆl,j = bg ), hen he corresponding observaion does no mach any objec. In his case, a new objec is creaed and iniialized wih his observaion. The number of racked objecs becomes k = k 1 + 1, and he new objec is iniialized as: O (k) = M (j). In pracice, he creaion of a new objec will only be validaed if he new objec is associaed o a (i) leas one observaion a ime + 1, i.e., if j {1...m +1 } such ha ˆl j,+1 = fg. 5 Segmening merged objecs Assume now ha he resuls of he segmenaions for differen objecs overlap, ha is i F O (i), where F denoes he curren se of objec indices. In his case, we propose an addiional sep o deermine wheher hese objecs ruly correspond o he same one or if hey should be separaed. A he end of his sep, each pixel of i FO (i) mus belong o only one objec. For his purpose, a new graph G = (Ṽ, Ẽ) is creaed, where Ṽ = i FO(i) and Ẽ is composed of all unordered pairs of neighboring pixel nodes of Ṽ. An exemple of such a graph is presened on figure 6. Figure 6: Graph example for he segmenaion of merged objecs. The goal is hen o assign o each node s of Ṽ a label ψs F. Defining L = {ψ s, s Ṽ} he labeling of Ṽ, a new energy is defined as: Ẽ ( L) = s Ṽ ln(p 3 (s, ψ s )) + λ 3 {s,r} Ẽ z s 1 (C) z r (C) 2 dis(s, r) e σ 3 2 (1 δ(ψ s, ψ r )). (16) The parameer σ 3 is here se as σ 3 = 4 (z (s) (i,c) z (r) (i,c) ) 2 wih he averaging being over i F and {s, r} Ẽ. The fac ha several objecs have been merged shows ha heir respecive feaure disribuions a previous insan did no permi o disinguish hem. A way o separae hem is hen o increase he role of he predicion. This is achieved by choosing funcion p 3 as: p 3 (s, ψ) = { p (ψ) 1 (z (s)) if s / O (ψ) 1, 1 oherwise. (17) RR n 6337

16 14 Bugeau & Pérez This muli-label energy funcion is minimized using he swap algorihms [6, 7]. Afer his minimizaion, he objecs O (i), i F are updaed. 6 Experimenal Resuls This secion presens various resuls of he racking and he separaion of merged objecs. Firs, we will consider a relaively simple sequence, wih saic background, in which he observaions are obained by background subracion (subsecion 2.2.1). Nex he racking mehod will be combined o he moving objecs deecor of [8] (subsecion 2.2.2). For all resuls, a color is associaed o each racked objec. This color only depends on he arbirary order in which he objecs are creaed. 6.1 Tracking objecs deeced wih background subracion We sar by demonsraing, on a sequence from he PETS 2006 daa corpus (sequence 1 camera 4), he validiy of he racking mehod as well as he robusness o parial occlusions and he individual segmenaion of objecs ha were iniially merged. Following [4], he parameer λ 3 was se o 20. However parameers λ 1 and λ 2 had o be uned by hand o ge beer resuls. Indeed, λ 1 was se o 10 while λ 2 o 2. Also, he number of classes for he Gaussian mixure models was se o 10. Firs resuls (figure 7) demonsrae he good behavior of our algorihm even in he presence of parial occlusions and of objec fusion. Observaions, obained by subracing reference frame (frame 10 shown on figure 1(a)) o he curren one, are visible in he second column of figure 7. The hird column conains he segmenaion of he objecs wih he use of he second energy funcion. In frame 81, wo objecs are iniialized using he observaions. Noe ha he conneced componen exraced wih he gap/mounain mehod misses he legs for he person in he upper righ corner. While his impacs he iniial segmenaion, he legs are included in he segmenaion as soon as he subsequen frame. The proposed mehods deals easily wih he enrance of new objecs in he scene. This resul also shows he robusness of our mehod o parial occlusions. Parial occlusions occur when he person a he op passes behind he hree oher ones (frames 176 and 206). Despie he similar color of all he objecs, his is well handled by he mehod, as he person is sill racked when he occlusion sops (frame 248). Finally noe ha, even if from he 102 nd frame he wo persons a he boom of he frames correspond o only one observaion and have a similar appearance (color and moion), our algorihm racks each person separaely (frames 116, 146). In figure 8, we show in more deails he influence of he second energy funcion by comparing he resuls obained wih and wihou i. Before frame 102, he hree persons a he boom generae hree disinc observaions while, passed his insan, hey correspond o only one or wo observaions. Even if he moions and colors of he hree persons are very close, he use of he secondary muli-label energy funcion allows heir separaion. 6.2 Tracking objecs in complex scenes We are now going o show he behaviour of our racking algorihm when he sequences are more complex (dynamic background, moving camera...). For each sequence, he observaions are he moving clusers deeced wih he mehod of [8]. In all his subsecion, he parameer λ 3 was se o 20, λ 1 o 10, and λ 2 o 1. The firs resul is on a waer skier sequence (figure 9). For each image, he moving clusers and he INRIA

17 Track and Cu: simulaneous racking and segmenaion of muliple objecs wih graph cus 15 (a) (b) (c) Figure 7: Resuls on sequence from PETS 2006 (frames 81, 116, 146, 176, 206 and 248). (a) Original frames. (b) Resul of simple background subracion and exraced observaions. (c) Tracked objecs on curren frame using he secondary energy funcion. RR n 6337

18 16 Bugeau & Pérez (a) (b) (c) Figure 8: Separaing merged objecs wih he secondary minimizaion (frames 101 and 102). (a) Resul of simple background subracion and exraced observaions. (b) Segmenaions wih primary energy funcions only. (c) Segmenaion afer pos-processing wih he secondary energy funcion. masks of he racked objecs are superimposed on he original image. The proposed racking mehod permis o rack correcly he waer skier (or more precisely his we sui) all along he sequence, despie he rajecory changes. As can be seen on he figure (for example a ime 58), he deecor someimes fails o deec he skier. No observaions are hen available. However, hanks o he use of he predicion of he objec, our mehod handles well his kind of siuaions and keeps racking and segmening correcly he skier. This shows he robusness of he algorihm o missing observaions. However if some observaions are missing for several consecuive frames, he segmenaion can be a bi deerioraed. Conversely, his means ha he incorporaion of observaions from he deecion module enables o ge beer segmenaions han when using only predicions. On several frames, some moving clusers are deeced in he waer. Neverheless, no objecs are creaed in his area. The reason is ha he creaion of a new objec is only validaed if he new objec is associaed o a leas one observaion in he following frame. This never happened in he sequence. We end by showing resuls on a driver sequence (figure 10). The firs objec deeced and racked is he face. Once again, racking his objec shows he robusness of our mehod o missing observaions. Indeed, even if from frame 19, he face does no move and herefore is no deeced, he algorihm keeps racking and segmening i correcly unil he driver sars urning i. The mos imporan resul on his sequence is he hands racking. In image 39, he masks of he wo hands are merged: hey have a few pixels in common. The sep of segmenaion of merged objecs is hen applied which allows he correc separaion of he wo masks and permis o keep racking hese wo objecs separaely. Finally, as can been seen on frame 57, our mehod deals well wih he exi of an objec from he scene. 7 Conclusion In his paper we have presened a new mehod o simulaneously segmen and rack objecs. Predicions and observaions, composed of deeced objecs, are inroduced in an energy funcion which is minimized using graph cus. The use of graph cus permis he segmenaion of he objecs a a modes compuaional cos (of course he compuaional ime depends on he objecs deecion and he disribuions compuaion). A novely is he use of observaion nodes in he graph which gives beer segmenaions bu also enables he direc associaion of he racked objecs o he observaions (wihou adding any associaion procedure). The algorihm is robus o parial occlusions, progressive illuminaion changes and o missing observaions. Thanks o he use of a secondary muli-label INRIA

19 Track and Cu: simulaneous racking and segmenaion of muliple objecs wih graph cus 17 = 31 = 177 = 51 = 215 = 58 = 225 = 80 = 243 Figure 9: Resuls on a waer skier sequence. The observaions are moving clusers deeced wih he mehod in [8]. A each ime, he observaions are shown on he lef image while he masks of he racked objecs are shown on he righ image. energy funcion, our mehod allows individual racking and segmenaion of objecs which where no disinguished from each oher in he firs sage. The observaions used in his paper are obained firsly by a simple background subracion based on a single reference frame and secondly by a more complicaed moving objec deecor. Noe however ha any objec deecion mehod could be used as well wih no change o he approach, as soon as he observaions can be represened by a mask of pixels. As we use feaure disribuions of objecs a previous ime o define curren energy funcions, our mehod breaks down in exreme cases of abrup illuminaion changes. However, by adding an exernal deecor of such changes, we could circumven his problem by keeping only he predicion and by updaing he reference frame when he abrup change occurs. Also, oher cues, such as shapes, could probably be added o improve he resuls. Apar from his raher specific problem, several research direcions are open. One of hem concerns he design of an unifying energy framework ha would allow segmenaion and racking of RR n 6337

20 18 Bugeau & Pérez = 13 = 39 = 16 = 43 = 29 = 57 = 35 = 63 Figure 10: Resuls on a driver sequence. The observaions are moving clusers deeced wih he mehod in [8]. A each ime, he observaions are shown on he lef image while he masks of he racked objecs are shown on he righ image. muliple objecs while precluding he incorrec merging of similar objecs geing close o each oher in he image plane. Anoher direcion of research concerns he auomaic uning of he parameers, which remains an open problem in he recen lieraure on image labeling (e.g., figure/ground segmenaion) wih graph-cus. References [1] Y. Bar-Shalom and X. Li. Esimaion and Tracking: Principles, Techniques, and Sofware. MA: Arech House, Boson, INRIA

21 Track and Cu: simulaneous racking and segmenaion of muliple objecs wih graph cus 19 [2] Y. Bar-Shalom and X. Li. Mulisensor-muliarge racking: Principles and Techniques. CT: YBS Publishing, Sorrs, [3] M. Beralmio, G. Sapiro, and G. Randall. Morphing acive conours. IEEE Trans. Paern Anal. Machine Inell., 22(7): , [4] A. Blake, C. Roher, M. Brown, P. Pérez, and P. Torr. Ineracive image segmenaion using an adapive gmmrf model. In Proc. Europ. Conf. Compuer Vision, [5] Y. Boykov and M. Jolly. Ineracive graph cus for opimal boundary and region segmenaion of objecs in n-d images. Proc. In. Conf. Compuer Vision, [6] Y. Boykov, O. Veksler, and R. Zabih. Markov random fields wih efficien approximaions. In Proc. Conf. Comp. Vision Paern Rec., [7] Y. Boykov, O. Veksler, and R. Zabih. Fas approximae energy minimizaion via graph cus. IEEE Trans. Paern Anal. Machine Inell., 23(11): , [8] A. Bugeau and P. Pérez. Deecion and segmenaion of moving objecs in highly dynamic scenes. Proc. Conf. Comp. Vision Paern Rec., [9] D. Comaniciu and P. Meer. Mean shif: A robus approach oward feaure space analysis. IEEE Trans. Paern Anal. Machine Inell., 24(5): , [10] D. Comaniciu, V. Ramesh, and P. Meer. Real-ime racking of non-rigid objecs using meanshif. In Proc. Conf. Comp. Vision Paern Rec., [11] D. Comaniciu, V. Ramesh, and P. Meer. Kernel-based opical racking. IEEE Trans. Paern Anal. Machine Inell., 25(5): , May [12] I. Cox. A review of saisical daa associaion for moion correspondence. In. J. Compuer Vision, 10(1):53 66, [13] D. Cremers and C. C. Schnörr. Saisical shape knowledge in variaional moion segmenaion. Image and Vision Compuing, 21(1):77 86, [14] A. Criminisi, G. Cross, A. Blake, and V. Kolmogorov. Bilayer segmenaion of live video. Proc. Conf. Comp. Vision Paern Rec., [15] D. Freedman and M. Turek. Illuminaion-invarian racking via graph cus. Proc. Conf. Comp. Vision Paern Rec., [16] N. Gordon, D. Salmond, and A. Smih. Novel approach o nonlinear/non-gaussian bayesian sae esimaion. IEEE Proceedings on Radar and Signal Processing, [17] M. Isard and A. Blake. Condensaion condiional densiy propagaion for visual racking. In. J. Compuer Vision, 29(1):5 28, [18] O. Juan and Y. Boykov. Acive graph cus. In Proc. Conf. Comp. Vision Paern Rec., [19] R. Kalman. A new approach o linear filering and predicion problems. J. Basic Eng., 82:35 45, [20] R. Kjeldsen and J. Kender. Finding skin in color images. Inernaional Conference on Auomaic Face and Gesure Recogniion, RR n 6337

22 20 Bugeau & Pérez [21] P. Kohli and P. Torr. Effcienly solving dynamic markov random fields using graph cus. In Proc. In. Conf. Compuer Vision, [22] S. Kullback and R. A. Leibler. On informaion and sufficiency. Annals of Mahemaical Saisics, 22(1):79 86, March [23] B.D. Lucas and T. Kanade. An ieraive echnique of image regisraion and is applicaion o sereo. Proc. In. Join Conf. on Arificial Inelligence, [24] J. MacCormick and A. Blake. A probabilisic exclusion principle for racking muliple objecs. In. J. Compuer Vision, 39(1):57 71, [25] A. Mansouri. Region racking via level se pdes wihou moion compuaion. IEEE Trans. Paern Anal. Machine Inell., 24(7): , [26] N. Paragios and R. Deriche. Geodesic acive regions for moion esimaion and racking. In Proc. In. Conf. Compuer Vision, [27] N. Paragios and G. Tzirias. Adapive deecion and localizaion of moving objecs in image sequences. Signal Processing: Image Communicaion, 14: , [28] D Reid. An algorihm for racking muliple arges. IEEE Trans. Auom. Conrol, 24(6): , [29] R. Ronfard. Region-based sraegies for acive conour models. In. J. Compuer Vision, 13(2): , [30] J. Shi and C. Tomasi. Good feaures o rack. Proc. Conf. Comp. Vision Paern Rec., [31] Y. Shi and W. Karl. Real-ime racking using level ses. Proc. Conf. Comp. Vision Paern Rec., [32] M. Singh and N. Ahuja. Regression based bandwidh selecion for segmenaion using parzen windows. Proc. In. Conf. Compuer Vision, 1, [33] D. Terzopoulos and R. Szeliski. Tracking wih kalman snakes. Acive vision, pages 3 20, [34] Y. Wang, J.F. Dohery, and R.E. Van Dyck. Moving objec racking in video. Applied Imagery Paern Recogniion (AIPR) Annual Workshop, [35] N. Xu and N. Ahuja. Objec conour racking using graph cus based acive conours. Proc. In. Conf. Image Processing, [36] A. Yilmaz. Conour-based objec racking wih occlusion handling in video acquired using mobile cameras. IEEE Trans. Paern Anal. Machine Inell., 26(11): , [37] A. Yilmaz, O. Javed, and M. Shah. Objec racking: A survey. ACM Compu. Surv., 38(4):13, INRIA

23 Unié de recherche INRIA Rennes IRISA, Campus universiaire de Beaulieu Rennes Cedex (France) Unié de recherche INRIA Fuurs : Parc Club Orsay Universié - ZAC des Vignes 4, rue Jacques Monod ORSAY Cedex (France) Unié de recherche INRIA Lorraine : LORIA, Technopôle de Nancy-Brabois - Campus scienifique 615, rue du Jardin Boanique - BP Villers-lès-Nancy Cedex (France) Unié de recherche INRIA Rhône-Alpes : 655, avenue de l Europe Monbonno Sain-Ismier (France) Unié de recherche INRIA Rocquencour : Domaine de Voluceau - Rocquencour - BP Le Chesnay Cedex (France) Unié de recherche INRIA Sophia Anipolis : 2004, roue des Lucioles - BP Sophia Anipolis Cedex (France) Édieur INRIA - Domaine de Voluceau - Rocquencour, BP Le Chesnay Cedex (France) hp:// ISSN