Multitarget Data Association with Higher-Order Motion Models

Transcription

1 Multitarget Data Assoiation with Higher-Orer Motion Moels Robert T. Collins The Pennsylvania State University University Park, PA 16802, USA Abstrat We present an iterative approximate solution to the multiimensional assignment problem uner general ost funtions. The metho maintains a feasible solution at every step, an is guarantee to onverge. It is similar to the iterate onitional moes (ICM) algorithm, but applie at eah step to a blok of variables representing orresponenes between two ajaent frames, with the optimal onitional moe being alulate exatly as the solution to a two-frame linear assignment problem. Experiments with groun-truthe trajetory ata show that the metho outperforms both network-flow ata assoiation an greey reursive filtering using a onstant veloity motion moel. 1. Introution The multi-target, multi-frame ata assoiation problem has a long history, with early works appearing in the target traking [6] an omputer vision [10] ommunities. It has seen a resurgene of interest in omputer vision ue to reent popularity of traking-by-etetion approahes, whih apply a etetor inepenently on every frame to fin aniate objets that are then assoiate aross frames [1]. Traitional ata assoiation problems onsier point-like objets (e.g. raar blips), with trajetory quality measure by smoothness an ontinuity of objet motion. Visionbase ata assoiation, on the other han, involves objets of extene spatial extent in an image, from whih isriminative appearane ues an be extrate to help isambiguate mathes. We argue that reent vision-base approahes have begun to rely too heavily on these appearane ues, to the point of ignoring motion harateristis. One example is the reent network flow approah to ata assoiation, whih formulates an objetive funtion ontaining only pairwise osts, an for whih a globally optimal solution an be foun in polynomial time [22, 3, 17]. An yet, see Figure 1 for a simple example where network flow is This work was fune by a grant from ObjetVieo an the AFOSR. Tehnial isussions with Davi Tolliver an Khurram Shafique of ObjetVieo were instrumental to the evelopment of this work. Figure 1. Comparison of trajetories ompute by network-flow ata assoiation (left) versus our approah using a ost funtion that inorporates a onstant-veloity smoothness term (right). Network flow approahes gain their effiieny by limiting the ost funtion to pairwise terms. Using higher-orer motion moels leas to smoother trajetories that reue the number of mismath errors, partiularly at low sampling rates. unable to fin the orret trajetories beause it is unable to represent onstant veloity motion onstraints. These network flow formulations gain effiieny by limiting the ost funtions they an hanle. Our experiments show that lak of regularizing motion moels has a etrimental effet on quality of the trajetories foun, partiularly when appearane onstraints are weak an etetion frame-rate is low. To fous our arguments, in this paper we o not use appearane terms objets are esribe solely by their 2D point loations. We o not eny that when objets are easily istinguishable by appearane, appearane terms an (an probably shoul) o most of the work. Our point is that we nee to retain the ability to leverage kinemati motion moels in ases where objets are very similar, or when there are rapi appearane hanges ue to pose or lighting. Although this paper fouses on kinemati ost funtions alone, we fully expet an improvement in ata assoiation performane when appearane terms are ae bak in. From a ombinatorial optimization stanpoint, searh for the best ata assoiation is governe by the form of the objetive funtion, whih requires two esign eisions: how to represent the set of all trajetories that an be forme from raw target observations; an how to alulate the ost (or affinity sore, if maximizing) of eah trajetory. Our approah borrows from network flow ieas to represent eah trajetory as a sequene of eges through a trellis graph, enabling effiient loal upate rules. However, network flow approahes also fator the ost funtion into pairwise osts, 1

2 one ost per ege in the graph, while we retain general ost funtions that are in priniple efine over entire trajetories. This is a major ifferene, an has both pros an ons. On the upsie is the inrease in power of our ost funtions to represent nontrivial motion onstraints. On the ownsie, the use of ost funtions efine over temporal winows longer than 2 frames yiels ata assoiation problems that are NP-har. However, this NP-harness is nothing new to the multi-target traking ommunity, whih has shown that approximate algorithms an proue multi-frame trajetories of high quality while working effiiently in pratie. Contributions. We present an iterative approximate solution to the multiimensional assignment problem uner general ost funtions. The metho maintains a feasible solution at every step, an is guarantee to onverge to a (loal) minimum. It is similar to the iterate onitional moes (ICM) algorithm, but is applie at eah step to a blok of variables representing possible orresponenes between two ajaent frames. The blok-optimal onitional moe at eah step is alulate exatly an effiiently as the solution to a two-frame linear assignment problem. Our approah iffers from both traitional multi-frame ata assignment approahes as well as from more reent network flow approahes. Unlike traitional multi-frame ata assoiation, we fator the eision variable for eah trajetory into a prout of variables efine over eges in a trellis graph. Unlike reent network flow formulations, we retain the full power of general ost funtions for esribing kinemati motion moels an long-range regularizers that improve the quality of estimate trajetories. We evelop a novel higher-orer ost funtion for ata assoiation that uses ative ontour ( snake ) spline energy to measure the quality of a propose trajetory. We show in our evaluation that this ost funtion ompares favorably with network-flow solutions an greey sequential filtering using a onstant veloity motion moel. 2. Bakgroun an Relate Work This setion reviews ifferent ombinatorial formulations of the ata assoiation problem. Data assoiation is the proess of partitioning a set of observations into trajetories. Although the ombinatorial formulations iffer, all are base on the funamental onstraint that trajetories must be isjoint; that is, no two trajetories an laim the same observation. To keep the isussion at a high level, observations will be enote as elements, an trajetories as subsets of elements. In the weighte Set Partition Problem (SPP) [2], we are given a universe of elements U = {e 1, e 2,..., e n }, a list of allowable sets S = {S 1, S 2,..., S m S j U}, an a ost j for eah set. The goal is to hoose a minimum ost olletion of sets that form a partition of U, i.e. eah element appears in one an only one set. Creating a solution vetor of binary eision variables, x = [x 1, x 2,..., x m ] T, with x j = 1 iff set S j is in the solution, SPP an be written as the binary integer program m { Ax = 1 min j x j s.t. (1) x j {0, 1} where A is an n m onstraint matrix with one row for eah element, one olumn for eah set, an a ij =1 if element i appears in set j, 0 otherwise. The i-th onstraint j a ijx j = 1 ounts the number of sets in the solution that ontain element i, an requires there to be only one. The losely-relate Set Paking (SP) problem is written as a maximization m { Ax 1 max p j x j s.t. (2) x j {0, 1} where p is now a vetor of sores to be maximize rather than osts to be minimize (for example, p j = j ). The main ifferene is the relation in the onstraints, making it possible to leave some elements unuse in the solution. In the ontext of ata assoiation, this makes it easier to ignore false positive etetions an spurious trajetory fragments, rather than expliitly aounting for them. The lassi paper by Morefiel on 0-1 programming for ata assoiation is an SP formulation [14]. Reent network flow algorithms for ata assoiation [22, 3, 17] an also be interprete as solving SP, although limite to sores/osts having a eomposable struture (Setion 3.3). The Maximum-Weight Inepenent Set (MWIS) problem, aka vertex paking, is also losely relate to SP an SPP. Consier the olumn intersetion graph G(S,E) assoiate with onstraint matrix A, with one vertex for eah set S j, an ege set E = {(i, j) S i S j }, that is, having an ege between two verties if their unerlying sets are not isjoint. As above, eah set S j has a sore p j an a binary eision variable x j iniating whether set/vertex S j is part of the solution. The MWIS problem solves for m { xj + x max p j x j s.t. j 1, (i, j) E (3) x j {0, 1}. The pros an ons of using MWIS for ata assoiation as ompare to multi-hypothesis traking (MHT) are isusse in [16]. A two-frame MWIS approah followe by hierarhial linking is presente in [7]. The MultiDimensional Assignment (MDA) problem is a speialization of SPP to the ase of hypergraphs where elements are organize into a k-partite graph an the allowable sets are hypereges ontaining one element per partite set. The SPP formulation is partiularly well-suite to

3 ata assoiation for multi-target traking, where elements are target observations in a series of frames, an allowable sets are paths onneting observations between frames. For example, assume a sequene of k frames with n observations in eah frame, from whih an optimal partition of n trajetories of length k is to be forme (this is generalize in Setion 3 to inlue varying numbers of observations an trajetory lengths). The k-partite struture of MDA enables enumeration over trajetories an ost funtion values by neste sums over observations an frames, i.e. subjet to min n n i 1=1 i 2=1 n i1i 2...i k (4) i k =1 I\i f = 1; f = 1, 2,..., k i 1 = 1, 2,..., n i 2 = 1, 2,..., n. i k = 1, 2,..., n {0, 1} (5) Here, I\i f enotes all observations in all frames other than frame f. As in SPP, there is one onstraint per observation, represente as a summation over all k-paths ontaining it. MDA is known to be NP-har for 3 or more frames. However, in the ase of two frames it reues to the 2D linear assignment problem on a bipartite graph n n min ij x ij s.t. j x ij = 1; i = 1, 2,..., n i x ij = 1; j = 1, 2,..., n i=1 x ij {0, 1} (6) for whih polynomial time exat solutions are known, e.g., the Kuhn-Munkres (Hungarian) algorithm [8]. Data assoiation problems formulate in an MDA framework have been wiely stuie, partiularly target traking appliations performing probabilisti MAP estimation [6, 20, 18]. Solution methos inlue both eterministi (e.g. MHT [16]) an stohasti (e.g. MCMCDA [15]) searh. Shafique presents a polynomial-time exat solution for pairwise ost funtions an a greey approximate solution for higher-orer motion moels [21]. We aopt a generalize MDA framework in Setion 3. Several reent vision papers show that ata assoiation an be formulate as a Network Flow (NF) problem for ost funtions that eompose into a prout of pairwise terms [22, 3, 17, 11]. Network flow an be solve in polynomial time using either linear programming [11], pushrelabel methos [22], or suessive shortest path algorithms [3, 17]. Effiient approximate solutions base on ynami programming also an be applie. However, as mentione in the introution, the network flow formulation is limite to relatively uninteresting motion moels suh as pairwise istane [22] or boune veloity [17]. This is so beause a stati set of pairwise interframe ege osts is not able to represent kinemati onstraints suh as onstant veloity that require ata over three or more frames. We show in our experimental results (Setion 4.2) that the inability to use higher-orer motion moels has a major etrimental effet on the quality of trajetories foun using network flow. All the previous approahes have been linear programming formulations. In ontrast, ata assoiation an also be formulate as a Quarati Boolean Optimization (QBO) problem [7, 13], inluing the problem of two frame assoiation via graph mathing [9]. The main benefit of quarati programming is the ability to represent onstraints between pairs of trajetories, suh as oupling them to enourage similar motions [7]. 3. Our Approah We aopt a generalize multiimensional assignment (MDA) formalism that hanles trajetory fragments an unassigne observations [18, 20], an present an iterative approximate solution for general ost funtions. While retaining generality of the ost funtion, we fator the trajetory eision variables into pairwise eges between observations in ajaent frames, failitating loal trajetory upates. We present an iterative approah that yles through the sequene, solving for eision variables between eah subsequent pair of frames while holing the rest of the urrent solution fixe. These two-frame solutions are ompute exatly an effiiently using the Hungarian algorithm. Eah step of eah iteration improves the value of the objetive funtion (more preisely, oes not inrease it s value) an iterations ontinue until no further improvement is seen. Our metho is similar to the Iterate Conitional Moes (ICM) algorithm of Besag [5], exept bloks of variables representing potential mathes between pairs of ajaent frames are upate simultaneously, instea of a single variable at a time. This blok upate strategy is helpful beause it maintains isjointness of trajetories. It is also expete that solution upates optimizing over multiple variables shoul be able to fin stronger loal optima than upate sheules that solve for a single variable at a time. In the next setion we lay out the generalize MDA problem formulation. Setion 3.2 presents our blok-icm iterative improvement strategy an skethes a proof (ue to Besag) that it is guarantee to onverge. Setion 3.3 shows that when the ost funtion an be fatore into a prout of pairwise osts between ajaent frames, our formalism reues to the network flow approah. Finally, Setion 3.4 isusses implementation issues relate to initialization an termination of the algorithm.

4 3.1. Problem Formulation Consier a sequene of k image frames where etetions have been observe. In eah frame f = 1,..., k we have n f observations, labele by an augmente inex set I f I f = {0, 1, 2,..., n f } ; 1 f k (7) where inex 0 is a virtual or ummy inex that allows us to reason over misse etetions an partial trajetories [18]. The observations from frames 1 through k form a k-partite graph G = (V, E) with V = I 1 I 2 I k (8) E = (I 1 xi 2 ) (I 2 xi 3 ) (I k 1 x I k ). (9) We treat G as an unirete graph. Denote the set of all paths of length k through graph G by T = I 1 I 2 I k, an efine a ost funtion : T R. Eah path t T represents one hypothesize target trajetory, while (t) is a ost quantifying trak quality. We write a k-path as an orere list of its inient inies, (i 1, i 2,..., i k ), with i f I f for 1 f k. The ummy inex 0 plays an important role in inreasing the flexibility of the k-path representation to hanle partial trajetories. For example, (0,0,a,b,) : path that starts at frame 3 (a,b,,0,0) : path that ens at frame 3 (0,0,a,0,0) : false positive etetion in frame 3 (a,b,0,0,) : misse etetions or olusion Thus all trajetory hypotheses, inluing partial trajetories an false positives, are represente by omplete paths of length k. The efinition of trajetory isjointness is moifie to exlue inex 0 so that a variable number of misse etetions an be represente in any given frame. We require eah path to ontain at least one real observation. Fining the best set of isjoint trajetories an now be formulate as a multiimensional assignment problem, with binary eision variables for eah k-path with assoiate osts i1i 2...i k. We seek an optimal set of k-paths with respet to the following linear objetive funtion an isjointness onstraints subjet to n 2 n 3 i 2=0 i 3=0 n 1 n 3 min n 1 n 2 i 1=0 i 2=0 n k i k =0 n k n k i k =0 i1i 2...i k (10) = 1; i 1 = 1, 2,..., n 1 (11) = 1; i 2 = 1, 2,..., n 2 (12) i 1=0 i 3=0 i k =0.. n 1 n 2 i 1=0 i 2=0 n k 1 i k 1 =0 = 1; i k = 1, 2,..., n k (13) Sine the problem is NP-har, it is infeasible to searh for the exat optimal solution, an an approximate solution metho is neessary. Greey forwar sequential methos are one obvious approah. These are ommonly foun in the target traking literature in the form of probabilisti reursive estimators, inluing single target filters suh as the Kalman filter an PDAF, as well as multitarget filters suh as JPDAF [19, 6]. These approahes require ausal ost funtions where omputations at time t are base only on information observe up to time t. Beause eisions, one mae, are fixe, these methos are suseptible to making mathes that are later reveale to be suboptimal. We use a greey sequential metho base on onstant veloity motion preition as a baseline for omparison in our experiments, as well as to initialize our iterative improvement approah. Multi-hypothesis traking (MHT) seeks stronger solutions through a eferre eision strategy [6]. Ambiguous mathes are maintaine until enough later information has been seen to isambiguate them. This leas to a ombinatorially large, branhing tree of hypotheses, an in pratie suboptimal heuristi pruning eisions must be mae [10]. Poore [18] an Deb et al. [20] present Lagrangian relaxation shemes for multiimensional assignment. These methos progressively relax the one-to-one mathing onstraints to generate a series of problems that are easier to solve, by inserting some of the mathing onstraints into the objetive funtion as soft onstraints using Lagrange multipliers. Both MHT an Lagrangean relaxation are ompliate algorithms that are iffiult to oe an analyze. In the next setion we propose a blok-icm iteration sheme that applies a series of optimal two-frame linear assignment eisions to monotonially improve an initial solution An Iterative Approximation To introue our solution strategy, first onsier the notation for a simple 4-frame problem with observations inexe in the first frame by I a = (0, 1,..., n a ), in the seon frame by I b = (0, 1,..., n b ), thir frame I = (0, 1,..., n ), an fourth frame I = (0, 1,..., n ). Letting binary eision variable x ab be 1 if path (a,b,,) is in the solution, an zero otherwise, our multiimensional assignment problem of Eq. 10 an be written as min n a a=0 n b b=0 n =0 n =0 ab x ab (14) s.t Ax = 1 (15) x {0, 1} (16)

5 where A is a matrix representing the linear onstraints in equations We further leverage the k-partite struture of the problem by noting that a path (a, b,, ) is uniquely efine by its ege list ((a, b), (b, ), (, )), so we an replae eision variables on the paths T = I 1 I 2 I 3 I 4 with eision variables on the eges E = (I 1 I 2 ) (I 2 I 3 ) (I 3 I 4 ). That is, we an fator x ab as f ab g b h, with x ab = 1 iff f ab = 1, g b = 1, an h = 1. This fatoring is avantageous beause the number of eision variables is now on the orer of (k 1) n 2 instea of n k. It also allows us to struture our objetive funtion as ab x ab (17) a = a = a b ab f ab g b h (18) b f ab g b h ab (19) b (20) For a general k-frame problem with trajetory variables an assoiate osts i1i 2...i k, we write the eision variable fatorization as = k 1 z j j+1 (21) where notation z j j+1 is introue to represent the fatore eision variables on graph eges. The supersripts enote an ege that runs between frame j an frame j + 1, an the supersripts enote that it is onneting observation i j (in frame j) to observation i j+1 (in frame j + 1). For example, z represents an ege onneting the thir observation in frame 1 with the fourth observation in frame 2. We then have i1i 2...i k (22) i 2 i k i 1 = i 1 i 2 = i 1 k 1 i1i 2...i k i k z 1 2 i 1i 2 z 2 3 i 2 i 3 i 2i 3 i k z j j+1 (23) z k 1 k i k 1 i k i1i 2...i k (24) Note that the ost funtion i1i 2...i k is still a general funtion ompute with respet to an entire trajetory. The onstraints also nee to be rewritten in terms of the new, fatore eision variables, but this is easily ahieve base on properties of the esire set partitioning. That is, for eah noe representing an observation, the sum of eision variables entering the noe an the sum of eision variables exiting the noe must be exatly one. Note that this is a stronger onstraint than the typial flow onservation onstraints in network flow, where the sum of flow into an out of a noe an be either 0 or 1. The ifferene arises beause our approah buils upon a set partitioning formulation rather than a set paking one. Loal improvement heuristi: We are now in a position to esribe our loal improvement heuristi, again illustrate with the 4-frame problem above. Assume we alreay have a feasible solution, meaning a binary labeling of eision variables f ab, g b an h satisfying all of the path isjointness onstraints. Let the value of the objetive funtion for this solution be C. Without loss of generality, hol the labelings of all variables f ab an h fixe, an onsier hanges only to variables g b representing eges between pairs of observations in frames 2 an 3. Define a b i to be the unique element {a f abi = 1}, an similarly j to be { h j = 1 }. Then ab f ab g b h (25) a b = b = b = b g b f ab h ab (26) a g b (a b)b( ) (27) g b ω(b, ) (28) Example: Consier two targets viewe through four frames as shown in the sketh below. The thik lines represent variable values being hel fixe, i.e. f 11 = f 22 = h 12 = h 21 = 1 an all other f an h eision variables are 0. The thin ashe lines represent the variables g 11, g 12, g 21, g 22 that we want to upate. Applying the reution from Eq 28 proues the two-frame ost matrix [ ] 1112 ω(b, ) = (29) It is easily seen that Eq 28 is equivalent to a weighte maximum mathing in a bipartite graph, aka the two-frame linear assignment problem. This subproblem an be solve exatly, in polynomial time, by a retangular-matrix variant of the Kuhn-Munkres Hungarian algorithm [8]. Our propose solution makes a series of iterations. At eah iteration we step through pairs of ajaent frames

6 from 1-2 through (k-1)-k, upating the ege eision variables between them while holing all other eision variables fixe. An iteration thus onsists of (k-1) two-frame linear assignment upates. In general notation, an upate step involves solving for eision variables z f f+1 i f i f+1 between frames f an f + 1 holing all other variables fixe. Define {p i f } to be the urrent subpath from frame 1 to f ening in observation i f. Similarly, efine {i f+1 q} to be the urrent subpath from observation i f+1 to frame k. The upate step reues to a two-frame assignment with objetive funtion i f z f f+1 i f i f+1 {p if }i f i f+1 {i f+1 q}. (30) i f+1 It is important to note that the objetive funtion value C after an upate step an be no worse than the value C of the urrent solution, that is, C C. This is so beause the urrent solution is among the set of solutions onsiere by eah reue two-frame assignment problem. As suh, blok-icm inherits the onvergene properties of regular ICM, in that it is guarantee to eventually onverge to a (loal) optimum [5]. We also fin experimentally that onvergene ours rapily, usually within 5 iterations Deomposable Cost Funtions Although we have maintaine the ability to use ost funtions efine over arbitrarily long trajetories, many ost funtions enountere in pratie are efine over smaller temporal winows. In these situations, aitional eomposition of the objetive funtion may be possible. For example, onsier the ase of ost funtions that eompose into a prout of pairwise osts i1i 2...i k = k 1 j j+1. (31) In this speial ase, Eq. 24 further simplifies to i 1 = i 1 z 1 2 i 1i 2 z 2 3 i 2 i 3 i 2i 3 z 1 2 i 1i 2 1 i 2 1i 2 z 2 3 i 2i i 2 i 3 k 1 z k 1 k i k 1 i k i k i 2i 3 i k j j+1 (32) z k 1 k i k 1 i k k 1 k i k 1 i k (33) an we see that eah ost fator is now paire with an assoiate ege eision variable. The objetive funtion therefore an be ompletely represente by a graph with weighte eges, as in the network flow formalism of [3]. This explains more preisely the relationship, as well as the limitations, of network flow ata assoiation with respet to the general ata assoiation problem. Network flow is a speial ase that only hanles ost funtions that an be fatore into a prout of osts ompute in sequene between pairs of observations along a hypothesize trajetory. As we have alreay state, this type of ost funtion has limite ability to represent motion moels, as there is little that an be ompute from pairs of loations other than istane Initialization an Termination Our approah is an iterative improvement strategy, an therefore requires an initial feasible solution to get starte. Beause it is monotonially improving the solution (i.e. hilllimbing), starting with a goo initial solution shoul yiel onvergene to a better loal optimum. In our experiments we use a greey baseline algorithm (Setion 4) for initialization this algorithm makes a series of bipartite assignments forwar in time while using onstant veloity motion preition. Typially, one runs an ICM-like algorithm until the value of the objetive funtion stops improving. However, we also note that our approah maintains a feasible solution at every step, unlike, say, Lagrangean relaxation [6]. In time-ritial appliations, one oul therefore use this approah as an anytime algorithm that an be terminate early while still proviing a usable result. 4. Experimental Evaluation In this setion we evaluate our propose iterative approximation approah against two baseline algorithms representing ommonly-use alternatives. The first, alle Flow, is a network flow approah using pairwise istane as the ost of an ege between potential mathes in ajaent frames. Sine there is no appearane information being use, the objetive funtion to be minimize reues to fining the k shortest isjoint paths over all trajetories. The globally optimal solution to this objetive is being foun. The seon baseline algorithm, Greey, is a forwar sequential filtering algorithm inorporating onstant veloity motion preition. For eah urrent trajetory, the last two point loations efine a veloity estimate that is use to preit a virtual point loation in the next frame. Distanes between preite loations an observe targets form the ost matrix for a two-frame mathing problem, solve by the Hungarian algorithm. Trajetories for whih no mathes are foun are arrie forwar using the onstant veloity preition for a short perio of time, but eventually ie out. Observations for whih no trajetory mathes are foun are use to start new trajetories. One a eision has been mae at a time step, it is fixe an annot be unone Spline-base Snake Energy" Cost Funtion Motivate by the ative ontour moel of Kass, Witkin an Terzopoulos[12], we efine a spline-base ost funtion for our higher-orer motion moel : ost(p ) = αeont + βeurv (34)

7 where Eont = Eurv = 1 n 1 n p i p i 1 (35) i=2 n 1 p i+1 2p i + p i 1 2. (36) i=2 This is a variant of the internal energy term of a snake ative ontour, an is applie to eah hypothesize trajetory to ompute the ost of that path. Eont is the average istane between suessive pairs of points, penalizing large jumps in position. Eurv is a sum of urvature terms over the length of the trajetory. In our experiments we set α = β = 1. When there are only 2 points in the trajetory, Eurv = 0, an the ost reues to istane between the points. We o not use any training ata to tune ost funtion parameters, e.g. no istane or veloity threshols an no knowlege of entry or exit regions. Note that the urve of least energy with respet to Eurve is a natural ubi spline. Also, note that the urvature term is a finite ifferene omputation of aeleration, an sine we are minimizing, the objetive funtion will automatially prefer pieewise onstant veloity trajetories Evaluation As a testset for evaluation, we have ollete an grountruthe two atasets of trajetories from peestrians walking in an atrium 1. One is a relatively sparse sequene, with an average of 5 observe people per frame. The seon ense sequene is more hallenging, with roughly 20 people observe per frame. The number of people in eah frame is variable, sine iniviuals may enter an exit the view at any time in the sequene. Eah sequene is 15 minutes long, an human-labele annotations were use to generate groun truth loations of all people in every 10th frame, in a groun plane oorinate system. This trajetory ata was then broken into 20 seon sliing winows, eah overlapping by 10 seons, to provie a olletion of smaller sequenes for testing. To stuy the effet of sampling rate, we subsample the ata into test sets with 3 observations per seon, 2 observations per seon an 1 observation per seon, expeting problems with a lower temporal sampling rate to be more iffiult. Table 1 shows a quantitative omparison of traking performane. Algorithms evaluate are network flow (Flow), greey sequential filtering (Greey), an our blok-icm approah (Ours) using the snake energy ost funtion. Although both Greey an our approah use onstant veloity motion information, the snake energy moel applies it to evaluate smoothness of entire trajetories, rather than to provie an inter-frame mathing riterion. 1 Dataset available from Table 1. Mismath error perentage for network flow (Flow), greey forwar sequential filtering (Greey) an our approah (Ours) using the snake energy ost funtion. Smaller numbers are better. Approahes are ompare on sparsely an ensely populate sequenes, for sampling rates ranging from 1 to 3 frames per seon. Also shown in parentheses for our approah is the average number of iterations to onvergene. Sparse Trajetories Dense Trajetories Flow Greey Ours Flow Greey Ours 3fps (1) (2) 2fps (1) (2) 1fps (2) (5) The error measure use is total mismath error perentage (mmep), whih is one omponent of the Multiple Objet Traking Auray (MOTA) error measure ommonly use by the target traking ommunity [4]. It is ompute as follows: let g(t) be the number of groun truth targets at time t, an let mme(t) be the number of mismathes (aka ientity swaps) that ourre at time t within the estimate trajetories. Mismath perentage mmep is then ompute as mmep = 100 ( t mme(t)/ t g(t)). A higher mismath perentage means a higher number of ases where an estimate trajetory inorretly jumps from one iniviual to another uring traking. Sample images omparing network flow results with our results are shown in Figure 2. Figure 2. Sample trajetories. Far left: network flow results, with 22 ID swaps. Seon from left: Our approah using snake energy on the same image yiels 2 ID swaps. Seon from right: network flow on a enser sequene, with 116 ID swaps. Far right: Our approah on the same image yiels 16 ID swaps. All trajetories are olor-oe with respet to groun truth; eges of goo trajetories appear in the same olor as the observations they onnet. Disussion: As expete, all methos perform better when the sampling rate is higher. However, at all sampling rates, even greey forwar seletion outperforms the globally optimal shortest path solution ompute by network flow, emonstrating the benefits of a onstant veloity motion moel for reuing mismath errors. The improvement in performane is partiularly notable for lower frame rates an for sequenes with larger numbers of losely-spae objets. Table 1 also shows average number of iterations until onvergene for our iterative approah. These numbers onfirm our observations that the blok-icm algorithm onverges quikly.

8 5. Summary an Future Work Starting with a generalize MDA framework [18], we fator the trajetory eision variables into a prout of variables on eges in a k-partite trellis network representing multiframe observations. However, unlike network flow formalisms, we retain fully general ost funtions that an use higher-orer motion moels to evaluate path quality. Although our problem formulation is NP-har, we have propose an iterative approximate solution metho similar to ICM that yles through pairs of ajaent frames, omputing blok-optimal two-frame linear assignment solutions while holing all other eision variables fixe. The metho is guarantee to onverge, an usually onverges rapily. Using a snake energy trajetory ost funtion, our approah has been shown to outperform two ommon baseline algorithms for ata assoiation. Our use of general ost funtions allows evaluation of path quality over an entire trajetory of state vetors. In this paper we have only use objet loation as the state vetor in orer to isolate the effets of kinemati motion features from the onfouning effets of appearane information. However, it is trivial to exten our approah to inlue shape an appearane information by augmenting eah observation state vetor with aitional features suh as bouning box with an height, etetor onfiene sores, normalize olor histograms, or HOG esriptors. The ost funtion woul then be able to ompute aitional quality measures base on average etetor onfiene, smoothness of variation of bouning box size/shape, an variane of appearane of the target with respet to its mean appearane over the whole trajetory. To properly inorporate this aitional information, we will nee to speify or learn relative weights for fusing ifferent types of appearane, shape an motion ues into a single path quality sore. Referenes [1] M. Anriluka, S. Roth, an B. Shiele. People-traking-byetetion an people-etetion-by-traking. IEEE Conf on Computer Vision an Pattern Reognition, June [2] E. Balas an M. Paberg. Set partitioning: A survey. SIAM Review, 18: , [3] J. Berlaz, F. Fleuret, E. Türetken, an P. Fua. Multiple objet traking using k-shortest paths optimization. IEEE Transations on Pattern Analysis an Mahine Intelligene, 33: , September , 2, 3, 6 [4] K. Bernarin an R. Stiefelhagen. Evaluating multiple objet traking performane: The CLEAR MOT metris. EURASIP Journal on Image an Vieo Proessing, Speial Issue on Vieo Traking in Complex Senes for Surveillane Appliations, 2008, May Artile ID [5] J. Besag. On the statistial analysis of irty pitures. Journal of the Royal Statistial Soiety, Series B, 48(1): , , 6 [6] S. Blakman an R. Popoli. Design an Analysis of Moern Traking Sys. Arteh House, Norwoo, MA, , 3, 4, 6 [7] W. Brenel, M. Amer, an S. Toorovi. Multiobjet traking as maximum weight inepenent set. In IEEE Conf on Computer Vision an Pattern Reognition, pages , , 3 [8] R. Burkar, M. Dell Amio, an S. Martello. Assignment Problems. Soiety for Inustrial an Applie Mathematis, Philaelphia, PA, , 5 [9] M. Cho, J. Lee, an K. M. Lee. Reweighte ranom walks for graph mathing. European Conferene on Computer Vision, pages , [10] I.J.Cox an S.L.Hingorani. An effiient implementation of rei s multiple hypothesis traking algorithm an its evaluation for the purpose of visual traking. IEEE Trans on Pat Analysis an Mah Intell, 18(2): , , 4 [11] H. Jiang, S. Fels, an J. J. Little. A linear programming approah for multiple objet traking. In IEEE Computer Vision an Pattern Reognition, pages , [12] M. Kass, A. Witkin, an D. Terzopoulos. Snakes: Ative ontour moels. International Journal of Computer Vision, 1(4): , [13] B. Leibe, K. Shinler, an L. J. V. Gool. Couple etetion an trajetory estimation for multi-objet traking. International Conferene on Computer Vision, pages 1 8, [14] C. Morefiel. Appliation of 0-1 integer programming to multitarget traking problems. IEEE Transations on Automati Control, 22(3): , June [15] S. Oh, S. Russell, an S. Sastry. Markov hain monte arlo ata assoiation for multi-target traking. IEEE Trans on Automati Control, 54(3): , Marh [16] D. J. Papageorgiou an M. R. Salpukas. The maximum weight inepenent set problem for ata assoiation in multiple hypothesis traking. Optimization an Cooperative Control Strategies, Leture Notes in Control an Information Sienes, 381: , , 3 [17] H. Pirsiavash, D. Ramanan, an C. Fowlkes. Globallyoptimal greey algorithms for traking a variable number of objets. IEEE Conf on Computer Vision an Pattern Reognition, pages , June , 2, 3 [18] A. B. Poore. Multiimensional assignment formulation of ata assoiation problems arising from multitarget an multisensor traking. Computational Optimization an Appliations, 3:27 57, Marh , 4, 8 [19] C. Rasmussen an G. Hager. Probabilisti ata assoiation methos for traking omplex visual objets. IEEE Trans on Pat Analysis an Mah Intell, 23(6): , June [20] S.Deb, M.Yeanapui, K.Pittipati, an Y.Bar-Shalom. A generalize S-D assignment algorithm for multisensormultitarget state estimation. IEEE Trans on Aerospae an Eletroni Systems, 33(2): , April , 4 [21] K. Shafique an M. Shah. A noniterative greey algorithm for multiframe point orresponene. IEEE Trans on Pattern Analysis an Mahine Intelligene, 27(1):51 65, [22] L. Zhang, Y. Li, an R. Nevatia. Global ata assoiation for multi-objet traking using network flows. IEEE Conf on Computer Vision an Pattern Reognition, pages , June , 2, 3