Tracking a Large Number of Objects from Multiple Views

Transcription

1 Boson Universiy Compuer Science Deparmen Technical Repor BUCS-TR Tracking a Large Number of Objecs from Muliple Views Zheng Wu 1, Nickolay I. Hrisov 2, Tyson L. Hedrick 3, Thomas H. Kun 2, Margri Beke 1 1 Deparmen of Compuer Science, Boson Universiy 2 Deparmen of Biology, Boson Universiy 3 Deparmen of Biology, Universiy of Norh Carolina a Chapel Hill Absrac We propose a muli-objec muli-camera framework for racking large numbers of ighly-spaced objecs ha rapidly move in hree dimensions. We formulae he problem of finding correspondences across muliple views as a mulidimensional assignmen problem and use a greedy randomied adapive search procedure o solve his NPhard problem efficienly. To accoun for occlusions, we relax he one-o-one consrain ha one measuremen corresponds o one objec and ieraively solve he relaxed assignmen problem. Afer correspondences are esablished, objec rajecories are esimaed by sereoscopic reconsrucion using an epipolar-neighborhood search. We embedded our mehod ino a racker-o-racker muli-view fusion sysem ha no only obains he hree-dimensional rajecories of closely-moving objecs bu also accuraely seles rack uncerainies ha could no be resolved from single views due o occlusion. We conduced experimens o validae our greedy assignmen procedure and our echnique o recover from occlusions. We successfully rack hundreds of flying bas and provide an analysis of heir group behavior based on 150 reconsruced 3D rajecories. 1. Inroducion The inerpreaion of he moion of large groups of individuals is a difficul problem in compuer vision. A complee racking sysem ypically consiss of wo phases: esimaion of he sae of each objec and across-ime daa associaion (i.e., he assignmen of curren measuremens o objec racks). Sae esimaion is difficul when objec moion is no smooh; daa associaion is difficul when he populaion of objecs is dense. This paper sresses he laer scenario in a muli-view seing. This means we also This repor will appear in he proceedings of The Twelfh IEEE Inernaional Conference on Compuer Vision, Kyoo, Japan, Sepember wuheng@cs.bu.edu, hp:// need o consider an across-view daa associaion problem: he deerminaion of corresponding measuremens in muliple views. Tracking is challenging here because i involves solving he problem of maching hundreds of deeced individuals from frame o frame and from camera view o camera view and reasoning abou heir occlusions. Pas effors have incorporaed models of he occlusion process [15] or he ineracion of individuals [12, 22], knowledge abou he appearance of he objecs [14, 10] or he homography of he scene [11, 8], or have applied rajecory relinking [21, 16, 23]. The racking scenarios ha have been considered in he pas have ypically involved inerpreing he aciviies of fewer han en individuals per image frame. Earlier mehods ypically do no scale well in cases when here are hundreds of objecs moving in hree-dimensional (3D) space and where objecs differ by only a few visual cues. Our work, on he oher hand, falls in he caegory of recen research effors o undersand he ineracion of significanly larger crowds of individuals [1, 2, 5, 6, 17]. Our conribuions are: A new formulaion for across-view daa-associaion in large crowds using a likelihood funcion ha is based on muli-view geomery. A new ieraive search procedure (IGRASP) o solve he across-view daa-associaion problem. A sereoscopic mehod o reconsruc he rajecories of objecs moving in 3D space ha employs a new epipolar-neighborhood search. A new informaion fusion echnique ha ensures inerpreaion of occlusion and consisency of racking. We formulae he problem of finding objec correspondences across muliple views as a mulidimensional assignmen problem. This problem is known o be NP-hard, bu here are subopimal algorihms ha can deermine assignmens efficienly. To handle scenarios where objecs occlude each oher, we modified a greedy randomied adapive search algorihm [18] ha does no adhere o he radi-

2 ional one-o-one correspondence assumpion ha each objec is represened by one measuremen. Afer esablishing correspondences beween he measuremens in each view and he objecs ha are being racked, our mehod compues he 3D rajecories of he objecs via sereoscopic reconsrucion. The accuracy of racking in each camera view was improved by examining he consisency of racker-oracker associaions. We incorporaed our algorihms ino a racking sysem ha can successfully reason abou he movemens of hundreds of individuals recorded from muliple views. In single-view racking, ambiguiy caused by occlusion can be solved by opimiing some global funcion ha considers rajecory smoohness over several frames. Wih his approach, rajecory pieces ( rackles ) are linked successfully and full rajecories can be recovered (e.g., [21, 16, 23]). The approach assumes ha he occlusion will disappear wihin he ypical racking period. However, his assumpion does no hold in siuaions when hundreds of objecs emerge a he same ime in he scene and occlusion occurs consanly, and for hese siuaions, single-view approaches are no promising. An alernaive way is using more han one camera o provide racking informaion from differen views [13]. Mos of he previous muli-view works on racking pedesrians use homography [8, 11] as a naural and effecive approach o find he correspondence across differen views. Occlusion can hen be resolved even if he objec is compleely occluded in some views. Exending homography-based approaches o he case when objecs are no moving in he plane, as in our case, is no inuiive. We sress he difficulies of our racking problem: The objecs are no easily disinguishable based on appearance, and, wih a large number of objecs moving in 3D space, occlusion frequenly occurs. This problem is relevan for he analysis of group behavior of animals [5, 12, 19], he applicaion we chose in his paper, and for rajecory-based abnormaliy deecion in surveillance sudies [1, 2, 6, 20]. The resuls of surveillance or animal-analysis sysems usually depend on he rajecories ha he racker produced. Our racking sysem can herefore have an impac in hese applicaions when i uses, as a pos processing sep, he same approaches o rajecory analysis. Our experimens show no jus he effeciveness of our racking sysem, bu also provide informaion valuable o mammalogiss, ecologis, and conservaion biologiss. In paricular, we produced he firs sereoscopic analysis of he emergence behavior of free-ranging bas. We repor he firs accurae and reproducible esimaes of 3D velociies of groups of emerging bas and heir spaio-emporal ineracions. Camera 1 Tracks in View 1 Camera 2 Tracks in View 2 Camera N Tracks in View N Cenral Node Across view Associaion Track o rack Fusion Trajecory Reconsrucion Figure 1. The hybrid archiecure of our racking sysem. Tracking is performed a each sensor level and racks and measuremens are sen o a cenral node for processing. Each sensor racker adjuss is across-ime associaions based on he fusion resul i receives from he cenral node. 2. Muli-objec Muli-view Tracking We firs describe our muli-objec racking approach and formulae he muli-view daa-associaion problem (Sec. 2.1). We hen inroduce an ieraive search procedure o efficienly solve his NP-hard problem (Sec. 2.3). We use sereoscopic reconsrucion o combine he wodimensional rajecories from each view ino a single hreedimensional rajecory for each objec and inroduce he echnique epipolar-neighborhood search (Sec. 2.2). We explain how we ensure he consisency of his sensor fusion in he presence of occlusions (Sec. 2.4). The archiecure of our racking sysem is shown in Fig Mulidimensional Assignmen Formulaion In his secion, we describe how we adaped he recursive Bayesian echniques from he radar lieraure [4, 7] o address he across-camera daa associaion problem. Our conribuion includes a formulaion of he likelihood funcion ha is based on muli-view geomery. The funcion deermines how likely i is ha associaed 2D measuremens are projecions of an objec in he 3D scene. Given N calibraed and synchronied cameras ha share overlapping fields of view and n s measuremens in he field of view of camera s, he sae x () (3D coordinaes) of an objec of ineres a ime can be assumed o evolve in ime according o he equaions as observed via measuremens x (+1) = Ax () + v () (1) () s,i s = H s x () + w (s) for s = 1,..., N, i s = 1,..., n s, (2) where v () and w (s) are independen ero-mean Gaussian noise processes wih respecive covariances Q() and R s (), A is he sae ransiion marix, and H s he projecion marix for camera s. Each measuremen () s,i s is eiher he projeced 2

3 image of some objec a in camera s plus addiive Gaussian noise N(0, R s ()), or a false-posiive deecion, which is assumed o occur uniformly wihin he field of view of camera s. For each camera, he deecion rae is P Ds < 1. We add dummy measuremens () s,0 o handle he case of missed deecions. In paricular, when objec a is no deeced in camera s a ime, dummy measuremen () s,0 from camera s is associaed wih objec a. For ease of noaion, we now drop he superscrip. We use he noaion Z i1i 2...i N o indicae ha he measuremens 1,i1, 2,i2,..., N,iN originaed from a common objec in he scene a ime. The likelihood ha Z i1i 2...i N describes objec sae x a is given as p(z i1i 2...i N x a ) = N {[1 P Ds ] 1 u(is) s=1 [P Ds p( s,is x a )] u(is) } (3) where u(i s ) is an indicaor funcion defined as u(i s ) = { 0 if is = 0 1 oherwise, and he condiional probabiliy densiy of a measuremen s,is, given i originaed from objec a, is (4) p( s,is x a ) = N( s,is ; H s x a, R s ). (5) The likelihood ha Z i1i 2...i N is unrelaed o objec a or relaed o dummy objec is p(z i1i 2...i N ) = N [ 1 ] u(is), (6) Φ s s=1 where Φ s is he volume of he field of view of camera s. Since we do no know he rue sae x a in Eq. 5, we replace i by ˆx a = arg min x a n d( s,is, H s x a ), (7) s=1 where d is Euclidean disance beween H s x a, he objec posiion projeced ono he image plane s, and he corresponding measuremen s,is. Using sereoscopy, 1 we esimae he sae ˆx a o be he reconsruced 3D posiion based on he corresponding measuremens 1,i1, 2,i2,..., N,iN in he N views. We now can define he cos of associaing N-uple Zi 1i 2...i N o objec a a ime is as he negaive 1 We seleced he Direc Linear Transformaion (DLT) algorihm [9] o perform he 3D reconsrucion because of is efficiency and sufficien accuracy. Oher mehods may replace DLT in our framework. log-likelihood raio: 2 c i1i 2...i N = ln p (Z i 1i 2...i N a) p (Zi 1i 2...i N ) N = {[u(i s ) 1] ln(1 P Ds ) s=1 u(i s )ln ( ) PDs Φ s 2πR s 1/2 +u(i s )[ 1 2 ( s,i s H sˆx a ) T R 1 s ( s,i s H sˆx a )]} (8) We use binary variable x i1i 2...i N o indicae if Z i1i 2...i N is associaed wih a candidae objec or no. Assuming ha such associaions are independen, our goal is o find he mos likely se of n-uples ha minimies he linear cos funcion c = min s.. n 1 n 2... n N i 1=0 i 2=0 i N=0 n 2 n 3 n N i 2=0 i 3=0 n 1 n i 1=0 i 3=0 n 1 n 2 i 1=0 i 2=0 i N=0 n N i N=0 i N 1=0 c i1i 2...i N x i1i 2...i N (9) x i1i 2...i N = 1; i 1 = 1, 2,..., n 1 x i1i 2...i N = 1; i 2 = 1, 2,..., n 2. n N 1... x i1i 2...i N = 1; i N = 1, 2,..., n N. Eq. 9 is known as he mulidimensional assignmen problem, which i is NP-hard for he dimension N 3. The processing ime for he opimal soluion is unaccepable in dense racking scenarios, even if a branch-and-bound search mehod is used, because such a mehod is ineviably enumeraive in naure. The alernaive is o search for a sub-opimal soluion o his combinaorial problem, using greedy approaches and is varians, Lagrangian relaxaion, simulaed annealing or abu search. We choose he Greedy Randomied Adapive Search Procedure (GRASP) [18] as he basic paradigm and modified i o handle occlusion reasoning (Sec. 2.3) Generic GRASP in Muli-view Scenario We briefly ouline a generic GRASP implemenaion for he mulidimensional assignmen problem [18] and hen adjus i o our muli-view scenario: In he local search phase, we adop he so-called 2- assignmen-exchange operaion. Tha is, for wo uples 2 We can append o his cos funcion oher ypes of coss, e.g., he measures of objec appearance, if such measures are available, and define a reasonable weighing scheme o yield normaliaion. 3

4 GREEDY RANDOMIZED ADAPTIVE SEARCH PROCEDURE: Iniialiaion by compuing he coss for all possible associaions For i = 1,..., maxier o 1 o 2 1. Randomly consruc a feasible greedy soluion, 2. Recursively improve he feasible soluion by local search, 3. Updae he bes soluion by comparing he oal coss, 1,1 1,3 2,1 2,2 Oupu he bes soluion found so far. Z i1...i j...i N and Z i 1...i from he feasible soluion, we j...i N exchange he assignmen o Z i1...i and Z j...in i if 1...ij...i N such operaion decreases he oal cos. The exchange akes place recursively unil no exchange can be made anymore. We adop a echnique similar o gaing during he iniialiaion sep o reduce he number of possible candidae uples as follows. Given a pair of calibraed views, our echnique esablishes he correspondence of he wo projeced images of an objec using epipolar geomery. Thus, we only need o evaluae he candidae uples ha lie wihin he neighborhood of corresponding epipolar lines. To enforce his neighborhood search, we se he cos of associaing measuremens ha violae he epipolar-geomery consrains o a large number. This pruning sep in building he mulidimensional assignmen problem, which we call epipolar-neighborhood search, becomes crucial for he overall efficiency, which will be demonsraed in Sec Ieraive GRASP in Muli-view Scenario The consrains in Eq. 9 imply he one-o-one correspondence beween measuremens and objecs, excep for he dummy measuremen and is corresponding objec. Each measuremen is eiher assigned o some objec or claimed o be a false-posiive deecion. An objec is eiher measured in each view or i is missed. This sric formulaion is no desirable in he muli-view racking scenario, as shown in Fig. 2. Wih he one-o-one correspondence consrain, he numeric opimal soluion migh associae ( 1,1, 2,1 ) o objec o 1 and ( 1,3, 2,2 ) o objec o 2 or decide objec o 2 is no deeced in view 1. This ambiguiy is difficul o resolve since boh inerpreaions have accepable oal coss. Our basic assumpion is ha if an occlusion occurs in one view, i does no happen in oher views a he same ime. This requires ha we relax he one-o-one correspondence consrain: Measuremens ha overlap due o occlusion or imperfec segmenaion during he deecion sage and hus are inerpreed as a single measuremen can be assigned o muliple objecs. We denoe he se of all possible N-uples as F = Z 1... Z i... Z N, where Z i is he se of all he measuremens in view i plus he dummy measuremen. Solving Eq. 9 yields a se of assignmens for he N-image measure- Figure 2. Sereoscopic reasoning for assessing occlusion. From a single view, wo objecs o 1 and o 2 occlude each oher and yield a single measuremen 1,1. A single-view racker may lose rack of one of he objecs or may misinerpre he nearby false-posiive deecion 1,3 as one of he objecs. If wo views are available, he objecs o 1 and o 2 can be mached o heir respecive measuremens 2,1 and 2,2. Sereoscopic reasoning reveals ha 1,1 is he image of boh objecs and 1,3 an unrelaed measuremen. men se Z, where a specific assignmen can be wrien as { i1i 2...i N x i1i 2...i N = 1}. We divide he se of assignmens ino wo subses as follows: 1. Confirmed associaions: M c = {Z i1i 2...i N x i1i 2...i N = 1; i 1 0;...; i N 0}. 2. Suspicious associaions: M s = Z \ M c. Suspicious associaions involve boh dummy measuremens s,0 ha indicae an objec was no deeced in some view and measuremens ha were assigned o he dummy objec (i.e., false posiive deecions). Eq. 9 does no conain consrains wih eros for index i. Associaions in se M s have a leas one ero in heir subscrips. The new version of GRASP ha we propose here (see pseudocode for Ieraive GRASP below) compues a soluion o an assignmen problem ha is described by Eq. 9, excep wih he already confirmed assignmens in M c removed from he feasible assignmen se F. During an ieraion of Ieraive GRASP, an assignmen found greedily in he consrucion phase can hus no involve a uple already in M c. The algorihm generaes wo subses from he resuling soluion and ieraes unil a maximum number of ieraion is reached or M c in he curren ieraion is empy Muli-Objec Tracking wih Fusion of Informaion from Muliple Views Thus far we described a mehod o solve muli-view daa associaion in a single ime sep. The resuling soluion allows us o esimae he curren 3D posiion of each objec in he scene using Eq. 7, which selecs he 3D posiion ha minimies he sum of he sereoscopic reconsrucion errors 4

5 ITERATIVE GREEDY RANDOMIZED ADAPTIVE SEARCH PRO- CEDURE (IGRASP): Building Phase Iniialiaion by compuing he coss for all possible associaions in se F ; Solving Phase For i = 1,..., maxier 1. Formulae mulidimensional assignmen problem on se F, 2. Run sandard GRASP described in Sec. 2.2, 3. Pariion he compued soluion ino confirmed se M c and suspicious se M s. 4. If Se M c is empy, erminae; else F = F \ M c Oupu he bes soluion found so far. compued for each view. To consruc 3D objec rajecories, we mus o solve anoher daa associaion problem, he assignmen of curren 3D objec posiions o he 3D racks esablished in previous ime seps. We can solve his problem indirecly by deermining, for each of he N camera views separaely, he assignmen of he 2D projecions of curren objec posiions o he 2D racks esablished in previous ime seps. For each objec in each camera view, we use a 2D Kalman filer o predic he objec posiion in he nex frame. Across-frame daa associaion can hen be accomplished by maching each 2D objec rack o he 2D measuremen ha is closes o he prediced 2D objec posiion. The 2D across-ime daa associaion mehod will likely resul in ambiguiies and mismaches due o occlusions in densely populaed scenes. If objecs do no disinguish hemselves by unique moving direcions, he occlusions mus be resolved o preven rack los or rack swich. We herefore analye he across-view correspondences, esablished wih IGRASP in each ime sep, which should be consisen hrough ime. In paricular, measuremens of an objec ha are associaed a ime should correspond o racks ha have been associaed a ime 1. We mainain a consisency able during racking ha records he consisency of correspondence across views (Fig. 3). If some measuremens ( () 1,i 1, () 2,i 2,..., () N,i N ) are associaed a curren ime sep, heir associaed 2D rackers (f 1,j1, f 2,j2,..., f N,jN ) form an enry in he consisency able. Here he racker f k,jk racks measuremen k,ik in view k independenly. If he 2D rackers perform well, his enry should be mainained in he able unil some racker ends. However, if some 2D racker incorrecly associaes a measuremen in is own view, i will be correced by looking a he corresponding enry in he able and hisorically comparing is consisency. The correcion is performed only when he associaions across a leas N/2 Camera 1 1,1 1,1 1,2 Camera 2 2,1 2,1 2,2 2,2 Camera 3 3,1 3,1 Track Consisency Table 3,2 3,2 ( ) 1,1 2,1 3,1 ( ) 1,2 2,2 3,2 Figure 3. Example of he racker-o-racker fusion. Two objecs are observed in hree cameras, and here are wo separae 2D rackers f i,1, f i,2 for each camera i. Based on he racking hisory, he racker-o-racker associaions are mainained in he rack consisency able, e.g., racker f 1,1 from camera 1, racker f 2,1 from camera 2, racker f 3,1 from camera 3 are associaed o form an enry in he able. When occlusion occurs in camera 1, eiher f 1,1 or f 1,2 will lose rack when hey compee for measuremen 1,1. However, by looking a he soluion of associaion across views {( 1,1, 2,1, 3,1),( 1,1, 2,2, 3,2)} and checking he enries in he able, he 2D racker ha iniially los he compeiion for 1,1 in camera 1 can recover and claim measuremen 1,1. As a resul, 1,1 is associaed wih he racks mainained by boh rackers. By a similar mechanism, our sysem recovers from he rack-swich problem. views are consisen (e.g., in 3-camera case, wo consisen inerpreaions are needed). The consisency able also provides a good parial feasible soluion for he assignmen problem because measuremens racked by esablished rackers (f 1,j1, f 2,j2,..., f N,jN ) are very likely o be associaed again. By comparing he assignmens compued by IGRASP wih he rack uples lised in he consisency able, our sysem can also preven assignmens ha could misakenly resul in a swich of racks. The idea behind our mehod is essenially racker-oracker sensor fusion. We mainain sensor-level rackers for each view and adjus heir individual esimaions afer finding correspondences across views. Our disribued racking syle is exremely imporan if he communicaion overload or burden of a cenral compuing node need o be minimied. The alernaive is o collec all measuremens from each view, reconsruc heir 3D posiions, and apply recursive Bayesian racking in 3D space. We do no currenly follow his cenralied syle because he reconsruced 3D posiions are no sufficienly accurae due o sub-opimal across-view associaions and deecion errors. Fuure work will compare he performance of he wo approaches. 3. Experimens and Resuls Observing he fligh behavior of large groups of bas or birds is fascinaing heir fas, collecive movemens provide some of he mos impressive displays of naure. Quaniaive sudies of cooperaive animal behavior have ypi- 5

6 OCCLUSION REASONING FOR 3 VIEWS IN ONE TIME STEP Inpu: Curren measuremens { s,is } and 2D-racks {f s,is } I. Wihin each view s, assign measuremens s,is o 2D-racks f s,js using biparie maching. II. Run IGRASP o find across-view associaions of measuremens {( 1,i1, 2,i2, 3,i3 )} and consruc he rack-o-rack associaions {(f 1,i1, f 2,i2, f 3,i3 )}. III. Check if he rack-o-rack associaion uples are consisen wih enries in he Track Consisency Table (Fig. 3). For each uple f {f 1,i1, f 2,i2, f 3,i3 }: New Track: If uple f consiss of a leas wo rack labels ha do no appear in he able, inser f ino he able as a new enry. Occlusion and Los Track: If uple f is parially mached o some enry in he able (i.e., 2 of 3 rack labels mach), is label f s,is differs from f s,i in he s able enry, and 2D-rack f s,i was found los in sep I, s hen assign measuremen s,is o f s,i. s For remaining uples, pair hem and check: Track Swich: If wo uples are parially mached o able enries a and b (i.e., 2 of 3 rack labels mach), label f s1,i s1 in uple 1 differs from he label in a, label f s2,i s2 in uple 2 differs from he label in b, and a rack label swich resuls in a mach for boh a and b, hen reassign s1,i s1 o f s2,i s2 and s2,i s2 o f s1,i s1. IV. Wihin each view, predic 2D-rack sae wih Kalman filer based on assignmens updaed in sep III. cally been limied o sparse groups of only a few individuals. The major limiaion in hese sudies has been he lack of ools o obain accurae 3D posiions of individuals in dense formaions. Alhough imporan progress has been made [3], robus general soluions o 3D racking, reconsrucion, and daa associaion have been lacking. In our experimens, we firs validaed our mehod using synheic daa for which we had ground ruh and hen applied i o infrared hermal video of colonies of Brailian free-ailed bas. We colleced his video while he colony was emerging from is cave roos a nigh. We reconsruced he 3D fligh pahs and hus provided he firs sereoscopic analysis of he emergence behavior of free-ranging bas Validaion of Across-View Daa Associaion We generaed synheic daa o es he performance of our IGRASP using a paricle dynamics environmen (Auodesk Maya). To simulae he scene near a cave in Texas where we recorded emerging bas, we generaed spherical paricles, 28 cm in radius, o move in a 20 x 5 x 5 m 3 space a a fixed speed of 2 m/s. We experimened wih incremenally increasing emergence raes beween 1 and 100 paricles per second. Sample images wih a high degree of densiy of paricles are shown in Fig. 4. The rajecories were randomied by placing an axial and radial consrain on he paricle movemen. Three virual cameras wih overlapping views were posiioned laerally and slighly below he average direcion of ravel of he paricles. Since he calibraion parameers for each camera and he 3D posiions of each paricle are known (i.e., he ground ruh ), we can es wheher our soluion of he mulidimensional assignmen problem (Eq. 9) maches paricles correcly ha are deeced in he hree views. Camera A Camera B Camera C Figure 4. Daa used for validaion of across-view associaion. The hree views show a scenario in which paricles emerged a he righ of he fields of view a a rae of 100 paricles/s and moved owards he lef side. We demonsrae he performance of our IGRASP as a funcion of differen paricle densiies in Fig. 5. As he number of paricles increases, an increasing number of paricles share overlapping regions in each field of view, which can hen be deeced as a single measuremen. We measure as he overlap densiy, he raio of number of overlapping paricle projecions over he oal number of paricles (Fig. 5 lef), and also he raio of correc maches as number of correc uples found by IGRASP over he ground ruh (Fig. 5 righ). Our resuls show ha even in very dense scenarios, IGRASP can recover up o 65% maches correcly. When 20 paricles/s are generaed, 105 paricles on average appear in a frame wih an overlap densiy of 16%, and 95 % of he maches IGRASP compues are correc. Overlap densiy Average number of paricles Raio of correc maches Average number of paricles Figure 5. Across-view daa associaion performance of IGRASP. IGRASP has very few parameers o be adjused. The execuion ime of he algorihm depends on he sparsiy of 6

7 he mulidimensional assignmen problem. We can use he epipolar consrain o reduce he number of feasible candidae uples (Sec. 2.1). This urns ou o be very imporan for he overall efficiency of he mehod. Compuing he cos for all feasible uples is much more expensive han deermining he assignmens (Fig. 6). We limi he coss of he ime-consuming Building Phase using a criical hreshold τ as follows: Only hose measuremens whose disances o he epipolar lines are wihin hreshold τ are considered o form feasible uples. A drawback of using a reduced feasible se is ha IGRASP may no find he opimal se of assignmens. Thus, parameer τ plays an imporan role in rading off accuracy and efficiency in dense racking scenarios. The number of ieraions maxier affecs he opimaliy of IGRASP: when he number becomes large, IGRASP approaches exhausive enumeraion. We se maxier = 20 hroughou our experimens because is increase did no improve he performance significanly. Execuion ime (s) Building Phase Solving Phase Parameer τ Execuion ime (s) Building Phase Solving Phase Parameer τ Figure 6. Execuion ime of IGRASP (our Malab version) wih differen values of τ for he across-view daa assignmen. Lef: 100 paricles/s. Righ: 50 paricles/s Infrared Thermal Video Analysis We recorded he emergence of a colony of Brailian freeailed bas from a naural cave in Blanco Couny, Texas. We used hree FLIR SC6000 hermal infrared cameras wih a resoluion of pixels a a frame rae of 125 H (Fig. 7). We implemened our algorihms in C++ and esed our sysem on a Inel Penium 2.36 GH plaform. Processing is performed in near real ime and depends on he densiy of he group (e.g., in a 100 bas/frame scenario, our sysem ook 3 s o process each frame). Our experimens showed ha we can rack each individual ba in he emergence column, reconsruc heir 3D fligh pahs, and provide insighs ino heir group behavior based on rajecory analysis. To deec moving bas, we applied adapive background subracion o idenify he conneced componens of ineres and hen used he pixel wih he highes inensiy wihin each componen as he posiion of a ba. We implemened 2D Kalman filers o rack bas in each view and solved he across-ime daa associaion wih biparie maching. If a 2D racker idenifies a rack loss, i keeps searching along he projeced 2D fligh pah of he ba for he nex 5 frames Camera 1 (m) y (m) 1 Camera x (m) 3D Trajecories of Bas Cave Enrance 4 6 Camera 3 Figure 7. Visualiaion of camera seup and 150 reconsruced 3D rajecories. The camera baselines are approximaely 1 m. We adjused camera pich, yaw, and roll o capure he full volume of he 3D column of emerging bas in overlapping field of views. In each view, here were as many as 200 bas a he same ime, wih an average sie of 5 5 pixels. The average speed of an emerging ba was 8.75 m/s. The average direcion of he emerging column can be described by he Euler angles 127, 97 and 38. The color differences in he hermal images are due o a lack of radiomeric calibraion of he cameras. o see if i can resume racking he ba or if i needs o wai for a reassignmen of a measuremen when across-view associaions are solved and racker-o-racker consisency is checked. Our resuls show ha our mehod correcly resolves ambiguiies due o occlusions (Fig. 8). However, we canno expec o resolve all ambiguiies in dense racking siuaions due o insufficien image resoluion. We invesigaed he performance of our sysem in resolving occlusions in scenarios wih four differen densiy levels of he column of emerging bas (Table 1). We couned he number of imes each 2D racker claimed o be los for all hree views. If he sysem could no resolve occlusion, i generaed a new racker once he bas were separaed again. The number of compued racks is herefore usually higher han he rue number of bas. In relaively sparse scenarios, our sysem successfully recovered from occlusions and avoided rack swiches (40/56=74%). In he highly dense cases, occlusions ypically occurred in wo or hree views a he same ime, and so i was significan ha we could correcly inerpre 88/368=24% of he occlusions. We reconsruced he full 3D rajecories of 150 bas and explored heir group behavior during emergence. We measured he average emerging speed of a ba o be 8.75 m/s ( 20 miles/h), which is consisen wih he low end of he range of emergence speeds repored in he mammalogy lieraure. We have also resolved he quesion abou he Euclidean disance beween emerging bas. Our resuls 7

8 Frame 043 Frame 049 Frame Camera A Camera A Camera A Average disance beween bas (m) Camera B Camera B Camera B Number of emerging bas per second Figure 9. Resuls of 3D rajecory analysis: Average disance beween emerging bas as a funcion of emerging rae, expressed by he average number of emerging bas per second. Camera C Camera C Camera C Figure 8. Occlusion Inerpreaion. Bas racked in infrared video from muliple views are shown as segmened foreground objecs (blue) wih heir racker number (whie). Frames 43, 49, and 63 are snapshos before, during, and afer occlusion occurred in he field of view of Camera B, involving four bas ha were flying close o each oher. In paricular, Bas 3 and 6 were difficul o disinguish in frame 49 recorded by camera B. Since heir projecions were well separaed in he oher wo views during he period of occlusion, our algorihm was able o correcly inerpre he occlusion by reasoning abou he 3D posiions of he four racked bas. The oupu from he algorihm indicaes ha Ba 3 occluded Ba 6 in frame 49 recorded by camera B. Table 1. Performance of racking sysem in resolving occlusions. Ground ruh was esablished by manual marking of four 100- frame sequences. Number of True Compued Number Number of Bas per Number Number of of Oc- Recovered Frame of Bas Tracks clusions Occlusions show ha when 1 15 bas emerge, heir average disance is 90 cm. The average disance drops o cm as soon as he emergence column conains more han 25 bas per second (Fig. 9). 4. Conclusions and Fuure Work Our experimens showed ha our mehod can reconsruc 3D rajecories of ighly-spaced, fas-moving objecs and can accuraely sele rack uncerainies ha could no be resolved from single views due o occlusion. Our work can be exended o incorporae boh acrossime and across-view associaions a he same ime in a single opimiaion framework. I would be ineresing o deermine wheher forward and backward inferences on he assignmen over ime could enhance he performance of our approach for highly dense groups. We also plan o do addiional daa mining on he group behavior of bas, once we generaed hundreds of housands of rajecories, which will be exremely valuable for scieniss in oher fields. Acknowledgemens This maerial is based upon work suppored by he Naional Science Foundaion under Grans and Any opinions, findings, and conclusions, or recommendaions expressed in he paper are hose of he auhors and do no necessarily reflec he views of he Naional Science Foundaion. References [1] S. Ali and M. Shah. Floor fields for racking in high densiy crowd scenes. In ECCV, [2] E. Andrade, S. Blunsden, and R. Fisher. Performance analysis of even deecion models in crowded scenes. In Workshop Towards Robus Visual Surveillance Techniques and Sysems a VIE, [3] M. Ballerini, N. Cabibbo, R. Candelier, A. Cavagna, E. Cisbani, I. Giardina, V. Lecome, A. Orlandi, G. Parisi, A. Procaccini, M. Viale, and V. Zdravkovic. Ineracion ruling animal collecive behavior depends on opological raher han meric disance: Evidence from a field sudy. Proceedings of he Naional Academy of Sciences, 105: , [4] Y. Bar-Shalom and X. R. Li. Muliarge - Mulisensor Tracking: Principles and Techniques. YBS Publishing, [5] M. Beke, D. E. Hirsh, A. Bagchi, N. I. Hrisov, N. C. Makris, and T. H. Kun. Tracking large variable numbers of objecs in cluer. In CVPR, [6] G. J. Brosow and R. Cipolla. Unsupervised Bayesian deecion of independen moion in crowds. In CVPR, [7] S. Deb, M. Yeddanapudi, K. Paipai, and Y. Bar-Shalom. A generalied s-d assignmen algorihm for mulisensormuliarge sae esimaion. IEEE Trans. AES, 33: ,

9 [8] R. Eshel and Y. Moses. Homography based muliple camera deecion and racking of people in a dense crowd. In CVPR, [9] R. I. Harley and A. Zisserman. Muliview view geomery in compuer vision. Cambridge Universiy Press, [10] Y. Huang and I. Essa. Tracking muliple objecs hrough occlusions. In CVPR, pages , [11] S. M. Khan and M. Shah. A muliview approach o racking people in crowded scenes using a planar homography consrain. In ECCV, pages , [12] Z. Khan, T. Balch, and F. Dellaer. MCMC daa associaion and sparse facoriaion updaing for real ime muliarge racking wih merged and muliple measuremens. IEEE Trans. PAMI, 28: , December [13] Y. Li, A. Hilon, and J. Illingworh. A relaxaion algorihm for real-ime muliple view 3d-racking. Image Vis Compu, 20: , [14] A. Mial and L. S. Davis. M2Tracker: A muli-view approach o segmening and racking people in a cluered scene. IJCV, 51(3): , [15] K. Osuka and N. Mukawa. Muliview occlusion analysis for racking densely populaed objecs based on 2-d visual angles. In CVPR, pages 90 97, [16] A. Perera, C. Srinivas, A. Hoogs, G. Brooksby, and W. Hu. Muli-objec racking hrough simulaneous long occlusions and spli-merge condiions. In CVPR, pages , [17] V. Rabaud and S. Belongie. Couning crowded moving objecs. In CVPR, [18] A. J. Roberson. A se of greedy randomied adapive local search procedure (GRASP) implemenaions for he mulidimensional assignmen problem. Compuaional Opimiaion and Applicaions, 19(2): , [19] A. Veeraraghavan, R. Chellappa, and M. Srinivasan. Shapeand-behavior-encoded racking of bee dances. IEEE Trans. PAMI, 30: , March [20] X. Wang, K. Ma, G. Ng, and W. Grimson. Trajecory analysis and semanic region modeling using a nonparameric bayesian model. In CVPR, [21] Q. Yu, G. Medioni, and I. Cohen. Muliple arge racking using spaio-emporal Markov Chain Mone Carlo daa associaion. In CVPR, pages 1 8, [22] T. Yu and Y. Wu. Collaboraive racking of muliple arges. In CVPR, pages , [23] L. Zhang, Y. Li, and R. Nevaia. Global daa associaion for muli-objec racking using nework flows. In CVPR,