MOTION TRACKING is a fundamental capability that

Transcription

1 TECHNICAL REPORT CRES , CENTER FOR ROBOTICS AND EMBEDDED SYSTEMS, UNIVERSITY OF SOUTHERN CALIFORNIA 1 Real-ime Moion Tracking from a Mobile Robo Boyoon Jung, Suden Member, IEEE, Gaurav S. Sukhame, Member, IEEE Absrac A mobile robo needs o perceive he moions of eernal objecs o perform asks successfully in a dynamic environmen. We propose a se of algorihms for muliple moion racking from a mobile robo equipped wih a monocular camera and a laser rangefinder. The key challenges are 1. o compensae he ego-moion of he robo for eernal moion deecion, and 2. o cope wih ransien and srucural noise for robus moion racking. In our algorihms, he robo ego-moion is direcly esimaed using corresponding feaure ses in wo consecuive images, and he posiion and velociy of a moving objec is esimaed in image space using muliple paricle filers. The esimaes are fused wih he deph informaion from he laser rangefinder o esimae he parial 3D posiion. The proposed algorihms have been esed wih various configuraions in oudoor environmens. The algorihms were deployed on hree differen plaforms; i was shown ha various ype of ego-moion were successfully eliminaed and he paricle filer was able o rack moions robusly. The muliple arge racking algorihm was esed for differen ypes of moions, and i was shown ha our muliple filer approach is effecive and robus. The racking algorihm was inegraed wih a robo conrol loop, and is realime capabiliy was demonsraed. Inde Terms mobile robo, moion racking, ego-moion compensaion, paricle filer. I. INTRODUCTION MOTION TRACKING is a fundamenal capabiliy ha a mobile robo mus have in order o operae in a dynamic environmen. Moving objecs (eg., people) are ofen subjecs for a robo o inerac wih, and in oher cones (eg., raffic) hey could be poenially more dangerous for safe navigaion compared o saionary objecs. Furher, capabiliies like localizaion and mapping criically depend on separaing moving objecs from saic ones. Finally moion is he mos criical feaure o rack for many surveillance or securiy applicaions. For insance, a building monioring sysem can wach for a burglar a nigh by deecing moion, or an auonomous rescue vehicle can search for vicims of naural disaser by sensing moion. Clearly, robus moion deecion and racking are key enablers for many mobile robo applicaions. The moion racking problem from a mobile robo is illusraed in Figure 1. There are muliple moving objecs in he viciniy of a mobile robo. Measuremens from sensors onboard he robo are conaminaed wih noise, and an esimaion process is required o compue he posiions and velociies of he moving objecs in he robo s local coordinae sysem. In he varian of he problem sudied here, we require realime esimaes wihou prior knowledge abou he number of moving objecs, he moion model of objecs, or he srucure of he environmen. As an addiional resricion, a populaed, Boyoon Jung and Gaurav S. Sukhame are wih he Roboic Embedded Sysems Laboraory, Cener for Roboics and Embedded Sysems, Universiy of Souhern California, Los Angeles, USA. boyoon,gaurav@roboics.usc.edu. Fig. 1. Moion racking from a mobile robo: The problem is o esimae he posiions and velociies of arge moions in he robo s local coordinae sysem. unsrucured oudoor environmen is assumed. Moving objec deecion in a srucured indoor environmen has been relaively well sudied [1], [2], [3]. In an indoor environmen, moving objecs are ofen clearly disinguishable from he res of he environmen due o disinc environmenal srucures (eg. sraigh and perpendicular walls). In conras, an oudoor environmen conains objecs of irregular shapes, and i is challenging o segmen moving objecs from he background. In addiion, oudoor environmens conain diverse moions (varying speeds, frequencies ec.). There are wo main challenges in he moion racking problem. Firs, here are wo independen moions involved - he ego-moion of he mobile robo and he eernal moions of moving objecs. Since hese wo moions appear blended in he sensor daa, he ego-moion of he robo needs o be eliminaed so ha he remaining moions, which are due o moving objecs, can be deeced. Second, here are various ypes of noise added a various sages. For eample, real oudoor images are conaminaed by various noise sources including poor lighing condiions, camera disorion, unsrucured and changing shape of objecs, ec. Perfec ego-moion compensaion is rarely achievable, hus i adds anoher ype of uncerainy o he sysem. Some of hese noise erms are ransien and some of hem are consan over ime. Our approach o he problem is o design a simple and fas ego-moion compensaion algorihm in he pre-processing sage for real-ime performance, and o develop a probabilisic filer in he pos-processing sage for uncerainy and noise handling. Since he sequence of camera images conains rich informaion of objec moion, a monocular camera is uilized for moion deecion and racking. A laser rangefinder provides deph informaion of image piels for parial 3D posiion esimaion. Figure 2 shows he processing sequence of our moion racking sysem. Frame differencing, which compares wo consecuive images and finds moions based on he difference, is eploied for moion deecion. However, when he camera moves (eg. when i is mouned on a mobile robo),

2 Repor Documenaion Page Form Approved OMB No Public reporing burden for he collecion of informaion is esimaed o average 1 hour per response, including he ime for reviewing insrucions, searching eising daa sources, gahering and mainaining he daa needed, and compleing and reviewing he collecion of informaion. Send commens regarding his burden esimae or any oher aspec of his collecion of informaion, including suggesions for reducing his burden, o Washingon Headquarers Services, Direcorae for Informaion Operaions and Repors, 1215 Jefferson Davis Highway, Suie 1204, Arlingon VA Respondens should be aware ha nowihsanding any oher provision of law, no person shall be subjec o a penaly for failing o comply wih a collecion of informaion if i does no display a currenly valid OMB conrol number. 1. REPORT DATE REPORT TYPE 3. DATES COVERED o TITLE AND SUBTITLE Real-ime Moion Tracking from a Mobile Robo 5a. CONTRACT NUMBER 5b. GRANT NUMBER 5c. PROGRAM ELEMENT NUMBER 6. AUTHOR(S) 5d. PROJECT NUMBER 5e. TASK NUMBER 5f. WORK UNIT NUMBER 7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) Universiy of Souhern California,Compuer Science Deparmen,Los Angeles,CA, PERFORMING ORGANIZATION REPORT NUMBER 9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR S ACRONYM(S) 12. DISTRIBUTION/AVAILABILITY STATEMENT Approved for public release; disribuion unlimied 13. SUPPLEMENTARY NOTES The original documen conains color images. 14. ABSTRACT 15. SUBJECT TERMS 11. SPONSOR/MONITOR S REPORT NUMBER(S) 16. SECURITY CLASSIFICATION OF: 17. LIMITATION OF ABSTRACT a. REPORT unclassified b. ABSTRACT unclassified c. THIS PAGE unclassified 18. NUMBER OF PAGES 18 19a. NAME OF RESPONSIBLE PERSON Sandard Form 298 (Rev. 8-98) Prescribed by ANSI Sd Z39-18

3 TECHNICAL REPORT CRES , CENTER FOR ROBOTICS AND EMBEDDED SYSTEMS, UNIVERSITY OF SOUTHERN CALIFORNIA 2 Fig. 2. Processing sequence for moion racking from a mobile robo: A monocular camera is uilized for moion deecion and racking in image space, and he resul is fused wih laser scans for moion esimaion in 2.5 dimension. sraighforward differencing is no applicable because a big difference is generaed by he moion of he camera even when nohing moves in he environmen. Therefore, he egomoion of he camera is compensaed before comparing he previous image (Image( 1)) wih he curren one (Image()). Assuming ha he ego-moion compensaion is perfec, he difference image would sill conain srucured noise on he boundaries of objecs because of he lack of deph informaion from a monocular image. We use a probabilisic model o filer such noise ou and o perform robus deecion and racking. The moions of moving objecs are modeled using a Bayesian framework, and heir probabiliy disribuion in image space is esimaed by applying percepion and moion models. Once he posiions and velociies of moving objecs are esimaed in 2D image space, he informaion is combined wih he parial deph informaion from a laser rangefinder in order o consruc a 3D moion model. By projecing range values ino an image space, he image piels a he same heigh as he laser rangefinder will have deph informaion. The performance of he proposed sysem has been analyzed in hree seps. Firs, he robusness of he moion racking algorihms has been esed on various robo plaforms ha have unique characerisics in erms of heir ego-moions. The eperimenal resuls show ha our moion racking sysem is able o cope wih various ypes of ego-moions and he paricle filer produces robus esimaion. Second, a muliple paricle filer approach for muliple moion racking has been validaed using various scenarios, and he eperimenal resuls show ha he muliple filer approach racks all moions successfully for he cases ha a single filer approach fails. Lasly, he proposed racking sysem has been inegraed wih a robo conrol loop, and is robusness and real-ime capabiliy have been eamined. The eperimens demonsrae ha he moion racking sysem is robus enough ha he sysem is no disurbed by oher moving objecs once i sars o rack an objec. The res of he paper is organized as follows. Secion II summarizes he relaed work on his opic. The deailed egomoion compensaion algorihm is given in Secion III, and he design of he probabilisic filer is eplained in Secion IV. Secion V describes how o fuse he esimaion resul in image space wih laser rangefinder daa, and Secion VI repors he eperimenal resuls and analyzes he performance of he proposed algorihms. The curren saus and possible improvemens are discussed in Secion VII. II. RELATED WORK The compuer vision communiy has proposed various mehods o sabilize camera moions by racking feaures [4], [5], [6] and compuing opical flow [7], [8], [9]. These approaches focus on how o esimae he ransformaion (homography) beween wo image coordinae sysems. However, he moions of moving objecs are ypically no considered, which leads o poor esimaion. Oher approaches ha eend hese mehods for moion racking using a pan/il camera include hose in [10], [11], [12]. However, in hese cases he camera moion was limied o ranslaion or roaion. When a camera is mouned on a mobile robo, he main moion of he camera is a forward/backward movemen, which makes he problem differen from ha of a pan/il camera. There is oher research on racking from a mobile plaform wih similar moions. [13] racks a single objec in forwardlooking infrared (FLIR) imagery aken from an airborne, moving plaform, and [14], [15] rack cars in fron using a camera mouned on a vehicle driven on a paved road. Once moion has been idenified, objecs in he scene need o be racked. Work focusing on robus muliple arge racking using probabilisic filers includes [2] which uses a paricle filer o rack people indoors (corridors) using a laser rangefinder, and [16] which also uses a paricle filer o rack muliple objecs using a saionary camera. A Kalman filer was used in [17] o deec and rack human aciviy wih he combinaion of a saic camera and a moving camera. III. EGO-MOTION COMPENSATION The ego-moion compensaion is a coordinae conversion procedure. Assume ha a sensory daa acquisiion process is as follows: (1) one se of daa D is acquired a ime when a robo is locaed a (, y, α), (2) he robo (and he sensors) moves o ( +, y + y, α + α) for, and (3) anoher se of daa D is acquired a ime +. In his case, he daa D and D canno be compared direcly because hey are capured in differen coordinae sysems. Therefore, he daa D should be compensaed for he egomoion (, y, α), which means ha he daa D should be ransformed as if i were acquired when a robo was locaed a ( +, y + y, α + α). The goal of he ego-moion compensaion sep is o compue his ransformaion T. The ransformaion can be esimaed direcly or indirecly. The indirec mehod is o esimae a robo pose each ime using various sensors (eg. gyroscope, acceleromeer, odomeer and/or GPS), and compue he ego-moion (, y, α) firs. Once he ego-moion is compued, he ransformaion T can be compued based on he geomeric properies of sensors. The pose esimaion echnique has been well sudied [18], [19], [20], and he indirec mehod may be effecive when he ransformaion error is linear in he ego-moion esimaion error. However, when a projecion operaion is involved in he ransformaion, as in our case, he indirec mehod is no appropriae since a iny angular error in he moion esimaion sep would induce a huge posiion error afer being projeced ino he daa space. Therefore, we choose he direc mehod.

4 TECHNICAL REPORT CRES , CENTER FOR ROBOTICS AND EMBEDDED SYSTEMS, UNIVERSITY OF SOUTHERN CALIFORNIA 3 The direc mehod is o infer he ransformaion T by corresponding salien feaures in he daa se D o hose in he daa se D. The feaure selecion and maching echniques for various ypes of sensors has been sudied by [4], [7], [21], [22], [23]. Since he ransformaion is esimaed using he corresponding feaure se direcly, he qualiy of he ransformaion relies on he qualiy of seleced feaures from he daa ses. Unforunaely, in our case, he qualiy of he feaures is poor due o independenly moving objecs in image daa. Therefore, a ransformaion model and oulier deecion algorihm needs o be designed so ha he esimaed ransformaion is no sensiive o hose objec moions. A. Feaure Selecion and Tracking We adop he KLT (Kanade-Lucas-Tomasi) feaure racking algorihm inroduced in [4], [7], [24], [25] o selec and correspond feaures beween wo images. The KLT algorihm has become a sandard echnique for feaure-based compuer vision algorihms. For compleeness, we describe he algorihms concisely here. Given wo consecuive images (he previous image I 1 and he curren I ), a se of feaures is seleced from he image I 1, and a corresponding feaure se is consruced by racking he same feaures on he image I. For feaure selecion, a small search window runs over he whole image I 1 o check if he window conains a reliably rackable feaure. For each search window, 1) Compue he boundary informaion, [ I(,y) ] T I(,y) y 2) Compue he covariance mari of he boundary piels 3) Compue wo eigenvalues (λ 1, λ 2 ) of he covariance mari 4) Selec a search window such ha min(λ 1, λ 2 ) > θ Search windows wih wo small eigenvalues conain no paern, and hose wih one small eigenvalue and one big eigenvalue conain unidirecional paerns, which are no easy o rack. Only search windows wih wo big eigenvalues are seleced for racking because hey conain a perpendicular paern (eg. corners) or divergen eures (eg. leaves) which are relaively unique enough o be racked. The feaure selecion algorihm runs on he image I 1, and generaes a se of feaures F 1. Figure 3 shows he feaures seleced from indoor and oudoor images. In he indoor image, mos of he seleced feaures are he corners of objecs, like desks, compuers, and bookshelves. In he oudoor image, some corners of bricks and cars, leaves and grass ha have comple eures were seleced as feaures. Once he feaure se F 1 is seleced, he feaures are racked on he subsequen image I and he se of racked feaures F is generaed. For efficiency, he search range was limied o a small consan disance (assuming a bounded robo speed). Figure 4 shows he robusness of he racking mehod. Figure 4 (a) shows he feaures seleced from he image I, and Figure 4 (b) shows he same feaures racked over 30 frames on he image I +30, which is an image capured 3 seconds laer. The erroneous feaures on image boundaries are eliminaed for subsequen processing. (a) indoor feaures (b) oudoor feaures Fig. 3. Salien feaures (filled circles) seleced for racking: Primarily perpendicular paerns (eg. corners) or divergen eures (eg. leaves) are seleced. (a) feaures a ime (b) racked feaures a ime + 30 Fig. 4. Feaure racking: (a) shows salien feaures seleced, and (b) shows he same feaures racked over 30 frames (3 seconds).

5 TECHNICAL REPORT CRES , CENTER FOR ROBOTICS AND EMBEDDED SYSTEMS, UNIVERSITY OF SOUTHERN CALIFORNIA 4 B. Transformaion Esimaion Once he correspondence F =<F 1, F > is known, he ego-moion of he camera can be esimaed using a ransformaion model and an opimizaion mehod. We have sudied hree differen models: affine model, bilinear model, and a pseudo-perspecive model. Affine [ ] Model : f = f y [ a0 f 1 [ Bilinear ] Model : f = f y a 3 f 1 [ a0 f 1 a 4 f 1 + a 1 f 1 y + a 2 + a 4 f 1 y + a 5 + a 1 f 1 y + a 5 f 1 y Pseudo-perspecive Model : a [ ] 0 f 1 f = a 5 f 1 f y + a 1 f 1 y + a 2 ] + a 2 + a 3 f 1 + a 6 + a 7 f 1 + a 3 f 1 + a 6 fy 1 + a 7 + a 4 f 1 fy a4 f 1 f 1 y f 1 y ] f 1 y + a 3 fy 1 2 When he inerval beween consecuive images is very small, mos ego-moions of he camera can be esimaed using an affine model, which can cover ranslaion, roaion, shearing, and scaling moions. However, when he inerval is long, he camera moion in he inerval canno be capured by a simple linear model. For eample, when he robo moves forward, he feaures in he image cener move slower ha hose near he image boundary, which is a projecion operaion, no a simple scaling. Therefore, a nonlinear ransformaion model is required for hose cases. On he oher hand, an over-fiing problem may be caused when a model is highly nonlinear, especially when some of he seleced feaures are associaed wih moving objecs (ouliers). There is clearly a rade-off beween a simple, linear model and a highly nonlinear model, and i needs more empirical research for he bes selecion. We used a bilinear model for he eperimens repored in his paper. When he ransformaion from he image I 1 o he image I is defined as T 1, he cos funcion for leas square opimizaion is defined as: J = 1 N { f 2 i T 1 ( )} f 1 2 i (1) i=1 where N is he number of feaures. The model parameers for ego-moion compensaion are esimaed by minimizing he cos. However, as menioned before, some of he feaures are associaed wih moving objecs, which lead o he inference of an inaccurae ransformaion. Those feaures (ouliers) should be eliminaed from he feaure se before he final ransformaion is compued. The model parameer esimaion is hus performed using he following wo-sep procedure: 1) compue he iniial esimae T 0 using he full feaure se F. 2) pariion he feaure se F ino wo subses F in and F ou as: { f i F in f i F ou if fi T 0 1(f 1 i ) < ɛ oherwise (2) Fig. 5. Oulier feaure deecion: Ouliers are marked wih filled circles, and inliers are marked wih empy circles. 3) re-compue he final esimae T using he subse F in only. Figure 5 shows he pariioned feaure ses: F in is marked wih empy circles, and F ou is marked wih filled circles. Noe ha all feaures associaed wih he pedesrian are deeced as ouliers. I is assumed for oulier deecion ha he porion of moving objecs in he images is relaively smaller compared o he background; he feaures which do no agree wih he main moion are considered as ouliers. This assumpion will break when he moving objecs are very close o he camera. However, mos of he ime, hese objecs pass by he camera in a shor period (leading o ransien errors), and a high-level probabilisic filer is able o deal wih he errors wihou oal failure. C. Frame Differencing The image I 1 is convered using he ransformaion model before being compared o he image I in order o eliminae he effec of he camera ego-moion. ( For each piel (, y): ) I c (, y) = I 1 T 1 1 (, y) (3) Figure 6 (c) shows he compensaed image of Figure 6 (a); he ranslaional and forward moions of he camera were clearly eliminaed. The valid region R of he ransformed image is smaller han ha of he original image because some piel values on he border are no available in he original image I 1. The invalid region in Figure 6 (c) is filled black. The difference image beween wo consecuive images is compued using he compensaed image: { (I c (, y) I (, y)) if (, y) R I d (, y) = (4) 0 oherwise Figure 7 compares he resuls of wo cases: frame differencing wihou ego-moion compensaion (Figure 7 (a)) and wih egomoion compensaion (Figure 7 (b)). The resuls show ha he ego-moion of he camera is decomposed and eliminaed from image sequences. The full descripion of he frame differencing process is given in Algorihm 1. IV. MOTION DETECTION IN 2D IMAGE SPACE The Frame Differencing sep in Figure 2 generaes he sequence of difference images, Id 0, I1 d,, I d, whose piel

6 TECHNICAL REPORT CRES , CENTER FOR ROBOTICS AND EMBEDDED SYSTEMS, UNIVERSITY OF SOUTHERN CALIFORNIA 5 (a) Image a ime 1 (b) Image a ime (c) Compensaed image of (a) Fig. 6. Image Transformaion: (c) is he ransformed image of (a) ino (b) coordinaes. The valid region of he compensaed image is smaller han ha of he original image due o he absence of daa on he border. (a) difference wihou compensaion camera posiions are differen when wo consecuive images are capured, i is ineviable ha some informaion is newly inroduced o he curren image I or some informaion of he previous image I 1 is occluded in he image I. To deal wih hose errors, a probabilisic approach is adoped. The normalized piel values in he difference images can be inerpreed as he probabiliy of he eisence of moving objecs in ha posiion, and he posiion and size of he moving objecs are esimaed over ime. This esimaion process can be modeled using a Bayesian formulaion. Le represen he sae (eg. he posiion and velociy) of a moving objec. = [ y ẋ ẏ ] T (5) The poserior probabiliy disribuion P m ( ) of he sae is derived as follows. (b) difference wih compensaion Fig. 7. Resuls of frame differencing: (b) shows ha he ego-moion of a camera was decomposed and eliminaed from image sequences. values represen he amoun of moion occurred in he posiion. However, as described earlier, he difference images conain wo differen ypes of errors. There are ransien errors caused by imperfec ego-moion compensaion, and his ype of error should be filered ou using heir emporal properies. There are also persisen errors caused during daa acquisiion. Since he P m ( ) = P ( I 0 d, I1 d,, I d ) = η P (Id, Id 0, I 1 d ) P ( Id 0, I 1 d ) = η P (Id ) P ( Id 0, I 1 d ) = η P (Id ) P ( Id 0, I 1 d, 1 ) P ( 1 Id 0, I 2 d ) d 1 = η P (Id ) P ( 1 ) P ( 1 Id 0, I 2 d ) d 1 = η P (Id ) P ( 1 )P m ( 1 ) d 1 Now he poserior probabiliy disribuion can be updaed recursively by applying a percepion model P (I d ) and a moion model P ( 1 ) over ime. (6)

7 TECHNICAL REPORT CRES , CENTER FOR ROBOTICS AND EMBEDDED SYSTEMS, UNIVERSITY OF SOUTHERN CALIFORNIA 6 Algorihm 1: Frame Differencing wih Ego-moion Compensaion Inpu: wo consecuive images I 1 and I Oupu: he difference image Id and he ego-moion ransformaion T 1 1 begin 2 selec a salien feaure se F 1 from I 1 ; 3 rack he feaures on I, and generae he corresponding feaure se F ; 4 F < F 1, F > ; 5 if F > he number of parameers in a ransformaion model hen /* compue he iniial esimae using he full feaure se. */ consruc an inpu mari X using F 1 ; consruc a arge vecor using F ; T 0 `X T X 1 X T /* remove ouliers from he feaure se. */ F in ; F ou ; foreach <f 1, f > in he se F do if f T 0 (f 1 ) < ɛ hen inser <f 1, f > ino F in ; else inser <f 1, f > ino F ou; end end /* compue he final esimae using he inliers only. */ 18 consruc an inpu mari X using only F 1 in ; 19 consruc a arge vecor using only Fin ; T 1 `X 20 T X 1 X T /* compensae ego-moion. */ 21 iniialize I c wih all 0 ; 22 forall (, y) in he image I c do I c(, y) I T (, y) ; 24 end /* frame differencing wih ego-moion compensaion. */ 25 Id I I c ; 26 else /* no compensaion is feasible wihou sufficien feaures. */ 27 T 1 he idenical ransformaion ; 28 Id I I 1 ; 29 end 30 reurn Id and T 1 31 end A. Bayesian Filer Design The derived equaion 6 shows how he sequence of difference images and he moion model of a moving objec are inegraed ino he sae esimaion process. However, here are sill hree quesions o answer: (1) how o define a percepion model, and (2) how o define a moion model, and finally (3) how o represen he poserior probabiliy disribuion. The percepion model P (Id ) capures he idea ha if here is a moion a posiion, hen he difference values of he piel in ha posiion and is neighbors should be big. For eample, le us assume a small moving objec ha occupies a single piel p 1 on a camera image. When he objec moves o is neighbor piel p, hen he difference values of boh piels p 1 and p would be big. This neighborhood can be modeled using a muli-variae Gaussian, and he percepion model can be defined as P (I d ) = Z I d 0 Id() 1 p 1 (2π)d Σ s e 2 ( ) T Σ 1 s ( ) d (7) when d is he dimension of he sae, and he covariance mari Σ s conrols he range of effecive neighborhood. Fig. 8. Bayesian filer racking wih piecewise consan represenaion: The lef window shows he inpu image and he deeced moving objec (ellipsoid), and he righ window shows he poserior probabiliy disribuion of he moving objec using a 1010 piel grid. The moion model P ( 1 ) capures he bes guess abou he moion of a moving objec. Since no prior knowledge of an objec moion is assumed, a consan velociy model is a naural choice. The uncerainy of an objec moion is modeled by he covariance mari Σ m of a muli-variae Gaussian. µ = P ( 1 ) = 1 + ẋ 1 y 1 + ẏ 1 ẋ 1 ẏ 1 (8) 1 (2π)d Σ m e 1 2 ( µ) T Σ 1 m ( µ) The choice of a represenaion for he poserior probabiliy disribuion is imporan for real-ime sysem design because he updae equaion (Equaion 6) conains an inegral operaion and he required compuaion is inensive. Even when he size of a camera image is small, he sae space is sizeable because he sae is four-dimensional. The mos compac represenaion is o use a single Gaussian, like he Kalman filer [26], [27], bu his approach is no appropriae because (1) he iniial sae of a moving objec is no given in priori, and (2) image segmenaion is avoided for real-ime response, which makes i hard o consruc a measuremen mari. A beer approach is o use a piecewise consan represenaion. By decomposing he sae space ino an equally spaced grid, he amoun of compuaion can be reduced drasically. For eample, Figure 8 shows he posiion esimaion resul using a 1010 piel grid. However, he compuaion was sill no efficien enough. In order o achieve a real-ime response, only he posiion of a moving objec was esimaed ( = [ y] T ), and a consan posiion model, insead of a consan velociy model, was uilized as he consequence of he simpler sae definiion. In addiion, he approimaion qualiy of he poserior probabiliy disribuion was sacrificed by using he sparse represenaion. The mos popular represenaion ha addresses hese concerns is a sample-based represenaion [28], [29]. The amoun of compuaion is reduced by using a small se of weighed samples, bu he approimaion qualiy is well preserved by concenraing samples on he area whose probabiliy is high. Also, he number of samples can change dynamically depending on he shape of he poserior probabiliy disribuion and available compuer power. In his paper, we adop he sample-based represenaion. (9)

8 TECHNICAL REPORT CRES , CENTER FOR ROBOTICS AND EMBEDDED SYSTEMS, UNIVERSITY OF SOUTHERN CALIFORNIA 7 B. Paricle Filer Design The Paricle filer [28], [29] is a simple bu effecive algorihm o esimae he poserior probabiliy disribuion recursively, which is appropriae for real-ime applicaions. In addiion, is abiliy o perform muli-modal racking is aracive for unsegmened objec deecion and racking from camera images. An efficien varian, called he Adapive Paricle Filer, was inroduced in [30]. This changes he number of paricles dynamically for a more efficien implemenaion. We implemened he Adapive Paricle Filer o esimae he poserior probabiliy disribuion in Equaion 6. As described in Secion IV-A, paricle filers also require wo models for he esimaion process: a percepion model and a moion model. The percepion model is used o evaluae a paricle and compue is weigh (or imporance). Equaion 7 provides a generic form of he percepion model. However, he percepion model is simplified for efficiency. A sep funcion is used insead of a muli-variae Gaussian, and he evaluaion range is also limied o m m fied area. The m m mask should be big enough (usually 5 5) so ha sal-and-pepper noise is eliminaed. The weigh ω i of he i h paricle (s i = [ i y i ẋ i ẏ i ]T ) is compued by ωi = 1 m/2 m 2 m/2 j= m/2 k= m/2 I d ( i j, y i k ) (10) As shown in Equaion 10, only he posiion informaion is used o evaluae paricles. The moion model is used o propagae a newly drawn paricle according o he esimaed moion of a moving objec. The moion model in and Equaion 9 describes how o compue he probabiliy of he new sae when he previous sae 1 is given. However, for paricle filer updae, he moion model should describe how o draw a new paricle s i when he previous paricle s 1 i and is weigh ωi are given. Therefore, he moion model is defined as s i = 1 i y 1 i + ẋ 1 i + ẏ 1 i ẋ 1 i ẏ 1 i + Normal( γp ) ωi + Normal( γp ω ) i + Normal( γv ω i + Normal( γv ω i ) ) (11) where is a ime inerval, and γ p and γ v are noise parameers for posiion and velociy componens respecively. The funcion N ormal(σ) generae a Gaussian random variae wih zero mean and he sandard deviaion σ. As shown in Equaion 11, he parameerized noise is added o he consan-velociy model in order o overcome an inrinsic limiaion of he paricle filer, which is ha all paricles move in a convergence direcion. However, a dynamic miure of divergence and convergence is required o deec newly inroduced moving objecs. [29] inroduced a miure model o solve his problem, bu in he image space he probabiliy P ( Id ) is uniform and he dual MCL becomes random. Therefore, we used a simpler, bu effecive mehod by adding inverse-proporional noise. As an implemenaion issue, a muli-dimensional kd-ree is consruced during he paricle filer updae. This serves wo purposes: (1) o compue he proper number of paricles Fig. 9. Paricle filer racking: The posiion of paricles are represened by small dos, and he horizonal bar on he op-lef corner shows he number of paricles being used. [30], and (2) o cluser paricles efficienly as described in Secion IV-C. In order o deermine he proper number of paricles, a kd-ree wih uniform-size nodes is buil. When he size of he ree is k, he error bound is ɛ, and he confidence quanile is z 1 δ, he proper number is compued as follows [30]. n = 1 2ɛ χ2 k 1,1 δ {. = k 1 2ɛ 1 2 9(k 1) + 2 9(k 1) z 1 δ } 3 (12) Figure 9 shows he oupu of he paricle filer. The dos represen he posiion of paricles, and he horizonal bar on he op-lef corner of he image shows he number of paricles being used. The final algorihm of he paricle filer is described in Algorihm 2. C. Paricle Clusering The paricle filer generaes a se of weighed paricles ha esimae he poserior probabiliy disribuion of a moving objec, bu he paricles are no easy o process in he following sep. More inuiive and meaningful daa can be eraced by clusering he paricles. A densiy-based algorihm using a kdree is inroduced for efficien paricle clusering. The main idea is o conver a se of weighed paricles ino a lowerresoluion, uniform-sized grid. The grids can be represened using a kd-ree efficienly, and all clusering operaions are performed using he grids insead of paricles. Therefore, he required compuaion is reduced drasically. However, since each grid mainains enough informaion abou he paricles in he grid, he saisics of each cluser can be calculaed wihou any accuracy loss. The algorihm consiss of he following four seps: 1) Tree Consrucion: Given a se of weighed paricles, a kd-ree represenaion is consruced. The sae space is pariioned ino uniform-sized grid cells, and only non-empy cells are mainained using a kd-ree. Since he Adapive Paricle Filer requires he kd-ree for compuing he proper number of paricles (as described in Secion IV-B), his sep can be combined wih he paricle filer updae. The informaion of each paricle is no necessary anymore for he subsequen seps, bu a few era saisics of each erminal node in he

9 TECHNICAL REPORT CRES , CENTER FOR ROBOTICS AND EMBEDDED SYSTEMS, UNIVERSITY OF SOUTHERN CALIFORNIA 8 Algorihm 2: Adapive Paricle Filer for Moion Tracking Inpu: he previous paricle se S 1, he difference image Id, and he ego-moion ransformaion T 1 Oupu: a new paricle se S and is kd-ree represenaion K 1 S 0 a se of uniformly weighed, random paricles wih he size n ma ; 2 begin /* ransform all paricles o compensae an ego-moion */ 3 foreach < s 1, ω 1 > in S 1 do 4 s 1 T 1 (s 1 ) ; 5 end /* perform he sandard paricle filer updae */ S ; K ; W 0 ; repea /* draw a paricle from S 1 according o is weigh values */ C 1 ω 1 1 ; for i=2 o S 1 do C i C i 1 + ω 1 i ; generae a random number r in he range [0, 1) ; selec he i h paricle s i from S 1 such ha C i r < C i+1 ; /* propagae he paricle using he moion & sensor models */ ω 1 P m/2 P m/2 m 2 j= m/2 k= m/2 I d ` i j, y i k ; 2 i s + ẋ i + Normal( γp 3 ω ) 6 y i 4 + ẏ i + Normal( γp ω ) 7 ẋ i + Normal( γv ω ) 5 ; ẏ i + Normal( γv ω ) ω 1 P m/2 P m/2 ` m 2 j= m/2 k= m/2 I d j, y k ; /* add he new paricle */ add < s, ω > o S ; add < s, ω > o K ; W W + ω ; unil S < n min n q o or S < K ɛ 9( K 1) + 2 3; 9( K 1) z 1 δ /* normalize he weighs of all paricles */ 21 if W > 0 hen 22 foreach < s, ω > in S do ω ω /W ; 23 else 24 S a se of uniformly weighed, random paricles wih he size n ma ; 25 end 26 reurn S and K 27 end ree need o be compued. For each erminal node k, he weigh w k, he mean µ k, and he covariance mari Σ k of he node are calculaed using he subse of paricles ha are associaed wih he node. w k = i w i µ k = i w is i / i w i Σ k = i w i(s i µ)(s i µ) T / i w i (13) 2) Candidae Selecion: Insead of re-consrucing a poserior probabiliy disribuion and hresholding he pdf, we selec candidae grid cells whose paricle densiy is bigger han a hreshold θ. Since each paricle has differen weigh, he densiy should ake he weigh ino accoun. Theoreically his means w k /volume(k) should be used as he deerminan. However, he number of paricles (n k /volume(k)) can be used alernaively for simpliciy assuming all paricles have uniform weighs. Fig. 10. Paricle clusering: Two ellipsoids represen he means and covariance of wo paricle clusers. 3) Grouping: Once he candidae nodes are seleced, clusering can be done by simply grouping he nodes by checking conneciviy among nodes. There are various known algorihms for his ask. The conneciviy can be defined using he disance beween he mean vecors of wo nodes, or can be deermined by checking if a node is a neighbor of anoher. 4) Saisics Compuaion: For each cluser, he saisics of he paricles in he cluser can be calculaed by summing he saisics of he nodes in he cluser incremenally. µ = w µ+w k µ k w+w k Σ = w{σ+(µ µ)(µ µ) T }+w k{σ k +(µ µ k )(µ µ k ) T } w+w k w = w + w k µ = µ (14) Figure 10 shows he oupu of he paricle clusering algorihm. The dos represen he posiion of paricles, and he ellipsoid represens he mean and covariance of each cluser. The full descripion of he clusering algorihm is in Table 3. D. Muliple Paricle Filers for Muliple Moion Tracking The paricle filer has many advanages as described in [31], [29]; one of he advanages is muli-modaliy. This propery is aracive for muli-arge racking because i raises he possibiliy ha a single se of paricles can rack muliple objecs in an image sequence. However, ha is rue only under wo condiions: 1) The percepion model should be bad enough so ha paricles converge slowly, and evenually say on muliple objecs. For eample, imagine he case in which here are wo moving objecs in he image sequence. I is desired ha a single se of paricles is spli ino wo groups, and each group converges o and rack each objecs coninuously. Apparenly a se of paricles would behave as desired if wo objecs show eacly he same amoun of moions (on average over ime) in he image sequence. However, his assumpion is no realisic. In mos cases, one would show a bigger moion han he oher due o differen size and shape of an objec, differen disance o a camera, ec. The behavior of paricles is quie differen wihou he assumpion. Since he amoun of moion is differen, he percepion model P (I d ) in Equaion 6 generaes a differen value for

10 TECHNICAL REPORT CRES , CENTER FOR ROBOTICS AND EMBEDDED SYSTEMS, UNIVERSITY OF SOUTHERN CALIFORNIA 9 Algorihm 3: Paricle Clusering for Pos-processing Inpu: he kd-ree represenaion K of a paricle se Oupu: m Gaussians (< µ 1, Σ 1 >,, < µ m, Σ m >) 1 begin /* selec a se of candidae nodes C based on heir densiy */ 2 C ; 3 forall a erminal node k in K do 4 he paricles in he node k are 5 < s 1, w 1 >, < s 2, w 2 >,, < s n, w n > ; if P i w i / volume(k) θ /* hresholding by weighed densiy */ 6 hen /* hese saisics can be compued when he ree is buil */ w k P 7 i w i ; µ k P i w is i / P 8 i w i ; 9 Σ k P i w i(s i µ)(s i µ) T / P i w i ; 10 add < w k, µ k, Σ k > o C ; 11 end 12 end /* group he candidae nodes using conneciviy */ 13 for i = 1 o C do 14 if c i / any group hen creae a new group G i ; 15 for j = i + 1 o C do 16 if c j / any group hen 17 if µ i µ j ρ hen add c j o he group G i 18 else if G i G j and µ i µ j ρ hen 19 combine he group G i and he group G j ; 20 end 21 end 22 end /* compue he saisics of each group */ 23 foreach a group G i do 24 w i 0 ; 25 µ i 0 ; 26 Σ i 0 ; 27 forall a candidae < w k, µ k, Σ k > in G i do 28 µ w i µ i +w k µ k w i +w k ; 29 Σ i w i{σ i +(µ µ i )(µ µ i ) T }+w k{σ k +(µ µ k )(µ µ k ) T } ; w i +w k 30 w i w i + w k ; 31 µ i µ 32 end 33 end 34 end 35 reurn < µ 1, Σ 1 >,, < µ m, Σ m > (a) single person a = 23 (b) hree people a = 109 Fig. 11. Problem of a single paricle filer: (a) shows ha he paricle filer converges on he person, and (b) shows ha he filer says on he person coninuously even when oher people are inroduced in he image laer. converged paricles do no diverge unless he racked objec disappears. For eample, Figure 11 shows a problemaic scenario. The paricle se converges when he person eners ino he field of view of he camera as in Figure 11 (a), and i concenraes on he person coninuously even when wo more people ener laer as in Figure 11 (b). This problem is no rivial o solve using a limied number of paricles. Therefore, we inroduce a racking sysem using muliple paricle filers. The main idea is o mainain an era paricle filer for a newly inroduced or deeced objec. Since he number of objecs is no known in priori, paricle filers should be creaed and desroyed dynamically. Whenever he era paricle filer converges on a newly deeced objec, a new paricle filer is creaed (as long as he number of paricle filers is smaller han he maimum limi N ma ). Similarly, whenever a paricle filer diverges due o he disappearance of a racked objec, i is desroyed. In order o preven wo paricle filers from converging on he same objec, whenever a paricle filer is updaed, he difference image is modified for subsequen processing such ha difference values covered by he filer are cleared. The deailed algorihm is described in Algorihm 4. each objec. As a resul, paricles on he smaller moion would shif o he bigger moion afer some ieraions even when he size of he wo paricle groups was he same in he beginning. This behavior is epeced because Equaion 6 is designed o esimae he posiion of a single objec. This limiaion can be overcome in wo ad-hoc ways: (1) By making he percepion model less sensiive so ha paricles converge very slowly, (2) By increasing he number of paricles. The firs echnique is feasible when he convergence speed is no imporan [32], [29]. However, convergence speed is a criical facor for moion racking. When a new objec is inroduced, a filer should be able o deec i and sar o rack i in a reasonable ime. The second echnique is no desirable since he required compuaion increases drasically. 2) All objecs should be inroduced in he beginning of esimaion process. As eplained in Secion IV-A, paricles shows a convergence endency, and consequenly V. POSITION ESTIMATION IN 3D SPACE A monocular image provides rich informaion for egomoion compensaion and moion racking in 2D image space. However, a single camera has limis on rerieving deph informaion, and an addiional sensor is required o consruc 3D models of moving objecs. Our robos are equipped wih a laser rangefinder, which provides deph informaion wihin a singe plane. Given he opical properies of a camera and he ransformaion beween he camera and he laser rangefinder, disance informaion from he laser rangefinder can be projeced ono he image coordinaes (Figure 12). Given he heading α and he range r of a scan, he projeced posiion (, y) in he image coordinae sysem is compued as follows: [ ] = y ( ) w 2 1 an(α) an(f h ) ( h ( ) d d r (r l)) 1 l an(f v) (15) where he focal lengh of he camera is l, he horizonal and verical field of view of he camera are f h and f v, he

11 TECHNICAL REPORT CRES , CENTER FOR ROBOTICS AND EMBEDDED SYSTEMS, UNIVERSITY OF SOUTHERN CALIFORNIA 10 Algorihm 4: Muliple Paricle Filers for Muliple Moion Tracking Inpu: he previous paricle filer se P 1, he difference image I d, and he ego-moion ransformaion T 1 Oupu: a new paricle filer se P 5 1 creae a paricle filer p 0 ; 2 P 0 {p 0 } ; 3 begin /* updae all paricle filers in he se */ 4 P ; I d Id ; 6 foreach a paricle filer p i in P 1 do 7 updae he paricle filer p i using he difference image I d ; 8 rerieve he cluser informaion C i from he updaed paricle 9 filer p i ; if C i > 0 /* check he convergence */ 10 hen 11 P P {p i } ; 12 P 1 P 1 {p i } ; 13 forall a cluser < µ j, Σ j > in Ci do 14 I d I d region(µ j, Σ j ) ; 15 end 16 end 17 end /* add an era paricle filer if allowed */ if P 1 > 0 hen selec a paricle filer p from P 1 ; rese he paricle filer p ; P P {p i } ; P 1 P 1 {p i } ; else if P < N ma hen creae a new paricle filer p ; P P {p i } ; end end /* desroy unused paricle filers */ 29 forall a paricle filer p i in P 1 do 30 desroy he paricle filer p i ; 31 end 32 end 33 reurn P heigh from he laser rangefinder o he camera is d, and he image size is w h. This projecion model assumes a very simple camera model (a pin-hole camera) for fas compuaion. As a resul of he projecion, he image piels a he same heigh as he laser rangefinder will have deph informaion as shown in Figure 13. For ground robos, his parial 3D informaion can be enough for safe navigaion assuming all moving obsacles are on he he same plane as he robo. In erms of moving objec racking, if he region of a moving objecs in image space and hose piels are overlapped, hen he disance beween a robo and he moving objec can be esimaed. This naive inegraion of he 2D moion esimaes and range scans from a laser rangefinder is a reasonable pracical soluion. However, using wo separae sensors requires anoher esimaion problem poenially, which is he fusion of muliple asynchronous inpus. A preferred roue (no invesigaed here) would be o use sereo vision for deph informaion rerieval. If compuaional power allows one can eploi he facs ha sereo (1) provides full deph informaion of an image space, and (2) a single inpu source provides synchronous daa and beer fusion resul can be epeced. Fig. 12. Projecion of laser scans ono he image coordinaes: The range scans from a laser rangefinder can be projeced ono he image coordinae sysem based on he opical properies of a camera and he ransformaion beween he camera and he laser rangefinder. Fig. 13. Projeced laser scans: The image piels a he same heigh as he laser rangefinder have deph informaion. VI. EXPERIMENTAL RESULTS AND DISCUSSION The proposed moion racking sysem was esed in various scenarios. Firs, he robusness of ego-moion compensaion and moion racking algorihms was esed using hree differen robo plaforms. The performance is analyzed in Secion VI- A. Second, he muliple-moion racking sysem described in Secion IV-D was esed using various scenarios. The resuls are presened in Secion VI-B. Finally, he racking sysem was inegraed wih an acual robo conrol sysem. The resul is discussed in Secion VI-C. A. Tracking a Moving Objec from Various Plaforms The ego-moion of a mobile robo is diverse according o is acuaor design and he way he camera is mouned on he plaform. For eample, a down-facing camera mouned on an UAV (Unmaned Aerial Vehicle) would show a differen ego-moion from a forward-facing camera mouned on a walking robo. In addiion, he compleiy of he ego-moion increases hrough he ineracion wih rough errain. The racking performance is also affeced by he disribuion of occlusive obsacles in an environmen. Therefore, he ego-moion compensaion and he moion racking algorihms should be esed on various environmens wih a wide variey of mobile plaforms. 1) Eperimenal Seup: The racking algorihms were implemened and esed in various oudoor environmens using hree differen robo plaforms: roboic helicoper, Segway RMP, and Pioneer2 AT. Each plaform has unique characerisics in erms of is ego-moion. The Roboic Helicoper [33] in Figure 14 (a) is an auonomous flying vehicle carrying a monocular camera facing

12 TECHNICAL REPORT CRES , CENTER FOR ROBOTICS AND EMBEDDED SYSTEMS, UNIVERSITY OF SOUTHERN CALIFORNIA 11 Fig. 14. (a) Roboic Helicoper (b) Segway RMP (c) Pioneer2 AT Robo plaforms for eperimens: Each plaform has unique characerisics in erms of is ego-moion. downward. Once i akes off and hovers, planar movemens are he main moion, and moving objecs on he ground say a a roughly consan disance from he camera mos of he ime; however, pich and roll moions sill generae complicae video sequences. Also, high-frequency vibraion of he engine adds moion-blur o camera images. The Segway RMP in Figure 14 (b) is a wo-wheeled, dynamically sable robo wih self-balancing capabiliy. I works like an invered pendulum; he wheels are driven in he direcion ha he upper par of he robo is falling, which means he robo body piches whenever i moves. Especially when he robo acceleraes or deceleraes, he pich angle increases by a significan amoun. Since all sensors are direcly mouned on he plaform, he pich moions preven direc image processing. Therefore, he ego-moion compensaion sep should be able o cope wih no only planar movemens bu also pich moions. The Pioneer2 AT in Figure 14 (c) is a ypical four-wheeled, saically sable robo. Since he Pioneer2 robo is he only saically sable plaform among hese robo plaforms, we drove he robo on he mos severe es environmen. Figure 17 shows he rocky errain where he robo was driven. In addiion, he moving objecs were occluded occasionally because of he rees in he environmen. The compuaion was performed on embedded compuers (Penium III 1.0 GHz) on he robos. Low resoluion ( piels) inpu images were chosen for real-ime response. The maimum number of paricles was se o 5,000, and he minimum number of paricles was se o Since he algorihm is supposed o run in parallel wih oher processes (eg. navigaion and communicaion), less han 70 percen of he CPU ime was dedicaed for racking; he racking algorihm was able o process five frames per second. 2) Eperimenal Resuls: The performance of he racking algorihm was evaluaed by comparing wih he posiions of manually racked objecs. For each video sequence, he recangular region of moving objecs were marked manually and used as ground ruh. Figure show his evaluaion process. The upper rows show he inpu image sequence, and he posiions of manually-racked objecs are marked wih recangles. The lower rows show he se of paricles and he clusering resuls. The posiion of each paricle is marked wih dos, and he horizonal bar on he op-lef corner of he image indicaes he number of paricles being used. The clusering resul is represened using an ellipsoid and a line inside. The ellipsoid shows he mean and covariance of he esimaed objec posiion, and he line inside of he ellipsoid represens he esimaed velociy vecor of he objec. The final evaluaion resul is shown in Table I. Frames is he number of image frames in a video sequence, and Moions is he number of moving objecs. Deeced is he oal number of deeced objecs, and True + and False + are he number of correc deecions and he number of falseposiives respecively. Deecion Rae shows he percenage of moving objecs correcly deeced, and Avg. Error is he average Euclidean disance in piels beween he ground ruh and he oupu of racking algorihm. The average disance error should no be considered as acual error measuremen since he racking algorihm does no perform an eplici objec segmenaion; i may rack a par of an objec ha generaes moion while he ground ruh always racks he whole objecs even hough only par of he objec moves. The Roboic helicoper resul shows ha he racking algorihm missed seven objecs, bu five of hem were he cases when a moving objec was inroduced and showed only parially on he boundary of he image plane. Once he whole objec enered ino he field of view of he camera, he racking algorihm racked i robusly. For he Segway RMP resul, he deecion rae was saisfacory, bu he average disance error was larger han he ohers. The reason was ha he walking person was closer o he robo and he racking algorihm ofen deeced he upper body only, which caused a consan disance error. The Pioneer2 AT resul shows he higher raio of falseposiives; however, as eplained in he previous secion, he errain for he eperimen was more challenging (rocky) and he inpu images were more blurred and unsable. Overall various ypes of ego-moions were successfully eliminaed from inpu images, and he paricle filer was able o rack moions robusly from diverse robo plaforms. B. Tracking Muliple Moving Objecs The muliple-moion racking sysem using muliple paricle filers was inroduced in Secion IV-D. Since he robusness of an individual filer was analyzed in Secion VI-A, we focus on analyzing how muliple filers are creaed and desroyed effecively when he number of moving objecs changes dynamically. 1) Eperimenal Seup: The Segway RMP in Figure 14 (b) was seleced for he eperimen because of is comple egomoion. The Segway RMP is a dynamically sable plaform, and is piching moions for self-balancing are combined ino

13 TECHNICAL REPORT CRES , CENTER FOR ROBOTICS AND EMBEDDED SYSTEMS, UNIVERSITY OF SOUTHERN CALIFORNIA 12 (a) = 13 (b) = 21 (c) = 34 (d) = 13 (e) = 21 (f) = 34 Fig. 15. Moving objec racking from Roboic helicoper: The upper row shows he inpu image sequence wih manually-racked objecs, and he lower row shows he paricle filer oupus and clusering resuls. (a) = 57 (b) = 119 (c) = 195 (d) = 57 (e) = 119 (f) = 195 Fig. 16. Moving objec racking from Segway RMP: The upper row shows he inpu image sequence wih manually-racked objecs, and he lower row shows he paricle filer oupus and clusering resuls. TABLE I PERFORMANCE OF MOVING OBJECT DETECTION ALGORITHM Plaform Frames Moions Deeced True + False + Deecion Rae Avg. Error Roboic helicoper % Segway RMP % Pioneer2 AT % 15.87

14 TECHNICAL REPORT CRES , CENTER FOR ROBOTICS AND EMBEDDED SYSTEMS, UNIVERSITY OF SOUTHERN CALIFORNIA 13 (a) = 35 (b) = 56 (c) = 191 (d) = 35 (e) = 56 (f) = 191 Fig. 17. Moving objec racking from Pioneer2 AT: The upper row shows he inpu image sequence wih manually-racked objecs, and he lower row shows he paricle filer oupus and clusering resuls. linear and angular moions. The combinaion of hese moions causes complicaed ego-moions even when he robo is driven on a fla errain or when he robo sops in place. The robo was driven on he USC campus during he dayime when here are diverse aciviies in he environmen including walking people and auomobiles. The racking performance is analyzed for hree differen cases. As eplained in Secion IV-D, if one of he following wo condiions is no saisfied, a single paricle filer fails o rack muliple objecs even hough i suppors muli-modaliy in heory: (1) all objecs should be inroduced before a paricle filer converges, and (2) he convergence speed of a paricle filer should be sacrificed by using a bad percepion model. The firs wo cases are when one or boh condiions can no be saisfied. In he firs case, here are hree people walking by, bu he people are inroduced in he inpu image sequence one by one, which violaes he firs condiion. In he second case, here are wo groups of auomobiles passing by, and hey are inroduced sequenially wih a big ime inerval beween hem. In addiion, he auomobiles move fas enough so ha he convergence speed of a paricle filer canno be sacrificed, which violaes he second condiion. The resuls for boh cases show how he muliple paricle filer approach overcomes he limiaion of a single paricle filer. The sabiliy of his approach is also clear. In he las case, i is observed how muliple paricle filers behave when wo people walk in differen direcions and inersec in he middle. The compuaion was performed on a Penium IV (2.1 GHz) compuer, and he image resoluion was fied o piels. The maimum number of paricle filers was fied o five, and for an individual paricle filer, he range of he number of paricles was se o ( ). The number of frames processed per second varies based on how many paricle filers have been creaed, bu roughly 10 frames were able o be processed. 2) Eperimenal Resuls: The snapshos of he muliple paricle filer racking muliple moving objecs are shown in Figure The upper rows of he figures show inpu image sequences and manually-racked moving objecs in he images. The manually-racked objecs are marked wih recangles. The lower rows show paricle filers and he covered area (he minimum recangular region enclosing each ellipsoid ha is generaed by he paricle clusering algorihm) by each paricle filer. Only converged paricle filer is visualized on he images. Each paricle filer is drawn wih differen colored dos, and he covered areas are marked wih recangles. The eperimenal resul of he firs case is shown in Figure 18. The esimaion process sars wih a single paricle filer. When he firs person eners ino he field of view of he camera as in Figure 18 (a), he paricle filer converges and sars o rack he person as in Figure 18 (d), and a new paricle filer is creaed o eplore he remained area. When he second person eners as in Figure 18 (b), he new paricle filer converges and sars o rack he second person as in Figure 18 (e), and anoher paricle filer is creaed. This process is repeaed whenever a new objec is inroduced. A he end when hree people are in he inpu image as in Figure 18 (c), he oal number of paricle filers becomes four; hree filers for people and one era filer o eplore. Figure 19 shows he eperimenal resul of he second case. The esimaion process is performed in he same way wih he firs case. Whenever a new auomobile is inroduced, a new paricle filer is creaed. When he auomobile leaves from he field of view of he camera, he paricle filer ha racks he

15 TECHNICAL REPORT CRES , CENTER FOR ROBOTICS AND EMBEDDED SYSTEMS, UNIVERSITY OF SOUTHERN CALIFORNIA 14 (a) = 49 (b) = 95 (c) = 132 (d) = 49 (e) = 95 (f) = 132 Fig. 18. Tracking people: There are hree people walking by. The upper row shows he inpu image sequence wih manually-racked objecs, and he lower row shows he paricle filer oupus and clusering resuls. (a) = 51 (b) = 66 (c) = 85 (d) = 51 (e) = 66 (f) = 85 Fig. 19. Tracking auomobiles: There are wo cars passing by followed by a hird car. The upper row shows he inpu image sequence wih manually-racked objecs, and he lower row shows he paricle filer oupus and clusering resuls. TABLE II PERFORMANCE OF MOVING OBJECT DETECTION ALGORITHM Case Frames Moions Deeced True + False + Deecion Rae Avg. Error (1) Pedesrians % 8.68 (2) Auomobiles % (3) Inersecion % 12.34

16 TECHNICAL REPORT CRES , CENTER FOR ROBOTICS AND EMBEDDED SYSTEMS, UNIVERSITY OF SOUTHERN CALIFORNIA 15 (a) = 27 (b) = 38 (c) = 67 (d) = 27 (e) = 38 (f) = 67 Fig. 20. Inersecional paricles: There are wo people inersecing in he image. The upper row shows he inpu image sequence wih manually-racked objecs, and he lower row shows he paricle filer oupus and clusering resuls. auomobile diverges and is desroyed evenually. The eperimenal resul of he hird case is shown in Figure 20. There are wo people walking in differen direcions as in Figure 20 (a), and wo paricle filers are creaed o rack hem individually as in Figure 20 (d). The pedesrians inersec in he middle, and keep walking in each direcion. Figure 20 (e) and (f) demonsrae ha he paricle filers rack hem successfully wihou being confused by he inersecion. The deailed evaluaion resul is shown in Table II. There were unracked moions in common; however, i happens only righ afer a new objec is inroduced and before a filer converges on he objec. Once a paricle filer converges and is associaed wih he objec, i never fails o rack he objec. For he second case, he deecion rae is lower han he oher wo cases. This is reasonable since he moion of an auomobile is much faser han ha of a person, and i ook longer for a paricle filer o converge on i. The false-posiives observed in he second case are also relaed o he faser moion. When an auomobile leaves from he camera field of view, i disappears quickly enough so ha he paricle filer says converged for one or wo frames. In general, he muliple paricle filer approach shows sable performance for all cases. C. Close he loop: Following a Moving Objec The proposed racking sysem is inegraed wih a robo conrol loop, and is robusness and real-ime capabiliy are esed. For his eperimen, he ask of a robo is o wai for a moving objec o appear and follow he objec. 1) Sysem Design and Implemenaion: A robo needs wo capabiliies o accomplish he ask: moion deecion and racking, and local navigaion. For moion deecion and racking, he sysem described in Secion III-V is uilized. This Fig. 21. Conrol archiecure for moion following sysem: The moion racker bo represens he proposed moion racking sysem. module akes he sequence of camera images and he laser range scans as inpus, and compues he eisence of moving objec(s) and he esimaion of arge posiions in he robo s local coordinaes. For local navigaion, VFH+ (Vecor Field Hisogram +) [34] algorihm is implemened. Inernally, VFH+ algorihm performs wo asks: (1) i rerieves range scans from a laser rangefinder, and build a local occupancy grid map for obsacle avoidance, and (2) when he esimaed arge posiion is given, he algorihm generaes boh ranslaional and roaional moor commands for poin-o-poin navigaion. The sysem archiecure is presened in Figure 21. The implemened sysem was deployed on a Segway RMP robo. The compuaion was performed on an embedded compuer (Ahlon 1.0 GHz) on he robo, and he image resoluion was fied a piels. 2) Eperimenal Resul: The snapshos of he robo following a person are shown in Figure 22. When he person enered ino he field of view of he camera as in Figure 22 (a), he paricle filer converged on him, and he Segway sared

17 TECHNICAL REPORT CRES , CENTER FOR ROBOTICS AND EMBEDDED SYSTEMS, UNIVERSITY OF SOUTHERN CALIFORNIA 16 o follow i. As shown in Figure 22 (b)-(e) and (h)-(i), here were auomobiles, a golf car, and pedesrians passing by in he background. However, he racking sysem was no confused by hem; he Segway was able o rack he person wihou a single failure. When he person sopped and sood sill as in Figure 22 (l), he paricle filer diverged, which made he robo sop. However, when he person re-sared o walk as in Figure 22 (m), he paricle filer was able o deec he moion again, and he robo re-sared o follow he person. VII. CONCLUSION We have presened a se of algorihms for muliple moion racking from a mobile robo in real-ime. There are hree challenges: 1) Compensaion for he robo ego-moion: The ego-moion of he robo was direcly measured using corresponding feaure ses in wo consecuive images obained from a camera rigidly aached o he robo. In order o eliminae he unfavorable effec of a moving objec in he image sequence, an oulier deecion algorihm has been proposed. 2) Transien and srucural noise: An adapive paricle filer has been designed for robus moion racking. The posiion and velociy of a moving objec were esimaed by combining he percepion model and he moion model incremenally. Also, he muliple arge racking sysem has been designed using muliple paricle filers. 3) Sensor fusion: The deph informaion from a laser rangefinder was projeced ino he image space, and he parial 3D posiion informaion was consruced in he region of overlap beween he range daa and he image daa. The proposed algorihms have been esed wih various configuraions in oudoor environmens. Firs, he algorihms were deployed on hree differen plaforms (Roboic Helicoper, Segway RMP, and Pioneer2 AT), and esed in differen environmens. The eperimenal resuls showed ha various ype of ego-moions were successfully eliminaed from inpu images, and he paricle filer was able o rack moions robusly. Second, he muliple arge racking algorihm was esed for differen ypes of moions. The eperimenal resuls show ha muliple paricle filers are creaed and desroyed dynamically o rack muliple arges inroduced a differen imes. Lasly, he racking algorihm was inegraed wih a robo conrol loop o es is real-ime capabiliy, and he ask of following a moving objec was successfully accomplished. The proposed algorihms are epeced o be uilized in various applicaion domains as a key enabler. Localizaion and mapping problems have been sudied acively by he mobile roboics communiy, and here are many well-developed echniques widely used. However, mos echniques assume a saic environmen, and heir performance is degraded significanly when here are dynamic objecs in an environmen. Our algorihm provides a robus mehod o deec dynamic objecs in an environmen, and i can be eploied in a pre-processing sep o filer ou daa ha are associaed wih he dynamic objecs. Safe navigaion is anoher fundamenal problem in mobile roboics, and mos soluions generae moion commands based on local posiions of obsacles. However, hose soluions become unreliable when obsacles are dynamic, especially when he obsacles move faser han he robo. In his case, a mobile robo needs o predic obsacle posiion in he near fuure o avoid collision. The moion velociy (speed and direcion) esimaion capabiliy of our algorihm could fulfill his requiremen. Human-robo ineracion is acive research area in service robo applicaions, and locaing a subjec o inerac wih is a key problem. Our algorihm is applicable in his regard. Needless o say, surveillance and securiy applicaions could make use of our algorihms o deec and rack moving arges. ACKNOWLEDGMENT This work is suppored in par by DARPA grans DABT , and A (via UPenn) under he Mobile Auonomous Robo Sofware (MARS) program, DOE RIM under gran DE-FG03-01ER45905, and NSF CAREER gran IIS REFERENCES [1] D. Wolf and G. S. Sukhame, Online simulaneous localizaion and mapping in dynamic environmens, in Proceedings of he Inernaional Conference on Roboics and Auomaion, New Orleans, Louisiana, April 2004, pp [2] D. Schulz, W. Burgard, D. Fo, and A. B. Cremers, Tracking muliple moving arges wih a mobile robo using paricle filers and saisical daa associaion, in Proceedings of he 2001 IEEE Inernaional Conference on Roboics and Auomaion, 2001, pp [3] M. Monemerlo, S. Thrun, and W. Whiaker, Condiional paricle filers for simulaneous mobile robo localizaion and people-racking, in Proceedings of he IEEE Inernaional Conference on Roboics and Auomaion, Washingon DC, May 2002, pp [4] C. Tomasi and T. Kanade, Deecion and racking of poin feaures, Carnegie Mellon Universiy, Pisburgh, PA, Tech. Rep. CMU-CS , April [5] A. Censi, A. Fusiello, and V. Robero, Image sabilizaion by feaures racking, in Proceedings of he 10h Inernaional Conference on Image Analysis and Processing, Venice, Ialy, Sepember 1999, pp [6] I. Zoghlami, O. Faugeras, and R. Deriche, Using geomeric corners o build a 2D mosaic from a se of images, in Proceedings of he IEEE Conference on Compuer Vision and Paern Recogniion, 1997, pp [7] B. D. Lucas and T. Kanade, An ieraive image regisraion echnique wih an applicaion o sereo vision, in Proceedings of he 7h Inernaional Join Conference on Arificial Inelligence, 1981, pp [8] S. Srinivasan and R. Chellappa, Image sabilizaion and mosaicking using he overlapped basis opical flow field, in Proceedings of IEEE Inernaional Conference on Image Processing, Ocober 1997, pp [9] M. Irani, R. Rousso, and S. Peleg, Recovery of ego-moion using image sabilizaion, in Proceedings of he IEEE Compuer Vision and Paern Recogniion, March 1994, pp [10] P. Nordlund and T. Uhlin, Closing he loop: Deecion and pursui of a moving objec by a moving observer, Image and Vision Compuing, vol. 14, pp , May [11] D. Murray and A. Basu, Moion racking wih an acive camera, IEEE Transacions on Paern Analysis and Machine Inelligence, vol. 16, no. 5, pp , May [12] G. L. Foresi and C. Micheloni, A robus feaure racker for acive surveillance of oudoor scenes, Elecronic Leers on Compuer Vision and Image Analysis, vol. 1, no. 1, pp , [13] A. Yilmaz, K. Shafique, N. Lobo, X. Li, T. Olson, and M. a. Shah, Targe-racking in FLIR imagery using mean-shif and global moion compensaion, in Workshop on Compuer Vision Beyond he Visible Specrum, Kauai, Hawaii, December 2001, pp [14] A. Behrad, A. Shahrokni, and S. A. Moamedi, A robus vision-based moving arge deecion and racking sysem, in he Proceeding of Image and Vision Compuing Conference, Universiy of Oago, Dunedin, New Zealand, November 2001.

18 TECHNICAL REPORT CRES , CENTER FOR ROBOTICS AND EMBEDDED SYSTEMS, UNIVERSITY OF SOUTHERN CALIFORNIA 17 (a) (b) (c) (d) (e) (f) (g) (h) (i) (j) (k) (l) Fig. 22. (m) (n) (o) Snapshos of a Segway RMP robo following a person