Motion analysis for event detection and tracking with a mobile omnidirectional camera

Transcription

1 Multimedi Systems 10: (2004) Digitl Oject Identifier (DOI) /s Multimedi Systems Motion nlysis for event detection nd trcking with moile omnidirectionl cmer Trk Gndhi, Mohn M. Trivedi Computer Vision nd Rootics Reserch Lortory, University of Cliforni Sn Diego, USA (e-mil: {tgndhi, Pulished online: 11 Octoer 2004 c Springer-Verlg 2004 Astrct. A moile pltform mounted with omnidirectionl vision sensor (ODVS) cn e used to monitor lrge res nd detect interesting events such s independently moving persons nd vehicles. To void flse lrms due to extrneous fetures, the imge motion induced y the moving pltform should e compensted. This pper descries formultion nd ppliction of prmetric egomotion compenstion for n ODVS. Omni imges give 360 view of surroundings ut undergo considerle imge distortion. To ccount for these distortions, the prmetric plnr motion model is integrted with the trnsformtions into omni imge spce. Prior knowledge of pproximte cmer clirtion nd cmer speed is integrted with the estimtion process using Byesin pproch. Itertive, corse-to-fine, grdient-sed estimtion is used to correct the motion prmeters for virtions nd other inccurcies in prior knowledge. Experiments with cmer mounted on vrious types of moile pltforms demonstrte successful detection of moving persons nd vehicles. Keywords: Motion detection Opticl flow Pnormic vision Dynmic vision Moile roots Intruder detection Surveillnce 1 Introduction nd motivtion Computer vision reserchers hve for long recognized the importnce of visul-surveillnce-relted pplictions while pursuing some of the outstnding reserch issues in dynmic scene nlysis, motion detection, feture extrction, pttern nd ctivity nlysis, nd iometric systems. Recent world events demnd prcticl nd roust deployment of videosed solutions for wide rnge of pplictions 11,16,30, 33]. Such wider cceptnce of the need for the technology does not men tht these systems re indeed redy for deployment. There re mny importnt nd difficult reserch prolems tht remin to e solved. In this pper we present study focused on one such chllenging reserch prolem, tht of developing n utonomous system tht cn serve s moile sentry to perform the tsks ssigned to someone posted on gurd duty round the perimeter of se. The moile sentry with video cmers should e le to detect interesting events nd record nd report the nture nd loction of the event in rel time for further processing y humn opertor. This is indeed n mitious gol nd it requires the resolution of severl importnt prolems from computer vision nd intelligent rootics. In this pper we focus on the prolem of detecting nd compensting for the egomotion of moile pltform. One of the novel fetures of our reserch is the omnidirectionl video strems we use s the input to the vision system. When cmer is sttionry, ckground sutrction is often used to extrct moving ojects 20,39]. However, when the cmer is moving, the ckground lso undergoes egomotion, which should e compensted. To distinguish ojects of interest from extrneous fetures on the ground, the ground is usully pproximted y plnr surfce whose egomotion is modeled using projective trnsform 12,29] or its linerized version. Using this model, the egomotion of the ground cn e compensted to seprte ojects with independent motion or height. This pproch hs een widely used for oject detection from moving pltforms 6,13,29]. Omnidirectionl vision sensors (ODVS), or omnicmers, tht give 360 field of view of the surroundings re very useful for pplictions such s surveillnce 5,8], root nvigtion 42], locliztion 26], nd wide seline stereo 38, 43]. The ook y Benosmn 3] gives comprehensive review of the theory nd pplictions of omnicmers. Motion estimtion from moving ODVS cmers hs recently een topic of gret interest. Rectiliner cmers usully hve smller field of view, which often cuses the focus of expnsion to lie outside the imge, mking motion estimtion sensitive to cmer orienttion. Also, the motion field produced y trnsltion long the horizontl direction is similr to tht from rottion out verticl xis. As noted y Gluckmn nd Nyr 18], ODVS cmers void oth these prolems thnks to their wide field of view. They project the imge motion on sphericl surfce using Jcoins of trnsformtions to determine egomotion of moving pltform in terms of trnsltion nd rottion of the cmer. Vsslo et l. 41] use the sphericl projection to determine the egomotion of moving pltform in terms of trnsltion nd rottion of the cmer. Ex-

2 132 T. Gndhi, M.M. Trivedi: Motion nlysis for event detection nd trcking with moile omnidirectionl cmer periments re performed using rootic pltforms in n indoor environment, nd the egomotion estimtes re compred with those from odometry. Shkerni et l. 35] use the concept of ck-projection flow, where the imge motion is projected to virtul curved surfce insted of sphericl surfce nd mke the Jcoins of trnsformtion simpler. Using this concept, they hve dpted egomotion lgorithms for use with ODVS sensors. Results using simulted imge sequences show the sic fesiility of the pproch. In our own reserch, the emphsis is on roustness, efficiency, nd pplicility in outdoor environments encountered in surveillnce nd physicl or se security. The min contriution of this pper is to perform detection of events, such s independently moving persons nd utomoiles from ODVS imge sequences, nd pply it to video sequences otined from moving pltform for surveillnce pplictions. 2 Egomotion compenstion frmework for ODVS video Prmetric motion estimtion sed on imge grdients, lso known s the direct method, hs een used for rectiliner cmers for plnr motion estimtion, ostcle detection, nd motion segmenttion 24,28]. The dvntge of the direct methods is tht they cn use motion informtion not only from cornerlike fetures ut lso from edges, which re usully more numerous in n imge. On the other hnd, direct methods re more chllenging for implementtion, especilly in outlier removl, nd it is more difficult to trck over frmes. A comprison of the corner-sed nd direct-grdient-sed methods is given in Tle 1. Here, the concept of direct method is extended for use with ODVS. This pproch ws lso used for detecting surrounding vehicles from moving cr in 15,23]. An ODVS gives full 360 view of the surroundings, which reduces the motion miguities often present in rectiliner cmers. However, the imges undergo considerle distortion, which should e ccounted for during motion estimtion. The lock digrm of the event detection system is shown in Fig. 1. The initil estimtes of the the ground plne motion prmeters re otined using pproximte knowledge out the cmer clirtion nd speed. Using these prmeters, one of the frmes is wrped towrd nother frme to compenste the motion of the ground plne. However, the motion of fetures hving independent motion or height ove the ground plne is not fully compensted. To detect these fetures, the normlized imge difference etween the two imges is computed using temporl nd sptil grdients. This suppresses the fetures on the ground plne nd enhnces the ojects of interest. Morphologicl nd other postprocessing is performed to further suppress the ground fetures resulting from ny resid- Fig. 1. Event detection nd recording system sed on egomotion compenstion from moving pltform: lock digrm ul motion nd get the positions of the ojects. The detected ojects re then trcked over frmes to form events. However, the clirtion nd speed of the cmer my not e known ccurtely. Furthermore, the cmer could virte during the motion. For this reson, the motion of the ground plne my not e fully compensted, leding to misses nd flse lrms. In order to improve the performnce, motion prmeters re itertively corrected using the sptil nd temporl grdients of the motion-compensted imges using opticl flow constrint in corse to fine frmework. The motion informtion contined in these grdients is optimlly comined with the prior knowledge of the motion prmeters using Byesin frmework similr to 29]. Roust estimtion is used to seprte the ground plne fetures from other fetures. The following sections del with the individul locks descried ove, long with the pproprite formultion for ODVS. 3 ODVS motion trnsformtions To compenste the motion of the omnidirectionl cmer, the trnsformtion due to ODVS should e comined with tht due to motion. These trnsforms re discussed elow. Tle 1. Comprison etween corner-sed nd grdient-sed motion estimtion Corner-sed methods Grdient-sed (direct) methods Determines motion of individul fetures Fits motion model to entire or prt of scene Only cornerlike fetures used Edge fetures used (more numerous) Esier to trck over frmes More difficult to trck over frmes Esier to identify outliers Outlier removl more difficult Esier to implement More difficult to implement

3 T. Gndhi, M.M. Trivedi: Motion nlysis for event detection nd trcking with moile omnidirectionl cmer 133 c Fig.2. Omnidirectionl vision sensor (ODVS). A typicl imge from n ODVS. c Trnsformtion to perspective pln view 3.1 Flt-plne trnsformtion The ODVS used in this work consists of hyperolic mirror nd cmer plced on its xis. It elongs to clss of cmers known s centrl pnormic ctdioptric cmers 3]. These cmers hve single viewpoint tht llows the imge to e suitly trnsformed to otin perspective views. Figure 2 shows photogrph of n ODVS mirror. An imge from cmer mounted with n ODVS mirror is shown in Fig. 2. It is seen tht the cmer covers 360 field of view round its center. However, the imge it produces is distorted with stright lines trnsformed into curves. A flt-plne trnsformtion is pplied to the imge to produce perspective view looking downwrds s shown in Fig. 2c, where the distortion is considerly reduced. Detils of this trnsformtion re discussed elow. The geometry of hyperolic ODVS is shown in Fig. 3. According to the mirror geometry, light ry from the oject towrd the viewpoint t the first focus O is reflected so tht it psses through the second focus, where conventionl rectiliner cmer is plced. The eqution of the hyperoloid is given y (Z c) 2 2 X2 + Y 2 2 =1, where c = Let P =(X, Y, Z) T denote the homogenous coordintes of the perspective trnsform of ny 3D point λp on ry OP, where λ is the scle fctor depending on the distnce of the 3D point from the origin. It cn e shown 1,22,35] tht the reflection in the mirror gives the point p =( x, y) T on the imge plne of the cmer using the flt-plne trnsform F : ( ) ( ) x q 1 X F (P )=p = =, (1) y q 2 Z + q 3 P Y where q 1 = c 2 2,q 2 = c 2 + 2,q 3 =2c, P = X 2 + Y 2 + Z 2. Note tht the expression for imge coordintes p is independent of the scle fctor λ. The pixel coordintes w =(u, v) T re then otined y using the clirtion mtrix K of the conventionl cmer composed of the focl lengths f u,f v, opticl center coordintes (u 0,v 0 ) T, nd cmer skew s. ( ) ( ) w p = K 1 1 or u v = f u s u 0 0 f v v 0 x y. (2) This trnsform cn e used to wrp n omni imge to pln perspective view. To convert perspective view ck to omni view, the inverse flt-plne trnsform p cn e used: ( ) p = x y = K 1 1 u v, (3) 1 1 F 1 (p) =P = X q 1 x Y = q 1 y. Z q 2 q 3 x2 + y 2 +1 (4) It should e noted tht the trnsformtion of omni to perspective view involves very different mgnifictions in different prts of the imge. For this reson, the qulity of the imge deteriortes if the entire imge is trnsformed t one time. Hence, it is desirle to perform motion estimtion directly in the ODVS domin ut use the ove trnsformtions to mp the loctions to the perspective domin s required. 3.2 Plnr motion trnsformtion To detect ojects with motion or height, the motion of the ground is modeled using plnr motion model 12,27]. Let P A nd P B denote the perspective trnsforms of point on the ground plne in the homogenous coordinte systems corresponding to two positions A nd B of the moving cmer. These re relted y Fig. 3. Omnidirectionl cmer geometry λ B P B = λ A RP A + D = λ A RP A + D/λ A ], (5)

4 134 T. Gndhi, M.M. Trivedi: Motion nlysis for event detection nd trcking with moile omnidirectionl cmer Fig. 4. Trnsforming pixel from omni imge A to omni imge B using (1) inverse clirtion mtrix K 1, (2) inverse flt-plne trnsform F 1, (3) projective trnsform H for plnr motion from A to B, (4) flt-plne trnsform F, (5) clirtion mtrix K where R nd D denote the rottion nd trnsltion etween the cmer positions nd λ A,λ B depend on the distnce of the ctul 3D point. Let the ground plne stisfy the following eqution t the cmer position A: or λ A K T P A =1 1/λ A = K T P A. Sustituting the vlue of 1/λ A into Eq. 5, it is seen tht P A nd P B re relted y projective trnsform: λ B P B = λ A R + DK T ] P A = λ A HP A (6) or P B HP A within scle fctor. This reltion hs een widely used to estimte plnr motion for perspective cmers. For performing motion compenstion using omnidirectionl cmers, the ove projective trnsform should e comined with the flt-plne trnsform s well s cmer clirtion mtrix to wrp every point in one imge towrd nother. The complete trnsform for wrping is shown in Fig Prmetric motion estimtion for ODVS This section descries the min contriution of the pper. Direct methods sed on imge grdients hve een pplied for estimting the motion prmeters for rectiliner cmers 4, 24]. Here, the direct method is generlized for ODVS cmers. Informtion from imge grdients is comined with the priori known informtion out the cmer motion nd clirtion in Byesin frmework to otin optiml estimtes of motion prmeters for egomotion compenstion. 4.1 Use of opticl flow constrint Under fvorle conditions, the sptil grdients (g u,g v ), the temporl grdient g t, nd the residul imge motion ( u, v) T fter current motion compenstion stisfy the opticl flow constrint 21]: g u u + g v v + g t =0. (7) Fig. 5.Aperture prolem. In the cse of n edge, only the component of motion norml to the edge cn e determined. In the cse of corner, the perture prolem is voided, nd the motion cn e uniquely determined However, there is only one eqution etween two unknowns for ech point. For this reson, only the norml flow, i.e., flow in the direction of the grdient, cn e determined using single point. This is known s the perture prolem nd is illustrted in Fig. 5. To solve this prolem, Lucs nd Knde 31] ssumed tht imge motion is pproximtely constnt in smll window round every point. Using this constrint, more equtions re otined using the neighoring points, nd the full opticl flow cn e estimted using lest squres. Such n estimte is relile ner cornerlike points where window hs grdients in different directions. This is s seen in Fig. 5. This method hs een used y Knde et l. 36] to find nd trck cornerlike fetures over n imge. However, in the cse of ODVS the ssumption of uniform opticl flow needs to e modified due to the nonliner ODVS trnsform. Dniilidis 9] hs generlized the opticl flow estimtion to ODVS cmers. However, tthis pproch would use the motion informtion only on cornerlike fetures. However, the edge fetures lso hve motion informtion. To use this informtion, the imge grdients cn e used directly to estimte the model prmeters. This pproch is known in the literture s the direct method of motion estimtion nd hs een extensively used in ostcle detection using rectiliner cmers 4,24]. Usully linerized version of projective trnsform is used: u = 1 u + 2 v u uv, v = 4 u + 5 v uv + 8 v 2. The expressions of imge motion re sustituted into the opticl flow constrint in Eq. 7 to give g u ( 1 u+ 2 v )+g v ( 4 u+ 5 v )+g t =0. This gives one eqution for every point in eight prmeters tht cn e solved using liner lest squres. Since the qudrtic prmeters re more sensitive to noise, six-prmeter ffine model is lso used. 4.2 Generliztion for ODVS To pply the motion estimtion to ODVS cmers, the nonliner flt-plne trnsform is used to go from omni to perspective domin nd ck. Since nonlinerity hs to e delt with nywy, the projective trnsform H is used insted of liner model so tht lrge motions cn e hndled etter. The motion prmeters in the projective trnsform re prmeterized s h = ( h 1 h 2 h 3 h 4 h 5 h 6 h 7 h 8 ) T

5 T. Gndhi, M.M. Trivedi: Motion nlysis for event detection nd trcking with moile omnidirectionl cmer 135 with H = h 1 h 2 h 3 h 4 h 5 h 6. H 33 h 7 h 8 1 The opticl flow constrint eqution is stisfied only for smll imge displcements up to 1 or 2 pixels. To estimte lrger motions, corse to fine pyrmidl frmework 25,37] is used. In this frmework, multiresolution Gussin pyrmid is constructed for djcent imges in the sequence. The motion prmeters re first computed t the corsest level, nd the imge points t the next finer level re wrped using the computed motion prmeters. The residul motion is computed t the finer level, nd the process is repeted until the finest level is reched. Even within ech level, multiple itertions of wrping nd estimtion cn e performed. Let h e the ctul vlue of the motion prmeter vector nd ĥ the current estimte. Using the current estimte, the second imge B is wrped towrd the first imge A to get the wrped imge B. Then, the trnsformtion etween A nd B cn e expressed pproximtely in terms of h = h ĥ. Let w A =(u A,v A ) T e the projection of point on the plnr surfce in imge A. Then, the projection w B of the sme point in wrped imge B is function of w A s well s h,given using composition of opertions shown in Fig. 4. The opticl flow constrint etween imges A nd B is then given y ( gu g v ) wb w A ]= g t + η, where η ccounts for the rndom noise in the temporl imge grdient. For N points on the plnr surfce, the constrints cn e expressed in mtrix form: z = c( h)+v, where every row i of the eqution represents the constrint for single imge point with c i ( h) = ( g u g v )i w B (w A; h) w A ] z i = (g t ) i, v i = η i. (8) Due to the flt-plne nd the projective trnsforms, the function c( ) is nonliner. Hence, stte estimte ĥ nd its covrince P re itertively updted using the mesurement updte equtions of the iterted extended Klmn filter 2], with C denoting the Jcoin mtrix of c( ). P γc T R 1 C + P 1 ] 1, (9) ] ĥ ĥ + ĥ = ĥ + P γc T R 1 z P 1 (ĥ h ), (10) where R is the covrince of the temporl grdient mesurements, h is the prior vlue of the stte otined from cmer clirtion nd velocity, nd P is the prior covrince. The mtrix R is tken s digonl mtrix to simplify clcultions. However, this would men ssuming tht the pixel grdients re independent, which my not relly e the cse since grdients re computed from multiple pixels. Hence, the fctor γ 1 is used to ccommodte the interdependence of the grdient mesurements. To compute the Jcoin C, ech row C i is expressed using the chin rule: ( ) ( ) ci c w B p B P B C i = =, (11) h w B p B P B h i where P B =(X B,Y B,Z B ) T, p B =(x B,y B ) T, nd w B = (u B,v B ) T re, respectively, the coordintes of point i in the mirror, imge, nd pixel coordinte systems for cmer position B. Differentiting Eq. 8 w.r.t. w B gives ( ) c = ( g u g v w )i. B The clirtion Eq. 2 cn e differentited to otin ( ) ( ) wb fu s =. p B 0 f v i i The Jcoin of the flt-plne trnsform is otined y differentiting Eq. 1 t P = P B s ( ) ( ) xb x B x B pb P B = i X B y B X B Y B y B Y B Z B y B Z B i 1 = (q 2Z B + q 3 P B ) i P B i ( ) q3x BX B q 1 P B q 3x BY B q 3x BZ B. q 3y BX B q 3y BY B q 1 P B q 3y BZ B Since the ODVS trnsforms giving p A nd p B do not chnge if the homogenous coordintes P A nd P B re chnged y scle fctor, we cn scle the right-hnd side of Eq. 6 to give P B = 1 HP A = h 1X B + h 2 Y B + h 3 Z B h 4 X B + h 5 Y B + h 6 Z B. H 33 h 7 X B + h 8 Y B + Z B Tking the Jcoin w.r.t. h =(h 1...h 8 ) gives ( ) ( ) XB PB Y B Z B = X B Y B Z B 0 0. h i X B Y B i 4.3 Outlier removl The estimte given ove is optiml only when ll points relly elong to the plnr surfce nd the underlying noise distriutions re Gussin. However, the estimtion is highly sensitive to the presence of outliers, i.e., points not stisfying the ground motion model. These fetures should e seprted using roust method. To reduce the numer of outliers, the rod region of interest is determined using clirtion informtion, nd the processing is done only in tht region to void extrneous fetures. To detect outliers, n pproch similr to the dt snooping pproch discussed in 10] hs een dpted for Byesin estimtion. In this pproch, the error residul of ech feture is compred with the expected residul covrince t every itertion, nd the fetures re reclssified s inliers or outliers. i

6 136 T. Gndhi, M.M. Trivedi: Motion nlysis for event detection nd trcking with moile omnidirectionl cmer 5 Dynmic event detection nd trcking Fig. 6. Hierrchicl motion estimtion lgorithm If point z i is not included in the estimtion of ĥ, i.e., is currently clssified s n outlier, then the covrince of its residul is ] Cov z i C i ĥ R + C i PC T i. However, if z in is included in the estimtion of ĥ, i.e., is currently clssified s n inlier, then it cn e shown tht the covrince of its residul is given y ] Cov z i C i ĥ R C i PC T i < R. Hence, to clssify in the next itertion, the Mhlnois norm of the residul is compred with threshold τ. For point currently clssified s inlier the following condition is used: R z i C i ĥ] + Ci PC T ] 1 ] i z i C i ĥ <τ. (12) For point currently clssified s inlier the covrince R is used in prctice insted of R C i PC T i in order to void nonpositive definite covrince ecuse of pproximtions due to nonlinerities. This would somewht increse the proility of clssifying s n outlier insted of inlier, which is to e on the sfer side. ] ] z i C i ĥ R 1 z i C i ĥ <τ (13) Note tht this method is effective only when there is some prior knowledge out the motion prmeters; otherwise the prior covrince P would ecome infinite. If there is no prior knowledge, roust estimtors cn e used s in 32]. The motion prmeter estimtion lgorithm is shown in Fig. 6. After motion compenstion, the fetures on the ground plne would e ligned etween the two frmes, wheres those due to sttionry nd moving ojects would e misligned. Imge difference etween the frmes would therefore enhnce the ojects nd suppress the rod fetures. However, the imge difference depends on residul motion s well s on the sptil grdients t tht point. In highly textured regions, the imge difference would e lrge even for smll residul motion, nd in less textured regions the imge difference would e smll even for lrge residul motion. To compenste this effect, normlized frme difference 40] is used. This is given t ech pixel y gt g 2 u + gv 2 k + (gu 2 + gv) 2, where g u,g v re sptil grdients nd g t is the temporl grdient. Constnt k is used to suppress the effect of noise in highly uniform regions. The summtion is performed over K K neighorhood of ech pixel. In fct, the normlized difference is smoothed version of the norml opticl flow nd hence depends on the mount of motion ner the point. Blos corresponding to oject fetures re otined using morphologicl opertions. Nery los re clustered into one, nd the cluster centroids re trcked from frme to frme y n lgorithm similr to 17]. For trcking, list contining the frme numer, unique ID, position, nd velocity of ech trck is mintined. The list is empty in the eginning. The following steps re performed to ssocite the trcks with fetures: At ech frme, ssocite ech existing trcks with the nerest feture in neighorhood window round the trck position. Use Klmn filter 2] to updte the trck with the feture. If no feture is found in the neighorhood window, only time updte is performed. For fetures not hving trcks in their neighorhood, crete new trck out of the feture nd updte it in the next frme. To keep the numer of trcks within ounds, delete the wekest trcks when the numer of trcks get too lrge. Merge the trcks tht re very close to ech other, hve nerly the sme velocity, nd tht re therefore ssumed to e from the sme oject. Disply trcks tht hve survived for stipulted numer of frmes long with the trck history. For ech trck tht survives over minimum numer of frmes, the originl ODVS imge is used to generte perspective view 22] of the event round the center of the ounding ox. 6 Experimentl vlidtion nd results A series of experimentl trils ws conducted to systemticlly evlute the cpilities nd performnce of the moile sentry system for event detection nd trcking. Three different types of ODVS mountings were utilized to exmine the generlity nd functionlities of the moile sentry system. The first tril involved cmer on n electric vehicle, the second

7 T. Gndhi, M.M. Trivedi: Motion nlysis for event detection nd trcking with moile omnidirectionl cmer 137 Fig. 7. An ODVS cmer mounted on n electric crt for moile sentry experimentl run ws using moile root, nd the third involved wlking person with helmet-mounted ODVS. The first experiment ws done most systemticlly in order to evlute the performnce. The other experiments re currently in the explortory stge nd more work is required to chrcterize their performnce. A relted ppliction pplying similr pproch to n utomoile-mounted ODVS is lso shown. 6.1 Cmer on electric crt The first experimentl tril of the moile sentry utilized the ODVS cmer mounted on n electric crt, s shown in Fig. 7. The crt ws driven on cmpus rod t speeds etween 2 nd 7 miles/h (pprox. 1 to 3 m/s). The pproximte speed of the crt ws determined using GPS nd used s n priori motion estimte. It ws lso oserved tht the ellipse corresponding to the entire FOV of the ODVS ws oscillting, due possily to reltive virtions etween the cmer nd the mirror or to the utomtic motion stiliztion in the cmer. These were suppressed y estimting the center of the FOV ellipse using Hough trnsform nd trnslting it to fixed position. The effect of the remining virtions were suppressed using the prmetric motion estimtion process. Figure 8 shows n imge from the ODVS video sequence. The estimted prmetric motion is shown using red rrows. Figure 8 shows the clssifiction of points into inliers (gry), outliers (white), nd unused (lck) points. It should e noted tht the outliers re usully identified when the edges re perpendiculr to the motion. When n edge is prllel to the motion, the perture prolem mkes it difficult to identify it. The estimtion is done using the inlier points only. An imge with the normlized frme difference etween the motioncompensted frmes is shown in Fig. 8c. It is seen tht the independently moving cr nd person stnd out wheres the sttionry fetures on the ground re ttenuted in spite of egomotion. Figure 8d shows the ounding oxes round the moving cr nd person fter postprocessing using morphologicl opertions. Since the lgorithm uses plnr motion model, sttionry ojects ove the ground induce motion prllx nd re c d Fig. 8. Detection of moving ojects from n ODVS mounted on n electric crt. Estimted prmetric motion of ground plne. Prts of the imge corresponding to the crt s well s distnt ojects re excluded from motion estimtion process. Fetures used for estimtion. Gry fetures re inliers, nd white fetures re outliers. c Motion-compensted difference imge. d Postprocessed imge showing detection of moving cr nd person. The ngle mde with x-xis in degrees is lso shown

8 138 T. Gndhi, M.M. Trivedi: Motion nlysis for event detection nd trcking with moile omnidirectionl cmer Tle 2. Performnce evlution. The right column shows the ground truth numer of relevnt events in the imge sequence. The other columns show the numer nd percentge of events detected y the system. The lst three rows show the numer of flse lrms due to sttionry ojects nd shdows. Ground truth is not relevnt here Minimum Ground trck length frmes frmes truth Totl events 14 (74%) 17 (89%) 19 Persons 9 (90%) 9 (90%) 10 Vehicles 5 (55%) 8 (89%) 9 Totl flse lrms 3 4 N.A. Sttionry ojects 1 2 N.A. Shdows 2 2 N.A. detected if they re sufficiently close to the cmer nd included in the region of interest. Figure 9 shows the detection of sttionry structure s well s moving person. Only the prts within the region of interest re detected. The centroids of the detected ounding oxes were trcked over time nd the trcks tht survived over ten or more frmes were identified. Typicl snpshots from these trcks were tken, nd the distortion due to ODVS ws corrected to get the perspective view looking towrd the trck position s in 22]. Figure 10 show the snpshots from these trcks, detecting the events. To evlute the lgorithm performnce, the detection results were compred with ground truth otined y mnully oserving the video sequence. The performnce ws compred for two different thresholds on the numer of frmes in which trck hs to survive to e detected s n event. Tle 2 shows the detection rte in terms of totl numer of events (ground truth) nd the numer of events ctully detected. Note tht sttionry ostcles nd shdows re clssified s flse lrms since they re currently not seprted from independently moving ojects. A lower threshold increses detection rte ut lso increses flse lrms. It ws oserved tht two events corresponding to moving person nd crt were not detected t ll for the following resons. The person nd crt were quite fr nd especilly the person ws smll in the imge. Furthermore, the cmer vehicle ws turning, inducing considerle rottionl egomotion. Also, the ojects were ner the oundry of the region of interest tht ws nlyzed. An imge of this person is shown in Fig. 11. Attriutes such s the time, durtion, nd position of the events were extrcted. The cmer position t the event time ws extrcted from the onord GPS. Assuming tht the point nerest to the cmer lies on the ground, the event loction with reference to the cmer could e computed. These were dded to the cmer position coordintes to otin the event position. the pproximte positions of the cmer s well s the ctul event were mpped s shown in Fig. 12. Tle 3 summrizes some of the events nd their ttriutes. 6.2 Cmer on moile root The second experimentl configurtion ws rootic pltform designed in our lortory. This pltform is clled the Moile Interctive Avtr (MIA) 19] in which cmers nd displys cn e mounted on semiutonomous root, s shown c d Fig. 9. Detection of sttionry structure in ddition to moving person. Note tht only the prt of the structure within the region of interest is detected

9 T. Gndhi, M.M. Trivedi: Motion nlysis for event detection nd trcking with moile omnidirectionl cmer 139 Fig. 12. Mp showing the position of the events in red nd the cmer position t tht time in lue. The egovehicle trck is mrked y the yellow line. The event IDs re leled in lck Fig. 10. Cptured events with their IDs. Detected persons nd vehicles. Shdows nd sttionry ostcles currently considered flse lrms Fig. 13. Moile Interctive Avtr: Semiutonomous rootic system used for evluting moile sentry Tle 3. Summriztion of event ttriutes. The left imge shows the originl ODVS imge t the time of the event, the middle imge shows the output of detection lgorithm, nd the right imge shows the snpshot of the event corrected for ODVS distortion Fig. 11. Originl imge corresponding to missed events. The moving person nd vehicle were fr from the cmer nd ner the region of interest oundry. Also, the cmer vehicle ws mking turn, inducing considerle rottionl egomotion in the imge Event ID: 65 Event time (snpshot): 16:01:08.3 Event durtion s]: 3.6 Event position m]: (7.1, -6.6) Cmer position m]: (7.8, -5.3) in Fig. 13, to interct with people t distnce. The root ws driven round the corridor of our uilding with people wlking round it. Figure 14 shows the detection of moving persons in one of the frmes. Snpshots of detected people re shown in Fig. 15. However, it ws noted tht the speed of the root ws much smller thn tht of people, which would men tht simpler methods could lso yield good results in this Event ID: 109 Event time (snpshot): 16:01:40.0 Event durtion s]: 1.9 Event position m]: (53.1, 1.8) Cmer position m]: (56.0, -3.1)

10 140 T. Gndhi, M.M. Trivedi: Motion nlysis for event detection nd trcking with moile omnidirectionl cmer Fig. 15. Some of the interesting events cptured y the moile root Fig. 16. ODVS cmer mounted on helmet. This configurtion enles cquisition of snpshots of surrounding people including their fces. However, there is considerle cmer rottion due to movement of hed nd ody tht should e compensted c d Fig. 14. Detection of moving persons in n imge sequence from moile root. Estimted prmetric motion of ground plne. The prt of the imge in the center, which imges the cmer itself, is not used for estimtion. Fetures used for estimtion. Gry fetures re inliers, nd white fetures re outliers. c Normlized imge difference fter motion compenstion. The moving person is detected ut the lines on the ground re not detected. d Postprocessing nd trcking output. The trck of the detected person with the ID is shown y the yellow line scenrio. Also, the height of the root ws smll, nd people s fces could not e effectively cptured. Fig. 16. In this configurtion, the cmer height ws pproximtely 2 m, which enled esy cpture of people s fces. The speed of the person with the helmet ws comprle to tht of other people. There ws lso considerle cmer rottion due to hed nd ody movement, which helped to test the lgorithm in the presence of lrge rottionl egomotions. Figure 17 shows the detection of moving person in one of the frmes. To reduce estimtion errors, the region of interest for motion nlysis ws truncted to remove the cmer s own imge, s well s the ojects ove the horizon. It is seen tht there is significnt rottionl motion etween two frmes. In spite of this motion, the moving person is seprted from ckground fetures such s the lines on the ground. However, if the rottionl motion is too lrge, the detection often deteriortes nd trcks get split into prts. Some of the snpshots of detected people shown re in Fig. 18. Unlike the first experiment, the events here consisted of the sme persons moving round the cmer. Also, there ws more reking of trcks due to lrge rottionl motion. Hence, insted of counting the numer of events detected, the orienttion of trcks in ech frme ws plotted ginst time in Fig. 19. The identities of the persons were mnully recorded nd re color-coded in the figure. The crosses show the trck reks. It ws oserved tht in sequence of 5 min (3000 frmes t 10 frmes per second), the persons were trcked when they were sufficiently close to the cmer nd the rottionl motion ws not very lrge. 6.3 Helmet-mounted cmer The third experimentl study for moile sentry involved person wlking with n ODVS mounted on helmet s shown in 6.4 Vehicle-mounted ODVS In relted ppliction 15,23], the event detection pproch ws pplied to n omnicmer mounted on n utomoile. The

11 T. Gndhi, M.M. Trivedi: Motion nlysis for event detection nd trcking with moile omnidirectionl cmer 141 Fig. 18. Some detected events from the helmet-mounted cmer Thet (rd) Time (s) Fig. 19. Time series showing the orienttion (thet) of the person with respect to the cmer. Ech color corresponds to different person. Trck reks re mrked y c ctul vehicle speed, otined from CAN us, ws used for the initil motion estimte. A video sequence of 36,000 frmes (20 min) ws processed nd vehicles on oth sides of the cr were detected y n lgorithm similr to the one descried ove, s shown in Fig. 20. The distortion due to omni imging ws removed to generte the ird s-eye view, s shown in Fig. 20. Figure 20c shows the plots of trck positions ginst time for segment of the video. 7 Summry nd discussion d Fig. 17. Detection of moving person in n imge sequence from helmet-mounted cmer. Estimted prmetric motion of ground plne. There is significnt rottionl motion tht is estimted y the lgorithm. Fetures used for estimtion. Gry fetures re inliers, nd white fetures re outliers. The prts of the imge in the center, ove the horizon, nd those hving smll imge grdients re not used in estimtion. c Normlized imge difference fter motion compenstion. The moving person is detected ut the lines on the ground re not detected. d Postprocessing nd trcking output. The trck of the detected person with the ID is shown y the yellow line. The position coordintes re computed y ssuming tht the point on the lo nerest the cmer is on the ground This pper descried n pproch for event detection using egomotion compenstion from moile omnidirectionl (ODVS) cmers. It pplied the concept of direct motion estimtion using imge grdients to ODVS cmers. The motion of the ground ws modeled s plnr motion, nd the fetures not oeying the motion model were seprted s outliers. An itertive estimtion frmework ws used for optimlly fusing the motion informtion in imge grdients with priori known informtion out the cmer motion nd clirtion. Corse to fine motion estimtion ws used nd the motion etween the frmes ws compensted t ech itertion. A scheme sed on dt snooping ws used to remove outliers. Experiments were performed y otining imge sequences from vrious types of moile pltforms nd detecting events such s moving persons nd utomoiles, giving stisfctory results. For future work, we pln to improve the roustness of the system especilly for correct locliztion of lrge ojects. The

12 142 T. Gndhi, M.M. Trivedi: Motion nlysis for event detection nd trcking with moile omnidirectionl cmer longitudinl position: Z m] time s] c Fig. 20. Detection of moving vehicles in n imge sequence using n omnidirectionl cmer mounted on moving cr. The trck history of the vehicle over numer of frmes is mrked. The trck ID nd the coordintes in the rod plne re lso shown. Bird s-eye view generted y removing the omni distortion, showing detected vehicles nd their coordintes. c Plot of the longitudinl position of vehicle trcks ginst time. The trcks re color coded s red, yellow, nd green ccording to incresing lterl distnce from the ego vehicle 100 such s size nd shpe. Lerning-sed pproches such s 34] would lso e useful for clssifiction. The method descried ove is pproprite for scenes where the ckground is predominntly plnr nd the foreground consists of outliers in form of smll ojects. If the scene is not tht simple, motion segmenttion should e performed long with estimtion. In the cse of scenes with multiple sttionry plnr surfces, the surfces hve the sme prmeters for rottion nd trnsltion ut different plnr normls 14]. Hence, the egomotion cn e prmeterized directly in terms of the liner nd ngulr velocity of the cmer nd the plne normls of ech plnr surfce. An itertive estimtion procedure tht estimtes ech plnr surfce seprtely ut uses the estimtes it otins for rottion nd trnsltion s strting points for estimting other plnr surfces could mke the process more roust to outliers. For exmple, if the scene consists of fetures fr wy from the cmer, their egomotion could e considered lmost pure rottion, hving only three degrees of freedom. These fetures could e used to estimte the pproximte rottion 40]. This rottion could e used s n initil estimte for the prts of the scene contining ground plne in order to determine the full plnr motion. This procedure could e comined with roust motion segmenttion method such s 32] to utomticlly seprte multiple plnr surfces. Alterntively, the motion prmeters cn e estimted using ootstrp method from smll ptches nd comine the ptches hving motion consistent with the ground plne s done y Ke nd Knde 28]. For 3D scenes with lrge vritions in depths, structure from motion pproch using epipolr constrint 7] is more pproprite. The plne+prllx method proposed y Irni ndanndn 24] cn lso e used for wide vriety of scenes including plnr, piecewise plnr, nd 3D. To discriminte etween independently moving ojects nd sttionry ojects ove the ground, the rigidity constrint 24] could e used in the plne+prllx frmework. We pln to generlize the piecewise plnr motion segmenttion s well s plne+prllx methods for use with ODVS cmers using nonliner motion models for complex scenes nd independent motion discrimintion. Acknowledgements. We re thnkful for the grnt wrded y the Technicl Support Working Group (TSWG) of the US Deprtment of Defense, which provided the primry sponsorship of the reported reserch. We lso thnk our collegues from the UCSD Computer Vision nd Reserch Lortory for their contriutions nd support. References lgorithm currently detects regions contining edges where motion informtion is significnt ut does not respond to uniform res of lrge ojects. Morphologicl opertions were helpful in comining the detected regions, ut systemtic pproch sed on region-sed segmenttion nd clustering my e more pproprite for getting ccurte locliztion in terms of ounding oxes. It ws lso oserved tht the trcks often got roken due to inccurte locliztion of the detected los. We pln to trck entire los insted of the centroids to otin more roust trcks. The events cn then e clssified into ctegories such s persons nd vehicles using criteri 1. Achler O, Trivedi MM (2002) Rel-time trffic flow nlysis using omnidirectionl video network nd fltplne trnsformtion. In: Workshop on Intelligent Trnsporttion Systems, Chicgo, IL, Br-Shlom Y, Li XR, Kirurjn T (2001) Estimtion with pplictions to trcking nd nvigtion. Wiley, New York 3. Benosmn R, Kng SB (2001) Pnormic vision: sensors, theory, nd pplictions. Springer, Berlin Heidelerg New York 4. Blck MJ, Anndn P (1996) The roust estimtion of multiple motions: prmetric nd piecewise-smooth flow fields. Comput Vision Imge Understnd 63(1):75 104

13 T. Gndhi, M.M. Trivedi: Motion nlysis for event detection nd trcking with moile omnidirectionl cmer Boult T, Erkin A, Lewis P, Michels R, Power C, Qin C, Yin W (1998) Frme-rte multi-ody trcking for surveillnce. In: Proc. DARPA Imge Understnding workshop 6. Crlsson S, Eklundh JO (1990) Oject detection using modelsed prediction nd motion prllx. In: Europen conference on computer vision, April 1990, pp Chng P, Herert M (2000) Omni-directionl structure from motion. In: IEEE workshop on omnidirectionl vision, Hilton Hed Islnd, SC, June IEEE Press, pp Cielnik G, Mildinovic M, Hmmrin D, Gornson L, Lilienthl A, Duckett T (2003) Appernce-sed trcking of persons with n omnidirectionl vision sensor. In: Proc. IEEE workshop on omnidirectionl vision 9. Dniilidis K, Mkdi A, Bulow T (2002) Imge processing in ctdioptric plnes: Sptiotemporl derivtives nd opticl flow computtion. In: Proc. IEEE workshop on omnidirectionl vision, June 2002, pp Dnuser G, Stricker M (1998) Prmetric model fitting: From inlier chrcteriztion to outlier detection. IEEE Trns Pttern Anl Mch Intell 20(2): Den KL (2003) Smrtcms tke im t terrorists. Wired News, June rticles/wired News Smrtcms.html 12. Fugers O (1993) Three-dimensionl computer vision: geometric viewpoint. MIT Press, Cmridge, MA 13. Gndhi T, Devdig S, Ksturi R, Cmps O (2000) Detection of ostcles using ego-motion compenstion nd trcking of significnt fetures. Imge Vision Comput 18(10): Gndhi T, Ksturi R (2000) Appliction of plnr motion segmenttion for scene text extrction. In: Proc. interntionl conference on pttern recognition, 1: Gndhi T, Trivedi MM (20040) Motion sed vehicle surround nlysis using omni-directionl cmer. In: Proc. IEEE symposium on intelligent vehicles (in press) 16. Gndhi T, Trivedi MM (2003) Motion nlysis of omnidirectionl video strems for moile sentry. In: 1st ACM interntionl workshop on video surveillnce, Berkeley, CA, Novemer 2003, pp Gndhi T, Yng MT, Ksturi R, Cmps O, Coror L, McCndless J (2003) Detection of ostcles in the flight pth of n ircrft. IEEE Trns Aerospce Electron Syst 39(1): Gluckmn J, Nyr S (1998) Ego-motion nd omnidirectionl cmers. In: Proc. interntionl conference on computer vision, pp Hll TB, Trivedi MM (2002) A novel interctivity environment for integrted intelligent trnsporttion nd telemtic systems. In: 5th interntionl IEEE conference on intelligent trnsporttion systems, Singpore, Septemer Hritoglu, Hrwood D, Dvis LS (2000) W4: Rel-time surveillnce of people nd their ctivities. IEEE Trns Pttern Anl Mch Intell 22(8): Horn B, Schunck B (1981) Determining opticl flow. In: DARPA Imge Understnding workshop, pp Hung KC, Trivedi MM (2003) Video rrys for rel-time trcking of persons, hed nd fce in n intelligent room. MchVision Appl 14(2): Hung K, Trivedi MM, Gndhi T (2003) Driver s view nd vehicle surround estimtion using omnidirectionl video strem. In: IEEE symposium on intelligent vehicles, Columus, OH, June Irni M, Anndn P (1998) A unified pproch to moving oject detection in 2D nd 3D scenes. IEEE Trns Pttern Anl Mch Intell 20(6): Jähne B, Hußecker H, Geißler P (1999) Hndook of computer vision nd pplictions, vol 2. Acdemic, Sn Diego, pp Jogn M, Leonrdis A (2000) Roust locliztion using pnormic view-sed recognition. In: Proc. interntionl conference on pttern recognition, pp Kntni K (1993) Geometric computtion for mchine vision. Oxford University Press, Oxford 28. Ke Q, Knde T (2003) Trnsforming cmer geometry to virtul downwrd-looking cmer: roust ego-motion estimtion nd ground-lyer detection. In: Proc. IEEE conference on computer vision nd pttern recognition, June 2003, pp I: Kruger W (1999) Roust rel time ground plne motion compenstion from moving vehicle. Mch Vision Appl 11: Lwson S (2002) Yes, you re eing wtched. PC World, 27 Decemer rticle/0,id,108121,00.sp 31. Lucs B, Knde T (1981) An itertive imge registrtion technique with n ppliction to stereo vision. In: Interntionl joint conference on rtificil intelligence, pp Odoez JM, Bouthemy P (1998) Direct incrementl modelsed imge motion segmenttion for video nlysis. Signl Process 66: Rmsey D (2003) Reserchers work with pulic gencies to enhnce super owl security, 15 Ferury superowl.html. 34. Spthrishi M, Hmpshire JB, Khosl PK (2000) Agent-sed moving oject correspondence using differentil discrimintive dignosis. In: Proc. IEEE conference on computer vision nd pttern recognition, June 2000, 2: Shkerni O, Vidl R, Sstry S (2003) Omnidirectionl egomotion estimtion from ck-projection flow. In: Proc. IEEE workshop on omnidirectionl vision, June Shi J, Tomsi C (1994) Good fetures to trck. In: Proc. IEEE conference on computer vision nd pttern recognition, pp Simoncelli EP (1993) Corse-to-fine estimtion of visul motion. In: Proc. 8th workshop on imge nd multidimensionl signl processing, Cnnes, Frnce, pp Sogo T, Ishiguro H, Trivedi M (2001) N-oculr stereo for rel-time humn trcking. In: Benosmn R, Kng RS (eds) Pnormic vision: sensors, theory, nd pplictions. Springer, Berlin Heidelerg New York, pp Stuffer C, Grimson WEL (1999) Adptive ckground mixture model for rel-time trcking. In: Proc. IEEE interntionl conference on computer vision nd pttern recognition, pp Trucco E, Verri A (1998) Computer vision nd pplictions: guide for students nd prctitioners. Prentice-Hll, Englewood Cliffs, NJ 41. Vssllo RF, Sntos-Victor J, Schneeeli HJ (2002) A generl pproch for egomotion estimtion with omnidirectionl imges. In: Proc. IEEE workshop on omnidirectionl vision, June 2002, pp Winters N, Gspr J, Lcey G, Sntos-Victor J (2002) Omnidirectionl vision for root nvigtion. In: Proc. IEEE workshop on omnidirectionl vision, pp Zhu Z, Kruppih D, Risemn E, Hnson A (2003) Omnidirectionl vision for root nvigtion. Root Automt Mg (in press)