Parametric ego-motion estimation for vehicle surround analysis using an omnidirectional camera

Transcription

1 Mchine Vision nd Applictions (25) 16: Digitl Oject Identifier (DOI) 1.17/s z Mchine Vision nd Applictions Prmetric ego-motion estimtion for vehicle surround nlysis using n omnidirectionl cmer Trk Gndhi, Mohn Trivedi Computer Vision nd Rootics Reserch Lortory, University of Cliforni t Sn Diego, L Joll, CA, USA (e-mil: {tgndhi,trivedi}@ucsd.edu) Received: 1 August 23 / Accepted: 8 July 24 Pulished online: 3 Ferury 25 c Springer-Verlg 25 Astrct. Omnidirectionl cmers tht give 36 pnormic view of the surroundings hve recently een used in mny pplictions such s rootics, nvigtion, nd surveillnce. This pper descries the ppliction of prmetric egomotion estimtion for vehicle detection to perform surround nlysis using n utomoile-mounted cmer. For this purpose, the prmetric plnr motion model is integrted with the trnsformtions to compenste distortion in omnidirectionl imges. The frmework is used to detect ojects with independent motion or height ove the rod. Cmer clirtion s well s the pproximte vehicle speed otined from CAN us re integrted with the motion informtion from sptil nd temporl grdients using Byesin pproch. The pproch is tested for vrious configurtions of n utomoilemounted omni cmer s well s rectiliner cmer. Successful detection nd trcking of moving vehicles nd genertion of surround mp re demonstrted for ppliction to intelligent driver support. Keywords: Motion estimtion Pnormic vision Intelligent vehicles Driver support systems Collision voidnce 1 Introduction nd motivtion Omnidirectionl cmers tht give pnormic view of surroundings hve ecome very populr in mchine vision. Benosmn nd Kng [5] give comprehensive description of pnormic imging systems nd their pplictions. There is considerle interest in motion nlysis from moving pltforms using omni cmers, since pnormic views help in deling with miguities ssocited with ego-motion of the pltforms [15]. In prticulr, vehicle surround nlysis system tht monitors the presence of other vehicles in ll directions is importnt for online s well s offline pplictions. Online systems re useful for intelligent driver support. On the other hnd, offline processing of video sequences is useful for study of ehviorl ptterns of the driver in order to develop etter tools for driver ssistnce. For such systems, complete surround nlysis system tht monitors the lnes nd vehicles round the driver Fig. 1,. Imges from omni cmer mounted on n utomoile. This cmer hs verticl FOV of 5 ove the horizon nd covers only nery surroundings ut gives lrger vehicle imges. This cmer hs verticl resolution of 15 ove the horizon nd covers frther surroundings, ut with smller vehicle imges is very importnt. An omni cmer mounted on the utomoile could provide complete pnormic view of the surroundings nd would e very pproprite for performing such tsk. The min contriution of this pper is to perform moving oject detection from omni imge sequences using direct prmetric motion estimtion method nd pply it to video sequences otined from n utomoile-mounted cmer to detect nd trck neighoring vehicles. Figure 1 shows the imges from omni cmers in different configurtions used for this work. It is seen tht the cmer covers 36 field of view (FOV) round its center. However, the imge it produces is distorted, with stright lines trnsformed into curves. Directly unwrping the imge to perspective imge would introduce severe lur in perspective imge, cusing prolems for susequent steps in motion nlysis. Insted, the omni cmer trnsformtions re comined with the motion trnsformtions to compenste the ego-motion in the omni domin itself. 1.1 Relted work in motion nlysis Motion estimtion from moving omni cmers hs recently een topic of gret interest. Rectiliner cmers usully hve smller FOV, due to which the focus of expnsion often lies outside the imge, cusing motion estimtion to e sensitive to the cmer orienttion. Also, the motion field produced y

2 86 T. Gndhi, M. Trivedi: Prmetric ego-motion compenstion trnsltion in the horizontl direction is similr to tht due to rottion out the verticl xis. As noted y Gluckmn nd Nyr [15], omni cmers void oth these prolems due to their wide FOV. They project the imge motion on sphericl surfce using Jcoins of trnsformtions to determine egomotion of moving pltform in terms of trnsltion nd rottion of the cmer. Vsslo et l. [3] propose generl Jcoin function tht cn descrie wide vriety of omni cmers. Shkerni et l. [26] use the concept of ck-projection flow, where the imge motion is projected to virtul curved surfce in plce of sphericl surfce to simplify the Jcoins. Using this concept, they hve dpted ego-motion lgorithms for rectiliner cmers for use with omni sensors. Svood et l. [28] use feture correspondences to estimte the essentil mtrix etween two frmes using the 8-point lgorithm. They lso note tht the motion estimtion is more stle with omni cmers compred to rectiliner cmers. Most of these methods first compute motion of imge pixels nd then use the motion vectors to estimte the motion prmeters. However, due to the perture prolem [17], the full motion informtion is relile only ner cornerlike points. The edge points only hve motion informtion norml to the edge. Direct methods cn optimlly use the motion informtion from edges s well s corners to get prmeters of motion. Direct methods hve often een used with rectiliner cmers for plnr motion estimtion, ostcle detection, nd motion segmenttion [7,21,2]. To distinguish ojects of interest from extrneous fetures, the ground is usully pproximted y plnr surfce whose ego-motion is modeled using projective trnsform [25,23] or its linerized version [3]. Using this model, the ego-motion of the ground is compensted in order to seprte the ojects with independent motion or height. 1.2 Relted work on intelligent vehicles In recent yers, considerle reserch hs een conducted on developing intelligent vehicles hving driver support systems tht enhnce sfety. Computer vision techniques hve een pplied to detecting lnes, other vehicles, nd pedestrins to wrn the driver of dngers such s lne deprture nd possile collision with other ojects. Stereo cmers re especilly useful for detecting ostcles in front tht re fr from the driver. Bertozzi nd Broggi [6] use stereo cmers for lne nd ostcle detection. They model the rod s plnr surfce nd use inverse perspective trnsform to register the rod plne etween two imges. The ostcles ove the rod would hve residul disprity nd re esily detected. In the cse of curved rods, [24] crete V-disprity imge sed on clustering similr disprities on ech imge row. A line or curve in this imge corresponds to stright or curved rod, respectively, nd the vehicles on the rod form other distinctive ptterns. Omni cmers with their pnormic FOV show gret potentil in intelligent vehicle pplictions. In [18], n omni cmer mounted inside the cr otined view of the driver s well s of the surroundings. The driver s pose ws estimted using hidden Mrkov models nd ws used to generte the driver s view of the surroundings using the sme cmer. In [2], feture-sed methods detecting specific chrcteristics of vehicles, such s wheels, were used to detect nd trck vehicles. Single-cmer motion nlysis hs een used for seprting ego-motion of the ckground to detect vehicles nd other ostcles on the rod. Roust rel-time motion compenstion for the rod plne for this purpose is descried in [23]. In [1], system for video-sed driver ssistnce involving lne nd ostcle detection using rectiliner cmer is descried. Direct prmetric motion estimtion discussed in the previous section is especilly useful for vehicle pplictions, since most of the fetures on the rod re line-sed nd very few corner fetures re ville. The direct estimtion pproch ws generlized for motion compenstion using omni cmers in [13,18], where prmeters of plnr homogrphy were estimted. A modifiction of tht pproch is used here s in [14] to estimte the vehicle ego-motion in terms of liner nd ngulr velocities. These re used to compenste the ego-motion for the rod plne nd detect vehicles hving residul motion to generte complete surround view showing the position nd trcks of the vehicles. 2 Ego-motion estimtion nd compenstion system The system lock digrm is shown in Fig. 2. The inputs to the system re sequence of imges from n omni cmer mounted on utomoile, the vehicle speed from the CAN us which gives informtion out the vehicle stte, nd the nominl clirtion of the cmer with respect to the rod plne. The stte of the vehicle contining vehicle velocity nd clirtion re used to compute the wrping prmeters to compenste the imge motion etween two frmes for points on the rod plne. The wrping trnsform is composition of the omni cmer trnsform nd the plnr motion model. It trnsforms the omni imge coordintes to perspective coordintes, pplies the plnr motion prmeters to compenste the rod motion, nd converts them ck to the omni view. Two consecutive frmes from the imge sequence re tken, nd the wrping prmeters re used to trnsform one imge to nother, to compenste the motion of the rod s much s possile. The ojects with independent motion nd height would hve lrge residul motion, mking it possile to seprte them from rod fetures. Fig. 2. System for ego-motion compenstion from moving pltform. The inputs to the system re the video sequence from n omni cmer nd vehicle speed informtion extrcted from the CAN us of the cr tht provides numer of vriles of the cr s dynmics. The output is surround mp with detected vehicles nd their trcks

3 T. Gndhi, M. Trivedi: Prmetric ego-motion compenstion 87 However, the fetures on the rod my lso hve some residul motion due to errors in the vehicle speed nd clirtion prmeters. To correct for these errors, sptil nd temporl grdients of the motion-compensted imges re otined. Byesin estimtion similr to [23] is pplied with grdients s oservtions to updte the prior knowledge of the stte of the vehicle using Klmn filter mesurement updte equtions. To minimize the effect of outliers, only the grdients stisfying constrint on the residul re used in the estimtion process. The updted vehicle stte is used to recompute the wrping prmeters, nd the residul grdients re recomputed. The process is repeted in corse-to-fine itertive mnner. The grdients computed using the finlly updted stte of the vehicle re used to seprte the vehicle fetures from the rod fetures. The vehicle fetures re comined using constrints on vehicle length nd seprtion to otin los corresponding to vehicles tht re trcked over the numer of frmes. The surround mp is generted y unwrping the omni imge to give pln view nd superimposing the vehicle los nd trcks over the resulting imge. The following sections descrie the processing steps in detil. 3 Motion trnsformtions for omni cmer Let c denote nominl cmer coordinte system, sed on the known cmer clirtion, with the Z-xis long the cmer xis nd the X Y plne eing the imging plne. Due to cmer virtions nd drift, the ctul cmer system t ny given time is ssumed to hve smll rottion with respect to this system due to virtions nd drift. Use of the nominl system llows us to tret smll rottions s ngulr displcement vectors. The ego-motion of the cmer is then descried using stte vector x contining the cmer liner velocity V, ngulr velocity W, nd ngulr displcement A etween nominl cmer system c nd ctul system, ll expressed in nominl cmer system c. 3.1 Plnr motion model To detect ostcles in the pth of moving cmer, the rod is modeled s plnr surfce. Let P nd P denote the perspective projections of point on the rod plne in coordinte systems corresponding to two positions nd of the moving cmer. These re relted y: λ P = λ RP + D = λ [RP + D/λ ], (1) where R nd D denote the rottion nd trnsltion etween the cmer positions, nd λ,λ depend on the distnce of the ctul 3D point. Let the eqution of the rod plne t the cmer position e: K T (λ P )=1, (2) where K is vector norml to the rod plne in the coordinte system of cmer position. Sustituting the vlue of λ from Eq. 2 in Eq. (1), it is seen tht P nd P re relted y projective trnsform [11]: λ P = λ [ R + DK T ] P = λ HP, (3) where H = R+DK T is known s the projective trnsform or homogrphy. This reltion hs een widely used to estimte plnr motion for rectiliner cmers. If the ngulr displcements with respect to the nominl cmer clirtion re smll, the mtrices cn e expressed s: R I W t, D [I W t A ] V t, K [I A ] K, (4) where W nd A represent the skew symmetric mtrices constructed from vectors W nd A, nd K represents the plne norml in the nominl cmer coordinte system. 3.2 Omni cmer trnsform To pply the ego-motion estimtion method to omni cmers, one needs the mpping from the cmer coordinte system to the pixel domin nd vice vers. Given this trnsformtion nd the plnr motion model, one cn generte trnsformtion tht compenstes the motion of the plnr surfce in the omni pixel domin. In prticulr, the omni cmer used in this work consists of hyperolic mirror nd cmer plced on its xis, with the center of projection of the cmer on one of the focl points of the hyperol. It elongs to clss of cmers known s centrl pnormic ctdioptric cmers [5]. These cmers hve single viewpoint tht permits the imge to e suitly trnsformed to otin perspective views. The geometry of hyperolic omni cmer is shown in Fig. 3. According to the mirror geometry, light ry from the oject towrd the viewpoint t the first focus O is reflected so tht it psses through the second focus, where conventionl rectiliner cmer is plced. The eqution of the hyperoloid is given y: (Z c) 2 2 X2 + Y 2 2 =1, (5) where c = Let P =(X, Y, Z) T denote the homogenous coordintes of the perspective trnsform of ny 3D point λp on ry OP, where λ is the scle fctor depending on the distnce of the 3D point from the origin. It cn e shown [1,19,26] tht the reflection in the mirror gives the point p =( x, y) T on the imge plne of the cmer using: ( ) x q 1 p = = y q 2 Z + q 3 P ( ) X, (6) Y where q 1 = c 2 2,q 2 = c 2 + 2,q 3 =2c, (7) nd P = X 2 + Y 2 + Z 2. Note tht the expression for imge coordintes p is independent of the scle fctor λ. The pixel coordintes w = (u, v) T re then otined y using the clirtion mtrix K of the conventionl cmer composed of the focl lengths f u,f v, opticl center coordintes (u,v ) T, nd cmer skew s, or: u x f u s u x v = K y = f v v y. (8)

4 88 T. Gndhi, M. Trivedi: Prmetric ego-motion compenstion deteriortes if the entire imge is trnsformed t one time. Hence, s noted y Dniilidis [8], it is desirle to perform motion estimtion directly in the omni domin, ut use the ove trnsformtions to mp the loctions to the perspective domin s required. Since the internl prmeters of the omni cmer re to e mesured only once, specilized setup ws used to otin the clirtion. The omni cmer ws set on tripod nd leveled to hve verticl cmer xis. A numer of fetures with known coordintes were tken on the ground nd verticl pole to cover the FOV of the omni cmer. The FOV covered y the omni cmer mps into the ellipse s seen in Fig. 3. The cmer center nd spect rtio were computed from the ellipse prmeters. Using these prmeters, the imge coordintes (u, v) cn e normlized to give (u,v ) corresponding to origin s center nd unit spect rtio. Assuming rdil symmetry round the imge center, we hve: d = u 2 + v 2 = X2 + Y 2 c 1 Z + c 2 P, (11) d [y pixels] c z/r Fig.3.Geometry of hyperolic omni cmer. The rys towrd the first focus of the mirror re reflected towrd the second focus nd imged y norml cmer. FOV of omni cmer with numer of points with known coordintes. c Curve fitting for internl prmeter estimtion where c 1 = q 2 /(q 1 f v ) nd c 2 = q 3 /(q 1 f v ). Using the known world nd imge coordintes of these points, the liner equtions in c 1 nd c 2 re formed nd solved using lest squres: dzc 1 + d P c 2 = X 2 + Y 2. (12) Figure 3 shows the plot of d ginst Z/ P of the smple points nd the curve fitted using estimted prmeters. It is seen tht the curve models the omni mpping quite fithfully. Nonliner lest squres cn then e used for improving the ccurcy. Though the method is designed for centrl pnormic cmers, if the distnce to the oserved scene is lrge compred to the mirror size, the method cn lso e pplied to noncentrl pnormic cmers provided the mpping from oject ry directions to pixel coordintes is known. In fct, it ws oserved tht for hyperolic mirror, the FOV is concentrted within close distnce round the cmer, which mde it somewht difficult to detect ojects frther from the cmer where resolution ws poor. Noncentrl cmers my e prticulrly useful since they give more flexiility in djusting the cmer resolution in different prts of the imge, s descried in [16]. This trnsform cn e used to wrp n omni imge to pln perspective view. To convert perspective view ck to omni view within scle fctor, the inverse trnsformtion cn e used: x u y 1 = K 1 v 1, (9) X q 1 x Y q 1 y. (1) Z q 2 q 3 x2 + y 2 +1 It should e noted tht the trnsformtion of omni to perspective view involves very different mgnifictions in different prts of the imge. For this reson, the qulity of the imge 4 Ego-motion estimtion To estimte the ego-motion prmeters, the prmetric imge motion is sustituted into the opticl flow constrint [17]: g u u + g v v + g t =, (13) where g u,g v re sptil grdients nd g t is the temporl grdient. Since the imge motion ( u, v) t ech point i cn e represented s function of the incrementl stte vector x, the opticl flow constrint Eq. 13 for imge points 1... N cn e expressed s: z = c( x)+v C x + v, (14)

5 T. Gndhi, M. Trivedi: Prmetric ego-motion compenstion 89 Tle 1. Chin of functions nd Jcoins leding from stte vector x to opticl flow constrint c. Rows 4 nd 5 correspond to the omni cmer trnsform tht converts the cmer coordintes to pixel coordintes V x = W H = R + DK T H = R + D.K T + D( K) T A R = W t h 1 h 2 h 3 R I W t D =(I W t A ) t V ( W t + A )V t H = h 4 h 5 h 6 D [I A ] V t K = A K h 7 h 8 h 9 K [I A ] K V/ V i = e i, W / W i = A / A i =(e i) ( ) T X h 1 h 2 h 3 X X Y Z P h = h 1... h 9 Y h 4 h 5 h 6 Y = h X Y Z Z h 7 h 8 h 9 Z X Y Z ( ) ( ) ( ) ( ) ( ) T x x q P = XY Z = = 1 X p q y y 2 Z+q 3 P P Y = 1 q 3xX q 1 P q 3xY q 3xZ (q 2 Z+q 3 P ) P q 3yX q 3yY q 1 P q 3yZ ( ) T u f u s u x ( ) w p = xy v = f v v y = f u s p f v ( T ( ) w = uv) ( ) u u ( ) c c = g u g v = g t + η w v v = g u g v where (g u u + g v v) 1 c( x) =. (g u u + g v v) N, (g t ) 1 z =. (g t ) N, (15) v is the vector of mesurement noise in the time grdients, nd C = c/ x is the Jcoin mtrix computed using the chin rule s in [13]. The function c(x) is nonliner. Row i of the N 9 Jcoin mtrix is given y the chin rule: C i = ( ) ( ) ci c w p P h =, (16) x w p P h x i where P = (X,Y,Z ) T, p = (x,y ) T nd w = (u,v ) T re the coordintes of the point in the cmer, imge, nd pixel coordinte systems for cmer position, nd h is the vector of elements of H. The individul Jcoins re computed similrly to [13]. The reltionship etween these vriles, nd their Jcoins, re shown in Tle 1. Since the points hving very low texture do not contriute much to the estimtion of motion prmeters, only those imge points hving grdient mgnitude ove threshold vlue re selected for performing estimtion. Alterntively, nonmximl suppression is performed on the imge grdients nd the imge points with locl mxim re used. This wy, insted of computing Jcoins using multiple imge trnsforms over the entire imge, the Jcoins re computed only t the selected points tht hve significnt informtion for estimting prmeters. The estimtes of the stte x nd its covrince P re itertively updted using the mesurement updte equtions of the iterted extended Klmn filter [4]: P [ C T R 1 C + P 1 ] 1 (17) ˆx ˆx + ˆx = ˆx + P [ C T R 1 z P 1 (ˆx x ) ]. (18) However, the opticl flow constrint eqution is stisfied only for smll imge displcements up to 1 or 2 pixels. To estimte lrger motions, corse-to-fine pyrmidl frmework [22,27] is used. In this frmework, multiresolution Gussin pyrmid is constructed for djcent imges in the sequence. The motion prmeters re first computed t the corsest level, nd the imge points t the next finer level re wrped using the computed motion prmeters. The residul motion is computed t the finer level, nd the process is repeted until the finest level. Note tht since the resolution of the mirror is not constnt, formtion of Gussin pyrmid could hve errors in the neighorhood. However, since the pyrmid is used itertively in corse-to-fine mnner, the errors t lower resolution re expected to e corrected t higher resolution. The prmeters cn lso e updted from frme to frme using time updte equtions of the Klmn filter: ˆx Bˆx, P BPB T + Q, (19) where B nd Q re determined from system dynmics. 4.1 Outlier removl The ove estimte is optiml only when ll points relly elong to the plnr surfce nd when the underlying noise distriutions re Gussin. However, the estimtion is highly sensitive to the presence of outliers, i.e., points not stisfying the rod motion model. These fetures should e seprted using roust method. For this purpose, first the region of

6 9 T. Gndhi, M. Trivedi: Prmetric ego-motion compenstion interest of rod is determined using clirtion informtion, nd the processing is done only in tht region to void extrneous fetures. To detect outliers, n pproch similr to the dt snooping pproch discussed in [9] hs een dpted for Byesin estimtion. In this pproch, the error residul of ech feture is compred with the expected residul covrince t every itertion, nd the fetures re reclssified s inliers or outliers. If point z i is not included in the estimtion of ˆx i.e., is currently clssified s n outlier then the covrince of its residul is: V [ z i C i ˆx] =V [ z i ]+CV [ˆx] C T = R + C i PC T i. (2) However, if z i is included in the estimtion of ˆx i.e., is currently clssified s n inlier then it cn e shown tht the covrince of its residul is given y: V [ z i C i ˆx] =R C i PC T i < R. (21) Hence, to clssify in the next itertion, the residul is compred with its covrince ccording to whether it is currently n outlier or n inlier. If the Mhlnois norm is greter thn given threshold, the point is clssified s outlier, otherwise s n inlier. Alterntively, roust-m estimtion [12] could e used to reduce the effect of outliers y itertively reweighting the contriution of smples ccording to their error residuls. 4.2 Algorithm for motion prmeter estimtion The lgorithm for itertive estimtion of motion prmeters is descried elow: Form Gussin pyrmid from imges A nd B from consecutive frmes. Set the initil prmeters nd the covrince mtrix to their priors s: ˆx = x nd P = P. Proceeding from the corsest to the finest level, perform multiple itertions of the following steps: 1. Wrp imge B using current estimte ˆx of motion prmeters to form imge W (B; ˆx). 2. Otin sptil nd temporl grdients etween imge A nd the wrped imge W (B; ˆx). 3. Use opticl flow constrint with prmetric motion model on inlier points to pply incrementl correction in motion prmeters nd their covrinces ccording to Eqs. 17 nd Compre the residuls of ll points with their expected covrinces in Eqs. 2 nd 21 to reclssify them s inliers nd outliers. 5 Vehicle detection nd trcking After motion compenstion, the fetures on the rod plne would e ligned etween the two frmes, wheres those due to ostcles would e misligned. Imge difference etween the frmes would therefore enhnce the ostcles nd suppress the rod fetures. To reduce the dependence on locl texture, the normlized frme difference [29] is used. This is given t ech pixel y: g t g 2 u + gv 2 k + gu 2 + gv 2, (22) where g u,g v re sptil grdients, g t is the temporl grdient fter motion compenstion, nd denotes Gussin weighted verging performed over K K neighorhood of ech pixel. In fct, the normlized difference is smoothed version of the norml opticl flow nd hence depends on the mount of motion ner the point. Due to the untextured interior of vehicle, los re usully detected t the sides of the vehicle. To get the full vehicle, it is ssumed tht if two los re within threshold distnce (5. m) in the direction of the cr s motion, they constitute vehicle. To detect this sitution, the originl imge is unwrped using the flt plne trnsform, nd morphologicl closing is performed on the trnsformed imge using 1 N verticl msk. After the los corresponding to moving ojects re identified, nery los re clustered nd trcked over frmes using Klmn filtering [4]. The points on the lo tht re nerest to the cmer center usully correspond to the rod plne nd re mrked s n ostcle mp. The vehicle position on the rod is computed y projecting the trck loction on the ostcle mp. Since the ostcle mp is ssumed to e on the rod plne, the loction of the vehicle cn e otined y inverse perspective trnsform. 6 Experimentl studies The ego-motion compenstion pproch ws pplied for detecting vehicles from n omni cmer mounted on n utomoile tested used for intelligent vehicle reserch. The tested is instrumented with numer of cmers nd computers to cpture synchronized video of the surroundings. In ddition, the CAN us of the vehicle gives informtion on vehicle speed, pedl nd rke positions, rdr, etc. The vehicle ws driven on freewy s well s on city rods. The mximum vehicle speed for the test ws 65 miles per hour (29 m/s). The ctul vehicle speed, otined from the CAN us, ws used for initil motion estimte. The first test run ws conducted with n omni cmer hving verticl FOV of only 5 ove the horizon. For this reson, only the vehicles ner the cr were oserved, ut the resolution ws s lrge s possile. To get s little of the cr s possile, the cmer ws rised 18 in. (45 cm) ove the cr using specilly designed fixture. Figure 4 shows n imge from the omni cmer on the cr eing driven on the freewy. The estimted prmetric motion is shown using red rrows. Note tht the motion is estimted only in the designted region of interest, which excludes the cr ody. Figure 4 shows the clssifiction of points into inliers (gry), outliers (white), nd unused (lck) points. The estimtion is done using only the inlier points. An imge with the normlized frme difference etween the motion-compensted frmes is shown in Fig. 4c, which enhnces the regions corresponding to independently moving vehicles. Figure 4d shows the detection nd trcking of vehicles mrked with trck ID nd the coordintes in the rod plne. The omni imge ws trnsformed to otin

7 T. Gndhi, M. Trivedi: Prmetric ego-motion compenstion 91 c d e Fig.4. Imge from sequence using n omni cmer mounted on moving cr with estimted prmetric motion of rod plne. Clssifiction of points into inliers (gry), outliers (white), nd unused (lck). c Normlized difference etween motion-compensted imges. d Detection nd trcking of moving vehicles mrked with trck ID nd the coordintes in rod plne. e Surround view generted y trnsforming the omni imge longitudinl position: Z [m] longitudinl position: Z [m] time [s] time [s] Fig. 5. Plot of the longitudinl position of vehicle trcks on two sides of the cr ginst time. The trcks re color coded s red, yellow, nd green in order of incresing lterl distnce from the cmer Fig. 6. Surround nlysis in different situtions with top-mounted cmer. City rod. Freewy

8 92 T. Gndhi, M. Trivedi: Prmetric ego-motion compenstion c d e Fig.7.Imge from sequence using n omni cmer with wider FOV mounted on moving cr. The rnge of the cmer is incresed ut the resolution is decresed. Clssifiction of points into inliers (gry), outliers (white), nd unused (lck). c Normlized difference etween motion-compensted imges. d Detection nd trcking of moving vehicles mrked with trck ID nd the coordintes in rod plne. e Surround view generted y dewrping omni imge Fig. 8. Smples showing surround vehicle detection with wider FOV omni cmer 1 1 longitudinl position: Z [m] longitudinl position: Z [m] time [s] time [s] Fig. 9. Plot of longitudinl position of vehicle trcks ginst time on two sides of cr ginst time. The trcks re color coded s red, yellow, nd green in order of incresing lterl distnce from the cmer

9 T. Gndhi, M. Trivedi: Prmetric ego-motion compenstion 93 c d e longitudinl position: Z [m] time [s] Fig. 1. Imge from sequence using side cmer mounted on moving cr with estimted prmetric motion of the rod plne. Clssifiction of points into inliers (gry), outliers (white), nd unused (lck). c Normlized difference etween motion-compensted imges. d Detection nd trcking of moving vehicles mrked with trck ID nd the coordintes in the rod plne. e Surround view generted y pplying inverse perspective trnsform. f Plot of longitudinl position of vehicle trcks ginst time. The trcks re color coded s red, yellow, nd green in order of incresing lterl distnce from the cmer f the pln view of the cr surround, s shown in Fig. 4e. The longitudinl position of the cr with reference to cmer ws recorded for ech trck. Figure 5 shows the plots of trck positions ginst time seprtely for vehicles on two sides of the cmer. The test run lso contined sections driven on city rods tht hd lne mrks nd other fetures tht were more prominent compred to the freewy. Figure 6 shows exmples of moving vehicle detection on city rods s well s in freewy conditions. The second test run ws conducted using n omni cmer with FOV 15 ove the horizon. It ws noted tht the cmer cn see vehicles t greter distnce thn the previous cmer. The trdeoff ws lower resolution due to which the vehicles hd smller imge size, mking them slightly more difficult to detect. Figure 7 shows the result of surround vehicle detection t lrger longitudinl distnce from the cmer. Figure 8 shows more exmples of vehicle detection. Figure 9 shows the plots of trck positions ginst time seprtely for vehicles on two sides of the cmer. It should e noted tht the simplified version of the surround nlysis lgorithm developed in this pper cn lso e used with commonly ville rectiliner cmers. We conducted severl experiments where video strems were cquired using rectiliner cmer mounted on cr window to get rer side view on the driver s side. Figure 1 shows the results of the detection lgorithm. Figure 1e shows the top view generted y pplying the inverse perspective trnsformtion using the known clirtion. Insted of the full surround view, which cn e cquired using n omni cmer, only prtil view on one side of the vehicle ws otined. 7 Summry This pper descried n pproch to oject detection using ego-motion compenstion from utomoile-mounted omni cmers using direct prmetric motion estimtion. The rod ws modeled s plnr surfce, nd the equtions for plnr motion trnsform were comined with the omni cmer trnsform. An opticl flow constrint ws used to optimlly comine the prior knowledge of ego-motion prmeters with the informtion in the imge grdients. Corse-to-fine motion estimtion ws used, nd the motion etween the frmes ws compensted t ech itertion. Experimentl results demonstrted vehicle detection in two different configurtions of omni cmers tht otin ner nd fr views of the surround.

10 94 T. Gndhi, M. Trivedi: Prmetric ego-motion compenstion Acknowledgements. This reserch ws supported y UC Discovery Progrm digitl medi grnt in collortion with the Nissn Reserch Center. We lso thnk our collegues from the CVRR Lortory for their contriutions nd support. We lso thnk Dr. Erwin Boer for his suggestions on visulizing the results. Finlly we thnk the reviewers for their insightful comments which helped us to improve the qulity of the pper. Author, plese note. The list of References in your dt is not exctly the sme s tht in your printout. Plese check crefully. Thnk you. References 1. Achler O, Trivedi MM (22) Rel-time trffic flow nlysis using omnidirectionl video network nd fltplne trnsformtion. In: Workshop on intelligent trnsporttion systems. Chicgo 2. Achler O, Trivedi MM (24) Vehicle wheel detector using 2d filter nks. In: Proc. IEEE symposium on intelligent vehicles, pp Adiv G (1985) Determining three-dimensionl motion nd structure from opticl flow generted y severl moving ojects. IEEE Trns Pttern Anl Mch Intell 7(4): Br-Shlom Y, Li XR, Kirurjn T (21) Estimtion with pplictions to trcking nd nvigtion. Wiley, New York 5. Benosmn R, Kng SB (21) Pnormic vision: sensors, theory, nd pplictions. Springer, Berlin Heidelerg New York 6. Bertozzi M, Broggi A (1998) Gold: prllel rel-time stereo vision system for generic ostcle nd lne detection. IEEE Trns Imge Proc 7(1): Blck MJ, Anndn P (1996) The roust estimtion of multiple motions: Prmetric nd piecewise-smooth flow fields. Comput Vis Imge Understnd 63(1): Dniilidis K, Mkdi A, Bulow T (22) Imge processing in ctdioptric plnes: sptiotemporl derivtives nd opticl flow computtion. In: IEEE workshop on omnidirectionl vision, pp Dnuser G, Stricker M (1998) Prmetric model fitting: from inlier chrcteriztion to outlier detection. IEEE Trns Pttern Anl Mch Intell 2(2): Enkelmnn W (21) Video-sed driver ssistnce: from sic functions to pplictions. Int J Comput Vis 45(3): Fugers O (1993) Three-dimensionl computer vision: geometric viewpoint. MIT Press, Cmridge, MA 12. Forsyth D, Ponce J (23) Computer vision: modern pproch. Prentice-Hll, Upper Sddle River, NJ 13. Gndhi T, Trivedi MM (23) Motion nlysis of omnidirectionl video strems for moile sentry. In: 1st ACM interntionl workshop on video surveillnce, pp 49 58, Berkeley, CA 14. Gndhi T, Trivedi MM (24) Motion sed vehicle surround nlysis using omni-directionl cmer. In: Proc. IEEE symposium intelligent vehicles, pp Gluckmn J, Nyr S (1998) Ego-motion nd omnidirectionl cmers. In: Proc. interntionl conference on computer vision, pp Hicks RA, Bjcsy R (1999) Reflective surfces s computtionl sensors. In: Proc. 2nd workshop on perception for moile gents, pp Horn B, Schunck B (1981) Determining opticl flow. In: DARPA81, pp Hung, K, Trivedi MM, Gndhi T (23) Driver s view nd vehicle surround estimtion using omnidirectionl video strem. In: IEEE symposium intelligent vehicles, Columus, OH, pp Hung KC, Trivedi MM (23) Video rrys for rel-time trcking of persons, hed nd fce in n intelligent room. Mch Vis Appl 14(2): Irni M, Anndn P (1998) A unified pproch to moving oject detection in 2D nd 3D scenes. IEEE Trns Pttern Anl Mch Intell 2(6): Irni M, Rousso B, Peleg S (1994) Computing occluding nd trnsprent motions. Int J Comput Vis 12: Jähne B, Hußecker H, Geißler P (1999) Hndook of Computer Vision nd Applictions, vol 2, chp 14, pp Acdemic Press, Sn Diego, CA 23. Kruger W (1999) Roust rel time ground plne motion compenstion from moving vehicle. Mch Vis Appl 11: Lyrde R, Auert D, Trel J-P (22) Rel time ostcle detection in stereovision on non flt rod geometry through v- disprity representtion. In: IEEE symposium intelligent vehicles, 2: Lourkis MIA, Orphnoudkis SC (1998) Visul detection of ostcles ssuming loclly plnr ground. In: Asin conference on computer vision, 2: Shkerni O, Vidl R, Sstry S (23) Omnidirectionl egomotion estimtion from ck-projection flow. In: IEEE workshop on omnidirectionl vision 27. Simoncelli EP (1993) Corse-to-fine estimtion of visul motion. In: Proc. 8th workshop on imge nd multidimensionl signl processing, Cnnes, Frnce, pp Svood T, Pjdl T, Hlváč V (1998) Motion estimtion using centrl pnormic cmers. In: IEEE interntionl conference on intelligent vehicles, pp Trucco E, Verri A (1998) Computer vision nd pplictions: guide for students nd prctitioners. Prentice Hll, Upper Sddle River, NJ 3. Vssllo RF, Sntos-Victor J, Schneeeli HJ (22) A generl pproch for egomotion estimtion with omnidirectionl imges. In: IEEE workshop on omnidirectionl vision, pp Trk Gndhi received his chelor of technology degree in computer science nd engineering t the Indin Institute of Technology, Bomy. He erned his M.S. nd Ph.D. from the Pennsylvni Stte University in computer science nd engineering, specilizing in computer vision. He worked t Adept Technology, Inc. on designing lgorithms for rootic systems. Currently, he is postdoctorl scholr t the Computer Vision nd Rootics Reserch lortory t the University of Cliforni t Sn Diego. His interests include computer vision, motion nlysis, imge processing, rootics, trget detection, nd pttern recognition. He is working on projects involving intelligent driver ssistnce, motion-sed event detection, trffic flow nlysis, nd structurl helth monitoring of ridges.

11 T. Gndhi, M. Trivedi: Prmetric ego-motion compenstion 95 Mohn Mnuhi Trivedi is professor of electricl nd computer engineering t the University of Cliforni t Sn Diego. Trivedi hs rod rnge of reserch interests in the intelligent systems, computer vision, intelligent ( smrt ) environments, intelligent vehicles nd trnsporttion systems, nd humn-mchine interfce res. He estlished the Computer Vision nd Rootics Reserch Lortory t UCSD. Currently, Trivedi nd his tem re pursuing systems-oriented reserch in distriuted video rrys nd ctive vision, omnidirectionl vision, humn ody modeling nd movement nlysis, fce nd ffect nlysis, nd intelligent vehicles nd interctive pulic spces. He serves on the executive committee of the Cliforni Institute for Telecommuniction nd Informtion Technologies [Cl-(IT) 2 ] s the leder of the Intelligent Trnsporttion nd Telemtics Lyer t UCSD. He lso serves s chrter memer of the executive committee of the University of Cliforni systemwide Digitl Medi Innovtion Progrm (DiMI). He serves regulrly s consultnt to industry nd government gencies in the USA nd rod. Trivedi ws editor-in-chief of the Mchine Vision nd Applictions Journl during He hs served on the editoril ords of journls nd progrm committees of severl mjor conferences. He served s chirmn of the Rootics Technicl Committee of the IEEE Computer Society. He ws elected to serve on the dministrtive committee (BoG) of the IEEE Systems, Mn nd Cyernetics Society. Trivedi hs received the Distinguished Alumnus wrd from the Uth Stte University nd Pioneer (Technicl Activities) nd Meritorious Service wrds from the IEEE Computer Society. He is fellow of the Interntionl Society for Opticl Engineering (SPIE).