Tracking Persons and Vehicles in Outdoor Image Sequences Using Temporal Spatio-Velocity Transform *

Transcription

1 Tracking Persons and Vehicles in Oudoor Image Sequences Using Temoral Saio-Veloci Transform * Koichi Sao and J. K. Aggarwal Dearmen of Elecrical and Comuer Engineering The Universi of Teas a Ausin Ausin, TX 7871, USA { koh, aggarwaljk }@mail.ueas.edu Absrac This aer resens a mehodolog for racking moving ersons and vehicles in oudoor image sequences. A erson or a vehicle in he image sequence is reaed as one or more coninuous objecs in he saio-emoral image cube. Slan clinders are used o model ersons and he vehicles. Our ssem firs deecs he moion iels using a moion subracion echnique and hen eracs iel velociies using a emoral saio-veloci (TSV) ransform. Eraced iels in he TSV image sequence are fied o slan clinders in he saioemoral cube. Our ssem erforms well for image sequences conaining noise such as ree movemen, shadow movemen and brighness change. I racks correcl, smaller and low-seed objecs such as humans. 1. Inroducion Tracking ersons and objecs is an imoran roblem ha has received significan aenion over he las several decades. Our grou has sudied various faces of racking as well as he reresenaion of moving objecs. In [1], Jain e al. eraced images of moving objecs in dnamic scenes using a differencing oeraion. In [], Yalamanchili e al. segmened moving objecs using a region-growing rocess based on he moion differencing oeraion. More recenl, in [3], Cai e al. racked ersons using mulile cameras and a combinaion of several echniques, including moion analsis of 3D geomer in mulile ersecives, moion deecion, segmenaion and Baesian aern recogniion. In [4], Ali e al. segmened and racked he human bod and recognized human aciviies using he skeleonized human bod shae. Oher grous who have worked on human racking include ha of Penland/Adelson a MIT [5-7], and Davis e al. a Universi of Marland [8,9] as well as several ohers. In [5], Oliver e al. segmened humans b subracing an eigenbackground, which was generaed using Princial Comonen Analsis (PCA) from several saic background images. The racked humans based on heir osiion and veloci, esimaed using a Kalman filer and he Probabili Densi Funcion (PDF) in each color comonen of he human blob. In [6], Niogi segmened humans using Hough ransform line deecion on a saio-emoral sliced image of background-subraced image sequences. The assumed ha he subjec crosses he field of view a a consan seed; ha is, he human rajecories aear as sraigh lines on he saioemoral sliced image. This mehod is useful for eracing a human rajecor; however, i does no al o image sequences ha conain ersons or vehicles moving randoml (soing and saring), in which case he human rajecories do no aear as sraigh lines. In [8,9], Hariaoglu e al. racked mulile eole using silhouee and eure of he human images. The racked humans b maching he segmened conour of he humans as well as b maching he head osiion maching. Then, he idenified each individual in he grou of eole using emoral eure emlaes. In [10], our revious ssem segmens and racks ersons and recognizes wo-erson ineracions in oudoor side-view image sequences. The ssem segmens he human image using a background subracion echnique followed b an objec eracion echnique and a emoral saio-veloci (TSV) ransform. Human racking uses several feaures such as a emoral eure emlae, blob size and so on. The TSV ransform is a echnique o erac velociies of moving iels from a sequence of binar images. The inu sequence is comosed of a one-dimensional binar image from which he TSV ransform generaes a wodimensional grascale image sequence ha consiss of a horizonal ais and a veloci ais. *This work was suored in ar b he Arm esearch Office under conracs DAAD , DAAG and DAAD (Johns Hokins Universi subconrac agreemen ). Proceedings nd IEEE In. Worksho on PETS, Kauai, Hawaii, USA, December 9 001

2 Our curren ssem uses he TSV ransform o erac wo-dimensional veloci vecors from wo-dimensional image sequences. Thus, he TSV image, which is he ouu of he TSV ransform, is a 4-dimensional image sequence ha consiss of horizonal and verical osiion aes and horizonal and verical veloci aes. The ssem racks objecs b fiing he TSV-els ino saioemoral slan clinder (STSC) models ha reresen moving objecs in he saio-emoral image cube. This aer is organized as follows: in secion, he TSV ransform echnique is described. In secion 3, feaures in he given image sequences are briefl menioned. In secion 4, mehodologies, including moion subracion and model fiing echniques, are resened. esuls and conclusion are described in secion 5 and secion 6, resecivel.. Temoral Saio-Veloci Transform The emoral saio-veloci ransform eracs iel velociies from sequences of binar images. B grouing similar velociies, he random moion of objecs such as ree movemen can be eliminaed. The TSV eracs he velociies along each ais. When he wo-dimensional binar image (based on he horizonal and verical aes) is used, TSV ransform eracs he wo-dimensional iel velociies along he horizonal and verical aes; herefore, he ouu TSV image sequence is a fourdimensional image sequence. In general, ersons and vehicles move randoml in a wo-dimensional lane. Thus, i is necessar o analze wo-dimensional osiion and wo-dimensional veloci. A wo-dimensional binar image sequences S * (,, is TSV ransformed ino fourdimensional TSV image sequences V n (,,v,v ) as follows. V n (,,v,v )=TSV{S * (,,} (1) where n is he frame number and TSV{} reresens he TSV ransform oeraor. The TSV ransform consiss of wo ses. Firs, i akes he windowing oeraion over he binar image sequence, and hen i akes he Hough Transform in erms of a line over he windowed image sequence. The windowed image sequence L n (,, is comued using windowing filer F n (, * Ln (,, S (,, F ( n () where S * (,, is he binar image in nh frame, n is he curren frame. An eonenial window is used o simlif comuaion, ( nn ) (1 e ) e n n (as or curre Fn ( (3) 0 n n (no defined for fuure) The windowed image sequence is Hough ransformed in erms of Line A(,,v,v ) and TSV image V n (,,v,v ) is comued: Line A: V (,, v, v ) Hough L n v ( n n ) (4) v LineA n Ln ( v ( n n ), v ( n n ), n (5) where n is he curren frame, (,) is he image coordinae, (v,v ) is he sloe of he line (ha is, he veloci), and (, ) is he osiion a n=n, (he osiion a curren frame). Since an eonenial window is used, (5) can be simlified ino: * V (,, v, v ) e V ( v, v, v, v ) (1 e ) S (,, n ) n (,, n 1 (6) This simle form seeds u comuaion. Figure 1 shows he ieraive oeraions ha generae he TSV image sequence. TSV image in he revious frame Vn-1(,,v,v) Shif oward he veloci ais Vn-1(-v,-v,v,v) e + TSV image Vn=e - Vn-1+ (1-e - )S * Binar image sequence S * (,, Srech so as o be he same size as TSV image. S * (,,v,v, 1 e Figure 1. Generaing a TSV image sequence Time Consan The TSV image eracs he iel velociies; however, if he iels have large acceleraion, TSV ma fail o erac he veloci. The ime consan in he TSV ransform deermines he acceleraion range of iels. When is large, i acces a smaller acceleraion range. When is small, i acces he larger acceleraion range. In he given daase, he acual acceleraion and veloci do no affec he image equall over he enire image. In oher words, wo eole moving a he same seed ma move a differen seeds in he image deending on he locaion along he verical ais. Therefore, we use differen ime consan values for differen verical locaions in he image. In his ssem, we se higher ime consan value where objec movemens are large and a lower ime consan value where objec movemens are small. Figure shows he ime consan values used for he Daase-Camera1. In his icure, he inensi of each image reresens he ime consan value. (High inensi means a high ime consan value). Obviousl, Proceedings nd IEEE In. Worksho on PETS, Kauai, Hawaii, USA, December 9 001

3 he objec movemen is larger when he objecs come closer o he camera, hus he image shows he lower inensiies in he closer ar. For eamle, here is a ree ha alwas aears in fron in he images in daase-camera1. The ree blocks our view of an moving objec in he daa. Thus we se a high ime consan value so ha he acceleraion range is small behind he ree. If he resence of he ree were no available, we would use a consan value in a horizonal locaion. Figure. Time Consan Field for Camera image 3. Descriion of he image sequences The images sequences used in his work have he following feaures: 1. The camera is fied and disan from he objecs.. The ground is almos geograhicall lane, wih few us and downs. 3. Some objecs ma occlude ohers, such as rees. 4. Vehicles move and so. 5. Persons walk randoml in he lane fields. 6. Brighness changes a imes. 7. Shadows aear and disaear. 8. Persons walk and ride on biccles. 4. Mehodolog The echniques for obaining he STSC model, are described below, along wih he rocessing order. Image Sequences Moion Subracion TSV Transform STSC Clusering Figure 3. Block diagram of his ssem Moion Subracion Since here are man facors ha ma conribue o changes in an oudoor image background, such as ree movemen or changing shadows and brighness from cloud movemen, a simle background subracion echnique ma generae significan noise. Thus, he moion subracion echnique is used in our ssem for segmening he moion of objecs. 1 I(,, I(,, n T ) Th S (,, (6) 0 oherwise where S(,, is a moion deeced image in nh frame, I(,, is an image in he nh frame, T is a ime consan and Th is a fied hreshold. This oeraion siml eracs he moving ars, he objec moion and he ree movemen. To reduce such noise as a ree movemen, our ssem alies an AND oeraion for a 13 block surrounding each iel. S * (,,=S(,-1, & S(,,& S(,+1, (7) where S * (,, is moion-eraced image of he noise reduced version and & reresens he binar AND oeraion. Temoral Saio-veloci Transform As deailed in secion, TSV ransforms generae a four-dimensional TSV image sequence. V n (,,v,v )=TSV{S * (,,} (1) Binarizing he TSV image To grou iels wih similar veloci, we binarized he emoral saio-veloci image b a secific hreshold T v. 1 if V (,,, ) * n v v T V (,,, ) V v v (8) n 0 oherwise where V * n (,,v,v ) is binar TSV image. The Slan Clinder Model The clinder model is used o reresen objecs in he saio-emoral image cube. Objecs in he image sequence form a slan clinder in he saio-emoral image cube. The clinder used in his aer is reresened as: a n v n an vn C(,, (9) where C(,, is he clinder model, he coordinae (a n +v n+,a n +v n+ ) reresens he cenroid of his objec, and (, ) reresens he horizonal and verical radius of he ellise fied o he objec. The objecive of his aer is o form he bes-fi clinders ha cover he scaered TSV-els. Hierarchical clusering is used for fiing in ever frame. One mehod for avoiding occlusion errors is o use long daa segmens in order o rack he objec before and afer he Proceedings nd IEEE In. Worksho on PETS, Kauai, Hawaii, USA, December 9 001

4 occlusion. B using slan clinder models and TSV-els and bes-fiing o he clinder model before and afer he occlusion, his ssem avoids occlusion errors. TSV-els Piels in TSV image Clusering o [w w w v w v] block Small clinder clusering Occlusion Merged ino final clinder model Figure 5. Clusring STSC model rocedures Figure 4. Clinder models and TSV-els Parameer Comuaion The arameers a,a,v,v,,,, are comued b a saisical aroach using he coordinaes of TSV-els belonging o he clinder. a, a v a a a a v a v v, v v v v a a a v, v (10) where k reresens he variance of he variable k, kl reresens he covariance of he variables k and l, and k reresens he mean value of he variable k. Also, he clinder densi r is defined as: N r h (11) where h reresens he heigh of he clinder, ha is, he ime range of he daa, and N reresens he number of TSV-els in he clinder. The denominaor is he clinder volume. Hierarchical Clusering A hierarchical clusering echnique is used o build he saio-emoral slan clinder model. Figure 5 shows an overview of he clusering ses. Firs, he iels in he binar TSV image are clusered ino [w w w v w v ] size blocks, which are equall disribued hroughou he enire TSV image field, where w,w,w v,w v are fied arameers. This seeds u he comuaion raher han comuing each iel in he laer sage. Second, each block is merged ino a small clinder model ha migh be a ar of he objec. Finall, each small clinder model is merged ino he final model. Figure 6 shows he deailed rocedure for bes fiing o he clinder models. Loo for all block cells Binar TSV-els Divide ino [w w wv wv] size blocks Comue cenroid and variance of each block es Loo for clinders Join o he clinder Transform o he virual coordinae Find he closes clinder o he block Comue d= c-b c: clinder cener, b:blob cenroid If dmin<thd? Comue he clinder feaures such as clinder radius (r), cener (c). Make a new clinder Comue he disance beween wo clinders d1= c1-c Picku smalles disance d1min no Densi es es Merge clinders no Leave he clinders un-merged Figure 6. Clinder model fiing rocedure Proceedings nd IEEE In. Worksho on PETS, Kauai, Hawaii, USA, December 9 001

5 Block Segmenaion The four-dimensional [w w w v w v ] TSV image is segmened ino a block, and hen he arameers of each block (such as cenroid coordinae and variance) are comued. K X K1 Y 1 K Y All he noaion is he same as equaion (13). (16) Coordinae Transformaion The acual osiion and image locaion are no linearl relaed, ha is, objec size a a far locaion is smaller han a a close locaion in he image. Thus, he segmened blocks were ransformed ino virual osiions ha have linear relaionshis wih he acual coordinaes. K1 X (13) Y 1 K * where (X,Y) is he virual osiion, K 1,K are consan values and (, * ) is he coordinae of he image. K 1,K is obained from he camera arameers such as camera heigh, focal disance and so on. In his case, he are obained from he raining image samle. Clinder Fiing Each block is clusered o a small clinder model under he following rules: 1. Comue he disances beween he block and all clinders.. If he minimum disance is less han a fied hreshold Th d, join he block o he clinder, oherwise, make a new clinder. The disance beween clinders o he block is comued: ( X B X C ) ( YB YC ) d (14) 1 where d is he disance, (X B,Y B ) is he block cenroid coordinae, (X C,Y C ) is a clinder s coordinae. Clinder Merging Clinders are merged under he following rule: 1. Comue he disances beween all clinders.. Focus on a clinder air whose disance is minimum. 3. Comue he densi of a combined clinder of he wo clinders. 4. If rue for he densi es, merge he wo clinders, oherwise, leave hem un-merged. The densi es is: r c Tr (15) ( r1 r) / where r c is he densi of a combined clinder, r 1 and r is he densi of he wo clinders. 5. esuls We used he daase (esing camera 1) image sequences wih he following secificaions. Table 1.Secificaions Size 3040 Color deh (8bi) G(8bi) B(8bi) Frame rae 9.97 [frame/sec] Daase (Tesing Camera 1) OS MS Windows 98 CPU Penium 4, 1GHz Inu Forma MPEG comressed AVI File Ouu Forma Uncomressed AVI File SDK MS Video For Windows SDK Processing Seed Table gives he rocessing ime for he daase. Table. Processing Time Wih Image Ouu 46 [sec] Wihou Image Ouu 431[sec] Wih image ouu means he rocessing ime when we use a rogram ha dislas 4 rocessing images and ouus 1 resul images. Wihou image ouu means he rocessing ime when we use he rogram ha ouus onl rajecor coordinaes, ha is, skis he disla rocedure. Thus he laer rocedure is slighl faser han he firs one. Tracking esul Table 3 shows he resul of he daase (Camera1,Tesing). Table 3. esul of Daase (Camera1,Tesing) Number Correc Incorrec Human Biccle 1 1 Vehicle 0 The images in his daase have a big ree ha occludes all oher objecs, making i difficul o idenif he objecs around he ree. Our ssem erforms well for smaller and low-seed objecs such as a human blob. However, our ssem was unable o idenif a vehicle ha soed near he ree. Transform o image coordinae To disla he rajecories of he racking objecs, he following ransformaion is used: Proceedings nd IEEE In. Worksho on PETS, Kauai, Hawaii, USA, December 9 001

6 Figure 7. One frame of he TSV image sequence Figure 8. One frame of he resul sequence The color line afer each erson reresens he rajecor of each erson. machine inelligence, vol. 1, No.1, , November [4] A. Ali and J. K. Aggarwal, Segmenaion and ecogniion of Coninuous Human Acivi, 001 IEEE Worksho on Evens in Video, Jul 001, Vancouver, Canada. [5] N. Oliver, B. osario and A. Penland, A Baesian comuer vision ssem for modeling human ineracions, Proceedings of Inernaional Conference on Vision Ssems '99, Gran Canaria, Sain,. 55-7, Januar [6] S. Niogi and E. Adelson, Analzing gai saioemoral surface, Proceedings of he 1994 IEEE Worksho,.64-69, [7] A. Penland and A. Liu. Towards augmened conrol ssems, IEEE Inelligen Vehicles '95, Deroi, MI, , Seember [8] I. Hariaoglu, D. Harwood and L. Davis, W4: Who, When, Where, Wha: A real ime ssem for deecing and racking eole, Third Inernaional Conference on Auomaic Face and Gesure, Nara, -7, Aril [9] I. Hariaoglu and L. Davis, Hdra: Mulile eole deecion and racking using silhouees, IEEE Worksho on Visual Surveillance,.6-13, [10] K. Sao and J. K. Aggarwal, Tracking and ecognizing Two-erson Ineracion in Oudoor Image Sequences, 001 IEEE Worksho on Muli-Objec Tracking,.87-94, Vancouver, CA, Jul, 001. [11]. Gonzalez and. Woods, Digial Image Processing, Addison Wesle, [1]. O. Duda, P. E. Har and D. G. Sork, Paern Classificaion, Second Ediion, A Wile-Inerscience Publicaion, Second Ediion, Conclusion In his aer, we have resened a ssem ha can rack humans and vehicles in oudoor image sequences. The ssem uses emoral saio-veloci ransform echnique on image sequences conaining noise such as ree movemen, shadow movemen and brighness change. This ssem erforms well for smaller and lowseed objecs such as humans. 7. eferences [1]. Jain, W. N. Marin and J. K. Aggarwal, Segmeaion hrough he Deecion of Changes Due o Moion, Comuer grahics and image Processing II, , [] S. Yalamanchili, W. N. Marin and J. K. Aggarwal, Eracion of Moving Objec Descriions via Differencing, Comuer grahics and image rocessing 18, , 198 [3] Q. Cai and J. K. Aggarwal, Tracking Human Moion in Srucured Environmens Using a Disribued-Camera Ssem, IEEE Transacions on aern analsis and Proceedings nd IEEE In. Worksho on PETS, Kauai, Hawaii, USA, December 9 001

7 8. Samle Images Figure 9. Original Image Figure 10. TSV image of Figure 7 All icures are slices of a TSV image sequence. The cener image is a TSV slice image a (v,v )=(0,0), he lef-o image is a TSV slice image a (v,v )=(-,-) [iel/frame]. All images are arranged in his wa. The smaller blob in each image is a human blob and he bigger blob is a vehicle blob. (This icure is comaible o he original image.) We can see ha he human blob has he highes inensi a he cener icure, and he vehicle blob has he highes inensi a he (v,v )=(-1,-) icure. Proceedings nd IEEE In. Worksho on PETS, Kauai, Hawaii, USA, December 9 001