Vision-Based Traffic Measurement System

Transcription

1 *J. M. Wang, **S. L. Chang, **Y. C. Chung, and **S. W. Chen Deparmen of Informaion and Compuer Educaion *Naional Taiwan Universiy **Naional Taiwan Normal Universiy Taipei, Taiwan Absrac The recen ongoing evoluion of compuer echnology has enabled a variey of pracical applicaions of video sequences. In his paper, a non-inrusive vision-based raffic measuremen sysem for auomaically collecing raffic daa from he video sequences of a roadway is described. The proposed sysem, composed of a video camcorder and a hos compuer wih embedded communicaion devices, is easily moved, insalled and operaed. The algorihm of he sysem consiss of an off-line sep (preprocessing) and hree on-line seps (vehicle deecion, vehicle racking, and raffic parameer esimaion). In he preprocessing sep, hree asks are carried ou: background image generaion, road feaure deecion, and camcorder calibraion. The generaed background image is used in he vehicle deecion sep for rapidly exracing vehicles from video images by way of background subracion. Road feaures, including lane markings and heir vanishing poin, are exraced from he background image and are uilized in he vehicle racking sep. Lane markings provide informaion for guiding vehicle racking and he vanishing poin serves as a reference poin for compuing he spaial aribues of vehicles. The relaionship beween he image plane and a prescribed real world coordinae sysem is esablished during he camcorder calibraion. The esablished relaionship is employed in he raffic parameer esimaion sep for ransforming 2D raffic daa colleced from video sequences ino 3D daa. A number of experimens wih video sequences acquired by wo camcorders: an ordinary camcorder available in our laboraory and a public camcorder deployed by our Traffic Adminisraion Bureau, were conduced o demonsrae he effeciveness of he proposed sysem. Keywords: Vision-based raffic measuremen sysem, Progressive background image generaion, Filer-based lane marking deecion, Camcorder calibraion, Illuminaion assessmen, Shadow removal, Muli-person decision making

2 1. Inroducion Video processing consumes a grea deal of compuaional power due o massive amoun of daa involved. The recen ongoing evoluion of compuer echnology significanly increases he compuaional power of compuers and, as a consequence, enables a variey of pracical applicaions of video images. In his paper, we presen a vision-based raffic measuremen sysem (VTMS) for auomaically collecing raffic daa from he video sequences of a roadway. Assored raffic parameers [25], such as flow rae, densiy, hroughpu, occupancy, and headway, are readily calculaed from he colleced raffic daa. Esimaed raffic parameers are of use for a number of raffic managemen purposes (e.g., adapive raffic signal conrol, congesion pricing, mainline meering, raveler informaion services, highway mainenance and consrucion, and raffic policy planning) as well as academic researches. Diverse sensors (e.g., inducive loops, capaciance mas, bending plaes, ulrasonic sensors, infrared deecors, and microwave radars) have been incorporaed in raffic measuremen sysems for collecing daa from raffic scenes. The choice of a sensor depends on boh he raffic scene (e.g., main sree, highway, freeway, corridor, or inersecion) under consideraion and he raffic parameers o be esimaed. In his sudy, a video camcorder is used as he sensor for our VTMS. Such a sensor possesses a number of advanages, including (a) large field of view i is suiable for wide-area monioring, (b) rich informaion collecion assored raffic parameers are readily derived from he informaion provided by he sensor, (c) flexibiliy i is feasible for various objecives of deecion, racking, classificaion, and recogniion, (d) reliabiliy i funcions under differen weaher and lighing condiions, (e) porabiliy i is convenien o ransfer, insall and operae, and (f) affordabiliy i s cos is low. A large number of VTMS [2, 7, 9, 10, 17, 19, 22, 24, 43, 46] have been developed for collecing raffic daa from video images of raffic scenes. The algorihms of VTMS may vary wih he raffic scene under consideraion, raffic condiion (e.g., inerruped/uninerruped raffic and downsream/upsream raffic flow), he configuraion of visual sensors (e.g., single/muliple cameras, saionary/mobile cameras, and heir locaions, heighs, and viewing direcions), and image qualiy (e.g., resoluion, accuracy, and repeaabiliy). Regardless of hese facors, here are ypically hree major seps involved in he algorihm of a VTMS: vehicle deecion, vehicle racking, and raffic parameer esimaion. Four echniques have commonly been employed for deecing vehicles from video images, including background subracion, emporal differencing, emplae maching, and moion-based segmenaion. The background subracion mehod [3, 15, 27, 31] exracs foreground objecs from an image by subracing he background from he image. The emporal differencing mehod [10, 13, 30, 36, 42] locaes moving objecs by searching for image regions having significan changes in inensiy beween wo or more successive images

3 The emplae maching mehod [23, 24, 27, 45] idenifies vehicles in an image by fiing deformable or pre-buil vehicle models o image regions. The moion-based segmenaion mehod [11, 33, 37] locaes moving vehicles by grouping image pixels wih akin moions. Since raffic congesion is a frequen occurence, we need a mehod capable of deecing moving, slowly moving, and even non-moving vehicles. The background subracion mehod and he emplae maching mehod saisfy his requiremen. However, he emplae maching mehod ypically akes ime due o he ieraive processing of deformable models or he repeaed processing of muliple vehicle models. In his sudy, we use he background subracion mehod o exrac vehicles from video sequences. Vehicle deecion is ypically imperfec. Deeced vehicles may be broken, disored, or conneced o each oher. Vehicle racking should be able o olerae hese imperfecions. Furhermore, he numbers of vehicles may vary from image o image due o missing/exra deecion, occlusion/disocclusion, and moving in/ou he monioring zone. Many echniques [5, 7, 8, 12, 28, 29, 33, 41] have been proposed for racking vehicles over video sequences. Basically, hey achieved he purpose by deermining correspondence for vehicles beween successive images. Kalman filers [24, 34, 38], acive conours [27], relaxaion labeling [18, 21], neural neworks [39], and flexible consrain propagaion and saisfacion [14] have been exploied o solve he correspondence problem. However, hese mehods are ieraive in naure; wihou addiional sraegies or mechanisms hey may no be adequae for real-ime purposes. Non-ieraive approaches, such as non-ieraive greedy algorihm [40], mean shif opimizaion [8], hidden Markov model [5], and graph maching [1, 4, 6, 32, 35, 44] become preferable. In his sudy, a non-ieraive fuzzy-se heoreic approach, called he muli-person decision making (MPDM) approach [26], is employed for esablishing correspondence for vehicles beween successive images. The res of his paper is organized as follows. In Secion 2, we describe he insallaion, operaion, and process of he proposed VTMS. Criical componens involved in he process of our VTMS are deailed in he subsequen secions, including he hree preprocessing asks of background image generaion, road feaure deecion, and camcorder calibraion in Secion 3, illuminaion assessmen and shadow removal for vehicle deecion in Secion 4, and he MPDM echnique for vehicle racking in Secion 5. Experimenal resuls are presened in Secion 6, followed by concluding remarks and fuure work in Secion In his secion, he insallaion, operaion, and process of he proposed VTMS are addressed. 2.1 Sysem Insallaion and Operaion The proposed VTMS is composed of a video camcorder and a hos compuer wih embedded communicaion devices. The camcorder ha can be mouned on an overpass, a all

4 building or a high pole coninuously acquires video sequences from a roadway. The acquired video sequences are immediaely ransmied o he hos compuer insalled in a raffic informaion cener, where he video sequences are processed and inerpreed. Figure 1 shows an example of sysem insallaion, in which he camcorder was mouned on a high pole locaed a he roadside of a highway (Figure 1(a)). An example video image acquired by he camcorder is displayed in Figure 1(b). Figures 1(c) and (d) depic he configuraion of he camcorder, in which he monioring zone indicaed by he shaded area, il angleθ, roll angle γ, pan angleλ, and heigh h of he camcorder are specified. These parameers and he focal lengh f of he camcorder can be known during insalling he camcorder. They are needed for he camcorder calibraion process. (a) (b) (c) Fig. 1 An example of sysem insallaion. (d) 2.2 Sysem Process A flowchar for illusraing he process of our VTMS is depiced in Figure 2. There are four major seps consiuing he process, including preprocessing, vehicle deecion, vehicle racking, and raffic parameer esimaion

5 Fig. 2 Flowchar for illusraing he process of he proposed VMTS. A. Preprocessing In he preprocessing sep, hree off-line asks are carried ou: background image generaion, road feaure deecion, and camcorder calibraion. A progressive approach o be addressed in Secion 3.1 is developed for generaing he background image of a scene from is video sequences. The generaed background image is o be used in he vehicle deecion sep for rapidly exracing vehicles from video images by way of background subracion. Road feaures, including lane markings and heir vanishing poin, are exraced from he background image. Boh will be used in he vehicle racking sep. Lane markings delimiing road lanes provide informaion for recifying divered rajecories of vehicles. The vanishing poin acs as a reference poin, for which a number of spaial aribues of vehicles are compued. Vehicle aribues provide criical informaion for racking vehicles over a video sequence. Secion 3.2 discusses road feaure deecion

6 Finally, he camcorder is calibraed, in which he relaionship beween he image plane of he camcorder and he real world coordinae sysem is esablished. This relaionship will be uilized in he raffic parameer esimaion sep o ransform he 2D raffic daa exraced from video sequences ino 3D daa. The camcorder calibraion ask is described in Secion 3.3. B. Vehicle Deecion Having compleed he preprocessing sep, he sysem is ready o coninually operae. Considering an inpu video image, foreground objecs are firs exraced from he image by means of background subracion. In general, he oucome of his sep is ineviably defecive. The exraced foreground objecs may be non-vehicle objecs, noise, broken, conneced or disored vehicles. We apply a series of operaions o he foreground objecs in order o eliminae defecs. The operaions include 1) non-vehicle removal by deleing he foreground objecs locaed ouside he prescribed monioring zone, 2) noise removal using conneced componen labeling followed by size filering, 3) morphological manipulaions for boh grouping broken vehicles and separaing conneced vehicles, and 4) hole filling for compleing vehicle shapes. However, he above operaions will no make up for he defecs of disored vehicles due o accompanying shadows as well as conneced vehicles resuling from shadows or occlusions. Clearly, o compensae for hese wo defecs, he issues of shadow and occlusion have o be resolved. In Secion 4.1, we presen an illuminaion assessmen process for evaluaing he brighness of a scene. This process is only invoked a designaed ime insans. Once a high degree of brighness of he scene is repored by he process, a shadow deecion and removal process o be addressed in Secion 4.2 is evoked. This process repeas for he subsequen video images unil a sop signal is received from he illuminaion assessmen process. To deal wih he issue of occlusion, we rely on wo sources of informaion: aribues of vehicle and emporal evidence. The former informaion can be aained from he curren image, while he laer informaion requires messages from he previous image. Since emporal informaion plays an imporan role in he nex sep of vehicle racking, in his sep we only use he aribues of color and inensiy o idenify and separae conneced vehicles resuling from occlusions and leave he deecion of occlusions based on emporal evidence o he nex sep. C. Vehicle Tracking Regardless of imperfecions associaed wih vehicle deecion, we rea vehicles as individual vehicles here, which are o be raced over he video sequence. To his end, we look for correspondence for vehicles beween successive images. Since vehicles deeced in he respecive images may be differen in boh number and ype. In his sudy, a modified muli-person decision making (MPDM) echnique [26] o be discussed in Secion 5 is employed for solving his correspondence problem. Since he modified MPDM allows

7 many-o-one, one-o-many and many-o-many mappings, he issues of missing/exra vehicle deecion, occlusion/disocclusion of vehicles, and vehicles moving in/ou he monioring zone can easily be resolved. D. Traffic Parameer Esimaion In he vehicle racking sep, we acually obain he number of vehicles and heir speeds passing hrough he monioring zone for each designaed ime inervals. In he raffic parameer esimaion sep, he vehicle speeds are ransformed ino he 3D space using he ransformaion parameers esimaed in he camcorder calibraion process. Furhermore, since we know he rajecories of vehicles, he number of vehicles passing any observaion line, a virual line crossing and perpendicular o he roadway being moniored, can be deermined. Based on he above informaion, we hen calculae he raffic parameers of flow rae, capaciy, densiy, hroughpu, headway, and occupancy for he roadway. The definiions and deails of calculaion of hese parameers can be referred o [25]. 3. Preprocessing In his secion, he hree off-line asks, background image generaion, road feaure deecion, and camcorder calibraion, involved in he preprocessing sep are addressed Progressive Background Image Generaion By he background image of a scene, we mean he saionary porion of he scene image. The proposed echnique can generae he background image of a scene wihou pre-evacuaing he scene. The background image is progressively generaed; is qualiy increases as he number of inpu images increases. A measure is associaed wih he background image generaed a any insan, which indicaes he degree of goodness of he image. The generaed background image is laer on coninually updaed in order o be consisen wih he acual background a any momen. The proposed algorihm generaes and updaes he background image in one process. To begin, for each inpu image I, we compue I(, x y) I(, x y) I 1(, x y) < ε P ( x, y) =, (1) 0 oherwise where ( x, y) represens any pixel locaion, ε is a small posiive ineger, and are he curren and he previous images, respecively. Equaion (1) preserves in array saionary porion of I and zeros ou he res of he area of P. We se he iniial background image B 0 o P 1 and hereupon updae he background image according o B I I 1 x, y) = arg max{ h( ( l) 0 l L 1}, (2) ( x,y) P he where L is he number of disinc gray levels and h ( x,y) is a 1 by L hisogram array associaed wih pixel (x,y). Iniially, 0 xy h(, ) () l = 0 for all l [1, L]. Thereafer, if P( x, y) 0, he

8 enries of h ( x,y) are updaed as follows. h 1 1 h if (, ) and (, )() (, )( ) xy l + A Pxy l ε hx y l+ A K K if Pxy (, ) l ε and h ( l) + A> K, (3) h () l D if Pxy (, ) l > and h ( l) D 0 0 oherwise 1 ( x, y) (, xy) ( l) = 1 1 (, xy) ε ( x, y) where A, D and K are posiive inegers. The above equaion increases h 1 ( x, y) l ( ) by A if pixel I I 1 (x,y) has he same inensiy value l in boh images and ; oherwise decreases by D. Adding A increases he opporuniy of gray level l o serve as a background value, while subracing D decreases he opporuniy. In our experimens, we always choose A > D (e.g., A = 3 and D = 1) in order o expedie background image generaion and in he mean ime resis variaions in he background image. The parameer K is inroduced here o preven he enries of from ceaselessly growing. We have chosen K = 15. h ( x,y) Turning back Equaion (2), i says ha a ime he background value B ( xy, ) a pixel * * (x,y) is deermined as he gray level, say l, for which h(, xy) ( l ) h(, xy) ( l), l. The value of h * (, xy ) ( l ) in a sense indicaes how good he gray level * l is when serving as he background value B ( xya, ) (x,y). We average h ( (, ))/ (, ) B x y K xy over all he pixels of he background image, i.e., 1 GB ( ) = h ( B( xy, ))/ K, where N is he number of pixels, ( x, y) N ( xy, ) and use GB ( ) o indicae he goodness of he background image, B, generaed a ime Road Feaure Deecion Road feaure deecion is performed on he background image. Two kinds of road feaures are exraced, including lane markings and heir vanishing poin. A. Lane Markings Deecion Many previous lane marking deecors based on edge deecion exraced wo boundary lines for each lane marking. Our deecor exracs one line wih a one-pixel widh for each lane marking. The proposed deecor is characerized by a seerable quadraure pair of filers [16], G2 and H2, where G2 is a second x-derivaive of a circularly symmeric Gaussian and H2 is he Hilber ransform of 2. The roaed versions, denoed by G G θ 2 and H θ 2, of G2 and H2 can be approximaed by linear combinaions of a finie se of basis

9 filers. We can pre-compue he responses of basis filers o he background image. Laer on, θ he responses, [G 2 ] θ and [H 2 ], of filers and H a any angle θ can easily be G2 2 compued hrough linear sums of he responses of basis filers. θ Based on he known [G 2 ] θ and [H 2 ], for each pixel p we calculae is orienaion energy by θ θ 2 θ 2 E2( p) = G 2( p) + H2( p). Le θ ( p ) be he direcion wih he maximal orienaion energy. We hen compue for pixel p is phase angle φ ( p) along he direcion θ d ( p ) by 1 2 φ( p) = an ([ H θ d 2 ( p)] / [ G θ d 2 ( p)] 2 ). Based on he calculaed φ ( p) value, we classify he conour ype of pixel p as follows. 0 p is a dark-line pixel If φ ( p) is close o π p is a ligh-line pixel. ± π / 2 p is an edge pixel Since lane markings are brigher han road surfaces, we preserve p only if i is classified as a ligh-line pixel. Furhermore, since he calculaed φ ( p) may no exacly equal π, we compue a measure Λ ( φ( p)) for φ ( p) defined as 2 cos ( φ( p)- π) π /2 φ( p) 3 π/2 Λ( φ( p)) =. 0 oherwise Only a ligh-line pixel wih a high enough Λ( φ( p)) is acually preserved. We repea he above process for all pixels and complee he ask of lane marking deecion. B. Vanishing Poin Deecion Based on he deeced ligh lines, we nex deermine he vanishing poin of lane markings. Since he exraced ligh lines may no be sraigh due o road curvaure or imperfec deecion, we fi each line segmen l wih a sraigh line L, which has he leas summed squares of disances, d 2 min d, from he poins on l o L. Le L be he line perpendicular o L and passing hrough he cenral poin of l. We calculae he summed squares of disances, d 2, from he 2 d min poins on l o L. We hen define he elongaion e of l as d 2 /. The smaller he elongaion, he shorer or he more curvilinear is l. We ignore he ligh lines wih small elongaion values (<10 in pracice). Le S be he se conaining he surviving ligh lines. For each pair of lines in S, we compue heir inersecion as well as he produc of heir elongaion values. The firs n (5 in our experimens) inersecions wih he larges elongaion producs are chosen. Finally, from he chosen inersecions he closes wo are seleced and heir average poin is

10 deermined as he vanishing poin of lane markings Camcorder Calibraion The inen of camcorder calibraion is o esablish he relaionship beween he image plane of he camcorder and a seleced real world coordinae sysem. Referring o Figure 3(a), he image plane is specified by he X-Y frame and he real world coordinae sysem is represened by he U-V-W frame. The origin o of he X-Y frame is on he opical axis of he camcorder and is locaed a disance f from he viewing cener C of he camcorder. The Y-axis perpendicular o he opical axis poins downward wih respec o he camcorder and he X-axis ogeher wih he opical axis and he Y-axis form a righ-handed coordinae sysem. The origin O of he U-V-W frame is locaed a he inersecion of he opical axis of he camcorder and he road surface. Is W-axis perpendicular o he road surface poins upward. The U-axis angen o he road surface direcs oward he camcorder and he V-axis ogeher wih he U- and W-axes form a righ-handed coordinae sysem. (a) (b) Fig. 3 Relaionship beween he image plane of he camcorder and he real world coordinae sysem

11 During he selecion of he real world coordinae sysem, our major concern is o reduce as much as possible he involvemen of personnel during camcorder calibraion. Referring o Figure 3(b), our calibraion process needs only o know he il angleθ, heigh h, and focal lengh f of he camcorder. Parameer f is known from he manufacurer of he camcorder and parameers h and θ can be measured locally. To describe he relaionship beween he X-Y and he U-V-W frames, we derive he ransformaion from any image poin o is corresponding spaial poin on he road surface. Referring o Figure 3(b), le p ( x, y ) be any image poin and P( uv,0), be is corresponding spaial poin on he road surface. The derivaion of he ransformaion consiss of hree seps. Firs, deermine he U-V-W coordinaes of he origin o of he X-Y frame. Second, deermine he U-V-W coordinaes of he image poin p. Finally, compue he U- and V-coordinaes of he spaial poin P. To deermine he U-V-W coordinaes of o, we firs projec o and C ono he U-axis. Le o and C be he projecion poins of o and C, respecively. The disance beween o and C is f cosθ and he disance beween O and C is h coθ. The U-coordinae of o is hence hcoθ f cosθ. Nex, consider he V-coordinae of o. Since o lies on he plane formed by he opical axis and he U-axis, his plane passes hrough O and is perpendicular o he V-axis. The V-coordinae of o is 0. Le o be he projecion of o ono he verical line connecing C and C. The disance from C o o is f sinθ. The W-coordinae of o is simply h f sinθ. Summarizing he above discussions, he U-V-W coordinaes of o is ( hcoθ f cos θ, 0, h f sin θ). Nex, deermine he U-V-W coordinaes of he image poin p. We know p is shifed from o by x in he X-direcion and y in he Y-direcion. In he following, we firs conver he shif ( x, y ) in he X-Y frame ino he shif ( u, v, w ) in he U-V-W frame and hen add he resul o he U-V-W coordinaes of o. Since he X-axis is parallel o he V-axis, a shif in he X-direcion will give rise o he same shif in he V-direcion, i.e., v = x. The shif y in he Y-direcion produces wo shifs in he U- and W-direcions, respecively. In he U-direcion, he shif is y sinθ, i.e., u = ysinθ. In he W-direcion, he shif is y cosθ, i.e., v = ycosθ. In summary, ( u, v, w ) = ( y sinθ, x, y cosθ ). Adding his ( u, v, w ) o he U-V-W coordinaes of o, we obain he U-V-W coordinaes of p in he following ( ysin θ, x, ycos θ)+( hcoθ f cos θ, 0, h f sin θ) = ( hcoθ f cosθ + ysin θ, x, h f sinθ ycos θ ). Finally, compue he U- and V-componens of he spaial poin P on he road surface. Since poins C, p and P lie on he same line, he parameric form of his line in erms of he coordinaes of C and p is

12 where w = 0 u = hco θ + (1 )( hcoθ f cosθ + ysin θ) v= (1 ) x, (4) w= h+ (1 )( h f sinθ ycos θ) R. Since poin P lies on he road surface, is W-componen is zero. Subsiuing ino he hird line of Equaion (4), we obain h + (1 )( h f sinθ y cos θ ) = 0. Solving his equaion for, we aain = 1 Equaion (4) he U- and V-componens of P are h. Finally, subsiuing ino f sinθ + ycosθ h 2 co θ + hx( hcoθ f cosθ + ysin θ) u = hcoθ f sinθ + ycosθ. (5) hx v = f sinθ + ycosθ In he above equaion, he parameers h, f, and θ are all known during camcorder insallaion. Given an image poin ( x, y ), we can calculae is corresponding spaial poin ( uv,,0) on he road surface using he above equaion. 4. Illuminaion Assessmen and Shadow Removal Shadows may cause wo issues: disored vehicles and conneced vehicles. Boh can confuse our VTMS. However, shadows do no always appear in a scene. In his secion, we presen an illuminaion assessmen mehod for evaluaing he brighness of a scene and an approach o deecing and removing shadows from scene images Illuminaion Assessmen Raher han applying he illuminaion assessmen process o he enire image, we apply he process o a seleced foreground figure exraced in he background subracion sep, By his, we reduce he processing ime and more imporanly preclude inerferences from he image areas of no ineres. Two parameers, n and ρ, are inroduced for choosing a foreground figure; n is he number of pixels of he figure and ρ is defined as ρ = nd/ nb, where nd and are he numbers of dark and brigh pixels in he figure, respecively. A dark pixel is n b defined as he pixel wih RGB values ha are all smaller han hose of he corresponding pixel in he background image. Similarly, a brigh pixel is he pixel wih RGB values all larger han hose of he corresponding background pixel. The foreground figure wih a large n (i.e., large size) and ρ 1 (i.e., dark area brigh areas) is seleced for illuminaion assessmen. Having chosen he foreground figure, we firs compue is brighness energy 1 Eb = ei, Sb i S b 1 where S b denoes he se of brigh pixels and e = i Ii - I, in which N j i is he Ni j Ni

13 neighbor se of pixel i and I i is is inensiy value. Since objecs are ypically brigher han shadows, in a sense reflecs he level of visibiliy of he objecs in he figure. A shadow E b deecion and removal process will be evoked if a large enough we furher esimae he direcion of illuminaion. E b is aained. In his case, To esimae he direcion of illuminaion, we define eigh direcions, N, NE, E, SE, S, SW, W, and NW, wih respec o he cener of graviy of he foreground figure (see Figure 4(a)). Nex, we define eigh overlapping halves wih heir bisecing lines aligned wih he eigh direcions, respecively. Then, wihin each half we coun dark pixels along he boundary of he n figure. Le,,,,,,, and denoe he numbers of dark N n NE boundary pixels in he eigh halves, respecively. The direcion of illuminaion, r, is simply deermined as r =arg max{ n }, where S = { n i S E i n SE n S n SW n W n NW N,NE,E,SE,S,SW,W,NW }. Figure 4(b) shows an example, in which he deermined direcion of illuminaion is indicaed by an arrow. (a) (b) Fig. 4 Deermining he direcion of illuminaion Shadow Deecion and Removal Referring o Figure 5, shadows can be broadly caegorized ino cas and self shadows. The self shadow is a par of he objec, which is no illuminaed by ligh sources. The cas shadow lying beside he objec belongs o he scene. In our applicaion, cas shadows are o be eliminaed, while self shadows should be preserved because hey are pars of objecs. However, boh self and cas shadows have similar characerisics, and disinguishing beween hem is always a challenge. Furhermore, if he surface of a vehicle has proximae properies o shadow, shadow deecion and removal could become exremely difficul. Fig. 5 Types of shadows

14 To illusrae he proposed shadow deecion and removal mehod, le us urn o he example shown in Figure 5(b). In his figure, he arrow indicaes he direcion of illuminaion. Along his direcion, we sample a number of foreground pixels nearby he border of he figure (indicaed by a small circle). Based on he se of sampling pixels, we compue shadow aribues, a and a, H I n 1 1 2R G B ah = Hi, in which H = cos ( ), n 2 i= 1 2 ( R G) + ( R B)( G B) n 1 ai = Ii, in which I = ( R+ G+ B) / 3,, n i = 1 where R, G, B are he color riples of a sampling pixel. Shadows are no removed simply based on he compued shadow aribues for he following reasons. Firs, he disribuion of inensiy wihin a shadow is no uniform. Second, self and cas shadows have proximae aribues. Third, surfaces of vehicles may possess aribues similar o shadows. Insead of removing shadows, we incremenally reconsruc vehicle shapes. See he example shown in Figure 6. The foreground figure displayed in Figure 6(a) conains wo vehicles, which are conneced o each oher hrough a shadow. Figure 6(b) depics he background region corresponding o he foreground figure. We apply Canny s edge deecor o boh he foreground figure and he background area. The resuls are shown in Figures 6(c) and (d), respecively. As can be seen in he figures, here are many edges along wih he vehicles, while quie a few wih he background. We exrac vehicle edges by subracing he background edges from he foreground ones. Figure 6(e) shows he exraced vehicle edges. (a) (b) (c) (d) (e) (f) (g) (h) (i) Fig. 6 Example illusraing shadow deecion and removal. Three rules are hen proposed for reconsrucing vehicle shapes from heir edges. The firs rule saes ha brigh foreground pixels are preserved. Figure 6(f) shows he resul of applying his rule o he foreground figure in Figure 6(a). However, his rule will miss he

15 vehicle pixels ha are darker han shadow pixels. We hence have a second rule, which saes ha foreground pixels wih aribues ha are considerably differen from shadow aribues are preserved. Seeing Figure 6(g), he windows of vehicles and several areas darker han shadows are regained due o he second rule. A his poin, self shadows are sill no recovered because hey have aribues similar o cas shadows. The hird rule hen says ha he foreground pixels adjacen o vehicle edges are preserved. Figure 6(h) shows he resul afer applying his rule. Finally, a hole-fill algorihm is evoked for filling holes in vehicles. Figure 6(i) shows he final resul. 5. Muli-Person Decision Making During vehicle racking, vehicles are raced over he video sequence. We accomplished his by looking for correspondence for vehicles beween successive images. The muli-person decision making (MPDM) echnique [26] is modified for his purpose. In his secion, we briefly inroduce he MPDM echnique and hen focus on how is modified version manages exra/missing vehicle deecion, occlusion/disocclusion of vehicles, and vehicles ha ener/exi he monioring zone. The MPDM mehod decides on he correspondences beween vehicles based on heir aribues. The aribues considered in his sudy include size, color, inensiy, locaion, velociy, he disance from vehicle o he vanishing poin of lane markings, and he angle beween he horizonal line and he line connecing vehicle and he vanishing poin. Wihou loss of generaliy, suppose ha here are m disinc aribues feasible for describing vehicles. Accordingly, m expers are involved in he MPDM process. In oher words, he number of expers is equal o he number of aribues. Le S 1 = { v 1, v 2,, v n } and S 1 = { v1, v2,, v n } be he ses of vehicles deeced in he monioring zone a imes -1 and, respecively. The numbers n 1 and n may be differen. Considering a vehicle v i in S 1, each exper hen ranks he vehicles in S by comparing hem wih v i based on only a single aribue. Since here are m expers (equivalenly, m disinc aribues), m ranking liss for he vehicles in can be obained. The MPDM inegraes he m ranking liss following m 1 P () r = 1 w r r, (6) ij k jk mn k = 1 where r specifies a rank in [1, n ], r is he rank of vehicle v in he kh ranking lis made by jk exper k, and wk is a weigh specifying he degree of imporance of exper k. Since expers correspond o aribues, he imporance of an exper can be viewed as he imporance of he corresponding aribue. The grade of imporance of an aribue can be evaluaed based on boh is reliabiliy of measuremen and is abiliy o discriminae among vehicles. Turning o S j

16 Equaion (6), P ( r) sands for he possibiliy ha vehicle v ranked r among all he ij vehicles in S when maching hem wih vehicle v i in S 1. Based on he above MPDM framework, i.e., Equaion (6), we inroduce sraegies o resolve he issues of missing/exra vehicle deecion, occlusions/disocclusions of vehicles, and vehicles moving in/ou he monioring zone. Reurning o Equaion (6), if we are only concerned wih he bes mach, i.e., r = 1, hen he bes mached vehicle in, say v, wih vehicle v is simply deermined by v* = arg max{ P (1)}. However, in order o deal wih he i vj S ij j S * issues menioned above, we selec a hreshold. If he possibiliy of he bes mach, (1) *, is smaller han, vehicle v i can no find a mach in S. This means ha vehicle v i may move ou he monioring zone or be occluded a ime, or ha vehicle v i is an exra P i deecion a ime -1 or a missing deecion of v occurs a ime. Nex, we accep P ( r) for i ij any r so long as Pij ( r) >. This allows vehicle v i o have several maches in S. Disocclusions and vehicles moving in he monioring zone can hence be handled. The above process only considers he maching condiion of a vehicle in S 1. To complee he correspondence for vehicles beween S 1 and S, he above process should be repeaedly applied o all he vehicles in S. 6. Experimenal Resuls 1 In our experimens, experimenal video sequences were supplied by wo camcorders: one was an ordinary camcorder available in our laboraory and he oher was a public camcorder deployed by our Traffic Adminisraion Bureau. Boh provided 30 images per second and each image had he size of 320 by 240. Our curren sysem running on a Penium-4 1.5GHz PC is capable of processing 3-5 images per second. Since a vehicle should ake several minues o pass hrough he enire monioring zone, our sysem would capure images of he vehicle for processing. One of he major difficulies encounered during our experimens was he collecion of ground ruhs for esifying our experimenal resuls. Originally, we looked for roadway segmens ha had been equipped wih some sors of raffic measuremen insrumens. Unforunaely, he raffic daa provided by he insrumens were eiher no comparable or incomplee. Hence, we aemped o collec daa from roadways by hand. While couning vehicles by hand was feasible, manually measuring he speeds of many vehicles simulaneously was formidable. Finally, we decided o collec ground ruhs from off-line video sequences ha had been processed by our sysem. As menioned above, he camcorders acquired 30 images per second, or equivalenly 1800 images per minue. Manually examining long video sequences was iresome and ineviably involved imprecision. In his secion,

17 comparisons beween auomaically and manually colleced raffic daa from shor video sequences (beween 1 2 minues) are presened. More video sequences and he associaed experimenal resuls are available on our websie a hp:// Laboraory Camcorder Our camcorder was insalled on an overpass across a freeway. Video sequences were hen acquired under differen viewing direcions, weaher, illuminaion, and raffic condiions. Figure 7 shows four images each from a differen video sequence. (a) (b) (c) (d) Fig. 7 Each image comes from a differen video sequence acquired using our laboraory camcorder The firs video sequence (Figure 7(a)) was exraced from a long sequence aken during he dayime wih a normal raffic flow. The background image generaed from he firs few images of he video sequence is displayed in Figure 8(a) and he lane markings exraced from he background image are depiced in Figure 8(b). The background image was hen used o exrac vehicles from he subsequen images of he video sequence by way of background subracion. The lane markings provided informaion for guiding vehicles during racking. Figure 9 shows he inermediae resuls of vehicle racking. There are wo vehicles marked (1) and (2) in he hird image. Our sysem repored he average speeds of 95.5 km/hr for vehicle one and 85.1 km/hr for vehicle wo. Visually examining he video sequence in Figure 9, he above resuls were consisen wih he observaion ha vehicle one ran faser han vehicle wo. Table 1 summarizes he experimenal resuls for he firs video sequence. Boh manual measuremens, if available, and auomaic esimaions are shown in he able for comparison. Our sysem achieved a saisfacory performance for his video sequence. (a) (b) Fig. 8 (a) The background image generaed from he firs video sequence, and (b) he lane markings exraced from he background image

18 Fig. 9 Inermediae resuls of vehicle racking. Table 1: Experimenal resuls for video sequence 1 aken by he laboraory camcorder. Video sequence 1 aken by Manual Auomaic he laboraory camcorder measuremen esimaion Number of vehicles Flow rae (vehicles/hr) Time headway (sec) Time mean speed (km/hr) Space mean speed (km/hr) Video sequence wo (Figure 7(b)) was aken by our camcorder wih a viewing direcion ha was considerably differen from ha of he firs video sequence. The roadway viewed under his direcion easily caused occlusions/disocclusions of vehicles. In his example, he acual number of vehicles passing hrough he monioring zone was 48. However, our sysem repored 46 vehicles passing hrough he monioring zone. The wo missing errors were boh due o occlusions of small cars by large vehicles all he way hrough he enire monioring zone. Acually, here were several occlusions/disocclusions of vehicles presen in his video sequence. Those occlusions/disocclusions, however, did no boher he sysem because hey occurred wihin some porions of he monioring zone raher han he enire monioring zone. Our sysem deermined he number of vehicles by couning he pahs of vehicles insead of couning he vehicles hemselves so as o avoid he confusion. However, wrong assignmens of pahs for vehicles may occur a disocclusions. If wrong assignmens occurred, incorrec esimaions of vehicle speeds could resul. The experimenal resuls for video sequence wo are summarized in Table 2. Alhough only wo vehicles were missed, i led o a large difference in flow rae beween manual (6171 vehicles/hr) and auomaic (5880 vehicles/hr) measuremens. This is because he ime uni has been exended from second (for number of vehicles) o hour (for flow rae)

19 Table 2: Experimenal resuls for video sequence 2 aken by he laboraory camcorder. Video sequence 2 aken by Manual Auomaic he laboraory camcorder measuremen esimaion Number of vehicles Flow rae (vehicles/hr) Time headway (sec) Time mean speed (km/hr) Space mean speed (km/hr) Our sysem can simulaneously measure boh downsream and upsream raffic flows. Video sequence hree (Figure 7(c)) conained boh sreams of raffic flow in opposie direcions. Table 3 collecs he experimenal resuls for his video sequence. In his example, one exra vehicle was deeced in he downsream raffic flow. The associaed flow rae, ime headway, and raffic densiy hence involved error. Excep his, our sysem performed well, for he upsream raffic flow. Table 3: Experimenal resuls for video sequence 3 aken by he laboraory camcorder. Video sequence 3 aken by he laboraory camcorder Manual Measuremen Auomaic Esimaion down up down up Number of vehicles Flow rae (vehicle/hr) Time headway (sec) Time mean speed (km/hr) Space mean speed (km/hr) Finally, we show a video sequence (Figure 7(d)) aken a sunshine for 90 minues. Alhough he inensiy conrass of he images in his sequence are changed over ime because of he clouds passing, our sysem sill achieved a saisfacory performance for his video sequence as shown in Table 4. Table 4: Experimenal resuls for video sequence 4 aken by he laboraory camcorder. Video sequence 4 aken by Manual Auomaic he laboraory camcorder measuremen esimaion Number of vehicles Flow rae (vehicle/hr) Time headway (sec) Time mean speed (km/hr) Space mean speed (km/hr)

20 7.2. Public Camcorder The public camcorder mouned on a high pole was locaed on he roadside of a highway. Three video sequences provided by he public camcorder were acquired during dayime, raining evening, and dusk, respecively. Each sequence is recorded for 50 minues. Figure 10 shows hree images each from one of he hree sequences. Since public camcorders were originally deployed for monioring raffic condiions (e.g., congesion) as well as unusual evens (e.g., accidens) raher han for raffic measuremen, he qualiy of he images provided by he public camcorder were, on average, worse han hose acquired by our laboraory camcorder. (a) (b) (c) Fig. 10 Video images of sequences (a) one, (b) wo, and (c) hree, respecively, acquired by he public camcorder. The experimenal resuls for he firs public video sequence (Figure 10(a)) are shown in Table 5. In his example, he number of vehicles couned by hand was However, our sysem deeced 6004 vehicles. During examining he video sequence, we found here were several clips in he video sequence, in which brighness changes abruply from image o image. Some of hese changes were deeced as vehicles. However, such a brighness insabiliy was grealy reduced in he oher wo public video sequences. From Figure 10, i is clear ha he images in he las wo video sequences are much darker han hose in he firs sequence. I seems ha he darker a scene he more sable is he public camcorder o illuminaion. Table 5: Experimenal resuls for he firs video sequence aken by he public camcorder. Video sequence 1 aken by Manual Auomaic he public camcorder measuremen esimaion Number of vehicles Flow rae (vehicle/hr) Time headway (sec) Time mean speed (km/hr) Space mean speed (km/hr)

21 The second public video sequence (Figure 10(b)) was aken on a rainy evening. In his sequence, boh he downsream and upsream raffic flows were measured. The experimenal resuls are colleced in Table 6. In he downsream raffic flow, 3536 vehicles were deeced by our sysem. However, 3540 vehicles acually passed he monioring zone. The reason for his good performance was due o he counerbalance beween he over couning of he headligh reflecions from he we road surface (see Figure 11) and he missing vehicle wih low conras. Fig. 11 Errors caused by he reflecance of vehicle headlighs from he we road surface. Table 6: Experimenal resuls for he second video sequence aken by he public camcorder. Video sequence 1 aken by Manual Auomaic he public camcorder measuremen esimaion Number of vehicles Flow rae (vehicle/hr) Time headway (sec) Time mean speed (km/hr) Space mean speed (km/hr) The las public video sequence (Figure 10(c)) was aken a dusk. In his example, he number of vehicles deeced by our sysem was However, he number of vehicles couned by hand was Mos of he errors were caused by he vehicles whose headlighs were urned off. As can be seen in Figure 12, hose vehicles are barely recognized even by a human. Our sysem missed hem. Table 7 shows he experimenal resuls for his video sequence. Fig. 12 Errors caused by vehicles wih heir headlighs urned off

22 Table 7: Experimenal resuls for he hird video sequence aken by he public camcorder. Video sequence 3 aken by he public camcorder Manual measuremen Auomaic esimaion Number of vehicles Flow rae (vehicle/hr) Time headway (sec) Time mean speed (km/hr) Space mean speed (km/hr) Concluding Remarks and Fuure Work In his paper, a vision-based raffic measuremen sysem for auomaically collecing raffic daa from roadways was described. The proposed sysem, composed of a few hardware devices, is easily insalled, ransferred and operaed. Several echniques were inroduced during he developmen of he sysem, including a progressive background image generaion approach, a filer-based lane marking deecion mehod, an illuminaion assessmen echnique, and a shadow deecion and removal approach. Besides, we inroduced a camcorder calibraion process, in which only wo parameric values, he il angle and heigh of he camcorder, should be provided. Boh are local parameers and are convenienly measured. During vehicle racking, he muli-person decision making sraegy was modified so as o effecively handle exra/missing vehicle deecion, occlusion/disocclusion of vehicles, and vehicles moving in/ou he monioring zone The proposed sysem was esed using video sequences provided by our laboraory camcorder and a public camcorder. The video sequences acquired using our camcorder were, on average, beer in qualiy han hose aken by he public camcorder. A reasonable performance of our sysem was achieved for he video sequences acquired by our camcorder. However, several difficulies were encounered during processing he video sequences provided by he public camcorder. As demonsraed, he reflecion of vehicle headlighs from we roadway surfaces during rainy days and vehicles wih headlighs urned on or off a dusk were coninually annoying our sysem. These difficulies have o be avoided before we can exend our curren sysem o deal wih raffic scenes in he nighime. As menioned earlier, he algorihm of a raffic measuremen sysem heavily relies on he raffic scenes (e.g., ypes of roadways and he raffic condiions) under consideraion. There is sill adequae room for us o improve our curren sysem or o develop new sysems for dealing wih differen raffic scenes

23 References [1] E. Bengoexea, P. Larranaga, I. Bloch, and A. Perchan, Inexac Graph Maching by Means of Esimaion of Disribuion Algorihms, Paern Recogniion, [2] D. Beymer, P. McLauchlan, B. Coifman, and J. Malik, A Real-Time Compuer Vision Sysem for Measuring Traffic Parameers, Proc. of IEEE Conf. on Compuer Vision and Paern Recogniion, pp , [3] A. Branca, G. Aolico, and A. Disane, Cas Shadow Removing in Foreground Segmenaion, Proc. of he 16 h In l Conf. on Paern Recogniion, pp , [4] R. Cesar, E. Bengoexea, and I. Bloch, Inexac Graph Maching Using Sochasic Opimizaion Techniques for Facial Feaure Recogniion, Proc. of 16 h In l Conf. on Paern Recogniion, [5] Y. Chen, Y. Rui, and T. S. Huang, JPDAF Based HMM for Real-Time Conour Tracking, Proc. of IEEE Conf. on Compuer Vision and Paern Recogniion, pp , [6] H. Chui and A. Rangarajan, A New Algorihm for Non-rigid Poin Maching, Proc. of IEEE Conf. on Compuer Vision and Paern Recogniion, vol. 2, pp , [7] B. Coifman, D. Beymer, P. McLauchlan, and J. Malik, A Real-ime Compuer Vision Sysem for Vehicle Tracking and Traffic Surveillance, Transporaion Research Par C: Emerging Technologies, vol. 6, no. 4, pp , [8] D. Comaniciu, V. Ramesh, and P. Meer, Kernel-Based Objec Tracking, IEEE Trans. On Paern Analysis and Machine Inelligence, vol. 25, no. 5, pp , [9] R. Cucchiara, M. Piccardi, and P. Mello, Image Analysis and Rule-based Reasoning for a Traffic Monioring Sysem, IEEE Trans. on Inelligen Transporaion Sysems, pp , [10] D. J. Dailey, F. W. Cahey and S. Pumrin, "An Algorihm o Esimae Mean Traffic Speed Using Un-Calibraed Cameras," IEEE Trans. on Inelligen Transporaion Sysems, vol. 1, no. 2, pp , [11] L. S. Davis, R. Bajcsy, M. Herman, and R. Nelson, RSTA on he Move, Proc. of ARPA Workshop on Image Undersanding, pp , [12] R. Deriche and O. D. Faugeras, Tracking Line Segmens, Image and Vision Compuing, vol. 8, no. 4, pp , [13] M. P. Dubuisson and A.K. Jain, Objec Conour Exracion Using Color And Moion, Proc. of IEEE Conf. on Compuer Vision and Paern Recogniion, pp , [14] D. Dubois, H. Fargier and H. Prade, Propagaion and Saisfacion of Flexible Consrains, Fuzzy Ses, Neural Neworks, and Sof Compuing, R. R. Yager and L. A. Zadeh, eds. Van Nosrand Reinhold, New York, pp , [15] A. Elgammal, R. Duraiswami, D. Harwood, and L. S. Davis, Background and Foreground Modeling Using Nonparameric Kernel Densiy Esimaion for Visual Surveillance, Proceedings of he IEEE, vol. 90, no. 7, pp ,

24 [16] W. T. Freeman and E. H. Adelson, The Design and Use of Seerable Filers, IEEE Trans. on Paern Analysis and Machine Inelligence, vol. 13, no. 9, pp , [17] D. Gao, J. Zhou, and L. Xin, SVM-Based Deecion of Moving Vehicles for Auomaic Traffic Monioring, Proc. of IEEE Conf. on Inelligen Transporaion Sysems, pp , [18] R. A. Hummel and S. W. Zucker, On he Fundaions of Relaxaion Labeling Processes, Readings in Compuer Vision, M. A. Fischler and O. Firschein, eds. pp , Morgan Kaufmann Publishers Inc., California, [19] Y. Inoue, H. Hobaake, T. Namai, and N. Hanba, A Sudy on he Measuremen of Two-Dimensional Movemen of Traffic, Proc. of 17 h Image Engineering Conf., [20] Y. K. Jung and Y. S. Ho, Traffic Parameer Exracion Using Video-Based Vehicle Tracking, Proc. of IEEE In l Conf. on Inelligen Transporaion Sysems, pp , [21] K. Kameyama and K. Toraichi, Relaxaion wih Model Swiching and is Applicaion o Shape Maching, Proc. of he In l Join Con. on Neural Neworks, vol. 2, pp , [22] S. Kamijo, Y. Masushia, K. Ikeuchi, and M. Sakauchi, Traffic Monioring and Acciden Deecion a Inersecions, IEEE Trans. on Inelligen Transporaion Sysems, pp , [23] T. Kawaguchi, R. Nagaa, and T. Sinozaki, Deecion of Targe Models in 2D Images by Line-based Maching and a Generic Algorihm, Proc. of In l Conf. on Image Processing, vol. 2, pp , [24] M. Kilger, A Shadow Handler in a Video-Based Real-Time Traffic Monioring Sysem, Proc. of IEEE Workshop on Applicaions of Compuer Vision, pp , [25] L. A. Klein, Sensor Technologies and Daa Requiremen for Inelligen Transporaion Sysems, Arech House, Boson, [26] G. J. Klir and B. Yuan, Fuzzy Ses and Fuzzy Logic, Theory and Applicaions, Chaper 15, pp , Prenice Hall, New Jersey, [27] D. Koller, J. Daniilidis, and H. H. Nagel, Model-Based Objec Tracking in Monocular Image Sequences of Road Traffic Scenes, In l J. Compuer Vision, vol.10, pp , [28] D. Koller, J. Weber, and J. Malik, Robus Muliple Car Tracking wih Occlusion Reasoning, Proc. of 3rd European Conf. on Compuer Vision, pp , [29] A. H. S. Lai, N. H. C. Yung and C. Zhang, An Inelligen Framework for Spaio-Temporal Vehicle Tracking, Proc. of IEEE In l Symp. on Circuis and Sysems, vol. 4, pp , [30] M. S. Lee, Deecing People in Cluered Indoor Scenes, Proc. of In l Conf. on Compuer Vision and Paern Recogniion, pp , [31] W. Long and Y. H. Yang, Saionary Background Generaion: an Alernaive o he