2004 IEEE Intellgent Transportaton Systems Conference Washngton, D.C., USA, October 3-6, 2004 WeA1.3 An Effcent Background Updatng Scheme for Real-tme Traffc Montorng Suchendra M. Bhandarkar and Xngzh Luo Department of Computer Scence, The Unversty of Georga Athens, Georga 30602-7404, USA Abstract Background updatng s an mportant problem n dynamc scene analyss. It s crtcal to be able to dstngush between long-term gradual changes n the background and shortterm rapd changes resultng from movng objects n the scene. In ths paper we propose an effcent background updatng scheme for real-tme traffc montorng. In partcular, we address the sleepng person problem, whch arses frequently n the context of real-tme traffc montorng. The proposed scheme combnes two levels of reasonng: lowlevel reasonng based on pxel status analyss and hgh-level reasonng based on movng object correspondence analyss. The proposed scheme s robust and fast enough to satsfy the real-tme constrants of traffc montorng. 1. Introducton Separatng the foreground from background s an mportant though dffcult problem n computer vson. Ths problem s even more complex n the case of dynamc scenes where the background typcally changes wth tme and hence needs to be updated perodcally. In computer analyss of dynamc scenes t s crtcal to be able to dstngush between long-term gradual changes n the background whch are typcally global n nature (such as changes n ambent llumnaton) and short-term rapd changes n the scene resultng from the presence of movng objects. Most object trackng systems need a background mage to extract movng objects n the scene. Systems that use known background mages for tranng [1] are not adaptve to changes n the background f the tranng mages do not span all possble varatons n the background. Moreover, n many practcal stuatons, t s dffcult to acqure tranng mages that do not contan a movng object. Some object trackng systems use adaptve technques to update the background mage on the fly such as by perodcally computng the temporal average of the mage frames [2]. A major shortcomng of the temporal averagng scheme s ts nablty to address the sleepng person problem [8]. The sleepng person problem arses frequently n the context of automated traffc montorng when a movng vehcle stops n the scene (such as at a traffc lght) and, on account of beng motonless, s mproperly merged wth the background mage [8]. Another shortcomng of temporal averagng s the presence of shadows n the background mage especally n areas contanng hgh/frequent moton. Replacng temporal averagng by medan flterng addresses the shadowng problem but not the sleepng person problem [4]. Koller et al. [3] propose the followng background updatng scheme as an mprovement over temporal averagng and medan flterng: B(x, y; t +1)=B(x, y; t)+α(f (x, y; t) B(x, y; t)) (1) where B(x, y; t) s background mage at tme t, α = α 1 (1 M(x, y; t)) + α 2 M(x, y; t), F (x, y; t) s the mage frame at tme t, M(x, y; t) s the moton hypothess mask gven by { 1 f F (x, y; t) F (x, y; t 1) >Tt M(x, y; t) = 0 otherwse (2) and 1 α 1 α 2 > 0. Ths technque ensures that there are no shadows at the busy pxels n the background mage, where movng objects pass frequently. However, t stll cannot solve the sleepng person problem. In [5] and [6] t s assumed that relable background pxels are those whch do not exhbt moton for a long perod of tme. Ths assumpton, however, s not vald f movng objects become statc and reman statc for a long tme. A more sophstcated background updatng scheme proposed n [7] uses coarse object segmentaton at the mage block level to construct a block smlarty matrx usng moton nformaton. Ths method can handle stuatons where a movng object becomes statc for a gven tme nterval. However, the sze of the smlarty matrx scales quadratcally wth the length of the tme nterval under consderaton. The memory requrement and computatonal overhead make ths technque unsutable for real-tme object trackng. Snce the tme nterval for analyss s constraned by the lmtatons of memory and processng speed, f a movng object remans statc for longer than ths tme nterval, t s mproperly merged wth the background mage. Technques based on Kalman flterng [9] and lnear predcton [8] have been proposed for background updatng resultng n update equatons smlar to equaton (1). Edge data s typcally used to 0-7803-8500-4/04/$20.00 2004 IEEE 859
track objects and these technques work well n the absence of occluson. However, n traffc montorng, occluson s a common occurrence thus lmtng the applcablty of these technques. Occluson occurs when portons of several objects n the 3D scene project onto a common regon n the 2D mage plane. Thus, only the object closest to the camera s vsble n that regon of the mage plane. The prevously mentoned technques perform poorly when confronted wth the sleepng person problem [8]. We classfy the sleepng person problem nto two categores: the mddle-statc-object problem where a movng object becomes statc n the mddle of the frame sequence and the ntally-statc-object problem where an object s statc at the begnnng of the frame sequence and eventually moves. Nether the mddle-statc-object problem nor the ntallystatc-object problem are handled adequately by the prevously descrbed methods. In ths paper we propose a background updatng scheme for real-tme traffc montorng that addresses the sleepng person problem by combnng low-level reasonng based on the analyss of the status of ndvdual pxels wth hgh-level reasonng based on correspondence analyss of the movng objects n the scene. The hgh-level reasonng about nterframe object correspondence enables robust background updatng and detecton of the ntally-statc-object. The low-level reasonng enables one to update the status of each pxel (ndcatng whether t belongs to the background or not) and also detect certan abnormal stuatons caused by an ntally-statc-object. In the proposed scheme, the background updatng s done on a frame-to-frame bass wthout requrng much hstory nformaton to be stored. After the ntally-statcobject problem s solved, the background updatng can be performed wthn a local wndow nstead of the entre mage. Ths reduces the processng overhead for background updatng leavng more CPU tme for object trackng and other tasks such as object recognton. By perodcally movng the wndow around the mage frame, the background can be refreshed to adapt to changes n llumnaton. Most of the prevously cted technques detect movng objects by computng some measure of the nter-frame dfference. The proposed technque, however, uses the background mage to detect movng objects whle smultaneously usng the knowledge of the movng objects to update and detect errors n the background mage. Thus, the processes of movng object detecton and background updatng are closely ntegrated. Expermental results show that t s possble to use the background mage, before t s fully ntalzed, to detect a mddle-statc-object. The ntegraton of movng object detecton and background updatng makes the system robust to large varatons n the speeds of the movng objects wthn the feld of vew. 2 Background Updatng Scheme The proposed background updatng scheme can be summarzed as consstng of two prmary steps: (a) Perform mage segmentaton on the frst frame to ntalze the background mage (Secton 2.1), and (b) Update the background usng correspondence analyss and reasonng (Secton 2.2). The background mage s mantaned n an mage buffer B(x, y; t). Each pxel n B(x, y; t) has the followng attrbutes: status, whch ndcates whether the background pxel ntensty s vald or not (1 represents vald, 0 represents nvald); sum, whch represents the accumulaton of gray levels at ths pxel locaton snce the last tme when t was vald; count whch s the total number of updates to the pxel snce the last tme when t was vald n the background mage buffer; and g, whch s the background gray level of ths pxel gven by g = sum/count. The temporal averagng used to compute the value of g produces a robust background, whch adapts to changes n ambent llumnaton wth the passage of tme. For each pxel, we desgn a set of operators to update the background mage. The operator Add adds the gray scale value of the current frame F (x, y; t) to B(x, y; t) usng the followng rules: 1. If B(x, y; t) s vald, that s status =1, then sum = sum + F (x, y; t) count = count +1 g = sum/count 2. If the status of B(x, y; t) s nvald, that s status =0, then status = 1 sum = F (x, y; t) (4) count = 1 g = F (x, y; t) Equaton (3) s used to update a background pxel whch s already vald whereas equaton (4) s used to ntalze a background pxel whch s stll nvald. The operator Invaldate smply nvaldates the status of the background pxel,.e., status =0. Intally, the status of all the background pxels s set to nvald. We also mantan a tmer varable for each pxel, whch accounts for the tme that ths pxel has been contnuously contaned wthn movng objects. If the pxel s not contaned wthn any movng object at any tme, we set the tmer to 0. If the pxel s contaned wthn movng objects contnuously for too long, we nvaldate the background at that pxel. Ths operator, termed as Tmer Invaldate, s dsabled when the background s fully ntalzed and s stable. The three operators: Add, Invaldate and Tmer Invaldate are the basc low-level pxel-based (3) 860
reasonng operators. The frst two are used to refresh the background mage as descrbed n Secton 2.3. The thrd operator s used to solve the ntally-statc-object problem at the pxel level. When correspondence analyss and reasonng (Secton 2.2) fal to detect the ntally-statc-object (whch s ntally msclassfed as vald background) whch then moves away, the object extracton method n Secton 2.1 detects a false object at the ntal locaton of the ntally-statc-object n every background frame thereafter. The tmers assocated wth the pxels at the ntal locaton of the ntally-statc-object count untl they reach a gven threshold value after whch the Tmer Invaldate operator nvaldates the status of these background pxels. If there s no movng object at these pxel locatons after the Tmer Invaldate operaton, the false object wll no longer be detected and these background pxels are refreshed to the real background value usng the frst rule descrbed n Secton 2.3. 2.1 Background Image Intalzaton We ntalze the background mage usng nter-frame moton detecton. We defne the dfference mage as: D(x, y; t) = F (x, y; t) F (x, y; t 1) (5) We use double thresholdng to segment the movng objects n D(x, y; t). The frst threshold τ 1 s used to extract the core regons of the movng objects. The second threshold τ 2 <τ 1 s used to grow the core regons usng spatal connectvty resultng n a bnary mage T (x, y; t). Connected component labelng (CCL) and sze flterng are used to dentfy sgnfcant connected regons n T (x, y; t) whch are then presumed to represent the movng objects n the scene. A boundng box s computed for each movng object. We ntalze the background mage as follows: for every pxel (x, y) n the current frame F (x, y; t), f(x, y) s not n the boundng box of any movng object, then we Add F (x, y; t) to B(x, y; t), else, do nothng (.e., keep B(x, y; t) nvald). The boundng box s used to reduce the effect of the object shadow. The background updatng process s prone to error f there exsts a statc object n the scene at the begnnng of the frame sequence. Snce ths object cannot be extracted usng moton detecton, ts pxels are msclassfed as vald background pxels, causng problems n future updates to the background mage. As descrbed n the followng subsectons, the proposed scheme solves ths problem usng two levels of reasonng, one based on object correspondence analyss and the other based on a tmer assocated wth each pxel locaton n the background mage B(x, y; t). Wth the ntalzed background, we are able to refne the object extracton method. Instead of usng equaton (5) to get the dfference mage, we use the followng equaton: D(x, y; t) = max( F (x, y; t) F (x, y; t 1), F (x, y; t) B(x, y; t) g B(x, y; t) status) (6) n whch, B(x, y; t) g and B(x, y; t) status are the attrbutes of B(x, y; t) mentoned prevously. Essentally, f the background pxel (x, y) s vald, we nclude t to extract the object, f not, gnore t. For a movng object that s gong to stop, the value of F (x, y; t) F (x, y; t 1) exhbts a decreasng trend. But the value of F (x, y; t) B(x, y; t) g B(x, y; t) status can stll be large f status s 1. Thus, the mddlestatc-object problem s automatcally prevented, snce these pxels wll not be msclassfed as belongng to the background on account of the hgh value of D(x, y; t) at these pxel locatons. If there s no ntally-statc-object n the frame sequence, then a smplfed verson of our background updatng scheme s as follows: (1) Compute the dfference mage usng equaton (6), (2) Extract movng object(s) usng double thresholdng and CCL, and, (3) Update the pxel locatons where there are no movng objects present. We revert to ths smplfed scheme to refresh the background durng object trackng when the ntally-statcobject problem has been solved and all the background pxels have been valdated. 2.2 Correspondence Analyss and Reasonng The correspondence analyss and reasonng phase detects the ntally-statc-object and refreshes the background mage at the same tme. Durng ths phase, we also solve the problems created by the presence of object shadows and unformty n object color. When the value of count s not large enough, the presence of object shadows wll cause the background pxel ntensty values to devate sgnfcantly. It s mportant to mnmze the effect of object shadows n the ntal stages when the background mage s not stablzed. Also, f an object wth unform color or ntensty such as a car or truck moves very slowly and f the background status s nvald for the pxels contaned wthn ths object, then the moton nformaton n the nteror pxels of the object may be dffcult to obtan. The background refreshng process tself makes the background mage adaptve to the changes n llumnaton. We use a smple scheme to predct the object speed for correspondence analyss nstead of Kalman flterng [3] or lnear predcton. We use an object s speed computed n a prevous frame as t s predcted speed n the current frame. The result of the correspondence analyss s used to compute the actual speed of the object n the current frame and to update the predcton for the current frame. Ths method 861
works very well for traffc montorng, where a movng object moves coherently. Correspondence analyss s observed to mprove the overall robustness of background updatng process. Moreover, errors n correspondence analyss are observed not to cause sgnfcant devatons n the gray level values of the background pxels. Suppose the movng objects at tme t and tme t 1 are denoted by O(t) = {o t 1,o t 2,..., o t n} and O(t 1) = {o t 1 1,o t 1 2,..., o t 1 m }, respectvely. For each object o t 1 we predct ts new poston ω t 1 usng the predcted speed. We compute the overlappng area between ω t 1 and every object n O(t) and construct the correspondence table C, where entry C[, j] denotes the sze of the overlappng regon between object ω t 1 and o t j. We perform nter-object correspondence analyss based on the followng categorzaton: 1. One-to-One Correspondence: Object o t 1 O(t 1), corresponds to only one object o t j O(t) and vce versa. Thus row and column j n C are all 0 s except for cell C[, j]. Hence objects o t 1 O(t 1) and o t j O(t) are deemed to be the same physcal object n successve frames. 2. One-to-None Correspondence: Object o t 1 1) does not correspond to any object n O(t). O(t 3. None-to-One Correspondence: For an object o t j O(t), there s no correspondng object n O(t 1). 4. One-to-Many Correspondence: For an object o t 1 n O(t 1), there s more than one correspondng object n O(t), however, object o t 1 s the only correspondng object for each of them. 5. Many-to-One Correspondence: For an object o t j there s more than one correspondng object n O(t 1), however, o t j s the only correspondng object for each of them. 6. Many-to-Many Correspondence: Objects o t 1 and o t 1 n O(t 1) correspond to the same object n o t j O(t). However, ot 1 also corresponds to another object o t j O(t). In ths case, we use the followng steps to smplfy the correspondence, so that the result falls n one of categores 1 5 descrbed above. (a) If object o t 1 O(t 1) corresponds to several objects n O(t), but one of the these objects,, corresponds to several objects n O(t 1) say o t j (ncludng o t 1 ), then we examne column j n C, and set the smallest non-zero correspondence value n ths column to 0. (b) If object o t j O(t) corresponds to several objects n O(t 1), but one of these objects, say o t 1, corresponds to several objects n O(t) (ncludng o t j ), then we examne the correspondng row n C and set the smallest non-zero correspondence value n ths row to 0. Steps (a) and (b) are performed repeatedly untl the resultng correspondence falls n one of categores 1-5 above. 2.3 Background Updatng wth Correspondence Reasonng The results of the correspondence analyss are used to update the background mage B as follows: 1. If a pxel (x, y) s not contaned n any movng object n ether O(t 1) or O(t), then we just Add the pxel value F (x, y; t) to the background pxel B(x, y; t) usng the Add operator descrbed prevously. 2. In the case of one-to-one correspondence, we consder the object to be movng normally. A common occurrence s when an object whch s statc at the begnnng of the frame sequence begns to move, t starts wth a small sze and then grows rapdly. Thus, when we detect an object whose area growth rate s larger than a threshold, we nvaldate the background at the pxels wthn the boundng box of that object. Ths solves some nstances of the ntally-statc-object problem. Otherwse, f a pont (x, y) s contaned n the boundng box of the object at tme t 1 but not n the boundng box of the correspondng object at tme t, then we Add the pxel value n the current frame to the background as n case 1 above. 3. In the case of many-to-one correspondence, we use a boundng box whch s the smallest one that ncludes the many boundng boxes and then handle the relaton as f t were a one-to-one correspondence. The reason we use the merged boundng box s to be able to handle the stuaton where an object has unform color. In such a case, t s dffcult to estmate the moton n the nteror pxels of the object usng temporal dfferencng. Consequently, the extracted object s often fragmented nto more than one component. It s possble for pxels lyng n areas between these components (whch truly belong to the object) to be msclassfed as background pxels. Ths results n false updates to the background whch could cause the background mage qualty to deterorate, especally when t has not been completely ntalzed. 4. In the case of one-to-many correspondence, we use the same reasonng as n the case of many-to-one correspondence. 862
5. In the case of one-to-none correspondence, we have an object movng out of the scene or just random nose. We smply Add the pxels n the boundng box to the background after a chosen tme delay, such as 100 frames. The tme delay s chosen to mnmze the effect of random moton n the mage (due to nose) on the background update. 6. In the case of none-to-one correspondence, we have an object that has stayed n the background snce the begnnng of the frame sequence and has now begun to move or just random nose. In ths case, we just nvaldate the background at pxels wthn the boundng box of the object. Ths solves most nstances of the ntally-statc-object problem. After the background s ntalzed, we can use a smple verson of background updatng that can run n real tme. We use equaton (6), double thresholdng and CCL to extract the movng object(s). However, we do not perform correspondence analyss. We only update the background pxels, where there s no object present, usng equaton (3). The computatonal requrements of the background updatng procedure can be scaled down, f requred, to meet the temporal constrants of real-tme traffc montorng. The background updatng procedure can be performed only wthn a wndow of predetermned sze and poston n the background mage. Thus, the object extracton and correspondence analyss, whch are used to update background, are performed only wthn the wndow, where the wndow sze s chosen to be larger than the largest object n the scene. The wndow s then moved perodcally wthn the mage frame n a prespecfed manner n order to update the entre background mage. The movng wndow-based approach can be used to update the background mage after t s ntalzed so that computatonal resources can be made avalable for other tasks such as object recognton and object trackng. 3 Expermental Results Our experments on traffc montorng vdeos show very good results when there s no statc object at the begnnng of the frame sequence. When a statc object moves nto the feld of vew and becomes motonless, t s never merged nto the background mage. When there s an ntallystatc-object, the correspondence reasonng sometmes fals to detect that t has begun to move because of the presence of an occludng object. As descrbed n Secton 2, the Tmer Invaldate operator can solve ths problem eventually f the traffc s not very heavy. When the ntally-statcobject moves out of ts poston, a false object wll be detected at that poston n the background mage. The tmer assocated wth the pxels at ths poston wll count untl the elapsed tme exceeds a threshold, after whch these pxels wll perform a Tmer Invaldate operaton. If there s no movng object at that poston at ths tme, the real background value wll be coped nto these pxels wth the Add operator n the next frame. We also notce that when there s heavy traffc, t takes a long tme for the pxel values n the background mage to converge to ther true background values, after the Tmer Invaldate operaton has been performed. Ths s because n heavy traffc t s possble to encounter another statc object at the very pxel poston that s beng nvaldated, thus preventng the true background value from beng coped nto that pxel va the Add operaton. Fgure 1 shows a seres of snapshots of the background updatng process. The ntal background mage (Fgure 1(b)) s just a copy of the second frame (Fgure 1(a)). However, the pxel status s nvald n the area where moton s detected. The whte car n the center of the feld of vew s statc n the begnnng and hence appears n the background mage (Fgure 1(b)). Even when the whte car moves away n frame 1328 (Fgure 1(c)), t perssts n the background mage due to the presence of other vehcles that partally occlude t (Fgure 1(d)). However, by frame 6700 (Fgure 1(e)), the tmer mechansm ensures that all pxels are updated to ther true background values (Fgure 1(e)). The result of the background updatng scheme based on smple temporal averagng of the frames s shown n Fgure 1(g). The background mage can be observed to contan the statonary vehcles watng at the traffc lght (.e., the sleepng person problem). Koller s scheme (equaton (1)) also has a serous problem when dealng wth statc objects, snce t can adapt very quckly to statc objects n the scene causng them to merge wth the background mage. Fgure 2 shows a comparson of the converge rates of the proposed scheme, Koller s scheme and temporal averagng scheme for background updatng. The y axs n Fgure 2 represents the percentage of pxels that do not match the real background values wthn a prespecfed threshold. The x-axs represents tme measured n terms of the number of frames (at a unform samplng rate of 30 frames/second). The proposed background updatng scheme can be seen to converge more rapdly to an overall background value that s much closer to the real background value compared to the other two schemes. 4 Conclusons In ths paper we proposed a background updatng scheme for a real-tme traffc montorng system. For the sleepng person problem, we consdered two cases termed as the ntally-statc-object problem and the mddle-statcobject problem. The former was handled wth the pro- 863
(a) Image Frame 2 (b) Background Frame 2 (c) Image Frame 1328 (d) Background Frame 1328 Fgure 2: Convergence curve References (e) Image Frame 6700 (f) Background Frame 6700 (g) Background Frame 6700 wth temporal averagng Fgure 1: Snapshots of the background updatng procedure posed two-level reasonng scheme. The two-level reasonng scheme was shown to also handle the problem of unform color object and shadows. The hgh-level reasonng based on nter-object correspondence solved the ntally-statcobject problem quckly when there was lttle occluson. In the event that the hgh-level reasonng faled to solve ths problem (n cases of heavy occluson), the low-level reasonng based on the Tmer Invaldate operator solved the problem (even n the presence of heavy occluson) though t took longer. The Tmer Invaldate also handled the mddlestatc-object problem. In summary, the proposed background updatng scheme was seen to be robust and scalable wth respect to the number of movng objects n the scene and changes n background llumnaton. It dd not ental tranng on typcal background mages, had low computatonal complexty and was fast enough to satsfy the temporal constrants of real-tme traffc montorng. [1] M. Isard and J. MacCormck, BraMBLe: A Bayesan Multple-Blob Tracker, Proc. Intl. Conf. Computer Vson, Vol.2, Vancouver, Canada, July 2001, pp. 34-41. [2] S. Kamjo, Traffc Montorng and Accdent Detecton at Intersectons, IEEE Trans. Intellgent Transportaton Systems, Vol. 1. No. 2, June 2000, pp. 108-118. [3] D. Koller, J.W. Weber and J. Malk. Robust Multple Car Trackng wth Occluson Reasonng, Proc. European Conf. on Computer Vson, Stockholm, Sweden, May 1994, pp. 189-196. [4] M. Massey and W. Bender, Salent stlls: Process and practce, IBM Systems Journal, Vol. 35, Nos. 3&4, 1996, pp. 557-573. [5] S.Y. Chen, S.Y. Ma, and L.G. Chen, Effcent Movng Object Segmentaton Algorthm Usng Background Regstraton Technque, IEEE Trans. Crcuts and Systems for Vdeo Technology, Vol. 12, No. 7, July 2002, pp. 577-586. [6] T. Meer and K. N. Ngan, Automatc Segmentaton of Movng Objects for Vdeo Object Plane Generaton, IEEE Trans. Crcuts and Systems for Vdeo Technology, Vol. 8, No. 5, Sept. 1998, pp. 525-538. [7] D. Farn, P.H.N. de Wth, and W. Effelsberg, Robust Background Estmaton for Complex Vdeo Sequences, Proc. IEEE Intl. Conf. Image Processng, Barcelona, Span, Sept. 2003, pp. 145-148. [8] K. Tooyama, J. Krumm, B. Brumt and B. Meyers, Wallflower: Prncples and Practce of Background Mantenance, Proc. Intl. Conf. Computer Vson, Corfu, Greece, Sept. 1999, pp. 255-261. [9] C. Rdder, O. Munkelt, and H. Krchner, Adaptve background estmaton and foreground detecton usng Kalmanflterng, Proc. Intl. Conf. Recent Adv. Mechatroncs, Istanbul,Turkey, August 1995, pp. 193-199. 864