Online codebook modeling based background subtraction with a moving camera

Transcription

1 Onlne codebook modelng based background subtracton wth a movng camera Lyun Gong School of Computer Scence Unversty of Lncoln, UK Emal: lgong@lncoln.ac.uk Mao Yu School of Computer Scence Unversty of Lncoln, UK Emal: myu@lncoln.ac.uk Tmothy Gordon School of Engneerng Unversty of Lncoln, UK Emal: tgordon@lncoln.ac.uk Abstract Ths paper proposes a new background subtracton method by a movng camera for the object detecton. Key ponts are frstly extracted and tracked. From the trackng results, spatal transformaton relatonshps for the background scenes n consecutve frames are obtaned whle the current frame s warped to the prevous mage plane for the camera movement compensaton. A codebook background model s constructed and updated n an onlne way by explotng the full RGB color nformaton, whch s used to dstngush the foreground/background regons. Both qualtatve and quanttatve expermental results show that the proposed method outperforms ts counterparts wth a better performance. Index Terms movng camera, background subtracton,codebook, mage matchng I. INTRODUCTION Currently there s an ncreasng demandng of for the montorng of publc and prvate spaces due to the steady ncrease for dfferent purposes (e.g., safety, healthcare and crme preventon) [1]. Large numbers of cameras are commonly deployed for montorng and traned people are asked to watch real-tme vdeos through closed-crcut TV (CCTV) systems to fnd any abnormal events. However, humans observers are not capable of watchng many cameras smultaneously and they wll be tred after a long tme montorng. For ths reason, automated vsual survellance system enjoys wde researches these days. For an automated vsual survellance system, one of the most mportant thngs s to detect movng objects n the montored envronment. As n [2], [3], the background subtracton technque s one of the most successful approaches for the movng object detecton. These methods buld statstcal background models and extract movng objects by fndng regons whch do not have smlar characterstcs to the background model. However, they have lmtaton that they are only applcable wth the statonary cameras. For detectng movng objects wth non-statonary cameras, Cucchara et al. [4] and Robnault et al. [5] propose panoramc background model based methods. Background models correspondng to panoramc mages are constructed through mage regstraton and movng objects are segmented based on the constructed panoramc models. Moreover, the key pont matchng method s adopted n [6] to solve the possble regstraton errors for more robust object segmentaton. However, the panoramc background model based methods need partcularly accurate camera moton model and also suffer from sttchng error accumulaton. Zhang [7] and Thakoor et al. [8] use a dense optcal flow based method. Movng objects are detected by comparng the estmated optcal flows wth the estmated camera moton. However, dense optcal flow requres heavy computaton and when the camera motons are large, the computatons of optcal flow usually fal. Recently, [9], [10] there has proposed a background subtracton method for non-statonary cameras wthout constructng a panoramc background model. Camera motons are estmated by the key ponts trackng whle the mage warpng technque s appled to match the current frame wth the background model correspondng to the prevous frame, for the movng objets detecton. Pxelwse Gaussan dstrbutons are exploted to model the background consderng both spatal and temporal nformaton. Compare wth the aforementoned methods [4] [6], t does not need panoramc background models thus problems caused by the sttchng error accumulaton can be avoded. Compared wth the optcal flow based methods, t s both tme-effcency and robust to the large camera motons as mentoned n [7], [8]. In ths work, a new method s proposed for the background subtracton detecton. Smlar to [9], [10], key ponts are extracted and tracked for consecutve frames. Image warpng technques are then appled to match the consecutve frames for the camera moton compensaton. However, dfferent from [9], [10] whch only explot ntensty nformaton to buld a scalar value based Gaussan dstrbuton for the background modelng, the full RGB color nformaton s exploted and a codebook based method as n [11] s mplemented onlne for the background model constructon, model updatng and background subtracton. The proposed method keeps the advantages of [9], [10] (e.g., low computatonal costs, avodng the sttchng error problems, etc.) and can obtan more accurate background subtracton results due to the explotaton of the full RGB color nformaton (nstead of only the ntensty nformaton as n [9], [10]).

2 Codebook model Frame t Key ponts extracton Ponts trackng Image warpng Background subtracton Output Frame t-1 Fg. 1. The flowchart of the proposed algorthm. The structure of ths work s proposed as follows: Secton II shows the framework of the proposed approach. Image warpng for the consecutve frames matchng s proposed n Secton III. The methodologes of codebook background model constructon and updatng, as well as ts applcaton for the background subtracton are proposed n Secton IV. Expermental results are gven n Secton V and we gve the fnal conclusons n Secton VI. II. F RAMEWORK OF THE PROPOSED APPROACH A flowchart of the proposed algorthm for consecutve frames s presented n Fg. 1. Frstly, key ponts n a mage (e.g., corner ponts) are extracted and tracked. Certan mage warpng transformatons could then be appled to match the background scenes n consecutve frames, for compensatng the background changes due to the camera movement. Based one the warpng results, a codebook model s constructed and updated, whch s used for the background subtracton to extract movng objects n the current frame. The detals of each block are presented n next few sectons. III. I MAGE WARPING Some key ponts (.e. corners) are frstly extracted from the mage and Lucas Kanade Trackng method (LKT) algorthm [9] s appled to track the key ponts n consecutve frames, as shown n Fg. 2. We defne Xt 1 = [x1t 1,..., xn t 1 ] the ensemble of N key ponts found at tme t 1 and Xt = [x1t,..., xn t ] s defned as the ensemble of the tracked ponts at tme t. Here xt = [ut, vt, 1] whle ut and vt represent ts 2D poston n the mage. Based on the defntons, we can solve the followng equaton for obtanng a transformaton matrx: Xt 1 = H Xt, (1) where the transform matrx H s a 3-by-3 matrx whch descrbes the spatal relatonshp between two consecutve frames. As n [9], H can be solved through a least square crtera wth: H = Xt 1 XtT (Xt XtT ) 1, (2) where ( )T represents the matrx transpose and ( ) 1 represents the matrx nverse. By multplyng the estmated transform matrx H on the pxels postons on the current mage, t s warped onto the mage plane at the prevous tme nstance and the same background scenes n consecutve frames can approxmately overlap wth each other, whch compensates for the camera movement. One mage warpng example s shown n Fg. 2 (c) (d). IV. O NLINE CODEBOOK BACKGROUND SUBTRACTION Certan background model can be constructed for the background subtracton, based on the mage warpng for algnng the same background scenes n consecutve frames. [9], [10] propose to model the background nformaton usng a sngle Gaussan dstrbuton based on the ntensty value; however, t s not realstc to assume a smple Gaussan dstrbuton model for every background pxel (.e., some background pxels may correspond to more than one color modalty [11]); besdes, t s not suffcent to only explot the ntensty nformaton nstead of the full RGB color nformaton for the background subtracton. In ths work, an onlne codebook modelng method s developed by explotng codewords to model the background nformaton n a pxel-wse way. For each pxel, there are two sets of codewords correspondng to t: background codewords set (denoted as Bcodewords ) and cache

3 (c) (d) Fg. 2. The llustraton of the key ponts extracton, trackng and mage warpng. The extracted corner ponts (blue stars) n a frame and the tracked ones (red stars) are shown n and. (c) and (d) show the orgnal frame and ts projecton on the prevous frame plane (enclosed by the red lnes) through the mage warpng. codewords set (denoted as Ccodewords ). Codewords n Bcodewords represent the background color nformaton for that pxel. The ones n Ccodewords represent the codewords whch are currently nconsstent wth the background color, but may potentally be updated to be the background codewords n Bcodewords due to the change of the envronment. One codeword denoted as c s composed of an RGB G, B) and a 5-tuple aux = I, ˇ I, ˆ f, λ, p, q. vector v = (R, The meanng of the elements n the 5-tuple aux s shown n Table I: TABLE I T HE MEANING OF THE ELEMENTS IN THE TUPLE AUX. ˇ Iˆ the mn and max brghtness of all pxels I, assgned to ths codeword f the frequency wth whch the codeword s matched λ the tme perod that the codeword s not matched p,q the frst and last matched tmes of the codeword beyond the mage dmensonalty (assumng every mage has the same dmensonalty of H W ), then the pxel belongs to the newly emergng regon due to the camera movement and a background codeword s ntalzed for that pxel. If xt 1 s wthn the mage dmensonalty, pt wll be compared wth codewords n both Bcodewords and Ccodewords assocated wth pxels wthn a small regon (denoted as Rt 1 ) around xt 1 consderng possble warpng errors, to determne whether pt s a background pxel or not. In ths work, Rt 1 s set to be square centerng at xt 1 wth the wdth beng l. Comparsons wll be made by both the color dstorton measurement and brghtness evaluaton [11], as llustrated n (3): p v 2 p 2 v { true f Ilow I Ih brghtness(p, c) = f alse otherwse colordst(p, c) = (3) where p = [r, g, b] representng the RGB vector for a pxel p n a three channels color mage and I represents the correspondng ntensty value. Ilow and Ih are calculated from the frst two components Iˇ and Iˆ of the aux vector n the codeword c as: Iˇ } (4) α where α and β are manually set parameters. As n [11], t s mentoned that a pxel p s matched wth a codeword c, f the compared colordst value s smaller than a threshold and brghtness s true. For the pxel pt, f there exsts a matched background codeword assocated, then the pxel s regarded as a wth any pxel wthn Rt 1 background pxel; otherwse, f no such codewords exst, pt s regarded as a foreground pxel representng a movng object. ˆ Ih = mn{β I, ˆ Ilow = αi, C. Onlne background model updatng A. Codebook model ntalzng The codebook model s ntalzed when the frst frame comes. For every pxel n the frame, an assocated codeword s constructed wth the correspondng v beng set to be the RGB values for that pxel and aux s set to {I, I, 1, 0, 1, 1} (where I represents the ntensty value), whch s put nto the background codewords set Bcodewords. The correspondng cache codewords set Ccodewords s set to be empty. B. Background subtracton For a newly ncomng frame at tme nstance t, frstly we explot the matrx H as (2) to project every pxel poston to the mage plane at t 1 accordng to (1). For a pxel denoted as pt, f ts projected poston (denoted as xt 1 ) s Codewords n the background model are desgned to update n an onlne way to account for the envronmental changes n vdeo streams. For any matched codeword, ts correspondng RGB vector v and components n the aux wll be updated by the [r, g, b] vector and ntensty value I of the pxel pt as n (5): fm R + r fm G + g fm B + b,, ), fm + 1 fm + 1 fm + 1 Iˇ mn{i, Iˇm }, Iˆ max{i, Iˆm }, v ( fm fm + 1, λ=0 q t. (5)

4 (c) (d) Fg. 3. Illustratons of the background subtracton results for dfferent scenes:. Orgnal frames. Ground truth movng objects regons (c). Background subtracton results based on the method n [9], [10] (d). Background subtracton results by the proposed method For the codewords are not matched, the λ component wll be updated as λ λ + 1 and all other components wll kept the same. Codewords n the set Ccodewords wll be added nto Bcodewords when matched frequently (fm > th1 ) ndcatng the background changes. Besdes, codewords wth λ > th2 wll be removed from the codewords set due to not beng matched for a certan tme perod. th1 and th2 are preset threshold values. V. E XPERIMENTAL RESULTS The proposed method s tested on a vdeo sequence for detectng movng objects (people, cyclsts, cars and vans) on the road, whch s recorded by a camera mounted on a vehcle. Recorded frames have a dmensonalty of Representatve frames are shown as n Fg. 4. Fg. 3 shows the qualtatve movng object detecton results of the proposed background subtracton method, as well as ts counterpart n dfferent traffc scenaros. Intutvely, we can observe the background subtracton results by the proposed method could better match the ground truth results wth a comparatvely smaller number of background pxels beng mstaken as foreground ones. The recall and precson values as n (6) are used for a quanttatve comparson. The hgher recall and precson values are, the better the background subtracton algorthm s as ndcated from ths equaton. A pece of vdeo Fg. 4. Selectve mages n the recorded vdeo sequence. sequence s chosen and the movng objects (as llustrated n Fg. 5) are extracted by dfferent background subtracton methods. The related recall and precson values for every frame are calculated and shown n Fg. 6, from whch we can see that for most of the tme nstances the proposed method could obtan both hgher recall and precson values, whch ndcate the better performance of the proposed algorthm. The reason for the better performance of the

5 Fg. 5. Movng objects (cyclst and vehcle) and the correspondng ground truth regons. proposed algorthm s that: compared wth the state-ofthe-art methods [9], [10] whch only explot the ntensty nformaton, the proposed methods explots the full RGB color nformaton for the background modelng; besdes, nstead of assumng a smple Gaussan dstrbuton for each pxel, the codebook based method can model the background nformaton n a more representatve way. precson = tp tp + fp, recall = tp tp + tn tp: foreground pxels whch are correctly detected fp: background pxels whch are mstaken as foreground ones tn: correctly detected backgrround pxels (6) VI. CONCLUSION In ths work, we propose a novel background subtracton method based on the movng camera. Key ponts are extracted and tracked whle mage warpng s appled to match the same background scenes for consecutve frames for compensatng the camera movement. A codebook model method s constructed and onlne updated, for extractng the movng foreground objects by explotng the full RGB color nformaton and codewords based representaton. Expermental results on a real vdeo sequence show the advantages of the proposed method over ts state-of-theart counterparts. For future works, we wll work on the mprovement of the current mage warpng method. Same background regons n consecutve frames can thus be matched n a better way, whch leads to better background subtracton results. REFERENCES [1] P. Remagnno, S. Velastn, G. Forest, and M. Trved, Novel concepts and challenges for the next generaton of vdeo survellance systems, Machne Vson Applcaton, vol. 18, no. 3, pp , [2] A. Elgammal, R. Duraswam, D. Harwood, and L. Davs, Background and foreground modelng usng nonparametrc kernel densty estmaton for vsual survellance, Proceedng of IEEE, vol. 90, no. 7, pp , [3] T. Ko, S. Soatto, and D. Estrn, Warpng background subtracton, n 2010 IEEE Conference on Computer Vson and Pattern Recognton (CVPR), 2010, pp Fg. 6. The precson and recall curves for the background subtracton results on a pece of vdeo sequence. [4] R. Cucchara, A. Prat, and R. Vezzan, Advanced vdeo survellance wth pan tlt zoom cameras, n In: Proceedngs ofworkshop onvsual Survellance (VS) at ECCV, [5] L. Robnault, S. Bres, and S. Mguet, Real tme foreground object detecton usng ptz camera, n Proceedngs of the Fourth Internatonal Conference on Computer Vson Theory and Applcatons, 2009, vol. 1. [6] C. Gullot, M. Taron,, P. Sayd, Q. Pham, C. Tlmant, and J. Lavest, Background subtracton adapted to ptz cameras by keypont densty estmaton, n Proceedngs of the Brtsh Machne Vson Conference, vol. 34, pp [7] Y. Zhang, S. Kselewch,, W. Bauson, and R. Hammoud, Robustmovng object detecton at dstance n the vsble spectrum and beyond usng a movng camera, n In: Conference on Computer Vson and Pattern Recognton Workshop, New York, USA, [8] N. Thakoor, J. Gao,, and H. Chen, Automatc object detecton n vdeo sequences wth camera n moton, n Advanced Concepts for Intellgent Vson Systems, Brussels, Belgum, [9] S. Km, K. Yun, K. Y, S. Km, and J. Cho, Detecton of movng objects wth a movng camera usng non-panoramc background model, Machne Vson and Applcatons, vol. 24, no. 5, pp , [10] A. Vswanath, R. Behera, V. Senthamlarasu, and K. Kutty, Background modellng from a movng camera, Proceda Computer Scence, vol. 58, pp , [11] K. Km, T. Chaldabhongse, D. Harwood, and L. Davs, Real-tme foregroundbackground segmentaton usng codebook model, Real Tme Imagng, vol. 11, no. 3, pp , 2005.