Real-time ghost removal for foreground segmentation methods

Transcription

1 Real-tme ghost removal for foreground segmentaton methods Fe Yn, Dmtros Makrs, Sergo Velastn To cte ths verson: Fe Yn, Dmtros Makrs, Sergo Velastn. Real-tme ghost removal for foreground segmentaton methods. The Eghth Internatonal Workshop on Vsual Survellance - VS2008, Oct 2008, Marselle, France <nra > HAL Id: nra Submtted on 29 Sep 2008 HAL s a mult-dscplnary open access archve for the depost and dssemnaton of scentfc research documents, whether they are publshed or not. The documents may come from teachng and research nsttutons n France or abroad, or from publc or prvate research centers. L archve ouverte plurdscplnare HAL, est destnée au dépôt et à la dffuson de documents scentfques de nveau recherche, publés ou non, émanant des établssements d ensegnement et de recherche franças ou étrangers, des laboratores publcs ou prvés.

2 Real-tme ghost removal for foreground segmentaton methods Fe Yn, Dmtros Makrs, Sergo A Velastn Dgtal Imagng Research Centre (DIRC) Faculty of Computng, Informaton Systems and Mathematcs Kngston Unversty, Surrey, KT1 2EE, UK {fe.yn, d.makrs, sergo.velastn}@kngston.ac.uk Abstract Among all the challenges for foreground detecton, we focus on detecton of ghosts caused by the startng or stoppng of obects. We propose a fast and effectve method for ghost detecton that compares the smlarty between the edges of the detected foreground obects and those of the current frame based on obect-level knowledge of movng obects. Fnally we demonstrate the performance of ghost detecton algorthm usng a seres of urban traffc vdeo sequences ncludng car parkng and departure whch normally cause ghost problems. The performance evaluaton results show that the proposed method can detect and reduce ghost obects effcently, brngs lttle computatonal load, and no sgnfcant sde-effect to the survellance system. 1. Introducton Vsual survellance n dynamc scenes, especally for trackng humans and vehcles movng through the feld of vew (FOV) of a CCTV camera has receved much attenton n the past decades [1]. Because detecton of movng obects s a key prelmnary, t ultmately lmts successful trackng n survellance applcatons such as traffc montorng and analyss, access control n specal areas, human and vehcle dentfcaton, and detecton of anomalous behavours. The most common approach for detectng movng obects s background subtracton, n whch each frame s compared aganst a background model. The pxels n the current frame that have sgnfcant dfference from the background are then consdered as movng pxels or foreground pxels. These foreground pxels wll be grouped to form obects whch are then tracked. A large number of background subtracton algorthms have been proposed so far [1], but problems stll reman for movng obects dentfcaton under certan condtons. One of the toughest problems s that background subtracton causes the detecton of false obects when an obect that belongs to the background starts to move away, often referred to as ghost. It s mportant to address the problem because ghost obects wll adversely affect many tasks such as obect classfcaton, trackng and event analyss (e.g. abandoned tem detecton). Therefore, ghost obect needs to be separated from other obects and elmnated. Ths paper focuses on the problem of ghost dentfcaton and elmnaton. We used a state-of-theart ndustral tracker [6] whch ncludes basc background subtracton and obect trackng. Then we ncluded our ghost detecton algorthm nto the basc tracker to dentfy and elmnate ghosts. Fnally, we systematcally evaluate and compare performance on some urban traffc vdeo sequences. The ground truth for all vdeos was manually generated usng Vper GT [8]. Ths paper s organzed as follows: Secton 2 defnes the problem of ghost detecton and surveys some prevous methods n the lterature. Secton 3 descrbes the methodology of ghost detecton. Results are presented and dscussed n secton 4. Fnally secton 5 concludes the paper. 2. Background Generally, a robust background model should be able to automatcally recover and update tself from a dynamc sequence and be nsenstve to llumnaton changes, shadows, weather condtons such as ran or snow. One of the most popular exstng background modellng algorthms was proposed by Stauffer and Grmson [2] n 1998 and further refned e.g. n [9]. Besdes background modellng, the accuracy of moton detecton also depends on rules of moton

3 segmentaton and obect classfcaton. Normally, moton segmentaton methods are based on the noton that dfferences n consecutve mages arse from movng obects. The moton s measured usng ether mage dfferencng or flow. There are some drawbacks of usng a background model. One problem related to our work s that, to account for changng llumnaton condtons, obects that become statonary are relatvely quckly ncorporated nto the background and when they move agan they leave behnd an area of foreground (or ghost ) that can be mstakenly taken as a new obect. So the ghost problem s a drect result of the approach to background modellng. An approach that would feedback hgh level obect detecton/trackng nformaton to the background model mght be able to reduce these effects, for example by enforcng constrants such as that slow or stopped obects cannot smply dsappear (merged nto the background). Cucchara et al [3] proposed a general frame work of background subtracton called Sakbot whch combnes statstcal assumptons to detect movng obects, apparent obects (ghosts), and shadows wth the obect level knowledge of those from prevous frames. For ghost detecton, they calculate the average optcal flow, over all the detected movng pxels. They assumed that movng obects should have sgnfcant moton, whle ghosts should have a nearto-zero average optcal flow snce ther moton s only apparent. However, optcal flow s computatonal expensve for real tme processng and also there s a danger of naccurate classfcaton of statonary obect nto ghost. Cheung et al [4], proposed a dfferent method to detect ghost n traffc survellance. They used a frame dfference mask whch was computed usng the ncomng frame and ts prevous frame. Obects wth no correspondng blob n the frame dfference mask were dentfed as ghosts. However, the method can not dstngush between ghosts and abandoned obects. In Guler s method [5], a specfc processng layer s used for detecton of obects that become statonary. The detected statonary foreground obects are mantaned n the specfc layer, and the moton hstory of the mmoble obects are recorded. However, the method needs moton hstory of the obect whch s not avalable f the obect belonged to background from the start of the vdeo sequence. Lu et al [10] proposed to recover the real background by usng a revsed mage npantng method. Thus they could dfferentate between ghost and abandoned obects: The obect wll be declared as ghost f there s no dfference between an ncomng frame and the real background mage, otherwse t s an abandoned tem. However, mage npantng s computatonal expensve for real tme processng, and ther method has not been evaluated quanttatvely. Desurmont et al [12] proposed a general framework for vdeo survellance systems and calculated the edge mean square gradent (EMSG) for each blob. They fnd out ghost blobs based on the assumpton that n the background the gradent s typcally low. Therefore, f the EMSG for a blob s very small (lower than a threshold), the blob wll be classfed as ghost, otherwse t wll be classfed as real obect. However, ths s not always true, e.g. a cluttered scene may result n a hgh textured background wth hgh EMSG. In ths paper, we propose usng edge comparson between extracted regons from multple mages to fnd out whether the detected foreground belongs to the current mage or not, and furthermore to dentfy detected foreground obects as real obects (whether movng or statonary) or ghosts. The method s dscussed n detal n secton 3 and evaluated n secton Methodology We have developed a framework that we use to detect ghosts durng trackng and furthermore to elmnate them. Ghosts manly appear n two cases: In the frst case, when a movng obect becomes statonary, t wll be adapted (merged) nto the background, and then, when t starts to move agan some tme later, there wll be a ghost left behnd. In the second case, an exstng obect that belongs to the background starts to move (e.g parked vehcle) and wll also cause a ghost problem. In the frst case, we may have the hstory of obect traectory, but n the second case, we may not have any prevous nformaton about the obect. So t s preferable to make the tracker able to recognse ghosts n tme wthout addtonal hstory nformaton rather than analyse the hstory of the obect. We consder that a basc tracker s avalable wth the followng assumptons: Trackng s performed at pxel level and no calbraton nformaton s needed. Trackng s performed on gray-level mage sequences (although extenson of the method to colour mage sequences s relatvely smple). The basc tracker contans a moton detecton module that can provde a dfference map whch

4 s the dfference of the values of each pxel between the background mage and the ncomng frame The basc tracker s able to estmate the boundng box and the velocty of movng obects at each frame. Our proposal of ghost detecton s descrbed as follows: For every k frames (k=12 n our mplementaton), we check the speed of each track. If the track s speed s below a threshold T v (e.g. T v = 0.1 pxel/frame), we proceed further to the ghost detecton. We apply Gaussan smoothng and Canny edge detecton n regons of the boundng boxes (bbox) on both the dfference map and the ncomng frame. and we get two edge maps for every track at the same tme (see Fgure 1). D D D D D D { x, y ),( x, y )...( x, y )} D = (1), ( Fgure 2 edges of dfference map and another set of edge ponts from the current mage for track (see Fgure 3):, C C C C C C { x, y ),( x, y )...( x, y )} C = (2) ( n m n m Fgure 3 edges of current mage (a) (b) Fgure 1 KND2-JULY-EAST-S2.av sequence frame 5880 a) a car starts to move out of ts parked area along the red arrow b) the detected movng area by background subtracton c) canny edge detecton of (a) n track regons d) canny edge detecton of (b) n track regons Note that red and yellow boxes ndcate the regons of two dfferent tracks Then, we compare the dfference between the edges from the two maps (ncomng frame and the frame dfference map) n the area of the boundng boxes (bbox) for each track respectvely. (e.g. n fgure 1 we compare the rght yellow bbox wth the left yellow bbox snce they are of the same track). We get a score for edge smlarty (S) between the edges for each track n each frame, and then set up a threshold for makng the decson of whether they are the same or dfferent (usually, the score s from 0% (do not match at all) to 100% (perfectly matched). Suppose for track, frame, we get a set of edge ponts from the dfference map (see Fgure 2): We calculate the score of S between two edge sets as follows: S sze ( D, I C, ) ( D U C ), = S, sze,, [0 ~1] (3) Normally, smlarty scores are very hgh when the tracks are from real exstng obects (typcally much hgher than 30%). If S, s larger than a threshold T (set to 30% accordng to experments), a ghost condton has been detected for track. We accumulate evdence over tme usng a counter G whch ndcates how many tmes the track has been detected as ghost. The ntal value for G s zero. If S, > T, G G + 1 (4) As long as G > 0, we label the track as ghost, the track s prevented from beng vald or real movng obect and does not trgger a false alarm. After several tmes of ghost suspcon (usng the crteron of equaton 4), f G s larger than a predefned number E (set to 5 n our work), then we confrm the track as ghost and vansh the track (.e. feedback the confrmaton to the moton detector so that the moton detector

5 enforces the ghost blobs to become part of the background). In order to makes sure that we do not detect a real obect as ghost and vansh t by mstake, we also take the moton of the track nto O O consderaton. Let ( x, y ) be the coordnates of the centrod of track where t appears frstly. x, y ) be the coordnates of the centrod of (,, track n current frame. W, H ) be the ( average wdth and heght of track durng ts lfe tme. Then, O 2 O 2 ( x, x ) + ( y, y ) >= 1 V 2 2 = W G G 1 + H G > 0 (5) If the track has moved a long enough dstance from where t appeared, then t s not be dentfed as ghost. The counter G wll be decremented to zero gradually. 4. Results We demonstrate the practcal value of the proposed method by usng an expermental ndustral tracker from BARCO [11] called the basc tracker here and then the same tracker wth the addton of our ghost detecton algorthm. We run the two trackers on a standard PC wth Pentum IV 3.4 GHz processor wth 1GB RAM. The trackng and ghost detecton were carred out on 4 vdeo sequences wth 6 ghost events and 2 abandoned tems. We managed to remove all the 6 ghost obects, and avod all false alarms caused by ghosts. (see Fgures 5,6,7) (a) ( b) Our ghost detecton algorthm s summarzed n Fgure 4: (e) (f) Fgure 5 PETS2001 PetsD1TeC1.av sequence s 2686 frames (00:01:29) long and depcts 4 persons, 2 groups of persons and 3 vehcles. Its man challenge s the multple obect ntersectons and ghosts. a) frame 2360 b) frame 2635 c) basc tracker at frame 2360 d) basc tracker at frame 2635 e) tracker wth ghost detecton at frame 2360 f) tracker wth ghost detecton at frame 2635 Fgure 4 Framework of ghost detecton algorthm Fgure 5 presents vsual results from the basc moton tracker as well as from the tracker wth ghost detecton algorthm usng a PETS2001 sequence. Red bboxes represent the vald tracks generated by the tracker, whle blue bboxes represent nvald tracks (ghost tracks). The coloured patches nsde each bbox present the foreground pxels whch are detected as movng pxels (the colours of each patch are randomly assgned by the tracker). In Fgures 5c and 5d, the whte mnbus moved away from where t had stopped,

6 leavng a ghost (red bbox wth green patch n Fgure 5d) behnd. In Fgures 5e and 5f, by usng the proposed ghost detecton method, we successfully dentfed the ghost obect (blue boundng bbox wth green patch wth the text ghost on t). Fgure 6 presents results from the KND2-JULYEAST-S2.av sequence. In Fgure 6c frame 5856 a car s about to start. After a whle, n frame 5880 (Fgure 6d), the car has left ts parked area and a ghost track has been formed (red box wth yellow patch). Fgures 6e and 6f present the tracker wth ghost detecton: n frame 5880 (Fgure 6f), the car left ts parked area, however, the ghost patch has been successfully dentfed (blue bbox wth yellow patch) and furthermore prevented from becomng vald and elmnated after a few frames. Fgures 7c and 7d present the results by the basc moton tracker and Fgures 7e and 7f are produced by Ghost detecton tracker. As we can see Fgure7d, after the vehcle (frame 9556, red bbox wth lght blue patch) has left, a ghost track was left behnd (red bbox wth purple patch). Wth ghost detecton n Fgures 7e and 7f, the ghost obect (blue bbox wth blue patch) has been successfully detected. (a) (e) (a) (b) (e) (f) Fgure 7 KND2-JULY-GATE1-S2.av sequence s frames (00:12:32) long and depcts 29 obects. The man challenges are quck llumnaton changes and 3 ghosts a) frame 9496 b) frame 9556 c) basc tracker at frame 9496 d) basc tracker at frame 9556 e) tracker wth ghost detecton at frame 9496 f) tracker wth ghost detecton at frame 9556 (b) (a) (b) (e) (f) (f) Fgure 6 KND2-JULY-EAST-S2.av sequence s frames (00:12:55) long and depcts 23 obects. The man challenges are quck llumnaton changes and 2 ghosts a) frame 5856 b) frame 5880 c) basc tracker at frame 5856 d) basc tracker at frame 5880 e) tracker wth ghost detecton at frame 5856 f) tracker wth ghost detecton at frame 5880 Fgure 8 KND2-JULY-EAST-S3.av sequence s frames (00:10:04) long and depcts 29 obects. The man challenges are quck llumnaton changes, abandoned bags. a) frame b) frame c) basc tracker at frame13460 d) basc tracker at frame e) tracker wth ghost detecton at frame f) tracker wth ghost detecton at frame 13510

7 Fgures 8e and 8f present the result from the Ghost detecton tracker. A box (frame 13510, red bbox wth green patch) was dropped from the car, and our algorthm can stll successfully detect the abandoned tem wthout mss classfyng as a ghost. From the fgures shown above, we can see that ghost detecton s effectve enough to detect ghost obects, to prevent the tracks from becomng vald, and furthermore to avod false alarms. At the same tme, the algorthm can dfferentate between ghost obects and abandoned tems. We expermented wth the frequency of ghost detecton by usng dfferent values of k (e.g. k = 1, 2, 5, 10, 12, 15, 20 ) and testng our algorthm. There s no effect on the performance as long as k s under 20. Our decson to set k to 12 ensures fast and relable ghost detecton. We also present some quanttatve evaluaton results (Tables 1-3) based on the performance evaluaton framework descrbed n detal n [6]. The mprovement s manly reflected by the metrc: False alarm tracks (FATs). In the PETS2001 sequence (table 1), there are two false alarms caused by ghost obects n the results of the basc tracker. In the results of ghost detecton tracker, the 2 false alarms caused by ghosts have been avoded. Smlarly, n KND2-JULY-EAST-S2 sequence (table 2), we had 2 cases of false alarms caused by ghosts, but the tracker wth ghost detecton avoded these 2 FATs. Hence, the number of FAT s reduced from 5 to 3. Smlarly, n KND2-JULY-GATE1-S2 sequence (table 3), we had 3 cases of FATs caused by ghosts, but the tracker wth ghost detecton avoded these 3 FATs. Hence, the number of FAT s reduced from 14 to 11. The speeds of the basc tracker and ghost detecton tracker are fast enough to operate n real-tme, although there s about 10% to 15% decrease n speed when addng ghost detecton nto the basc tracker. In addton, except false alarm track, other metrcs such as correct detected track, track detecton falure, track fragmentaton, average latency, dstance error reman more or less the same for both basc tracker and ghost detecton tracker, whch means that the ghost detecton does not cause any undesrable sde-effects on trackng the obects. Accordng to the overall evaluaton results, the ghost detecton algorthm can effectvely elmnate ghost obects and brngs no sgnfcant sde-effects caused by the ghost detecton algorthm to the survellance system. Table 1 Performance evaluaton for Pets2001.av PetsD1TeC1.av Basc tracker Ghost detecton Total num of frames Speed 77.4 f/s 63 f/s GT Tracks 9 9 System Tracks CorrectDetectedTrack 9 9 FalseAlarmTrack 3 1 TrackDetectonFalure 0 0 Trackfragmentaton 3 4 ID Change 5 6 AverageLatency AverageOverlap OverlapDevaton DstanceError DstanceErrorDevaton AverageCompleteness Table 2 Performance evaluaton for KND2-JULY-EAST-S2.av KND2-JULY-EAST- S2.av Basc tracker Ghost detecton Total num of frames Speed 28.9 f/s 25.5 f/s GT Tracks System Tracks CorrectDetectedTrack FalseAlarmTrack 5 3 TrackDetectonFalure 4 3 Trackfragmentaton 1 2 ID Change 1 0 AverageLatency AverageOverlap OverlapDevaton DstanceError DstanceErrorDevaton AverageCompleteness

8 Table 3 Performance evaluaton for KND2-JULY-GATE1-S2.av KND2-JULY-GATE1-S2.av Basc Ghost tracker detecton Total num of frames Speed 25.2 f/s 21.6 f/s GT Tracks System Tracks CorrectDetectedTrack FalseAlarmTrack TrackDetectonFalure 3 4 Trackfragmentaton 5 6 ID Change 0 0 AverageLatency AverageOverlap OverlapDevaton DstanceError DstanceErrorDevaton AverageCompleteness Concluson We have presented an effcent ghost detecton and removal algorthm that can be appled to any moton detecton and trackng system that produces a dfference map per frame and boundng boxes for foreground obects. Accordng to the vsual results and performance evaluaton results, we can effectvely avod false alarms caused by ghost obects whle stll dfferentatng between ghost obects and abandoned obects. The method works well n several scenaros: PETS2001 and another two car park scenaros. The Ghost removal verson of basc tracker runs almost as fast as the orgnal and do not have any negatve sdeeffect on detecton and trackng. However, our method s lmted by the performance of the tracker, e.g. f the tracker can not recognse ndvdual obects n crowd scenes, our method wll not be able to detect ghosts ether. In future work, we want to test our algorthm n vdeos wth textured background to check f that would cause confuson between the edges of the pattern and the obect. 6. Acknowledgements The authors would lke to acknowledge fnancal support from BARCO Vew, a maor multnatonal manufacturer of CCTV dsplay systems, whose headquarters are n Belgum. They are also grateful to the Engneerng and Physcal Scences Research Councl's REASON proect for provdng access to data. 7. References [1] W. Hu, T. Tan, L. Wang, and S. Maybank. A survey on vsual survellance of obect moton and behavors. IEEE Trans. on Systems, Man, and Cybernetcs - Part C, 34(3): , August 2004 [2] C. Stauffer, W.E.L. Grmson, Adaptve background mxture models for real-tme trackng, n: Computer Vson and Pattern Recognton, vol 2, p.2246, Santa Barbara, CA, June, 1998 [3] R. Cucchara, M. Pccard, and A. Prat, "Detectng movng obects, ghosts, and shadows n vdeo streams," IEEE Transactons on Pattern Analyss and Machne Intellgence 25, pp , Oct 2003 [4] Sen-Chng S. Cheung, Chandrka Kamath, Robust Background Subtracton wth Fore-ground Valdaton for Urban Traffc Vdeo, EURASIP Journal on Appled Sgnal Processng, pp , [5] Sadye Guler, Jason A. Slversten, Ian H. Pushee, "Statonary Obects n Multple Obect Trackng", IEEE Internatonal Conference on AVSS, pp , 2007 [6] Fe Yn, D. Makrs, S.A. Velastn, "Performance Evaluaton of Obect Trackng Algorthms", 10th IEEE Internatonal Workshop on Performance Evaluaton of Trackng and Survellance (PETS2007), October, Ro de Janero, Brazl, 2007 [7] OpenCV Computer Vson Lbrary opencv/ndex.htm, [Last accessed: August 2007] [8] Gude to Authorng Meda Ground Truth wth VPERGT, [Last accessed: July 2007] [9] D.S. Lee, Effectve Gaussan mxture learnng for vdeo background substracton, IEEE Transactons on Pattern Analyss and Machne Intellgence 27 (5) pp , 2005 [10] Sun Lu Jan Zhang Feng, D. "An effcent method for detectng ghost and left obects n survellance vdeo",

9 IEEE Conference on Advanced Vdeo and Sgnal Based Survellance, pp , London, Sept [11] Barco Dsplay and vsualzaton solutons, [Last accessed: May 2008] [12] X. Desurmont, C. Chaudy, A. Bastde, C. Parsot, J.F. Delagle, B. Macq, Image analyss archtectures and technques for ntellgent systems, IEE Proceedngs on Vson, Image and Sgnal Processng, Specal ssue on Intellgent Dstrbuted Survellance Systems, pp , Aprl 2005.