An abandoned object detection system based on dual background segmentation

Transcription

1 2009 Advaced Video ad Sigal Based Surveillace A abadoed object detectio system based o dual backgroud segmetatio A. Sigh, S. Sawa, M. Hamadlu Departmet of Electrical Egieerig I.I.T. Delhi Delhi, Idia abhiavkumar.sigh@mail2.iitd.ac.i V.K. Madasu, B.C. Lovell School of ITEE NICTA ad The Uiversity of Queeslad Brisbae, Australia v.madasu@uq.edu.au Abstract A abadoed object detectio system is preseted ad evaluated usig bechmark datasets. The detectio is based o a simple mathematical model ad works efficietly at QVGA resolutio at which most CCTV cameras operate. The pre-processig ivolves a dual-time backgroud subtractio algorithm which dyamically updates two sets of backgroud, oe after a very short iterval (less tha half a secod) ad the other after a relatively loger duratio. The framework of the proposed algorithm is based o the Approximate Media model. A algorithm for trackig of abadoed objects eve uder occlusio is also proposed. Results show that the system is robust to variatios i lightig coditios ad the umber of people i the scee. I additio, the system is simple ad computatioally less itesive as it avoids the use of expesive filters while achievig better detectio results. Keywords- video surveillace, left baggage detectio, backgroud segmetatio, trackig I. INTRODUCTION Recet years have see a stark rise i terrorist attacks o crowded public places such as airports, trai statios ad subways, ightclubs, shoppig malls, markets, etc. May surveillace tools have bee employed i the fight agaist terror. Although video surveillace systems have bee i operatio for the past two decades, the aalysis of the CCTV footage has seldom vetured out of the hads of huma operators. Recet studies [1-3] have brought ito fore the limits to huma effectiveess i aalyzig ad processig crowded scees, particularly i video surveillace systems cosistig of multiple cameras. The advet of smart cameras with higher processig capabilities has ow made it possible to desig systems which ca possibly detect suspicious behaviors (i geeral) ad abadoed objects (i particular). A umber of algorithms [5, 7, 8] have bee suggested i the recet past to deal with the problem of abadoed-object-detectio. Due to their depedece o complex probabilistic mathematics, most of these algorithms have failed to perform satisfactorily i real time scearios. I additio, the other difficulty of detectig a abadoed object uder occlusio adds to the overall complexity. Some proposed algorithms [4-5] have dealt with partial occlusio (by movig people) but complete or prologed occlusio (by aother object) has ot yet bee tackled. Furthermore, the backgroud subtractio methods employed i the above methods are either computatioally itesive or lack dyamically updatig features. I this paper, we preset a abadoed object detectio system based o a simplistic ad ituitive mathematical model which works efficietly at QVGA resolutio which is the idustry stadard for most CCTV cameras. The proposed system cosists of a ovel selfadaptive dual backgroud subtractio techique based o the Approximate Media model [6] framework. Algorithms for trackig abadoed objects with or without occlusio are also icluded. A. System Overview The overall system (see Fig. 1) is modular i ature ad cosists of four disparate blocks with each block actig as a discreet processig uit makig it easy to modify ay block, provided the iput ad output data types remai compatible with the coectig blocks. The 4 blocks are: Data extractio ad coversio uit; Backgroud subtractio module; Still object trackig ad occlusio detectio block ad Alarm raisig ad display of result uit A live video stream is iitially segmeted ito idividual images from which a regio of iterest is extracted ad coverted to 3D itesity matrices (height * width * itesity value of each pixel). These matrices are the fed as iput to the Backgroud Subtractio module. II. BACKGROUND SEGMENTATION Numerous backgroud subtractio methods are available i the literature. The most popular beig the oes based o Gaussia mixture models, the first of which was proposed by Friedma ad Russell [12] ad the modified by several authors [13-14] to suit their specific eeds. I this work, a ew backgroud subtractio techique based o the Approximate Media algorithm is developed. This method is adaptive, dyamic, o-probabilistic ad ituitive i ature. Like the majority of other methods (for ex. [6]), we also use pixel color/itesity iformatio for backgroud processig. But istead of havig oe referece frame, we maitai two differet referece frames for self adaptability resultig i less computatio due to o-iclusio of ay complex mathematics. Movig crowd/objects, lightig chages ad uecessary details like shadows, reflectios o floors ad walls are filtered off efficietly with oly statioary objects remaiig i the scee, thus leavig us with the prime motive of detectig abadoed objects. Moreover, havig two backgrouds has a added advatage that the user ca adjust the time iterval betwee the update of referece backgroud frames to suit differet eeds ad eviromets. 1

2 Figure 1: Flowchart of the overall system operatio A. Algorithm The proposed algorithm to separate backgroud ad foregroud i the icomig image is based o the Approximate Media Model [6]. However, our techique requires two referece backgroud images, amely, Curret Backgroud ad Buffered Backgroud. This techique of storig two backgrouds ca be cosidered as a dual backgroud method. Oe of the iterestig features of this techique is that both the backgrouds are updated dyamically. The first oe is updated frequetly while the secod oe has a slower update rate. The first frame of the icomig video is iitialized as Curret Backgroud. Subsequetly, the itesity of each pixel of this curret backgroud is compared with the correspodig pixel of the ext frame (after every 0.4 secods). If it is less, the the itesity of that pixel of curret backgroud is icremeted by oe uit, otherwise it is decremeted by oe uit. I case of equality, the pixel itesities remai uchaged. This way, eve if the foregroud is chagig at a fast pace, it will ot affect the backgroud but if the foregroud is statioary, it gradually merges ito the backgroud. Sice we are iterested i all those objects which are statioary for a log period of time (ad thus have gradually merged ito the backgroud), we maitai aother set of backgroud images called Buffered Backgroud. Here, all those pixels which do ot belog to the prospective abadoed objects set are made equal to that of Curret Backgroud. This is doe at a iterval of every 20 secods. Differece of the two backgrouds is represeted as a biary image with the white portio represetig foregroud (blobs). B. Illustratio The Dual Backgroud techique is illustrated i Fig. 2. Frame 2A shows all the objects that are detected. Frame 2D shows the curret backgroud, which is updated every half a secod. The loger a perso or object stays i A, stroger its impressio is imprited o D. Frame 2E shows the buffered backgroud, which is updated every half a miute, ad does ot cotai abadoed object(s). Hece the differece of 2D ad 2E gives the positio of abadoed objects, which is highlighted i frame 2C, after the object has bee left abadoed for a log eough time. Frame 2B shows the foregroud which comes from differece of 2A ad 2E. Figure 2: Dual Backgroud Segmetatio III. OBJECT DETECTION I this module, we divide the biary image from the previous uit ito a umber of legitimate blobs (rectagular regios eclosig cotiuous regios of foregroud). Oce the blobs ad their various properties like area, cetroid positio etc. have bee geerated, we apply the trackig algorithms. A. Mathematical Model Let us suppose that after blob aalysis we get N umber of blobs, each with its eclosig regio R (t, l, h, w), its area A ad cetroid C (i,j). Figure 3: A typical blob where, t is the top positio value of pixel; l is the left positio value of pixel; h gives the height of the blob; ad w is the width of the blob; ad 1 N Let T be the set of tracked blobs such that, [ B : B = { R ( t, l, h, w), A, C ( i, j), t, m }] T = 1 M 2

3 where, M is the umber of tracked blobs; t is the umber of frames for which the blob has bee tracked ad m is the umber of cosecutive frames for which the blob beig tracked previously has bee ot detected Let us call the preset set of blobs which we get after aalysis of the preset frame as P ad its objects as b, which are N i umber. The the set of blobs is: [ b : b = { r ( t, l, h, w), a, c ( i, j) }] T = 1 N B. Blob detectio The blob aalysis takes as a iput a biary image, applies a algorithm similar to the oe described i [11] ad returs various properties of the detected blobs like boudig box, area, cetroid positio etc. A simplified versio of the algorithm is as follows: 1. Create a regio couter. 2. Sca the image from left to right ad from top to bottom. 3. For every pixel check the orth ad west pixel (4- coectivity) or the ortheast, orth, orthwest, ad west pixel (8-coectivity) for a itesity value of 1 i the biary image (termed as criterio of blob aalysis) 4. If oe of the eighbors fit the criterio the assig to regio value of the regio couter. Icremet regio couter. 5. If oly oe eighbor fits the criterio, assig pixel to that regio. 6. If multiple eighbors match ad are all members of the same regio, assig pixel to their regio. 7. If multiple eighbors match ad are members of differet regios, assig pixel to oe of the regios ad idicate that all of these regios are the equivalet. 8. Sca image agai, assigig all equivalet regios the same regio value. C. Trackig The ext process i object detectio is trackig the differet blobs so as to fid which blobs correspod to abadoed objects. The first step i this process is to create a set, Track, whose elemets have three variables: blob- Properties, hitcout ad misscout. The ext step is to aalyze the icomig image for all the blobs. If the area chage ad the cetroid positio chage, as compared to ay of the elemets of the set Track are below a threshold value, we icremet hitcout ad reiitialize misscout with a zero; otherwise we create a ew elemet i the Track-set, iitializig the blob-properties variable with the properties of icomig blob ad hitcout ad misscout are iitialized to zero. We the ru a loop through all the elemets of the set. If the hitcout goes above a user defied threshold value, a alarm is triggered. If the misscout goes above a threshold, we delete the elemet from the set. These two steps are repeated util there are o icomig images. Pseudo Code for Trackig Take area, cetroid, boudig boxes (bbox) ad total umber of blobs () as iput from Blob Aalysis block. Let Track=empty set of vectors of type t where t=(area, cetroid, bbox, hitcout, misscout,active,occluded) m= Track.Size For i=1 to c=0 For j=1 to m If (percetage backgroud i Track[j].area<50) The Track[j].occluded=true Ed If ( area[i] Track[j].area /area[i] <.05 ad cetroid[i] Track[j].cetroid /cetroid[i]<.05) The Track[j].active =true, c=1, break from loop Ed Ed If c=0 The k=track.size++, Track[k].area = area[i]; Track[k].cetroid = cetroid[i]; Track[k].bbox = bbox[i]; Track[k].hitcout = 1; Track[k]. active = true; Ed Ed m= Track.Size For j=1 to m If (Track[j].active==true) The Track[j].hitcout=Track[j].hitcout+1; Track[j].miscout=0; If (Track[j].hitcout > 4) Do t update pixels of Track[j].bbox i buffered backgroud Ed If (Track[j].hitcout > 40) The raise alarm for Track[j] Ed If (Track[j].active==false ad misscout >3) The delete Track[j] Ed Ed Update the buffered backgroud D. Occlusio Detectio ad Trackig A tracked blob is cosidered to be occluded if its major regio (say 80 %) is covered by foregroud ad it should cotiue to be tracked if either it is occluded or its area ad cetroid is matched with ay of the blobs of set P. A alarm is raised if t > threshold. The blob is removed from T if m >3. This idea is similar to the method used i [7] ad [8] for occlusio detectio, but istead of 3

4 keepig track of two differet foregrouds, we propose the followig modificatio. Let us assume that a particular portio of the frame cotaiig the blob which is beig tracked (i.e. preset i the Track set of blobs) is ow occluded. Due to this occlusio, the blob sigifyig that particular object wo t be icluded i the preset set of Track. Mathematically, t = t + 1 ad m = 0 if R A > 0.8 ( i, j) c ( i, j) C k or A 1 k m else m = m < 0.1ad A a A k < 0.1 Followig are the possibilities i the ew frame of the blob that was beig tracked up to previous frame:- Object is removed from the locatio. I this case, the blob area represetig the object should cotai backgroud pixels. There may be a ew object at the same locatio. There is a ew object which completely or partially occludes the old object. A exceptio to the above cases is whe a tracked object is removed while beig occluded or aother object of similar size is placed i camera s lie of view. To deal with occlusio we have added the followig two steps to the trackig algorithm: Step 1: Calculate the umber of pixels of buffered backgroud which are same as that of curret backgroud for that elemet of set Track which has suddely stopped beig tracked (due to occlusio, or removal from scee). If it s below a threshold value, say 50 percet, ad the hitcout is above a threshold value (makig sure the blob has bee tracked log eough), we label this elemet of the set Track as occluded. Step 2: Go to step 2 of the trackig algorithm. A blob labeled occluded remais i the Track set; i.e. its hitcout is icremeted ad misscout is reiitialized. Rest of the trackig algorithm remais same. E. Alarm ad Display We use the Raise-alarm flag from previous uits ad highlight that part of the video for which the alarm has bee raised. We also display the biary-image (without backgroud) video so that the operator ca fie tue the value of D for shadow ad reflectio subtractio. IV. RESULTS AND ANALYSIS All algorithms described above are applied o stadard bechmark datasets for obtaiig experimetal results. A. Datasets The experimets were computed o Itel Core 2 Duo processor with 1 GB RAM. Every video was scaled dow to QVGA resolutio (320x240) ad 10 fps frame rate before further aalysis. Five differet situatios ivolvig differet crowd desities ad various sizes of objects were selected from the PETS 2006 dataset ad two differet situatios were aalyzed from AVSS 2007 i-lids Abadoed Baggage Traiig dataset, oe with a low crowd desity while the other had a medium crowd desity. Both datasets ivolved surveillace feed from metro statios, sapshots of which are give below. Figure 4: Sapshot of PETS & AVSS datasets The PETS dataset was recorded at a metro statio ad each scee ivolved a perso with a bag who loiters for a while before leavig the bag uatteded. The details of videos which were aalyzed are as follows: Dataset S1 (Take 1-C) : 1 perso, 1 luggage item, difficulty 1/5 Dataset S2 (Take 3-C) : 2 people, 1 luggage item, difficulty 3/5 Dataset S5 (Take 1-G) : 1 perso, 1 luggage item, difficulty 2/5 Dataset S6 (Take 3-H) : 2 people, 1 luggage item, difficulty 3/5 Although groud truth for all the videos was available, it was ot utilized as our model dyamically updates the backgroud ad is therefore ot scee specific. Each scee of PETS dataset was recorded from four differet agles, resultig i 20 differet videos from PETS dataset ad two videos from AVSS dataset. B. Setup The miimum time for which the object remais statioary ad abadoed, before the alarm is raised, ca be varied ad should be ideally 2 to 3 miutes, but sice the total time for which a video i our dataset rus is less tha 3 miutes, we have kept the miimum time as 30 secods or 300 frames fps). Oce the object starts gettig tracked, aythig which occludes it is take care by our algorithm. Hece the trackig time remais costat at aroud 30 secods, i ay situatio for ay statioary object before the alarm is raised. 4

5 Time take for the alarm to be stopped after the object is removed from the site, depeds o how soo the impressio of the object o the curret backgroud disappears ad hece the differece betwee the itesities of buffered backgroud ad curret backgroud pixels becomes isigificat. This time depeds o curretbackgroud-update-rate ad how differet the object is from the thigs which are behid it i the scee. Sice this time depeds o the object ad scee textures, it varies from five to te secods. Figure 5: Abadoed object detectio process False alarm is raised or the object is ot detected i oly three cases. Each case ad the reaso for failure are give as follows: Object gets camouflaged by the backgroud ad fails detectio ( o ): 2 videos Object is correctly detected but a very still perso also gets detected ( p ): 1 video Object is correctly detected, o perso is detected but a uwated blob is also icorrectly detected as a abadoed object ( b ): 3 videos Figure 6: Occluded abadoed object detectio Although, the system performace ca be measured via commo metrics such as the ROC curves, we have defied the success rate of our algorithm usig a score which is equivalet to the ratio of the total umber of videos aalyzed to the umber of successfully aalyzed videos. Mathematically, S = ( ) o p We defie the parameter for measurig successful videos i the above fashio because our algorithm completely fails if o object is detected, it partially (50% failure) succeeds if object is detected but alog with it a still perso is also detected, ad it fails very little (25% failure) if the object is correctly detected, o perso is detected but a uwated blob is icorrectly detected as a abadoed object. The overall results are illustrated i Table 1. A total of 22 videos were aalyzed from both the PETS ad AVSS datasets. There were two complete failures as the system was uable to detect the abadoed objects i the video frames. There was oe partial failure i which a still perso was also detected as a abadoed object. I additio, there were three videos i which uwated blobs were sometimes classified as abadoed objects. By applyig the formula for the performace score as explaied above, we achieved a success rate of 85.2%. The results are comparable to the methods preseted i the literature ad are sigificat because of lesser computatio ad faster processig. C. Discussio of Results Based o the results ad aalysis, we ca coclude that low to medium desity crowd has o effect o processig speed or accuracy of the model. I a high desity sceario, there is a possibility that the object is proe to be hidde from camera view for most of the time or i other words it is camouflaged by the backgroud leadig to a failure i detectio. Aother achievemet of this model was that differece i lightig coditios had almost egligible effect o the operatig performace. This ca be attributed to the use of Dyamic Backgroud techique. The system will thus work perfectly i a ope eviromet (uder sulight) too. Additioally, shadow effects ad reflectio of light from bright objects do ot pose ay problems. The algorithm works i real time at QVGA resolutio ad 10 fps frame rate, ad has a high success rate of 85%. Eve decreasig the frame rate to as low as 3 or 4 fps has isigificat effects o the accuracy of the model. Processig speed is iversely proportioal to the square of resolutio of the video for a give aspect ratio ad also iversely proportioal to the frame rate of the video. The model ca detect ay umber of abadoed objects i a give video sequece. Although speed is compromised with a icrease i the umber of objects to be detected but such cases are rare to ecouter. Some oticeable limitatios of the model are that a completely immovable perso gets mistake for a abadoed object. Also, the object must be i clear view of the camera for at least five secods, otherwise it gets merged ito the backgroud. b 5

6 V. CONCLUSIONS This paper preseted a abadoed object detectio system based o a dual backgroud segmetatio scheme. The backgroud segmetatio is adaptive i ature ad based o the Approximate Media Model. It cosists of two types of referece backgrouds, Curret ad Buffered backgroud, each with a differet time iterval. Blob aalysis is doe o the segmeted backgroud ad a dyamic trackig algorithm is devised for trackig the blobs eve uder occlusio. Detectio results show that the system is robust to variatios i lightig coditios ad the umber of people i the scee. I additio, the system is simple ad computatioally less itesive as it avoids the use of expesive filters while achievig better detectio results. REFERENCES [1] C. Sears ad Z. Pylyshy, Multiple Object Trackig ad Attetioal Processig, Caadia Joural of Experimetal Psychology, vol. 54, 2000, pp [2] P. Cavaaugh ad G. Alvarez, Trackig Multiple Targets with Multifocal Attetio, Treds i Cogitive Scieces, vol. 9(7), 2005, pp [3] M. Bhargava, C-C. Che, M.S. Ryoo, ad J.K. Aggarwal, Detectio of Abadoed Objects i Crowded Eviromets, i Proceedigs of IEEE Coferece o Advaced Video ad Sigal Based Surveillace, 2007, pp [4] G.L. Foresti, L. Marcearo, ad C.S. Regazzoi, Automatic Detectio ad Idexig of Video-Evet Shots for Surveillace Applicatios, vol. 4, 2002, pp [5] R. Mathew, Z. Yu ad J. Zhag, Detectig New Stable Objects i Surveillace Video i Proceedigs of the IEEE 7 th Workshop o Multimedia Sigal Processig, 2005, pp [6] N.J.B. McFarlae ad C.P. Schofield, Segmetatio ad trackig of piglets i images, Machie Visio ad Applicatios, vol. 8, 1995, pp [7] N. Bird, S. Atev, N. Caramelli, R. Marti, O. Masoud ad N. Papaikolopoulos, Real Time, Olie Detectio of Abadoed Objects i Public Areas, i Proceedigs of IEEE Iteratioal Coferece o Robotics ad Automatio, 2006, pp [8] F. Porikli, Y. Ivaov, ad T. Haga, Robust Abadoed Object Detectio Usig Dual Foregrouds, Eurasip Joural o Advaces i Sigal Processig, vol. 2008, [9] J.O. Aguilar, Omidirectioal Visio Trackig with Particle Filter, i Proceedigs of 18 th Iteratioal Coferece o Patter Recogitio, vol. 3, 2006, pp [10] I. Haritaoglu, D. Harwood, L. S. David, W4: Real-time surveillace of people ad their activities, IEEE Tras. Patter Aal. Mach. Itelligece, vol. 22 (8), 2000, pp [11] F. Chag, C-J. Che, ad C-J. Lu, A liear time Compoet- Labelig Algorithm usig Cotour Tracig Techique, Computer Visio ad Image Uderstadig, vol. 93, 2004, pp [12] N. Friedma ad S. Russell, Image segmetatio i video sequeces: a probabilistic approach, i Proceedigs of 13 th Aual Coferece o Ucertaity i Artificial Itelligece, 1997, pp [13] C. Stauffer, ad W. Grimso, Adaptive backgroud models for realtime trackig, i Proceedigs of IEEE Coferece o Computer Visio ad Patter Recogitio, vol. 2, pp , [14] C.R. Wre, A. Azarbayejai, T. Darrell, ad A.P. Petlad, Pfider:Real-time trackig of the huma body, IEEE Tras. Patter Aal. Mach. Itell., vol. 19, o. 7, pp , Jul TABLE I. SUMMARY OF RESULTS Video Nomeclature Time Take for Object to be detected after beig left abadoed (i secods) Time Take for alarm to be removed after the object is removed from the site (i secods) Agle1 Agle2 Agle3 Agle4 Agle1 Agle2 Agle3 Agle4 S1 (Take 1-C) S2 (Take 3-C) S5 (Take 1-G) Failed Failed Failed Failed 9 10 S6 (Take 3-H) S7 (Take 6-B) AVSS (low desity) 28 _ 6 _ AVSS (medium desity) 30 _ 7 _ 6