Performance Evaluation of Surveillance Systems Under Varying Conditions

Transcription

1 Performance Evaluaton of Survellance Systems Under Varyng Condtons Lsa M. Brown, Andrew W. Senor, Yng-l Tan, Jonathan Connell, Arun Hampapur, Chao-Fe Shu, Hans Merkl, Max Lu IBM T.J. Watson Research Center, Abstract Effectvely evaluatng the performance of movng object detecton and trackng algorthms s an mportant step towards attanng robust dgtal vdeo survellance systems wth suffcent accuracy for practcal applcatons. As systems become more complex and acheve greater robustness, the ablty to quanttatvely assess performance s needed n order to contnuously mprove performance. In ths paper, we refne the methods used to estmate performance and use these methods to measure the performance of our system under several dfferent condtons ncludng: ndoor/outdoor, dfferent weather condtons (precptaton, wnd, and brghtness), dfferent cameras/vewponts, and as a standard benchmark, the PETS 2001 datasets. To test the extensblty/valdty of our results, we have also evaluated our system on four longer data sets (20-30 mn each) from four dfferent cameras. We evaluate the performance of the background subtracton alone and wth a smple trackng system usng two dfferent sets of metrcs. Vsualzaton of the performance results has proven crtcal for understandng the weaknesses of the system. 1. Introducton In practce, dgtal vdeo survellance needs to operate around the clock, as the weather vares, the seasons change, and the daly events unfold. Performance evaluaton of automatc survellance systems s stll typcally lmted to short sequences. Unfortunately, these sequences and ther annotaton are often avalable only to the researchers who created them. There s a need to create benchmark datasets avalable to all researchers and to agree on standardzed performance metrcs for ther evaluaton. Furthermore, we need to understand the extensblty of these results to the long-term operaton of systems for real-world applcatons. In an effort to begn to understand the ssues nvolved n runnng real-tme real-world around-the-clock dgtal vdeo analyss, we have developed and evaluated a test bed of sequences, across a range of condtons. The condtons we have studed nclude: ndoor/outdoor, varyng weather condtons, and dfferent cameras/vewponts. We have also evaluated our system on the standard datasets provded by the PETS 2001 workshop. In ths paper, we suggest several metrcs for system performance evaluaton, and test our system on these metrcs for several dfferent types of sequences. When possble, we provde the sequences and ther ground truth annotaton onlne for general accessblty and further testng. [ performanceevaluaton.html] Performance evaluaton systems have been developed to analyze the two prmary levels of processng: background subtracton and trackng. We beleve these are both mportant elements of any dgtal vdeo survellance system. However, snce the ultmate performance of such a system, reles on correct trackng and subsequent trggerng of specfc alarms and possbly archvng the correct vdeo clps, t s mportant to dstngush raw background subtracton detecton accuracy from the eventual hgh level trackng and trggerng of an event alarm. Background subtracton may nfluence the effectveness of the trackng, but the fnal trackng wll reflect the system s capabltes. In the next secton of the paper, we dscuss the stateof-the-art n performance evaluaton ncludng the absolute performance of systems.e., what are the current performance values reported by researchers on ther systems. In Secton 3, we descrbe the annotaton process used to generate ground truth. In Secton 4, we descrbe our background subtracton evaluaton system. In Secton 5, we explan our track evaluaton system. In Secton 6, we descrbe the datasets used for evaluaton. In Secton 7 we show the results on each of the datasets. Fnally we gve our conclusons n Secton Background Some of the earlest efforts n performance evaluaton of movng object detecton and trackng began at the

2 PETS (Performance Evaluaton of ng and Survellance) workshops. In 2000 and 2001, the workshop provded general outdoor survellance benchmark datasets for the partcpants to evaluate ther systems. [Senor 01] suggested several metrcs for evaluatng performance of trackng ncludng: # track false postves, # track false negatves, average poston error, average area error, average detecton lag, and average track ncompleteness. However, the results of these metrcs are hghly dependent on the nput sequence and practcal systems need to perform well on a wde range of nput data. [Toyama 99] was the frst to analyze the performance of nne background subtracton methods usng the number of pxels erroneously classfed as foreground (false postves) or not detected (false negatves). The results were based on the manual and somewhat arduous annotaton of seven short (several mnutes) sequences usng the outlne of each foreground object at 4Hz and a resoluton of 160x120. The sequences were chosen to exemplfy each of 7 challengng stuatons for background subtracton methods. In partcular, the stuatons were: moved object (statc objects are moved), tme of day (gradual lghtng change), lght swtch (sudden lght change), wavng trees (unnterestng moton), camouflage (foreground smlar to background), bootstrap (constant moton, no tme to learn background), and foreground aperture (unformly colored object moves). Ther results show the comparatve performance of the nne algorthms and the advantages of background subtracton methods whch determne foreground objects based on spatally varyng crtera (.e., not just based on pxel level models.) For further comparson, ther sequences need to be made accessble to other researchers. More recently (at PETS 03), two new methods have been proposed to evaluate background subtracton and n the second case, trackng performance. [Chaldabhongse03] proposed a background subtracton evaluaton system n whch the false alarm rate (FAR) s fxed, typcally n the range of percent of pxels/frame, and unform random contrast dfferences are generated to determne the just notceable dfference (JND) for background subtracton. They compared 4 methods and found ther own codebook approach whch uses a nonparametrc quantzaton/clusterng model to be superor. Ther analyss compares the raw performance dfferences of pxel-based approaches. Ths s a useful, repeatable (although of course, t depends on the sequences) metrc but lmted to measurng raw (.e. pxel level) models. The degree to whch the role of the base performance of background subtracton pxel-level model effects the ultmate results of trackng 24/7 n the real world are stll unclear. [Black 03] propose a methodology to mnmze the panstakng manual annotaton necessary to provde accurate ground truth nformaton. They suggest generatng a range of trackng stuatons based on a small set of manually annotated tracks by usng dfferent combnatons of tracks on dfferent background scenes. Although ths clearly elmnates sgnfcant labor, t s not clear ths wll smulate many of the ssues present n actual vdeo such as wnd and shadows or effects due to naccurate background subtracton healng (e.g. ghosts left behnd when statonary objects begn to move.) Most of these effects are complexly related to the tracks themselves and wll not occur wth smulaton but may severely affect performance. Furthermore there s a need to dscover these real-world ssues. They also report the performance of ther system on the PETS 2001 data for one sequence (dataset 2, camera 2). For the full resoluton (768x576) color vdeo, they report a false alarm rate of.01 and a track detecton rate of 98%. In terms of our metrcs (descrbed n Secton 4, ths s equvalent to FP=.02, FN=.02, and track fragmentaton=1.2. Ths s one of the few quanttatve and comparable results reported for a publcly avalable sequence albet for a sngle sequence and no tmng nformaton. System to ground truth algnment was based on mnmzng the dstance metrc (match each system track to ts best GT track) weghted nversely by the length of the temporal overlap [Senor01]. We wll dscuss ths metrc and ts lmtatons n Secton 5. [Pngal96] descrbe an approach to measure trackng accuracy based on trajectores and trajectory events such as crossngs. Matchng system to ground truth tracks s based on comparng trajectores and trajectory events. Ther prmary contrbuton allows performance evaluaton of specfc applcaton goals such as countng. As far as we know, the work of [Black04], [Pngal96] and [Senor01] are the only papers whch quanttatvely evaluate the performance of full trackng algorthms,.e., trackng of multple objects through occlusons. Other researchers have studed trackng systems only wth regard to segmentaton accuracy,.e., wthout evaluatng occlusons, mergng and splttng. [Tssanayagam02] evaluated the performance of contour trackers but they only consder performance of the trackers n terms of the accuracy of the contour and assume a sngle object wthout occluson. [Erdem04] measure trackng performance based on segmentaton accuracy usng spatal dfferences of color and moton along the boundary. [Dahlkamp04] vsually compare two vehcle trackng methods by analyzng ther respectve falure modes and comparng the tme ntervals n whch each vehcle s successfully tracked.

3 In ths paper, we descrbe a method to evaluate trackng performance for real-world stuatons n whch multple objects traverse the scene, there s sgnfcant background clutter, and objects are occluded by the scene and each other. 3. Ground Truth Acquston Ground Truth (GT) s were obtaned manually usng an annotaton tool. Annotaton s performed every 30 frames and at start/end of each track. The user draws the approprate boundng box around each foreground object whch s assocated wth a track. If the object s temporarly predomnantly occluded, the user marks t as such. The system tracks are obtaned by an automatc scrpt. More nformaton about our system, the Smart Survellance System can be found n [Hampapur03]. 4. Background Subtracton Evaluaton The background subtracton evaluaton compares every ground truth frame aganst the results of a specfc background subtracton algorthm. Each comparson determnes f there s a false negatve (FN): no system foreground object centrod nsde the ground truth boundng box or false postve (FP): system foreground object does not ntersect wth any ground truth boundng box. If a foreground movng object becomes statonary, we do not measure performance for ths regon.e., whether the system contnues to detect ths object for longer or no longer detects t, we do not consder t to be ether a FP or FN because of the ambguty of the stuaton. The evaluaton determnes the number of true postves (TP) over all ground truth frames. The fnal FP measure represents the average number of false postves per ground truth frame. The fnal FN measure s the percentage of TP whch are mssed by the system. For both false postves and false negatves, we also measure the average area (n pxels) of ther respectve nstances. These values are referred to as FPSze and FNSze. We compare two background subtracton algorthms. The frst uses a varaton of the adaptve mxture of Gaussans model (MOG) [Stauffer 99]. We use three Gaussans per pxel and a threshold of.3. The multadaptve model learnng rate s.01 and the weght update learnng rate s.005. The second method, we call Salence-Based (SAL) and s descrbed n [Connell04]. Ths method combnes evdence from dfferences n color, texture and moton. The method also has several bult-n mechansms to handle changng ambent condtons and scene composton. Frst, t contnually updates ts overall RGB channel nose parameters to compensate for changng lght levels. Second, t estmates and corrects for automatc gan control and whte balance shfts nduced by the camera. Fnally, t mantans a map of hgh actvty regons and slowly updates ts background model only n areas deemed as relatvely quescent. 5. ng Evaluaton In addton to measurng the performance of the background subtracton, we also measure the performance of the full system (background subtracton followed by trackng.) For ths evaluaton, we need to determne whch system tracks correspond to whch ground truth tracks. In [Senor 01], the evaluaton matched system tracks to ground truth tracks. The correspondence was many-toone,.e., several system tracks could be matched to one ground truth track but not vce versa. A match was based on proxmty and the overlap duraton: where MatchDuraton * MatchDuraton GT p s the centrod of the ground truth track at the th Sys ground truth frame, p s the centrod of the system track, and Dst s the Eucldean dstance. MatchDuraton s the number of frames n the overlap. If the match score s below a threshold, the two tracks are matched. Ths metrc s useful for smple scenes and trackng scenaros but has several lmtatons. Ths can be best explaned n terms of the four types of trackng errors: spatal fragmentaton, temporal fragmentaton, spatal mergng and temporal mergng. Fg 1 shows examples of track fragmentaton error. Ths can be due to ether spatal error (e.g. a sngle person results n an upper and lower body track) or temporal (e.g. a small object s only ntermttently observed). In the latter case, the horzontal axs represents tme. Fg 2 shows examples of track merge error. Temporal mergng s often due to the track of one object extng just as the track of another object enters the scene. Spatal mergng s often the result of the tracks of two objects mergng when they appear close together. S1 GT1 S2 Fg 1. fragmentaton error. The system dentfes multple tracks for a sngle real track. GT1 MatchDuraton Dst S1 ( p GT, p Sys GT2 Fg 2. merge error. The system dentfes a sngle track for multple real tracks. )

4 The problem wth matchng tracks usng the prevously descrbed match score based on proxmty s twofold. Proxmty between the centrods of the system and ground truth tracks s often nsuffcent to correctly match tracks when multple tracks are nearby; sze and shape nformaton should also be consdered. Second, snce a many-to-one match s performed (many ground truth tracks to each system track) only track fragmentaton errors are addressed. We have found when evaluatng a wde range of sequences, that ths metrc s nadequate. We propose a new two-pass matchng scheme to address these lmtatons as seen n Fgure 3. In phase 1, each system (S) track s allowed to match to many ground-truth (GT) tracks. A GT track s matched to the system track f there s both temporal overlap and spatal overlap. Temporal overlap s wth respect to the duraton of the system track. Spatal overlap s based on the centrod of the system lyng nsde the boundng box of the ground truth track. If multple GT tracks are matched to a partcular system track, then the cumulatve temporal/spatal overlap s computed,.e, percent of frames whch overlap both spatally and temporally. Ths s used to fnd track false postves.e. system tracks wth nsuffcent matches. We threshold the cumulatve overlap to dentfy system tracks wth nsuffcent matches (track false postves). By measurng temporal and spatal overlap, we address the problems of temporal and spatal mergng respectvely. After ths matchng phase s completed, we can fnd nstances of track mergng system tracks whch are explaned by multple ground truth tracks. In phase 2, each GT track s matched to many system tracks. Ths s used to determne track false negatves,.e., ground truth tracks wth nsuffcent matches. In ths case, temporal and spatal overlap s used to dentfy nstances of temporal and spatal fragmentaton respectvely. By determnng spatal overlap based on the system track centrod locaton nsde the GT track boundng box (or vce versa for phase 1), we have a more precse estmate of track concdence than proxmty. We enlarge the boundng box by 20% (E1=E2=.2) to account for small errors n segmentaton. After ths matchng phase s completed, we can fnd nstances of track fragmentaton ground truth tracks whch are explaned by multple system tracks. After track matchng s performed, t s possble to measure the number of track false negatves (TFN) and track false postves (TFP). These should be tracks whch have ether been mssed by the system or ncorrectly found by t. We have set the temporal overlap thresholds for TFP to T1=.5 and for TFN to T2=.01. Vsualzng these tracks has been very useful n understandng the causes of these problems and the ultmate system performance. In addton to measurng the number of track false postves (TFP) and track false negatves (TFN), we also measure the average sze and duraton of TFPs and TFNs. The fragmentaton error s defned as the number of system tracks per ground truth track (phase 1). The merge error s defned as the number of GT tracks per system track (phase 2). 6. Data Sets PETS 01 4 sequences from the Performance Evaluaton n ng for Survellance (PETS) Workshop 2001, 2 dfferent cameras of an outdoor campus scene, hgh qualty (from dgtal camera), wth resoluton (358,288) 30fps, stored as avs wth no compresson. The data from PETS01 was orgnally of hgher resoluton and stored as JPEG mages. Hawthorne Outdoor 10 sequences from an IBM buldng entrance and parkng lot, from 4 dfferent Sensormatc NTSC cameras, many dfferent vewponts, range of NY weather condtons, 320x240 resoluton, 30fps, MPEG1 compressed, 2-5mnutes each. Longer Sequences 4 longer sequences from 4 dfferent IBM Sensormatc NTSC cameras, 320x240 resoluton, 30fps, MPEG1 compressed, mnutes each. Sgnfcant lghtng changes and wndy condtons ncludng camera nstablty Indoor 11 sequences, 5 dfferent NTSC cameras, 320x240 resoluton, 30fps, MPEG1 compressed, less than 3 mnutes each. Three sequences are taken smultaneously by 3 cameras n our laboratory as two or three people walk by and around each other. Two other sequences are taken from two dfferent cameras n our lobby and by our elevators, of two people, each walkng along a corrdor, followng one another, and then walkng past each other. 7. Results We frst report our results on the PETS01 datasets. Tables 1 and 2 show the results of the background subtracton and trackng evaluaton for the two dfferent background subtracton methods. We compare the results for varyng resoluton (ds1 = full resoluton, ds2= half resoluton), varyng the mnmum connected component sze threshold (30 or 100 pxels for full resoluton), and for grey-scale (8-bt) vs. color (RGB 24bt). In each case (resoluton, component sze threshold and grey vs. color) there s a clear trade-off between mproved detecton (lower FN) and over-senstvty (ncreased FP). Ths relatonshp s depcted n the plots shown n Fgure 4. These plots show the relatonshp between the false negatves and false postves for each background

5 subtracton method. Each lne segment represents the two values obtaned at the two resolutons (ds1/ds2). In addton to showng the trade-off between FN/FP, the plot for the salence-based approach ndcates lttle change wth grey to color or from low (ds2) to hgh (ds1) resoluton Ths s not true for the MOG method. Table 2 shows the average number of FP per frame based on the background subtracton evaluator and the number of track false postves and ther average duraton based on the track evaluator. Although the number of false postves s hgh, the average duraton s typcally short (<100 frames or 3 seconds). The percentage of false negatves wth respect to the total number of true postves and ther average sze based on the background subtracton evaluator s also shown. In addton, the number of track FN and ther average sze per frame n square pxels are also gven. FN are typcally less than 100 square pxels. The best results are obtaned usng the SAL method, at half resoluton (DS2), wth mnmum connected component sze of 30 and color pxels. At these settngs, there were 6 TFP and 8 TFN. The evaluator automatcally creates a vdeo of these tracks for vsualzaton of the results. Fgure 4 shows an example frame from the TFPs and one of the TFN. The other FNs are very smlar along the same dstant partally occluded road. The TFP are due to (1) parked cars begnnng to move leavng a hole (2) movng tree and movng object resultng n extra object and (3) shadows. Table 3 shows the results of the background subtracton and trackng evaluaton on the 10 Hawthorne outdoor sequences usng the MOG background subtracton method, CCMn=100, and grey-scale pxels. MOG performed modestly better than SAL overall. Fgure 5 shows several examples of track FPs and FNs. Table 4 shows the results of the evaluaton on the four longer outdoor sequences. It can be seen wth more data, our results are not yet suffcent to generalze. For these longer sequences the SAL method was sgnfcantly more robust to the strong lghtng changes whch caused nnumerable FP for MOG method. Table 5 shows the results of the evaluaton on the 11 ndoor sequences. The movng objects n the ndoor data were substantally larger than outdoors and not subject to the lghtng changes, weather and camera moton due to wnd. Hence the ndoor data had no TFN and only one TFP due to shadows. For ndoor data, the performance was most nfluenced by the accuracy of trackng through occluson. For our smple appearance-based tracker there were sgnfcant amounts of fragmentaton and mergng. Some of the mergng s due to actually mergng of GT tracks. In ths case, a system track wll correctly match to multple GT tracks. Ths type of mergng should not be reported as merge errors. Fgures 7 and 8 show examples of track fragmentaton and merge errors. Fgure 7 shows an example of spatal mergng and temporal mergng. Fgure 8 shows an example of temporal fragmentaton and temporal mergng due to track crossng (the system follows one track then loses ths track whch contnues and ncorrectly follows a dfferent track.) For the appearance based tracker used n these examples spatal fragmentaton dd not occur. The tme requred by the system (background subtracton and trackng) to process each frame for a gven vdeo sequence (from the 10 outdoor Hawthorne vdeos) s shown n Fgure 6. Ths s based on MOG background subtracton followed by appearance based trackng. The graph s a hstogram showng the relatve frequency of frame tmes n mcroseconds on a 2.4 GHz machne. Ths plot shows that most frames take < 12ms to process and very few take more than 20ms (~50fps). The left hand peak (7ms/frame) corresponds to frames n whch no foreground s detected. The rght hand (and broader) peak corresponds to frames n whch trackng must be carred out n addton to background subtracton. 8. Conclusons In ths paper we have presented a new method for evaluatng the performance of background subtracton and trackng ncludng a track evaluaton based on matchng ground truth tracks to system tracks n a two-way matchng system. We have shown the quanttatve results of ths evaluaton on the PETS benchmark data, over 100 mnutes of outdoor data wth a wde range of camera vewponts, weather condtons, lghtng changes and camera nstablty. We have also shown results on ndoor data. We have made some of the data and annotaton avalable publcly (when possble) n order to enable the communty to work together to understand the relatve merts of dfferent algorthms. Consequently many of our results can be openly compared to results wth other algorthms. We have llustrated the trade-off between FN and FP detecton based on varyng background subtracton method, resoluton, color vs. grey-scale, and the mnmum connected component sze. But, we have also shown, va the use of longer sequences, that nsuffcent data s yet avalable to determne the performance of systems for around-the-clock operaton. 9. References [Black03] Black, J. et al., A Novel Method for Vdeo ng Performance Evaluaton,, Jont IEEE Int l Workshop on Vsual Survellance and Performance Evaluaton of ng and Survellance (VS-PETS), Nce France, October 11-12, 2003, p [Chaldabhongse03] Chaldabhongse, T.H., et al., A Perturbaton Method for Evaluatng Background

6 Subtracton Algorthms, Jont IEEE Int l Workshop on Vsual Survellance and Performance Evaluaton of ng and Survellance (VS-PETS), Nce France, October 11-12, 2003, p [Connell04] Connell, J., Detecton and ng n the IBM PeopleVson System, IEEE ICME, June [Erdem 04] Erdem, C.E. et al., Performance Measures for Vdeo Object Segmentaton and ng, IEEE Trans. On Image Processng, Vol 13, No. 7, July [Dahlkamp04] Dahlkamp, H. et al., Dfferental Analyss of Two Model-Based Vehcle ng Approaches, DAGM 2004, LNCS 3175, pp71-78, [Erdem04] Erdem.C. et al., Performance Measures for Vdeo Object Segmentaton and ng, IEEE Trans. On Image Processng, Vol 13, No. 7, July [Hampapur03] Hampapur, A. et al., Smart Survellance: Applcatons, Technologes and Implcatons, IEEE Pacfc-Rm Conference On Multmeda, Sngapore, Dec [Pngal96] Pngal, S. and Segen, J., Performance Evaluaton of People ng Systems, IEEE Workshop on Applcatons of Computer Vson, p33-38, [Senor01] Senor, A. et al., Appearance Models for Occluson Handlng, IEEE Int. Workshop on Performance Evaluaton of ng and Survellance, Kaua, HI, December 9, [Stauffer99] Stauffer, C. and Grmson, W.E.L., Adaptve Background Mxture Models for Real-tme ng, Int l Conf. Computer Vson and Pattern Recognton, Vol. 2, pp , [Tssanayagam02] Tssanayagam, P., and Suter, D., Performance Measures for Assessng Contour ers, IEEE Int. Journal of Image and Graphcs, Vol 2, p , Aprl [Toyama99] Toyama, K. et al., Wallflower: Prncples and Practce of Background Mantenance, Seventh Int l Conf on Computer Vson, pp , System--Matchng for every system track fnd all GT-matches GT-match = Temporal-Overlap AND Spatal-Overlap Temporal-Overlap = overlap/(system duraton) Spatal-Overlap = GT centrod nsde E1% enlarged system boundng box If cumulatve temporal/spatal overlap < T1, then system track has nsuffcent matches and s labeled a FP. If multple GT-matches, then ths system track has merge error = # matched GT tracks 2. GT--Matchng for every GT track fnd all system-matches System-match = Temporal-Overlap AND Spatal-Overlap Temporal-Overlap = overlap/(gt duraton) Spatal-Overlap = system centrod nsde E2% enlarged GT boundng box If cumulatve temporal/spatal overlap < T2, then GT track has nsuffcent matches and s labelhbghed a FN. If multple system-matches, then ths GT track has fragmentaton error = # matched Sys tracks Fgure 3. Two-pass many-to-many system to ground truth (GT) track matchng crtera MOG 30 MOG 100 SAL 30 SAL 100 FP FN FP FN FP FN FP FN COLOR DS DS GREY DS DS Table 1. Performance results on PETS01 data usng dfferent background subtracton methods (SAL/MOG), dfferent resoluton(ds1/ds2), and mnmum connected component sze (30 or 100 pxels) for Color/Grey-Scale data. Fle Frames True Postves False Postves False Negatves TP FP TFP duraton FN TFN area MOG SAL Table 2. Performance results ncludng performance of trackng on PETS01 data usng two best results (SAL-color-ds2 and MOG-grey-ds1.

7 Salence-Based Background Subtracton ROC on PETS data MOG Background Subtracton ROC on PETS data % False Negatve Color CCSze30 Grey CCSze 30 Color CCSze100 Grey CCSze100 %False Negatves Color CCSze30 Grey CCSze30 Color CCSze100 Grey CCSze False Postves/Frame False Postves/Frame Fgure 4. ROC plots of the performance of two background subtractons plots. Each lne segment represents the results at two resolutons. Performance vares from upper left (hgh FN, low FP usng grey-values, low resoluton, and large sze threshold to bottom rght (low FN, hgh FP) for color, hgh resoluton and small sze threshold. Fgure 5. Top left, track FN example car behnd trees on upper left. Top rght, track FP due to statonary truck movng and leavng a ghost. Bottom left, track FP due to movng tree near movng object. Bottom rght, track FP due to shadow on grass (number on mage where FP occurred.) Fle Frames True Postves False Postves False Negatves TP FP TFP duraton FN TFN area MOG SAL Table 3. Performance Results on 10 Hawthorne Outdoor Vdeos (CCSze100,ds2,grey) Frames True False False Fle Postves Postves Negatves Table 4. Performance results on 4 longer Hawthorne outdoor sequences usng SAL background subtracton. Fgure 6. Two pctures on left are example frames from track FNs - the frst s not detected because of nsuffcent contrast, the second because t les on the border of the vdeo mage (upper left). Two pctures on rght are examples from track FPs the frst s the result of sgnfcant shadows, the second from reflectons off glass of buldng (bottom rght of mage.)

8 Indoor Data GT TP FP FN s TFP TFN #sys/gt #GT/sys Frames Total /Ave Table 5. Performance results on ndoor data ncludng track fragmentaton and merge errors. Fgure 7. Left two mages: spatal merge system combnes two people nto one track, Mddle two mages: 1 st person walks across, Last two mages: 2 nd person then walks, causng temporal merge system combnes tracks of both people, one after the other. Fgure 8. Top Row frst person walks halfway across room, stops and then contnues. System s confused when the person stops and creates another track when he restarts resultng n temporal fragmentaton (s S0 and S2). Second Row second person walks around the frst person. System ntally tracks ths person as S1 but then ncorrectly connects hs fnal ext to track S0. Fgure 9. Hstogram showng the relatve frequency of frames executon tmes (n mcroseconds on a 2.4GHz machne). The left hand peak (7ms/frame) corresponds to frames n whch no foreground s detected. The rght hand (and broader) peak corresponds to frames n whch trackng must be carred out n addton to background subtracton.