Human Activity Tracking for Wide-Area Surveillance

Transcription

1 Huma Activity Trackig for Wide-Area Surveillace Patrick D. O Malley Michael C. Nechyba A. Atoio Arroyo pomalley@mil.ufl.edu echyba@mil.ufl.edu arroyo@mil.ufl.edu Machie Itelligece Laboratory Departmet of Electrical ad Computer Egieerig Uiversity of Florida, Gaiesville, FL Abstract We preset a method for trackig ad idetifyig movig persos from video images take by a fixed field-ofview camera. Specifically, we have developed a system to first track movig persos i a give scee ad geerate color-based models of those persos to accomplish idetificatio. The trackig is o-ivasive meaig that it does ot require persos to wear ay particular electroics or clothig to be able to track them. Trackig is accomplished usig a positio-based data associatio algorithm while the color modelig is accomplished usig a mixture-of- Gaussias statistical model. The expectatio-maximizatio algorithm is used to geerate the color models over a sequece of frames of data i order to make the etire process ear-realtime. Oce a color model is developed for a give perso, that model will be placed i a database where it will be used for future idetificatio. Ay similar idetificatio scheme ca be used i place of color modelig for greater accuracy.. Itroductio Trackig people usig oly the data from a video camera is a difficult computer visio problem. However, research i this area is spurred o by the promise of may ehacemets to curret techology that a robust perso trackig system would provide. Already we are seeig the use of computer geerated overlays to add iformatio to sports broadcasts. These overlays, however, are usually depedet o special sesors ad emitters placed o the tracked object. The RACEf/x(c) [] system, for example, uses a GPS system to accomplish trackig of NASCAR (c) racecars i order to superimpose car statistics such as the ame of the driver, velocity ad positio oto the broadcast. The people trackig problem, o the other had, caot usually be solved by usig similar trasmitters. I surveillace applicatios wherei people are tracked i order to detect uauthorized etry or suspicious behavior for example, there is o possibility of usig trasmitters. A o-ivasive method usig oly the iformatio obtaied by a passive sesor such as a video camera is ecessary. The specific problem which we wish to address is surveillace over a wide-area. A typical coverage zoe would be a airport, the dowtow of a large city or ay large public area. Surveillace over these wide areas would be utilized to search for suspicious behavior - persos loiterig, for example - or uauthorized access - a perso attemptig to eter a restricted zoe, for example. Curretly, surveillace of this type is accomplished by a huma operator who does all of the detectio of uauthorized activity over may cameras. Give the challege of watchig multiple moitors, it is easy to see how some suspicious activity would go uoticed by a huma operator. A computer visio system, however, ca moitor both immediate uauthorized behavior ad log-term suspicious behavior. The system would the alert a huma operator for a closer look. I this way, oe operator could moitor more locatios or moitor the same umber of locatios more closely. The most commo method of trackig ay movig object, icludig persos, is to use some form of image subtractio to fid motio regios. These motio regios are the classified as either oise or a perso. Researchers such as [2] use the differece betwee cosecutive video images to fid the regios of motio. Others such as [3] ad [4] use a backgroud kow to be empty of ay movig objects (a referece image ) which is the subtracted from curret or foregroud images to fid the motio regios. It should be oted that this use of a referece image is limited to fixed field-of-view cameras whereas the cosecutive image subtractio method ca be used with movable cameras. The more difficult step occurs after fidig the motio regios. This step is to determie which motio regios should be tracked ad which are oise. A simple tree swayig i the breeze, for example, could cause severe oise problems leadig to false positives (oise tracked as a perso) or false egatives (a perso ot tracked because of earby oise.) [5] assumes that a ukow object is to be tracked (ad thus a perso) if it ca be tracked over several frames usig a secod-order motio model. Other trackig methods usig color aloe [3] have a difficult time trackig ew objects - oise or a perso - because they do ot have color iformatio available iitially. Idetificatio of a give idividual is difficult uder ay circumstaces. If the video imagery is take from a distace such that the face is obscured, it is questioable whether a good likelihood of positive idetificatio is pos-

2 sible. If the video ca be take close-up, a face recogitio algorithm such as that by [6] or [7] could give good idetificatio accuracy. A full-body color modelig scheme as used by [8] would be useful assumig the statistical model chose ca robustly compare two slightly dissimilar persos. It ca be see that a simple color model caot be used over great time periods for idetificatio but should be limited to short spas ad localized regios - a city block, for example. I order to track a perso over a wider area ad to be able to coect tracks foud across multiple cameras, a sigle frame of referece will be eeded to trasform coordiates from idividual cameras ito a global coordiate system. Oe method (for two cameras) is give by [9]. Exteded to may more cameras, we ca see how a huma trackig system ca be used for wide-area surveillace. 2. Overview The method for trackig people preseted here is fudametally based o utilizig two classificatio systems: positio ad color. As with [5], we will use a first-order motio model to track the idividual people as they move throughout the scee. The secod classificatio system is a mixture-of-gaussias model of the color of each perso. Iitially, these two classifiers act separately - that is, they do ot share iformatio betwee them to support more robust classificatio. However, further work i the area (see Sectio 7) will make the trackig more robust by fusig the two systems. By separatig the trackig system from the color modelig system, we ca solve the bootstrappig problem faced by systems such as [3] which use oly color modelig as the trackig system. We ca maitai trackig usig the positio-based tracker while at the same time geeratig a color model of the tracked perso. Oce that color model is developed, we ca use it for idetificatio or to resolve occlusio evets as described i Sectio 7. Before ay trackig work ca be doe, the video frames (images) eed to be reduced to cetroids of motio regios usig image processig techiques. 3. Image Processig I order to determie where a potetial perso is i a give scee, we reduce the scee to the cetroids of motio regios. Those cetroids are what fially gets tracked by the positio-based tracker. The origial frame is retaied, however, to extract color iformatio later. Like [3] we use a backgroud subtractio scheme based o makig a ormal distributio model for each pixel. Each frame is first coverted from RGB (red, gree, blue) colorspace to HSV (hue, saturatio, value.) We chose HSV because it teds to reduce the effects of shadows o the subtractio process. The ormal distributio for each pixel is foud by takig the first frames of the video sequece (which we assume to ot have ay foregroud or movig objects i them.) A three dimesioal vector, xk ij give i () where k is the frame umber ad (i,j) is the pixel coordiate, is the used to model the mea ad variace as give i (2) ad (3). xk ij = h s v µ ij = -- xk ij k = Σ ij = ( xk ij µ ij ) xk ( ij µ ij ) T k = The Mahalaobis distace is used to classify a pixel from a frame as either foregroud or backgroud. However, to speed the calculatio, we oly use the diagoals of the covariace foud i (3). We experimetally choose a distace threshold above which a pixel is classified as foregroud. Oce motio regios are foud, we get masks such as that i Figure. Usig these masks, we apply a stadard coected-regio segmetatio algorithm to determie the followig properties of each regio: cetroid coordiates (x,y), extets (xmax, ymax, xmi, ymi) ad size i pixels. Size thresholds are applied to elimiate ay motio regios too small to be cosidered a perso. The remaiig cetroids are set to the positio trackig system to be classified as a trackable object (a perso) or oise. 4. Positio Trackig System The positio trackig system is based o similar systems used i RADAR trackig situatios where there is a high oise desity. The first stage of trackig is kow here Fig. : Motio regio mask () (2) (3) 2

3 as the Track Iitiatio Processor (TIP). After the TIP has acquired a target (assumed here to be a perso) it passes that target to the Data Associatio Filter (DAF) which is desiged to keep a lock o the target. The fial stage is the track-drop ruleset which is a heuristic approach to determiig whe a target is o loger trackable such as whe it has goe behid a object or has left the scee. 4. Track Iitiatio Processor The job of the Track Iitiatio Processor (TIP) is to extract from a series of video frames, all of the objects which could potetially be people. At this stage we assume that the image processig has reduced each video frame to a series of cetroids of motio regios that are as yet uclassified as targets. Therefore, the TIP will classify them as either trackable objects (people) or oise. If oise, those regios are t elimiated outright because they could be asyet utracked objects so the TIP algorithm takes them ito cosideratio. The TIP is a faster implemetatio of a commo algorithm used i RADAR kow as retrospective processig [0] wherei three assumptios made about movig objects are evaluated o all potetial targets. By evaluatig all potetial targets - that is, all uclassified motio regios whether oise or people - over may frames, we will evetually fid oly those targets which ca be people. That is cosidered a good track. The three assumptios we wish to follow are very simple: First, that our oise is radomly distributed through the scee. Secod, that a perso movig through the scee moves with liear velocity over short distaces. Thirdly, that a perso movig through the scee moves with a limited maximum velocity. These three assumptios are ot ureasoable give that people do ot ted to move with a quickess that is goig to violate either of the last two assumptios. We fid objects which follow these three assumptios by usig frames of data. Three frames is the miimum. The first frame of cetroid data is assumed to be all potetial targets (potetial people.) We will prue this assumptio over the ext two frames. The situatio whe the first frame of data is preset is show i Figure 4a. I this simulatio (of radomly geerated 2-d data), we see simulated cetroids of objects. The first image (a) shows the objects as blue dots with gree circles idicatig the maximum distace that this object (if it is i fact a perso) could travel from this frame to the ext. The secod image (b) shows the frame #2 objects as red dots. Oly those red objects iside the gree circles ca be people because the others are movig too fast (or are oise.) Isolatig oly those pairs of blue ad red objects (frame # ad #2 objects, respectively) we predict where those objects would be i the third frame if they follow the costat velocity assumptio. The third image (c) shows these predictios as mageta circles. Fially, whe the third frame of data arrives (show i Figure 4d) we ca see that oly oe of those objects satisfies the costat liear velocity assumptio. Therefore, we ca elimiate all the other objects as oise. The three-poit track we have foud ca ow be used to coverge the Kalma filter described i the ext sectio. The algorithm described here ca be exteded from three frames of data to ay umber. The more frames used to fid a valid track, the more likely it is to be a valid track ad ot simply well-placed oise. I this way, our image processig ca be poor i that it leave a lot of oise yet trackig would ot suffer seriously. 4.2 Data Associatio Filter To cotiue the trackig process, a data associatio filter is used. We use a first-order Kalma filter to geerate predictios of the positios of each object i the ext frame. Whe that frame arrives, we associate the predictios with the icomig objects usig a earest-eighbor algorithm. We determie earess based o the Mahalaobis distace from the icomig cetroid to the predictio with its associated ucertaity covariace give by the Kalma filter. This method teds to give more robust results tha usig a simple Euclidea distace measure. 4.3 Track Drop Rules I order to determie whe to drop a track, a heuristic approach is used. Curretly, we allow the DAF to miss - that is, fail to associate with a icomig object - twice before that object is cosidered goe ad the track is dropped. A miss is defied to be whe the DAF caot make a earest-eighbor associatio because there are o objects withi a certai distace to associate with. The distace threshold is determied experimetally. 5. Color Modelig Color modelig is used for local idetificatio. That is, because it is assumed that a perso would chage color because they chage clothig from oe day to the ext or eve from oe hour to the ext, usig color models for idetificatio does ot work over log time spas. Therefore, we use color models i localized temporal ad spatial regios. The dowtow of a city for oe day, for example. After that day, the color models would be expired. But for that day, the idetificatio is assumed to work well eough to allow systems such as this to follow a give perso eve if they disappear from the reappear i the camera s field of view. I this way, the perso s piecemeal path ca be recorded. 5. Mixture-of-Gaussias Modelig We chose to use a mixture-of-gaussias model of the perso s color data i RGB (red-gree-blue) colorspace. Mixture modelig like this is typically a good fit to the data 3

4 give that most persos ted to appear as oly a few regios of color. Blue jeas ad a t-shirt, for example, are usually oly a few colors. Ski toes are well modeled as a regio of color. To simplify the mixture model, we fix the umber of Gaussias to three. Three was chose based o the Expectatio-Maximizatio (EM) algorithm we use to fit the Gaussias to the data. Equatios (4), (5) ad (6) give the update equatios for the iterative EM solutio to the mixture-of-gaussias problem for i {, 2,..., k} where k is the umber of Gaussias i the model ad is the umber of poits i the data set. As ca be see, reducig the umber of Gaussias ad the amout of data to be modeled ca sigificatly icrease the computatioal speed of this algorithm. Therefore we used oly three Gaussias ad limited the data to 300 poits i RGB colorspace which were foud by subsamplig all of the perso s color data. Note that each of these equatios eeds to be iterated may times util the Gaussias coverge. P( ω i ) µ i = = -- P ω ( i x j, Θ) j = j = P( ω i x, Θ)x j j j = P( ω i x j, Θ) P( ω i x j, Θ) x j µ i ( )( x j µ i ) T Σ j = i = P( ω i x j, Θ) j = It was foud that further speed ehacemets were required to improve the algorithm. The choice was betwee fidig ways to do the above computatios more efficietly or tryig to hide the computatio time etirely. We chose the latter by istead of modelig the color all i oe frame, we spread the iteratios out over multiple frames. We also allowed the color data which the model is traied o to be chaged with each frame. I this way we were able to assure that we would be modelig more tha oe view of the perso. We use the icomig color data (data that has ot yet bee icluded i the color model) as ukow validatio data by takig the sum log-likelihood of the data give that perso s color model. Oce the log-likelihood coverges (4) (5) (6) (plateaus) we cosider it well-formed ad do ot cotiue to iclude ay more data i the model. 6. Results Figure 3 shows a typical sequece usig the trackig ad idetificatio algorithms together. The images are osequetial from the top-left to the bottom-right. The first frame shows the perso beig tracked iitially as he eters the scee. (A gree box idicates a tracked object. A blue box idicates a ukow object, probably oise.) Two idetifiers are give to the perso: a ame - i this case a - ad a target idetifier - i this case 0. The ame is associated with the color model ad is therefore uique to a give perso. The target idetifier is ot uique to a perso as it is merely the positio-based tracker s idetificatio iformatio. The ame ( a ) is assiged to this perso iitially because it did ot have a pre-recorded color model to be associated with. As this perso is tracked, a color model is developed ad will from the all be associated with the ame a. The secod image shows that the positio-based tracker loses the perso because of a image processig problem. (This problem was later fixed but re-itroduced ito the code to cause the code to lose trackig for demostratio purposes.) The third image shows that the perso is still ot tracked. I the fourth image, the perso is agai tracked. The code compares this perso s color to the color models i the database ad matches it to the ame a. (I this demostratio there was oly oe model i the database from whe this perso was tracked previously - as show i frame oe. However, it could have determied that this perso did t exist i the database at all ad assiged a ew ame. Because it did ot, it idicates that it successfully matched this perso to the previously developed color model.) The last two images show how it cotiues to track the subject. Not show is a failure to track caused whe the subject is occluded by the sig. 7. Future Work Two major areas of work could be doe to improve this trackig system. They are related, however, ad could be accomplished at the same time. The first problem is to improve the robustess of the trackig by fusig the positio- -based trackig system with the color modelig system. I this way, if a situatio happeed as i Figure 3, frame 2 wherei the tracker loses the perso because of a image processig problem, we could istead decide if the two halves of the previously coected perso are really supposed to be oe usig the color iformatio. Similarly, we ca resolve occlusio such as that show i Figure 2 where the motio regio is see as oly oe ob- 4

5 ject (bouded by the blue box) but which is really two persos who were previously ucoected. It ca be see how, oce we detect that occlusio of this sort has occurred (usig a heuristic techique, for example) we ca resolve the occlusio ad separate the people by usig the previously determied color models. We do this by classifyig each pixel of the big motio regio as either belogig to oe or the other color model. By those classificatios, we ca make a fairly good guess as to the relative locatios of the persos. 8. Coclusio We have show how a combiatio of two methods - positio-based trackig ad color modelig - ca be used to robustly track people o-ivasively usig oly stadard statioary cameras. We have also show how this method ca be expaded to recover from difficult problems such as occlusio by allowig the positio-based tracker to essetially bootstrap the trackig util a color model ca be developed which will the make the whole system more robust. We believe this system ca be useful i a wide variety of surveillace situatios. 9. Refereces [] Sportvisio website [olie]. URL: Retrieved o April 9, [2] A. J. Lipto, H. Fujiyoshi, ad R. S. Patil, Movig target classificatio ad trackig from real-time video, Proc. IEEE Image Uderstadig Workshop, pp , 998. [3] S. Kha ad M. Shah, Trackig People i Presece of Occlusio, Asia Coferece o Computer Visio, Taipei, Taiwa, Jauary [4] Q. Cai, A. Mitiche, ad J. K. Aggarwal. Trackig huma motio i a idoor eviromet, 2d Itl. Cof. o Image Processig, pages 25-28, Washigto D.C., October 995. [5] I. Haritaoglu, D.Harwood, ad L.Davis, "Who, whe, where, what: A real time system for detectig ad trackig people", Proceedigs of the Third Face ad Gesture Recogitio Coferece, pp , 998. [6] C. Beumier, M.P. Acheroy, Automatic Face Autheticatio from 3D Surface, British Machie Visio Coferece BMVC 98, Uiversity of Southampto UK, 4-7 Sep, 998, pp [7] R. Bruelli ad D. Falaviga. "Perso idetificatio usig multiple cues", IEEE Trasactios o Patter Aalysis ad Machie Itelligece, Vol. 7, No. 0, pp , October (995). [8] I. Zapata, Detectig Humas i Video Sequeces Usig Statistical Color ad Shape Models, Master s Thesis, Departmet of Electrical ad Computer Egieerig, Uiversity of Florida, 200. [9] S. Nichols, Improvemet of the Camera Calibratio Through the Use of Machie Learig Techiques, Master s Thesis, Departmet of Electrical ad Computer Egieerig, Uiversity of Florida, 200. [0] E. Brooker, Trackig ad Kalma Filterig Made Easy, Wiley, New York, 998, pp. -6. Fig. 2: Example of two-perso occlusio 5

6 Fig. 3: Example of trackig ad idetificatio video sequece 6

7 Fig. 4: Track Iitiatio Processor simulatio 7