A SALIENCY BASED OBJECT TRACKING METHOD

Transcription

1 A SALIENCY BASED OBJECT TRACKING METHOD Shje Zhang and Fred Stentford Unversty College London, Adastral Park Campus, Ross Buldng Martlesham Heath, Ipswch, IP5 3RE, UK {j.zhang, ABSTRACT A novel three-stage framework for object trackng under statonary background condtons s proposed n ths paper. The frst stage uses an attenton based method to etract moton nformaton. The second stage then apples a regon growng and matchng technque to moton vectors to obtan moton segmentaton. Fnally the movng objects are tracked based on the dsplacements of regon centrods. The method s tested on varous real-world vdeo data and emprcal results show that the proposed approach can track movng objects and etract moton nformaton from non-rgd objects such as movng people wthout pror knowledge of the object s sze or shape. 1. INTRODUCTION The demand for automated moton detecton and object trackng systems has promoted consderable research actvty n the feld of computer vson [1-9]. Ths paper proposes a method to detect and measure moton based upon trackng salent features usng a model of vsual attenton. Stauffer and Grmson [1] presented a novel probablstc method for background subtracton for multple object trackng. It modeled each pel as a separate mture Gaussan model. After the background subtracton process, foreground pels were dentfed, labeled and grouped nto regons by a connected components algorthm. The model was updated wth an on-lne appromaton. It copes well wth lghtng changes, repettve motons from clutter, and long-term scene changes wth dfferent weather condtons nvolvng dfferent cameras. However, problems arse when movng objects occlude each other and one object enters the scene whle another s leavng. In addton movng shadows are not removed durng trackng. Comancu [2] proposed a method for real tme non-rgd object trackng wth a movng camera based on the mean shft algorthm. One advantage s that the ntense blurrng due to camera moton dd not affect the tracker performance, whch s always a problem for contour based trackers. The tracker handles partal occlusons, background clutter and target scale varatons. It also works under low qualty sequences wth compresson artfacts but the coarse appearance models can fal to track accurately regons that share smlar statstcs (colour) wth nearby regons. Sh and Tomas [3] proposed a method for feature selecton, a trackng algorthm based on affne change models, and a technque for montorng features durng trackng. The feature selecton crteron depended entrely on how well the tracker worked. The trackng algorthm etended the prevous Newton-Raphson style search methods to work under affne transformatons. The bad features were abandoned based on a measure of dssmlarty that used an affne moton model. The proposed method works under occlusons. However, the method cannot handle deformable objects and the teratve trackng algorthm can take a relatvely long tme to converge. In [4] a new approach was proposed for vsual trackng usng dynamc geodesc snakes. The method combned state nformaton (velocty) wth every partcle on a contour descrbed by a level set functon. It works under partal occlusons but also tracks shadows due to the edgebased contour model. Isard and MacCormck [5] proposed a multple-person trackng system for sngle camera real-tme survellance applcatons. A mult-blob lkelhood functon was adapted from the theory of Bayesan correlaton based on learned statstcs, but assumed a statc camera to create more specfc background and foreground modelng. Then a Bayesan multple-object partcle flter was used for trackng. The observaton model proposed for object lkelhood used a synthess of learnt background patches, pooled foreground patches and geometrc reasonng from the camera calbraton. It works wth background clutter. One dsadvantage of ths approach s ts falure under occlusons. Bouthemy [6] proposed a novel probablstc parameter-free method for detectng ndependently movng objects usng the Helmholz prncple. Optcal flow felds were estmated wthout makng assumptons on moton presence and allowed for possble llumnaton changes. The method mposes a requrement on the mnmum sze for the detected regon and detecton errors arse wth small and low contrast objects. Black and Jepson [7] proposed a method for optcal flow estmaton based on the moton of planar regons plus local deformatons. The approach used brghtness nformaton for moton nterpretaton by usng segmented regons of pecewse smooth brghtness to hypothesze planar regons n the scene. The proposed method has problems dealng wth small and fast movng

2 objects. It s also computatonal epensve. Black and Anandan [8] then proposed a framework based on robust estmaton that addressed volatons of both brghtness constancy and spatal smoothness assumptons caused by multple motons. It was appled to two common technques for optcal flow estmaton: the area-based regresson method and the gradent-based method. To cope wth motons larger than a sngle pel, a coarse-to-fne strategy was employed n whch a pyramd of spatally fltered and sub-sampled mages was constructed. Separate motons were recovered usng estmated affne motons, however, the method s relatvely slow. Vola and Jones [9] presented a pedestran detecton system that ntegrated both mage ntensty (appearance) and moton nformaton, whch was the frst approach that combned moton and appearance n a sngle model. The system works relatvely fast and operates on low resoluton mages under dffcult condtons such as ran and snow, but t does not detect occluded or partal human fgures. The use of vsual attenton (VA) methods [1-13] to defne the foreground and background nformaton n a statc mage for scene analyss has motvated ths nvestgaton. We propose n ths paper that smlar mechansms may be appled to the detecton of salency n moton and thereby derve an estmate for that moton. The object trackng framework s presented n Secton 2. Results are shown n Secton 3 along wth some dscusson. Fnally, Secton 4 outlnes conclusons and future work. 2. OBJECT TRACKING FRAMEWORK OUTLINE The proposed framework contans three stages. The frst stage uses an attenton based method to estmate and etract moton nformaton [14]. The second stage then apples a regon growng and matchng technque to moton vectors etracted [15]. In the last stage movng objects are tracked based on lnkng regon centrods. The outlne of the method s gven below Moton estmaton based on vsual attenton Regons of statc salency have been dentfed usng an attenton method descrbed n [12]. Those regons whch are largely dfferent to other parts of the mage wll be salent and are lkely to be n the foreground. Ths concept has been etended nto the tme doman and s appled to frames from vdeo sequences to detect salent moton. The approach [14] does not requre an ntal segmentaton process and depends only upon the detecton of anomalous movements. The method estmates the shft of locatons between frames by obtanng the dstrbuton of dsplacements of correspondng salent features around these locatons. In ths paper canddate regons of moton are detected by generatng the ntensty dfference between the current frame and a background reference frame obtaned by averagng a seres of frames n an unchangng vdeo sequence. A threshold s then appled producng a potental moton template. The ntensty dfference I between pels n the current frame and the reference s gven by I = { r2 r1 + g 2 g1 + b2 b1 }, (1) where parameters ( r 1, g1, b1 ) & ( r 2, g 2, b2 ) represent the rgb colour values for pel n reference frame and the current frame. The ntensty I s calculated by takng the sum of the dfferences of rgb values between the two frames. The canddate regons R t n the frame t are then dentfed where I > T where T s a fed threshold n R t correspond to colour a. Let F() = a and let be n R t n frame t. Consder a neghbourhood G of wthn a wndow of radus ε where { G ff ε }. (2) Select a set of m ponts S n G (called a fork) where S = { 1, 2,..., m}. (3) Forks are only generated whch contan pels that msmatch each other. Ths means that they are selected n mage regons possessng hgh or certanly non-zero attenton scores, such as on edges or other salent features as reported earler [12]. In ths case the crtera s set so that at least one pel n the fork wll dffer wth one or more of the other fork pels by more than δ n one or more of ts rgb values.e. Fk ( ) - Fk ( j ) > δk, for some, j, k. (4) Defne the radus of the regon wthn whch fork comparsons wll be made as V (vew radus). Select another locaton y n the net frame regon R t + 1 wthn a radus V of. Defne the second fork S y = { y 1, y2,..., ym } where = y y (5) and y V. S y s a translated verson of S. The fork centred on s sad to match that at y ( S matches S y ) f all the colour components of correspondng pels are wthn threshold δ, k F k ( ) Fk ( y ) δk, k = r, g, b, = 1,2,..., m. (6) 2 All pels ( N = V ) wthn the vew radus are searched to fnd matches and the correspondng dsplacements are recorded as follows: Let a pel = (, y) components = ( r, g, b)

3 For the jth of N 1 < N matches defne the correspondng +1 dsplacement between and y as σ t j = ( σ p, σ q ) where σ p = p y p, σ q = q y, (7) q and the cumulatve dsplacements and match counts Γ as t + 1 ( ) ( ) = + σ j j = 1,..., Nl < N, (8) Γ( ) = Γ( ) + 1 where N 1 s the total number of matchng forks and N s the total number of matchng attempts. t+1 The dsplacement σ correspondng to pel averaged over the matchng forks s t+ 1 ( ) σ =. (9) Γ( ) Ths process s carred out for every pel n the canddate moton regon R t. All nternally msmatchng forks S wth m = 2 at each pel locaton are used for matchng between the two frames. The dsplacements are saved n the moton vector mapo and a copy no. MV MV 2.2. Moton segmentaton based on regon growng and matchng A regon growng and matchng process [15] s appled to obtan homogeneous regons wth moton nformaton. The moton vectors generated n the prevous secton tend to be assocated wth salent regons such as leadng and tralng edges of movng objects; non-salent homogeneous regons are not assgned moton vectors and for ths reason n the second stage a regon growng algorthm s ntroduced whch nfers moton n these homogeneous regons. Frst homogeneous regons are dentfed. Then the poston of the largest moton vector s taken as a seed for regon growng and the value of ths vector s assgned to pels n the homogeneous regon f ths translaton would lead to a pel match n the net frame. Ths s repeated for the same homogeneous regon to allow a dfferent moton vector to be assgned to the remanng part of the same homogeneous regon to obtan a match wth the net frame. Regons whch are changng shape would be affected by ths process. Seed moton vectors are rejected f ther locatons are not present n a dfference frame between the current and net frame. Ths elmnates the spurous analyss of statonary objects not present n the reference frame Object trackng Once the salent moton regons are obtaned after growng, ther correspondng centrods are then lnked across multple frames and used for trackng n the vdeo sequence. Regons are lnked f they overlap each other between successve frames. The areas of regons n each frame are normalzed accordng to perspectve n the mage and ordered n descendng order accordng to ther szes. A regon s selected for trackng f ts sze s larger than a certan threshold K. K s set to 1 for a mage and scaled accordng other mage szes. Trackng trajectores are plotted separately n the and y drectons aganst the frame number. 3. RESULTS AND DISCUSSION The attenton based regon growng algorthm s llustrated on varous data ncludng road scenes from an MPEG-7 traffc sequence [16] and a London Tran Staton pedestran sequence from PETS26 [17]. The parameters of all eperments are ε = 1, m = 2, δ = (4,4,4), T = 9. 1 frames from each vdeo are used for trackng. The varyng parameters are the vew radus, V and K. V s selected accordng to the mamum velocty epected n the clp Traffc sequence A traffc sequence of frame sze (Fgure 1) was analysed wth results shown n Fgure 2. The reference frame was obtaned by averagng over 1418 frames. Areas of canddate moton were obtaned by takng the dfference between each frame and the reference frame. The network of moton trajectores arses from the dvdng and rejonng of homogeneous regons as the moton progresses across frames. V was set to 2 ths correspondng to the mamum epected velocty of the objects. Regons contanng more than 25 pels were tracked. Fg.1. Frst frame (left) and reference frame (rght)

4 A B D C B D A C (a) (b) (c) (d) (e) (f) Fg.2. (a) X-frame plot; (b) Y-frame plot; (c) frame 7; (d) frame 14; (e) frame 66; (f) frame 85 Fgures 2(a) and 2(b) show the X and Y trajectores aganst frame number for 7 vehcles. The red crcle (A) ndcates the pont when the whte truck enters the scene at frame 7; the green crcle (B) ndcates the pont where the whte van starts beng tracked at frame 14; the blue crcle (C) ndcates the pont when the fnal truck enters the scene at frame 66; the orange crcle (D) ndcates the pont where the whte van ceases beng tracked. The moton estmaton process for each frame takes appromately 3 seconds whle the growng process takes around 1 seconds runnng n C++ on a 2.8 GHz machne wth 512 MB RAM London tran staton The trackng results are generated from a London Tran Staton sequence of frame sze The reference frame was obtaned by averagng over 1 frames taken from the vdeo. Moton trajectores n the drecton for each pedestran are plotted n Fgure 4. V was set to 15 ths beng the mamum epected object velocty n ths scene. The average tmes for moton estmaton and regon growng are 2 seconds and 3 seconds respectvely for each frame. Fg.3. Frst frame (left) and reference frame (rght) Two pedestrans occlude each other at frames 69 (B) and 88 (C). The effect of shadows s ndcated by a red crcle at frame 5 (A) where the trackng s dsturbed. Trackng stopped for the backpack pedestran when he ceased movng around frame 4 (D). Three frames and the correspondng regon growng maps are also shown n Fgure 5 to llustrate these effects. Shadows from both pedestrans overlap each and lead to the lnkng of the two regon centrods.

5 D B C A Fgure.4. X-frame plot Fg.5. (a) (b) (c) (d) (a) frame 5 and ts regon map; (b) frame 69 and regon map; (c) frame 88 and regon map; (d) frame 4 and regon map 4. CONCLUSIONS AND FUTURE WORK A salency based object trackng method has been proposed. The framework ncludes: an attenton based approach that etracts object dsplacement between frames by comparng salent regons; a regon growng technque that classfes moton regons accordng to moton nformaton etracted and a smple matchng process that assgns moton vectors to the classfed regons; fnally a trackng process whch lnks regon centrods between frames to form moton trajectores. The trackng method was llustrated on varous vdeo data wth a statonary background n both ndoor and outdoor scenes. The method does not requre a tranng stage or pror object models. More accurate object trackng may be obtaned by applyng a shadow dentfcaton technque. However, future work s amed at analyzng the network of moton trajectores to obtan more detaled object moton nformaton whch may also reveal dstngushng propertes for shadows and objects. Trackng through occlusons wll be developed. The proposed method wll be compared wth conventonal trackng technques such as mean-shft and

6 partcle flterng approaches. In addton more precse evaluaton wll be carred out usng ground truth data. 5. ACKNOWLEDGEMENT The project s sponsored by European Commsson Framework Programme 6 Network of Ecellence MUSCLE (Multmeda Understandng through Semantcs, Computaton and Learnng) [18]. [17] Performance Evaluaton of Trackng and Survellance (PETS), [18] Multmeda Understandng through Semantcs, Computaton and Learnng, 25. EC 6th Framework Programme, FP , 6. REFERENCES [1] C. Stauffer and W.E.L Grmson, Adaptve background mture models for real-tme trackng, n Proc. of CVPR, Ft. Collns, CO, USA, vol. 2, pp , June 23-25, [2] D. Comancu, V. Ramesh, and P. Meer, Real-tme trackng of non-rgd objects usng mean shft, n Proc. of CVPR, Hlton Head, SC, USA, vol. 2, pp , June 13-15, 2. [3] J. Sh and C. Tomas, Good features to track, n Proc. of CVPR, Seattle, WA, USA, pp , June 21-23, [4] M. Nethammer and A. Tannenbaum, Dynamc geodesc snakes for vsual trackng, n Proc. of CVPR, Washngton, DC, USA, vol. 1, pp , June 27-July 2, 24. [5] M. Isard and J. MacCormck, BraMBLe: a Bayesan multpleblob tracker, n Proc. of ICCV, Vancouver, Canada, vol. 2, pp , July 7-14, 21. [6] T. Vet, F. Cao, and P. Bouthemy, Probablstc parameter-free moton detecton, n Proc. of CVPR, Washngton, DC, USA, vol. 1, pp , June 27-July 2, 24. [7] M.J. Black and A.D. Jepson, Estmatng optcal flow n segmented mages usng varable-order parametrc models wth local deformatons, IEEE Trans. on PAMI, vol. 18, Issue 1, pp , Oct [8] M.J. Black and P. Anandan, The robust estmaton of multple motons: parametrc and pecewse-smooth flow felds, CVIU, vol. 63, Issue 1, pp , [9] P. Vola, M.J. Jones, and D. Snow, Detectng pedestrans usng patterns of moton and appearance, n Proc. of ICCV, Nce, France, vol. 2, pp , Oct. 23. [1] L. Itt, C. Koch, and E. Nebur, A model of salency-based vsual attenton for rapd scene analyss, IEEE Trans. on PAMI, vol. 2, Issue 11, pp , Nov [11] L. Itt and P. Bald, A prncpled approach to detectng surprsng events n vdeo, n Proc. of CVPR, San Dego, CA, USA, vol. 1, pp , June 25. [12] F.W.M. Stentford, An estmator for vsual attenton through compettve novelty wth applcaton to mage compresson, Pcture Codng Symposum, Seoul, pp , Aprl 21. [13] L. Wson, Detectng salent moton by accumulatng drectonally-consstent flow, IEEE Trans. on PAMI, vol. 22, No. 8, pp , Aug. 2. [14] S. Zhang and F.W.M. Stentford, Moton detecton usng a model of vsual attenton, n Proc. of ICIP, San Antono, USA, pp , Sept , 27. [15] S. Zhang and F.W.M. Stentford, Moton segmentaton usng regon growng and an attenton based algorthm, European Conference on Vsual Meda Producton, London, UK, Nov. 27. [16] MPEG-7 Content Set, Atlantc Cty, USA, Oct