Object Contour Tracking Using Multi-feature Fusion based Particle Filter

Transcription

1 Object Contour Tracng Usng Mult-feature Fuson based Partcle Flter Xaofeng Lu 1,3, L Song 1,2, Songyu Yu 1, Nam Lng 2 Insttute of Image Communcaton and Informaton Processng 1 Shangha Jao Tong Unversty, Shangha, Chna Department of Computer Engneerng 2 Santa Clara Unversty, Santa Clara, Calforna, USA School of communcaton and nformaton engneerng 3 Shangha Unversty, Shangha, Chna Emal:{luxaofeng_sjtu, song_l, syyu}@sjtu.edu.cn, {lsong, nlng}@scu.edu Abstract In ths paper, a novel object contour tracng framewor ntegratng ndependent mult-feature fuson object rough locaton and regon-based temporal dfferencng model s proposed. In our model, the object rough locaton tracng s realzed by color hstogram and Harrs corner features fuson method n partcle flter framewor. Thus t can acheve more robust tracng performance n many challenge scenes. And ths partcle flter framewor s based on our prevous CamShft guded partcle flter [7]. Wth the rough object locaton, effcent regon-based temporal dfferencng model s adopted for object contour detecton, then ths method s faster and more effectve compared to actve contour models or conventonal global temporal dfferencng models. Moreover, exact contour tracng result can be used to gude the partcle propagaton of next frame, to enable more effcent partcle redstrbutons and reducng partcle degeneraton. Expermental results demonstrate that ths proposed method s smple but effectve n object locaton and contour tracng. Keywords-object tracng; contour tracng; mult-feature fuson; partcle flter; harrs corner; regon-based temporal dfferencng I. INTRODUCTION Interested object tracng n vdeo sequences has always been a very mportant problem n several research and applcaton felds, such as behavor understandng, vdeo compresson, robotcs, survellance, and so on. Tracng accuracy, mult-object tracng, occluson and dsoccluson adaptaton are stll challenge ssues about object tracng [1]. In object tracng, the ey tas s how to assocate object accurate locaton over contnuous frames through dfferent methods. But most methods smplfy ths tas to represent the traced objects wth rectangles or ellpses [5][6][7]. In contrast, exact object contours are represented nstead of the rough locatons, especally for non-rgd targets [8]. And ths exact object locaton representaton s much more useful for further behavor analyss le abnormty detecton. Partcle flter s a popular optmal Bayesan algorthm for nonlnear and non-gaussan object tracng and t obtans the most lely posteror estmaton based on sequental Monte Carlo framewor. Due to ts multple hypothess property and ts good tracng result, we can apply ths method to any statespace model to estmate the trajectory of an target n vdeo sequences effectvely [2][3]. Followng the early wor of Isard [2], a varety of extended partcle flter framewors have been proposed for object tracng tass [4][5][6]. Reference [7] s our prevous wor, a CamShft guded partcle flter (CAMSGPF). In CAMSGPF, we enhance conventonal partcle flter algorthm by ntegratng mean shft algorthm to gude partcle propagaton to mprove the tracng accuracy n llumnaton changng and severe object deformng scenes. In recent years, object tracng methods based on a varety of object detecton per frame n a vdeo sequence have been proposed. And foreground extracton and bacground extracton are two man categores. In bacground modelng category, one of the most wdely used approaches s the Gaussan mxtures models (GMM) [9]. However, the Gaussan assumpton for every pxel ntensty dstrbuton s not always satsfed n dynamc scenes and GMM may fal n many scenes. In [10][11], a local bnary pattern (LBP) algorthm and ts extended model are proposed for object detecton. However, LBP algorthm s not so effcent snce ts senstvty to nose and long hstograms, then t can not realze real-tme detecton snce ts hgh computatonal load. At the same tme, temporal dfferencng method (TDM) [12][13] computes the dfference between adjacent two or more frames n a sequence to obtan the movng regons as a foreground extracton model. TDM s sutable for real-tme object detecton because of ts fast computaton. However, global TDM s easy to detect false object and cause the ghost n dynamc bacground scenes for bacground nose or llumnaton changng. The partcle flter method and bacground models are always dependng on the pxel or regon statstc features. And they wll be senstve to llumnaton, other objects nterference, or target deformaton. So the boundary-based object tracng models, mostly based on actve contour models (ACMs), are wdely adopted n vson applcaton. ACMs or snae model was frst proposed by Kass et al. [14] for target contour detecton. Ths method utlzed an energy mnmzng functon to a deformable contour so as to approach the true target

2 boundares. ACMs can be broadly classfed nto the parametrc models and geometrc models accordng to ther representaton and mplementaton dfference. Parametrc model based on regon segmentaton [15] teratvely searches ts new poston n current frame to reach object boundares by mnmzng the energy functon. However, ths model has heavy senstvty to ntal contour settngs and wll be confused the real objects wth smlar color regons. Geometrc models [16][17] are superor to parametrc models snce they can automatcally handle the topologcal changes of the propagatng curve. These geometrc models are motvated by a curve evoluton approach, and the deformaton s mplemented usng a level set method [18]. The man advantage of ACMs s that t can detect the deformable object contours accurately. However, they requre ntal contour settngs and need a very long computatonal tme to acheve the energy convergence. In general, the selected features, that can effcently dstngush the objects of nterest from the bacground, determne the performance of tracng methods. Regular features le color, moton, texture, and frame dfference are modeled by nds of tracng algorthms [6][7][8]. Recent approaches combne dfferent features for robust object tracng [19] or select the most dscrmnatve features from canddates onlne [20]. In ths paper, we propose a new object contour tracng model framewor to ntegratng ndependent mult-feature fuson object rough locaton and regon-based temporal dfferencng model. Our wor has three major contrbutons: 1) the object rough locaton tracng s realzed by color hstogram and Harrs corner features [23] fuson method n partcle flter framewor. Thus t can acheve more robust tracng performance n many challenge scenes le llumnaton changes, transformaton, short tme occluson; 2) regon-based temporal dfferencng model (RTDM) s adopted n our object contour detecton step, and ths effectve regon s the rough locaton tracng result. Thus, ths model s smpler and faster compared to snae or level set method. In addtonally, t has more effcent computng regon than conventonal global TDM, and t can tolerate complex bacground or smlar color objects to get more robust object contour; 3) contour tracng result, that s exact object boundary locaton, can be used to gude the current partcles propagaton, whch enables more effectve redstrbutons of partcles towards locatons assocated wth hgh probablty and reduces the partcle degeneraton possblty. In ths case, our new partcle flter framewor just uses a very small number of partcles (N=20 s used). Ths paper s organzed as follows. Secton II gves general descrpton of the algorthm framewor. Secton III descrbes the proposed algorthm scheme n detal. And Secton IV shows the expermental results. Conclusons are presented n Secton V. II. GENERAL FRAMEWORK DESCRIPTION The proposed object contour tracng general framewor can be dvded nto two procedures,.e. the object tracng ntal procedure and the exact contour-guded mult-feature fuson object contour tracng procedure as show n Fg. 1. In ntal procedure, we obtan the object rough locaton wth a rectangle manually as a reference template. Based on ths rough locaton, the state vectors ntalzaton of partcles s fnshed, and we can acqure the reference color hstogram. Smlarly, the object boundary n rough locaton s computed by Canny operator [21] for Harrs corner extracton [23] to form reference Harrs corner feature vector. Next, object ntal contour s used to gude partcle propagaton n current frame. Independent color and Harrs corner features are ntegrated by Bayesan flters to produce a detaled representaton of the object of nterest. When ths object rough regon s obtaned, the model apples the RTDM to canddate locaton to get exact contour of ths traced object. Vdeo sequence nput Object tracng ntal procedure for reference color and corner templates Contour-guded mult-feature fuson partcle flter contour tracng procedure Fgure 1. Bloc dagram of the proposed general algorthm framewor. III. THE PROPOSED ALGORITHM PROCEDURES These two procedures about object locaton ntalzaton and object contour tracng are descrbed n detal as follows. A. Object Contour Intal Algorthm Procedure Fg. 2 shows the bloc dagram of the object contour ntal procedure. Frst, the ntal frame s captured and we defne the traced nterested object wth a rectangle manually. Then, we begn to ntalze the partcle flter state vector based on our prevous CAMSGPF framewor [7] n that rectangle area. 1) State vectors of partcles ntalzaton: The canddate s modeled by a state vector as n (1). And we defne the rectangle centered at coordnate ( xc, y c) wth wdth w and heght h as the ntal rough locaton. Equaton (2) and (3) show the nonlnear state transton model f and observaton model E. Then x presents the object state and z presents the object observaton at tme, wth n and θ denotng process and observaton nose. The posteror densty functon p( x z 1: ) s approxmated by a set of N samples (partcles) N x, w. Gven the posteror densty functon = wth weghts { } 1

3 ( 1 z1: 1) p x, under the frst order Marov assumpton, regressve Bayesan theory s optmal. (,,, ) c c T x = x y w h (1) (, ) x = f x n (2) 1 1 (, ) z = E x θ (3) obtan Harrs corner feature, and that s nsenstve to llumnaton changng, rotaton or zoom operaton. Generally, Harrs operator s realzed by calculatng each pxel s gradent and judge the absolute gradent values n two drectons. Harrs corner detecton algorthm s defned as follows: 2 g x gg x y M = G( σ ) 2 gg x y g y (6) R M M 2 = det( ) Tr ( ), (7) Where G( σ ) s a Gaussan functon wth devaton σ ; gx and g y s the frst dervatves of pxel ( x, y ).det( M ) means the determnant of matrx M, and Tr denotes the trace of matrx M. And R s the corner strength functon at one pxel. If the strength s above a partcular threshold, then a Harrs corner s found. The Harrs corner s presented as Fg. 3(f). Fgure 2. Bloc dagram of othe object ntal algorthm procedure. In partcle flter framewor, each sample s weghted n terms of the observaton wth probablty computng formula (4). In ths method, the proposal mportance samplng dstrbuton functon q( x, x0: 1 z) s substtuted by pror p x x. And the lelhood dstrbuton s dstrbuton ( 0: 1 ) presented as (5). Here D[] s Bhattacharyya smlarty metrc [24] to compare reference template h 0 and canddate template hx [ ]. [ ( ) ( 0: 1 )]/ ( 0: 1, ) 2 ( ) exp( λ 0, ( ) ) w p z x p x x q x x z (4) p z x D h h x 2) Harrs corner computng: Frst, we utlze a temporal dfferencng [13] between the adjacent two frames to obtan the subtracted result as Fg.3 (c) shown. However, the dfferencng boundary s not accurate enough. To solve ths problem, we adopt Canny edge detector for current frame, and apply a logc AND operaton wth dfferencng boundary mage to obtan the fnal moton contour mage as Fg. 3(d)(e). In order to reduce the computatonal cost for measurng contnued contours, we choose a set of Harrs feature ponts to represent the object contour that can eep sgnfcant object contour nformaton. Harrs corner detector [23] s appled to (5) Fgure 3. Object tracng ntal procedure: (a) and (b) are adjacent frames; (c) result of temporal dfferencng model; (d) result of Canny edge detecton; (e) result of AND logc operaton; (f) result of Harrs corner detecton. B. Object Contoure Tracng Algorthm Procedure 1) Partlce propagaton: When we obtan the ntal reference color hstogram template and reference Harrs corner set, the system begns to gettng nto the object contour tracng algorthm procedure as Fg. 5 shown. Frst, the dynamc state transton model s defned as (8). Ths propagaton mechansm satsfed to a standard 2nd order autoregressve process s an effcent partcles evoluton. n 1 xˆ = x 1 + x l x ( l+ 1) + n (8) n l= 1 2) Color weght estmaton: After partcle propagaton, we can get N canddate partcle samples. In our prevous wor CAMSGPF, we employ Bhattacharyya dstance and Bhattacharyya coeffcent [24] to present the smlarty measurement between one canddate and the reference color template as n (9) shown, where pref and q can are two hstograms. From ths smlarty measurement, we can use (10) to defne the color weght of ths canddate. And σ s the varance determned emprcally.

4 N color ref o can = 1 d = 1 p ( y ) q ( y ) (9) weght j color = πσ σ (10) 2 2 (1/ 2 )exp( dcolor / 2 ) 3) Harrs corner weght estmaton: For every object canddate, we must compare the smlarty bewteen canddate Harrs corner set and reference Harrs corner set. In ths procedure, we use smlarty Hausdorff dstance [22] to represent ths dstance between two pont sets. The Harrs corners are changng ther postons and number, and smularty Hausdorff dstance does not need to establsh the pont-to-pont relatonshps. To two fnte pont sets, A={ a 1, a 2,..., a n } and B{ b 1, b 2,..., b n }, the Hausdorff dstance s defned wth (11), (12) and (13) [22]. Where d( A, B) s the Hausdorff dstance between set A and set B, and habs (, ) the dstance from set A to B. defnes the Eucldean norm between two corner set, d( A, B) = max[ h( A, B), h( B, A)] (11) t 6) Object exact locaton guded partcle propagaton: If the random nose n partcle propagaton s small, the partcle resample mpovershment wll be obvous. On the contray, f the random nose s large, t wll need more partcles to realze exact state predcton [6]. n = r n' = ( a wr + b hr ) n' (16) hab (, ) = maxmn a b j (12) a A b B hba (, ) = maxmn b a j (13) b B a A Smultaneously, we can defne the Harrs corner weght of the canddate from the smlarty measurement based on Hausdorff dstance as (14) shown comparng to color weght. weght j contour = πσ σ (14) 2 2 (1/ 2 )exp( dcontour / 2 ) weght = αweght + β weght (15) j j j t t ( color ) t ( contour) 4) Object rough regon estmaton: From (15), we can obtan the feature weght of ths current canddate. α and β mean constant factors. Ths mult-feature fuson scheme combnes color and corner feature wth dfferent effectve feature smlarty measurements, and t acheves object locaton that can be tolerant to llumnaton and object deformaton. Through weght sortng of all partcles, t s easy to get the object rough locaton regon n sequence. 5) Regon-based TDM: As Fg. 4 shown, here we apply a regon-based TDM algorthm to ths rough object locaton to obtan the object bnary regon mage. And the exact object contour can be receved followed by morphology operaton and contour fnder. Generally, the regon sze we need to compute s much smaller than the whole mage, So ths fast algorthm can really reduce the computatonal cost of whole algorthm structure. Fgure 4. Regon-based TDM results: (a) current frame. (b) object bnary regon mage. (c) object rough locaton and exact contour. Fgure 5. Object contour tracng algorthm procedure. Equaton (16) s the new contour scale guded partcle propagaton nose model we proposed, where n s new nose vector, r s exact contour scale coeffcent, and n ' meets zeromean Gaussan dstrbuton. In ths model, we defne a and b as vertcal and horzontal coordnate nose coeffcent emprcally, and they satsfy the defnton a+ b = 1 ( a = 0.7, b = 0.3 ). Then the exact contour object locaton trends can be used to gude the current partcles propagaton, whch enables a more effectve redstrbutons of partcles towards locatons assocated wth hgh probablty and reduce the partcle degeneraton possblty. In ths case, our new partcle flter framewor just uses a very small number of partcles (N=20 s used) to realze exact object rough locaton and exact contour tracng. IV. EXPERIMENTANL RESULTS In ths secton, to verfy the performance of ths proposed method, some common test vdeo sequences from Toronto Unversty and our own datasets are used. And the conventonal partcle flter (CPF) [3] and CAMSGPF [7] are also performed for comparson. A. Mult-feature Fuson Rough Object Locaton Results Fg. 6 compares the object rough face locaton tracng results presented by CPF [3], CAMSGPF [7] and our proposed method. In ths Davd2_WMM_xvd vdeo sequence, llumnaton on object face s changng rapdly, and object scale s changng when Davd movng around the camera. We can

5 see that CPF wll lose traced object qucly, and CAMSPF can not deal wth llumnaton changng very well. Whle our proposed method can tolerate object scale and llumnaton changes durng the whole tracng process by fuson color and corner features. Another advantage of our proposed mult-feature fuson tracng strategy s the tracng robustness to the short occluson,.e. our proposed model can qucly re-obtan satsfactory tracng results even after a short whole object occluson as Fg. 7 shown. One can observe that CPF method wll qucly lose the traced object when same color feature object appears n frame 186. On the other hand, when a short occluson happens, CAMSGPF method can not recover the target and turn nto tracng error. But to our proposed method, short occluson wll not result n loss of trac, and t can retarget the object exactly after occluson. B. Contour Tracng Results Wth mult-feature fuson partcle mportance evaluaton scheme, we can obtan more exact rough object locaton than conventonal partcle flter. Then, based on ths rough locaton, we further assess the algorthm performance on object exact contour extracton model. In ths proposed algorthm, we apply the exact regon-based temporal dfferencng model to extract object contour as Secton III part B descrbed. Ths model s smpler and faster compared to snae or level set method. In addtonally, t s more effcent than conventonal global TDM to get more robust object contour. Fg. 8 presents the Lu_1 vdeo sequence tracng result. We ntalze the tracng object wth a green rectangle n frame 1, and ths method s capable to get object rough locaton wth green rectangle and exact contour wth blue connecton lne accurately as frame 65, frame 135, frame 228 shown. And n another experment for Zhang_2 vdeo sequence, the llumnaton s always changng and the tracng results are presented n Fg. 9. One can observe that n ths sequence, ths method can tolerate object deformaton, but the contour n face regon s not very good for the severe llumnaton affect to temporal dfferencng model. Frame 1 Frame 45 Frame 81 Frame 120 Fgure 6. Davd2_WMM_xvd vdeo sequence: Blue ellpse s the object tracng result from CPF; Green ellpse s the tracng result from CAMSGPF; And Red ellpse s the tracng result of our proposed. Fgure 8. Object rough locaton and exact contour tracng n room scene for Lu_1 vdeo sequence. Frame 1 Frame 186 Fgure 9. Object rough locaton and exact contour tracng n wea llumnaton scene for Zhang_2 vdeo sequence. Frame 216 Frame 224 Fgure 7. Anvar1GlobalNormal vdeo sequence: Blue ellpse s the object tracng result from CPF; Green ellpse s the tracng result from CAMSPF; And Red ellpse s the tracng result of our proposed. V. CONCLUSION In ths paper, a new object contour tracng framewor s proposed. Ths method combnes mult-feature fuson strategy based object rough locaton and exact regon-based object contour extracton for accurate and robust object contour tracng. Specfcally, n our proposed model, the object rough locaton s realzed by color hstogram and Harrs corner features fuson method n partcle flter framewor. Regon-

6 based TDM s appled for exact contour detecton, whch s smpler than actve contour models. And n turn, object exact contour scale s also appled to gude new partcle propagaton procedure. The expermental results demonstrate the mproved performance of ths proposed model. In the future wor, we are plannng to embed accurate object local moton nformaton nto contour exact step to mprove the contour edge accuracy based on TDM. And t s also need to realze tradtonal actve contour model for algorthm comparson. ACKNOWLEDGMENT Ths research was supported n part by NSFC ( ), NSFC ( ), 973 Program (2010CB731400) and the 111 project. REFERENCES [1] Alper Ylmaz, Omar Javed, and Mubara Shah, Object tracng: a survey, ACM Computng Surveys, Vol.38, No.4, Artcle 13, Dec [2] Isard M., Blae A., CONDENSATION-condtonal densty propagaton for vsual tracng, Internatonal Journal of Computer Vson, 29(1), pp.5-28, [3] A. Doucet, S. Godsll, and C. Andreu, On sequental Monte Carlo samplng methods for Bayesan flterng, Statstcs and Computng, 10(3), pp , [4] M. S. Arulampalam, S. Masell, N. Gordon, and T. Clapp, A tutoral on partcle flters fot onlne nonnear/non-gaussan Bayesan tracng, IEEE Trans. on Sgnal Processng, Vol.50(2), pp , [5] Zulfqar Hasan Khan, Irene Yu-Hua Gu, and Andrew G. Bachouse, A robust partcle flter-based method for tracng sngle vsual object through complex scenes usng dynamcal object shape and appearance smlarty, J Sgn Process Syst, Vol.65, pp.63-79, [6] S.K. Zhou, R. Chellappa and B. Moghaddam, Vsual tracng and recognton usng appearance-adaptve models n partcle flter, IEEE Trans. on Image Processng, Vol.13, No.11, Nov [7] Z. Wang, X. Yang, Y. Xu, and S. Yu, CamShft guded partcle flter for vsual tracng, ACM Pattern Recognton Letters, Vol.30, pp , Mar [8] Lng Ca, Le He, Yamasta Taayosh, Yren Xu, Yumng Zhao, and Xn Yang, Robust contour tracng by combnng regon and boundary nformaton, IEEE Trans. on Crcuts and System for Vdeo Technology, n press. [9] C. Stauffer and W.E.L. Grmsom, Adaptve bacground mxture models for real-tme tracng, Proc. IEEE CS Conf. Computer Vson and Pattern Recognton, Vol.2, pp , [10] M. Hela and M. Petanen, A texture-based method for modellng the bacground and detectng movng objects, IEEE Trans. on Pattern Analyss and Machne Intellgence, Vol.28, No.4, pp , [11] Xue Gengjan, Sun Jun, and Song L, Dynamc bacground subtracton based on spatal extended center-symmetrc local bnary pattern, IEEE Internatonal Conference on Multmeda & Expo, ICME, pp , July, [12] Q. Ca and J. K. Agarwal, Tracng human moton n structured envronments usng a dstrbuted-camera system, IEEE Trans. on Pattern Analyss and Machne Intellgence, Vol.21, No.11, pp , Nov [13] A. J. Lpton, H. Fujyosh, and R. S. Patl, Movng target classfcaton and tracng from real-tme vdeo, n Proc. of the IEEE Worshop on Applcatons of Computer Vson, pp.8-14, Oct [14] M. Kass, A. Wtn, and D. Terzopoulos, Snaes: actve contour models, Internatonal Journal of Computer Vson, Vol.1(4), pp , [15] C. Xu and J. L. Prnce, Snaes shapes, and gradent vector flow, IEEE Trans. on Image Processng, Vol.7(3), March, [16] N. Paragos and R. Derche, Geodesc actve contours and level sets for the detecton and tracng of movng objects, IEEE Trans. Pattern Analyss and Machne Intellgence, Vol.22(3), pp , [17] R. Goldenberg and R. Kmmel, Fast geodesc actve contours, IEEE Trans. Image Processng, 10(10), pp , 2001 [18] J. A. Sethan, Level set methods, Cambrdge Un. Press, [19] V. Taala and M. Petnen, Mult-object tracng usng color, texture and moton, n Proc. IEEE Conf. Computer Vson and Pattern Recognton, pp.1-7, [20] R. Collns, Y. Lu, and M. Leordeanu, On-lne selecton of dscrmnatve tracng features, IEEE Trans. Pattern Analyss and Machne Intellgence, Vol.27, No.10, pp , [21] J. Canny, A computatonal approach to edge detecton, IEEE Trans. on Pattern Analyss and Machne Intellgence, Vol.8, pp , [22] D. P. Huttenlocher, G. A. Klanderman, and W. A. Rucldge, Comparng mages usng the Hausdorff dstance, IEEE Trans. on Pattern Analyss and Machne Intellgence, Vol.15, pp , [23] C. Harrs and M. J. Stephens, A combned corner and edge detector, In Alvey Vson Conference, pp , [24] T. Zhang, S. Fe, X. L and H. Lu, An mproved partcle flter for tracng color object, IEEE Computer Socety Internatonal Conf. on Intellgent Computaton Technology and Automaton, pp , 2008.