Scale and Orientation Adaptive Mean Shift Tracking

Transcription

1 Scale and Orentaton Adaptve Mean Shft Trackng Jfeng Nng, Le Zhang, Davd Zhang and Chengke Wu Abstract A scale and orentaton adaptve mean shft trackng (SOAMST) algorthm s proposed n ths paper to address the problem of how to estmate the scale and orentaton changes of the target under the mean shft trackng framework. In the orgnal mean shft trackng algorthm, the poston of the target can be well estmated, whle the scale and orentaton changes can not be adaptvely estmated. Consderng that the weght mage derved from the target model and the canddate model can represent the possblty that a pxel belongs to the target, we show that the orgnal mean shft trackng algorthm can be derved usng the zero th and the frst order moments of the weght mage. Wth the zero th order moment and the Bhattacharyya coeffcent between the target model and canddate model, a smple and effectve method s proposed to estmate the scale of target. Then an approach, whch utlzes the estmated area and the second order center moment, s proposed to adaptvely estmate the wdth, heght and orentaton changes of the target. Extensve experments are performed to testfy the proposed method and valdate ts robustness to the scale and orentaton changes of the target. Keywords: object trackng, mean shft, moment, scale and orentaton estmaton Correspondng author. Le Zhang s wth the Bometrcs Research Center, Dept. of Computng, The Hong Kong Polytechnc Unversty, Kowloon, Hong Kong, Chna. Emal: cslzhang@comp.polyu.edu.hk. Ths work s supported by the Natonal Scence Foundaton Councl of Chna under Grants , and 635 and the Chnese Unversty Scentfc Fund under Grant No.QN909. Jfeng Nng s wth the College of Informaton Engneerng, Northwest A&F Unversty, Yanglng, Chna, and the Bometrcs Research Center, Dept. of Computng, The Hong Kong Polytechnc Unversty, Kowloon, Hong Kong, Chna and the State Key Laboratory of Integrated Servce Networks, Xdan Unversty, X an, Chna. Emal: jf_nng@sna.com. Davd Zhang s wth the Bometrcs Research Center, Dept. of Computng, The Hong Kong Polytechnc Unversty, Kowloon, Hong Kong, Chna. Emal: csdzhang@comp.polyu.edu.hk. Chengke Wu s wth the State Key Laboratory of Integrated Servce Networks, Xdan Unversty, X an, Chna. Emal: ckwu@xdan.edu.cn.

2 . Introducton Real-tme object trackng s a crtcal task n computer vson, and many algorthms have been proposed to overcome the dffcultes arsng from nose, occlusons, clutters, and changes n the foreground object and/or background envronment [4]. Among varous trackng methods, the mean shft trackng algorthm s a popular one due to ts smplcty and effcency. The mean shft algorthm was orgnally developed by Fukunaga and Hostetler [] for data analyss, and later Cheng [3] ntroduced t to the feld of computer vson. Bradsk [6] modfed t and developed the Contnuously Adaptve Mean Shft (CAMSHIFT) algorthm for face trackng. Comancu and Meer successfully appled mean shft algorthm to mage segmentaton [8] and object trackng [7, 9]. Some optmal propertes of mean shft were dscussed n [3, 5]. In the classcal mean shft trackng algorthm [9], the estmaton of scale and orentaton changes of the target s not solved. Although t s not robust, the CAMSHIFT algorthm [6], as the earlest mean shft based trackng scheme, could actually deal wth varous types of movements of the object. In CAMSHIFT, the moment of the weght mage determned by the target model was used to estmate the scale (also called area) and orentaton of the object beng tracked. Based on Comancu et al s work n [9], many trackng schemes [0,, 7, 8, 3] were proposed to solve the problem of target scale and/or orentaton estmaton. Collns [0] adopted Lndeberg et al s scale space theory [9, 0] for kernel scale selecton n mean-shft based blob trackng. However, t cannot handle the rotaton changes of the target. An EM-shft algorthm was proposed by Zvkovc and Kröse n [], whch smultaneously estmates the poston of the local mode and the covarance matrx that can approxmately descrbe the shape of the local mode. In [3], a dstance transform based asymmetrc kernel s used to ft the object shape through a scale adaptaton followed by a segmentaton process. Hu

3 et al [7] developed a scheme to estmate the scale and orentaton changes of the object by usng spatal-color features and a novel smlarty measure functon [, 6]. In ths paper, a scale and orentaton adaptve mean shft trackng (SOAMST) algorthm s presented under the mean shft framework. Unlke CAMSHIFT, whch uses the weght mage determned by the target model, the proposed SOAMST algorthm employs the weght mage derved from the target model and the target canddate model n the target canddate regon to estmate the target scale and orentaton. Such a weght mage can be regarded as the densty dstrbuton functon of the object n the target canddate regon, and the weght value of each pxel represents the possblty that t belongs to the target. Usng ths densty dstrbuton functon, we can compute the moment features and then estmate effectvely the wdth, heght and orentaton of the object based on the zero th order moment, the second order center moment and the Bhattacharyya coeffcent between target model and target canddate model. The expermental results demonstrate that SOAMST can deal wth varous movements of the tracked object flexbly and robustly. The rest of the paper s organzed as follows. Secton ntroduces the classcal mean shft algorthm. Secton 3 analyzes the moment features of the target canddate regon and then descrbes n detal the proposed SOAMST approach. Secton 4 performs extensve experments to test the proposed SOAMST algorthm n comparson wth state-of-the-art schemes. Secton 5 concludes the paper.. Mean Shft Trackng Algorthm. Target Representaton In object trackng, a target s usually defned as a rectangle or an ellpsodal regon n the mage. Currently, a wdely used target representaton s the color hstogram because of ts ndependence of scalng and rotaton and ts robustness to partal occlusons [9, ]. Denote 3

4 by { } x = n the normalzed pxels n the target regon, whch s supposed to be centered at the orgn pont and have n pxels. The probablty of the feature u (u=,,, m) n the target model s computed as [9] { qˆ } q= ˆ u u= m n qˆ ( x ) * ( x * u = C k δ b ) u = where ˆq s the target model, q ˆu s the probablty of the u th () element of ˆq, δ s the x * Kronecker delta functon, ( ) b assocates the pxel * x to the hstogram bn, and k(x) s an sotropc kernel profle. Constant C s a normalzaton functon defned by n = * ( ) C = k x () Smlarly, the probablty of the feature u n the target canddate model from the canddate regon centered at poston y s gven by { } ( ) pˆ ( y) pˆ y = u u= m n h y x pˆ u (y) = Ch k δ b( x) u = h (3) C h n h y x = k (4) = h where ˆp( y ) s the target canddate model, ˆ ( y) p s the probablty of the u th element of u ˆp( y ), { x } = n h are pxels n the target canddate regon centered at y, h s the bandwdth and C h s the normalzaton functon whch s ndependent of y [9]. In order to calculate the lkelhood of the target model and the canddate model, a metrc based on the Bhattacharyya coeffcent [] s defned by usng the two normalzed hstograms pˆ (y) and qˆ as follows 4

5 [ pˆ ( y), qˆ ] = ˆ ( y) m ρ p u qˆ u (5) u= The dstance between pˆ (y) and qˆ s then defned as [ pˆ ( y),qˆ ] = ρ[ pˆ ( y),qˆ ] d (6). Mean Shft Mnmzng the dstance d ˆ( ) p y,qˆ Bhattacharyya coeffcent ρ ˆ( ) p y,qˆ n Eq. (6) s equvalent to maxmzng the n Eq. (5). The optmzaton process s an teratve process and s ntalzed wth the target poston, denoted by y 0, n the prevous frame. By usng the Taylor expanson around p ( ) ˆ u y 0, the lnear approxmaton of the Bhattacharyya coeffcent ρ ˆ( ) p y,qˆ n Eq. (5) can be obtaned as: h ρ [ pˆ(y),qˆ ] pˆ u (y 0 ) qˆ u + (7) m n C h y x w k u= u= h where w m = qˆ pˆ u ( y ) u= u 0 [ b( x ) u] δ (8) Snce the frst term n Eq. (7) s ndependent of y, to mnmze the dstance n Eq. (6) s to maxmze the second term n Eq. (7). In the mean shft teraton, the estmated target moves from y to a new poston y, whch s defned as n h y x x w g = = h y (9) n h y x w g = h When we choose the kernel k(x) wth the Epanechnkov profle, there s g(x)=-k(x)=, and Eq. (9) can be reduced to [9] 5

6 nh = x w y = (0) nh w By usng Eq. (0), the mean shft trackng algorthm fnds n the new frame the most smlar regon to the object. From Eq. (0) t can be observed that the key parameters n the mean shft trackng = algorthm are the weghts w. In ths paper we wll focus on the analyss of w, wth whch the scale and orentaton of the tracked target can be well estmated, and then a scale and orentaton adaptve mean shft trackng algorthm can be developed. 3. Scale and Orentaton Adaptve Mean Shft Trackng In ths secton, we frst analyze how to calculate adaptvely the scale and orentaton of the target n sub-sectons 3. ~ 3.5, then n sub-secton 3.6, a scale and orentaton adaptve mean shft trackng (SOAMST) algorthm s presented. The enlargng or shrnkng of the target s usually a gradual process n consecutve frames. Thus we can assume that the scale change of the target s smooth and ths assumpton holds reasonably well n most vdeo sequences. If the scale of the target changes abruptly n adjacent frames, no general trackng algorthm can track t effectvely. Wth ths assumpton, we can make a small modfcaton of the orgnal mean shft trackng algorthm. Suppose that we have estmated the area of the target (the area estmaton wll be dscussed n sub-secton 3.) n the prevous frame, n the current frame we let the wndow sze or the area of the target canddate regon be a lttle bgger than the estmated area of the target. Therefore, no matter how the scale and orentaton of the target change, t should be stll n ths bgger target canddate regon n the current frame. Now the problem turns to how to estmate the real area and orentaton from the target canddate regon. 6

7 3. The Weght Images n Target Scale Changng (a) (b) (c) (d) (e) (f) (g) (h) () (j) (k) Fg. : Weght mages n CAMSHIF [6] and mean shft trackng [9] algorthms when the object scale changes. (a) A syntheszed target wth three gray levels. (b) A target canddate wndow that s bgger than the target. (c), (f) and () are the target canddate regons enclosed by the target canddate wndow (dashed box) when the scale of the target decreases, keeps nvarant and ncreases, respectvely. (d), (g) and (j) are respectvely the weght mages of the target canddate regons n (c), (f) and () calculated by CAMSHIFT. (e), (h) and (k) are respectvely the weght mages of the target canddate regons n (c), (f) and () calculated by mean shft trackng. In the CAMSHIFT and the mean shft trackng algorthms, the estmaton of the target locaton s actually obtaned by usng a weght mage [0, 4]. In CAMSHIFT, the weght mage s determned usng a hue-based object hstogram where the weght of a pxel s the probablty of ts hue n the object model. Whle n the mean shft trackng algorthm, the weght mage s defned by Eq. (8) where the weght of a pxel s the square root of the rato of ts color probablty n the target model to ts color probablty n the target canddate model. Moreover, t s not accurate to use the weght mage by CAMSHIFT to estmate the locaton 7

8 of the target, and the mean shft trackng algorthm can have better estmaton results. That s to say, the weght mage n the mean shft trackng algorthm s more relable than that n the CAMSHIFT algorthm. As n the CAMSHIFT algorthm, n the SOAMST scheme to be developed, the scale and orentaton of the target wll be estmated by usng the moment features [4-6] of the weght mage. Snce those moment features depend only on the weght mage, a properly calculated weght mage could lead to accurate moment features and consequently good estmates of the target changes. Therefore, let s analyze the weght mages n the CAMSHIFT and mean shft trackng methods n order for the development of the SOAMST algorthm. As mentoned at the begnnng of Secton 3, we wll track the target n a larger canddate regon than ts sze to ensure that the target wll be wthn ths canddate regon when the trackng process ends. Wth ths strategy, let s compare the weght mages n CAMSHIFT and mean shft trackng under dfferent scale changes by usng the followng experments. Fgure -(a) shows a syntheszed target that has three gray levels. Fgure -(b) shows the canddate regon that s a lttle bgger than the target. Fgures -(c), (f) and () are the trackng results when the scale of the syntheszed target decreases, keeps nvarant and ncreases, respectvely. Fgures -(d), (g) and (j) llustrate the weght mages calculated by the CAMSHIFT algorthm n the three cases, whle Fgures -(e), (h) and (k) llustrate the weght mages calculated by the mean shft trackng algorthm n the three cases. From Fgure, we can see clearly the dfference of the weght mages between CAMSHIFT and mean shft trackng. Frst, the weght mage n the CAMSHIFT algorthm s constant and t only depends on the target model, whle the weght mage n the mean shft trackng algorthms wll change dynamcally wth the scale changes of the target. Second, the weght mage s closely related to the target scale change n mean shft trackng. The closer the real scale of the target s to the canddate regon, the better the weght mage approaches to 8

9 . That s to say, the weght mage n mean shft trackng can be a good ndcator of the scale change of the target. However, the weght mage n CAMSHIFT does not reflect ths. Based on the above observaton and analyss, we could consder the weght mage n the mean shft trackng algorthm as a densty dstrbuton functon of the target, where the weght value of a pxel reflects the possblty that t belongs to the target. In the followng sectons, we can see that the scale and orentaton of the target can be well estmated by usng ths densty dstrbuton functon together wth the moment features of the weght mage. 3. Estmatng the Target Area Snce the weght value of a pxel n the target canddate regon represents the probablty that t belongs to the target, the sum of the weghts of all pxels,.e., the zero th order moment, can be consdered as the weghted area of the target n the target canddate regon: M n = ( x ) = w () In mean shft trackng, the target s usually n the bg target canddate regon. Due to the exstence of the background features n the target canddate regon, the probablty of the target features s less than that n the target model. So Eq. (8) wll enlarge the weghts of target pxels and suppress the weght of background pxels. Thus, the pxels from the target wll contrbute more to target area estmaton, whle the pxels from the background wll contrbute less. Ths can be clearly seen n Fgures -(e), -(h) and -(k). On the other hand, the Bhattacharyya coeffcent (referrng to Eq. (5)) s an ndcator of the smlarty between the target model ˆq and the target canddate model ˆp ( y ). A smaller Bhattacharyya coeffcent means that there are more features from the background and fewer features from the target n the target canddate regon, vce versa. If we take M as the In the remanng of the paper, for the convenence of expresson we wll only use Bhattacharyya coeffcent to represent the Bhattacharyya coeffcent between the target model and the target canddate model. 9

10 estmaton of the target area, then accordng to Eq. (), when the weghts from the target become bgger, the estmaton error by takng M as the area of the target wll be bgger, vce versa. Therefore, the Bhattacharyya coeffcent s a good ndcator of how relable t s by takng M as the target area. Table lsts the real area of the target n Fgure and the estmaton error by takng M as the target area. We can see that wth the ncrease of the Bhattacharyya coeffcent, the estmaton accuracy by takng M as the target area wll also ncrease (e.g., the estmaton error wll decrease). Based on the above analyss, we see that the Bhattacharyya coeffcent can be used to adjust M n estmatng the target area, denoted by A. We propose the followng equaton to estmate t: A= c( ρ) M () where c(ρ) s a monotoncally ncreasng functon wth respect to the Bhattacharyya coeffcent ρ ( 0 ρ ). As can be seen n Fgures -(e), -(h) and -(k) and Table, M s always greater than the real target area and t wll monotoncally approach to the real target area wth ρ ncreasng. Thus we requre that c(ρ) should be monotoncally ncrease and reach maxmum when ρ s. Such a correcton functon c(ρ) s possble to shrnk M back to the real target scale. There can be alternatve canddate functons of c(ρ), such as lnear functon c(ρ)=ρ, Gaussan functon, etc. Here we choose the exponental functon as c(ρ) based on our expermental experence 3 : ρ c( ρ) = exp σ (3) From Eqs. () and (3) we can see that when ρ approaches to the upper bound,.e., when the target canddate model approaches to the target model, c(ρ) approaches to and n 3 By our expermental experence, both exponental and Gaussan functons can acheve satsfyng results, and we choose the former here for smplcty. 0

11 ths case t s more relable to use M as the estmaton of target area. When ρ decreases,.e., the canddate model s not dentcal to the target model, M wll be much bgger than the target area but c(ρ) s less than so that A can avod beng based too much from the real target area. When ρ approaches to 0,.e., the tracked target gets lost, c(ρ) wll be very small so that A s close to zero. Table. The area estmaton (pxels) of the target under dfferent scale changes by the proposed method. Trackng result Fg. (e) Fg. (h) Fg. (k) Real area of target Background area Bhattacharyya coeffcent Estmated area A under dfferent σ and the relatve estmaton error (%) n comparson wth M. M % % 40 0% σ = % % 40 0% σ = % % 40 0% σ = % % 40 0% σ = % % 40 0% Table lsts the area estmaton results of the target by usng Eq. () under dfferent scale changes n Fgures -(e), -(h) and -(k). Though an optmal value of σ should be adaptve to the vdeo content, by our expermental experences t was found that when the target model s approprately defned (contanng not too many background features), settng σ between and can acheve very robust trackng results for most of the testng vdeo sequences. 3.3 The Moment Features n Mean Shft Trackng In ths sub-secton, we analyze the moment features n mean shft trackng and then combne them wth the estmated target area to further estmate the wdth, heght and orentaton of the target n the next sub-secton. Lke n CAMSHIFT, we can easly calculate the moments of the weght mage as follows: nh nh (4) M = wx M = wx 0, 0, =

12 nh nh nh 0 = x,, 0 = x,, = x, x, = = = (5) M w M w M w where par (x,, x, ) s the coordnate of pxel n the canddate regon. Comparng Eq. (0) wth Eqs. () and (4), we can fnd that y s actually the rato of the frst order moment to the zero th order moment: ( ) y = x,x = ( M / M, M / M ) (6) 0 0 where ( x, x ) represents the centrod of the target canddate regon. The second order center moment could descrbe the shape and orentaton of an object. By usng Eqs. (0), (), (5) and (6), we can convert Eq. (9) to the second order center moment as follows μ = M / M x μ = M / M x x μ = M / M x (7) Eq. (7) can be rewrtten as the followng covarance matrx n order to estmate the wdth, heght and orentaton of the target: Cov μ μ 0 = μ μ 0 (8) 3.4 Estmatng the Wdth, Heght and Orentaton of the Target By usng the estmated area (sub-secton 3.) and the moment features (sub-secton 3.3), the wdth, heght and orentaton of the target can be well estmated. The covarance matrx n Eq. (8) can be decomposed by usng the sngular value decomposton (SVD) [] as follows 0 T u u λ u u Cov = U S U = u u 0 λ u u T (9) where U u u = u u and λ 0 S =. 0 λ λ and λ are the egenvalues of Cov. The vectors (u, u ) T and (u, u ) T represent, respectvely, the orentaton of the two man axes of the real target n the target canddate regon.

13 Because the weght mage s a relable densty dstrbuton functon, the orentaton estmaton of the target provded by matrx U s more relable than that by CAMSHIFT. Moreover, n the CAMSHIFT algorthm, λ and λ were drectly used as the wdth and heght of the target, whch s actually mproper [4, pp. -4]. Next, we present a new scheme to more accurately estmate the wdth and heght of the target. Suppose that the target s represented by an ellpse, for whch the lengths of the sem-major axs and sem-mnor axs are denoted by a and b, respectvely. Instead of usng λ and λ drectly as the wdth a and heght b, t has been shown [4, pp. -4] that the rato of λ to λ can well approxmate the rato of a to b,.e., λ λ ab. Thus we can set a = kλ and b= kλ, where k s a scale factor. Snce we have estmated the target area A, there s π π ( λ )( λ ) ab = k k = A. Then t can be easly derved that Now the covarance matrx becomes ( ) k = A/ πλλ (0) ( ) ( ) a = λ A/ πλ b= λ A/ πλ () u u a 0 u u Cov = u u 0 b u u T () The adjustment of covarance matrx Cov n Eq. () s a key step of the proposed algorthm. It should be noted that the EM-lke algorthm by Zvkovc and Kröse [] estmates teratvely the covarance matrx for each frame based on the mean shft trackng algorthm. Unlke the EM-lke algorthm, our algorthm combnes the area of target,.e., A, wth the covarance matrx to estmate the wdth, heght and orentaton of the target. In Secton 4., we lsted the estmated wdth, heght and orentaton of the synthetc ellpse sequence n Fgure together wth the relatve estmaton error by usng the proposed SOAMST algorthm. It can be seen that the estmaton accuracy s very satsfyng. 3

14 3.5 Determnng the Canddate Regon n Next Frame Once the locaton, scale and orentaton of the target are estmated n the current frame, we need to determne the locaton of the target canddate regon n the next frame. Wth Eq. (), we defne the followng covarance matrx to represent the sze of the target canddate regon n the next frame ( a+δd) 0 Cov = U U 0 ( b+δd) T (3) where Δ d s the ncrement of the target canddate regon n the next frame. The poston of the ntal target canddate regon s defned by the followng ellpse regon (x y ) Cov (x y ) (4) T 3.6 Implementaton of the SOAMST Algorthm Based on the above analyses n sub-sectons 3. ~ 3.5, the scale and orentaton of the target can be estmated and then a scale and orentaton adaptve mean shft trackng algorthm,.e. the SOAMST algorthm, can be developed. The mplementaton of the whole algorthm s summarzed as follows. Algorthm of Scale and Orentaton Adaptve Mean Shft Trackng (SOAMST) ) Intalzaton: calculate the target model ˆq and ntalze the poston y 0 of the target canddate model n the prevous frame. ) Intalze the teraton number k 0. 3) Calculate the target canddate model ˆp(y 0) n the current frame. 4) Calculate the weght vector { w } = usng Eq. (8). n 5) Calculate the new poston y of the target canddate model usng Eq. (0). 6) Let d y y 0, y0 y. Set the error threshold ε (default 0.) and the maxmum Iteraton number N (default 5). 4

15 If ( d<ε or k N) Stop and go to step 7; Otherwse k k+ and go to step 3. 7) Estmate the wdth, heght and orentaton from the target canddate model usng Eq. (). 8) Estmate the ntal target canddate model for next frame usng Eq. (4). 4. Expermental Results Ths secton evaluates the proposed SOAMST algorthm n comparson wth the orgnal mean shft algorthm,.e., mean shft trackng wth a fxed scale, the adaptve scale algorthm [9] and the EM-shft algorthm 4 [, 5]. The adaptve scale algorthm and the EM-shft algorthm are two representatve schemes to address the scale and orentaton changes of the targets under the mean shft framework. Because the weght mage estmated by CAMSHIFT s not relable, t s prone to errors n estmatng the scale and orentaton of the object. So CAMSHIFT s not used n the experments. We selected RGB color space as the feature space and t was quantzed nto bns for a far comparson between dfferent algorthms. It should be noted that other color space such as the HSV color space can also be used n SOAMST. One synthetc vdeo sequence and three real vdeo sequences are used n the experments. The MATLAB source codes and all the expermental results of ths paper can be downloaded n the webste 4. Experments on a Synthetc Sequence We frst use a synthetc ellpse sequence to verfy the effcency of the proposed SOAMST algorthm. As shown n Fgure -(d), the wndow sze of the ntal target (blue ellpse) s 4 We thank Dr. Zvkovc for sharng the code n [5]. 5

16 We select Δ k =0 n the proposed SOAMST algorthm so that the wndow sze of the ntal target canddate regon (red ellpse n Fgure -(b)) s n frame. For other frames n the SOAMST results, the external ellpses represent the target canddate regons, whch are used to estmate the real targets,.e., the nner ellpses. The expermental results show that the proposed SOAMST algorthm could relably track the ellpse wth scale and orentaton changes. Meanwhle, the expermental results by the fxed-scale mean shft s not good because of sgnfcant scale and orentaton changes of the object. The adaptve scale algorthm does not estmate the target orentaton change and has bad trackng results. The EM-shft algorthm fals to correctly estmate the scale and orentaton of the synthetc ellpse, although the target n ths sequence s very smple. (a) The fxed-scale mean shft trackng algorthm (b) Adaptve scale algorthm (c) The EM-shft algorthm (d) The proposed SOAMST algorthm Fg. : Trackng results of the synthetc ellpse sequence by dfferent trackng algorthms. The red ellpses represent the target canddate regon whle the blue ellpse represents the estmated target regon. The frames, 0, 30, 40, 50, 70 are dsplayed. Table lsts the estmated wdth, heght and orentaton of the ellpse n ths sequence by 6

17 usng the SOAMST scheme. The orentaton s calculated as the angle between the major axs and x-axs. The frst frame of the sequence was used to defne the target model and the rest frames were used for testng. It can be seen that the proposed SOAMST method acheves good estmaton accuracy of the scale and orentaton of the target. Table. The estmaton result and accuracy of the wdth, heght and orentaton of the ellpse by the proposed SOAMST method. Sem-major length a Sem-mnor length b Orentaton Frame Real Estmated Error Real Estmated Error Real Estmated Error no. length length (%) length length (%) angle angle (%) Average error over 7 frames Experments on Real Vdeo Sequences The proposed SOAMST algorthm s then tested by usng three real vdeo sequences. The frst vdeo s a palm sequence (Fgure 3) where the object has clearly scale and orentaton changes. Nether the fxed-scale mean shft algorthm nor the adaptve scale algorthm acheves good trackng results. On the other hand, we see that both EM-shft and SOAMST track the palm well n the sequence. However, when the palm s movng fast, such as n frames 7 and 94, the estmated target scale and orentaton by EM-shft are not as accurate as those by the SOAMST algorthm. The second vdeo s a car sequence where the scale of the object (a whte car) ncreases gradually as shown n Fgure 4. The expermental results show that the proposed SOAMST algorthm estmates more accurately the scale changes than the adaptve scale and the EM-shft algorthms. 7

18 (a) The fxed-scale mean shft trackng algorthm (b) Adaptve scale algorthm (a) The EM-Shft algorthm (b) The proposed SOAMST algorthm Fg. 3: Trackng results of the palm sequence by dfferent trackng algorthms. The frames 0, 7, 94, and 40 are dsplayed. (a) Adaptve scale algorthm (b) The EM-Shft algorthm (c) The proposed SOAMST algorthm Fg. 4: Trackng results of the car sequence by dfferent trackng algorthms. The frames 5, 40, 60 and 75 are dsplayed. 8

19 The last experment s on a more complex sequence of walkng man. The object exhbts large scale changes wth partal occluson. To save space we only show the results by EM-shft and SOAMST here. As can be seen n Fgure 5, both EM-shft and SOAMST algorthm can track the target over the whole sequence. However, the SOAMST scheme works much better n estmatng the scale and orentaton of the target, especally when occluson occurs. (a) The EM-shft algorthm (b) The proposed SOAMST algorthm Fg. 5: Trackng results of the walkng man sequence wth occluson by the EM-shft and SOAMST algorthms. The frames 0, 60, 0 and 50 are dsplayed. Table 3. The average number of teratons by dfferent methods on the four sequences. Methods Fxed-scale Adaptve mean shft scale EM-shft SOAMST Synthetc ellpse Palm sequence Car sequence Walkng-man sequence Table 3 lsts the average numbers of teratons by dfferent schemes on the four vdeo sequences. The average number of teratons of the proposed SOAMST s approxmately equal to that of the orgnal mean shft algorthm wth fxed scale. The teraton number of the adaptve scale algorthm s the hghest because t runs mean shft algorthm three tmes. The man factors whch affect the convergence speed of the EM-shft and the SOAMST algorthms are the computaton of the covarance matrx. EM-shft estmates t n each teraton whle 9

20 SOAMST only estmates t once for each frame. So SOAMST s faster than EM-shft. To better evaluate the competng methods, n Table 4 we lst the mean localzaton errors (MLE) and the true area ratos (TAR) by the three trackers on the three real vdeo sequences, palm, car and walkng man. The TAR s defned as the rato of the overlapped area between the trackng result and ground truth to the area of ground truth. The MLE and TAR are closely related to scale and orentaton estmaton of the target beng tracked. Table 4 shows that the proposed SOAMST method acheves the best performance among the three trackng methods. Table 4. The MLE and TAR values by the competng trackng methods. Method Adaptve scale EM-shft SOAMST MLE TAR MLE TAR MLE TAR Palm % % % Car % % % Walkng man % % % In general, the proposed SOAMST algorthm, whch s motvated by the CAMSHIFT algorthm [6], extends the mean shft algorthm when the target has large scale and orentaton varatons. It nherts the smplcty and effectveness of the orgnal mean shft algorthm whle beng adaptve to the scale and orentaton changes of the target. 5. Conclusons By analyzng the moment features of the weght mage of the target canddate regon and the Bhattacharyya coeffcents, we developed a scale and orentaton adaptve mean shft trackng (SOAMST) algorthm. It can well solve the problem of how to estmate robustly the scale and orentaton changes of the target under the mean shft trackng framework. The weght of a pxel n the canddate regon represents ts probablty of belongng to the target, whle the zero th order moment of the weghts mage can represent the weghted area of the canddate 0

21 regon. By usng the zero th order moment and the Bhattacharyya coeffcent between the target model and the canddate model, a smple and effectve method to estmate the target area was proposed. Then a new approach, whch s based on the area of the target and the corrected second order center moments, was proposed to adaptvely estmate the wdth, heght and orentaton changes of the target. The proposed SOAMST method nherts the merts of mean shft trackng, such as smplcty, effcency and robustness. Extensve experments were performed and the results showed that SOAMST can relably track the objects wth scale and orentaton changes, whch s dffcult to acheve by other state-of-the-art schemes. In the future research, we wll focus on how to detect and use the true shape of the target, nstead of an ellpse or a rectangle model, for a more robust trackng. References [] Kalath T.: The Dvergence and Bhattacharyya Dstance Measures n Sgnal Selecton, IEEE Trans. Communcaton Technology, 967, 5, (), pp [] Fukunaga F., Hostetler L. D.: The Estmaton of the Gradent of a Densty Functon, wth Applcatons n Pattern Recognton, IEEE Trans. on Informaton Theory, 975,, (), pp [3] Cheng Y.: Mean Shft, Mode Seekng, and Clusterng, IEEE Trans on Pattern Anal. Machne Intell., 995, 7, (8), pp [4] Mukundan R., Ramakrshnan K. R.: Moment Functons n Image Analyss: Theory and Applcatons, World Scentfc, Sngapore, 996. [5] Wren C., Azarbayejan A., Darrell T., Pentland A.: Pfnder: Real-Tme Trackng of the Human Body,IEEE Trans. Pattern Anal. Machne Intell., 997, 9, (7), pp [6] Bradsk G.: Computer Vson Face Trackng for Use n a Perceptual User Interface, Intel Technology Journal, 998, (Q), pp. -5. [7] Comancu D., Ramesh V., Meer P.: Real-Tme Trackng of Non-Rgd Objects Usng Mean

22 Shft. Proc. IEEE Conf. on Computer Vson and Pattern Recognton, Hlton Head, SC, June, 0, vol., pp [8] Comancu D., Meer P.: Mean Shft: a Robust Approach toward Feature Space Analyss, IEEE Trans Pattern Anal. Machne Intell.,, 4, (5), pp [9] Comancu D., Ramesh V., Meer P.: Kernel-Based Object Trackng, IEEE Trans. Pattern Anal. Machne Intell., 3, 5, (), pp [0] Collns R.: Mean-Shft Blob Trackng through Scale Space, Proc. IEEE Conf. Computer Vson and Pattern Recognton, Wsconsn, USA, 3, pp [] Zvkovc Z., Kröse B.: An EM-lke Algorthm for Color-Hstogram-Based Object Trackng, Proc. IEEE Conf. Computer Vson and Pattern Recognton, Washngton, DC, USA, 4, vol., pp [] Yang C., Raman D., Davs L.: Effcent Mean-Shft Trackng va a New Smlarty Measure, Proc. IEEE Conf. Computer Vson and Pattern Recognton, San Dego, CA, 5, vol., pp [3] Fashng M., Tomas C.: Mean Shft s a Bound Optmzaton, IEEE Trans. Pattern Anal. Machne Intell., 5, 7, (3), pp [4] Ylmaz A., Javed O., Shah M.: Object Trackng: a Survey, ACM Computng Surveys, 6, 38, (4), Artcle 3. [5] Carrera-Perpnan M. A. Gaussan Mean-Shft s an EM Algorthm, IEEE Trans. Pattern Anal. Machne Intell., 7, 9, (5), pp [6] Brchfeld S., Rangarajan S.: Spatograms versus hstograms for regon-based trackng, Proc. IEEE Conf. on Computer Vson and Pattern Recognton, 5, vol., pp , 5. [7] Hu J., Juan C., Wang J.: A spatal-color mean-shft object trackng algorthm wth scale and orentaton estmaton, Pattern Recognton Letters, 8, 9, (6), pp [8] Srkrshnan V., Nagaraj T., Chaudhur S.: Fragment Based Trackng for Scale and Orentaton Adapton, Proc. Indan Conf. on In Computer Vson, Graphcs & Image Processng, 8, pp [9] Lnderberg T.: Feature Detecton wth Automatc Scale Selecton, Internatonal Journal of

23 Computer Vson. 998, 30, (), pp [0] Bretzner L., Lndeberg T.: Qualtatve Mult-Scale Feature Herarches for Object Trackng, Journal of Vsual Communcaton and Image Representaton, 0,, (), pp.5-9. [] Nummaro K., Koller-Meer E., Gool L. V.: An Adaptve Color-Based Partcle Flter, Image and Vson Computng, 3,, (), pp [] Horn R. A., Johnson C. R., Topcs n Matrx Analyss, Cambrdge Unversty Press, U.K., 99. [3] Quast K., Kaup A.: Scale and Shape adaptve Mean Shft Object Trackng n Vdeo Sequences, Proc. European Sgnal Processng Conference, Glasgow, Scotland, 9, pp [4] Collns R.: Lecture on Mean Shft, [5] Zvkovc Z.: EM-shft code, 3