Joint Registration and Active Contour Segmentation for Object Tracking

Transcription

1 Jont Regstraton and Actve Contour Segentaton for Object Trackng Jfeng Nng a,b, Le Zhang b,1, Meber, IEEE, Davd Zhang b, Fellow, IEEE and We Yu a a College of Inforaton Engneerng, Northwest A&F Unversty, Yanglng, Shaanx, Chna b Dept. of Coputng, The Hong Kong Polytechnc Unversty, Hong Kong, Chna Abstract:Ths paper presents a novel object trackng fraework by jont regstraton and actve contour segentaton (JRACS), whch can robustly deal wth the non-rgd shape changes of the target. The target regon, whch ncludes both foreground and background pxels, s plctly represented by a level set. A Bhattacharyya slarty based etrc s proposed to locate the regon whose foreground and background dstrbutons can best atch those of the tracked target. Based on ths etrc, a trackng fraework whch conssts of a regstraton stage and a segentaton stage s then establshed. The regstraton step roughly locates the target object by odelng ts oton as an affne transforaton, and the segentaton step refnes the regstraton result and coputes the true contour of the target. The robust trackng perforance of the proposed JRACS ethod s deonstrated by real vdeo sequences where the objects have clear non-rgd shape changes. Index Ters: Object trackng, actve contour odel, level set, segentaton, regstraton 1 Correspondng author: cslzhang@cop.polyu.edu.hk. Ths work s supported by the Natonal Scence Foundaton Councl of Chna under Grants and the Fundaental Research Funds for the Central Unverstes under Grant No.QN

2 1. Introducton Vsual trackng s an portant yet challengng task n varous coputer vson applcatons. In the past decades any algorths have been proposed for robust object trackng, ang to overcoe the dffcultes arsng fro noses, occlusons, clutters, and changes n the foreground object and/or n the background envronent [1-5]. In partcular, how to desgn the trackers that can handle the target object shape (or contour) changes s one of the hottest topcs of object trackng [6-15]. Because level set can deal wth object topologcal changes sealessly, any trackers a to descrbe the oton change nforaton of the object usng the actve contour ethod. Freedan and Zhang [11] located the best canddate regon by atchng object dstrbutons usng the Bhattacharyya slarty and Kullback-Lebler dstance, respectvely. Afterwards, they proved t by cobnng foreground atchng flow and background satchng flow, proposng the so-called the cobnaton flow ethod [12]. However, both the ethods n [11] and [12] are level set based age segentaton ethod wth pror dstrbuton. They often need any tes of teratons to converge when the ntal contour s far fro the target true contour. On the other hand, soe teplate (e.g., usng a sple rectangle or an ellpse) based trackers [16-17] often perfor well n real te under coplex scenes, but they are dffcult to track the target wth coplex contour. Methods n [18] and [19] track the target as a changng ellpse obtaned by estatng the covarance atrx n scale and orentaton. Ylaz [8] ade an attept to deal wth the scale and orentaton changes of the target. He extended the tradtonal ean shft tracker [17] by usng an asyetrc kernel (level set) to represent the target. Rkln-Ravv et al. [20] used projectve transforaton to segent an age by usng a 2

3 sngle pror shape but wthout usng any pont correspondences. Recently, Chverton et al. [21] proposed an onlne actve contour based learnng odel, whch conssts of two coponents: a oton based bootstrapper to extract the target shape nforaton fro prevous trackng results and a regon based tracker to optze the extracted shape by actve contour. In suary, t s of hgh nterest to develop a trackng ethod whch can possess the advantages of both the teplate based trackers (e.g., havng less teraton nubers for converge) and the segentaton based trackers (e.g., trackng the true contour of the target). To acheve the above entoned goal, n ths paper we propose a novel trackng fraework to track the true contour change of the target. In the proposed ethod, the target regon, ncludng the foreground and background coponents, s represented by a level set. By usng the Bhattacharyya slarty [22-24] as a easure for target atchng, we locate the canddate target regon such that ts foreground dstrbuton and background dstrbuton are ost slar to the user defned target odel. A novel trackng algorth va jont regstraton and actve contour segentaton (JRACS) s then developed. The proposed JRACS conssts of two stages. Frst, the regstraton procedure estates the affne deforaton of the target. Ths stage can be consdered as a teplate based tracker, whle t uses arbtrary shape (level set), nstead of the sple rectangle or ellpse teplate, to represent the target. Ths akes the proposed ethod powerful to estate non-rgd oton of the target. Second, the segentaton procedure refnes the affne transforaton estated n the regstraton stage, and coputes the target s true shape accurately. Fnally, on-lne target appearance updatng s used to reove trackng drft. Extensve experents on typcal vdeos valdate the effectveness of our ethods. 3

4 The proposed ethod s partally nspred by the works n [11, 12, 16, 17]. However, dfferent fro ean-shft [16] and foreground flow [11] ethods whch perfor only foreground atchng, the proposed JRACS odel atches both foreground and background. In addton, the classcal ean-shft tracker consders the target oton as a translaton transforaton; the lately developed EM-shft [18] and SOAMST [19] ethods track the target wth a changng ellpse; ethods n [11] and [12] are actually two age segentaton ethods; n contrast, the proposed JRACS ethod consders the non-rgd arbtrary shape changes of the target object. The rest of ths paper s organzed as follows. Secton 2 descrbes the target representaton. Secton 3 descrbes n detal the proposed JRACS ethod. Secton 4 presents the pleentaton of the JRACS. Secton 5 presents the experental results and Secton 6 concludes the paper. 2. Target Representaton To track robustly the target object n the vdeo sequence, we need to represent the target as robust as possble. The regon where the target locates can be represented as an ellpse, a rectangle or an arbtrary contour. Once the regon s fxed, there are varous features that can be used to descrbe the target defned n t, such as color, edge and texture features, or the cobnaton of the. In ths paper, we select color hstogra to odel the target object because of ts erts such as ndependence of scalng and rotaton, robustness to partal occlusons, low coputatonal cost, etc. Of course, when object and background dffers uch fro each other on texture features, the texture hstogra can also be used to represent the target. Follow the notaton n [16], we denote by u the feature space of the object ndcated by the user, denote by the nuber of features, and defne the foreground dstrbuton q and 4

5 background dstrbuton o as follows: q 1,, 1 q 1 q (1) u u u o 1,, 1 o 1 u u u u o (2) u To track the target object whose regon s ntalzed n the frst frae, n the subsequent fraes we attept to fnd the best canddate regon under a certan etrc. We use level set to represent the target regon because of ts flexblty n representng an arbtrary target contour. Let level set functon denote a canddate target. We use p() to stand for the dstrbuton of foreground regon 0, and use v() to stand for the dstrbuton of background regon d<<0, where threshold d s used to restrct the nterested regon nto a sall area. The two dstrbutons p() and v() wll atch q and o, respectvely, under certan etrcs. Fg. 1 shows an exaple by usng level set to represent the contour of a car. The regon enclosed by the nternal curve (n blue color) s the foreground, whle the regon enclosed between the nternal and external (n red color) curves s the background. p() and v() are respectvely defned as follows: u u1 p p = p 1 v u1,, u 1 v = v 1 u1,, u u u (3) (4) Fgure 1: Target representaton plctly by usng level set. 5

6 Let x * f, and * x 1 n b, f 1 nb be the pxels fallng nto the foreground part and background part, respectvely. Next we defne the foreground odel and background odel. 2 The functon b: R 1 * aps the pxel at locaton x nto the bn x b n the quantfed feature space. Then, the probablty of the feature u=1, 2,, n the foreground odel and background odel are respectvely calculated as n f * x, f (5) f 1 1 qu b u n 1 ou b u n n b * xb, (6) b 1 where s the Kronecker delta functon. Noralzed constants n f and n b are the nubers of foreground pxels and background pxels, whch ake qu 1 and u1 ou 1. u1 Slarly, we copute foreground dstrbuton and background dstrbuton n the current canddate regon d 0 as follows: n 1 pu Hx bx u A (7) f 1 n 1 v 1Hx bx u (8) u Ab 1 where n s the nuber of pxels n the canddate regon, the Heavsde functon H() s used to select foreground regon, and thus 1H() s eployed to select background regon. The noralzaton factors derved as follows: A f and A f A b, akng pu 1 and vu u1 u1 1, are n H x (9) 1 n H (10) A 1 x b 1 6

7 We select H x 1 2 x 1 arctan 2 as the Heavsde functon n the level set because of ts any advantages [25]. As shown n Fg. 2, the Heavsde functon can also be regarded as a kernel functon, lke the Epanechnkov kernel or Gaussan kernel n the ean shft tracker [16] Fgure 2: Heavsde functon wth = The jont regstraton and segentaton In the proposed ethod, the Bhattacharyya slarty s adopted for target atchng. Based on ths etrc, we derve a regstraton forula and a segentaton forula. The regstraton forula s used to estate the affne deforaton of the target, and the segentaton forula s used to refne the regstraton results so that the contour of the target can be obtaned. 3.1 Matchng etrc Soe recently proposed trackng ethods acheve good results by constructng an onlne dscrnatve classfer usng both the foreground and background features [26-28]. The cobnaton flow ethod [12] also deonstrates that the background nforaton s portant to obtan good result for level set based trackng ethods. Nonetheless, because both the foreground and background nforaton of the target changes soothly n ost vdeos, the background nforaton s also useful to estate accurately the target contour change. 7

8 Our goal s to fnd a regon n the current frae such that ts foreground dstrbuton p() and background dstrbuton v() can best atch the odel foreground dstrbuton q and the odel background dstrbuton o, respectvely. There are any knds of crtera [29] that can be used to copare the slarty of these dstrbutons. In ths paper, we adopt the Bhattacharyya slarty [22-24] because of ts successful applcatons n object trackng. The Bhattacharyya coeffcent s a dvergence-type easure whch has a straghtforward geoetrc nterpretaton. It s the cosne of the angle between q and p() or between o and v(). The hgher the Bhattacharyya coeffcent between target odel and canddate target odel, the hgher the slarty between the. In our odel, snce both the foreground and background are consdered, we defne the slarty dstance easure as 1 u u u u (11) u E p q v o where the weght balances the contrbutons of foreground and background n the atchng. 3.2 Deforaton estaton Deforaton estaton s used to handle the shape change of the target. The teplate based trackers [16-19] usually consder the oton of the target as a translaton transfor or sple zoong change, but they encounter dffcultes when the contour of the target presents large non-rgd change. On the other hand, although the segentaton based trackng ethods can handle non-rgd oton, they usually requre ore te to converge and are prone to local na. Therefore, our goal s to present a novel fraework whch can cobne the advantages of the above two knds of trackers. Let 0 be the ntal poston of the target n the current frae. The contour of the target can be obtaned by lettng 0 =0. Thus, the two probabltes p p 0 u 0 u 1,, and v v = 0 u 0 u 1,, can be coputed frst. Suppose that (>-d) s the next poston of 8

9 the target, whch s adjacent to 0. Slar to the dervaton of classcal ean shft tracker [16], by applyng Taylor expanson around p u ( 0 ) and v u ( 0 ), we have 1 E p q p 2 u u 0 u u u1 u1 pu u vu 0 ou vu u1 u1 vu 0 q o (12) Furtherore, by substtutng Eq. (7) and Eq. (8) nto Eq. (12), we have 1 n 1 E p q w H 2 u1 Af 1 u0 u f, x n 1 1 v o w H 2 0, 1 x u u b u1 Ab 1 (13) where q w b u f, b, u x u p (14) 1 u 0 o w b u u x u v (15) 1 u 0 It s worth notng that the orgnal ean-shft trackng ethod [16] consders the oton of the target as purely translaton, and the weght w f, plays a key role n fndng the new centrod of the target. It ndcates the possblty that pxel x belongs to the foreground. In our ethod, w f, plays the slar role. Slarly, w b, ndcates the possblty that pxel x belongs to the background. As we wll see later, w f, and w b, wll gude the regstraton and segentaton of level set. Maxzng Eq. (13) s equvalent to axzng the Bhattacharyya coeffcent defned n Eq. (11), whch s a functon about locaton x and level set functon. Next, we derve the forulas for optzng locaton and contour, respectvely. 9

10 3.2.1 Regstraton. We odel the oton of the target as an affne transforaton. To ths end, we ntroduce a warp x=w x, T [30] nto Eq. (13) to odel and estate the affne transforaton of the target: x 1 p1 p3 p5 x=w x, T y p2 1 p4 p 61 (16) where the colun vector T p, p, p, p, p, p ' has 6 paraeters to characterze the pose change of the object, and x, y s the colun and row coordnate at pxel x. Substtutng Eq. (16) nto Eq. (13) and ottng the ters that are not a functon of T, we have 1 1 E H T w H T w 2A 2A f n n w x, f 1 wx,, (17) b. 1 b 1 T s the ncreental warp of the shape kernel represented plctly by level set functon. When T tends to 0, the affne deforaton estaton wll converge. In order for the convenence to derve T, we rewrte H wx, T and H T as H wx, T and 1 H wx, T 1 w x, 2 2, respectvely [9]. The Taylor expresson around T leads to 2 wx, 1 H T H JT 2 H x w x, 1 -J H T H T 2 H x 2 2 (18) (19) where J H W xx W x T T (20) 10

11 x xx, yx (21) x, T x 0 y W T 0 x 0 y 0 1 (22) By dfferentatng Eq. (17) wth respect to T, we have 1 n 1 w n f, 1 w, 1 1 b T T J J f, b, J 1 Af 2H x Ab 21 x 1 Af A b T w w H The level set functon can then be updated by usng Eq. (16). Note that the denonator H and H x (23) 1 x n Eq. (23) wll never be zero wth the used Heavsde functon H x 1 2 x 1 arctan 2 and each x corresponds to a dfferent J accordng to Eq. (20). Ths regstraton step by solvng an affne transforaton proble can be vewed as a teplate based trackng process. It teratvely estates the shape change untl convergence. For the ajorty of target shape change types, the affne transforaton can usually descrbe the well Segentaton. The affne transforaton tracker proposed n Secton can estate the non-rgd oton better than tradtonal teplate based trackers, such as ean shft tracker [16] and the EM-shft tracker [18], but t cannot extract the contour of the target accurately. Therefore we propose to refne the regstraton result by usng a novel segentaton based trackng procedure. By vewng Eq. (13) as the functon of x, we optze t wth respect to x by calculus of varatons [31]. The frst varaton of the functonal s E x 1 x 1 1 wf, wb, x 2 Af A b (24) 11

12 where x x 2 2 s the dervatve of the soothed Heavsde step functon,.e., a soothed Drac delta functon. tends to copute a global nzaton [25]. x acts on all level curves, and t We seek E x x 0 by carryng out the steepest-ascent ethod usng the followng gradent flow: x, x t x E t E (25) Eq. (24) s actually a segentaton based actve contour tracker, where the sgn of 1 1 w w deternes f the pxel x belongs to foreground or background. Because A f f, b, Ab the contour obtaned by the regstraton step s very close to the true edge of the target, we evolve Eq. (24) by usng the result fro the regstraton step as the ntal curve Target update. In dynac scenes, the llunaton and vewpont ght change and the object ght be occluded, and thus the foreground dstrbuton and background dstrbuton of the target often change gradually. Snce our ethod can estate the shape deforaton accurately, the target odel updatng becoes easy and ths can prevent effectvely the target fro drftng. Our updatng ethod s very sple: q_update = q+ 1- p o_update = o+ 1- v t t (26) where p t and v t are respectvely the foreground and background odel at te t. In our experents, we set, [0.7, 0.95] by experence. 12

13 4. Ipleentaton In ths secton, we present the nuercal pleentaton of the proposed algorth n detal. Let h be the step length and obvously h=1 n age grd. Let x, y be the spatal coordnate correspondng to pxel x. For convenence, we represent x as x, y n another for. n Eq. (20),.e., x, y x dfference schee as follows:, s approxated by the center x, y x 1, y x 1, y; x, y 1 x, y 1 (27) For the regstraton step, we frst copute J n Eq. (20) to estate the affne transforaton vector T n Eq. (23), where T p, p, p, p, p, p ' s a 61 vector. In Eq. (20), x s a scalar, x s 1 2 vector, and W T s a 2 6 atrx. So J T whch corresponds to each x s a 1 6 vector and J J s a 6 6 atrx. Thus we can get T T by Eq. (23) after coputng J J and J T. In practce, we can only use those pxels around zero level set to estate T, and ths can speed up the estaton of T. For segentaton, Eq. (25) can be pleented by usng a nuercal schee on a dscrete grd. Let t be the te step. Then we copute l1 by the followng dscretzaton and lnearzaton of Eq. (25), l x y x y l 1,, 1 l 1 l 1 l x, y w l f, w l b, t 2 Af A b (28) Eq. (28) can be rewrtten as 1 1 x, y x, y t x, y w 1 w l1 l l l l l f, l b, 2 Af A b (29) The convergence of Eq. (29) s guaranteed by 13

14 1 1 1 t 1 ax x, y w w 1,, n 2 Af A b l l l l, l, f b accordng to the Courant-Fredrchs-Lewy step-sze restrcton [32]. The level set s represented by a sgned dstance functon and ts re-ntalzaton can be solved effcently by usng the ethod proposed n [33]. In general, the procedures of JRACS can be suarzed as follows. ) Calculate the foreground odel q and background odel o. ) Intalze the poston 0 of the canddate regon n the current frae. ) Calculate the foreground canddate p( 0 ) and background canddate v( 0 ). v) Intalze the teraton nuber k 0 for regstraton. v) Calculate T by usng Eq. (23). v) Let et, k k 1. Set the error threshold and the axu teraton nuber N. If e and k N, then update 0 by usng (16) and go to step. Otherwse, regstraton converges and then executes segentaton. v) Refne the regstraton result by usng Eq. (25). v) Update the foreground odel and background odel by usng Eq. (26). 5. Experental results and dscussons Snce the proposed JRACS s nspred by both teplate based trackng ethods and segentaton based trackng ethods, we evaluate t n coparson wth the cobnaton flow algorth [12], whch s a representatve and state-of-the-art segentaton-based trackng ethod, and the EM-shft algorth 2 [18], whch proves the teplate based ean shft tracker by estatng teratvely the scale and orentaton changes of the target. In addton, the recently developed SOAMST algorth [19], whch s robust to scale and orentaton changes of the target, and the trackng ethod developed by Chverton et al. [21], 2 We thank Dr. Zvkovc for sharng the code n [35]. 14

15 whch conssts of a bootstrap stage and an adaptve shape eory based actve contour stage, are also eployed n the experent. Note that the results of Chveton et al. s ethod were obtaned by usng only the actve trackng part wthout the bootstrap stage 3 for a far coparson wth the other ethods whch use a non-autoatc (anual) ntalzaton. In the JRACS ethod, there are several paraeters to set. Frst, d (>-d) s used to select the sze of background regon. In our experents, the sze of background regon s set approxately that of the foreground regon. Second, s used to balance the foreground atchng score and background atchng score. In the experents, we set Ab Af, and 1 1 then wf, wb, A A f can be rewrtten as wf, wb, b 1 A f, n whch wf, wb, wll gude the regstraton and segentaton and 1 A f s a noralzaton constant. Ab Af ples that soe foreground pxels wll be consdered as the background. Thrd, a bg n the Heavsde functon can speed up the convergence of regstraton and segentaton process. We select [1, 5] and set the foreground and background dstrbuton update paraeters and n Eq. (25) as 0.9 n our experents. We frst use an exaple to llustrate the regstraton and the segentaton stages of the proposed JRACS ethod. Then we present the experental results on fve real vdeo sequences. The algorths are pleented n the envronent of MATLAB 7.10 and run on a PC wth Intel Core 7 CPU (2.93 GHZ) and 8GB RAM. The RGB color odel s selected as the feature space to represent the target. The vdeos of all the trackng results n the followng experents can be downloaded at An exaple of the proposed JRACS ethod In ths secton, we deonstrate the proposed JRACS ethod by an exaple of hand, whch 3 We thank Dr. Chverton for provdng the experental results of ther trackng algorth. 15

16 shows obvous non-rgd oton. We frst deonstrate the regstraton perforance of JRACS,.e., estatng the affne transforaton of the hand, n Fgure 3. Due to the hgh flexblty of huan hand, t s not accurate to represent the hand shape by usng an ellpse teplate [18, 19], whle the level set can be used to accurately represent the hand shape. Fgure 3 (b) shows the ntal poston of level set n a certan frae and t can be clearly seen that the target has large non-rgd deforaton. Fgures 3 (c)-(f) llustrate the regstraton process of JRACS. It can be seen that although there s a sgnfcant change of scale and orentaton, the JRACS can stll estate the hand shape well. The cobnaton flow ethod [12] uses 180 teratons (about 1.95s) to convergence, whle JRACS uses only 8 teratons to converge (about 0.13s) snce t atches the teplate quckly by estatng the affne transforaton of the hand. It can also be seen fro Fgure 3 that although the regstraton process n JRACS can estate the general deforaton of hand, the boundary s not very accurate. Therefore we further perfor segentaton to refne the regstraton result for a ore accurate hand shape contour. The segentaton process s llustrated n Fgure 4. (a) (b) (c) (d) (e) (f) Fgure 3: The regstraton process by the proposed JRACS ethod. (a) The tracked target (ncludng foreground and background). (b) Intal level set locaton n the current frae. (c)-(f) The regstraton process and the fnal results. 16

17 (a) (b) (c) Fgure 4: The segentaton process n the proposed JRACS ethod. 5.2 Experental results on real vdeo sequences We then use several real vdeo sequences to valdate the proposed JRACS ethod. Because Chverton et al. s ethod [21] fals to track the sequences of car, head and outdoor face, we only show ts trackng results on the hand and fsh sequences n the followng fgures. The frst one s a car sequence, whose scale grows gradually. In ths sequence, soe features of the car present n the background ght dsturb the estaton of scale and orentaton. As can be seen n Fgure 5(d), the JRACS tracks the target over the whole sequence wth good scale estaton, whle the cobnaton flow algorth (refer to Fgure 5(c)) does not perfor well because the an features of background are slar to soe features of the car, whch dsturbs the evoluton of the level set. The EM-shft (Fgure 5(a)) and SOAMST (Fgure 5(b)) ethods, whch are teplate based trackers, cannot capture the true contour of the car. 17

18 (a) EM-shft (b) SOAMST (c) Cobnaton flow (d) JRACS Fgure 5: Trackng results on the car sequence wth large scale changes by the copetng ethods. The second sequence was the one used n [36]. The target s an ndoor face that oves quckly. In ths sequence, the face shows soe coplex changes of vewpont, background and pose. Fgure 6 llustrates the trackng results. The skn color of neck (whch s part of the background) s close to that of the face. Because the cobnaton flow ethod consders the dsslarty between the background and target, the neck of the an s wrongly regarded as a part of the target, leadng to naccurate trackng results. On the other hand, the proposed JRACS ethod sultaneously atches foreground and background of the target, and t perfors uch better n trackng the target face. Because of the bg pose change of the face, the EM-shft ethod and SOAMST ethod do not perfor well. 18

19 (a) EM-shft (b) SOAMST (c) Cobnaton flow (d) JRACS Fgure 6: Trackng results on the ndoor face sequence wth obvous vewpont and background changes by the copetng ethods. In the thrd sequence, our goal s to track a ovng hand wth large non-rgd deforaton (whch was used n Secton 5.1). In ths hand sequence, the fast stretchng and clenchng actons and the dsturbance of soe background features rase any dffcultes to estate the contour changes of hand. In the proposed JRACS ethod, the regstraton step estates rgd deforaton of the target, and then the segentaton step akes the contour of the target coplete. The trackng results n Fgure 7 show that JRACS perfors uch better than the cobnaton flow ethod on ths sequence. We can see that EM-shft and SOAMST are hard to handle the coplex shape change of the target, whle the actve contour based trackng ethod n [21] does not perfor well. 19

20 (a) EM-shft (b) SOAMST (c) Cobnaton flow (d) Chverton et al. s ethod (e) JRACS Fgure 7: Trackng results on the hand sequence wth obvous non-rgd changes. (Note that the result of Chveton et al. s ethod was obtaned by the actve trackng part only.) The fourth vdeo s an outdoor face sequence whch has obvous vewpont and llunaton changes and occluson. Fgure 8 shows the experental results by the four trackng ethods. Because of the llunaton and vewpont changes, segentaton based cobnaton flow algorth fals to track after the 60 th frae. The teplate based trackers such as EM-Shft and SOAMST perfor better than cobnaton flow. For the proposed JRACS ethod, the regstraton step of t estates the non-rgd deforaton of the object by affne transforaton and overcoes effectvely the effect of the llunaton and vewpont change, and then the segentaton step further optzes the object area by actve contour. 20

21 Note that n Fgure 8 we only show the selected results fro the frst 60 fraes for the cobnaton flow ethod because t fals to track after the 60 th frae. For EM-Shft, SOAMST and JRACS ethod, we show the representatve experental results selected fro all the 380 fraes. The last experent s on a fsh sequence, where the oton and shape of the fsh are very rregular, and there are slar objects appearng around the target object. The experental results n Fgure 9 show that our ethod perfors well even when a slar fsh s presented near the target fsh. However, the cobnaton flow ethod cannot handle t well because t consders another fsh as the desred target as well. Meanwhle, the actve contour stage of the trackng ethod n [21] does not estate the contour change of the target well. (a) EM-shft (b) SOAMST (c) Cobnaton flow (d) JRACS Fgure 8: Trackng results on the outdoor face sequence wth llunaton and vewpont changes and occluson. 21

22 (a) EM-shft (b) SOAMST (c) Cobnaton flow (d) Chverton et al. s ethod (e) JRACS Fgure 9: Trackng results on fsh sequence wth obvous non-rgd changes and slar objects. (Note that the result of Chveton et al. s ethod was obtaned by the actve trackng part only.) In order to copare quanttatvely the proposed JRACS ethod wth the other four ethods, we anually labeled the ground truth contours of the target objects n the fve vdeos, and evaluate the trackng perforance by applyng the overlap ndex (OI) [34]: OI A A G G A A M M (30) where A G represents the ground truth area of the nterestng object and A M represents the area of trackng outputs. A bg OI generally ples that the trackng ethod has good localzaton 22

23 accuraces. Table 1 lsts the OI values of the fve trackng ethods on the fve vdeo sequences. Because the trackng ethod n [21] fals to track the car and two face sequences, we only show ts target localzaton accuraces on the hand and fsh sequences. A slar case arses for the cobnaton flow ethod to handle the outdoor face sequence. The proposed JRACS ethod acheves the hghest OI aong the fve trackng ethods. Table 1: The target localzaton accuraces for the fve trackng ethods accordng to OI. (Note that the result of Chveton et al. s ethod was obtaned by the actve trackng part only.) Method Car Face (ndoor) Hand Fsh Face (outdoor) EM-shft [18] 57% 45% 55% 63% 25% SOAMST [19] 60% 40% 66% 70% 48% Cobnaton flow [12] 64% 66% 81% 71% - Chverton et al. s ethod [21] % 50% - JRACS 75% 67% 82% 82% 69% 10 Iteraton nuber Frae ndex Fgure 10: The teraton nuber by the proposed JRACS on the ndoor face sequence n regstraton stage. The proposed JRACS algorth has a regstraton stage and a segentaton stage. By any experents, we found that the regstraton stage needs 3~5 teratons n average to converge for sall deforaton, but ore teratons are necessary for severe deforaton. Fgure 10 plots the nuber of teratons of the proposed algorth on the face sequence for each frae. The average nuber of teraton s 3.8. After estatng the affne transforaton 23

24 of the target n the regstraton step, the segentaton step furtherore refnes the result of regstraton n order to better approxate the true shape of the target. Theoretcally, the ore teratons used n the segentaton stage, the ore accurate result can be obtaned, but t ay consue ore coputatonal te. Accordng to our experents, 5~15 teratons are approprate. For the cobnaton flow ethod, t wll requre about 35 teratons n average because segentaton-based trackng ethods often requre ore teratons than teplate atchng based trackng ethods. Table 2: Average speed for the four trackng ethods (fraes/per second) Method Car Face (ndoor) Hand Fsh Face (outdoor) Cobnaton flow [12] EM-shft [18] SOAMST [19] JRACS In MATLAB envronent, the trackng speed of JRACS s faster than foreground flowng and the cobnng flowng ethods reported n [11, 12]. Certanly, JRACS s slower than EM-shft and SOAMST because the te coplexty of estatng affne transforaton (n JRACS) s hgher than that of estatng covarance atrx (n EM-Shft and SOAMST). Table 2 copares the average trackng speed for the cobnaton flow, EM-shft, SOAMST and JRASC ethods 4. The proposed JRACS ethod cobnes the advantages of the segentaton based tracker and the teplate based tracker. The regstraton step estates adaptvely the target shape change by usng affne transforaton based on level set ethod. Because the regstraton result s close to the true contour of the object, the segentaton step can easly optze t and get accurate object contour. Actually, the work by Chverton et al [21] shares ths ert wth our work. 4 Note that the speed of Chverton et al. s ethod [21] s not lsted here because the results of ths algorth were run on a dfferent PC and software syste. 24

25 6. Conclusons Ths paper presented a novel trackng fraework wth jont regstraton and actve contour segentaton (JRACS). The tracked target was plctly represented by usng a level set, whch can handle sealessly the topologcal changes of the target. The goal s to fnd a canddate regon, whose foreground dstrbuton and background dstrbuton can best atch the teplate foreground and background based on the Bhattacharyya slarty. A jont regstraton and segentaton schee was developed, whch frst estates the rgd deforaton of the object and then refnes the regstraton result. The advantages of JRACS coe fro the two key weghts, whch ndcate the possblty of a pxel n the canddate regon belongng to foreground or background, and gude the evoluton process of regstraton and segentaton. The good perforance of JRACS was deonstrated by representatve testng vdeos where the targets show large scale non-rgd shape changes. Experental results valdated that JRACS overcoes soe ltatons of prevous works, ncludng EM-shft tracker, SOAMST tracker and the cobnaton flow tracker, and JRACS can be consdered as an extenson of the. In the future, we wll consder how to extend JRACS by ntegratng the spatal nforaton nto the target representaton. References [1] A. Ylaz, O. Javed and M. Shah, Object Trackng: a Survey, ACM Coputng Surveys, vol. 38, no. 4, Artcle 13, [2] M. Isard, and A. Blake, CONDENSATION - condtonal densty propagaton for vsual trackng, Int. J. Coputer Vson, vol. 29, no.1, pp. 5-28, [3] P. Chockalnga, N. Pradeep, and S. Brchfeld, Adaptve fragents-based trackng of non-rgd objects usng level sets, In Proceedngs of IEEE Internatonal Conference on Coputer Vson, [4] Zulfqar Hasan Khan, Irene Yu-Hua Gu, and Andrew G. Backhouse, Robust Vsual Object Trackng Usng Mult-Mode Ansotropc Mean Shft and Partcle Flters, IEEE Trans. Crcuts and Systes for Vdeo Technology, vol. 21, no. 1, 74-87, [5] N. M. Artner, A. Ion, and W. G. Kropatsch, Mult-scale 2D trackng of artculated objects usng herarchcal sprng systes, Pattern Recognton, vol. 44, no. 4, pp , [6] N. Paragos, R. Derche, Geodesc actve contours and level sets for the detecton and trackng of ovng objects, IEEE Trans. Pattern Anal. Machne Intell., vol. 22, no.3, pp ,

26 [7] Qang Chen, Quan-Sen Sun, Pheng Ann Heng, and De-Shen Xa. Two-Stage Object Trackng Method Based on Kernel and Actve Contour. IEEE Trans. Crcuts and Systes for Vdeo Technology, vol. 20, no. 4, , [8] A. Ylaz, Object Trackng by Asyetrc Kernel Mean Shft wth Autoatc Scale and Orentaton Selecton, In Proc. IEEE Conf. on Coputer Vson and pattern Recognton, Mnnesota, USA, Vol. I, pp.1-6, [9] C. Bbby and I. Red. Robust Real-Te Vsual Trackng usng Pxel-Wse Posterors, In Proc. European Conf. on Coputer Vson, part II, pp , [10] X. Sun, H. Yao and S. Zhang. A novel supervsed level set ethod for non-rgd object trackng, In Proceedngs of IEEE Internatonal Conference on Coputer Vson, pp , [11] D. Freedan and T. Zhang, Actve Contours for Trackng Dstrbutons, IEEE Trans. Iage Processng, vol. 13, no. 4, pp , [12] T. Zhang, D. Freedan, Iprovng perforance of dstrbuton trackng through background atchng, IEEE Trans. Pattern Anal. Machne Intell., vol. 27, no. 2, pp , [13] D. Creers, Dynacal statstcal shape prors for level set based trackng, IEEE Trans. Pattern Anal. Machne Intell., vol. 28, no. 8, pp , [14] D. Creers. M. Rousson, R. Derche, A revew of statstcal approaches to level set segentaton: Integratng color, texture, oton and shape, Int l Journal of Coputer Vson, vol. vol. 72, no. 2, pp , [15] M. Roh, T. K, J. Park, and S. Lee, Accurate object contour trackng based on boundary edge selecton, Pattern Recognton, vol. 40, no. 3, pp , [16] D. Coancu, V. Raesh and P. Meer, Kernel-Based Object Trackng, IEEE Trans. Pattern Anal. Machne Intell., vol.25, no. 5, pp , May, [17] C. Yang C, D. Raan, and L. Davs, Effcent Mean-Shft Trackng va a New Slarty Measure, Proc. IEEE Conf. Coputer Vson and Pattern Recognton, San Dego, CA, 2005, vol. 1, pp [18] Z. Zvkovc and B. Kröse, An EM-lke Algorth for Color-Hstogra-Based Object Trackng, In Proc. IEEE Conf. on Coputer Vson and Pattern Recognton, Washngton, D.C., USA, vol. I, pp , [19] J. Nng, L. Zhang, D. Zhang, and C. Wu, Scale and Orentaton Adaptve Mean Shft Trackng, to appear n IET Coputer Vson, [20] T. Rkln-Ravv, N. Kryat and N. Sochen. Unlevel-Set: Geoetry and Pror-based Segentaton, In Proc. European Conf. on Coputer Vson. pp , [21] J. Chverton, X. Xe, and M. Mrehd. Autoatc Bootstrappng and Trackng of Object Contours. IEEE Trans. on Iage Processng, vol. 21, no. 3, pp, , [22] T. Kalath, The dvergence and bhattacharyya dstance easures n sgnal selecton, IEEE Trans. Councaton Technology, vol. 15, no. 1, pp , [23] O. Mchalovch, Y. Rath, and A. Tannenbau, Iage Segentaton Usng Actve Contours Drven by the Bhattacharyya Gradent Flow, IEEE Trans. Iage Processng. vol. 16, no. 11, pp , [24] F. Goudal, P. Refreger, and G. Delyon, Bhattacharyya dstance as a contrast paraeter for statstcal processng of nosy optcal ages, J. Opt. Soc. A. A, vol. 21, no. 7, pp , July [25] T. F. Chan and L. A. Vese, Actve Contours wthout Edges, IEEE Trans. Iage Processng, vol. 10, no. 2, pp , [26] Grgoros Tsagkataks and Andreas Savaks, Onlne Dstance Metrc Learnng for Object Trackng IEEE Trans. Crcuts and Systes for Vdeo Technology, vol. 21, no. 12, ,

27 [27] B. Babenko, B., M. Yang, and S. Belonge, Trackng wth Onlne Multple Instance Learnng, IEEE Trans. Pattern Anal. Machne Intell., vol.33, no. 8, pp , [28] Yng-Ja Yeh and Chou-Tng Hsu, Onlne Selecton of Trackng Features Usng AdaBoost, IEEE Trans. Crcuts and Systes for Vdeo Technology, vol. 19, no. 3, , [29] T. M. Cover and J. A. Thoas, Eleents of nforaton theory, New York: Wley, [30] S. Baker, and I. Matthews, Lukas-kanade 20 years on: A unfyng fraework, Int l Journal of Coputer Vson, vol. 69, no. 3, pp , [31] G. Aubert, K. Perre, Matheatcal probles n age processng: partal dfferental equatons and the calculus of varatons. Sprnger, [32] W. F. Aes, Nuercal Methods for Partal Dfferental Equatons, 3rded. New York: Acadec, [33] K. Zhang, L. Zhang, H. Song and W. Zhou., Actve contours wth selectve local or global segentaton: a new forulaton and level set ethod., Iage and Vson Coputng, vol.28, ssue 4, pp , Aprl 2010 [34] G. H. Rosenfeld and K. Ftzpatrck Lns, A coeffcent of agreeent as a easure of theatc classfcaton accuracy, Photogra. Eng. Reote Sens., vol. 52, no. 2, pp , [35] Z. Zvkovc, EM-shft code, [36] 27