AGOOD appearance model is one of the most critical

Transcription

1 1510 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 24, NO. 5, MAY 2015 Robust Dscrmnatve Trackng va Landmark-Based Label Propagaton Yuwe Wu, Mngtao Pe, Mn Yang, Junsong Yuan, Member, IEEE, and Yunde Ja, Member, IEEE Abstract The appearance of an object could be contnuously changng durng trackng, thereby beng not ndependent dentcally dstrbuted. A good dscrmnatve tracker often needs a large number of tranng samples to ft the underlyng data dstrbuton, whch s mpractcal for vsual trackng. In ths paper, we present a new dscrmnatve tracker va landmarkbased label propagaton (LLP) that s nonparametrc and makes no specfc assumpton about the sample dstrbuton. Wth an undrected graph representaton of samples, the LLP locally approxmates the soft label of each sample by a lnear combnaton of labels on ts nearby landmarks. It s able to effectvely propagate a lmted amount of ntal labels to a large amount of unlabeled samples. To ths end, we ntroduce a local landmarks approxmaton method to compute the cross-smlarty matrx between the whole data and landmarks. Moreover, a soft label predcton functon ncorporatng the graph Laplacan regularzer s used to dffuse the known labels to all the unlabeled vertces n the graph, whch explctly consders the local geometrcal structure of all samples. Trackng s then carred out wthn a Bayesan nference framework, where the soft label predcton value s used to construct the observaton model. Both qualtatve and quanttatve evaluatons on the benchmark data set contanng 51 challengng mage sequences demonstrate that the proposed algorthm outperforms the state-of-the-art methods. Index Terms Vsual trackng, label propagaton, appearance changes, Laplacan regularzer. I. INTRODUCTION AGOOD appearance model s one of the most crtcal prerequstes for successful vsual trackng. Desgnng an effectve appearance model s stll a challengng task due to appearance varatons caused by background clutter, object deformaton, partal occlusons, and llumnaton changes, etc. Numerous trackng algorthms have been proposed to address ths ssue [1], [2], and exstng trackng algorthms can be Manuscrpt receved Aprl 21, 2014; revsed October 8, 2014 and November 29, 2014; accepted February 11, Date of publcaton February 19, 2015; date of current verson March 6, Ths work was supported n part by the 973 Program of Chna under Grant 2012CB720000, n part by the Natural Scence Foundaton of Chna under Grant , n part by the Specalzed Fund for Jont Buldng Program of Bejng Muncpal Educaton Commsson, and n part by the Mnstry of Educaton Ter-1 under Grant M The assocate edtor coordnatng the revew of ths manuscrpt and approvng t for publcaton was Dr. Nlanjan Ray. (Correspondng author: Mngtao Pe.) Y. Wu, M. Pe, M. Yang, and Y. Ja are wth the Bejng Laboratory of Intellgent Informaton Technology, School of Computer Scence, Bejng Insttute of Technology, Bejng , Chna (e-mal: wuyuwe@bt.edu.cn; pemt@bt.edu.cn; yangmnbt@bt.edu.cn; jayunde@bt.edu.cn). J. Yuan s wth the School of Electrcal and Electroncs Engneerng, Nanyang Technologcal Unversty, Sngapore (e-mal: jsyuan@ntu.edu.sg). Color versons of one or more of the fgures n ths paper are avalable onlne at Dgtal Object Identfer /TIP roughly categorzed as ether generatve [3] [7] or dscrmnatve [8] [14] approaches. Generatve methods buld an object representaton, and then search for the regon most smlar to the object. However, generatve models do not take nto account background nformaton. Dscrmnatve methods tran an onlne bnary classfer to adaptvely separate the object from the background, whch are more robust aganst appearance varatons of an object. In ths paper, we focus on the dscrmnatve trackng method. In vsual trackng scenaros, samples obtaned by the tracker are drawn from an unknown underlyng data dstrbuton. The appearance of an object could be contnuously changng and thus t s mpossble to be ndependent and dentcally dstrbuted (..d). A good dscrmnatve tracker often needs a large number of labeled samples to adequately ft the real data dstrbuton [15]. Ths s because f the dmensonalty of the data s large compared to the number of samples, then many statstcal learnng methods wll be overfttng due to the curse of dmensonalty. However, precsely labeled samples only come from the frst frame durng trackng,.e., the number of labeled samples s very small. To acqure more labeled samples, n most exstng dscrmnatve trackng approaches, the current trackng result s used to extract postve samples and the surroundng regons are used to extract negatve samples. Once the tracker locaton s not precse, the assgned labels may be nosy. Over tme, the accumulaton of errors can degrade the classfer and cause drft. Ths stuaton makes us wonder: wth a very small number of labeled samples, whether we can desgn a new dscrmnatve tracker whch makes no specfc assumpton about the sample dstrbuton. In ths paper, we take full advantage of the geometrc structure of the data and thus present a new dscrmnatve trackng approach wth landmark-based label propagaton (LLP). The LLP locally approxmates the soft label of each sample by a lnear combnaton of labels on ts nearby landmarks. It s able to effectvely propagate a lmted amount of ntal labels to a large amount of unlabeled samples, matchng the needs of dscrmnatve trackers. Under the graph representaton of samples, we employ a local landmarks approxmaton (LLA) method to desgn a sparse and nonnegatve adjacency matrx characterzng relatonshp among all samples. Based on the Nesterov s gradent projecton algorthm, an effcent numercal algorthm s developed to solve the problem of the LLA wth guaranteed quadratc convergence. Furthermore, the objectve functon of the label predcton provdes a promsng paradgm for modelng the geometrcal structures of samples va Laplacan regularzer. Preservng the local manfold structure IEEE. Personal use s permtted, but republcaton/redstrbuton requres IEEE permsson. See for more nformaton.

2 WU et al.: ROBUST DISCRIMINATIVE TRACKING VIA LLP 1511 Fg. 1. Landmark-based label propagaton for vsual trackng. The proposed method treats both labeled and unlabeled samples as vertces n a graph. For each new frame, canddates predcted by partcle flter are consdered as unlabeled samples and utlzed to consttute a new graph representaton. The label of each sample s a locally weghted average of the labels on landmarks. Then the classfcaton scores f of canddates are used to construct the observaton model of the partcle flter to determne the best canddate. of samples can make our tracker have more dscrmnatng power to handle appearance changes. Fg. 1 shows the flow dagram of vsual trackng usng the LLP. Specfcally, the proposed method treats both labeled and unlabeled samples as vertces n a graph and bulds edges whch are weghted by the affntes (smlartes) between the correspondng sample pars. For each new frame, canddates predcted by the partcle flter are consdered as unlabeled samples and utlzed to consttute a new graph representaton together wth the collected samples stored n the sample pool. A small number of landmarks obtaned from the entre sample space enable nonparametrc regresson that calculates the soft label of each sample as a locally weghted average of labels on landmarks. Trackng s carred out wthn a Bayesan nference framework where the soft label predcton value s used to construct the observaton model. A canddate wth the hghest classfcaton score s consdered as the trackng result. To allevate the drft problem, once the tracked object s located, the labels of the newly collected samples are assgned accordng to the classfcaton score of the current trackng results, n whch no self-labelng s nvolved. The proposed tracker adapts to drastc appearance varatons, as valdated n our experments. The remander of ths paper s organzed as follows. For the ease of readng, we frstly dscuss the related work n Sect. II. In Sect. III, we ntroduce the landmark-based label propagaton method to tran an effectve classfer. Then the trackng algorthm based on the LLP s presented n Sect. IV. Expermental results and demonstratons are reported and analyzed n Sect. V and the concluson s gven n Sect. VI. II. RELATED WORK Dscrmnatve trackng has receved wde attenton for ts adaptve ablty to handle appearance changes. In ths secton, we only dscuss the most relevant lterature wth our method. Interested readers may refer to [2] for a comprehensve revew. The essental component of dscrmnatve trackers s the classfer learnng. Many trackers employ onlne supervsed learnng methods to tran the classfers. Avdan [16] ntroduced an ensemble trackng method n whch a set of weak classfers s traned and combned for dstngushng the object and the background. The features used n [16] may contan redundant and rrelevant nformaton whch affects the classfcaton performance. Collns et al. [17] developed an onlne feature selecton mechansm usng the two-class varance rato to fnd the most dscrmnatve RGB color combnaton n each frame. Grabner et al. [18] proposed an onlne boostng feature selecton method for vsual trackng. However, above-mentoned methods [16] [18] only utlze one postve sample (.e., the trackng result n the current frame) and multple negatve samples to update the classfer. If the object locaton s not perfectly detected by the current classfer, the appearance model would be updated wth a suboptmal postve example. Over tme the accumulaton of errors can degrade the classfer, and can cause drft. Numerous approaches also apply multple postve samples and negatve samples to tran classfers. Babenko et al. [11] ntegrated multple nstance learnng (MIL) nto onlne boostng algorthm to allevate the drft problem. In the MIL tracker, the classfer s updated wth postve and negatve bags rather than ndvdual labeled examples. Zhang and Maaten [19] developed a structure-preservng object tracker that learns spatal constrants between objects usng an onlne structured SVM algorthm to mprove the performance of sngle-object or mult-object trackng. Wu et al. [20] and Jang et al. [21] addressed vsual trackng by learnng a sutable metrc matrx to effectvely capture appearance varatons, such that dfferent appearances of an object wll be close to each other and be well dstngushed from the background. Dscrmnatve trackers also explot the sem-supervsed learnng scheme to address the appearance varatons. Grabner et al. [9] employed an onlne sem-supervsed learnng framework to tran a classfer by only labelng samples n the frst frame and leavng subsequent samples unlabeled. Although ths method has shown to be less susceptble to drft, t s not adaptve enough to handle fast appearance changes. Kalal et al. [13] developed a P-N learnng method to tran a bnary classfer wth structured unlabeled data. Zesl et al. [22] presented a coherent framework whch s able to combne both onlne sem-supervsed learnng and multple nstance learnng. Recently, researchers utlzed the graph-based dscrmnatve learnng to construct the object appearance model for vsual trackng. Zha et al. [23] employed the graph-based transductve learnng to capture the underlyng geometrc structure of samples for trackng. Wth the 2 nd -order tensor representaton, Gao et al. [24] desgned two graphs for characterzng the ntrnsc local geometrcal structure of the tensor space. Based on the least square support vector machne, L et al. [25] exploted a hypergraph propagaton method to capture the contextual nformaton on samples, whch further mproves the trackng accuracy. Kumar and Vleeschouwer [26] constructed a number of dstnct graphs (.e., spatotemporal, appearance and excluson) to capture the spato-temporal and the appearance nformaton. Then, they formulated the mult-object trackng as a consstent labelng problem n the assocated graphs. In works of [9], [11], [13], and [20], canddates are not used to tran a classfer, and therefore the class labels of

3 1512 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 24, NO. 5, MAY 2015 them are assgned by the prevous classfer. Dfferent from these works, n our tracker, canddates are consdered as unlabeled samples and utlzed to consttute a new graph representaton to update the current classfer for each new frame, as llustrated n Fg. 1. Explctly takng nto account the local manfold structure of labeled and unlabeled samples, we ntroduce a soft label propagaton method defned over the graph, whch has more dscrmnatng power. In addton, once the tracked object s located, the new tranng samples are collected both n a supervsed and unsupervsed way whch makes our tracker more stable and adaptve to appearance changes. More detals are dscussed n Sect. IV. Our method dffers from [23] [25] both n the graph constructon and the label propagaton method. Methods n [23] [25] construct the graph representaton usng knn whose computatonal cost s expensve. In contrast, employng local landmarks approxmaton, we desgn a new form of the adjacency matrx characterzng the relatonshp between all samples. The total tme complexty scales lnearly wth the number of samples. More mportantly, our method s an nductve model whch can be used to nfer the labels of unseen data (.e., canddates). The label of each sample can be nterpreted as the weghted combnaton of the labels on landmarks. Graph Laplacan s ncorporated nto the objectve functon of soft label predcton as a regularzer to preserve the local geometrcal structure of samples. III. LANDMARK-BASED LABEL PROPAGATION In ths secton, we ntroduce a smple yet effectve lnear classfer. The core dea of our model s that the label of each sample can be nterpreted as the weghted combnaton of the labels on landmarks. Employng local landmarks approxmaton, we desgn a new form of the adjacency matrx characterzng the relatonshp between all samples. Graph Laplacan s ncorporated nto the objectve functon of sem-supervsed learnng as a regularzer to preserve the local geometrcal structure of samples, whch makes our model have more dscrmnatve power compared to tradtonal sem-supervsed learnng methods. A. Problem Descrpton Suppose that we have l labeled samples {(x, y )} l =1 and u unlabeled samples {x } l+u =l+1,wherex R d,andy R c s the label vector. Snce dscrmnatve models take trackng as a bnary classfcaton task to separate the object from ts surroundng background, the number of classes c equals 2. Denote X = {x 1, x 2,, x n } R d n and Y l ={y 1, y 2,, y l } R l c,wheren = l + u. Ifx belongs to the kth class (1 k c), the kth entry n y s 1 and all the other entres are 0 s. In ths paper, the data X s represented by the undrected graph G ={X, E}, where the set of vertces s X = {x } and the set of edges s E = {e j }, where e j denotes the smlarty between x and x j.defneasoft label predcton (.e., classfcaton) functon f : R d R c. A crucal component of our method s the estmaton of a weghted graph G from X. Then, the soft label of any sample can be nferred usng G and known labels Y l. The tme complexty of tradtonal graph-based semsupervsed learnng methods s usually O(n 3 ) wth respect to the data sze n, because n n kernel matrx (e.g., multplcaton or nverse) s calculated n nferrng the label predcton. Snce full-sze label predcton s nfeasble when n s large, the work of [27] nspres us to explot the dea of landmark samples. To accomplsh the soft label predcton, we employ an economcal and practcal predcton functon expressed as f (x) = m K (x, d k )a k, (1) k=1 where d k denotes the k-th landmark, a k s the label of the k-th landmark, and K (x, d k ) represents the cross-smlarty weght between the data x and the landmark d k. The dea of Eq. (1) s that the label of each sample can be nterpreted as the locally weghted average of varables a k s defned on m landmarks [27], [28]. As a trade-off between computatonal effcency and effectveness, n ths paper, k-means algorthm s used to select the centers as the set of landmarks D ={d k } m k=1 Rd m. Eq. (1) s deemed as a label propagaton model, because t can dffuse the label of landmarks to all unlabeled samples, as dscussed n Sect. III-D. It avods optmzng the labels of all the samples, by just concentratng on the labels of the landmarks. Unlke the tradtonal label propagaton method [29], our model takes full advantage of the geometrc structure of the data and makes no specfc assumpton about the sample dstrbuton. The above model can be wrtten n a matrx form F = HA, (2) where F = [f (x 1 ), f (x 2 ),, f (x n )] R n c s the landmark-based label predcton functon on all samples. A = [f (d 1 ), f (d 2 ),, f (d m )] = [A 1, A 2,, A c ] R m c denotes the label of landmarks d k s. H R n m s the cross-smlarty matrx between the whole data X and landmarks d k, H k = K (x, d k )>0, 1 n, 1 k m. In what follows, we wll elaborate how to effectvely solve A and H. B. Solvng Optmal H Typcally, we may employ Gaussan kernel or Epanechnkov quadratc kernel [30] to compute H. However, choosng approprate kernel bandwdths s dffcult. Instead of adoptng the predefned kernel, we learn an optmal H by consderng the geometrc structure nformaton between labeled and unlabeled samples. We reconstruct x as a combnaton of ts s closest landmarks n the feature space. In ths work, we employ the Eucldean dstance to select the s = 10 closest landmarks for the gven sample x. Recently, Wang et al. [31] proposed localty-constraned lnear codng (LLC) whch uses the localty constrants to project each descrptor nto ts localcoordnate system [32]. To enhance the codng effcency,

4 WU et al.: ROBUST DISCRIMINATIVE TRACKING VIA LLP 1513 approxmated LLC s proposed n [31], n whch the localty constrant functon s replaced by usng the s closest landmarks. For each x approxmated LLC s defned as Algorthm 1 Solvng the Eucldean Projecton Operator C (v) mn h R s x Dh 2, (3) where D R d s s the s closest landmarks of x. Inspred by the dea of LLC, our goal s to desgn a both sparse and optmal cross-smlarty matrx H between the whole data X and landmarks D. A Local Landmarks Approxmaton (LLA) method s proposed to optmze the coeffcent vector h R s for each data pont x, correspondng to the followng problem: mn g(h ) = 1 s h R s 2 x d j h j 2, j=1 s.t. 1 h = 1, h j 0 (4) where h j s the coeffcent actvated by the j th nearby landmark of x.thes entres of the vector h correspond to the s coeffcents contrbuted by the s nearest landmarks. The constrant 1 h = 1 follows the shft-nvarant requrements. The man dfference between LLC and our method s that we ncorporate nequalty constrants (.e., non-negatve constrants) nto the objectve functon as we requre the smlarty measure to be a postve value. Therefore we need to develop a dfferent optmzaton algorthm to solve Eq. (4). It s easy to see that the constrants set C ={h R s : 1 h = 1, h j 0} s a convex set. Standard quadratc programmng (QP) algorthms can be used to solve Eq. (4) but most of them are computatonally expensve for computng an approxmaton of the Hessan. To speed up the convergence rate, Nesterov s gradent projecton (NGP) method [33], a frstorder optmzaton procedure, s employed to solve the constraned optmzaton problem Eq. (4). A key step of NGP s how to effcently project a vector h onto the correspondng constrant set C. 1) Eucldean Projecton Onto the Smplex: For smplcty, let v R s denote the vector whch needs to be mapped onto C, and v be the output. Therefore, the Eucldean projecton of v R s onto C s to solve the followng optmzaton problem: 1 C (v) = arg mn v C 2 v v 2 2 s.t. 1 v = 1, v 0, (5) where C (v) denotes the Eucldean projecton operator on any v R s. The Lagrangan of the problem n (5) s L(v,ω) = 1 k 2 v v μ( v 1) ω v, (6) where μ s a Lagrange multpler and ω s a vector of non-negatve Lagrange multplers. By settng the dervatve w.r.t. v to zero, we have L v =1 = v v + μ ω = 0. (7) The complementary slackness KKT condton mples that whenever v > 0 we have ω = 0. Thus, we can get v = max{v μ, 0}, where μ = ρ 1 ( ρ =1 z 1 ) and { ρ = max [1 : s] : z 1 ( r=1 z r 1 ) > 0}. z denotes the vector obtaned by sortng v n a descendng order. The projecton operator C ( ) can be mplemented effcently n O(s log s) [34]. The eucldean projecton onto the smplex s summarzed n Algorthm 1. For more detals, please refer to [34]. 2) Nesterov s Gradent Projecton (NGP): We use NGP to solve the constraned optmzaton problem Eq. (4) by adoptng the Eucldean projecton. Denote Q β,v (h ) = g(v) + g(v) (h v) + β 2 h v 2 2, (8) whch s the frst-order Taylor expanson of g(h ) at v wth the squared Eucldean dstance between h and v as a regularzaton term. Here g(v) s the gradent of g(h ) at v. Accordng to Eq. (5), we can easly obtan ( arg mn Q β,v(h ) = C v 1 ). h C β g(v) (9) From Eq. (9), the soluton of Eq. (4) can be obtaned by generatng a sequence {h (t) } at v (t) = h (t) ( (t) + α t h h (t 1) ),.e., h (t+1) = C (v (t) 1 ) g(v (t) ) β t = arg mn Q β h C t,v (t)(h ). (10) In NGP, choosng proper parameters β t and α t s also sgnfcant for the convergence property. Smlar to [33], we set α t = (δ t 1 1)/δ t wth δ t = ( δt 1) 2 /2, δ0 = 0and δ 1 = 1. β t s selected by fndng the smallest nonnegatve nteger j such that g(h ) Q βt,v (t)(h ) wth β t = 2 j β t 1. In [35], Nesterov states that NGP has a convergence rate O(1/t 2 ). The convergence property s summarzed n Theorem 1. The solvng process of Eq. (4) s summarzed n Algorthm 2. Theorem 1: Employng NGP to solve the constraned optmzaton problem (4) by adoptng the Eucldean projecton, for any t, we have g(s (t+1) ) mn g(s ) 2 β L s (0) s 2 2 s C (t + 1) 2, (11) where β L = max(2β L,β 0 ), β 0 s the ntal estmaton of gradent Lpschtz constant β L of g(s ). For s and v, β L satsfes g(s ) g(v) 2 β L s v 2. Besdes, the frst t steps of the method requre t evaluatons of g(s ) and no more than 2t + log 2 ( β L /β 0 ) evaluatons of g(s ).

5 1514 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 24, NO. 5, MAY 2015 Algorthm 2 Nesterov s Gradent Projecton for Solvng the Optmal H G s = dag( 1, 2,, n ). To compute the label predcton matrx A, we explot the followng optmzaton framework [27]: mn η 2 f G + L( f l, y l ). (13) After gettng the optmal weght vector h, we set H, = h,where s the vector of ndces correspondng to the s nearest landmarks and the cardnalty = s. Forthe remanng entres of H,, we set 0 s. Apparently, H j = 0 when landmark d j s far away from x and H j = 0 s only for the s closest landmarks of x. In contrast to weghts defned by kernel functon (e.g., Gaussan kernel), the LLA s able to provde optmzed and sparser weghts, as valdated n our experments. C. Solvng Label Predcton Matrx A Note that the adjacency matrx W R n n between all samples encountered n practce usually has low numercalrank compared to the matrx sze [36]. We consder whether we can construct a nonnegatve and emprcally sparse graph adjacency matrx W wth the nonnegatve and sparse H R n m ntroduced n Sect. III-B. Interestngly, each row H n H can be a new representaton of raw sample x. x H s remnscent of sparse codng [31] wth the bass D snce x Dh = DH,where D R d s s a sub-matrx composed of s nearest landmarks of x. That s to say, samples X R d n can be represented n the new space, no matter what the orgnal features are. Intutvely, we can desgn the adjacency matrx W to be a low-rank form W = HH, (12) where the nner product s regarded as the metrc to measure the adjacent weght between samples. Eq. (12) mples that f two samples are correlatve (.e., W j > 0), they share at least one landmark, otherwse W j = 0. W defned n Eq. (12) naturally preserves some good propertes (e.g., sparseness and nonnegatveness). The effectveness of W wll be demonstrated n Sect. V-E2. We defne the degree of x as = n j=1 W j. Therefore, the vertex degree matrx of the whole The frst term f G n Eq. (13) enforces the smoothness of f wth regard to the manfold structure of the graph, and s formulated as n f 2 G = f (x ) f (x j ) 2 W j = =, j=1 n, j=1 ( ) f (x ) 2 + f (x j ) 2 2 f (x ) f (x j ) W j n f (x ) 2 + =1 2 n, j=1 n f (x j ) 2 jj, j=1 f (x ) f (x j )W j = 2Tr ( F F F WF ) = 2Tr ( F LF ) (14) where L = W s the graph-based regularzaton matrx L R n n,andtr( ) s a matrx trace operaton. Substtutng W = HH nto Eq. (14), Laplacan graph regularzaton can be approxmated as F LF = F (dag(hh 1) HH )F, (15) where nonnegatve W guarantees the postve sem-defnte (PSD) property of L. Keepng L PSD s mportant as t ensures that the graph regularzer F LF s convex. The second term L(, ) n Eq. (13) s an emprcal loss functon, whch requres that the predcton f should be consstent wth the known class labels. η s a postve regularzaton parameter. f l R l c s the sub-matrx correspondng to the labeled samples n f R n c. By pluggng F = HA nto Eq. (13) and choosng the loss functon L(, ) as the L2-norm, the convex dfferentable objectve functon for solvng label predcton matrx A can be formulated as mn L(A) = η Tr ( F LF ) + H l A Y l 2 F A = η Tr ( (HA) L(HA) ) + H l A Y l 2 F. (16) Here, H l R l m s made up of the rows H that corresponds to the labeled samples, and L s defned n Eq. (15). We easly obtan L A = 2η( H LHA ) + 2Hl (H l A Y l ). (17) By settng the dervatve w.r.t. A to zero, the globally optmal soluton of Eq. (16) s gven by A = ( Hl H l + ηh LH ) 1 H l Y l. (18)

6 WU et al.: ROBUST DISCRIMINATIVE TRACKING VIA LLP 1515 model prevously predefned n Eq. (2). The fnal trackng result can be determned by the sum of the classfcaton scores of the fve mage patches nsde the object regon: Fg. 2. Object representaton usng fve dfferent mage patches. The canddate s normalzed to the same sze (24 24 n our experment), each mage patch s wth D. Soft Label Propagaton Through applyng the label propagaton model Eq. (2), we are able to predct the soft label for any sample x (unlabeled tranng samples or new test samples) as H(x ) A k f (x ) = max k {1,2} 1, (19) HA k where {A k } c k=1 Rm 1 s the column vector of A, and H(x ) R 1 m represents the weght between x and landmarks d k s. Specfcally, f x belongs to unlabeled tranng samples, H(x ) = H where H denotes the -th row of H, = l + 1,, n. Ifx s a new test sample, we need to compute the vector H as H(x ) descrbed n Algorthm 2, then update H R (n+1) m,.e., H [H; H ]. After dervng the soft label predcton (.e., classfcaton) of each sample, the classfcaton score can be utlzed as the smlarty measure for trackng. In the next secton, we wll elaborate the applcaton of the proposed landmark-based label propagaton n trackng. IV. LLP TRACKER In ths secton, wth the landmark-based label propagaton ntroduced n Sect. III, we propose the LLP tracker based on Bayesan nference. In our tracker, the patch-based mage representaton s able to handle partal occluson. Once the tracked object s located, the labels of the newly collected samples are determned by the classfcaton score of the current trackng results, n whch no self-labelng s nvolved. Ths labelng strategy s effectve to allevate the drft problem. A. Object Representaton In order to potentally allevate the drft caused by partal occlusons, we employ the part-based scheme to tran the classfer n our trackng framework. As a trade-off between computatonal effcency and effectveness, the object s dvded nto 5 dfferent mage patches emprcally. That s, an object s represented by fve mage feature vectors nsde the object regon. The frst patch s the entre object. Then the object s parttoned nto 2 2 subsets whch consttute the 4 remanng patches. These fve mage patches correspond to the fve parts of an object, respectvely, as exemplfed n Fg. 2. Fnally, mage patches correspondng to the same part of all samples construct a sub-sample set X (τ), τ = 1, 2,, 5. For example, the frst patch of all samples consttute the frst sub-sample set. Each sub-sample set X (τ) s used to tran a sngle classfer f (τ) usng the label propagaton SC = 5 ω τ f (τ), (20) τ=1 where ω τ s the weght of the τ-th mage patch ( 5 τ=1 ω τ = 1 and ω τ = 0.2 n the experments). Ths part-based scheme could potentally allevate the drft caused by partal occlusons. B. Classfer Intalzaton To ntalze the classfer n the frst frame, we draw postve and negatve samples around the object locaton. Suppose the object s labeled manually, perturbaton (e.g., shftng 1 or 2 pxels) around the object s performed for collectng N p postve samples X Np. Smlarly, N n negatve samples X Nn are collected far away from the located object (e.g., wthn an annular regon a few pxels away from the object). X 1 = X Np X Nn s the ntalzed labeled sample set. Accordng to dscusson n Sect. IV-A, each sample n X 1 s parttoned nto 5 dfferent patches. X 1 thus contans 5 subsets. The k-means algorthm s used to select the centers as the set of landmarks D n each subset. Usng labeled samples and landmarks, we can tran pror classfers va the LLP. C. Updatng the Samples and Landmarks For each new frame, canddates predcted by the partcle flter are consdered as unlabeled samples X. Accordng to Eq. (19), we can get the classfcaton score of each canddate. A canddate wth hgher classfcaton score ndcates that t s more lkely to be generated from the target class. The most lkely canddate s consdered as the trackng result for ths frame. Then, perturbaton (.e., the same scheme n the frst frame) around the trackng result s performed for collectng sample set X C. If the classfcaton score of the located object s hgher than the predefned threshold ɛ (.e., the current trackng result s relable), samples n X C are regarded as labeled ones, otherwse regarded as unlabeled ones. That s, samples are collected both n a supervsed and unsupervsed way, and thus the stablty and adaptvty n trackng objects of changng appearance are preserved. To make our tracker more adaptve to appearance changes, we construct a sample pool X P and a sample buffer pool X to update the samples and landmarks, as shown n Fg. 3. We keep a set of T prevous X C to consttute the sample buffer pool X,.e., X = [X C T +1 ; X C T +2 ; ;X C ], where X C denotes the sample set collected from the current frame. Every T frames, X s utlzed to update X P.After updatng the sample pool, we wll leave X empty and then reconfgure t. In our experment, we set the sample pool capacty (X P ). 1 If the total number of samples n the sample pool s larger than (X P ), some samples n X P wll be randomly replaced wth samples n X. To reduce the rsk of 1 The cardnalty (X P ) denotes the number of samples n the sample pool.

7 1516 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 24, NO. 5, MAY 2015 Fg. 3. Illustraton of constructng the sample pool and sample buffer pool. For each new frame, canddates cropped by the partcle flter are consdered as unlabeled samples. Every T frames, the sample pool s updated by the sample buffer pool. After updatng, we wll leave the sample buffer pool blank and then reconfgure t. Fg. 4. The set of landmarks updatng. The updated set of landmarks s obtaned by carryng out twce k-means. vsual drft, we always retan the samples X 1 obtaned from the frst frame n the sample pool. That s, X P =[X 1 ; X ]. Note that canddates are consdered as unlabeled samples and utlzed to tran the classfer together wth collected samples stored n the sample pool. Smlarly, landmarks also should be updated usng the sample pool every T frames. Specfcally, we frst mplement the k-means algorthm n the current sample pool X P to obtan a new landmarks set. Then, the updated set of landmarks can be ganed by carryng out the k-means algorthm agan usng both the new and the prevous landmarks set whch are able to better characterze the samples dstrbuton. The landmarks updatng are llustrated n Fg. 4. D. Bayesan State Inference Object trackng can be consdered as a Bayesan nference task n a Markov model wth hdden state varables. Gven the observaton set of the object O 1:t = {o 1, o 2,, o t }, the optmal state s t of the tracked object s obtaned by the maxmum a posteror estmaton p ( st ) O 1:t,wheres t ndcates the state of the -th sample. The posteror probablty p ( ) ( ) s O1:t t s formulated by Bayes theorem as p st O1:t p(o t s t ) p ( ) ( ) s t s t 1 p st 1 O1:t 1 dst 1. Ths nference s governed by the dynamc model p ( ) s t s t 1 whch models the temporal correlaton of the trackng results n consecutve frames, and by the observaton model p(o t s t ) whch estmates the lkelhood of observng o t at state s t. Wth partcle flterng, the posteror p ( ) s t O 1:t s approxmated by a fnte set of N s samples or partcles { st } Ns =1 wth mportance weghts { ωt } Ns =1. The partcle sample s t s drawn from an mportance dstrbuton q(s t s 1:t 1, O 1:t ), whch for smplcty s set to the dynamc model p(s t s t 1 ). The mportance weght ωt of partcle s equal to the observaton lkelhood p(o t st ). We apply an affne mage warp to model the object moton between two consecutve frames. Let s t = {x t, y t,θ t, s t,η t,ψ t },wherex t, y t, θ t, s t, η t, ψ t denote x, y translatons, rotaton angle, scale, aspect rato and skew at tme t, respectvely. The state transton dstrbuton p ( ) ( ) s t s t 1 s modeled by Brownan moton,.e., p st s t 1 = N (s t ; s t 1, ), where s a dagonal covarance matrx whose dagonal elements are the correspondng varances of respectve parameters. The observaton model p(o t s t ) s defned as p(o t s t ) SC t, (21) where SC t = f ( x (t)) s the classfcaton score at tme t based on Eq. (19). The detaled descrpton of the proposed trackng method s summarzed n Algorthm 3. V. EXPERIMENTS We run our tracker on the benchmark dataset [37] ncludng 51 challengng mage sequences. The total number of frames on the benchmark s more than We evaluate the proposed tracker aganst the 11 state-of-the-art trackng algorthms ncludng ONNDL [38], RET [39], CT [40], VTD [5], MIL [11], SCM [41], Struck [12], TLD [13], ASLSA [3], LSST [4] and SPT [14]. For far comparsons, we use the source codes provded by the benchmark wth the same parameters except ONNDL, RET, LSST and SPT whose parameters of the partcle flter are set as n our tracker. Snce the trackers nvolve randomness, we run them 5 tmes and report the average result for each sequence. The MATLAB source code and expermental results of the 12 trackers are avalable at A. Expermental Setup Note that we fx the parameters of our tracker for all sequences to demonstrate ts robustness and stablty. The number of partcles s 400 and the state transton matrx s [8, 8, 0.01, 0.001, 0.005, 0] n the partcle flter. We resze

8 WU et al.: ROBUST DISCRIMINATIVE TRACKING VIA LLP 1517 Algorthm 3 The Proposed Trackng Algorthm Fg. 5. Overall performance comparsons of precson plot and success rate. The performance score for each tracker s shown n the legend (best vewed on hgh-resoluton dsplay). the object mage to pxels and each mage patch s pxels, as llustrated n Fg dmensonal gray scale feature and 128 dmensonal HOG feature are extracted from each mage patch, and they are concatenated nto a sngle feature vector of 272 dmensons. In the frst frame, N p = 20 postve samples and N n = 100 negatve samples are used to ntalze the classfer. The predefned threshold of classfcaton score ɛ s set to 0.3. Gven the object locaton at the current frame, f SC 0.3, 2 postve samples and 50 negatve samples are used for the supervsed learnng. If SC < 0.3, the trackng result s treated as the unrelable one and 100 unlabeled samples are utlzed for the unsupervsed learnng. The sample pool capacty (X P ) s set to 310, n whch the number of postve, negatve and unlabeled samples are 50, 160 and 100, respectvely. The number of landmarks s set to 30 emprcally and the regularzaton parameter expressed n Eq. (18) s set to η = The set of landmarks D s updated every T = 10 frames. B. Evaluaton Crtera One wdely used evaluaton method to measure the trackng results s the center locaton error. It s based on the relatve poston errors (n pxels) between the central locatons of the tracked object and those of the ground truth. Ideally, an optmal tracker s expected to have a small error. However, when the tracker lost the object for some frames, the output locaton can be random and therefore the average center locaton errors may not evaluate the trackng performance correctly [37]. In ths paper, the precson plot s also adopted to measure the overall trackng performance. It shows the percentage of frames whose estmated locaton s wthn the gven threshold dstance (e.g., 20 pxels) of the ground truth. More accurate trackers have hgher precson at lower thresholds. If a tracker loses the object t s dffcult to reach a hgher precson [42]. The trackng overlap rate ndcates stablty of each algorthm as t takes the sze and pose of the target object nto account [43]. It s defned by score = area(roi T ROIG ) area(roi T ROIG ), where ROI T s the trackng boundng box and ROI G s the ground truth. Ths can be used to evaluate the success rate of any trackng approach. Generally, the trackng result s consdered as a success when the score s greater than the gven threshold t s. It may not be far or representatve for tracker evaluaton usng one success rate value at a specfc threshold (e.g., t s = 0.5). Further, we count the number of successful frames as the thresholds vary from 0 to 1 and plot the success rate curve for our tracker and the compared trackers. The area under curve (AUC) of each success rate plot s employed to rank the trackng algorthms. More robust trackers have hgher success rates at hgher thresholds. C. Overall Performance The overall performance for the 12 trackers s summarzed by the precson plot and the success rate on the 51 sequence, as shown n Fg. 5. For precson plots, we use the results at error threshold of 20 pxels for rankng these 12 trackers. The AUC score for each tracker s shown n the legend. In success rate, our tracker s 4.6% above the SCM, and outperforms the Struck by 3.4% n precson plot. SCM, ASLSA and LSST trackers also perform well n success rate, whch suggests sparse representatons are effectve models to account for the appearance change, especally for occluson. Snce the Struck does not handle scale varaton, the success rate of Struck s hgher than SCM, LSST and ALSA when the overlap threshold t s s small, but less than SCM, LSST and ASLSA when t s s large (e.g., t s = 0.4). Overall, our tracker outperforms the other 11 trackers both n precson plot and success rate. The good performance of our method can be attrbuted to the fact that the classfer generalzes well on the new data from a lmted number of tranng samples. That s, our method has excellent generalzaton ablty. In addton, the local manfold structure of samples makes the classfer have more dscrmnatng power.

9 1518 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 24, NO. 5, MAY 2015 Fg. 6. Attrbute-based performance analyss n success rate. The performance score of each tracker s shown n the legend (best vewed on hgh-resoluton dsplay). D. Attrbute-Based Performance Apart from summarzng the performance on the whole sequences, we also construct the 11 subsets correspondng to dfferent attrbutes to test the trackng performance under specfc challengng condtons. Because the AUC score of the success plot s more accurate than the score at one threshold (e.g., 20 pxels) of the precson plot, n the followng we manly analyze the rankngs based on success plots, as shown n Fg. 6. On the OCC subset, SCM, ASLSA, LSST and our method get better results than others. The results suggest that local mage representatons are more effectve than holstc templates n dealng wth occlusons. On the SV subset, we see that trackers wth affne moton models (e.g., our method, SCM, ASLSA and LSST) are able to cope wth scale varaton better than others that only consder translatonal moton (e.g., Struck and MIL). Smlarly, on the OPR and IPR subsets, besdes our tracker, the SCM and ASLSA trackers are also able to obtan satsfactory results. The performance of SCM and ASLSA trackers can be attrbuted to the effcent spare representatons of local mage patches. When the object undergoes fast moton and/or moton blur, our method performs worse than the Struck, SPT trackers due to the poor dynamc models n the partcle flter. Our tracker can be further mproved wth more effectve state transton matrx of the partcle flter. In the LR subset, our tracker does not perform well, because low-resoluton objects whch are reszed to may not capture suffcent vsual nformaton to represent objects for trackng. E. Dagnostc Analyss In ths secton, we analyze two aspects of our landmarkbased label propagaton that are mportant for good trackng results,.e., the weght H between the whole samples and landmarks, and the label predcton matrx A. 1) Effectveness of the Optmal H: To evaluate the contrbuton of the optmal H descrbed n Sect. III-B to the overall performance of our tracker, we compute the Nadaraya-Watson kernel regresson [30] for comparson. It assgns weghts smoothly wth K σ (x, d k ) H k = mj=1 1 n, 1 j m. (22) K σ (x, d j ) Two kernel functons are exploted n the Nadaraya-Watson kernel regresson to measure the cross-smlarty matrx between the whole data X and landmarks d k s. We frst adopt Gaussan kernel K σ (x, d k ) = exp ( x d k 2 /σ 2) for the kernel regresson. Therefore, the correspondng trackng method s called as the BaseLne1 tracker. Epanechnkov quadratc kernel expressed as 3( 1 x d K σ (x, d k ) = k 2) f x d k 1; 4 0 otherwse. s also employed for the kernel regresson, whose correspondng trackng method s referred to as the BaseLne2 tracker. We use a more robust way to get σ whch uses the nearest neghborhood sze s of x to replace σ,.e., σ(x ) = x d s 2, where d s s the sth closest landmarks of x.

10 WU et al.: ROBUST DISCRIMINATIVE TRACKING VIA LLP 1519 Fg. 7. Dagnostc analyss of our tracker on the 51 sequence. Wth fxed A, BaseLne1 and BaseLne2 use dfferent cross-smlarty matrx H. Smlarly, BaseLne3 and BaseLne4 use dfferent soft label predcton matrx A wth fxed H. The only dfference between baselne algorthms and Ours s that baselne algorthms utlze the predefned kernel functons to solve cross-smlarty matrx H whle Ours takes advantage of the LLA method to optmze H. The overall trackng performance of these two baselne algorthms and our method on the benchmark s presented n Fg. 7. On the whole, our method obtans more accurate trackng results than baselne algorthms. 2) Effectveness of the Predcton Matrx A: We desgn another two baselne algorthms to evaluate the effectveness of the soft label predcton matrx A descrbed n Sect. III-C. In the BaseLne3, we do not consder the Laplacan graph regularzer n Eq. (16),.e., η = 0, and thus A becomes the least-squares soluton. In the BaseLne4, we drectly construct the adjacent matrx W usng the knn algorthm nstead of W = HH. If x s among the k-neghbors of x j or x j s among the k-neghbors of x, W j = 1, otherwse, W j = 0. The overall trackng performance on the benchmark s llustrated n Fg. 7. Surprsngly, even wthout Laplacan graph regularzer, the BaseLne3 produces the precson score of and the success score of 0.504, outperformng the SCM tracker, whch mples that the success s due to the framework of the landmark-based label propagaton. The overall performance can be further mproved usng our scheme of solvng A descrbed n Sect. III-C. F. Qualtatve Comparsons 1) Sgnfcant Pose Varatons: Fg. 8 shows trackng results of three challengng sequences wth sgnfcant pose varatons to verfy the effectveness of our method. In the Basketball sequence, the object appearance change drastcally as the players run sde to sde, especally for close-fttng defence between players. We see that SPT, CT, RET and SCM trackers are easy to drft at the begnnng of the sequence (e.g., 60). The TLD, ONNDL, Struck and MIL algorthms drft to another player as the appearance between players n the same team s very smlar (e.g., 473). VTD, ASLSA and our methods are able to track the whole sequence successfully. We note that the VTD perform better than the other methods. Ths can be attrbuted to that the object appearance can be well approxmated by multple basc observaton models. The Freeman4 sequence s used to test the performance of our method n handlng pose changes. There are partal occlusons and scale changes when the object walks toward the camera. Most methods fal to track the object. For example, CT does not manage to get a stable result due to potental randomness. Although TLD has a re-ntalzaton mechansm after occluson, t locks onto the wrong person as the surroundng background s very smlar to the object (e.g., 142). In comparson, our method s able to provde a trackng boundng box that s much more accurate and consstent. In the Shakng sequence, the target undergoes llumnaton change besdes pose varatons. the Struck, LSST, TLD, CT and RET trackers drft from the object quckly when the spotlght blnks suddenly (e.g., 60). SCM, VTD and our trackers are able to successfully track the object throughout the sequence wth relatvely accurate szes of the boundng box. SPT, ONNDL, MIL and ASLSA methods are also able to track the object n ths sequence but wth a lower success rate than our method. In ths sequence, the VTD performs better than the other methods. 2) Heavy Occluson: Fg. 9 shows results from three challengng sequences wth heavy occluson. Images of the Woman sequence are acqured by a movng camera and the object color sometmes appears smlar to the background clutter. Many methods cannot keep trackng of the object after occluson. The CT, SCM, MIL, VTD, TLD and ONNDL trackers fal to capture the object after the woman walks behnd the whte car (e.g., 127). The appearance model fuses more background nterference due to an occluson, whch sgnfcantly nfluences the samples onlne updatng of the MIL, TLD and ASLSA trackers. The LSST tracker fals gradually over tme (e.g., 297). Although the RET method tracks well, our method, SPT and Struck trackers acheve more stable performance n the entre sequence. In the SUV sequence, most of the trackers drft when the long-term occluson happens (e.g., 552). Trackng such an object s extremely challengng because the vehcle s almost ndstngushable behnd the trees, even for human eyes. Although VTD, SPT and ASLSA trackers take partal occluson nto account, the results are not satsfed. The Struck, RET and ONNDL trackers get slghtly better results. In comparsons, our tracker and SCM have relatvely lower center locaton errors and hgher success rates. The TLD tracker s equpped wth a detecton procedure to succeed n trackng after occlusons, whch can explan why the TLD tracker obtans relatvely hgh success rate but wth hgh center locaton error n ths sequence. In the Lquor sequence, the object suffers from background clutter besdes heavy occlusons for many tmes. The CT, MIL, LSST and ASLSA trackers drft frst when the occluson occurs (e.g., 361). Although the TLD, RET, VTD, SPT and Struck trackers obtan slghtly better results than SCM and ONNDL trackers, they lose the object after several occlusons (e.g., 733). Overall, our method acheves both the lowest trackng error and the hghest overlap rate. The ASLSA and LSST methods are generatve models that do not take nto account the useful nformaton from the background, and they are not effectve n separatng two nearby objects wth smlar appearance. Though the SCM tracker ncorporates the dscrmnatve model, ts classfer does not update onlne, makng

11 1520 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 24, NO. 5, MAY 2015 Fg. 8. Qualtatve trackng results of the 12 trackers over sequences Basketball, Freeman4 and Shakng from top to bottom (best vewed on hgh-resoluton dsplay). Object appearance changes drastcally due to large varatons of pose. Fg. 9. Qualtatve trackng results of the 12 trackers over sequences Woman, Lquor and SUV from top to bottom (best vewed on hgh-resoluton dsplay). Objects undergo heavy occluson. Fg. 10. Qualtatve trackng results of the eleven trackers over sequences Trells, Snger1 and Snger2 from top to bottom (best vewed on hgh-resoluton dsplay). Objects undergo llumnaton changes. t unable to adaptvely capture the dfference between the object and the background over tme. Although the localzed Haar-lke features used n the MIL and TLD trackers are robust to partal occluson [11], they cannot perform well n ths sequence because of the large scale appearance changes caused by frequent occlusons and background clutter. Our tracker performs well as t assgns the sample labels both n a supervsed and unsupervsed way durng the classfer learnng whch makes the updated classfer better dfferentate the object from the cluttered background. 3) Illumnaton Changes: Fg. 10 shows trackng results of three challengng sequences to evaluate whether our method s able to tackle drastc llumnaton changes. In Trells sequence, a man walks under a trells. Sufferng from large changes n envronmental llumnaton and head pose, the CT, TLD, MIL, SPT and LSST trackers drft gradually (e.g., 214). In contrast, RET, ASLSA, SCM, Struck and our trackers have relatvely hgh overlap rates. Note that the ASLSA get the best results, whch s attrbuted to the effcent algnment poolng on the sparse codng of local mage patches. For the Snger1

12 WU et al.: ROBUST DISCRIMINATIVE TRACKING VIA LLP 1521 Fg. 11. Qualtatve trackng results of the eleven trackers over sequences Tger1, Boy, Couple and FaccOcc2 from top to bottom (best vewed on hgh-resoluton dsplay). The challenges nclude camera jtter, fast moton, n-plane rotaton, occluson and background clutter, etc. sequence, there are large scale changes of the object and unknown camera moton n addton to llumnaton change. The SPT tracker gets lost n trackng the object after drastc llumnaton changes (e.g., 121) whereas ONNDL, LSST and RET algorthms perform slghtly better. The CT, Struck and MIL trackers perform reasonably well n terms of the center locaton error but wth lower overlap rate, because they can not deal wth scale changes well (e.g., 207, 279 and 321). In the Snger2 sequence, the contrast between the foreground and the background s very low besdes llumnaton change. Most trackers drft away at the begnnng of the sequence when the stage lght changes drastcally (e.g., 59). The VTD tracker performs slghtly better as the edge feature s less senstve to llumnaton change. In contrast, our method succeeds n trackng the object accurately. Overall, the SCM, ASLSA and our trackers obtans the relatvely robust trackng results n the presence of llumnaton changes. The reason that these three methods perform well can be explaned as follows. In SCM and ASLSA trackers, part-based sparse representatons wth poolng strategy are less senstve to llumnaton and pose change, thereby achevng good trackng performance. Our tracker uses an onlne update mechansm to account for the appearance varatons of the object and background over tme. More mportantly, wth the graph representaton, our tracker provdes a promsng paradgm for modelng the manfold structures of samples, whch makes the classfer have more dscrmnatng power. Therefore, our tracker s more adaptve to handle appearance changes. 4) Other Challenges: Fg. 11 presents the trackng results where the objects suffer other challenges ncludng moton blur, rotaton and scale, etc. In the Tger1 sequences, the appearances of the object change sgnfcantly as a result of scale, pose varaton, llumnaton change and moton blur at the same tme. The LSST and ASLSA trackers drft to the background at the begnnng of ths sequence (e.g., 39). The ONNDL, TLD, MIL, VTD and SCM fal gradually when the object frequently undergoes occluson and pose changes (e.g., 180, 233). In comparsons, the CT, Struck, RET, SPT and our methods track the object well untl the end of ths sequence. In the Struck, RET, SPT and our trackers, the dscrmnatve appearance models are updated n an onlne manner, whch take nto account the dfference between the foreground and the background over tme and thereby allevatng the drft problem. Note that the CT tracker gets the best results as t effectvely selects the most dscrmnatve random features for updatng the classfer, thereby better handlng drastc appearance change n ths sequence. In the Boy sequences, a boy jumps rregularly where the object undergoes fast moton and out-of-plane. It s dffcult to predct ther locatons. Most methods acheve relatvely lower center locaton errors and hgher success rates except CT, SCM and ASLSA trackers. As demonstrated n Fg. 6, SCM and ASLSA trackers do not perform well n ths sequence as the drastc appearance changes cause by fast moton and/or moton blur, are not effectvely accounted for the sparse representaton. The object n the Couple sequence s dffcult to track as t moves through the scene wth camera jtter and partal occluson. The TLD, Struck, MIL and our trackers perform well wth hgher success rates and lower locaton errors. Whle the ONNDL, RET and SPT methods perform better than the CT, SCM, ASLSA, VTD and LSST trackers, they all lose the object when occluson occurs (e.g., 112). In the sequence FaceOcc2, the object undergoes n-plane rotaton and frequent occlusons. The MIL, RET and TLD trackers fal after the object suffers from the partal occluson (e.g., 425). Struck, ASLSA, LSST and ONNDL are slghtly better but gradually drfts after frequent occluson (e.g., 600). Though CT, VTD, and SPT trackers are able to keep track of the object to the end, SCM and our methods acheve both the lowest trackng error and the hghest overlap rate. G. Computatonal Complexty The most tme consumng part of our trackng algorthm s the computaton of the label predcton functon f.

13 1522 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 24, NO. 5, MAY 2015 Specfcally, the tme complexty of seekng m landmarks usng k-means clusterng s O(mn) where n s the number of samples. The tme complexty of solvng the optmal H and the predcton matrx A s O(smn) and O(m 3 + m 2 n), respectvely, where s s the number of nearest landmarks of each sample. We use a fxed number m n of landmarks for calculatng f, whch s ndependent of the sample sze n. Thus, the total tme complexty s O(m 2 n) whch scales lnearly wth the n. The proposed approach was mplemented n MATLAB on a Intel Core2 2.5 GHz processor wth 4GB RAM. Our tracker s about 1.5 frame/sec for all experments. No code optmzaton s performed. VI. CONCLUSION In ths paper, we have proposed the landmark-based label propagaton for vsual trackng, n whch the label of each sample can be nterpreted as the weghted combnaton of labels on landmarks. Through solvng the cross-smlarty matrx H and the label predcton matrx A, our model s able to effectvely propagate a lmted number of landmarks labels to all the unlabeled canddates, matchng the needs of the dscrmnatve tracker. Explctly consderng the local geometrcal structure of the samples, the graph-based regularzer s ncorporated nto the LLP tracker, whch makes our method have better dscrmnatng power and thus s more adaptve to handle appearance changes. Comparson wth 11 state-of-the-art trackng methods on the benchmark dataset have demonstrated that the LLP tracker s more robust to llumnaton changes, pose varatons and partal occlusons, etc. ACKNOWLEDGEMENTS The authors would lke to thank both the edtor and the revewers for the nvaluable comments and suggestons that help a great deal n mprovng the manuscrpt. The authors would lke to thank Yang He for hs efforts on conductng experments. REFERENCES [1] S. Salt, A. Cavallaro, and L. D Stefano, Adaptve appearance modelng for vdeo trackng: Survey and evaluaton, IEEE Trans. Image Process., vol. 21, no. 10, pp , Oct [2] X. L, W. Hu, C. Shen, Z. Zhang, A. Dck, and A. van den Hengel, A survey of appearance models n vsual object trackng, ACM Trans. Intell. Syst. Technol., vol. 4, no. 4, 2013, Art. ID 58. [3] X. Ja, H. Lu, and M.-H. Yang, Vsual trackng va adaptve structural local sparse appearance model, n Proc. IEEE Conf. Comput. Vs. Pattern Recognt., Jun. 2012, pp [4] D. Wang, H. Lu, and M.-H. Yang, Least soft-threshold squares trackng, n Proc. IEEE Conf. Comput. Vs. Pattern Recognt., Jun. 2013, pp [5] J. Kwon and K. M. Lee, Vsual trackng decomposton, n Proc. IEEE Conf. Comput. Vs. Pattern Recognt., Jun. 2010, pp [6] X. Me and H. Lng, Robust vsual trackng usng l1 mnmzaton, n Proc. IEEE 12th Int. Conf. Comput. Vs., Sep./Oct. 2009, pp [7] D. A. Ross, J. Lm, R.-S. Ln, and M.-H. Yang, Incremental learnng for robust vsual trackng, Int. J. Comput. Vs., vol. 77, nos. 1 3, pp , [8] B. Zhuang, H. Lu, Z. Xao, and D. Wang, Vsual trackng va dscrmnatve sparse smlarty map, IEEE Trans. Image Process., vol. 23, no. 4, pp , Apr [9] H. Grabner, C. Lestner, and H. Bschof, Sem-supervsed on-lne boostng for robust trackng, n Proc. 10th Eur. Conf. Comput. Vs., 2008, pp [10] M. Yang, J. Yuan, and Y. Wu, Spatal selecton for attentonal vsual trackng, n Proc. IEEE Conf. Comput. Vs. Pattern Recognt. (CVPR), Jun. 2007, pp [11] B. Babenko, M.-H. Yang, and S. Belonge, Robust object trackng wth onlne multple nstance learnng, IEEE Trans. Pattern Anal. Mach. Intell., vol. 33, no. 8, pp , Aug [12] S. Hare, A. Saffar, and P. H. S. Torr, Struck: Structured output trackng wth kernels, n Proc. IEEE Int. Conf. Comput. Vs., Nov. 2011, pp [13] Z. Kalal, K. Mkolajczyk, and J. Matas, Trackng-learnng-detecton, IEEE Trans. Pattern Anal. Mach. Intell., vol. 34, no. 7, pp , Jul [14] R. Yao, Q. Sh, C. Shen, Y. Zhang, and A. van den Hengel, Part-based vsual trackng wth onlne latent structural learnng, n Proc. IEEE Conf. Comput. Vs. Pattern Recognt., Jun. 2013, pp [15] Q. Yu, T. B. Dnh, and G. Medon, Onlne trackng and reacquston usng co-traned generatve and dscrmnatve trackers, n Computer Vson. Berln, Germany: Sprnger-Verlag, 2008, pp [16] S. Avdan, Ensemble trackng, IEEE Trans. Pattern Anal. Mach. Intell., vol. 29, no. 2, pp , Feb [17] R. T. Collns, Y. Lu, and M. Leordeanu, Onlne selecton of dscrmnatve trackng features, IEEE Trans. Pattern Anal. Mach. Intell., vol. 27, no. 10, pp , Oct [18] H. Grabner, M. Grabner, and H. Bschof, Real-tme trackng va on-lne boostng, n Proc. Brt. Mach. Vs. Conf., 2006, vol. 1, no. 5, p. 6. [19] L. Zhang and L. J. P. van der Maaten, Preservng structure n modelfree trackng, IEEE Trans. Pattern Anal. Mach. Intell., vol. 36, no. 4, pp , Apr [20] Y. Wu, B. Ma, M. Yang, Y. Ja, and J. Zhang, Metrc learnng based structural appearance model for robust vsual trackng, IEEE Trans. Crcuts Syst. Vdeo Technol., vol. 24, no. 5, pp , May [21] N. Jang, W. Lu, and Y. Wu, Learnng adaptve metrc for robust vsual trackng, IEEE Trans. Image Process., vol. 20, no. 8, pp , Aug [22] B. Zesl, C. Lestner, A. Saffar, and H. Bschof, On-lne semsupervsed multple-nstance boostng, n Proc. IEEE Conf. Comput. Vs. Pattern Recognt., Jun. 2010, p [23] Y. Zha, Y. Yang, and D. B, Graph-based transductve learnng for robust vsual trackng, Pattern Recognt., vol. 43, no. 1, pp , [24] J. Gao, J. Xng, W. Hu, and S. Maybank, Dscrmnant trackng usng tensor representaton wth sem-supervsed mprovement, n Proc. IEEE Int. Conf. Comput. Vs., Dec. 2013, pp [25] X. L, C. Shen, A. Dck, and A. van den Hengel, Learnng compact bnary codes for vsual trackng, n Proc. IEEE Conf. Comput. Vs. Pattern Recognt., Jun. 2013, pp [26] K. C. A. Kumar and C. De Vleeschouwer, Dscrmnatve label propagaton for mult-object trackng wth sporadc appearance features, n Proc. IEEE Int. Conf. Comput. Vs., Dec. 2013, pp [27] K. Zhang, J. T. Kwok, and B. Parvn, Prototype vector machne for large scale sem-supervsed learnng, n Proc. 26th Annu. Int. Conf. Mach. Learn., 2009, pp [28] W. Lu, J. Wang, and S.-F. Chang, Robust and scalable graph-based semsupervsed learnng, Proc. IEEE, vol. 100, no. 9, pp , Sep [29] X. Zhu and Z. Ghahraman, Learnng from labeled and unlabeled data wth label propagaton, School Comput. Sc., Carnege Mellon Unv., Pttsburgh, PA, USA, Tech. Rep. CMU-CALD , [30] T. Haste, R. Tbshran, and J. Fredman, The Elements of Statstcal Learnng: Data Mnng, Inference, and Predcton. New York, NY, USA: Sprnger-Verlag, [31] J. Wang, J. Yang, K. Yu, F. Lv, T. Huang, and Y. Gong, Localtyconstraned lnear codng for mage classfcaton, n Proc. IEEE Conf. Comput. Vs. Pattern Recognt., Jun. 2010, pp [32] K. Yu, T. Zhang, and Y. Gong, Nonlnear learnng usng local coordnate codng, n Advances n Neural Informaton Processng Systems. Red Hook, NY, USA: Curran & Assocates Inc., 2009, pp [33] Y. Nesterov, Introductory Lectures on Convex Optmzaton: A Basc Course, vol. 87. New York, NY, USA: Sprnger-Verlag, [34] J. Duch, S. Shalev-Shwartz, Y. Snger, and T. Chandra, Effcent projectons onto the l 1 -ball for learnng n hgh dmensons, n Proc. 25th Int. Conf. Mach. Learn., 2008, pp

14 WU et al.: ROBUST DISCRIMINATIVE TRACKING VIA LLP 1523 [35] Y. Nesterov, A method of solvng a convex programmng problem wth convergence rate O(1/k 2 ), Sovet Math. Doklady, vol. 27, no. 2, pp , [36] C. K. I. Wllams and M. Seeger, Usng the Nyström method to speed up kernel machnes, n Advances n Neural Informaton Processng Systems. Cambrdge, MA, USA: MIT Press, [37] Y. Wu, J. Lm, and M.-H. Yang, Onlne object trackng: A benchmark, n Proc. IEEE Conf. Comput. Vs. Pattern Recognt., Jun. 2013, pp [38] N. Wang, J. Wang, and D.-Y. Yeung, Onlne robust non-negatve dctonary learnng for vsual trackng, n Proc. IEEE Int. Conf. Comput. Vs., Dec. 2013, pp [39] Q. Ba, Z. Wu, S. Sclaroff, M. Betke, and C. Monner, Randomzed ensemble trackng, n Proc. IEEE Int. Conf. Comput. Vs., Dec. 2013, pp [40] K. Zhang, L. Zhang, and M.-H. Yang, Real-tme compressve trackng, n Proc. 12th Eur. Conf. Comput. Vs., 2012, pp [41] W. Zhong, H. Lu, and M. Yang, Robust object trackng va sparstybased collaboratve model, IEEE Trans. Image Process., vol. 23, no. 5, pp , May [42] J. F. Henrques, R. Casero, P. Martns, and J. Batsta, Explotng the crculant structure of trackng-by-detecton wth kernels, n Proc. 12th Eur. Conf. Comput. Vs., 2012, pp [43] T. Nawaz and A. Cavallaro, A protocol for evaluatng vdeo trackers under real-world condtons, IEEE Trans. Image Process., vol. 22, no. 4, pp , Apr Yuwe Wu receved the Ph.D. degree n computer scence from the Bejng Insttute of Technology (BIT), Bejng, Chna, n He s currently a Research Fellow wth the Rapd-Rch Object Search Laboratory, School of Electrcal and Electronc Engneerng, Nanyang Technologcal Unversty, Sngapore. He has strong research nterests n computer vson, medcal mage processng, and object trackng. He receved the Natonal Scholarshp for Graduate Students and the Academc Scholarshp for the Ph.D. Canddates from the Mnstry of Educaton n Chna, the Outstandng Ph.D. Thess Award and the XU TELI Excellent Scholarshp from BIT, and the CASC Scholarshp from the Chna Aerospace Scence and Industry Corporaton. Mngtao Pe receved the Ph.D. degree n computer scence from the Bejng Insttute of Technology (BIT), n He served as an Assocate Professor wth the School of Computer Scence, BIT. He was a Vstng Scholar wth the Center of Image and Vson Scence, Unversty of Calforna at Los Angeles, from 2009 to Hs man research nterest s computer vson wth an emphass on event recognton and machne learnng. He s a member of the Chna Computer Federaton. Mn Yang receved the B.S. degree from the Bejng Insttute of Technology (BIT), n He s currently pursung the Ph.D. degree wth the Bejng Laboratory of Intellgent Informaton Technology, School of Computer Scence, BIT, under the supervson of Prof. Y. Ja. Hs research nterests nclude computer vson, pattern recognton, and machne learnng. Junsong Yuan (M 08) receved the Ph.D. degree from Northwestern Unversty, USA, and the M.Eng. degree from the Natonal Unversty of Sngapore. Before that, he graduated from Specal Class for the Gfted Young of Huazhong Unversty of Scence and Technology, Chna. He s currently a Nanyang Assstant Professor and the Program Drector of Vdeo Analytcs wth the School of Electrcal and Electronc Engneerng, Nanyang Technologcal Unversty, Sngapore. He has authored over 100 techncal papers, and hold three U.S. patents and two provsonal U.S. patents. Hs research nterests nclude computer vson, vdeo analytcs, large-scale vsual search and mnng, and human computer nteracton. He receved the Nanyang Assstant Professorshp and the Tan Chn Tuan Exchange Fellowshp from Nanyang Technologcal Unversty, the Outstandng EECS Ph.D. Thess Award from Northwestern Unversty, the Best Doctoral Spotlght Award from the IEEE Conference on Computer Vson and Pattern Recognton (CVPR 2009), and the Natonal Outstandng Student Award from the Mnstry of Educaton, Chna. He served as an Area Char of the IEEE Wnter Conference on Computer Vson n 2014, the IEEE Conference on Multmeda Expo (ICME 2014), and the Asan Conference on Computer Vson (ACCV 2014), the Organzng Char of ACCV 2014, and the Co-Char of workshops at CVPR 2012 and 2013, and the IEEE Conference on Computer Vson n He serves as an Assocate Edtor of The Vsual Computer journal, and the Journal of Multmeda. He recently gves tutorals at the IEEE ICIP13, FG13, ICME12, SIGGRAPH VRCAI12, and PCM12. Yunde Ja (M 11) receved the B.S., M.S., and Ph.D. degrees n mechatroncs from the Bejng Insttute of Technology (BIT), Chna, n 1983, 1986, and 2000, respectvely. He s currently a Professor of Computer Scence wth BIT, where he currently serves as the Drector of the Bejng Laboratory of Intellgent Informaton Technology. He was a Vstng Scentst wth Carnege Mellon Unversty, USA, from 1995 to 1997, and a Vstng Fellow wth Australan Natonal Unversty, Australa, n He served as the Executve Dean of the School of Computer Scence wth BIT from 2005 to Hs current research nterests nclude computer vson, meda computng, and ntellgent systems.