Fast Global Kernel Density Mode Seeking with application to Localisation and Tracking

Transcription

1 Fast Global Kernel Density Mode Seeking wit application to Localisation and Tracking Cunua Sen, Micael J. Brooks, Anton van den Hengel Scool of Computer Science, University of Adelaide, SA 55, Australia Abstract We address te problem of seeking te global mode of a density function using te mean sift algoritm. Mean sift, like oter gradient ascent optimisation metods, is susceptible to local maxima, and ence often fails to find te desired global maximum. In tis work, we propose a multi-bandwidt mean sift procedure tat alleviates tis problem, wic we term annealed mean sift, as it sares similarities wit te annealed importance sampling procedure. Te bandwidt of te algoritm plays te same role as te temperature in annealing. We observe tat te over-smooted density function wit a sufficiently large bandwidt is uni-modal. Using a continuation principle, te influence of te global peak in te density function is introduced gradually. In tis way te global maximum is more reliably located. Generally, te price of tis annealing-like procedure is tat more iterations are required. Since it is imperative tat te computation complexity is minimal in real-time applications suc as visual tracking. We propose an accelerated version of te mean sift algoritm. Compared wit te conventional mean sift algoritm, te accelerated mean sift can significantly decrease te number of iterations required for convergence. Te proposed algoritm is applied to te problems of visual tracking and object localisation. We empirically sow on various data sets tat te proposed algoritm can reliably find te true object location wen te starting position of mean sift is far away from te global maximum, in contrast wit te conventional mean sift algoritm tat will usually get trapped in a spurious local maximum. 1. Introduction & Motivation Kernel-based density estimation tecniques for computer vision ave recently attracted a great deal of attention. One example is te mean sift tecnique wic as been applied to image segmentation and visual tracking [1 6], etc. Mean sift is a versatile nonparametric density analysis tool introduced in [7 9]. In essence, it is an iterative mode detection algoritm in te density distribution space. Te mean sift algoritm uses kernels to compute te weigted average of te observations witin a smooting window. Tis computation is repeated until convergence is attained at a local density mode. Tis way te density modes can be elegantly located witout explicitly estimating te density. Ceng [8] notes tat mean sift is fundamentally a gradient ascent algoritm wit an adaptive step size. Recently Fasing et al. sow te connection between mean sift and te Newton-Rapson optimisation algoritm [1]. Tey also discover tat mean sift is actually a quadratic bound optimisation bot for stationary and evolving sample sets [1]. Mean sift is also a fixed-point iteration procedure. Since Comaniciu et al. first introduced mean sift based object tracking [], it as proven to be a promising alternative to popular particle filtering based trackers. Incremental researc as been reported in te literature. In [3] te selection of kernel scale via linear searc is discussed. Elgammal et al. reformulate te tracking framework as a general form of joint feature-spatial distributions [, 6]. Compared wit te approac of Comaniciu et al. [], te advantage is tat spatial structure information of te tracked region is incorporated into te measure. In [5] multiple spatially distributed kernels are adopted to accurately capture canges in te target s orientation and scale. Anoter approac is developed in [11] for te same purpose. Furtermore Fan et al. present a teoretical analysis of similarity measure and arrive at a criterion, leading to kernel design strategies wit prevention of singularity in kernel visual tracking [1]. All of tese trackers adopt mean sift or similar optimisation strategies to acieve tracking. Despite successful applications, mean sift trackers require tat te displacement of te tracked target in consecutive frames is small. If tis is not te case, tey are likely to become trapped in spurious local maxima of te multi-modal density distribution space 1. Tis appens because mean sift is a purely local optimisation metod. Fundamentally, mean sift as two important inerent drawbacks. First of all, it can only be used to find local modes. Being trapped in a local maximum/minimum is a common problem for traditional nonlinear optimisation algoritms. Simulated annealing is a well-known strategy wic aims to acieve global optimisation. It starts by initially sampling wit a reduced sensitivity to te underly- 1 In contrast, particle filtering based trackers (e.g. [13]) perform better in tis situation. However weak dynamical modelling also presents callenges to particle filters.

2 ing modes (on a flattened cost function surface) and ten progressively increasing te sensitivity to drive samples towards peaked cost regions [1]. Recently te idea of annealing as been merged into importance sampling, yielding annealed importance sampling [15] and it as been introduced to 3D articulated tracking [16]. Motivated by te success of bot simulated annealing and annealing importance sampling, we propose a novel multi-bandwidt mean sift procedure, termed annealed mean sift (ANNEALEDMS). It sares similarities wit te annealed importance sampling procedure in te sense tat it also gradually smoots te cost function surface and gently introduces te influence of te global peak. We observe tat te over-smooted density function wit a sufficiently large bandwidt M is uni-modal. Ten, wit a continuation principle, we slowly decrease te bandwidt = M > M 1 > > and, at eac bandwidt, we maximise te density (cost) function wit mean sift, starting from te convergence position of te previous run. Tis multi-bandwidt mean sift iteration process is similar to te multi-layered annealing procedure of annealed importance sampling. Te main differences are: (1) In AN- NEALEDMS, it is te degree of smootness of te cost function tat is annealed, wile in annealed importance sampling, it is te degree of flatness of te cost function; () Most importantly, in ANNEALEDMS, te number and positions of te modes evolve slowly wile in te annealed importance sampling, te temperature does not cange te number of modes or teir positions. In teory, as long as te cange of te bandwidt is sufficiently slow, te global maximum can usually be found successfully. 3 We provide tecnical details later. A second drawback of mean sift is tat in many cases it converges slowly. Te proposed ANNEALEDMS involves even more iterations, and especially wen it is applied to localisation, in wic case we ave no knowledge of were to start searcing. It is imperative tat te computational complexity is minimal in real-time applications suc as visual tracking. To our knowledge, few attempts ave been made to speed up te convergence to mean sift. In [1] locality sensitive asing is used to reduce te computational complexity of finding te nearest neigbours of a sample point involved in mean sift. Altoug a dramatic decrease in te execution time is acieved for ig-dimensional clustering, tis tecnique is not tat attractive for relatively lowdimensional problems suc as visual tracking. Te speedup of [1] is not obtained by reducing te iteration steps. In tis paper, we advance an accelerated version of te mean sift algoritm. Compared wit te conventional mean sift By a sufficiently large bandwidt, we mean a bandwidt wic is muc larger tan te optimal bandwidt wit te minimum asymptotic mean integrated square error (AMISE). 3 For continuous variables, te assertion of success is probabilistic. algoritm, it can significantly decrease te number of iterations to convergence. Te accelerated mean sift is inspired by te successful accelerated variants of bound optimisation algoritms suc as Expectation Maximisation (EM). An over-relaxed strategy is ten adopted to accelerate te convergence. Muc effort as been expended to improve te efficiency of bound optimisation algoritms (e.g. EM, [17 19]). A teoretical analysis of te convergence properties for a class of bound optimisation algoritms as been given in [19], and is used as te basis for a novel adaptive over-relaxed sceme. Our proposal is inspired by tis approac. Based on te findings in [1], wic bridge te gap between mean sift and general bound optimisation algoritms, we promote an adaptive over-relaxed mean sift algoritm wic is simple to implement yet significantly more efficient tan te standard counterpart. As applications of te proposed fast, globally modeseeking mean sift, a fast ANNEALEDMS based object localiser and a visual tracker are developed. Substantially more promising results ave been acieved over te conventional mean sift based algoritms. In summary, our key contributions comprise: 1. Development of a novel annealed mean sift algoritm wic can reliably find te global mode of a density distribution. Tis is introduced in Section 3.. Reinterpretation of te mean sift algoritm, resulting in a faster version of mean sift. We discuss tese issues in Section. 3. Application of ANNEALEDMS to te problem of visual tracking using kernel-weigted colour istogram features. Given a target model, te tracker is able to initialise automatically. It also as te capability to recover from tracking failures caused by occlusions, drastic illumination canges, etc., in tat te tracker itself can also be a localiser. In contrast, conventional mean sift trackers lack tese desirable properties. Tese developments, including experimental results, are presented in Sections 5.1 and 5.. A brief review of te standard mean sift algoritm is presented in Section. We conclude te paper in Section 6 wit a discussion of some important issues.. Mean Sift Analysis We first review te basic concepts of te mean sift algoritm wit notation similar to [9]. One of te most popular nonparametric density estimators is kernel density estimation. Given n data points x i, i =1,,n, drawn from a population wit density function f(x), x IR d, te general multivariate kernel density estimate at x is defined by ˆf K (x) = 1 n n K H (x x i ), (1) i=1

3 Density estimate (arbitrary units) Gaussian kernel, bandwidt:38,3,6,,13,8, x Density estimate (arbitrary units) Gaussian kernel, bandwidt:5 ( ) x Figure 1: Multi-bandwidt density estimate on 1D galaxy velocity data. (left) Curves from outside to inside indicate te annealing process wit successively decreasing bandwidts. In tis case, te optimal bandwidt is = 5. Te evolution of te modes is clearly sown: wit a multibandwidt mean sift mode detection, it is possible to find te global maximum witout being distracted by local modes. (rigt) Convergence positions at eac bandwidt are marked wit circles in te last curve. Note tat te unit of te vertical axis is arbitrary. were K H (x) = H 1 K(H 1 x). Here K( ) is a kernel function (or window) wit a symmetric positive definite bandwidt matrix H IR d d. A kernel function is bounded wit support satisfying te regularity constraints as described in [8, 9]. For simplicity one usually assumes an isotropic bandwidt wic is proportional to te identity matrix, i.e. H = I. Employing te profile definition, te kernel density estimator becomes ˆf K (x) = c k n d ( n x x i ) k, () i=1 were k( ) is te profile of te kernel K( ) and c k is a normalisation constant. Te optimisation problem of seeking te local modes is solved by setting te gradient equal to zero. Tus we ave were ˆ f K (x) = def ˆf K (x) = c k ˆfG (x) m G (x) =, (3) c g ˆf G (x) = c g n d ( n x x i ) g, () i=1 ( n i=1 x x x ig i ) m G (x) = ( n i=1 g x x i ) x, (5) and g(x) = k (x). Herek( ) is defined to be te sadow of te profile g( ) [1], and m G (x) is te mean sift vector. Clearly ˆ f K (x) = m G (x) =, and te incremental iteration sceme is obtained immediately: ( n i=1 x x x ig i ) x ( n i=1 g x x i ). (6) 1. Determine te set of values for m, (m = M ) (a.k.a. te annealing scedule).. Randomly select an initial starting location for te first annealing run and get te convergence location of ˆf M,K( ), wic is x (M),usingmean sift. 3. for eac m = M 1,M,,, runmean sift to get te convergence position x (m) wit te initial position x (m+1), i.e., te convergence position from te previous bandwidt. x () is ten te final global mode. Figure : Te ANNEALEDMS algoritm. 3. Annealed Mean Sift Let m (m = M,M 1,, ) be a monotonically decreasing sequence of bandwidts suc tat is te optimal bandwidt for te considered data set and usually M. A series of kernel density functions ˆf M,K( ), ˆf M 1,K( ),, ˆf,K( ) are applied to te sample data, were te subscripts of ˆf,K ( ) denote te bandwidt and kernel type respectively. Figure 1 illustrates a 1D example 5, were M =6.Wit a large bandwidt, te function ˆf M,K( ) is uni-modal, merely representing te overall trend of te density function. Tus te starting point of te first annealing run does not affect te mode detection. Te ANNEALEDMS algoritm is given in Figure Remarks 1. Apparently te annealing scedule is a trade off between efficiency and efficacy: slow annealing is more likely to find a global maximum, but could also be proibitively expensive.. A justification of wy ANNEALEDMS works is tat te number of modes of a kernel density estimator wit a Gaussian kernel is monotonically non-increasing []. In order to convey convergence information, te monotonicity of number of modes wit respect to bandwidts is compulsory. Note tat te monotonicity result only applies to te Gaussian kernel: compactly supported kernels suc as te Epanecnikov kernel may not ave tis property. However, as pointed out in [], te lack of monotonicity appens only for relatively very small bandwidts. Te notion of a critical bandwidt 6 for te popular kernels suc as te Tere is a tremendous amount of literature on ow to select te optimal bandwidt in order to produce a minimum AMISE estimate (see, e.g., [, 1].) In tis work, we assume can be obtained by existing tecniques. 5 Te 1D galaxy velocity data set is also used in [1]. 6 Te smallest bandwidt above wic te number of modes is monotone.

4 Epanecnikov kernel is still well defined. Moreover, just as in te Gaussian case, te critical bandwidt is of te same size as te bandwidt ( ) tat minimises mean square error of te density estimator []. Tis conclusion serves as one of te teoretical bases of our ANNEALEDMS: we are not interested in te bandwidt under. Rater, we only take advantage of te property of over-smootness at bandwidts above. Terefore, for te applications we are interested in, e.g., visual localisation and tracking, te problem of nonmonotonicity does not arise. We ave also empirically sown tis important proposition. 3. Unless oterwise specified, in our examples te (truncated) Gaussian kernel is used because it is well suited to fast computation [6, 3]. Wen Gaussian kernel is adopted, te mecanism is related wit te well-developed multi-scale teory [, 5]. Te oversmooted kernel density is essentially a Gaussian smooted version of te true density, via convolution wit extra Gaussians. We note tat, in te statistics literature, [] as proposed an algoritm SiZer to explore te significant modes in an estimated curve across multiple scales. SiZer employs a similar principle. In computer vision, a similar strategy, variable-bandwidt density-based fusion (VBDF), as also been adopted to find te most significant mode of a density function in te context of information fusion for multiple motion estimation [6]. However, tere are no teoretical details given in [6]. We independently develop AN- NEALEDMS mainly inspired by simulated annealing and annealed importance sampling. We ave sown a connection between ANNEALEDMS and tese annealing tecniques. Furtermore, we use it in a novel way to solve some problems in robust visual localisation and tracking. 3.. Numerical Examples 1D example. Figure 1 sows a simple 1D example on te galaxy data [1]. Because of te density estimator s uni-modal property at a large bandwidt ( M ), te start position at M as no effect on te final convergence. Figure 1 sows tat te global maximum is successfully located wit a roug seven-step annealing scedule. For tis particular case, actually only two steps are needed to locate te global mode. D example. For tis example, te data are drawn from a Gaussian mixture.1 N ( [ 1, ],.13I ) +. N ( [1, ], I ) +.7 N ( [1, ], I ). A four-step AN- NEALEDMS wit bandwidts {, 1.,.66,.5} is used to locate te global mode. Figure 3 depicts te annealing process. Again due to te uni-modal property, no matter from wic initial position ANNEALEDMS starts, te global mode is always obtained eventually. A video se- Density estimate Density estimate 6 6 x 1 x 1 x 1 Gaussian kernel, bandwidt: x Gaussian kernel, bandwidt: x 1 x Density estimate 6 6 x 1 x 1 Gaussian kernel, bandwidt:1.18 x 1 x Gaussian kernel, bandwidt:.5 Figure 3: Multi-bandwidt density estimate on D artificial Gaussian mixture data. A four-step annealing scedule is employed to find te global mode. Te modes found by mean sift across bandwidts are marked wit circles. See te sequence GMMD.avi wic demonstrates a slower evolution across bandwidts. quence (GMMD.avi) 7 is also generated to sow te mode evolution process more elaborately. For tese two examples, we do not assume any prior information about te distribution structure of te data. Te only information needed is te approximate range of te data, wic is usually available.. Fast Mean Sift Generally, te price of global convergence of AN- NEALEDMS is tat more iterations are required. Tis is te case particularly wen te start point is far away from te convergence position. It is imperative tat te computational complexity is minimal in real-time applications suc as visual tracking. We introduce an adaptive over-relaxed accelerated mean sift in tis section..1. Adaptive Over-Relaxed Mean Sift Te following two teorems serve te bases of te adaptive over-relaxed mean sift algoritm. Teorem.1 (Ceng [8]): Mean sift wit kernel G( ) finds te modes of te density estimate wit kernel K( ), i.e. ˆfK ( ), were K( ) is te sadow of te kernel G( ). Density estimate Wit te analysis in Section, Teorem.1 is evident. Teorem. (Fasing et al. [1]): Mean sift wit kernel K( ) is a quadratic bound optimisation over a density estimate wit a continuous sadow of K( ). 7 Te videos mentioned in tis paper can be accessed at ttp:// adelaide.edu.au/ vision/demo/index.tml. x 1 x

5 Tese two teorems sow tat mean sift is actually a bound maximisation. One step of te mean sift procedure of Equation (6) finds te exact maximum of te lower bound of te objective function ˆf K (x (κ) ). 8 From Equation (3) we ave m G (x) ˆ f K(x), wic means mean sift is a gradient ascent algoritm wit adaptive step size. Hence its con- ˆf G(x) vergence rate is better tan conventional fixed-step gradient algoritms. As we will see, owever, from te viewpoint of bound optimisation, te learning rate can be over-relaxed to make its convergence faster. It is well known tat, in order to guarantee increasing te cost function at eac iteration, bound optimisation metods must usually build conservative bounds, leading to slow convergence [17, 19]. A lot of work as been carried out to speed up bound optimisation metods, especially for te EM algoritm due to its popularity [18]. In [19] it is sown tat by over-relaxing te step size, acceleration can be acieved. Denote te bound function as ρ(x, x (κ) ), ten te over-relaxed bound optimisation iteration becomes: [ x (κ+1) = x (κ) + β arg max ρ(x, x x(κ) ) x (κ)]. (7) Apparently wen te learning rate β =1, over-relaxed optimisation reduces to te standard bound optimisation algoritm. It is easily seen tat wen β > 1 acceleration is realised. Neverteless by simply assigning a fixed value to β, no convergence is secured and it seems quite difficult, if not impossible, to obtain te optimal value for β. Xu proves tat in te case of te Gaussian Mixture Model (GMM) parameter estimation wit EM, convergence can be guaranteed using tis metod wen we are close toalocal maximum and < β < [17]. Tis conclusion is generalised to te case of general bound optimisation metods in [19]. Based on tis important proposition, a simple adaptive over-relaxed bound optimisation is readily available: te learning rate β can be adjusted by evaluating te cost function. Wen one observes for some β>1tat te cost function becomes worse, ten β as been set too large and needs to be reduced. By simply setting β =1immediately, convergence can be acieved. By regarding mean sift as a special case of bound optimisation, tese teoretical conclusions also apply to mean sift. Te accelerated mean sift algoritm obtained in tis way is sown in Figure. One can easily ceck tat te following relation olds (up to a translation and a scale factor): ˆf K (x (κ+1) )=ρ(x (κ+1), x (κ+1) ) ρ(x (κ+1), x (κ) ) ρ(x (κ), x (κ) )= ˆf K (x (κ) ). Note tat in te above analysis we do not take te mean sift wit a weigt function into consideration, but te accelerated algoritm also applies for te weigted case, because 8 κ =1,,, denotes te iteration index. 1. Initialisation: Set te iteration index κ = 1, te learning rate β =1, and te step parameter α>1.. Iterate until convergence condition is met: (a) Calculate x (κ+1) wit Equation (6). Calculate te mean sift m G (x (κ+1) )= x (κ+1) x (κ). (b) x (κ+1) = x (κ) + β m G (x (κ+1) ). (c) if ˆf K (x (κ+1) ) > ˆf K (x (κ) ), Accept x (κ+1) and β = α β; else Reject x (κ+1), x (κ+1) = x (κ+1), and β =1. (d) Set κ = κ +1. Start a new iteration. Figure : Te over-relaxed adaptive mean sift algoritm. te two teorems concerned are derived from te weigted mean sift [8,1]. Te only overead is te evaluation of te cost function. However, we will see tat for te (truncated) Gaussian kernel, its special structure means tat computing te mean sift iteration wit Equation (6) also results in evaluation of te cost function ˆf K (x). Because te sadow of te Gaussian kernel is itself, we ave ˆf K (x) = ˆf G (x). A question naturally arises, wat if a kernel oter tan a Gaussian, e.g., Epanecnikov, is adopted? Te observation tat we can reliably judge te beaviour of ˆf K (x) troug te estimate ˆf G (x) is only satisfied wen tese two kernel functions generate density estimates of te same degree of smootness. For different kernels, as long as te bandwidts are adjusted accordingly, all te kernels are asymptotically equivalent under te AMISE error criterion. Terefore te kernel type is not of importance in mean sift analysis but te bandwidt plays a critical role. For a non-gaussian kernel, te sadow is different from itself ( ˆf K (x) ˆf G (x)). Te smootness of two kernel density estimates wit te same bandwidt but different kernels migt be quite different. As a consequence, usually we cannot reuse te density ˆf G (x) calculated in Equation (6) and an extra evaluation of te cost function ˆf K (x) needs to be made. In fact, if te bandwidts of two different kernels A, B satisfy A B = δ,a δ,b, were δ is a kernel s canonical bandwidt, ten te density estimates based on tese two kernels ave te same degree of smootness [7]. Utilising tis knowledge, in practice, if te canonical bandwidts associated wit a kernel and its sadow kernel are comparable, we still can reuse ˆf G (x). Altoug no details on tis topic are presented in tis paper, we ave validated tis conclusion wit numerical experiments. However, one sould be aware tat te measurement of comparable is application dependent.

6 .. Numerical Experiments We compare te performance of te proposed accelerated mean sift algoritm wit te standard mean sift algoritm on bot syntetic data and real application data sets. Note tat rejected iterations are also counted for te accelerated mean sift algoritm. Due to limited space, te detailed description of te data sets is omitted, wic can be found in [3]. In all te tests, we use α = 1.5 and te convergence tolerance ε = ˆf K(x κ+1 ) ˆf K(x κ ) =.1. Te resulting mode locations found by te two algoritms are so close tat te ˆf K(x κ ) difference is negligible. We run te comparison wit tree arbitrarily selected start points on eac data set. Te experiment results are reported in Table.. Te proposed algoritm is significantly more efficient tan te standard mean sift. Te evaluation results are promising: a speedup by a factor of about 5 can be acieved in tese evaluations. We ave also developed an accelerated mean sift tracker, wic outperforms te conventional mean sift tracker [3]. Te accelerated mean sift s performance wit fewer convergence iterations as proven commensurate wit its standard counterpart. In teory wen te start point is extremely close to te local maximum, te rejection in te proposed accelerated mean sift procedure migt appen frequently, resulting in a resource waste. In practice tese cases are very rare. Moreover one can devise smarter stepsize adjustment strategies to survive in tis extreme case. data set initial number of iterations fast mean sift mean sift data set # data set # ( 5, ) 1 3 data set #3 ( 1, 16) 11 9 (, 1) (1, 1.) data set # (1.5,.) (.3,.3) 1 36 Table 1: Comparison of number of iterations for convergence. Te initial location for eac run is sown in te second column. 5. Fast Annealed Mean Sift Based Visual Localisation and Tracking In tis section we apply te two improvements on mean sift to visual localisation and tracking. In all te localisation and tracking experiments we use RGB colour istograms, consisting of bins. Te tracking framework presented in [] is adopted, but we use an an- (5) (6) (1) () (3) () (6) (1) () (5) () (5) (3) Figure 5: We locate a specified uman face (left top) beside two spurious faces, te STARBUCKS logo (rigt top), a CD (left bottom) and a book cover (rigt bottom) under cluttered backgrounds. Te ANNEALEDMS is started at arbitrarily selected positions. Dased lines indicate te mean sift searcing trajectories for eac run. Dots indicate te start and convergence positions of mean sift for eac bandwidt. See te accompanying videos localiser{1,, 3, }.avi for an intuitive demonstration on te annealing convergence processes. (6) (6) (1) nealing procedure for global mode seeking Visual Localisation Up to date, mean sift as typically been used for tracking motions wit small displacements due to its lack of global mode seeking capability. Armed wit AN- NEALEDMS, it is possible to locate a target no matter from wic initial position te mean sift localiser starts, given te target template. In our experiments, te ANNEALEDMS localiser starts at arbitrarily selected positions. All result on successful location of te target. Six runs for eac example are marked in Figure 5. Four objects are located successfully in different environments. For te first example, te bandwidts are {6,,, 1}. We plot te cost functions of tis example in Figure 6 to explore ow ANNEALEDMS works in tis case. Te influence of te most significant peak is introduced gradually, wic guides searc towards te global mode. One can see tat even at 1 =, tere ave been plenty of local modes wic can easily make te searc stop prematurely. At =1, tere are tree major modes corresponding to te tree faces in te figure. Note tat mean sift does not converge to te exact modes in Figure 6 due to te Taylor approximation []. However it converges to a position close to te true mode. For localisation and tracking, tis accuracy loss is negligible. Te oter tree examples begin at =8andafivestep annealing guarantees a global mode in tese cases. For () (5) () (1) (3) (3) () ()

7 1 Gussian kernel, bandwidt:6 1 Gussian kernel, bandwidt: Gussian kernel, bandwidt: Gussian kernel, bandwidt:1 Figure 7: Te face tracking sequence wit standard mean sift (top) and ANNEALEDMS (bottom). Frames #5, #1, # and #5 are sown. Te object is accurately detected and tracked by ANNEALEDMS despite large displacements. In contrast, mean sift is more likely to trap into local modes and gives inaccurate results (#5, #1 and #) or even fails completely (#5). See te video facetracker.avi for details Figure 6: Te cost functions (corresponding to te first example in Figure 5) at different bandwidts: 6,,, and 1, are plotted as contours of D translations. Te true mode is marked wit a square. te CD and book cover localisation, we take te template models from oter images wit large geometry and sligt illumination differences. Te success proves tat te colour istogram is a robust feature. It is straigtforward to include oter features, e.g., intensity gradient, to make te localiser more robust. Witout te annealing procedure, most runs stop at a local maximum only wen te initial positions are located in te small area close to te global mode, can standard mean sift find te target. Wen no prior knowledge is available about te global maximum we are seeking, it is always beneficial to employ a relatively broad bandwidt mean sift procedure, wic can provide a coarse location of te global mode. In te experiments, altoug we do not carefully design te annealing scedule, global modes are acieved. 5.. Visual Tracking Many tracking algoritms fail to perform well in practice. Tey ave several fundamental drawbacks: (1) Tey work well only wen te displacements between consecutive frames are relatively small; () Usually tey cannot self-start; (3) Tey are not robust to occlusions and are unable to recover from momentary tracking failures. Standard mean sift trackers are no exception. Our ANNEALEDMS tracker alleviates tese weaknesses by incorporating an efficient bottom-up localisation functionality. Face tracking example. Te tracked target moves fast ence leading to large displacements between consecutive frames. An annealing scedule {6, 3, 18} is used by AN- NEALEDMS. Te ANNEALEDMS tracker is automatically started by a localisation process, wile te mean sift tracker is manually started. As in mean sift tracking, AN- NEALEDMS also starts at te position of te previous frame. Unless oterwise noted, in all te tracking experiments, te convergence tolerance is te l -norm distance between two iterations ε =. pixels. Figure 7 summarises te tracking results. Te ANNEALEDMS tracker is more robust and accurate tan te standard mean sift tracker: Wen te displacement is large, te standard mean sift tracker becomes easily stuck in spurious modes. Implementation issues. Mean sift migt get stuck at false modes caused by discrete colour values of pixels. Wang et al. observe tis penomenon in grey image istogram clustering [8]. Teir analysis also applies to colour image istograms. We avoid tis problem by simply ceiling te mean sift step m G (x) (Equation (5)). Tis modification increases te size of sift steps, ence leading to a quicker convergence. Te drawback is tat it migt lose accuracy. We use te original step by Equation (5) at te last bandwidt. Because we are only interested in te last convergence position, accuracy is retained. Bot in localisation and tracking, it as been observed tat tis simple treatment results in satisfactory convergence witout accuracy loss. We compare te number of convergence iterations per frame for te face tracking video in Figure 8. 9 One can see tat in tis example, teir convergence speeds are similar. In many frames, ANNEALEDMS is even faster. Te reason is tat mean sift at te first few bandwidts ( M...1 ) can move close to te mode quickly. However, larger bandwidts of ANNEALEDMS mean tat sligtly more computation migt be needed to build istograms. We also ave implemented te proposed accelerated mean sift algoritm (Section ) into tracking, and a considerable speedup as been acieved [3]. Basketball tracking example. Tis example sows te ANNEALEDMS tracker s ability to recover from temporal failures. Te original sequence is down sampled by a factor of to make te target s displacements larger. Te mean 9 Only frames #1... are compared because from #3 on, te mean sift tracker fails. For ANNEALEDMS, we count te sum of iterations at eac bandwidt.

8 # of convergence iterations mean sift tracker AnnealedMS tracker Frame index Figure 8: Comparison of te number of iterations per frame: mean sift (marked wit circles) vs. ANNEALEDMS (marked wit squares), for te face tracking sequence. sift tracker fails as early as at Frame #6. Terefore we only sow te tracking results of ANNEALEDMS in Figure 9. ANNEALEDMS tracks across bandwidts {3, 15, 8} and works successfully. It is not always necessary to perform a global searc in tracking. In ANNEALEDMS te ierarcical bandwidts control te size of te searcing area in te cost function. For tis video, te first bandwidt is not set very large because a global searc migt not be desired. At #18, ANNEALEDMS loses te target due to illumination canges. However, it recovers immediately at #19. It drifts sligtly because of te basket s occlusions at # and recovers at te next frame. Again we observe ANNEALEDMS tracking is efficient: only an average 8.1 iterations per frame is needed. Weetbix box tracking example. We track a part of a weetbix box, wic is recorded by an extremely unstable camera. ANNEALEDMS sows its robustness over te mean sift tracker again. See weetbixbox.avi for tracking results. Figure 9: Te basketball tracking results wit ANNEALEDMS. Frames #18, # and #9 are sown. See basketball.avi for details. 6. Conclusion We ave presented a new global mode seeking mean sift, termed ANNEALEDMS. Improvements are acieved over te standard mean sift wen te density as multiple peaked modes. We ave also introduced te new ANNEALEDMS strategy into localisation and tracking. Promising results ave been obtained in bot applications, even wit simple annealing scedules, wic are not carefully designed. An adaptive over-relaxed mean sift is also advanced to accelerate te convergence speed. Compared wit te standard mean sift algoritm, te number of convergence iterations is almost always significantly decreased. It provides an additional speedup to existing tecniques suc as locality sensitive asing [1] and fast Gaussian transformation [6]. Future work will explore te effects of annealing scedule design on te localisation and tracking performances. Oter discriminative features will be adopted for better localisation and tracking performances, rater tan relying solely on simple colour istograms. References [1] B. Georgescu, I. Simsoni, P. Meer, Mean sift based clustering in ig dimensions: A texture classification example, in: IEEE Int l Conf. on Comp. Vision, Vol., Nice, France, 3, pp [] D. Comaniciu, V. Rames, P. Meer, Kernel-based object tracking, IEEE Trans. on Patt. Anal. and Mac. Intell. 5 (5) (3) [3] R. Collins, Mean-sift blob tracking troug scale space, in: IEEE Conf. on Comp. Vision and Patt. Recog., Vol., Madison, Wisconsin, 3, pp. 3. [] A. Elgammal, R. Duraiswami, L. S. Davis, Probabilistic tracking in joint feature-spatial spaces, in: IEEE Conf. on Comp. Vision and Patt. Recog., Vol. 1, Madison, Wisconsin, 3, pp [5] G. D. Hager, M. Dewan, C. V. Stewart, Multiple kernel tracking wit SSD, in: IEEE Conf. on Comp. Vision and Patt. Recog., Vol. 1, Wasington, D.C.,, pp [6] C. Yang, R. Duraiswami, L. Davis, Efficient spatial-feature tracking via te mean-sift and a new similarity measure, in: IEEE Conf. on Comp. Vision and Patt. Recog., Vol. 1, San Diego, CA, 5, pp [7] K. Fukunaga, L. D. Hostetler, Te estimation of te gradient of a density function, wit applications in pattern recognition, IEEE Trans. on Information Teory 1 (1975) 3. [8] Y. Ceng, Mean sift, mode seeking, and clustering, IEEE Trans. on Patt. Anal. and Mac. Intell. 17 (8) (1995) [9] D. Comaniciu, P. Meer, Mean sift: A robust approac toward feature space analysis, IEEE Trans. on Patt. Anal. and Mac. Intell. (5) () [1] M. Fasing, C. Tomasi, Mean sift is a bound optimization, IEEE Trans. on Patt. Anal. and Mac. Intell. 7 (3) (5) [11] Z. Zivkovic, B. Krose, An EM-like algoritm for color-istogram-based object tracking, in: IEEE Conf. on Comp. Vision and Patt. Recog., Vol. 1,, pp [1] Z. Fan, Y. Wu, Multiple collaborative kernel tracking, in: IEEE Conf. on Comp. Vision and Patt. Recog., Vol., San Diego, CA, 5, pp [13] P. Pérez, C. Hue, J. Vermaak, M. Gangnet, Color-based probabilistic tracking, in: 7t European Conf. on Comp. Vision, Vol. 35 of LNCS, Springer, Copenagen, Denmark,, pp [1] P. van Laaroven, E. Aarts, Simulated Annealing: Teory and Applications, Springer Verlag, [15] R. M. Neal, Annealed importance sampling, Statistics and Computing 11 () (1) [16] J. Deutscer, I. Reid, Articulated body motion capture by stocastic searc, Int l J. of Comp. Vision 61 () (5) [17] L. Xu, On convergence properties of te EM algoritm for Gaussian mixtures, Neural Computation 8 (1) (1996) [18] L. E. Ortiz, L. P. Kaelbling, Accelerating EM: An empirical study, in: Uncertainty in Artificial Intell., Stockolm, Sweden, 1999, pp [19] R. Salakutdinov, S. Roweis, Adaptive overrelaxed bound optimization metods, in: Int l Conf. on Mac. Learning, AAAI Press, Wasington DC, 3, pp [] D. Comaniciu, An algoritm for data-driven bandwidt selection, IEEE Trans. on Patt. Anal. and Mac. Intell. 5 () (3) [1] M. C. Jones, J. S. Marron, S. J. Seater, A brief survey of bandwidt selection for density estimation, J. of te American Statistical Association 91 (33) (1996) 1 7. [] P. Hall, M. C. Minnotte, C. Zang, Bump unting wit non-gaussian kernels, Te Annals of Statistics 3 (5) () [3] C. Sen, M. J. Brooks, Adaptive over-relaxed mean sift, in: 8t Int l Symp. on Signal Processing and Its Applications, Sydney, Australia, 5. [] P. Cauduri, J. S. Marron, Scale space view of curve estimation, Te Annals of Statistics 8 () () 8 8. [5] T. Lindeberg, Scale-Space Teory in Comp. Vision, Kluwer Academic Publisers, 199. [6] D. Comaniciu, Nonparametric information fusion for motion estimation, in: IEEE Conf. on Comp. Vision and Patt. Recog., Vol. 1, Madison, Wisconsin, 3, pp [7] W. Härdle, M. Müller, S. Sperlic, A. Werwatz, Nonparametric and Semiparametric Models, Springer Series in Statistics, Springer,. [8] H. Wang, D. Suter, False-peaks-avoiding mean sift metod for unsupervised peak-valley sliding image segmentation, in: 8t Digital Image Computing: Tecniques and Applications, Sydney, 3, pp