A novel framework for motion segmentation and tracking by clustering incomplete trajectories

Transcription

1 A novel framewor for moton segmentaton and tracng by clusterng ncomplete trajectores Vasleos Karavasls, Konstantnos Bleas, Chrstophoros Nou Department of Computer Scence, Unversty of Ioannna, PO Box 1186, Ioannna, Greece, 1 Abstract In ths paper, we present a framewor for vsual object tracng based on clusterng trajectores of mage ey ponts extracted from an mage sequence. The man contrbuton of our method s that the trajectores are automatcally extracted from the mage sequence and they are provded drectly to a model-based clusterng approach. In most other methodologes, the latter consttutes a dffcult part snce the resultng feature trajectores have a short duraton, as the ey ponts dsappear and reappear due to occluson, llumnaton, vewpont changes and nose. We present here a sparse, translaton nvarant regresson mxture model for clusterng trajectores of varable length. The overall scheme s converted nto a Maxmum A Posteror approach, where the Expectaton-Maxmzaton (EM) algorthm s used for estmatng the model parameters. The proposed method detects the dfferent objects n the nput mage sequence by assgnng each trajectory to a cluster, and smultaneously provdes ther moton. Numercal results demonstrate the ablty of the proposed method to offer more accurate and robust solutons n comparson wth other tracng approaches, such as the mean shft tracer, the camshaft tracer and the Kalman flter. Index Terms Moton segmentaton, vsual feature tracng, trajectory clusterng, sparse regresson modelng. I. INTRODUCTION Vsual target tracng s a preponderant research area n computer vson wth many applcatons such as survellance, targetng, acton recognton from moton, moton-based vdeo compresson,

2 teleconferencng, vdeo ndexng and traffc montorng. Tracng s the procedure of mang nference about apparent moton gven a sequence of mages, where t s generally assumed that the appearance model of the target (e.g. color, shape, salent feature descrptors etc.) s nown a pror. Hence, usng a set of measurements from mage frames the target s poston s estmated. Tracng algorthms can be classfed nto two man categores [1], [2]. The frst one s based on flterng and data assocaton. It assumes that the movng object has an nternal state, where the poston of an object s estmated by combnng the measurements wth the model of the state evoluton. Kalman flter [3] belongs to such famly whch can be successfully appled when the assumptons made about the model of the moton are adequately satsfed, ncludng cases of occluson. Alternatvely, partcle flters [4], ncludng the condensaton algorthm [5], are more general tracng methods wthout assumng any specfc type of denstes. Fnally, there are methods relyng on feature extracton and tracng usng optcal flow technques [6]. A general drawbac of ths famly s that the type of object s moton must be nown and modeled correctly. On the other hand, there are approaches whch are based on target representaton and localzaton and assume a probablstc model for the object s appearance n order to estmate ts locaton. More specfcally, color or texture features of the object mased by an sotropc ernel are used to create a hstogram [1]. Then, the object s poston s estmated by mnmzng a cost functon between the model s hstogram and canddate hstograms of the next mage. A characterstc method n ths category s the mean shft algorthm [1] and ts extensons [7], [8], where the object s supposed to be surrounded by an ellpse whle the hstogram s constructed from ts nternal pxel values. Also, algorthms based on the mnmzaton of the dfferental Earth mover s dstance [9], [10] belong to ths category. The above methods trac only one object at a tme. Other methods that smultaneously trac many objects have also been proposed [11], [12]. In [13], multple objects are traced by usng Graph Cuts [14] over some observatons (.e. possble locatons of the object) whch are extracted. Another approach s to employ level-sets to represent each object to be traced [15], [16] whch may also be useful to handle the case of multple object tracng [17], [18] by optmally groupng regons whose pxels have smlar feature sgnatures. An applcaton n vehcle tracng s presented n [19] where multple vehcles are traced by ntally assgnng each pxel ether to bacground, or foreground and applyng next Kalman flter to estmate the vehcle poston and assocate each foreground pxel wth a sngle object. In cases of artculated objects (e.g. pedestrans) stereo depth nformaton have been used n order to acqure a 3D artculated structure per object and then estmate the 3D trajectory of each object [20]. In any case, combnng multple object representatons could mae the tracng 2

3 procedure more robust [21], [22]. Moton segmentaton consttutes a sgnfcant applcaton of tracng algorthms, whch ams at dentfyng movng objects n an mage sequence. It can be seen ether as the post-processng step of a tracng algorthm, or as an assstve mechansm of the tracng algorthms by ncorporatng nowledge to the number of ndvdual motons or ther parameters. It has been consdered as an optcal flow estmaton [23] where volatons of brghtness constancy and spatal smoothness assumptons are addressed. In [24], an alternatve scheme s used where small textured patches wth unform optcal flow are detected and clustered nto layers, each one havng an affne flow. Lewse n [25], mage features are clustered nto groups and the number of groups s updated automatcally over tme n. Trajectores of mage ey-ponts extracted from an mage sequence have also been used for tracng. Groupng 3D trajectores s proposed n [26] usng an agglomeratve clusterng algorthm where occlusons are handled by multple tracng hypotheses. Fnte mxtures of Hdden Marov Models (HMMs) were also employed n [27] where tranng s made usng the EM algorthm. Zappela et al. [28] assume that trajectores belongng to dfferent objects le n dfferent subspaces, thus, the segmentaton can be obtaned by groupng together all the trajectores that generate these subspaces. The groupng s obtaned by the egenvectors of an affnty matrx whch contans the parwse dstances between trajectores computed n the correspondng subspaces. The number of motons can be estmated based on the number of egenvalues of the symmetrc normalzed Laplacan matrx. In [29], a two stage procedure s proposed: Intally, an teratve clusterng scheme s appled that groups temporally overlappng trajectores wth smlar velocty drecton and magntude. Next, the clusters created are merged each other coverng larger tme spans. Furthermore, spectral clusterng approaches were also proposed, such as n [30] where the motons of the traced feature ponts are modeled by lnear subspaces, and the approach n [31] where mssng data from the trajectores are flled n by a matrx factorzaton method. Moreover, Yan et al. [32] estmate a lnear manfold for every trajectory and then spectral clusterng s employed to separate these subspaces. In [33], moton segmentaton s accomplshed by computng the shape nteracton matrces for dfferent subspace dmensons and combne them to form an affnty matrx that s used for spectral clusterng. Fnally, many methods proposed ndependently rely on the separaton of the mage nto layers. For example, n [34], tracng s performed n two stages: at frst foreground extracted blobs are traced usng graph cut optmzaton and then pedestrans are assocated wth blobs and ther moton s estmated by a Kalman flter. In ths wor, we present a novel framewor for vsual target tracng based on model-based 3

4 clusterng trajectores of ey ponts (Fg. 1). The proposed method creates trajectores of mage features (e.g Harrs corner [35]). However, ey pont tracng ntroduces an addtonal dffculty snce the resultng feature trajectores have a short duraton, as the ey ponts dsappear and reappear due to occluson, llumnaton and vewpont changes. Therefore, we are dealng wth tme-seres of varable length. Moton segmentaton s then converted nto a clusterng problem of these nput trajectores, n a sense of groupng together feature trajectores that belong to the same object. For ths purpose, we use an effcent regresson mxture model, whch has three sgnfcant features: a) Sparse propertes over the regresson coeffcents, b) t s allowed to be translated n measurement space and c) ts nose covarance matrx s dagonal and not sphercal as t s commonly used. The above propertes are ncorporated through a Bayesan regresson modelng framewor, where the EM algorthm can be appled for estmatng the model parameters. Specal care s gven for ntalzng the EM algorthm where an nterpolatng scheme s proposed based on executng successvely the -means algorthm over the duraton of trajectores. Experments show the robustness of our method to occlusons and hghlght ts ablty to dscover better the objects moton n comparson wth the state-of-the art mean shft algorthm [1]. A prelmnary verson of ths wor was presented n [36]. Here, we made a more extended evaluaton of the proposed method. The expermental results nvolve comparsons wth more methods, the evaluaton of the method usng mage sequences of the Hopns 155 dataset [37] and the analyss of the algorthm usng more features for the generaton of trajectores. The rest of the paper s organzed as follows: the procedure of feature extracton and tracng n order to create the trajectores s presented n secton II. The trajectores clusterng algorthm s presented n secton III, expermental results are shown n secton IV and conclusons s drawn n secton V. II. EXTRACTING TRAJECTORIES Trajectores are constructed by tracng ponts n each frame of the mage sequence. The man dea s to extract some salent ponts from a gven mage and assocate them wth ponts from prevous mages. To ths end, we employ the so called Harrs corners [35] and standard optcal flow for the data assocaton step [38]. Let us notce that any other scale or affne nvarant features [39], [40] would also be approprate. In ths wor we use Harrs corner features due to ther smplcty, as they rely on the egenvalues of the matrx of the second order dervatves of the mage ntenstes. Let T be the number of mage frames and Y = {y } =1,...,N be a lst of trajectores wth N beng unnown beforehand. Each ndvdual trajectory y conssts of a set of 2D ponts, the tme of 4

5 0 100 y (2) y (1) 500 (a) 500 (b) y (1) y (2) (c) T (d) T Fg. 1. Trajectores extracted from an mage sequence. (a) The frst frame of the mage sequence showng four robots and ther mean trajectores. The group of robots perform a square-le movement. (b) The trajectores of the features extracted from the mage sequence. The two axes represent the horzontal and vertcal coordnates. (c) The horzontal trajectores along tme. (d) The vertcal trajectores along tme. Notce that there s a large number of short and ncomplete trajectores because the features dsappear and reappear durng the mage sequence due to llumnaton changes and the dstance of the object from the camera. appearance of ts corner pont nto the trajectory, (.e. the number of the mage frame) and the optcal flow vector of the last pont n the lst. Intally, the lst Y s empty. In every mage frame, Harrs corners are detected and the optcal flow vector at each corner s estmated [6]. Then, each corner found n the current mage frame s attrbuted to a trajectory that already exsts, or a new trajectory s created havng only one element, 5

6 the corner under consderaton. Accordng to ths scheme, three cases are possble: If any ey pont of the prevous frame has an optcal flow vector pontng out the ey pont under consderaton, then the current corner s attrbuted to an exstng trajectory. In ths case, a trajectory follows the optcal flow dsplacement vector, meanng that the corner s apparent n consecutve frames. If there s no such ey pont n the prevous frame, we apply a wndow around the last corner whch s smlar to the current corner. If there are more than one smlar corners then we choose the closest one. Otherwse, a new trajectory s constructed contanng only the corner under consderaton. In Fg. 2, an ntutve example s presented where three corners are consdered for demonstraton purposes and fve tme nstances are depcted. In the frst frame, three corners are detected and three trajectores are created. In the second frame, the same corners are detected and assocated wth exstng trajectores due to optcal flow constrant. Next, one corner s detected and attrbuted to an exstng trajectory due to optcal flow constrants whle the other two ey ponts are occluded. Durng the fourth frame, the ey pont that was not occluded s also detected and attrbuted to an exstng trajectory. One of the other two corner ponts that reappear s attrbuted to a trajectory due to local wndow matchng. The other corner s not assocated wth any exstng trajectory, so a new trajectory s created. In the last frame, three corners are detected and assocated wth exstng trajectores due to optcal flow. Thus, four trajectores have been created: two trajectores correspondng to the same ey pont and two addtonal trajectores nvolvng two dstnct ey ponts. Fg. 2. Example of trajectores constructon. The red dots represent the mage ey ponts and the green lnes represent ther trajectores. The fgure s better seen n color. Once the lst Y s created, the trajectores of the corner ponts that belong to the bacground are elmnated. Ths s acheved by removng the trajectores havng a small varance, accordng to a predefned-threshold value, as well as the trajectores of small length (e.g. 1% of the number of frames). The complete procedure s descrbed n Algorthm 1. 6

7 Algorthm 1 Trajectores constructon algorthm 1: functon CREATETRAJECTORIES(Im) 2: Input: an mage sequence Im. 3: Output: a lst of trajectores Y. 4: Y =. 5: for every mage {m (t) } t=1,...,t do 6: Detect corners {c (t) l } l=1,...,l (t) and estmate optcal flow {f (t) l } l=1,...,l (t) n each one. 7: for every corner {c (t) l } l=1,...,l (t) detected n m (t) do 8: f y has ts last corner c last current corner,.e. c last + f last 9: Insert c (t) l nto y. c (t) l n the mage t 1 and ts optcal flow f last then 10: else f y has ts the wndow around ts last corner c last thw current corner c (t) l then 11: Insert c (t) l nto y. 12: else 13: Insert a new trajectory y wth only c (t) l nto Y. 14: end f 15: end for 16: end for 17: Elmnate trajectory y wth small varaton n ts corners coordnates. 18: end functon ponts to the smlar to the wndow around III. CLUSTERING TRAJECTORIES OF VARIABLE LENGTH Suppose we have a set of trajectores of N traced feature ponts over T frames obtaned from the prevous procedure. The am of tracng s to detect K ndependently movng objects n the scene and smultaneously estmate ther characterstc moton. Ths can be seen as a clusterng problem. A 2D trajectory y = (y (1), y (2) ) conssts of two drectons: (1) horzontal and (2) vertcal. It s defned by a set of T ponts {(y (1) 1, y(2) 1 ),..., (y(1) T, y (2) T )}, correspondng to T mage postons (t 1,..., t T ) of the mage sequence. It s mportant to note that we deal wth trajectores of varable length T snce occlusons or llumnaton changes may bloc the vew of the objects n certan mage frames. Lnear regresson model consttutes a powerful platform for modelng sequental data that can be adopted n our case. In partcular, we assume that a trajectory y (j) of any drecton j = {1, 2} can 7

8 be modeled through the followng functonal form: y (j) = Φ w (j) + d (j) + e (j), (1) where Φ s the desgn ernel matrx (common for both drectons) of sze T T, and w = (w 1,..., w T ) s the vector of the T unnown regresson coeffcents. At frst, we assume that the nput space conssts of the tme nstances (t 1,..., t T ) whch correspond to the T frames of the mage sequence. Havng that n mnd, equaton (1) s the standard lnear regresson model [41] wth a global translaton term added, where y (j) s a vector of length T contanng the values of the -th observaton of the horzontal (j = 1) and vertcal (j = 2) drectons. The -th desgn matrx Φ s generated by eepng the T rows of the global desgn matrx Φ that correspond to tme nstances (t 1,..., t T ), where the -th ey pont exsts. The global matrx Φ s a ernel matrx of sze T T wth elements calculated by a ernel functon,.e. Φ(, l) = (t, t l ), where t, t l {t 1,..., t T }. In our wor, we consdered the mexcan hat wavelet ernel (t, t l ) = ( ) (t l t ) 2 3σπ 4 σ e (t l t ) 2 2 2σ 2, though any other ernel functon could be used (e.g. Gaussan). Also, n the above equaton, we assume a translaton scalar term d (j) that allows for the entre trajectory to be translated globally [42], see Fg. 3. Incorporatng such term results n a regresson model that allows for arbtrary translatons n measurement space. In our case, the features are dstrbuted around the edges of the objects and the trajectores of the ey ponts are translated n order to be algned wth the trajectory of the center of gravty of the object. Fnally, the error term e (j) n the above formulaton s a T -dmensonal vector that s assumed to be zero-mean Gaussan and ndependent over tme: wth a dagonal covarance matrx Σ (j) e N (0, Σ ), (2) = dag(σt 2(j) 1,..., σt 2(j) T ). More specfcally, Σ s a bloc matrx of a T T dagonal covarance matrx that corresponds to the nose varance of T frames. Under these assumptons, the condtonal densty of a trajectory s also Gaussan: p(y (j) w (j), Σ (j), d (j) ) = N (Φ w (j) + d (j), Σ (j) ). (3) Moreover, we consder the scalar d (j) to be a zero-mean Gaussan varable wth varance u (j) : d (j) N (0, u (j) ). (4) Thus, we can further ntegrate out d to obtan the margnal densty for y (j) whch s also Gaussan: p(y (j) θ) = p(y (j) w (j), Σ (j), d (j) )p(d (j) )dd (j) = N (Φ w (j), Σ (j) + u (j) 1), (5) 8

9 (a) (b) Fg. 3. The effect of the translaton parameter. (a) A set of trajectores. (b) Algnment of the trajectores. where 1 s a matrx of 1 s of sze T T. The margnal densty s mplctly condtoned on the parameters θ = {w, u, Σ}. In our study we consder that every object can be descrbed by a unque functonal regresson model, as gven by the set parameters θ = {θ (1), θ(2) }, where θ(j) = {w (j), u(j), Σ(j) }, that fts to all trajectores belong to ths object. Therefore, the objectve s to dvdng the set of N trajectores nto K clusters. Ths can be descrbed by the followng regresson mxture model: p(y Θ) = K π p(y θ ) = =1 K π p(y (1) =1 θ (1) )p(y(2) θ (2) ), (6) where we have assumed ndependence between that trajectores of both drectons (y (1), y (2) ). In addton, π are the mxng weghts satsfyng the constrants π 0 and K =1 π = 1, whle Θ s the set of all the unnown mxture parameters, Θ = {π, θ } K =1. An mportant ssue, when dealng wth regresson models s how to determne ther order. Models of small order can lead to under-fttng, whle larger order lead to curve over-fttng. Both cases may cause to serous deteroraton of the clusterng performance. Sparse modelng [43] offers a sgnfcant soluton to ths problem by employng models havng ntally many degrees of freedom than could unquely be adapted gven data. Sparse Bayesan regresson can be acheved through a herarchcal pror defnton over regresson coeffcents w (j) = (w (j) 1,..., w(j) T )T. In partcular, we assume frst that coeffcents follows a zero-mean Gaussan dstrbuton: T p(w (j) α(j) ) = N (w(j) 0, A 1(j) ) = N (w (j) l 0, α 1(j) l ) (7) where A (j) s a dagonal matrx contanng the T elements of precsons α (j) l=1 = (α (j) 1,..., α(j) T )T. 9

10 We mpose next a Gamma pror on these hyperparameters: p(α (j) T ) = Gamma(α (j) T l a, b) l=1 l=1 α (j)a 1 l exp( bα (j) l ), (8) where a and b denote parameters that are a pror set to values near zero. The above two-stage herarchcal pror s actually a Student s t-dstrbuton [43]. Ths s a heavy taled pror dstrbuton that enforces most of the values α (j) l to be large, thus the correspondng w (j) l are set to zero and elmnated from the model. In ths way, the complexty of the regresson model s controlled n an automatc and elegant way and over-fttng s avoded. Now, the clusterng procedure has been converted nto a maxmum a posteror (MAP) estmaton approach, n a sense of estmatng the mxture model parameters that maxmze the MAP log-lelhood functon gven by: L(Θ) = N K K log{ π p(y θ )} + =1 =1 2 =1 j=1 {log p(w (j) α(j) ) + log p(α(j) )}. (9) The model can be traned usng the Expectaton - Maxmzaton (EM) algorthm [44] that teratvely performs two man stages: The E-step where the expected values of the hdden varables are calculated. In our case, ths ncludes the cluster labels of trajectores as gven by the posteror probabltes: z = P ( y ) = as well as the mean value of the translatons d (j) fact that the posteror densty of translatons s also Gaussan: where ˆd (j) ( = V (j) y (j) p(d (j) y(j) where 1 s a T -length vector of 1 s., ) p(y (j) θ (j) ) Φ w (j) T Σ 1 (j) 1 and V (j) = π p(y θ ) π p(y θ ), (10) at any drecton. The latter s obtaned by usng the )p(d(j) (j) ) = N ( ˆd, V (j) ), (11) ( ) 1 1 T Σ 1(j) u (j), (12) Durng the M-step, the maxmzaton of the expected value of the complete log-lelhood functon (referred as Q-functon n the machne learnng bblography [42]) s performed: ( ) N K 2 Q(θ t, θ t 1 ) = z log π + 1 y (j) log Σ(j) 2 µ (j) T (j) Σ 2 =1 =1 + K =1 j= log A(j) j=1 w(j)t A (j) w(j) + 2 K 2 T =1 j=1 l=1 ( y (j) log α (j)a 1 l µ (j) ) bα (j) l. (13) 10

11 Maxmzng (13) wth respect to the model parameters leads to the followng update rules [42], [45]: ˆπ = ŵ (j) = α (j) l = ˆσ 2(j) l = û (j) = N =1 z, N [ N ] 1 N (14) z Φ T Σ 1(j) Φ + A (j) z Φ T Σ 1(j) (y (j) (j) ˆd ), (15) =1 = a, l = 1,..., T, (16) ŵl 2(j) + 2b { ( ) } N =1 z y (j) l [Φ ŵ (j) ] (j) 2 (j) l ˆd + V N =1 z, l = 1,..., T. (17) N =1 z ( ˆd2 (j) N =1 z + V (j) ). (18) where [.] l ndcates the l-th component of the T -dmensonal vector that corresponds to tme locaton t l. After convergence of the EM algorthm, two nds of nformaton are avalable: At frst, the cluster labels of the trajectores are obtaned accordng to the maxmum posteror probablty value (Eq. 10). Moreover, the moton of objects are obtaned from the predcted mean trajectores per cluster,.e. µ = (µ (1), µ(2) ) = (Φw(1), Φw(2) ). A. Intalzaton Strategy A fundamental ssue when applyng the EM algorthm, s ts strong dependence on the ntalzaton of the model parameters due to ts local nature. Improper ntalzaton may lead to reachng poor local maxma of the log-lelhood, a fact that may affect sgnfcantly the performance of the method. A natural way for ntalzaton s by randomly selectng K samples from the set of nput trajectores, one for each cluster. Then, we can obtan the least-squares soluton for ntalzng the regresson coeffcents. In addton, the nose varance Σ s ntalzed by a small percentage of the total varance of all trajectores equally for each clusters, whle we set the mxng weghts π equal to 1/K. Fnally, one step of the EM algorthm s executed for further refnng these parameters and calculatng the MAP log-lelhood functon. Several such dfferent trals are executed and the soluton wth the maxmum MAP lelhood functon s selected for ntalzng model parameters. However, the above scheme cannot be easly appled to our approach due to the large varablty n length (T ) of the nput trajectores whch brngs a practcal dffculty n obtanng the leastsquared soluton. For ths reason, we have followed a more robust ntalzaton scheme based on the nterpolaton among successve tme steps. More specfcally, startng from the frst tme step we perform perodcally (e.g. at every 5T % frames) the -means clusterng over the ponts (y (1) t, y(2) t ). 11

12 Then, the resultng K centers are assocated wth those of the prevous tme accordng to the mnmum dstance crteron. Fnally, a lnear nterpolaton (per cluster) s performed and thus the resultng K curves are used for ntalzng the parameters of the K clusters. It must be noted that n cases where there s a large number of features representng the bacground, the ntalzaton may dverge from the desred soluton snce the exstence of a sgnfcant amount of outlers affects the -means soluton. Even f durng our experments we have not faced wth any such problem, treatng wth ths stuaton stll remans a future plan of study. An example of the proposed process s gven n Fg. 4 adopted from an expermental data set, where both the ntal nterpolated trajectores (Fg. 4c) and the fnal clusterng soluton are shown (Fg. 4d). IV. EXPERIMENTAL RESULTS We have evaluated the performance of our approach usng both smulated and real examples. Demonstraton vdeos are avalable at varavas/moton segmentaton and tracng. Some mplementaton detals of our method are the followng: At frst, we have normalzed spatal and temporal coordnates nto the nterval [0, 1] whle the extracted trajectores ether wth length less than 1% of the number of frames T, or wth varance less than 0.01 were not taen nto account. ( ) We have selected the mexcan hat wavelet ernel (t, t l ) = (t l t ) 2 3σπ 4 σ e (t l t ) 2 2 2σ 2 for the desgn ernel matrx Φ, where the scalar parameter too the value of σ = 0.3. Experments have shown that the performance of our approach s not very senstve to ths parameter, snce we too smlar results for values n the range of [0.1, 0.5]. Comparson has been made wth the mean shft algorthm [1], the camshft algorthm [46] and the Kalman flter [3], whch are establshed algorthms n vsual tracng. For the mean shft algorthm, the mages were represented n the RGB color space usng hstograms of 16 bns for each component. For the camshft algorthm, the hue component of HSV color space was used to construct a 16-bn hstogram. For the Kalman flter, camshft was used n order to provde measurements (the poston and the sze of the object). For ntalzng all these algorthms, we have manually selected the poston and the sze of each object n the frst frame of the mage sequence. Then, the objects are traced, usng a dstnct tracer per each target. The centers of the ellpses surroundng the targets are used to construct the mean trajectory of each object. Ths comparson favors these algorthms n cases where the features are not unformly dstrbuted around the object, as the center estmated by the features may vary from the geometrc center of the object (Fg. 5). 12

13 µ (1) y (1) T T 500 (a) 500 (b) µ (1) µ (1) (c) T (d) T Fg. 4. The overall progress of our method appled to an expermental mage sequence of 250 mages wth = 4 objects wth dfferent motons. (a) Real trajectores, (b) nput trajectores, (c) ntal estmaton of mean trajectores usng the proposed technque, (d) the estmated trajectores after EM convergence. A. Experments wth smulated data sets The frst seres of experments nvolves seven (7) smulated mage sequences wth spheres movng n predefned drectons as shown n Fg. 6 and Fg. 7. Each mage sequence contans 130 frames of sze The value of N vares from 1500 to 2000 trajectores per case, wth average length around T = 60 frames each trajectory. In the frst fve problems, no occlusons are smulated (all objects are vsble n every frame). In the rest two problems, occluson s happened, where n Sm6 a sphere dsappears whle n Sm7 a sphere dsappears and reappears. 13

14 Fg. 5. Features are not unformly dstrbuted over the object and the center of gravty of the ey ponts does not concde wth the center of gravty of the object. The small dots represent the features and the bg dot represents ther barycenter. The fgure s better vsualzed n color. Snce we are aware of the ground truth, the performance of all tracng approaches was evaluated usng two crtera: The mean squared error (MSE) measured n pxels, between the ground truth r and the estmated mean trajectores µ as gven by MSE = 1 K T K 2 =1 j=1 r (j) µ (j) 2. The accuracy (ACC) that counts the percentage of correctly classfed trajectores. It must be noted that the nput trajectores created by our method have been also chosen to evaluate the mean shft algorthm, the camshft algorthm and the Kalman flter tracer. In partcular, we assgn every nput trajectory to an object accordng to the smallest dstance wth the predcted mean trajectory. The depcted results are presented n Table I. As t s obvous, we too comparable results n the frst thee approaches Sm1, Sm2 and Sm3. However, n the fourth problem (Sm4), our approach and mean shft successfully trac the objects, whle camshft and the Kalman flter are faled. Ths s happened n the case of camshft due to the fact that the objects are overlappng n the ntal frame, and after some teratons the result s an ellpse wth ts center located at the center of the mage and whose sze s ncreased n order to contan all the objects nsde t. Moreover, Kalman flter fals because t uses camshft to obtan the measurements. On the other hand, mean shft tracs the objects due to the fact that n ths mplementaton we decded not to ntegrate a scale change, as n camshft. In problem Sm5, camshft and Kalman flter fal agan, because two objects (the one wth 14

15 Smulated datasets (A) Sm1 Sm2 Sm3 Sm4 True moton Input trajectores Our approach Mean shft Camshft Kalman flter Fg. 6. Comparatve results wth four artfcal datasets. For each problem we gve the true objects moton, the created nput trajectores and the estmated moton by all approaches. the green trajectory and the one wth the blue trajectory n Fg. 7) pass near each other and camshft maes the target ellpse bgger n order to nclude both objects. In problem Sm6 one object suddenly dsappears (the one wth the red trajectory n Fg. 7). Fnally, n problem Sm7 the object dsappears 15

16 Smulated datasets (B) Sm5 Sm6 Sm7 True moton Input trajectores Our approach Mean shft Camshft Kalman flter Fg. 7. Comparatve results wth three artfcal datasets. For each problem we gve the true objects moton, the created nput trajectores and the estmated moton by all approaches. and reappears n another poston after some tme. Mean shft fals to trac the sphere that dsappears and reappears and tracs the object only as long as t s vsble. Camshft and Kalman flter, as they tae nto account scale changes, ncrease the sze of the ellpse and trac a nearby object. On the other 16

17 TABLE I THE PERFORMANCE OF THE COMPARED METHODS IN TERMS OF CLASSIFICATION ACCURACY (ACC) AND MEAN SQUARED ERROR (MSE). Problem Our approach Mean shft Camshft Kalman MSE ACC MSE ACC MSE ACC MSE ACC Sm % % 4 96% 0 100% Sm % % % % Sm % % % % Sm % % lost lost lost lost Sm % % lost lost lost lost Sm % % lost lost lost lost Sm % lost lost lost lost lost lost hand, the proposed method correctly assocates the two separated trajectores of the sphere. Fnally, t must be noted that n the case of both problems Sm6 and Sm7, the frames n whch a sphere s not vsble are not taen nto account for computng the MSE crteron. B. Experments wth real data sets We have also evaluated our moton segmentaton approach usng fve (5) real datasets shown n Fg. 8 contanng mages of sze All of these vdeos were created n our laboratory and they may be downloaded at varavas/moton segmentaton and tracng. The frst three of them (Real1-3) show moble robots movng n varous drectons: In Real1 (T = 250), the robots are movng towards the borders of the mage formng the vertces of a square. In Real2 (T = 680), the robots are movng around the center of the mage formng a crcle, and fnally n Real3 (T = 500), the robots are movng forward and bacward. The rest two datasets (Real4-5) show two persons walng, and so occluson tae place as one person gets behnd the other. In partcular, n Real4 (T = 485) two persons are movng from one sde of the scene to the other and bacwards, whle n Real5 (T = 635) the persons addtonally move forward and bacward n the scene. As ground truth s not provded n these cases we have evaluated our approach only vsually. In problems Real1-3 all algorthms produce approxmately the same trajectores. On the contrary, n the cases of Real4 and Real5 problems, where we deal wth artculated objects and occluson, mean shft and camshft fal to properly trac the persons and one of them s lost. Moreover, the trajectory of the center of the ellpse (whch represents the object n mean shft and camshft) s not smooth. Ths s due to the change n the appearance of the target. When the person wals, there are frames where both arms and legs of a person are vsble and nstances where only one of them s present. In these 17

18 cases, mean shft and camshft may produce abrupt changes n moton estmaton because the person n the frame has change ts appearance wth respect to ts appearance n the frst frame (whch s used to ntalze the model representng the object). On the other hand, our method s more accurate and produces a smooth trajectory. Loong n more detal the problem Real4, we can see the person n blac dsappears (because he gets behnd the other person) twce durng ths mage sequence: at frst, when he s movng from rght to left, and then, as he s movng from left to rght. Mean shft successfully follows the person before and after the frst occluson, but t fals to trac t when the second one taes place. Camshft fals to trac the person after the frst occluson, but t follows t after the second occluson when the person turns bac. The Kalman flter successfully follows the person. Ths s better depcted n Fg. 9 where the predcted trajectory, correspondng to the person n blac, s shown n green. The part of the trajectory where the person s lost s hghlghted by an ellpse. Mean shft (Fg. 9(b)) follows the correct person from rght to left and from left untl the mddle of the mage when the second occluson taes place. Then t follows the other person whch moves from the mddle of the mage to the left. Camshft (Fg. 9(c)) follows the person untl the frst occluson at the mddle of the mage. Then, t follows the other person whch moves from the mddle to the rght and from the rght to the mddle. After the second occluson s occurred, t fnds the correct person agan whch moves from the mddle to the rght. On the other hand, the proposed method successfully tracs the object n all frames (Fg. 9(a)). C. Experments usng the Hopns 155 dataset We have also used the Hopns 155 dataset [37] to evaluate our approach, where we have selected the traffc subset that conssts of 31 scenaros wth two motons and 7 scenaros wth three motons. For each vdeo, the extracted trajectores and the ground truth are provded. Here we must hghlght that these trajectores have been manually corrected or flled by the researchers that constructed the dataset [37], so each trajectory has a value for every frame. Some representatve frames of the traffc subset are presented n Fg. 10. In Tables II and III, the performance of our algorthm s presented n terms of the classfcaton error. For comparson purposes, we also summarze the results obtaned by 15 other methods as they are presented n the web page of the dataset ( More specfcally, the average value of the classfcaton error as calculated by our method over the correspondng dataset s shown n Table II. For the rest of the compared methods, we present the mnmum, the maxmum, the mean and the medan values for the average classfcaton error over 18

19 Real datasets Real1 Real2 Real3 Real4 Real5 True moton Input trajectores Our approach Mean shft Camshft Kalman flter Fg. 8. Comparatve results wth fve real datasets. For each problem we gve the true objects moton (chosen manually), the created nput trajectores and the estmated moton by all approaches. all the 15 state of art methods. Ths means that we have ncluded the average classfcaton error of each of the compared methods n the computaton of each statstc (mn, max, mean, medan). As t may be observed, our method provdes satsfactory results compared to all other methods. Ths s 19

20 (a) Fg. 9. (b) (c) Estmated trajectores for the dataset Real4. (a) Our method, (b) mean shft, (c) camshft. The green (prnted n lght gray n blac and whte) trajectory n (b) and (c) corresponds to the person n blac movng from the rght sde of the mage to the left and bacwards. The ellpse hghlghts the part of the trajectory where the person s lost, because mean shft or camshft fals to trac the object due to occluson. The fgure s better vsualzed n color. Fg. 10. Representatve frames of the Hopns 155 dataset. The feature ponts are mared usng dfferent colors n order to denote the cluster they belong to. also confrmed by the ranng nformaton, shown n the last column of Table II. In the case of two motons, our method s raned frst among the 15 compared algorthms. Let us notce that the method sparse subspace clusterng (SSC) [47] provdes a smlar average error. Ths s ndcated n the last column of the Table II, showng how many methods provde the same error n average. In the case of three motons, the performance of our method s raned n the second place behnd SSC, whose average error s 0.58%. Smlar n sprt statstcs are shown n Table III concernng the medan classfcaton error. In that case, our method accurately classfes more than half of the tme seres whch s also the case for 7 out of 15 methods for the problems nvolvng two motons. For the three motons problems, where the complexty ncreases, our algorthm s also raned at the frst place along wth other three methods, namely mult stage learnng (MSL) [48], local lnear manfold clusterng wth projecton to a space of dmenson 5 (LLMC 5) [49] and SSC [47]. 20

21 TABLE II STATISTICS ON THE AVERAGE OF CLASSIFICATION ERROR FOR THE TRAFFIC SUBSET OF THE HOPKINS 155 DATASET. Our approach Other approaches Our Ran In te Mn Max Mean Medan Two motons 0.02% 0.02% 5.74% 2.63% 2.23% 1/15 1 Three Motons 0.98% 0.58% 27.02% 9.53% 8.00% 2/15 - TABLE III STATISTICS ON THE MEDIAN OF CLASSIFICATION ERROR FOR THE TRAFFIC SUBSET OF THE HOPKINS 155 DATASET. Our approach Other approaches Our Ran In te Mn Max Mean Medan Two motons 0.00% 0.00% 1.55% 0.51% 0.21% 1/15 7 Three Motons 0.00% 0.00% 34.01% 7.22% 2.06% 1/15 3 D. Experments usng other ey pont descrptors In secton II we descrbed how to create trajectores from an mage sequence and as an example we used Harrs corners [35] n order to detect salent mage features and optcal flow [38] to assocate them between mages. Apart from Harrs corners, numerous nterest-pont detectors have been proposed n the computer vson lterature, such as the scale nvarant feature transform dfference of Gaussan ey ponts (SIFT DoG or SIFT for smplcty) [39], the maxmally stable extremal regons (MSER) [50], the Hessan matrx-based affne features [40] and the speeded-up robust features (SURF) [51]. They manly dffer to the level of the tradeoff between repeatablty and complexty [40], [52]. The SIFT features for example are hghly repeatable but requre a large computatonal cost. Ths s why SURF ey pont detectors have ganed ncreasng nterest, as they are faster (they are based on box flters and ntegral mages [53]). In order to study the consstency of the proposed method we have evaluated t n terms of the technque used for generatng trajectores. Table IV summarze the comparatve results on the seven smulated datasets usng Harrs corners, SIFT and SURF features. For every case we gve also the number of the generated trajectores. It may be observed that when the trajectores are computed usng Harrs corners a smaller MSE s obtaned whle trajectores computed by SIFT and SURF detectors have approxmately smlar errors but n any case hgher than Harrs corners. In terms of accuracy, all of the methods exhbt, n average, comparable rates whch ndcates a larger number of trajectores, provded for example by SURF, does not necessary ensure a better result. 21

22 TABLE IV THE PERFORMANCE OF THE DIFFERENT KEY POINT EXTRACTION METHODS IN TERMS OF CLASSIFICATION ACCURACY (ACC) AND MEAN SQUARED ERROR (MSE). Problem Harrs corners SIFT SURF #Trajectores MSE ACC #Trajectores MSE ACC #Trajectores MSE ACC Sm % % % Sm % % % Sm % % % Sm % % % Sm % % % Sm % % % Sm % % % V. CONCLUSIONS In ths study we have presented a compact methodology for objects tracng based on model-based clusterng trajectores of Harrs corners extracted from an mage sequence. Clusterng s acheved through an effcent sparse regresson mxture model that embodes effcent characterstcs n order to handle trajectores of varable length, and to be translated n measurement space. Experments have shown the abltes of our approach to automatcally detect the moton of objects wthout any human nteracton and also demonstrated ts robustness to occluson and feature msdetecton. The man advantage of our method wth respect to Kalman flter s that the former handles both tracng and moton segmentaton whle the latter only tracs the target. Also, Kalman flter should be provded wth the moton model whle the method proposed heren needs as nput only the number of the objects to be traced. Moreover, our method may be appled wthout the nowledge of the full trajectores by usng only tme seres data up to the current tme nstant f ths s mposed by the applcaton. Fnally, lnear regresson model can be appled n order to predct the next state [41]. Some drectons for future study nclude an alternatve strategy for ntalzng regresson mxture model parameters durng EM procedure especally n cases of occluson, as well as a mechansm to smultaneously estmate the number of objects K n the mage sequence. Also, our wllng s to study the performance of our method n other nterestng computer vson applcatons, such as human acton recognton [54] and fully 3D moton estmaton [30]. REFERENCES [1] D. Comancu, V. Ramesh, and P. Meer, Kernel-based object tracng, IEEE Transactons on Pattern Analyss and Machne Intellgence, vol. 25, no. 5, pp ,

23 [2] A. Ylmaz, O. Javed, and M. Shah, Object tracng: A survey, ACM Computng Surveys, vol. 38, no. 4, pp. 1 45, [3] E. Cuevas, D. Zaldvar, and R. Rojas, Kalman flter for vson tracng, Freer Unverstat Berln, Insttut fur Informat, Tech. Rep. B 05-12, [4] M. Yang, Y. Wu, and G. Hua, Context-aware vsual tracng, IEEE Transactons on Pattern Analyss and Machne Intellgence, vol. 31, no. 7, pp , [5] M. Isard and A. Blae, Condensaton - condtonal densty propagaton for vsual tracng, Internatonal Journal of Computer Vson, vol. 29, pp. 5 28, [6] J. Sh and C. Tomas, Good features to trac, n 1994 IEEE Conference on Computer Vson and Pattern Recognton (CVPR94), 1994, pp [7] Z. Wang, X. Yang, Y. Xu, and S. Yu, Camshft guded partcle flter for vsual tracng, Pattern Recognton Letters, vol. 30, no. 4, pp , [8] H. Zhoo, Y. Yuan, and C. Sh, Object tracng usng SIFT features and mean shft, Computer Vson and Image Understandng, vol. 113, no. 3, pp , [9] Q. Zhao and H. Tao, Dfferental Earth Mover s Dstance wth ts applcaton to vsual tracng, IEEE Transactons on Pattern Analyss and Machne Intellgence, vol. 32, no. 5, pp , [10] V. Karavasls, C. Nou, and A. Las, Vsual tracng usng the earth mover s dstance between Gaussan mxtures and Kalman flterng, Image and Vson Computng, vol. 29, no. 5, pp , [11] A. A. Argyros and M. I. A. Louras, Real-tme tracng of multple sn-colored object wth a possbly movng camera, n Proceedngs of the European Conference on Computer Vson (ECCV), 2004, pp [12] D. Beymer and K. Konolge, Real-tme tracng of multple people usng contnuous detecton, n Proceedngs of Internatonal Conference on Computer Vson (ICCV99), [13] A. Bugeau and P. Perez, Trac and cut: Smultaneous tracng and segmentaton of multple objects wth graph cuts, EURASIP Journal on Image and Vdeo Processng, no. ID:317278, [14] Y. Boyov, O. Vesler, and R. Zabh, Fast approxmate energy mnmzaton va graph cuts, IEEE Transactons on Pattern Analyss and Machne Intellgence, vol. 23, no. 11, pp , [15] A. Mansour, Regon tracng va level set pdes wthout moton computaton, IEEE Transactons on Pattern Analyss and Machne Intellgence, vol. 24, pp , [16] D. P. Muherjee, S. Member, N. Ray, S. T. Acton, and S. Member, Level set analyss for leuocyte detecton and tracng, IEEE Transactons on Image Processng, vol. 13, pp , [17] E. Horbert, K. Rematas, and B. Lebe, Level-set person segmentaton and tracng wth mult-regon appearance models and top-down shape nformaton, n Proceedngs of Internatonal Conference on Computer Vson (ICCV11), 2011, pp [18] C. Bbby and I. Red, Real-tme tracng of multple occludng objects usng level sets, n IEEE Conference on Computer Vson and Pattern Recognton (CVPR 10), 2010, pp [19] D. R. Magee, Tracng multple vehcles usng foreground, bacground and moton models, Image and Vson Computng, vol. 22, no. 2, pp , [20] S. Gammeter, A. Ess, T. Jaeggl, K. Schndler, B. Lebe,, and L. van Gool, Artculated multbody tracng under egomoton, n European Conference on Computer Vson (ECCV 08), ser. LNCS. Sprnger, 2008, pp [21] B. Han, S.-W. Joo, and L. S. Davs, Probablstc fuson tracng usng mxture ernel-based Bayesan flterng, n Proceedngs of Internatonal Conference on Computer Vson (ICCV07), 2007, pp

24 [22] F. Moreno-Noguer, A. Sanfelu, and D. Samaras, Dependent multple cue ntegraton for robust tracng, IEEE Transactons on Pattern Analyss and Machne Intellgence, vol. 30, no. 4, pp , [23] M. Blac and P. Anandan, The robust estmaton of multple motons: parametrc and pecewse-smooth flow felds, Computer Vson and Image Understandng, vol. 63, no. 1, pp , [24] K. Y. Wong and M. E. Spetsas, Moton segmentaton by EM clusterng of good features, n Proceedngs of the 2004 Conference on Computer Vson and Pattern Recognton Worshop (CVPRW 04) Volume 11, ser. CVPRW 04. IEEE Computer Socety, 2004, pp [25] S. J. Pundl and S. T. Brchfeld, Real-tme moton segmentaton of sparse feature ponts at any speed, IEEE Transactons on Systems, Man, and Cybernetcs, vol. 38, pp , [26] D. Buzan, S. Sclaroff, and G. Kollos, Extracton and clusterng of moton trajectores n vdeo, n Internatonal Conference on Pattern Recognton, 2004, pp [27] J. Alon, S. Sclaroff, G. Kollos, and V. Pavlovc, Dscoverng clusters n moton tme-seres data, n Proceedngs of the IEEE Computer Socety Conference on Computer Vson and Pattern Recognton, 2003, pp [28] L. Zappella, X. Llado, E. Provenz, and J. Salv, Enhanced local subspace affnty for feature-based moton segmentaton, Pattern Recognton, vol. 44, pp , [29] M. Zeppelzauer, M. Zahareva, D. Mtrovc, and C. Breteneder, A novel trajectory clusterng approach for moton segmentaton, n Advances n Multmeda Modelng, 2010, vol. 5916, pp [30] F. Lauer and C. Schnorr, Spectral clusterng of lnear subspaces for moton segmentaton, n Proc. of the 12th IEEE Int. Conf. on Computer Vson, ser. ICCV 09, [31] C. Julà, A. Sappa, F. Lumbreras, J. Serrat, and A. López, Moton segmentaton from feature trajectores wth mssng data, n Proceedngs of the 3rd Iberan conference on Pattern Recognton and Image Analyss, Part I, ser. IbPRIA 07, 2007, pp [32] J. Yan and M. Pollefeys, A general framewor for moton segmentaton: Independent, artculated, rgd, non-rgd, degenerate and non-degenerate, n Proceedngs of the European Conference on Computer Vson, ser. ECCV 06, [33] J.-H. Km and L. Agapto, Moton segmentaton usng the Hadamard product and spectral clusterng, n Proceedngs of the 2009 nternatonal conference on Moton and vdeo computng, ser. WMVC 09. IEEE Computer Socety, 2009, pp [34] O. Masoud and N. P. Papanolopoulos, A novel method for tracng and countng pedestrans n real-tme usng a sngle camera, IEEE Transactons on Vehcular Technology, vol. 50, no. 5, pp , [35] C. Harrs and M. Stephens, A combned corner and edge detecton, n Proceedngs of The Fourth Alvey Vson Conference, 1988, pp [36] V. Karavasls, K. Bleas, and C. Nou, Moton segmentaton by model-based clusterng of ncomplete trajectores, n European Conference on Machne Learnng and Prncples and Practce of Knowledge Dscovery n Databases (ECML 11-PKDD 11), Athens, Greece, 5-9 September [37] R. Tron and R. Vdal, A benchmar for the comparson of 3d moton segmentaton algorthms, n IEEE Conference on Computer Vson and Pattern Recognton (CVPR 07), vol. 1, [38] B. Lucas and T. Kanade, An teratve mage regstraton technque wth an applcaton to stereo vson, n Proceedngs of the 7th Internatonal Jont Conference on Artfcal Intellgence (IJCAI 81), 1981, pp [39] D. G. Lowe, Dstnctve mage features from Scale-Invarant eypont, Internatonal Journal of Computer Vson, vol. 60, no. 2, pp , [40] K. Molajczy, T. Tuytelaars, C. Schmd, A. Zsserman, J. Matas, F. Schaffaltzy, T. Kadr, and L. V. Gool, A comparson of affne regon detectors, Internatonal Journal of Computer Vson, vol. 65, no. 1/2, pp ,

25 [41] C. M. Bshop, Pattern Recognton and Machne Learnng. Sprnger, [42] S. Gaffney, Probablstc curve-algned clusterng and predcton wth regresson mxture models, Ph.D. dssertaton, Department of Computer Scence, Unversty of Calforna, Irvne, [43] M. Tppng, Sparse bayesan learnng and the relevance vector machne, Journal of Machne Learnng Research, vol. 1, pp , [44] A. P. Dempster, N. M. Lard, and D. B. Rubn, Maxmum lelhood from ncomplete data va the EM algorthm, J. Roy. Statst. Soc. B, vol. 39, pp. 1 38, [45] K. Bleas, C. Nou, N. Galatsanos, and N. Tseos, A regresson mxture model wth spatal constrants for clusterng spatotemporal data, Intern. Journal on Artfcal Intellgence Tools, vol. 17, no. 5, pp , [46] G. R. Brads, Computer Vson Face Tracng For Use n a Perceptual User Interface, Intel Technology Journal, no. Q2, [47] E. Elhamfar and R. Vdal, Sparse subspace clusterng, n IEEE Conference on Computer Vson and Pattern Recognton (CVPR 09), 2009, pp [48] Y. Sugaya and K. Kanatan, Mult-stage optmzaton for mult-body moton segmentaton, IEICE Transactons on Informaton and Systemsg, vol. E87-D, no. 7, pp , [49] A. Goh and R. Vdal, Segmentng motons of dfferent types by unsupervsed manfold clusterng, n IEEE Conference on Computer Vson and Pattern Recognton (CVPR 07), [50] J. Matas, O. Chum, M. Urban, and T. Pajdla, Robust wde baselne stereo from maxmally stable extremal regons, n In Brtsh Machne Vson Conference (BMVC 02), vol. 1, 2002, pp [51] H. Bay, T. Tuytelaars, and L. V. Gool, SURF: Speeded up robust features, n Proceedngs of the European Conference on Computer Vson (ECCV 06), 2006, pp [52] V. Chandrasehar, D. Chen, A. Ln, G. Taacs, S. Tsa, N.-M. Cheung, Y. Rezn, R. Grzeszczu, and B. Grod, Comparson of local feature descrptors for moble vsual search, IEEE Sgnal Processng Magazne, vol. 28, no. 4, pp , [53] P. Vola and M. Jones, Robust real-tme face detecton, Internatonal Journal of Computer Vson, vol. 57, pp , [54] A. Oonomopoulos, I. Patras, M. Pantc, and N. Paragos, Trajectory-based representaton of human actons, n Proceedngs of the ICMI 2006 and IJCAI 2007 Internatonal Conference on Artfcal Intellgence for Human Computng, vol. 4451, 2007, pp