Coherent Filtering: Detecting Coherent Motions from Crowd Clutters

Transcription

1 Coherent Flterng: Detectng Coherent Motons from Crowd Clutters Bole Zhou, Xaoou Tang,3, and Xaogang Wang 2,3 Department of Informaton Engneerng, The Chnese Unversty of Hong Kong 2 Department of Electronc Engneerng, The Chnese Unversty of Hong Kong 3 Shenzhen Insttutes of Advanced Technology, Chnese Academy of Scences Abstract. Coherent motons, whch descrbe the collectve movements of ndvduals n crowd, wdely exst n physcal and bologcal systems. Understandng ther underlyng prors and detectng varous coherent moton patterns from background clutters have both scentfc values and a wde range of practcal applcatons, especally for crowd moton analyss. In ths paper, we propose and study a pror of coherent moton called Coherent Neghbor Invarance, whch characterzes the local spatotemporal relatonshps of ndvduals n coherent moton. Based on the coherent neghbor nvarance, a general technque of detectng coherent moton patterns from nosy tme-seres data called Coherent Flterng s proposed. It can be effectvely appled to data wth dfferent dstrbutons at d- fferent scales n varous real-world problems, where the envronments could be sparse or extremely crowded wth heavy nose. Expermental evaluaton and comparson on synthetc and real data show the exstence of Coherence Neghbor Invarance and the effectveness of our Coherent Flterng. Introducton Coherent moton s a unversal phenomenon n nature and wdely exsts n many physcal and bologcal systems. For example, tornadoes, storms, and atmospherc crculaton are all caused by the coherent movements of physcal partcles n the atmosphere. Meanwhle, the collectve behavors of organsms such as swarmng ants and schoolng fshes have long captured the nterests of socal and natural scentsts [, 2]. Detectng coherent motons and understandng ther underlyng prncples are related to many mportant scentfc research topcs such as self-organzaton of bologcal systems [3]. There s also a wde range of practcal applcatons. For example, n vdeo survellance, detectng coherent moton patterns of pedestran groups n crowd has mportant applcatons to object countng [4, 5], crowd trackng [6], and crowd management [7]. Furthermore, clusters of coherent motons provde a md-level representaton of crowd dynamcs, and could be used for hgh-level semantc analyss such as scene understandng and crowd actvty recognton [8 ]. Generally speakng, coherent moton detecton can be formulated as fndng clusters of dynamc partcles wth coherent moton from ther tme-seres observaton and removng background nose as outlers. Under dfferent scene context, the detected coherent motons may be nterpreted as dfferent semantc behavors. As shown n Fgure Source codes and vdeos are avalable from

2 2 Bole Zhou, Xaoou Tang, and Xaogang Wang 9, movng keyponts tracked n the scenes exhbt a wde varety of coherent moton patterns, correspondng to ndvdual and group movements, traffc mode, crowd flow etc. These examples show that detectng coherent motons from nosy observatons s of great mportance to crowd behavor analyss and scene understandng. The goal of ths work s to explore the underlyng pror n the dynamcs of coherent motons and to leverage t for coherent moton detecton. We propose a pror called Coherent Neghbor Invarance, whch exsts n the local neghborhoods of ndvduals n coherent motons, and show that t well dstngushes coherent and ncoherent motons. Then we develop a general coherent moton detecton technque called Coherent Flterng based on such a pror. Experments show that t can work robustly n onlne mode for varous applcatons.. Related Works Crowd moton analyss has been an actve research topc n computer vson. It s closely related to many low-level and hgh-level computer vson problems n crowded or cluttered envronments. For example, Rabaud et al. [4] and Brostow et al. [5] proposed approaches to detect ndependent motons n order to count movng objects. Ln et al. [2] used Le algebra of affne transform to learn the global moton patterns of crowds. Al et al. [6] used floor felds from flud mechancs for the segmentaton of crowd flows. Hu et al. [3] clustered the sngle-frame optcal flows to learn the moton patterns. Brox et al. extended spectral clusterng to group long-term dense trajectores for the segmentaton of movng objects n vdeo. Meanwhle, the hgh-level semantc analyss n crowded scenes focuses on modelng scene structures and recognzng crowd behavors. Wang et al. [8, 4] and Zhou et al. [] used herarchal topc models to learn the models of semantc regons from the co-occurrence of optcal flow and tracks. Zhou et al. [9] proposed a mxture model of dynamc pedestran-agents to learn the collectve crowd behavor patterns n crowded scenes. In 3D moton segmentaton [5], under the assumpton of affne transform there are several subspace approaches proposed, such as Generalzed Prncpal Component Analyss (GPCA) [6] and RANSAC [5]. Although these problems are closely related to coherent motons, they were studed separately and had dfferent lmted assumptons and solutons n the past lterature. For example, the approaches n [4, 5] are specfcally desgned for object countng and had assumptons on the shape and dstrbuton of coherent motons. The models n [2] and [6] assume the geometrc transformaton of moton. The clusterng methods n [3, 7] are easly affected by noses. GPCA [6] and RANSAC [8] consder the moton trajectores les on moton subspace and are projected under the affne camera model. Therefore, these approaches cannot be well generalzed to other problems related to coherent motons. We develop a general coherent moton technque whch can be well appled to the varous problems dscussed above. It acheves comparable results or even outperforms the state-of-the-arts n dfferent applcaton domans, especally when the background nose s heavy. 2 The Pror: Coherent Neghbor Invarance Although coherent motons are the macroscopc observatons of collectve movements of ndvduals, recent studes [2, 9] show that t actually can be characterzed by the

3 Coherent Flterng:Detectng Coherent Motons from Crowd Clutters 3 tme t tme t+ tme t+2 center nvarant neghbor removed added unrelated Fg.. Illustraton of coherent neghbor nvarance. The green dots are the nvarant K nearest neghbors of the central black dot over tme (here K = 7). The nvarant neghbors have a hgher probablty to be the dots movng coherently wth the central dot, snce ther local spatotemporal relatonshps and velocty correlatons wth the central dot are nclned to reman nvarant over tme. The red and blue dots change ther neghborshp over tme (removed or added), so that they have a small probablty to move coherently wth the central dot. nteractons among ndvduals wthn local neghborhoods. Inspred by these observatons and results, we propose a pror underlyng the dynamcs of coherent moton as Coherent Neghbor Invarance. There are two key propertes of Coherent Neghbor Invarance, whch dstngush coherent motons from random motons: Invarance of spatotemporal relatonshps: the neghborshp of ndvduals wth coherent motons tends to reman nvarant over tme. Invarance of velocty correlatons: the velocty correlatons of neghborng ndvduals wth coherent motons reman hgh when beng averaged over tme. On the contrary, ncoherently movng ndvduals do not have such propertes because of the mutual ndependence of ther movements. Coherent neghbor nvarance exsts n the dynamcs of K nearest neghbors of ndvduals for consecutve tme. It reflects the self-organzaton of coherent motons n neghborshp, and explans the formaton of global coherent moton patterns from local coordnatons of ndvduals. An llustraton of the pror s shown n Fgure. To quanttatvely analyze ths pror, we frst defne two measurements of Coherent Neghbor Invarance: the coherent neghbor nvarance of spatotemporal relatonshps (n Secton 2.2), and the coherent neghbor nvarance of velocty correlatons (n Secton 2.3). From the followng experments, we show that ths pror not only helps reveal the mechansm of coherent motons, but also can be effectvely leveraged to separate varous coherent moton patterns from background nose usng a technque called Coherent Flterng. 2. Random Dot Knematogram We take Random Dot Knematogram (RDK) as an example to analyze the coherent neghbor nvarance, because t s easy to understand and can be well generalzed. RDK s a classc psychophyscal stmulus, and s often used for nvestgatng coherent moton percepton of vsual systems [2]. The stmulus conssts of a completely random array of thousands of tny dots that move ether coherently or randomly. An llustraton s shown n Fgure 2A. Incoherently movng dots(referred as ncoherent dots) are randomly placed over the whole scene and serve as background nose. In the central

4 4 Bole Zhou, Xaoou Tang, and Xaogang Wang The cardnalty of a set. The ntersecton of two sets. F F = {F,..., F N }. The set of coherent dots. B The set of ncoherent dots. N t N t = { t,..., K t }. The set of the K-NNs of dot at tme t. M M = N t N t+... N t+d.the dth order nvarant neghbor set of. C C = M F. The coherent nvarant neghbor set of g k Averaged velocty correlaton between and k from t to t + d. R R = {(, k ) g k > λ, k A}. The set of parwse connectons. S S = {s I}. The set of the cluster ndex s for each dot. Table. Notatons used n the paper. P.8.6 Invarant Neghbor Rato mxed coherent ncoherent w Coherent Invarant Neghbor Rato.8.6 mxed cohrent ncoherent A) Coherence Level=.3 B) tme nterval d C) 5 5 tme nterval d Fg. 2. A) Illustraton of random dot knematogram. Here the number of coherent moton pattern N =. The cyan arrows ndcate the random movng drectons of some nosy dots wth ncoherent motons. The green arrows ndcate the drecton of coherent moton. B) Averaged nvarant neghbor ratos P wth tme nterval d. C) Averaged coherent nvarant neghbor ratos W wth tme nterval d. All these measurements are computed and averaged for coherent dots, ncoherent dots and all the dots respectvely for comparson. rectangular area, a group of coherently movng dots(referred as coherent dots) are also randomly placed. The proporton of coherent dots to all the dots n the rectangular area (mxng of coherent dots and ncoherent dots) s called the coherence level. In psychophyscal study, human subjects are requred to dentfy coherent motons of dots from the background nose. The coherence level determnes the dffculty level of the dentfcaton task. Formally, we denote I as the set of all the dots (mxed dots) n the central rectangular area, F as the set of coherent dots, and B as the set of ncoherent dots. There could be N dfferent coherent moton patterns whch dvde F nto subsets {F,..., F N }. Thus the problem of detectng coherent motons from nosy observatons of dot movements s formulated as estmatng the separaton I = {F, B}, and the sub-separaton F = {F,..., F N }. In Sectons 2.2 and 2.3, we wll analyze two coherent neghbor nvarance measurements and study ther dynamc behavors. Table shows the notatons used n the paper. 2.2 Invarance of Spatotemporal Relatonshps Ths subsecton analyzes the nvarance of spatotemporal relatonshps n coherent motons. We frst defne some related concepts. At tme t, the set N t contans the K n- earest neghbors of dot under Eucldean dstance. It evolves nto N t+ at tme t + 2. We denote M t as the st order nvarant neghbor set, whch contans the nvarant neghbors among the K nearest neghbors of dot from tme t to t+. Smlarly, M s denoted as the d th order nvarant neghbor set whch contans the nvarant neghbors 2 The correspondence of dots over tme s assumed.

5 Coherent Flterng:Detectng Coherent Motons from Crowd Clutters 5 from tme t to t + d. We denote C as the ntersecton of M and F, so that t only contans the nvarant neghbors wth coherent moton. It s called the coherent nvarant neghbor set of dot at d th order. Two ratos are defned. The frst s the nvarant neghbor rato P, measurng the proporton of nvarant neghbors among the K nearest neghbors durng tme t to t+d. The second s the coherent nvarant neghbor rato W, measurng the proporton of coherent dots among the nvarant neghbors durng tme t to t + d. Specfcally, P = M K, W = C M, where P and W [, ]. Obvously P and W wll change wth the value of d and they descrbe the temporal behavors of dots wth coherent and ncoherent motons. For an ncoherent dot, snce most of the dots n ts neghborhood move ndependently wth dot, ts K nearest neghbors would vary greatly over tme. Thus P s expected to decrease quckly when d ncrease, and gradually to. On the contrary, for a coherent dot, dots n ts neghborhood, whch move coherently wth, would tend to reman n the neghborhood durng the whole tme nterval d because of ther consstent movements wth, whle other ncoherent dots n ts neghborhood wll easly move out of the neghborshp. Thus P s expected to decrease slower than that of ncoherent dots and then remans as a constant when d ncreases further. On the other hand, W measures the proporton of coherent dots n the nvarant neghbor set M. Obvously only the dots whch move coherently wth dot tend to reman n the neghborhood of from tme t to t + d. For an ncoherent dot, because all dots n M are movng ndependently wth dot, W s expected to be low over tme. For a coherent dot, snce majorty of the remanng dots n M move coherently wth as d ncreases, W s expected to ncrease wth d. Snce the groundtruth of dots as F or B are known n RDK, we can compute and analyze the two ratos for coherent dots, ncoherent dots, and mxed dots respectvely. We set the coherence level as.3, whch means there are 3% dots ( 8) movng coherently n the central rectangular area of Fgure 2A. As shown n Fgures 2B and 2C, the expermental results of the two ratos n RDK verfy our analyss. We can see that as d ncreases the averaged nvarant neghbor ratos P for coherent dots and ncoherent dots are clearly separated. Meanwhle, the averaged coherent nvarant neghbor rato W for coherent dots ncreases almost to, and that for ncoherent dots decreases to. Our analyss and llustratve results n RDK show the nvarant neghbor rato and the coherent nvarant neghbor rato have good dscrmnablty for coherent and ncoherent motons. We call ths property of coherent moton as coherent neghbor nvarance of spatotemporal relatonshps. 2.3 Invarance of Velocty Correlatons The other property of coherent moton s the nvarance of velocty correlatons between neghborng dots. Suppose that dot k belongs to the nvarant neghbor set of dot. Ther velocty correlaton averaged from tme t to t + d s g k = d + t+d τ=t v τ v k τ v τ v k τ,

6 6 Bole Zhou, Xaoou Tang, and Xaogang Wang where v τ s the velocty of at tme τ. If dot k moves ncoherently wth dot, g k would be low as d ncreases. Otherwse, g k remans hgh. Therefore, the velocty correlatons of coherently movng dots and ncoherently movng dots n local regons can be well separated as d ncreases. Fgure 3 shows the hstograms of g k generated from the nvarant neghbor set n RDK, wth d =,, 3, 5, respectvely. The expermental results verfy our analyss above. We can see that as d ncreases, the hstogram gradually separates nto two modes: one near and the other near. Ths property of coherent motons s called coherent neghbor nvarance of velocty correlatons. Because of ths property, t s smple to remove the ncoherent dots k from the nvarant neghbor set M : settng a threshold λ on the value of g k and then removng k from M f g k < λ. After thresholdng, we can create a set R contanng the thresholded parwse connectons, n whch (, k ) are connected f k stll remans n M. Then coherent motons can be easly detected accordng to R, usng the algorthm proposed n Secton x d= d= d=3 d=5 d= Fg. 3. Hstograms of g k generated from the nvarant neghbor set n RDK, wth d =,, 3, 5, respectvely. As d ncreases, g k of coherently movng dots and ncoherently movng dots are well separated. The bar near s the hstogram of g k of coherently movng dots, and the hump near s the hstogram of g k of ncoherently movng dots. 3 Coherent Flterng Usng Coherent Neghbor Invarance Based on Coherent Neghbor Invarance, a clusterng technque Coherent Flterng s proposed for coherent moton detecton from tme-seres data. Coherent Flterng conssts of two algorthms. The frst s to detect coherent moton patterns at one tme, the second s to assocate detected coherent moton and update exstng coherent moton over tme. Algorthms are lsted n Table 2 and Table 3 respectvely. Coherent Flterng has some mportant merts. ) It stems from the coherent neghbor nvarance, whch s a pror wdely observed n coherent motons. Therefore t s a general technque for clusterng tme-seres data and detectng coherent motons n varous real-world problems, such as object countng, group movement detecton, traffc mode recognton, and crowd flow segmentaton. 2) It only reles on local spatotemporal relatonshps and velocty correlatons wthout any assumpton on the global shape of coherent moton patterns and the dstrbuton of background nose. Therefore t can be robustly appled to data at dfferent scales and dstrbutons wthout substantal change. 3) In practcal applcatons, t mght be dffcult to obtan the correspondence of keyponts over a long tme, especally n crowded envronments. Experments show that our algorthm only requres correspondence over a short perod (normally 4 or 5 frames). Ths means that t can work robustly n crowded scenes and n an onlne mode. 4) The cluster number N s automatcally decded from data wthout knowng as a pror.

7 Coherent Flterng:Detectng Coherent Motons from Crowd Clutters 7 3. Algorthm for detectng coherent motons In Algorthm CoheFlterDet we frst obtan the nvarant neghbor set M by examnng the neghborshp n Nτ from t to t + d for each dot I. Accordng to coherent neghbor nvarance of spatotemporal relatonshps, most dots n M should be coherent dots. However, t does not guarantee that all the dots n A are coherent dots especally when d s small. Then, accordng to the coherent neghbor nvarance of velocty correlatons, we set a threshold λ on the averaged velocty correlatons to remove ncoherent dots and obtan the parwse connecton set R. Fnally a connectvty graph s bult, where nodes are dots and edges are defned by connecton relatonshps n R. Wth ths graph, ncoherent dots B are dentfed as solated nodes and coherent moton clusters {F,..., F N } are dentfed as the connected components of the graph. FUNCTION (F,..., F N ) = CoheFlterDet(I) :for τ = t to t + d 2: search the K nearest neghbor set as Nτ for each dot I 3:for each dot I 4: search the nvarant neghbor set as M 5: for each k M 6: compute the averaged velocty correlatons g k 7: nclude (, k ) n R f g k > λ. 8:Buld a graph from R. Remove ncoherently movng ndvduals as the solated nodes and dentfy coherent moton {F,..., F N } as the connected components of the graph. Table 2. Algorthm CoheFlterDet for detectng coherent moton patterns. 3.2 Algorthm for assocatng contnuous coherent moton Coherent moton clusters wll contnue and evolve, and new cluster of coherent moton wll emerge over tme. Based on the temporal overlaps of trajectores we develop another algorthm CoheFlterAssoc to assocate and update the clusters of coherent moton over consecutve frames. To assocate the clusters of coherent moton over tme, we defne a varable s as the cluster ndex for each trajectory. As llustrated n Fgure 4A, the algorthm wll update the cluster ndce of trajectores by majorty votng and keep on detectng new coherent moton cluster over tme. The detal of algorthm s lsted n Table 3. FUNCTION (S t+ ) = CoheFlterAssoc(S t, I t+ ) : (F,..., F Nt+ ) = CoheFlterDet(I t+ ) /*detect new coherent moton at frame t+*/ 2:for each I t I t+ 3: s t+ = s t /*frstly assume there s no cluster ndex changng for dot */ 4:M=max(S t ) /*maxmum cluster ndex value n S t */ 5:for each F nt+ 6: S=mode(H), where H = {s t+ F nt+ }/*get the most frequent value n H*/ 7: f S== then S=M+ /*add a new cluster ndex*/ 8: for each F nt+ let s t+ =S 9:dentfy dot I t+ as foreground and ts cluster ndex as s t+ f s t+ > Table 3. Algorthm CoheFlterAssoc for assocatng coherent motons.

8 8 Bole Zhou, Xaoou Tang, and Xaogang Wang ts A) ts 2 wth assocatng s = 2 te s = 2 2 te 2 s = 2 3 ts 3 te 3 tme B) Frame 26 Frame 4 wthout assocatng Frame 26 Frame 4 Fg. 4. Illustraton of assocatng contnuous coherent motons. A) There are temporal overlaps between trajectory and 2, trajectory 2 and 3. If trajectory and 2 are detected nto one coherent moton cluster at one tme, and trajectory 2 and 3 are detected nto one coherent moton cluster at next tme, the ndex s = 2 of trajectory wll be transferred to the other two trajectores. Red crcles ndcate trajectores are detected nto one coherent moton cluster, ts and te denote the startng and endng tme of trajectory. B) Two representatve frames of coherent moton detecton result wth assocatng and wthout assocatng respectvely. Dots n the same color belong to one coherent moton cluster over tme and space. 4 Expermental Results In ths secton, we wll evaluate the robustness and effectveness of Coherent Flterng on complex synthetc data, real 3D moton segmentaton database, and crowd vdeos. On the synthetc data, we test the technque by detectng coherent moton patterns wth dfferent dynamcs from hgh-densty Brownan moton nose. Then we evaluate the technque on the 3D affne moton segmentaton Hopkns55 database [5], and compare t to several baselne methods on the database n the presence of outlers. Lastly we test Coherent Flterng on vdeos by detectng coherent moton patterns n real scenes wth a varety of scales and crowdedness. 4. Coherent Moton n Synthetc Data Fgure 5A shows the two synthetc datasets for evaluaton. The coherent moton patterns emergng n the datasets vary greatly n scales, shapes, and dynamcs. For the 2D dataset, the parametrc forms of parabola, crcle, lne, and dsk are used to generate four 2D coherent moton patterns. For the 3D dataset, the parametrc forms of helx and spral surface are used to generate two 3D coherent moton patterns. Fgure 5B llustrates the traces of the synthetc coherent moton patterns. Intal postons of the coherent dots are randomly sampled from Gaussan dstrbuton along the traces of the coherent moton patterns. As the detecton results shown n Fgure 5C, our technque detects well these coherent moton patterns from rather nosy data. The good performance of detectng varous coherent moton patterns shows the robustness and generalzaton ablty of our technque. Fgures 5D and 5E show that the nvarant neghbor ratos and the coherent nvarant neghbor ratos for the two datasets. We can see they have good dscrmnablty between coherent dots and ncoherent nosy dots. It also verfes the exstence of coherent neghbor nvarance n the synthetc data. Three representatve clusterng methods,.e., Normalzed Cuts (Ncuts) [2], Kmeans and Mean-shft [22], are selected for comparson. Ncuts, K-means, and Meanshft are often extended for tme-seres data clusterng [23] and trajectory clusterng [24]. By conventon, we treat the trajectory of each dot from tme t to t + d as a feature vector (xt, yt, vx,t, vy,t,..., xt+d, yt+d, vx,t+d, vy,t+d ), where (xτ, yτ ) and (vx,τ, vy,τ ) are the locaton and velocty of dot at tme τ. Dots are clustered based

9 Coherent Flterng:Detectng Coherent Motons from Crowd Clutters A) 3 4 tme nterval d mxed coherent ncoherent mxed coherent ncoherent B) Coherent Invarant Neghbor Rato tme nterval d Invarant Neghbor Rato mxed coherent ncoherent mxed coherent ncoherent.6 2 Coherent Invarant Neghbor Rato Invarant Neghbor Rato C) 2 D) tme nterval d E) tme nterval d Fg. 5. Coherent moton detecton on synthetc 2D and 3D datasets. A) The shapes and the numbers of coherently movng dots (colors ndcate dfferent coherent moton patterns) and nosy dots (n blue). B) The traces of each coherent moton patterns. C) The detected coherent moton patterns by Coherent Flterng. D) Invarant neghbor ratos for dfferent types of dots over tme. E) Coherent nvarant neghbor ratos for dfferent types of dots over tme. The algorthm parameters are K = 5, d = 4 and λ =.6. NMI 3D 2D Ncuts K-means Mean-shft Ours Ours Ncuts K-means Mean-shft Fg. 6. The qualtatve and quanttatve results of the four methods for comparson. Colors ndcate dfferent clusters. NMI s used to quanttatvely evaluate the clusterng results. Our technque acheves the best performance. on the feature vectors. For Ncuts and K-means, the cluster numbers are chosen as 5 and 3 for the two datasets. Mean-shft automatcally determnes the cluster number. The clusterng results are shown n Fgure 6. The quanttatve result s measured by the Normalzed Mutual Informaton (NMI) [25]. Larger NMI ndcates better clusterng performance. The tme nterval d s set as 4, whch means that the trajectory of each dot has sx samples as the nputs of all the algorthms. Our algorthm acheves the best performance. 4.2 A) 3D Moton Segmentaton B) Fg. 7. A) Representatve sequences from Hopkns55 database and the segmentaton results of Coherent Flterng. B) The frst mage s one representatve frame wth groundtruth. There are 2 clusters, one cluster s on the movng object, the other cluster s on the statc background objects, whch results from movng camera. The second mage s the segmentaton result of our method. Snce Coherent Flterng has no assumpton on the number of clusters, t tends to segment some dspersed background cluster nto several clusters of separated objects. Yellow + dots are the detected noses.

10 Bole Zhou, Xaoou Tang, and Xaogang Wang There are many potental applcatons of our Coherent Flterng algorthm n realworld problems. We test t on 3D affne moton segmentaton on the Hopkns55 Database [5]. We choose ths applcaton because ts ground truth s avalable and t can provde quanttatve evaluaton of our technque. Ths database conssts of 2 vdeo sequences wth two motons and 35 vdeo sequences wth three motons. Trajectores of feature ponts on each moton are clustered as ground truth and nput for each testng method. The sequences can be categorzed nto three man groups, checkboard, traffc, and artculated, whch contan a varety of motons, such as degenerate and non-degenerate motons, ndependent motons, and motons from camera movement. Fgure 7A shows the representatve frames of sequences n the database. For comparson, typcal subspace moton segmentaton methods, Generalzed Prncpal Component Analyss (GP- CA) [6] and RANSAC [5] are taken as the baselne. These approaches utlze the fact that object movements n ths database are rgd and under affne transform. However, our method does not need the assumpton snce t s used to detect motons n more general form. Fgure 8A shows the segmentaton performance n terms of Normalzed Mutual Informaton and average computaton tme. 3 Coherent Flterng acheves comparatve performance to these subspace segmentaton methods, though t s not specfcally desgned for 3D affne moton segmentaton. The major error of our method comes from that our method has no assumpton on the number of moton clusters n the sequence, so that t would tend to segment some dspersed background cluster of ground truth nto several small clusters nstead of one background cluster, as shown n Fgure 7B. Ths problem s hard to be solved usng our method alone wthout assumng the affne transform snce these small clusters are far away to each other n space. Strctly speakng, they do not belong to a sngle coherent moton, but the database ground truth treats them as one pattern. To further evaluate the robustness of dfferent methods n the NMI GPCA RANSAC CF Checkboard Traffc Artculated All sequences NMI Average Tme.2s 7.58s 32ms A) B) CF GPCA RANSAC outler percentage % Fg. 8. A) NMI of dfferent methods on the Hopkns55 Database, along wth the average computaton tme. Though Coherent Flterng s not specfcally desgned for 3D moton segmentaton, t acheves comparatve performance to other subspace segmentaton methods wth a better computatonal effcency. B) NMI of dfferent methods as the functon of the outler percentage(from % to 4%). presence of outlers, we add outler trajectores from % to 4% nto the groundtruth 3 Codes of comparson methods are downloaded from authors webstes. Average computaton tme s tested on a computer wth 3GHz Core Quad CPU and 4GB RAM wth Matlab mplementaton. Note that snce the number of clusters detected by our method may not accurately correspond to the ground truth cluster number, NMI s a more sutable measurement than the msclassfcaton rate reported n [5].

11 Coherent Flterng:Detectng Coherent Motons from Crowd Clutters trajectores of every 55 sequences, then test the performances of these methods. Outler trajectores are generated by randomly choosng ntal ponts n the frst frame and then extendng the trajectores wth random walk. Fgure 8B shows the NMI of these methods wth dfferent outler percentages. Coherent Flterng works more robust wth heavy nose. 4.3 Coherent Moton Detecton n Crowd Experments are conducted on 8 vdeo clps wth coherent moton emergng n dfferent scales and dstrbutons. The frst vdeo clp s captured on the USC campus [4]. The scene s relatvely sparse and the scales of pedestrans are hgh. The second one s from [26] wth hgher crowd densty, and t contans both ndvdual and groups of pedestrans. In the thrd one, many mddle-szed people are comng n and out. The fourth s acqured from a far-vew ralway staton. The resolutons of pedestran are very small. And the last four clps are selected from Getty Image webste, contanng hgh-densty crowd runnng and traversng. Some of them have been used n [7]. In ntalzaton, KLT keypont tracker [27] s used to automatcally detect keyponts and extract ther trajectores as the observaton I for the nput of algorthm. Trackng termnates when severe clutters or occlusons arse, and new tracks wll be ntalzed later. For all vdeos, the parameters of Coherent Flterng are λ =.6, d = 4, and K =. We further dscuss the nfluence of parameters on the clusterng results n Secton 4.4. Fgure 9 shows the representatve frames and coherent moton clusters detected by Coherent Flterng n dfferent scenes. Coherent Flterng detects well the underlyng coherent moton patterns from the nosy tme-seres observatons of detected keyponts. From these results, we can see that the detected coherent moton clusters correspond to a varety of semantc behavors and actvtes, whch are of great mportance to further vdeo analyss, survellance and scene understandng. However, t s dffcult to quanttatvely evaluate these behavors from detected coherent motons snce t s hard to obtan the ground truth. To quanttatvely evaluate the detecton performance, we conduct the people countng experment on the scene shown n Fgure, and compare wth trajectory clusterngbased people countng method ALDENTE [4] and Bayesan Detecton countng method BayDet [5]. The expermental settng s the same as [5]. We count the number of pedestran detected at each key frame (every key frame per 5 frames), and use the average people countng error as the evaluaton crtera for the three methods. Meanwhle, snce at each frame the detected clusters can be ether True Postve or False Postve, and the False Postve also can be counted as the False Negatve (the undetected one) n the number of pedestrans detected n each key frame. That makes people countng evaluaton crtera not so accurate. Thus we further evaluate the Detecton Rate (DR) and False Alarm Rate (F AR) as DR = T P, F AR = F P T P, + F P GT where T P, F P, and GT are the number of True Postve, False Postve, and groundtruth at frame. Fgure A shows the numbers of pedestran detected by the three methods and the groundtruth at each key frames, and Fgure B shows the average people

12 2 Bole Zhou, Xaoou Tang, and Xaogang Wang A) B) C) D) E) F) G) H) Fg. 9. Representatve frames and the coherent moton clusters detected by Coherent Flterng. Movng keyponts from vdeos exhbt a varety of coherent moton patterns at dfferent scales n dfferent scene context. A) The majorty of detected coherent moton clusters result from the ndependent walkng pedestrans, snce the scale of pedestrans s rather bg. B) Coherent motons from both ndvdual pedestrans and groups of pedestrans walkng together are detected. C) Dfferent queues of walkng-n-and-out people are detected. D) From the far vew to the ralway staton, there are merely one or two keyponts tracked on each pedestran n the scene. Thus the emergent coherent motons of keyponts represent the clusters of nearby pedestrans headng n the same drectons, and they are related to dfferent traffc modes. E) Two major lanes of vehcles on the road are detected, among them several small clusters representng jaywalkers are also detected because of ther dfference n moton drectons to the major lanes. F) Two groups of pedestrans are detected to pass each other on the crosswalk. G) There s one crcular coherent moton cluster detected as athletes runnng. H) The populaton densty n the scene s extremely hgh, the detected coherent moton patterns characterze the domnant crowd flows. The crowd s separated nto several bdrectonal flows: the yellow flow s movng to the left, the orange flow s movng to the rght, and the blue flow s movng aganst the orange flow dvdng the orange flow of people. countng error, DR, and F AR for the three methods respectvely. Coherent Flterng acheves the best performance. On the other hand, ALDENTE and BayDet work poorly when the densty of the crowd and the level of the nose ncrease. As shown n Fgure C, they both fal to detect the coherent motons n crowded scenes. Number of People 4 ALDENTE Bayesan Detecton A) DR FAR CountError 2 2 CF Frame No. BayDet ALDENTE Ground Truth CF BayDet ALDENTE B) Fg.. A) The number of pedestrans detected at each key frame wth respect to Frame No., and Detecton Rate(DR), False Alarm Rate(FAR), and countng error(counterror) for Coherent Flter(CF), BayDet[5], and ALDENTE[4]. B) BayDet and ALDENTE fal to detect the coherent motons when the crowdedness and the level of nose arse. 4.4 Further Analyss of the Algorthm Necessty of two flterng steps n the algorthm. The algorthm CoheFlterDet of Coherent Flterng can be dvded nto two steps: frst removng varant neghbors and

13 Coherent Flterng:Detectng Coherent Motons from Crowd Clutters 3 then flterng out the neghbors wth low averaged velocty correlatons. As dscussed n Secton 2.2, the coherent neghbor nvarance of spatotemporal relatonshps does not guarantee that all the dots n A are coherent dots, especally when d s small. Fgure A shows the clusterng results drectly obtaned from A wthout thresholdng when d = 6,, and 2 respectvely, and the plot of Normalzed Mutual Informaton (NMI) under dfferent d wth thresholdng and wthout thresholdng respectvely. No thresholdng, when d s small there remans a sgnfcant amount of nose. As d ncreases, NMI ncreases, whch means the clusterng performance s mproved. Then NMI wth thresholdng and NMI wthout thresholdng gradually converge. In prncple, all the ncoherent dots can be removed by settng a large d, such as when d = 2. However n practce, t s dffcult to obtan the correspondence of dots over a long perod. Thus flterng wth thresholdng the averaged velocty correlatons on A s necessary NMI thresholdng no thresholdng A) d=6 d= d= d B) K=5 K=25 K=5 K=25 Fg.. A) The clusterng results wthout thresholdng when d = 6,, 2, and the plot of NMI under dfferent d wth thresholdng and wthout thresholdng. B) The clusterng results on the 2D synthetc data and the real data wth K = 5 and K = 25 respectvely. Influence of K. K decdes the sze of the neghborhood. Fgure B shows the clusterng results wth K = 5 and K = 25 on the 2D synthetc data and the real data. When K s small, the detected coherent moton patterns are nclned to be dvded nto parts. However, when K s too large, some nose mght be ncluded. Thus the choce of K s related to the scale of coherent moton patterns to be detected n specfc vdeos. 5 Concluson In ths paper, we study the Coherent Neghbor Invarance as a pror of coherent motons and propose an effectve clusterng technque Coherent Flterng. Expermental evaluaton and comparson on synthetc and real datasets show the exstence of coherent neghbor nvarance n varous dynamc systems and the effectveness of our technque under hgh-densty nose. In the future work, we wll study coherence neghbor nvarance n a wder range of physcal and bologcal systems and explore more potental applcatons. 6 Acknowledgement Ths work s partally supported by the General Research Fund sponsored by the Research Grants Councl of Hong Kong (Project No. CUHK47 and CUHK47) and Natonal Natural Scence Foundaton of Chna (Project No.6557). It s also supported through Introduced Innovatve R&D Team of Guangdong Provnce 2D and Shenzhen Key Lab of Computer Vson and Pattern Recognton.

14 4 Bole Zhou, Xaoou Tang, and Xaogang Wang References. Couzn, I.: Collectve cognton n anmal groups. Trends n Cogntve Scences (29) 2. Moussad, M., Garner, S., Theraulaz, G., Helbng, D.: Collectve nformaton processng and pattern formaton n swarms, flocks, and crowds. Topcs n Cogntve Scence (29) 3. Camazne, S.: Self-organzaton n bologcal systems. Prnceton Unv Pr (23) 4. Rabaud, V., Belonge, S.: Countng crowded movng objects. In: Proc. CVPR. (26) 5. Brostow, G., Cpolla, R.: Unsupervsed bayesan detecton of ndependent moton n crowds. In: Proc. CVPR. (26) 6. Al, S., Shah, M.: Floor felds for trackng n hgh densty crowd scenes. In: Proc. ECCV. (28) 7. Al, S., Shah, M.: A lagrangan partcle dynamcs approach for crowd flow segmentaton and stablty analyss. In: Proc. CVPR. (27) 8. Wang, X., Ma, X., Grmson, W.: Unsupervsed actvty percepton n crowded and complcated scenes usng herarchcal bayesan models. IEEE Trans. on PAMI (28) 9. Zhou, B., Wang, X., Tang, X.: Understandng collectve crowd behavors:learnng a mxture model of dynamc pedestran-agents. In: Proc. CVPR. (22). Zhou, B., Wang, X., Tang, X.: Random feld topc model for semantc regon analyss n crowded scenes from tracklets. In: Proc. CVPR. (2). Wang, X., Teu, K., Grmson, E.: Learnng semantc scene models by trajectory analyss. Proc. ECCV (26) 2. Ln, D., Grmson, E., Fsher, J.: Learnng vsual flows: A le algebrac approach. In: Proc. CVPR. (29) 3. Hu, M., Al, S., Shah, M.: Learnng moton patterns n crowded scenes usng moton flow feld. In: Proc. ICPR. (28) 4. Wang, X., Ma, K., Ng, G., Grmson, W.: Trajectory analyss and semantc regon modelng usng nonparametrc herarchcal bayesan models. Int l Journal of Computer Vson (2) 5. Tron, R., Vdal, R.: A benchmark for the comparson of 3-d moton segmentaton algorthms. In: Proc. CVPR. (27) 6. Vdal, R., Ma, Y., Sastry, S.: Generalzed prncpal component analyss (gpca). IEEE Trans. on PAMI (25) 7. Brox, T., Malk, J.: Object segmentaton by long term analyss of pont trajectores. In: Proc. ECCV. (2) 8. Fschler, M., Bolles, R.: Random sample consensus: a paradgm for model fttng wth applcatons to mage analyss and automated cartography. Communcatons of the ACM (98) 9. Pett, O., Bon, R.: Decson-makng processes: the case of collectve movements. Behavoural Processes (2) 2. Lamme, V.: The neurophysology of fgure-ground segregaton n prmary vsual cortex. The Journal of neuroscence (995) 2. Sh, J., Malk, J.: Normalzed cuts and mage segmentaton. IEEE Trans. on PAMI (2) 22. Comancu, D., Meer, P.: Mean shft: A robust approach toward feature space analyss. IEEE Trans. on PAMI (22) 23. Lao, W., et al.: Clusterng of tme seres data a survey. Pattern Recognton (25) 24. Morrs, B., Trved, M.: A survey of vson-based trajectory learnng and analyss for survellance. IEEE Transactons on Crcuts and Systems for Vdeo Technology (28) 25. Strehl, A., Ghosh, J.: Cluster ensembles a knowledge reuse framework for combnng multple parttons. The Journal of Machne Learnng Research (23) 26. Pellegrn, S., Ess, A., Van Gool, L.: Improvng data assocaton by jont modelng of pedestran trajectores and groupngs. Proc. ECCV (2) 27. Tomas, C., Kanade, T.: Detecton and Trackng of Pont Features. In: Int l Journal of Computer Vson. (99)