High Dimensional Data Clustering

Transcription

1 Hgh Dmensonal Data Clusterng Charles Bouveyron 1,2, Stéphane Grard 1, and Cordela Schmd 2 1 LMC-IMAG, BP 53, Unversté Grenoble 1, Grenoble Cede 9, France charles.bouveyron@mag.fr, stephane.grard@mag.fr 2 INRIA Rhône-Alpes, Projet Lear, 655 av. de l Europe, Sant-Ismer Cede, France cordela.schmd@nralpes.fr Summary. Clusterng n hgh-dmensonal spaces s a recurrent problem n many domans, for eample n object recognton. Hgh-dmensonal data usually lve n dfferent lowdmensonal subspaces hdden n the orgnal space. Ths paper presents a clusterng approach whch estmates the specfc subspace and the ntrnsc dmenson of each class. Our approach adapts the Gaussan mture model framework to hgh-dmensonal data and estmates the parameters whch best ft the data. We obtan a robust clusterng method called Hgh- Dmensonal Data Clusterng (HDDC). We apply HDDC to locate objects n natural mages n a probablstc framework. Eperments on a recently proposed database demonstrate the effectveness of our clusterng method for category localzaton. Key words: Model-based clusterng, hgh-dmensonal data, dmenson reducton, dmenson reducton, parsmonous models. 1 Introducton In many scentfc domans, the measured observatons are hgh-dmensonal. For eample, vsual descrptors used n object recognton are often hgh-dmensonal and ths penalzes classfcaton methods and consequently recognton. Popular clusterng methods are based on the Gaussan mture model and show a dsappontng behavor when the sze of the tranng dataset s too small compared to the number of parameters to estmate. To avod overfttng, t s therefore necessary to fnd a balance between the number of parameters to estmate and the generalty of the model. In ths paper we propose a Gaussan mture model whch determnes the specfc subspace n whch each class s located and therefore lmts the number of parameters to estmate. The Epectaton-Mamzaton (EM) algorthm [5] s used for parameter estmaton and the ntrnsc dmenson of each class s determned automatcally wth the scree test of Cattell. Ths allows to derve a robust clusterng method n hgh-dmensonal spaces, called Hgh Dmensonal Data Clusterng (HDDC). In order to further lmt the number of parameters, t s possble to make addtonal assumptons on the model. We can for eample assume that classes are sphercal n ther subspaces or f some parameters to be common between classes.

2 2 Charles Bouveyron, Stéphane Grard, and Cordela Schmd We evaluate HDDC on a recently proposed vsual recognton dataset [4]. We compare HDDC to standard clusterng methods and to the state of the art results. We show that our approach outperforms estng results for object localzaton. Ths paper s organzed as follows. Secton 2 presents the state of the art on clusterng of hgh-dmensonal data. In Secton 3, we descrbe our parameterzaton of the Gaussan mture model. Secton 4 presents our clusterng method,.e. the estmaton of the parameters and of the ntrnsc dmensons. Epermental results for our clusterng method are gven n Secton 5. 2 Related work on hgh-dmensonal clusterng Many methods use global dmensonalty reducton and then apply a standard clusterng method. Dmenson reducton technques are ether based on feature etracton or feature selecton. Feature etracton bulds new varables whch carry a large part of the global nformaton. The most known method s Prncpal Component Analyss (PCA) whch s a lnear technque. Recently, many non-lnear methods have been proposed, such as Kernel PCA and non-lnear PCA. In contrast, feature selecton fnds an approprate subset of the orgnal varables to represent the data. Global dmenson reducton s often advantageous n terms of performance, but loses nformaton whch could be dscrmnant,.e. clusters are often hdden n dfferent subspaces of the orgnal feature space and a global approach cannot capture ths. It s also possble to use a parsmonous model [7] whch reduces the number of parameters to estmate. It s for eample possble to f some parameters to be common between classes. These methods do not solve the problem of hgh dmensonalty because clusters are usually hdden n dfferent subspaces and many dmensons are rrelevant. Recent methods determne the subspaces for each cluster. Many subspace clusterng methods use heurstc search technques to fnd the subspaces. They are usually based on grd search methods and fnd dense clusterable subspaces [8]. The approach "mtures of Probablstc Prncpal Component Analyzers" [10] proposes a latent varable model and derves an EM based method to cluster hgh-dmensonal data. Bocc et al. [1] propose a smlar method to cluster dssmlarty data. In ths paper, we ntroduce an unfed approach for class-specfc subspace clusterng whch ncludes these two methods and allows addtonal regularzatons. 3 Gaussan mture models for hgh-dmensonal data Clusterng dvdes a gven dataset { 1,..., n } of n data ponts nto k homogeneous groups. Popular clusterng technques use Gaussan Mture Models (GMM), whch assume that each class s represented by a Gaussan probablty densty. Data { 1,..., n } R p are then modeled wth the densty f(, θ) = k =1 π φ(, θ ), where φ s a mult-varate normal densty wth parameter θ = {µ, Σ } and π are mng proportons. Ths model estmates full covarance matrces and therefore the number of parameters s very large n hgh dmensons. However, due to the empty

3 Hgh Dmensonal Data Clusterng 3 E P () d(,e ) E X µ P () d(µ, P ()) Fg. 1. The class-specfc subspace E. space phenomenon we can assume that hgh-dmensonal data lve n subspaces wth a dmensonalty lower than the dmensonalty of the orgnal space. We therefore propose to work n low-dmensonal class-specfc subspaces n order to adapt classfcaton to hgh-dmensonal data and to lmt the number of parameters to estmate. 3.1 The famly of Gaussan mture models We remnd that class condtonal denstes are Gaussan N(µ, Σ ) wth means µ and covarance matrces Σ, = 1,..., k. Let Q be the orthogonal matr of egenvectors of Σ, then = Q t Σ Q s a dagonal matr contanng the egenvalues of Σ. We further assume that s dvded nto two blocks: = a a d 0 0 b b 9 = ; d 9 >= (p d ) >; where a j > b, j = 1,..., d. The class specfc subspace E s generated by the d frst egenvectors correspondng to the egenvalues a j wth µ E. Outsde ths subspace, the varance s modeled by the sngle parameterb. Let P () = Q t Q ( µ )+µ be the projecton of on E, where Q s made of the d frst columns of Q supplemented by zeros. Fgure 1 summarzes these notatons. The mture model presented above wll be n the followng referred to by [a j b Q d ]. By fng some parameters to be common wthn or between classes, we obtan a famly of models whch correspond to dfferent regularzatons. For eample, f we f the frst d egenvalues to be common wthn each class, we obtan the more restrcted model [a b Q d ]. The model [a b Q d ] s often robust and gves satsfyng results,.e. the assumpton that each matr has only two dfferent egenvalues s n many cases an effcent way to regularze the estmaton of. In

4 4 Charles Bouveyron, Stéphane Grard, and Cordela Schmd ths paper, we focus on the models [a j b Q d ], [a j bq d ], [a b Q d ], [a bq d ] and [abq d ]. 3.2 The decson rule Classfcaton assgns an observaton R p wth unknown class membershp to one of k classes C 1,..., C k known a pror. The optmal decson rule, called Bayes decson rule, affects the observaton to the class whch has the mamum posteror probablty P( C ) = π φ(, θ )/ k l=1 π lφ(, θ l ). Mamzng the posteror probablty s equvalent to mnmzng 2 log(π φ(, θ )). For the model [a j b Q d ], ths results n the decson rule δ + whch assgns to the class mnmzng the followng cost functon K (): K () = µ P () 2 Λ + 1 d P () 2 + log(a j )+(p d )log(b ) 2 log(π ), b where. Λ s the Mahalanobs dstance assocated wth the matr Λ = Q t Q. The posteror probablty can therefore be rewrtten as follows: P( C ) = 1/ k l=1 ep ( 1 2 (K () K l ()) ). It measures the probablty that belongs to C and allows to dentfy dubously classfed ponts. We can observe that ths new decson rule s manly based on two dstances: the dstance between the projecton of on E and the mean of the class; and the dstance between the observaton and the subspace E. Ths rule assgns a new observaton to the class for whch t s close to the subspace and for whch ts projecton on the class subspace s close to the mean of the class. If we consder the model [a b Q d ], the varances a and b balance the mportance of both dstances. For eample, f the data are very nosy,.e. b s large, t s natural to balance the dstance P () 2 by 1/b n order to take nto account the large varance n E. Remark that the decson rule δ + of our models uses only the projecton on E and we only have to estmate a d -dmensonal subspace. Thus, our models are sgnfcantly more parsmonous than the general GMM. For eample, f we consder 100- dmensonal data, made of 4 classes and wth common ntrnsc dmensons d equal to 10, the model [a b Q d ] requres the estmaton of parameters whereas the full Gaussan mture model estmates parameters. 4 Hgh Dmensonal Data Clusterng In ths secton we derve the EM-based clusterng framework for the model [a j b Q d ] and ts sub-models. The new clusterng approach s n the followng referred to by Hgh-Dmensonal Data Clusterng (HDDC). By lack of space, we do not present proofs of the followng results whch can be found n [2]. 4.1 The clusterng method HDDC Unsupervsed classfcaton organzes data n homogeneous groups usng only the observed values of the p eplanatory varables. Usually, the parameters are estmated

5 Hgh Dmensonal Data Clusterng 5 by the EM algorthm whch repeats teratvely E and M steps. If we use the parameterzaton presented n the prevous secton, the EM algorthm for estmatng the parameters θ = {π, µ, Σ, a j, b, Q, d }, can be wrtten as follows: E step: ths step computes at the teraton q the condtonal posteror probabltes j = P( j C (q) j ) accordng to the relaton: j = 1/ k ( ) 1 ep 2 (K(q 1) ( j ) K (q 1) l ( j )), (1) l=1 where K s defned n Paragraph 3.2. M step: ths step mamzes at the teraton q the condtonal lkelhood. Proportons, means and covarance matrces of the mture are estmated by: ˆπ (q) = n(q) n, ˆµ(q) = 1 n (q) ˆΣ (q) = 1 n (q) n n j j, n (q) = n j. (2) j ( j ˆµ (q) )( j ˆµ (q) ) t. (3) The estmaton of HDDC parameters s detaled n the followng subsecton. 4.2 Estmaton of HDDC parameters Assumng for the moment that parameters d are known and omttng the nde q of the teraton for the sake of smplcty, we obtan the followng closed form estmators for the parameters of our models: Subspace E : the d frst columns of Q are estmated by the egenvectors assocated wth the d largest egenvalues λ j of ˆΣ. Model [a j b Q d ]: the estmators of a j are the d largest egenvalues λ j of ˆΣ and the estmator of b s the mean of the (p d ) smallest egenvalues of ˆΣ and can be wrtten as follows: 1 ˆb = Tr( (p d ) ˆΣ d ) λ j. (4) Model [a b Q d ]: the estmator of b s gven by (4) and the estmator of a s the mean of the d largest egenvalues of ˆΣ : â = 1 d λ j, (5) d Model [a bq d ]: the estmator of a s gven by (5) and the estmator of b s: 1 k ˆb = (np n k =1 n d ) Tr(Ŵ) d n λ j, (6) =1

6 6 Charles Bouveyron, Stéphane Grard, and Cordela Schmd where Ŵ = k =1 ˆπ ˆΣ. Model [abq d ]: the estmator of b s gven by (6) and the estmator of a s: â = 1 k =1 n d 4.3 Intrnsc dmenson estmaton k =1 λ j. (7) n d We also have to estmate the ntrnsc dmensons of each subclass. Ths s a dffcult problem wth no unque technque to use. Our approach s based on the egenvalues of the class condtonal covarance matr ˆΣ of the class C. The jth egenvalue of ˆΣ corresponds to the fracton of the full varance carred by the jth egenvector of ˆΣ. We estmate the class specfc dmenson d, = 1,..., k, wth the emprcal method scree-test of Cattell [3] whch analyzes the dfferences between egenvalues n order to fnd a break n the scree. The selected dmenson s the one for whch the subsequent dfferences are smaller than a threshold. In our eperments, the threshold s chosen by cross-valdaton. We also compared to the probablstc crteron BIC [9] whch gave very smlar results. 5 Epermental results In ths secton, we use our clusterng method HDDC to recognze and locate objects n natural mages. Object category recognton s one of the most challengng problems n computer vson. Recent methods use local mage descrptors whch are robust to occlusons, clutters and geometrc transformatons. Many of these approaches form clusters of local descrptors as an ntal step; n most cases clusterng s acheved wth k-means, dagonal or sphercal GMM and EM estmaton wth or wthout PCA to reduce the dmenson. Dorko and Schmd [6] select dscrmnant clusters based on the lkelhood rato and use the most dscrmnatve ones for recognton. Bag-of-keypont methods [11] represent an mage by a hstogram of cluster labels and learn a Support Vector Machne classfer. 5.1 Protocol and data We use an approach smlar to Dorko and Schmd [6]. Local descrptors of dmenson 128 are etracted from the tranng mages (see [6] for detals) and then are organzed nto k groups by a clusterng method (k = 200 n our eperments). We then compute the dscrmnatve capacty of the class C for a gven object category O through the [ posteror probablty ] R = P(C O C ). Ths probablty s estmated by R = (Ψ t Ψ) 1 Ψ t Φ, where Φ j = P( j O j ) and Ψ jl = P( j C l j ). Learnng can be ether supervsed or weakly supervsed. In the supervsed framework, the objects are segmented usng boundng boes and only the descrptors located nsde the boundng boes are labeled as postve n the learnng step. In the

7 Hgh Dmensonal Data Clusterng 7 HDDC [ Q d ] GMM Pascal Learnng [a jb ] [a jb] [a b ] [a b] PCA+dag. Dagonal Sphercal Best of [4] Supervsed Weakly-sup / Table 1. Object localzaton on the database Pascal test2: mean of the average precson on the four object categores. Best results are hghlghted. weakly-supervsed scenaro, the object are not segmented and all descrptors from mages contanng the object are labeled as postve. Note that n ths case many descrptors from the background are labeled as postve. In both cases, we consder that P( j O j ) = 1 f j s postve and P( j O j ) = 0 otherwse. For each descrptor of a test mage, the probablty that ths pont belongs to the object O s then gven by P( j O j ) = k =1 R P( j C j ) where the posteror probablty P( j C j ) s obtaned by the decson rule assocated to the clusterng method (see Paragraph 3.2 for HDDC). We compare the HDDC clusterng method to the followng classcal clusterng methods: dagonal Gaussan mture model, sphercal Gaussan mture model, and data reducton wth PCA combned wth a dagonal Gaussan mture model. The dagonal GMM has a covarance matr defned by Σ = dag(σ 1,..., σ p ) and the sphercal GMM s characterzed by Σ = σ Id. For all the models the parameters were estmated va the EM algorthm. The EM estmaton used the same ntalzaton based on k-means for both HDDC and classcal methods. The object category database used n our eperments s the Pascal dataset [4] whch contans four categores: motorbkes, bcycles, people and cars. There are 684 tranng mages and two test sets: test1 and test2. We evaluate our method on the set test2, whch s the most dffcult of the two test sets and contans 956 mages. There are on average 250 descrptors per mage. From a computatonal pont of vew, the localzaton step s very fast. For the learnng step, computng tme manly depends of the number of groups k and s equal on average to 2 hours on a recent computer. To locate an object n a test mage, we compute for each descrptor the probablty to belong to the object. We then predct the boundng bo based on the arthmetc mean and the standard devaton of descrptors. In order to compare our results wth those of the Pascal Challenge [4], we used ts evaluaton crteron "average precson" whch s the area under the precson-recall curve computed for the predcted boundng boes (see [4] for further detals). 5.2 Object localzaton results Table 1 presents localzaton results for the dataset Pascal test2 wth supervsed and weakly-supervsed tranng. Frst of all, we observe that HDDC performs better than standard GMM wthn the probablstc framework descrbed n Secton 5.1 and partcularly n the weakly-supervsed framework. Ths ndcates that our clusterng method dentfes relevant clusters for each object category. In addton, HDDC

8 8 Charles Bouveyron, Stéphane Grard, and Cordela Schmd (a) car (b) motorbke (c) bcycle Fg. 2. Object localzaton on on the database Pascal test2: predcted boundng boes wth HDDC are n red and true boundng boes are n yellow. provdes better localzaton results than the state of the art methods reported n the Pascal Challenge [4]. Note that the dfference between the results obtaned n the supervsed and n the weakly-supervsed framework s not very hgh. Ths means that HDDC effcently dentfes dscrmnatve clusters of each object category even wth weak supervson. Weakly-supervsed results are promsng as they avod tme consumng manual annotaton. Fgure 2 shows eamples of object localzaton on test mages wth the model [a b Q d ] of HDDC and supervsed tranng. Acknowledgments Ths work was supported by the French department of research through the ACI Masse de données (Movstar project). References 1. Bocc, L., Vcar, D., Vch, M.: A mture model for the classfcaton of three-way promty data. Computatonal Statstcs and Data Analyss, 50, (2006). 2. Bouveyron, C., Grard, S., Schmd, C.: Hgh-Dmensonal Data Clusterng. Techncal Report 1083M, LMC-IMAG, Unversté J. Fourer Grenoble 1 (2006). 3. Cattell, R.: The scree test for the number of factors. Multvarate Behavoral Research, 1, (1966). 4. D Alche Buc, F., Dagan, I., Qunonero, J.: The 2005 Pascal vsual object classes challenge. Proceedngs of the frst PASCAL Challenges Workshop, Sprnger (2006). 5. Dempster, A., Lard, N., Rubn, D.: Mamum lkelhood from ncomplete data va the EM algorthm. Journal of the Royal Statstcal Socety, 39, 1 38 (1977). 6. Dorko, G., Schmd, C.: Object class recognton usng dscrmnatve local features. Techncal Report 5497, INRIA (2004). 7. Fraley, C., Raftery, A.: Model-based clusterng, dscrmnant analyss and densty estmaton. Journal of Amercan Statstcal Assocaton, 97, (2002). 8. Parsons, L., Haque, E., Lu, H.: Subspace clusterng for hgh dmensonal data: a revew. SIGKDD Eplor. Newsl. 6, (2004). 9. Schwarz, G.: Estmatng the dmenson of a model. Annals of Statstcs, 6, (1978). 10. Tppng, M., Bshop, C.: Mtures of probablstc prncpal component analysers. Neural Computaton, (1999). 11. Zhang, J., Marszalek, M., Lazebnk, S., Schmd, C.: Local features and kernels for classfcaton of teture and object categores. Techncal Report 5737, INRIA (2005).