Linearized Cluster Assignment via Spectral Ordering

Transcription

1 Lnearzed Cluster Assgnment va Spectral Orderng Chrs Dng Computatonal Research Dvson, Lawrence Berkeley Natonal Laboratory, Berkeley, CA 9472 Xaofeng He Computatonal Research Dvson, Lawrence Berkeley Natonal Laboratory, Berkeley, CA 9472 Abstract Spectral clusterng uses egenvectors of the Laplacan of the smlarty matrx. They are most convenently appled to 2-way clusterng problems. When applyng to mult-way clusterng, ether the 2-way spectral clusterng s recursvely appled or an embeddng to spectral space s done and some other methods are used to cluster the ponts. Here we propose and study a K-way cluster assgnment method. The method transforms the problem to fnd valleys and peaks of a 1-D quantty called cluster crossng, whch measures the symmetrc cluster overlap across a cut pont along a lnear orderng of the data ponts. The method can ether determne K clusters n one shot or recursvely splt a current cluster nto several smaller ones. We show that a lnear orderng based on a dstance senstve objectve has a contnuous soluton whch s the egenvector of the Laplacan, showng the close relatonshp between clusterng and orderng. The method reles on the connectvty matrx constructed as the truncated spectral expanson of the smlarty matrx, useful for revealng cluster structure. The method s appled to newsgroups to llustrate ntroduced concepts; experments show t outperforms the recursve 2-way clusterng and the standard K-means clusterng. 1. Introducton In recent years spectral clusterng emerges as sold approach for data clusterng. Spectral clusterng ncludes a class of clusterng methods (Bach & Jordan, 23; Appearng n Proceedngs of the 21 st Internatonal Conference on Machne Learnng, Banff, Canada, 24. Copyrght 24 by the authors. Chan et al., 1994; Dng et al., 22; Hagen & Kahng, 1992; Mela & Xu, 23; Ng et al., 21; Sh & Malk, 2; Yu & Sh, 23) that use egenvectors of the Laplacan of the symmetrc matrx W = (w j ) contanng the parwse smlarty between data objects, j. Spectral clusterng has well-motvated clusterng objectve functons and many nterestng and useful propertes can be proved. Spectral clusterng s most convenently appled to 2-way clusterng problem usng a sngle egenvector. When applyng to mult-way (K-way) clusterng, there are two man approaches: (1) the 2-way spectral clusterng s recursvely appled or (2) an embeddng to spectral space usng several egenvectors s frst done and some other methods, such as K-means (Ng et al., 21; Zha et al., 22; Bach & Jordan, 23), are used to cluster the ponts. These cluster assgnment methods are ndrect (except perhaps a recent study (Yu & Sh, 23)). 2. Lnearzed cluster assgnment Here we propose and study a drect K-way cluster assgnment method. The method transforms the problem to one of fndng valleys and peaks of a 1-D quantty called cluster crossng, whch measures the cluster overlap across a cut pont along lnear orderng of data objects. In other words, the method lnearzes the clusterng assgnment problem. The lnearzed assgnment algorthm depends crucally on an algorthm for orderng objects based on a parwse smlarty metrc. The orderng s such that adjacent objects are smlar whle objects far away along the orderng are ds-smlar. We show that for such an orderng objectve functon the nverse ndex permutaton has a contnuous (relaxed) soluton whch s the egenvector of the Laplacan of the smlarty matrx. Ths spectral orderng approach has been prevously consdered for reducton of the envelope of a sparse

2 symmetrc matrx. (Barnard et al., 1993). Our contrbutons are (1) provdng a clear orderng objectve functon and a new dervaton, and (2) ntroducng a modfcaton that sgnfcantly mproves the orderng. Ths s dscussed n 3. The actual lnearzaton s performed va the cluster crossng, the sum of smlartes symmetrcally across a cut pont along the lnear orderng. Computatonally, ths s the sum along ant-dagonal drecton on W wthn a pre-specfed bandwdth. Detals are dscussed n 4. If the clusters n a dataset are well-separated,.e., the smlarty matrx W s nearly dsconnected, the clusterng crossng along the spectral orderng can easly detect the clusters. For datasets where clusters moderately or strongly overlap, cluster crossng drectly computed from the smlarty matrx W provdes weak sgnals for revealng cluster structure. The connectvty matrx (Dng et al., 22) provdes sharper cluster structure and s adoped n the lnearzed assgnment algorthm. Ths s brefly dscussed n 5. In summary, the lnearzed assgnment algorthm depends on three technques: () an orderng of the data objects, () cluster crossng, () the connectvty matrx. These are dscussed n the followng sectons. 3. Dstance senstve orderng Gven n objects and the smlartes between them W = (w j ), the objectve of orderng s to nsure that () adjacent objects are smlar () the larger the dstance between the objects, the less smlar the two objects are. The orderng s defned by the ndex permutaton π(1,2,,n) = (π 1,,π n ). For a vector x = (x 1,,x n ) T, the permuted vector s π(x) = (x π1,,x πn ) T. The permuted smlarty matrx s (πwπ T ) j = w π,π j. Let J l (π) = n l =1 w π,π +l represent the parwse smlartes between objects wth fxed dstance l on the permuted order. We defne the global orderng objectve as n 1 mn J(π), J(π) = l 2 J l (π) = π l=1 n 1 n l l 2 w π,π +l l=1 =1 (1) Here larger dstance smlartes are mnmzed more heavly than smaller dstances, to ensure that the larger the dstance between a par of objects, the less smlar these two objects. Let us compute the optmal π. Frst, let j = + l or l = j, J(π) can be rewrtten as J(π) = 1 ( j) 2 w π,π 2 j = 1 ( j) 2 w π,π 2 j π,π j,j Replacng π by n the summaton and notng that ndex s permuted to π 1, where π 1 s the nverse permutaton, we obtan = n2 8,j J(π) = 1 2,j (π 1 π 1 j ) 2 w j ( ) 2 π 1 (n + 1)/2 π 1 j (n + 1)/2 w j. n/2 n/2 For smplcty, we defne the shfted and rescaled nverse ndex permutaton q π 1 (n + 1)/2 { 1 n n/2 n, 3 n n,, n 1 n }, (2) whch satsfes q =, q 2 = 1. (3) where q s further scaled by q (n 3 /12 n/3) 1/2 q whch does not change the permutaton. Note that (q q j ) 2 w j = (q 2 +qj 2 2q q j )w j = 2q T (D W)q j j where D s a dagonal matrx wth each dagonal element beng the sum of the correspondng row (d = j w j). Therefore, we need to mnmze q T (D W)q for q takng those dscrete values of Eq.(2), subject to the constrants n Eq.(3). Usng a Lagrangan multpler for the second constrant n Eq.(3), mnmzaton of J(π) becomes mn q J1, J1 = qt (D W)q q T q (4) Fndng the optmal soluton for the dscrete values of q s a combnatoral optmzaton problem, and s lkely to have no polynomal-tme optmal algorthms. However a contnuous soluton for q can be computed. We relax the restrcton that q must take dscrete values of Eq.(2) n [ 1,1], and let q take contnuous values n [ 1,1]. Wth ths, J1 can be mnmzed by solvng an egenvalue problem. It s well-known that q s an egenvector of the equaton (D W)q = ζq. (5) Clearly q = 1 = (1,,1) T s an egenvector wth ζ =. All other egenvector are orthogonal to q,.e.,

3 the frst constrant n Eq.(3) s also satsfed. Therefore q1 s the desred contnuous soluton of the dstance senstve orderng. We note that earler work on sparse matrx envelope reducton (Barnard et al., 1993) based on dfferent motvaton, reaches the same egenvector soluton. Our contrbuton here s to ntroduce the dstance senstve objectve functon J(π), and provde the detaled dervaton usng shfted nverse permutaton vector π 1 to show that the soluton s q1. Thus π 1 can be unquely recovered from q1. A smple mplementaton to recover the permutatons s to sort the elements of q1 n ncreasng order. Ths sortng nduces the desred ndex permutaton π Now we make a crucal modfcaton on the above soluton whch (a) mproves the qualty of the soluton, and (b) makes a drect connecton to the scaled PCA and connectvty matrx n 5. The modfcaton s made on the constrants n Eq.(3); we weght each pont wth the degree d, the column sum of the smlarty matrx W. In graph theory (Chung, 1997), d s called the volume of node. The new constrants are X X q d =, q2 d = 1. (6) (more dscusson later). Wth these constrants, the mnmzaton problem of J(π) becomes mn J 2, q T q (D W )q J 2 =. qt Dq 3 (7) Relaxng q to contnuous values n ( 1, 1). the soluton for q satsfes the egenvalue equaton (D W )q = ζdq. (8) 15 2 Let q = D 1/2 z. Substtutng t nto Eq.(8), we obtan D 1/2 W D 1/2 z = λz, λ = 1 ζ. 3 (9) Ths s a standard egenvalue equaton. Thus the egenvectors zk and qk have the orthogonalty relaton ½ 1 f k = ℓ T T (1) zk zℓ = qk Dqℓ = δkℓ = f k 6= ℓ The trval egenvector s q = e wth ζ =. Thus the constrants n Eq.(6) are automatcally satsfed. Snce (D W ) s sempostve defnte, we have ζk, λk = 1 ζk 1. (11) In dstance-senstve orderng, we seek qk wth the smallest ζk, or the largest λk. The desred soluton for the permutaton π 1 s q1 () (the th element of q1 ), subject to the rescalng and a constant shft condton accordng to Eq.(2). Note that q1 () < q1 (j) = π 1 < πj (12) Fgure 1. The connectvty matrx of 5-newsgroup (see 5) s dsplayed usng the J 1 orderng of Eq.(4) (top), and usng the J 2 orderng of Eq.(7) (mddle). The orgnal cosne smlarty of the 5 newsgroups s shown n bottom. In Fgure 1, we show a matrx where J 1 orderng s compared to J 2 orderng. Clearly, J 2 orderng provdes a better dstance-senstve orderng. The values of the ntal orderng objectve J(π) are J(π)/hJ =.846, usng J 1 usng J 2 J(π)/hJ =.584, (13) J(π)/hJ =.949, usng random orderng P P where hj = ( j wj /n2 ) j ( j)2 s the expected

4 mean value for J(π). Another way to measure the effects of orderng s to use the bandwdth and envelope of a symmetrc sparse matrx C. The bandwdth b() at row s the largest dstance between the dagonal element and any nonzero element n row. The bandwdth of the entre matrx s the largest of b() and the envelope s the sum of b(). For the J 1 orderng of C, bandwdth = 495, envelope = 156,24. For the J 2 orderng of C, bandwdth = 32, envelope = 48, 65. Clearly, J2 orderng s better. J 2 orderng uses the weghted constrants of Eq.(6) whle J 1 uses the unweghted constrants of Eq.(3). To understand why the weghtng leads to better orderng, frst observe that objects wth large d wll get smaller q to balance out the equaton. Now we rewrte Eq.(2) as π 1 = q (n+1)/2 gnorng the overall scalng factor. Snce π 1 = {1,,n}, (n + 1)/2 s n the mddle. Smaller q ndcates π 1 s near the mddle, thus objects wth large d are more lkely to be permuted towards the mddle usng the weghted constrants of Eq.(6). Ths s favorable, snce objects wth large d are more lkely to have more edges; and movng these objects towards mddle decreases the dstances among these smlar objects, therefore mproves J(π). Connecton to spectral clusterng Note that egenvector of Eq.(5) s used n the Rato cut spectral clusterng (Hagen & Kahng, 1992) and egenvector of Eq.(8) s used n the normalzed cut (Sh & Malk, 2) and mn-max cut (Dng et al., 21) spectral clusterng. In dervng 2-way spectral clusterng, only the sgns of the cluster ndcator vector are useful and all objects n a cluster have the same magntude. Ths ndcator vector s then relaxed nto the egenvector. In our dervaton of spectral orderng, both the sgn and magntude of the scaled and shfted permutaton vector are useful, see Eq.(2). Snce an egenvector has both sgn and magntude, the relaxaton of the permutaton vector s therefore better qualty approxmaton than the relaxaton of cluster ndcator. From ths analyss, we beleve the better reason for the success of spectral clusterng s due to the orderng, nstead of relaxng the dscrete cluster ndcators. In fact, ths orderng perspectve s used n actual mplementaton of spectral clusterng (Hagen & Kahng, 1992; Sh & Malk, 2): one frst sort q 1 to provde a lnear orderng, then along ths orderng, search for the cut that optmzes the cluster objectve functon. Thus our orderng analyss provdes a deeper understandng of spectral clusterng. Our results ndcates that J 2 orderng s better than J 1 orderng. Ths s due to the weghtng of d, the node degree, n Eq.(6). Smlar motvaton s used n normalzed cut. Let s 12 be the cut between two subgraphs C 1,C 2. All three graph clusterng objectve functon can be wrtten as J = s 12 /a 1 + s 12 /a 2. For Rato cut, a k = C k 1. For normalzed cut, a k = C k d. Ths weghtng of the subgraph volume mproves upon the smple weghtng of the subgraph sze n rato cut. For mn-max cut, a k =,j C k w j, the sum of edge weghts nsde C k. That J 2 orderng s better than J 1 orderng mples that normalzed cut and mn-max cut n general provdes a better clusterng than rato cut. Ths fact s observed n experments (Sh & Malk, 2; Dng et al., 21). It s sometmes happens that there s a symmetry among several nodes n the graph,.e., G(W)G T = W, where G s a permutaton specfyng an element n the nvarant symmetry group. In ths case, an egenvector of W,D W,D 1/2 WD 1/2 wll have several nodes wth the same value. If ths happens, nether the orderng nor clusterng problems can be unquely determned. Ths s not necessarly a weakness of the spectral methods, although t become obvous from the perspectve of the egenvector. In practce, ths happens rarely for weghted graphs. 4. Cluster crossng and assgnment We start wth cluster overlap. Gven two clusters C k,c l, the cluster overlap can be defned as the sum of parwse assocatons between two clusters, s kl = C k,j C l w j. (14) In spectral clusterng, s kl are mnmzed. Cluster overlap nvolves all C k C l parwse smlartes. We defne cluster crossng as the sum of a small fracton of the parwse smlartes. Ths s aded by lnear orderng data ponts. Gven a lnear order o of all objects, at each ste of the order, we can sum over w j wthn a wndow sze 2m + 1 across the ste, ρ() = m j=1 w o( j),o(+j) Ths corresponds to sum along the ant-dagonal drectons n the smlarty matrx W wth a bandwdth m. There are 2n 1 ant-dagonals n a matrx, among them ρ() are n full-step ant-dagonals. We also utlze

5 the other n 1 half-step ant-dagonals,.e., ρ( ± 1/2) = m w o( j),o(+j±1). j=1 The fnal crossng s the weghted average ρ() = ρ( + 1/2)/4 + ρ()/2 + ρ( 1/2)/4. (When s close to the two ends,.e., n m or m, the sum should be properly weghted to reflect the fact that the number of smlartes n the sum s less than the normal case.) Clearly, cluster crossng ρ() should have a mnmum at the cluster boundary between C k,c l. As moves away from the boundary, ρ() ncreases. Ths form the bass of the lnearzed cluster assgnment. Ths approach works for K > 2 as well as for K = 2; In essence, t reformulate a problem of K-way clusterng wth parwse smlartes nto a K-way clusterng problem n 1-dmenson (a) (b) (c) Fgure 2. Crossng curves for dataset A. Top: computed based on the smlarty matrx W shown n Fg.1(bottom). Mddle: computed based on the connectvty matrx C usng J 2 orderng shown n Fg.1(mddle). Bottom: computed based on C usng J 1 orderng shown n Fg.1(top). To llustrates the basc deas n ths approach. we compute the crossng of the matrx C shown n Fgure 1(bottom). The crossng curves are shown n Fgure 2. For matrx C usng J 2 orderng, the crossng (mddle panel n Fgure 2) exhbt clearly the fve-cluster structure. In cluster crossng curve, the valleys are more mportant than the peaks. Ths s because between two consecutve valleys, there could be several overlappng clusters so that the peaks are not as pronounced as the valleys. Usng the valleys, we can clearly separate sets of clusters (composte cluster). Ths suggest a dvde-and-conquer approach,.e., recursvely apply the algorthm on each set of clusters, untl the total number of cluster reach the pre-specfed K, or some other crtera are met (lke the top-down dvsve clusterng approach). For example n Fgure 1 (mddle), the 2nd and 3rd clusters overlap slghtly, and the correspondng valley pont n the crossng curve (mddle panel n Fgure 2) s not as low as others (although unambguously clear). We mght consder these two cluster as one composte cluster. Thus we cut the crossng nto 4 clusters at present round and cut the composte cluster n next round. 5. Scaled prncpal components and connectvty matrx Gven a symmetrc smlarty matrx W, one can obtan a spectral decomposton to get the prncpal component analyss (PCA), a wdely used technque n multvarate statstcs. In scaled PCA proposed n (Dng et al., 22), one performs the followng spectral decomposton W = D 1/2 (D 1/2 WD 1/2 )D 1/2 K = D 1/2 ( z kˆλk z T k )D 1/2 k=1 here the spectral decomposton s performed on the scaled matrx Ŵ = D 1/2 WD 1/2. Clearly, the egenvectors z k are governed by Eq.(9). The magntudes of all egenvalues are less than 1, as n Eq.(11). The connectvty matrx C s obtaned by truncatng the PCA expanson at K terms and settng the egenvalues to unty, λ k = 1, as K K C = D 1/2 z k z T k D 1/2 = D q k q T k D (15) k=1 k=1 where q k = D 1/2 z k s called scaled prncpal components due to ts smlarty to the usual PCA. q s governed by Eq.(8), and s closely related to spectral clusterng. It s shown va a perturbaton analyss that C has a so-called self-aggregaton property that connectvtes (matrx elements n C) between dfferent clusters are suppressed whle connectvtes wthn clusters are enhanced. Thus C s useful for revealng cluster structure. Connectvty matrx approach nvolves a nose reducton procedure. The probablty that two objects, j

6 belong to the same cluster s p j = C j /C 1/2 C 1/2 jj. To reduce nose one set C j = f p j < β, (16) where β =.8. For a range of problems, β =.5.9 leads to very smlar results. As an llustraton, the orgnal smlarty matrx (based on 5 newsgroups n 6) s shown n Fgure 1(bottom) usng J 2 orderng, where cluster structure s not apparent. Connectvty matrx (shown n Fgure 1) constructed based on ths smlarty matrx has clear cluster structure. 6. Complete assgnment algorthm Prespecfy K as the numnber of clusters, and set bandwdth m = n/k (or the expected largest cluster sze). The complete algorthm s as follows: (1) Compute connectve matrx C; (2) Compute the J 2 orderng of C; (3) Compute the crossng of C based on J 2 orderng. (4) Locate valley ponts n the crossng curve. Assgn each regon sandwched between two valley ponts or ends to one composte cluster. (5) If the total number of current composte cluster s less than K, recursvely apply the algorthm to the largest to further splt t. Note that ths recursve clusterng algorthm dffers from the usual recursve 2-way clusterng n that, a current composte cluster s parttoned nto several clusters dependng on the crossng curve, not restrcted to 2 clusters. It s possble that all K clusters are dentfed usng one crossng curve as n 4. Thus the total number of recurson s less than or equal to K-1, whch s requred by the usual recursve 2-way clusterng. In step (5), the choce of next cluster to splt s based on the largest (sze) cluster. Ths smple choce s more orented towards cluster balance. More refned choces for cluster splt s dscussed n (Dng & He, 22). The man advantage of ths approach s that clusters are formed consstently. In 2-way recursve clusterng, each current cluster s formed va a certan clusterng objectve functon whch s correctly motvated for only true clusters, not for composte clusters. For example n normalzed cut, the cluster objectve functon for K-way clusterng can not be recursvely constructed from the 2-way clusterng objectve. Therefore the 2- way recursve procedure s only a heurstc for K-way clusterng. In our lnearzed assgnment, cluster are assgn based on the crtera that the connectvty between clusters are small, whch s vald for both true clusters and composte clusters. 7. Experments The lnearzed cluster assgnment method s appled to Internet newsgroup artcles. A 2-newsgroup dataset s from 11/www/nave-bayes.html. Word-document matrx s frst constructed. 1 words are selected accordng to the mutual nformaton between words and documents n unsupervsed manner. Standard tf.df term weghtng s used. Each document s normalzed to 1. We focus on two sets of 5-newsgroup combnatons. The choce of K = 5 s to have some varety n the recursve steps (we avod K = 4,8). These two newsgroup combnatons are lsted below: A B NG2: comp.graphcs NG2: comp.graphcs NG9: rec.motorcycles NG3: comp.os.ms-wndows NG1: rec.sport.baseball NG8: rec.autos NG15: sc.space NG13: sc.electroncs NG18: talk.poltcs.mdeast NG19: talk.poltcs.msc Datasets wth well separated clusters are easy to handle; we are nterested n clusterng medum and large overlappng clusters. To measure cluster separaton, we compute s kl and defne the symmetrcally scaled cluster overlap as the cluster separaton ndex between clusters C k,c l as µ kl = s kl / s kk s ll. The over-all separaton s defned as 2 µ = K(K 1) k,l;k l µ kl. Clearly µ 1 and µ kk = 1. For a complete graph µ kl = 1 and µ = 1. Cluster overlap s kl and separaton ndex µ kl can be convenently stored n a matrx S, where S(upper-rght trangle ncludng dagonals) = s kl and S(lower-left trangle)= µ kl. For datasets A,B, ther overlap-separatons are S(A) = S(B) = Ther average separaton are µ(a) =.695, µ(b) =.755. These nformaton are useful. For example, for dataset B, NG18 (mdeast) s a coherent cluster because s 55 = 996 s relatvely large, whereas NG13(electroncs) s less coherent because s 44 = 472 s relatvely small. The overlap between NG2 (graphcs)

7 and NG3(wndows OS) s relatvely large: µ 12 =.578 whle the overlap between NG8 (auto) and NG13 (electroncs) s also large: µ 34 =.483. Overall, dataset A s moderately overlappng and dataset B s strongly overlappng. The above s based on a random sample of documents from the newsgroups. Each cluster has 1 documents. To accumulate suffcent statstcs, for each newsgroup combnaton, we generate 5 samples and average ther performance. Dataset A The cosne smlarty matrx W among documents shown n Fgure 1(bottom). The cluster structure s not clear from the smlarty matrx W. The connectvty matrx C s dsplayed n Fgure 1(top) whch exhbts the cluster structure. Clusterng crossngs based on W and C are shown n Fgure 2. The crossng for C based on J 2 orderng (mddle panel) shows clear cluster structure, whereas crossng for C based on J 1 orderng (bottom pannel) shows less clear cluster structure. Crossng for W (top pannel) show even less cluster structure. Based on the crossng of C usng J 2 orderng, local mnma n the valleys are dentfed usng a smple smoothng procedure, where the new smoothed value on each pont s the average of old values on 5 nearest ponts. Ths smoothng procedure overcomes the local abrupt changes and automatcally compute the more stable or consensus valley ponts, although the dfference wth un-smoothed one s often small. Data ponts between two valleys or ends are assgned to one cluster. Thus all ponts are assgned nto 5 clusters n one shot. Each of newsgroup artcle s cluster label s known (although not necessarly perfect). Usng ths, the confuson matrx T = (t kl ) for the clusterng results are computed, where t kl = number of ponts belongng to cluster k but clustered to cluster l. Based on the lnearzed cluster assgnment results, T s computed as T = For ths results, the clusterng accuracy, Q = k t kk/n = 9.5%. The clusterng experment s repeated for 5 dfferent random samples. The accuracy for ths lnearzed orderng approach s lsted n Table 1. For the same Table 1. Clusterng accuracy as of dfferent methods on the 5-newsgroup datasets. LA: lnearzed assgnment; R2W: recursve 2-way clusterng; usng MnMaxCut wth cluster choce for splt based on largest sze cluster; K-means. Method LA R2W K-means Data A 89.% 82.8% 75.1% Data B 75.7% 67.2% 56.4% dataset, the cluster accuracy usng recursve 2-way spectral clusterng and standard K-means are also lsted n Table 1. One see that the lnearzed assgnment outperform slghtly over the recursve 2-way clusterng and sgnfcantly over the K-means. Dataset B The cosne-smlarty matrx W s shown n Fgure 3 (top panel, usng J 2 orderng). The connectvty matrx C of ths dataset s shown n Fgure 4. The overlap between NG2 (computer graphcs) and NG3 (Wndows OS) s large; the overlap between NG8 (autos) and NG13 (electroncs) s large as well. These are expected from the cluster separaton ndexes S(B), and also can be confrmed by nspectng the cosnesmlarty W shown n Fgure 3 (bottom panel), usng J 2 orderng based on C. The crossng based on C s shown n Fgure 5 (top panel). The lower panels show the crossng curves after four successve applcatons of smoothng. Based on ths crossng, we can dentfy three composte clusters by two clear and low-lyng valley ponts. The two large composte clusters are further clustered usng the same lnearzed algorthm. Repeatng the experments on 5 random samples from dataset B, the clusterng accuracy s lsted n Table 1. The lnearzed assgnment outperforms the recursve 2-way spectral clusterng and the standard K-means. 8. Summary In summary, we propose and study a drect K-way cluster assgnment method that lnearze the clusterng problem nto 1-D clusterng crossng curve. The method depends on an effectve lnear orderng provded by the spectral orderng. We prove a clear dervaton of the dstance senstve orderng and show the shfted and scaled ndex permutaton vector s relaxed nto egenvectors of the Laplacan of the smlarty matrx. Our results provdes a deeper nsghts to spectral clusterng as well. Ths work s supported by U.S. Department of Energy,

8 (a) Fgure 4. The connectvty matrx of dataset B. (b) Fgure 3. The cosne smlarty matrx W of dataset B, dsplayed usng (a) the J 2 orderng based on W, (b) the J 2 orderng based on the connectvty matrx C n Fg.(4). Offce of Scence, Offce of Laboratory Polcy and Infrastructure, through an LBNL LDRD, under contract DE-AC3-76SF98. References Bach, F. R., & Jordan, M. I. (23). Learnng spectral clusterng. Neural Info. Processng Systems 16 (NIPS 23). Barnard, S. T., Pothen, A., & Smon, H. D. (1993). A spectral algorthm for envelope reducton of sparse matrces. Proc. Supercomputng 93, IEEE, Chan, P., M.Schlag, & Zen, J. (1994). Spectral k-way rato-cut parttonng and clusterng. IEEE Trans. CADIntegrated Crcuts and Systems, 13, Chung, F. (1997). Spectral graph theory. Amer. Math. Socety. Dng, C., & He, X. (22). Cluster merge and splt n herarchcal clusterng. Proc. IEEE Int l Conf. Data Mnng, Dng, C., He, X., Zha, H., Gu, M., & Smon, H. (21). A mn-max cut algorthm for graph parttonng and data clusterng. Proc. IEEE Int l Conf. Data Mnng. Dng, C., He, X., Zha, H., & Smon, H. (22). Unsupervsed learnng: self-aggregaton n scaled prncpal component space. Proc. 6th European Conf. Prncples of Fgure 5. Crossngs computed based on the connectvty matrx n Fg.(4). Frst curve s the crossng. Lower curves show the smoothng of the crossngs. The two ponts n bottom curve ndcate the cut ponts. Data Mnng and Knowledge Dscovery (PDKK 22), Hagen, L., & Kahng, A. (1992). New spectral methods for rato cut parttonng and clusterng. IEEE. Trans. on Computed Aded Desgn, 11, Mela, M., & Xu, L. (23). Multway cuts and spectral clusterng. U. Washngton Tech Report. Ng, A., Jordan, M., & Wess, Y. (21). On spectral clusterng: Analyss and an algorthm. Proc. Neural Info. Processng Systems (NIPS 21). Sh, J., & Malk, J. (2). Normalzed cuts and mage segmentaton. IEEE. Trans. on Pattern Analyss and Machne Intellgence, 22, Yu, S. X., & Sh, J. (23). Multclass spectral clusterng. Int l Conf. on Computer Vson. Zha, H., Dng, C., Gu, M., He, X., & Smon, H. (22). Spectral relaxaton for K-means clusterng. Advances n Neural Informaton Processng Systems 14 (NIPS 1),