Cone Cluster Labeling for Support Vector Clustering

Transcription

1 Cone Cluster Lbeling for Support Vector Clustering Sei-Hyung Lee Deprtment of Computer Science University of Msschusetts Lowell MA 1854, U.S.A. Kren M. Dniels Deprtment of Computer Science University of Msschusetts Lowell MA 1854, U.S.A. Abstrct Clustering forms nturl groupings of dt points tht mximize intr-cluster similrity nd minimize intercluster similrity. Support Vector Clustering (SVC is clustering lgorithm tht cn hndle rbitrry cluster shpes. One of the mjor SVC chllenges is cluster lbeling performnce bottleneck. We propose novel cluster lbeling lgorithm tht relies on pproximte coverings both in feture spce nd dt spce. Comprison with existing cluster lbeling pproches suggests tht our strtegy improves efficiency without scrificing clustering qulity. 1 Introduction 1.1 Clustering Overview Clustering is nturl grouping or unsupervised clssifiction of dt into groups [6]. Most clustering lgorithms use one or combintion of the following techniques: grph-bsed [5, 9, 1, 7], density-bsed [8], model-bsed methods using either sttisticl pproch or neurl network pproch, or optimiztion of clustering criterion function. Constructing cluster boundries is nother populr technique [1, 11]. SVC is similr to boundryfinding clustering method except tht it only finds certin points (clled support vectors on the boundry of ech cluster. 1.2 Support Vector Clustering As with support vector mchines [4], SVC uses nonliner mpping of the dt from the dt spce into high-dimensionl feture spce. Wheres support vector mchines use liner seprtor in feture spce in order to seprte nd clssify dt points, SVC uses miniml hypersphere encompssing feture spce imges of dt points [2, 3]. The miniml hypersphere corresponds to severl disjoint bounded regions in the dt spce tht re contours nd re interpreted s clusters. The cluster lbeling chllenge of SVC is to ssocite ech dt point with cluster, using only the vilble opertions, without explicitly constructing the contours in dt spce. Given finite set X R d of N distinct dt points, the miniml hypersphere of rdius R enclosing ll dt points imges in the feture spce cn be described by the following, s in [2]: (1.1 Φ(x 2 R 2 x X, where Φ is nonliner mpping from dt spce to feture spce, Φ(x is the feture spce imge of dt point x, is the Eucliden norm, nd is the center of the sphere. Therefore, dt points cn be ctegorized into three groups bsed on the loction of their imges in feture spce: (1 points whose imges re on the surfce of the miniml hypersphere re Support Vectors (SVs, (2 points whose imges re outside of the miniml hypersphere re Bounded Support Vectors (BSVs, nd (3 points whose imges re inside the miniml hypersphere. The mpping from dt spce to feture spce is determined by kernel function, K : X X R, tht defines the inner product of imge points. As in other SVC literture [1, 2], we use Gussin kernel given by Eq. 1.2 below: (1.2 K(x i, x j = e q xi xj 2 = Φ(x i Φ(x j, where q is the width of the Gussin kernel. Eq. 1.2, it immeditely follows tht, (1.3 K(x i, x i = 1. From This implies tht ll dt point imges re on the surfce of the unit bll in feture spce. Cluster lbeling, the focus of this pper, hs gol of grouping together dt points tht re in the sme contour obtined from the miniml hypersphere. Prior work is discussed in Section 1.3. In tht work, cluster lbeling hs been n SVC performnce bottleneck. 1.3 Cluster Lbeling Trditionl SVC cluster lbeling lgorithms re Complete Grph (CG [2], Support Vector Grph (SVG [2], Proximity Grph (PG 482

2 [7], nd Grdient Descent (GD [1]. All these lgorithms group together dt points by representing pirs of dt points using n djcency structure (typiclly mtrix. Ech element records whether there is sufficient evidence to conclude tht the ssocited pir of dt points is in the sme contour nd therefore the sme cluster. Ech connected component in the djcency mtrix is regrded s cluster. CG requires O(N 2 sized djcency structure becuse it represents ll dt point pirs. SVG represents pirs in which one point is support vector, so its djcency structure only uses O(N sv N spce, where N sv is the number of support vectors. PG forms n djcency structure from proximity grph tht hs only O(N edges. The GD method finds Stble Equilibrium Points (SEPs tht re the nerest minimum point (of Eq. 1.4 below for ech dt point nd then tests pirs of SEPs. GD s djcency structure therefore uses O(Nsep 2 spce, where N sep is the number of SEPs. R 2 (x = Φ(x 2 (1.4 = 1 2 j β j K(x j, x + i,j β i β j K(x i, x j, where x R d, β i is the Lgrnge multiplier for x i, nd = i β iφ(x i is the center of the miniml hypersphere [2]. Deciding if pir of dt points x i nd x j is in the sme contour is problemtic becuse dt spce contours cnnot be explicitly constructed due to the nture of the feture spce mpping Φ. The forementioned lgorithms therefore use n indirect strtegy tht relies on the fct tht every pth connecting two dt points in different dt spce contours exits the contours in dt spce nd the imge of the pth in feture spce exits the miniml hypersphere. The lgorithms use, s pth in dt spce, the stright line segment x i x j connecting x i nd x j. The line segment x i x j is smpled. For smple point x in dt spce, Φ(x is outside the miniml hypersphere if R 2 (x > R 2, where R 2 (x is defined by Eq If the imge of every smple point long x i x j is inside the miniml hypersphere, then x i nd x j re determined to be in the sme contour (hence cluster. Unfortuntely, the smpling pproch cretes running time versus ccurcy trdeoff. If m is the number of smple points long line segment, then solving Eq. 1.4 for ech smple point introduces multiplictive fctor of mn sv beyond the time proportionl to the size of the djcency structure. The lgorithms limit m to be smll constnt (typiclly 1 to 2 in order to limit running time. However, smll vlues of m cn lso cuse two types of errors, flse positive nd flse negtive, s shown in Figure 1. xi ( xj Figure 1: Problems in using line segment x i x j, depicted in dt spce: ( smple points on x i x j re ll inside the miniml hypersphere lthough x i nd x j re in different contours; (b ll smple points re outside the miniml hypersphere lthough x i nd x j re in the sme contour. 1.4 Overview This pper voids the cluster lbeling problems listed in Section 1.3 by using novel pproch tht decides if two dt points re in the sme cluster without smpling pth between them. The min ide of this pper is to try to cover key portion of the miniml hypersphere in feture spce using cones tht re nchored t ech support vector in the feture spce nd lso correspond to hyperspheres in dt spce. The union of the hyperspheres forms n pproximte covering of the dt spce contours. The union need not be constructed; pirs of support vectors cn be quickly tested during the cluster lbeling process nd then the remining dt points cn be esily clustered. The lgorithm 1, presented in Section 2, does not use smple points. It works for dt sets of rbitrry dimension, nd hs been tested for up to 2 dimensions. The results in Section 3 show tht our new lgorithm is fster thn trditionl cluster lbeling lgorithms. 2 Cone Cluster Lbeling 2.1 High-Level Approch We provide new cluster lbeling lgorithm tht is quite different from trditionl methods. For given kernel width vlue, our method does not smple pth between two dt points in order to judge if they belong in the sme cluster. Insted, we leverge the geometry of the feture spce to help perform cluster lbeling in the dt spce. First we find n pproximte covering for the miniml hypersphere in feture spce. This is described in Section 2.2. Strictly speking, the covering is not for the miniml hypersphere but for the intersection P between the surfce of the unit bll nd the miniml hypersphere. The pproximte covering consists of union of cone-shped regions. One region is ssocited with ech support vector s feture spce imge. Let V = {v i v i is support vector, 1 i N sv } be the set of SVs for given q vlue. The region for support vector v i is clled support vector cone nd is denoted by E vi. xi xj (b 1 An erlier version of this work ppers in [14]. 483

3 We cll our lgorithm Cone Cluster Lbeling (CCL becuse of its relince on cones. Let B be the surfce of the feture spce unit bll. A hypersphere S vi centered on v i in the dt spce mps to subset of E vi B. Section 2.3 derives the rdius of S vi, which is shown to be the sme for ll support vectors. Hving only one rdius contributes to the speed of CCL. The union i (S vi is n pproximte covering of the dt spce contours P, where Φ(P P. However, the union is not explicitly constructed. Rther, cluster lbeling is done in two phses, s described in Section 2.4. The first phse clusters support vectors while the second clusters the remining dt points. We regrd two support vectors v i nd v j s connected if their hyperspheres overlp: (S vi S vj. Forming the trnsitive closure of the connected reltion yields set of support vector clusters. The finl step is clustering for dt points tht re not support vectors. For ech such dt point, we ssign the closest support vector s cluster lbel. Proofs re in [14]. 2.2 Approximte Covering in Feture Spce This section forms collection of support vector cones tht pproximtely cover P. Let i = (Φ(v i O, where O is the feture spce origin nd is the center of the miniml hypersphere (see Figure 2(. In feture spce, ech support vector hs its own cone E vi tht covers prt of P. We define the support vector cone E vi to be the infinite cone with xis Φ(v i nd bse ngle i. Lemm 2.1 below shows tht i = j = for ll v i, v j V. Lemm 2.1. (Φ(v i O = (Φ(v j O, v i, v j V. To pproximtely cover P, we denote by the intersection of with the surfce of the unit bll (see Figure 2(b. The point is common point of intersection for ll the support vector cones. Thus, (( i (E vi P P. 2.3 Approximte Covering in Dt Spce To pproximtely cover P in dt spce using support vector cones, we find hypersphere S vi in dt spce ssocited with ech E vi in feture spce. Since ll support vector cones hve the sme bse ngle, ll S vi hve the sme rdius nd ech is centered t v i. Lemm 2.2. Ech dt point whose imge is inside (E vi P is t distnce v i g i from v i, where g i R d is such tht (Φ(v i OΦ(g i =. The clim implies tht (E vi P corresponds to hypersphere S vi in the dt spce centered t v i with rdius v i g i. Since ( i (E vi P pproximtely covers P, i (S vi pproximtely covers P. The next tsk is to obtin v i g i in dt spce. Becuse Φ(v i = 1 =, cos( = cos( (Φ(v i OΦ(g i = cos( (Φ(v i O = Φ(v i. Thus, we cn solve for v i g i s follows: (2.5 v i g i = ln(cos(. q Note tht becuse = (Φ(v i O is the sme for ll v i V, ll S vi hve the sme rdii. We therefore denote v i g i by Z. We now need to obtin Φ(v i in order to clculte cos(. To do this, we first show in Lemm 2.3 tht Φ(v i =, v i V (see Figure 2 (c. We then show tht = 1 R 2 (see Figure 2 (d. Lemm 2.3. Φ(v i =, v i V. Corollry 2.1. Φ(v i is orthogonl to. Lemm 2.4. Φ(v i = 1 R 2 = 2, v i V. (2.6 Consequently, we hve: Z = ln( 1 R 2. q 2.4 Assign Cluster Lbels Tble 1 below shows the CCL lgorithm. For the given q vlue, it first computes Z using Eq Next, support vectors re clustered by finding connected components in the resulting djcency structure. Connected components cn be efficiently found using stndrd lgorithm such s Depth First Serch. Ech connected component corresponds to cluster. Therefore, the output of FindConnComponents( is n rry of size N tht contins cluster lbels. Finlly, the remining dt points re clustered. Ech is ssigned the cluster lbel of the nerest support vector. CCL(X,q,V compute Z for q using Eq. 2.6 AdjcencyMtrix Construct connectivity mtrix // ssign cluster lbels to the support vectors Lbels FindConnComponents(AdjcencyMtrix // ssign cluster lbel to the rest of dt for ech x X, where x / V idx find the nerest SV from x Lbels[x] Lbels[x idx ] end for return Lbels Tble 1: Min lgorithm of Cone Cluster Lbeling 484

4 ' Φ ( v j Φ ( v j ( (b R R Φ ( v j ' R R Φ ( v j Γ ' = Φ( v i... Φ( v i ' = Φ( ' v i (c (d Figure 2: Developing n pproximte cover of prt of P, where v i nd v j re support vectors nd Φ(v i nd Φ(v j re their feture spce imges, respectively. ( = (Φ(v i O= (Φ(v j O. (b is intersection of with the surfce of the unit bll. (c the length is < Φ(v i >. (d = < Φ(v i > = 1 R 2 = 2. Note this is only two-dimensionl illustrtion; the ctul feture spce is high-dimensionl. The worst-cse symptotic running time complexity for FindConnComponents( is in O(Nsv. 2 The time complexity of the CCL for loop is in O((N N sv N sv. Therefore, this lgorithm uses time in O(NN sv for ech q vlue. Note tht, unlike previous cluster lbeling lgorithms, this time does not depend on number of smple points. Detiled execution time comprisons re given in Section 3. 3 Results CCL provides high-qulity clustering using less computtion time thn existing cluster lbeling lgorithms. We compre three different types of execution times s well s totl time of CCL with respect to 4 cluster lbeling pproches introduced in Section 1.3: CG, SVG, PG, nd GD. Times re for 1 preprocessing nd constructing the djcency mtrix, 2 finding connected components from the djcency mtrix (this clusters some dt points, nd 3 clustering remining dt points. Worst-cse symptotic time complexity nd ctul running time comprisons re given in Tble 2 with the dt set of Figure 3. The totl of the three ctul execution times is me- Figure 3: Left: dtset (N =98, right: cluster result (2 clusters. Dt is subset of dt from the uthors of [1]. sured nd divided by the number of q vlues 2 to compute verge times. Since cluster lbeling lgorithm receives q s n input nd produces cluster result, the verge time is pproprite s mesurement for execution time comprison. To test high-dimensionl dtsets without outliers or strongly overlpping clusters, we creted dtsets with different dimensions. CCL worked well with these high-dimensionl dt. Detils of these nd ll of our two nd three-dimensionl experiments cn be found t [14]. 2 A list of q vlues is generted by our method in [13]. 485

5 CG SVG PG GD CCL Adjcency Mtrix Size O(N 2 O(N svn O(N 2 O(N 2 sep O(N 2 sv Preprocessing O(NlogN O(mN 2 k O(1 Adjcency O(mN 2 N sv O(mNNsv 2 O(mNN sv O(mNsepN 2 sv O(Nsv 2 Connected Component O(N 2 O(N sv N O(N 2 O(Nsep 2 O(Nsv 2 Remining Clustering O((N N sv N sv O(N N sep O((N N sv N sv Totl Asymptotic Time O(mN 2 N sv O(mNNsv 2 O(N 2 + mnn sv O(mN 2 (k + N sv O(NN sv Preprocessing Adjcency Connected Component Remining Clustering Totl Execution Time Tble 2: Worst-cse symptotic running times for single q vlue nd verge execution times for 22 q vlues in seconds, where k is the number of itertions for GD to converge to SEP. 4 Conclusion nd Future Work Existing SVC cluster lbeling lgorithms, such s CG, SVG, PG, nd GD, smple line segment to decide whether pir of dt points is in the sme cluster. This cretes trdeoff between cluster lbeling time nd clustering qulity. Our Cone Cluster Lbeling method uses novel covering pproch tht voids smpling. Using the geometry of the feture spce, we find n pproximte covering for the intersection of the miniml hypersphere nd the surfce of the unit bll. This mps to n pproximte cover of the contours in dt spce. The cover uses hyperspheres in dt spce, centered on support vectors. Without constructing the union of these hyperspheres, dt points re clustered quickly nd effectively. Cone Cluster Lbeling quickly produces highqulity clusterings. Our experiments suggest tht it opertes well even in high dimensions. Future work will seek tighter nd more complete coverge in the feture spce nd dt spce. We would lso like to reduce the size of the kernel width vlue t which this method produces its best clustering. References [1] J. Lee nd D. Lee, An Improved Cluster Lbeling Method for Support Vector Clustering, IEEE Trnsctions on pttern nlysis nd mchine intelligence, 27 (25, pp [2] A. Ben-Hur, D. Horn, H. Siegelmnn, nd V. Vpnik, Support vector clustering, Journl of Mchine Lerning Reserch 2, 21, pp [3] B. Schölkopf, O. Burges, nd V. Vpnik, Extrcting support dt for given tsk, First Interntionl Conference on Knowledge Discovery nd Dt Mining, 1995, pp [4] C. J. Burges, A Tutoril on Support Vector Mchines for Pttern Recognition, Dt Mining nd Knowledge Discovery, 2 (1998, pp [5] E. Hrtuv nd R. Shmir,A clustering lgorithm bsed on grph connectivity, Informtion Processing Letters, 76 (2, pp [6] A. Jin, N. Murty, nd P. Flynn, Dt Clustering: A Review, ACMCS, 31 (1999, pp [7] J. Yng, V. Estivill-Cstro, nd S. Chlup, Support Vector Clustering Through Proximity Grph Modeling, Proceedings, 9th Interntionl Conference on Neurl Informtion Processing (ICONIP 2, 22, pp [8] M. Ester, H. Kriegel, J. Snder, nd X. Xu, A densitybsed lgorithm for discovering clusters in lrge sptil dtbses with noise, Proc. 2nd Int. Conf. Knowledge Discovery nd Dt Mining(KDD 96, Portlnd, OR, 1996, pp [9] I. Jonyer, L. Holder, nd D. Cook, Grph-Bsed Hierrchicl Conceptul Clustering, Interntionl Journl on Artificil Intelligence Tools, 1 (21, pp [1] D. Hrel nd Y. Koren, Clustering sptil dt using rndom wlks, Knowledge Discovery nd Dt Mining (KDD 1, 21, pp [11] V. Estivill-Cstro nd I. Lee, Automtic Clustering vi Boundry Extrction for Mining Mssive Pointdt sets, In Proceedings of the 5th Interntionl Conference on Geocomputtion, 2. [12] C. Blke nd C. Merz, UCI repository of mchine lerning dtbses, [13] S. Lee nd K. Dniels, Gussin Kernel Width Genertor for Support Vector Clustering, Interntionl Conference on Bioinformtics nd its Applictions, 24, pp [14] S. Lee, Gussin Kernel Width Selection nd Fst Cluster Lbeling for Support Vector Clustering, Doctorl Thesis nd Tech. Report 25-9, Deprtment of Computer Science, University of Msschusetts Lowell,