CLUSTERING that discovers the relationship among data

Transcription

1 Ramp-based Twn Support Vector Clusterng Zhen Wang, Xu Chen, Chun-Na L, and Yuan-Ha Shao arxv: v [cs.lg] 0 Dec 208 Abstract Tradtonal plane-based clusterng methods measure the cost of wthn-cluster and between-cluster by quadratc, lnear or some other unbounded functons, whch may amplfy the mpact of cost. Ths letter ntroduces a ramp cost functon nto the plane-based clusterng to propose a new clusterng method, called ramp-based twn support vector clusterng (RampTWSVC). RampTWSVC s more robust because of ts boundness, and thus t s more easer to fnd the ntrnsc clusters than other planebased clusterng methods. The non-convex programmng problem n RampTWSVC s solved effcently through an alternatng teraton algorthm, and ts local soluton can be obtaned n a fnte number of teratons theoretcally. In addton, the nonlnear manfold-based formaton of RampTWSVC s also proposed by kernel trck. Expermental results on several benchmark datasets show the better performance of our RampTWSVC compared wth other plane-based clusterng methods. Index Terms Nonlnear clusterng, plane-based clusterng, ramp cost, twn support vector machnes, unsupervsed learnng. I. INTRODUCTION CLUSTERING that dscovers the relatonshp among data samples, s one of the most fundamental problems n machne learnng [] [4]. It has been appled to many realworld problems, e.g., marketng, text mnng, and web analyss [5], [6]. In partcular, the partton clusterng methods [], [7] are wdely used n real applcaton for ther smplcty, e.g., the classcal kmeans [8] wth ponts as cluster centers, the k-plane clusterng (kpc) [9] and proxmal-plane clusterng (PPC) [0], [] wth planes as cluster centers. As an extenson of pont center, the plane center has the ablty to dscover comprehensve structures n the sample space. The plane-based clusterng seeks the cluster centers dependng on the current cluster assgnment. When a cluster center s constructed, the smlarty of wthn-cluster s ntensfed (n some methods, the dssmlarty of between-cluster s also ntensfed smultaneously). Therefore, the noses or outlers would sgnfcantly nfluence the cluster centers n plane-based clusterng. For nstance, kpc mnmzes the cost of wthncluster by a quadratc functon, and PPC mnmzes the cost of wthn-cluster and between-cluster by the same one. Subsequently, the twn support vector clusterng (TWSVC) [2] was Submtted n December, 208. Ths work s supported n part by Natonal Natural Scence Foundaton of Chna (Nos. 5030, , 8783, and ), n part by Natural Scence Foundaton of Hanan Provnce (No. 8QN8), and n part by Scentfc Research Foundaton of Hanan Unversty (No. kyqd(sk)804). Zhen Wang s wth the School of Mathematcal Scences, Inner Mongola Unversty, Hohhot, 0002 P.R.Chna (e-mal: wangzhen@mu.edu.cn). Xu Chen s wth the School of Mathematcal Scences, Inner Mongola Unversty, Hohhot, 0002 P.R.Chna (e-mal: pohuozhe@63.com). Chun-Na L s wth the Zhjang College, Zhejang Unversty of Technology, Hangzhou 30024, P.R.Chna (e-mal: na03na@63.com). Yuan-Ha Shao (*Correspondng Author) s wth the School of Economcs and Management, Hanan Unversty, Hakou, , P.R.Chna (e-mal: shaoyuanha2@63.com). (a) Wthn-cluster (b) Between-cluster Fg.. Functons used n kpc, PPC, TWSVC, RTWSVC, FRTWSVC, and RampTWSVC to measure the cost of wthn-cluster and between-cluster. The horzontal axs denotes the devaton of a sample from the cluster center, and the vertcal one denotes the cost to ft the sample. When a cost value s negatve, the cost becomes a reward. proposed, whch hred a pecewse lnear functon to measure the cost of between-cluster but perssted n usng the quadratc functon for the wthn-cluster. Recently, another plane-based clusterng method, called robust twn support vector clusterng (RTWSVC) [3], was proposed by hrng a lnear functon to measure the cost of wthn-cluster and between-cluster. Both TWSVC and RTWSVC reduce the nfluence of noses or outlers to some extent. The ramp functon [4], whch has been appled n semsupervsed and supervsed learnng successfully [5] [7], s a bounded pecewse lnear functon. Therefore, n ths letter, we propose a ramp-based twn support vector clusterng method (RampTWSVC) to further reduce the nfluence of noses or outlers both from wthn-cluster and between-cluster, by ntroducng the ramp functon n the constructon of the cluster center planes. The problem of RampTWSVC s a non-convex programmng problem, and t s recast to a mxed nteger programmng problem. We propose an teratve algorthm to solve the mxed nteger programmng problem, and we prove that the algorthm termnates n a fnte number of teratons at a local soluton. In addton, RampTWSVC s extended to nonlnear case by kernel trck to cope wth the manfold clusterng [8], [9]. Fg. exhbts the cost functons used n several plane-based clusterng methods, where FRTWSVC [3] s a plane-based clusterng method called fast robust twn support vector clusterng. It s obvous from Fg. that only our RampTWSVC uses the bounded cost functons both n wthn-cluster and between-cluster, whch can reduce the nfluence of noses or outlers much more than other methods. Expermental results on the benchmark datasets show the better performance of the proposed RampTWSVC compared wth other plane-based clusterng methods.

2 2 II. REVIEW OF PLANE-BASED CLUSTERING In ths paper, we consdermdata samples{x, x 2,...,x m } n the n-dmensonal real vector space R n. Assumng these m samples belong to k clusters wth ther correspondng labels n {, 2,...,k}, and they are represented by a matrx X = (x,x 2,...,x m ) R n m. We further organze the samples from X wth the current label nto a matrx X and those wth the rest labels nto a matrx ˆX, where =, 2,...,k. For readers convenence, the symbols X, X, and ˆX wll also refer to the correspondng sets, dependng on the specfc context they appear. For example, the symbol X can be comprehended as a matrx belongng to R n m or a set that contans m samples. The th cluster center plane ( =,...,k) s defned as f (x) = w x+b = 0, () where w R n and b R. The followng plane-based clusterng methods share the same kmeans-lke clusterng procedure. Startng from an ntal assgnment of the m samples nto k clusters, all the cluster center planes () are constructed by the current cluster assgnment. Once obtaned all the cluster center planes, the m samples are reassgned by y = argmn w x+b, (2) where denotes the absolute value. The cluster center planes and the sample labels are updated alternately untl some termnate condtons are satsfed. In the followng, we brefly descrbe the dfferent constructons of the cluster center plane by kpc, PPC, TWSVC, and RTWSVC. A. kpc and PPC kpc [9] wshes the cluster center plane close to the current cluster samples. Further on, PPC [0] consders t should also be far away from the dfferent cluster samples. Therefore, the th ( =,...,k) cluster center plane n PPC s constructed by solvng the followng problem mn X w +b e 2 c ˆX w +b e 2 w,b s.t. w 2 =, where denotes the L 2 norm, e s a column vector of ones wth approprate dmenson, and c > 0 s a user-set parameter. The optmzaton problem n kpc s just of the frst term of the objectve of (3). From the objectve of (3), t s obvous that a sample from the current cluster receves a quadratc cost, and a sample from a dfferent cluster receves a quadratc reward. Therefore, noses or outlers from the current cluster or dfferent clusters wll have great mpact on the potental cluster center plane. B. TWSVC and RTWSVC PPC may obtan a cluster center plane whch s far from the current cluster, because the samples from dfferent clusters receve hgh rewards when they are far away from the cluster center plane. In contrast, TWSVC [2] degrades the reward of (3) the samples from dfferent clusters by consderng the problem wth =,...,k as mn w 2 X w +b e 2 +ce (e ˆX w +b e ) +, (4),b where ( ) + replaces the negatve part by zeros. From the second part of (4), t s avalable that a sample wth a devaton n[0,) has mpact on the cluster center plane. Thus, TWSVC s more robust than PPC. However, the ssue of current cluster also exsts because of the quadratc cost n the frst part of (4). Thus, RTWSVC [3] was proposed to decreases the nfluence of current cluster by replacng the L 2 norm n (4) wth L norm. RTWSVC nherts the advantage of TWSVC and decreases the requrement from current cluster. However, the cost of RTWSVC from wthncluster s unbounded from Fg.. In order to eradcate the nfluence of noses or outlers, t s reasonable to hre a bounded functon for the wthn-cluster, whose prncple s smlar to the cost for the between-cluster used n TWSVC. III. RAMPTWSVC Smlar to the above plane-based clusterng methods mentoned n secton 2, our RampTWSVC starts wth an ntal sample labels, then computes each cluster center plane for the current sample labels teratvely, untl some termnate condtons are satsfed. In the followng, we consder to obtan one of the cluster center planes for the gven samples wth ther labels. A. Formaton To obtan the th ( =,...,k) cluster center plane, our RampTWSVC consders the followng problem mn w 2 ( w 2 +b 2 )+c R (x j )+c 2 R 2 (x j ),,b x j X x j ˆX where c > 0 and c 2 > 0 are parameters. R (x) and R 2 (x) are two pecewse lnear functons w.r.t. the devaton f (x) = w x+b (see Fg. ) as 0 f f (x), R (x) = s f f (x) 2 s, f (x) + otherwse, (6) 2+2 f f (x) s, R 2 (x) = + s f f (x) +, f (x) +2+2 s otherwse, (7) where [0,) ands (,0] are two parameters to control the functon form (typcally, we set = 0.3 and s = 0.2 n ths letter). It s obvous that both of the cost functons R (x) (for the current cluster X ) and R 2 (x) (for the dfferent clusters ˆX ) have bounds for large devaton. Thus, the noses or outlers much further from the cluster center plane do not have greater mpact on the cluster center plane when they meet the bound. (5)

3 3 The above property ndcates our RampTWSVC s more robust than RTWSVC. In the followng, we extend the RampTWSVC to nonlnear manfold clusterng, and the solutons to the problems n lnear and nonlnear RampTWSVC are elaborated n next subseton. The plane-based clusterng method can be extend to nonlnear manfold clusterng easly by the kernel trck [20], [2]. By ntroducng a pre-defned kernel functon K(, ), the planebased nonlnear clusterng seeks k cluster center manfolds n the kernel generated space as g (x) = K(x,X) w +b = 0, =,...,k. (8) Then, the nonlnear RampTWSVC consders to ntroduce the ramp functons nto the plane-based nonlnear clusterng. By replacng f (x) wth g (x) n (6) and (7), and substtutng them nto (5), one can easly obtan k optmzaton problems to construct the cluster center manfolds (8). When we obtan the k cluster centers (8), a sample x s assgned to whch cluster dependng on y = argmn K(x,X) w +b. (9) The procedure of the nonlnear case s the same as the lnear one, so the detals are omtted. B. Soluton In ths subsecton, we study the soluton to the problem (5). The correspondng problem n nonlnear RampTWSVC s smlar to the one n lnear case. For convenence, let u = (w,b ), Z be a matrx whose jth column z j s x j wth an addtonal feature (where the correspondng x j belongs to the th cluster), and Ẑ be a matrx whose column smlar as z j (where the correspondng x j does not belongs to the th cluster). Then, the problem (5) s recast to mn u 2 u j 2 +c e ( + Z u ) + +c e ( + +Z u ) + +c 2 e (+ Ẑ u ) + +c 2 e (+ +Ẑ u ) + c e (s 2+ Z u ) + c e (s 2+ +Z u ) + c 2 e (s Ẑ u ) + c 2 e (s+ẑ u ) +. (0) It s easy to see that the above problem s a non-convex programmng problem because of the concave part ( ) +. By ntroducng two auxlary vectors p {,0,} m and p 2 {,0,} m m (where m s the sample number of the current th cluster), the above problem s equvalent to the followng mxed-nteger programmng problem mn u 2 u 2 +c e ( + Z u ) +,p,p 2 +c e ( + +Z u ) + +c 2 e (+ Ẑ u ) + +c 2 e (+ +Ẑ u ) + +c p Z u +c 2 p 2 Ẑ u f zj u > 2 s, s.t. p (j) = f zj u < 2+ +s, z j Z p 2 (j) = 0 otherwse, f zj u > s, f zj u < s, 0 otherwse, z j Ẑ () TABLE I DETAILS OF THE BENCHMARK DATASETS Data m n k (a) Arrhythma (b) Dermatology (c) Ecol (d) Glass (e) Irs (f) Lbras (g) Seeds (h) Wne () Zoo (j) Bupa (k) Echocardogram (l) Heartstatlog (m) Housevotes (n) Ionosphere (o) Sonar (p) Soybean (q) Spect (r) Wpbc *m s the number of samples, n s the one of dmenson, and k s the one of classes. where p (j) and p 2 (j) are the correspondng jth elements of p and p 2, respectvely. Here, we propose an alternatng teraton algorthm to solve the mxed-nteger programmng problem (). Startng wth an ntalzed u (0), t s easy to calculate p (0) and p (0) 2 by the constrants of (). For fxed p (t ) and p (t ) 2 (t =,2,...), the problem () becomes to an unconstraned convex problem and ts soluton can be obtaned by many algorthms easly (e.g., sequental mnmal optmzaton (SMO) [22] and fast Newton-Amjo algorthm [23]). Once obtaned u (t), p (t) and p (t) 2 are updated agan. The loop wll be contnued untl the objectve of () does not decrease any more. Theorem III.. The above alternatng teraton algorthm to solve () termnates n a fnte number of teratons at a local optmal pont, where a local optmal pont of the mxed nteger programmng problem () s defned as the pont (u,p,p 2) f u s the global soluton to the problem () wth fxed (p,p 2 ) and vce versa. Proof. From the procedure of the alternatng teraton algorthm, t s obvous that the global solutons to the problem () wth fxed u or (p,p 2 ) are obtaned n teraton. Snce there s a fnte number of ways to select p and p 2, there are two fnte numbers r,r 2 > 0 such that (p (r) 2 ) = (p (r2),p (r2) 2 ). Thus, we have u (r) = u (r2). That s to say, the objectve values are equal n the r th and r 2 th teratons. Snce p Z u 0 and p 2 Ẑ u 0 are always holds, the objectve value of () keeps non-ncreasng n teraton. Therefore, the objectve s nvarant after the r th teraton, and then the algorthm would termnate at the r th teraton. Let us consder the pont (u (r),p (r) 2 ). From the above proof, we have G(u (r),p (r) 2 ) = G(u (r),p (r+),p (r+) 2 ), where G( ) s the objectve value of (). If there are more than one global soluton to the problem () wth fxed u, we always select the same one for the same u. Thus, we have (p (r) 2 ) = (p (r+),p (r+) 2 ), 2 ) s a local optmal whch ndcates the pont (u (r) pont.,p (r)

4 4 IV. EXPERIMENTAL RESULTS In ths secton, we analyze the performance of our RampTWSVC compared wth kmeans [8], kpc [9], PPC [0], TWSVC [2], RTWSVC [3], and FRTWSVC [3] on several benchmark datasets [24]. All the methods were mplemented by MATLAB207 on a PC wth an Intel Core Duo processor (double 4.2 GHz) wth 6GB RAM. The parameters c n PPC, TWSVC, RTWSVC, FRTWSVC, and c, c 2 n RampTWSVC were selected from {2 = 8, 7,...,7}. For nonlnear case, the Gaussan kernel K(x,x 2 ) = exp{ µ x x 2 2 } [20] was used, and ts parameter µ was selected from {2 = 0, 9,...,5}. The random ntalzaton was used on kmeans, and the nearest neghbor graph (NNG) ntalzaton [2] was used on other methods. In the experments, we used the metrc accuracy (AC) [2] and mutual nformaton (MI) [25] to measure the performance of these methods. Table I shows the detals of the benchmark datasets. Tables II and III exhbt the lnear and nonlnear clusterng methods on the benchmark datasets, respectvely. The hghest metrcs among these methods on each dataset are n bold. Besdes, we also reported the statstcs of these methods n the last rows n Tables II and III, whch s the number of the datasets that each method s the hghest one n terms of AC, MI, or both. From Table II, t can be seen that our lnear RampTWSVC performs better than other lnear methods on fve datasets n terms of both AC and MI, and t s more accurate than other methods on other fve datasets. On the rest eght datasets, our lnear RampTWSVC s also compettve wth the best one. From Table III, t s obvous that our nonlnear RampTWSVC has much hgher AC and MI over other methods on many datasets. V. CONCLUSIONS A plane-based clusterng method (RampTWSVC) has been proposed wth the ramp functon. It contans both the lnear and nonlnear formatons. The cluster center planes n RampTWSVC are obtaned by solvng a seres of non-convex problems, and ther local solutons are guaranteed by a proposed alternatng teraton algorthm n theory. Expermental results on several benchmark datasets have ndcated that our RampTWSVC performs much better than other plane-based clusterng methods on many datasets. For practcal convenence, the correspondng RampTWSVC Matlab code has been uploaded upon Future work ncludes the parameter regulaton and effcent solver desgn for our non-convex problems. REFERENCES [] A. Jan, M. Murty, and P. Flynn, Data clusterng: a revew, ACM computng surveys (CSUR), vol. 3, no. 3, pp , 999. [2] Z. Kang, C. Peng, and Q. Cheng, Robust subspace clusterng va smoothed rank approxmaton, IEEE Sgnal Processng Letters, vol. 22, no., pp , 205. [3] A. Malhotra and I. Schzas, Mlp-based unsupervsed clusterng, IEEE Sgnal Processng Letters, vol. 25, no. 2, pp , 208. [4] Z. Zhou, A bref ntroducton to weakly supervsed learnng, Natonal Scence Revew, vol. 5, no., pp , 207. TABLE II LINEAR CLUSTERING ON BENCHMARK DATASETS kmeans kpc PPC TWSVC RTWSVC FRTWSVC Ours Data AC(%) AC(%) AC(%) AC(%) AC(%) AC(%) AC(%) MI(%) MI(%) MI(%) MI(%) MI(%) MI(%) MI(%) (a) 65.72± ± (b) 69.76± ± (c) 82.9± ± (d) 65.58± ± (e) 84.57± ± (f) 90.84± ± (g) 87.35± ± (h) 7.06± ± () 87.49± ± (j) 50.39± ± (k) 66.4± ± (l) 5.45± ± (m) 78.83± ± (n) 58.89± ± (o) 50.22± ± (p) 93.4± ± (q) 52.97± ± (r) 56.03± ± AC MI Both TABLE III NONLINEAR CLUSTERING ON BENCHMARK DATASETS kmeans kpc PPC TWSVC RTWSVC FRTWSVC Ours Data AC(%) AC(%) AC(%) AC(%) AC(%) AC(%) AC(%) MI(%) MI(%) MI(%) MI(%) MI(%) MI(%) MI(%) (a) 47.32± ± (b) 7.66± ± (c) 79.93± ± (d) 69.27± ± (e) 87.63± ± (f) 90.60± ± (g) 87.02± ± (h) 52.07± ± () 87.4± ± (j) 5.08± ± (k) 7.4± ± (l) 50.83± ± (m) 79.79± ± (n) 62.32± ± (o) 50.6± ± (p) 00.0± ± (q) 60.68± ± (r) 63.40± ± AC MI Both

5 5 [5] M. Berry, Survey of Text Mnng I: Clusterng, Classfcaton, and Retreval. Sprnger, 2004, vol.. [6] R. Iln, Unsupervsed learnng of categorcal data wth competng models, Neural Networks and Learnng Systems, IEEE Transactons on, vol. 23, no., pp , 202. [7] P. Tan, M. Stenbach, and V. Kumar, Introducton to Data Mnng, (st Edton). Boston, MA, USA: Addson-Wesley Longman Publshng Co., Inc., [8] A. Jan and R. Dubes, Algorthms for clusterng data. Prentce-Hall, Inc., 988. [9] P. Bradley and O. Mangasaran, k-plane clusterng, Journal of Global Optmzaton, vol. 6, no., pp , [0] Y. Shao, L. Ba, Z. Wang, X. Hua, and N. Deng, Proxmal plane clusterng va egenvalues, Proceda Computer Scence, vol. 7, pp. 4 47, 203. [] Y. Shao, Y. Guo, Z. Wang, Z. Yang, and N. Deng, k-proxmal plane clusterng, Internatonal Journal of Machne Learnng and Cybernetcs, vol. 8, no. 5, pp , 207. [2] Z. Wang, Y. Shao, L. Ba, and N. Deng, Twn support vector machne for clusterng, IEEE transactons on neural networks and learnng systems, vol. 26, no. 0, pp , 205. [3] Q. Ye, H. Zhao, Z. L, X. Yang, S. Gao, T. Yn, and N. Ye, L- norm dstance mnmzaton-based fast robust twn support vector k- plane clusterng, IEEE transactons on neural networks and learnng systems, vol. 29, no. 9, pp , 208. [4] R. Collobert, F. Snz, J. Weston, and L. Bottou, Large scale transductve svms, Journal of Machne Learnng Research, vol. 7, pp , [5] H. Cevkalp, Best fttng hyperplanes for classfcaton, IEEE transactons on pattern analyss and machne ntellgence, vol. 39, no. 6, pp , 207. [6] Y. Tan, M. Mrzabagher, S. Bamakan, H. Wang, and Q. Qu, Ramp loss one-class support vector machne: A robust and effectve approach to anomaly detecton problems, Neurocomputng, vol. 30, no. 8, pp , 208. [7] D. Lu, Y. Sh, Y. Tan, and X. Huang, Ramp loss least squares support vector machne, Journal of computatonal scence, vol. 4, pp. 6 68, 206. [8] R. Souvenr and R. Pless, Manfold clusterng, n Computer Vson, ICCV Tenth IEEE Internatonal Conference on, vol.. IEEE, 2005, pp [9] W. Cao and R. Haralck, Nonlnear manfold clusterng by dmensonalty, n Pattern Recognton, ICPR th Internatonal Conference on, vol.. IEEE, 2006, pp [20] R. Khemchandan, Jayadeva, and S. Chandra, Optmal kernel selecton n twn support vector machnes, Optmzaton Letters, vol. 3, pp , [2] Z. Wang, Y. Shao, L. Ba, C. L, L. Lu, and N. Deng, Insenstve stochastc gradent twn support vector machnes for large scale problems, Informaton Scences, vol. 462, pp. 4 3, 208. [22] J. Platt, Fast tranng of support vector machnes usng sequental mnmal optmzaton, n Advances n kernel methods-support vector learnng, Cambrdge, MA: MIT Press, 999, pp [23] Z. Wang, Y. Shao, and T. Wu, A ga-based model selecton for smooth twn parametrc-margn support vector machne, Pattern Recognton, vol. 46, no. 8, pp , 203. [24] C. Blake and C. Merz, UCI Repostory for Machne Learnng Databases, mlearn/mlrepostory.html, 998. [25] H. Lu and Y. Fu, Clusterng wth partton level sde nformaton, n In Data Mnng (ICDM), 205 IEEE Internatonal Conference on, IEEE, 205, pp