Flatten a Curved Space by Kernel: From Einstein to Euclid

Transcription

1 Flatten a Curved Space by Kernel: From Ensten to Eucld Quyuan Huang, Dapeng Olver Wu Ensten s general theory of relatvty fundamentally changed our vew about the physcal world. Dfferent from Newton s theory, Ensten s space and tme are not flat but can be warped by matter. For a curved space such as Ensten s space, Eucldean geometry s no longer sutable, and Remannan geometry s usually used nstead. In parallel wth physcs, due to an exploson of data from all felds of scence, there s an ncreasng need for pattern analyss tools, whch are capable of analyzng patterns of data n a non-eucldean (curved) space. To handle data n a curved space, lnear approaches are not drectly applcable, and nstead nonlnear approaches are the rght weapon. However, early-day nonlnear approaches were usually based on gradent descent or greedy heurstcs, and suffered from local mnma and overfttng [1]. In contrast, ernel methods provde a powerful means for transformng data n a non-eucldean curved space (such as Ensten space) nto ponts n a hghdmensonal Eucldean flat space, so that lnear approaches can be appled to the transformed ponts n the hgh-dmensonal Eucldean space. Wth ths flattenng capablty, ernel methods combne the best features of lnear approaches and nonlnear approaches,.e., ernel methods are capable of dealng wth nonlnear structures whle enjoyng a low computatonal complexty le lnear approaches. In ths column, we provde mportant nsghts nto ernel methods and llustrate the power of ernel methods n two mportant pattern analyss problems: feature extracton and clusterng. INSIGHT INTO KERNEL METHODS: A TRANSDUCTIVE PARADIGM A lnear pattern analyss method A can be extended to a ernel method va the followng procedure: 1. Select a ernel sutable for a gven nonlnear pattern analyss problem. A ernel s a functon κ that for all x and z n the data space X, satsfes κ(x,z) = ϕ(x),ϕ(z), where ϕ s a mappng from X to a Hlbert space, and, s an nner product. 2. Gven a tranng data set {x : = 1,2,,N}, calculate the ernel functon, j = κ(x,x j ) for each par of x and x j. The resultng N N matrx K wth entres, j s called the ernel matrx. 3. Tran the gven lnear pattern analyss method A usng the ernel matrx and the tranng data set, and obtan a pattern functon f (x) = N =1 α κ(x,x), where α ( = 1,,N) are obtaned by tranng. The term N =1 α κ(x,x) s called the dual representaton of f (x) [1], and α ( = 1,,N) are called the dual varables. In essence, under dual representaton, f (x) s a lnear combnaton of ernel functons evaluated at each tranng data pont and a gven x. Hence, a ernel method actually conducts transducton,.e., drectly draws conclusons about new data from the tranng data, wthout constructng a model; n other words, transducton s a type of nference from observed, specfc (tranng) cases 1

2 x x x (a) Nonlnear surface n the nput data space (b) Hyperplane n the feature space Fgure 1: Flatten a curved space by a nonlnear mappng ϕ(x). to specfc (test) cases (e.g., a gven x). Ths s dfferent from nducton, whch s a type of nference from specfc (tranng) cases to a general rule/model. Under an nductve paradgm, once the general rule/model s obtaned through learnng, the tranng data wll be dscarded and wll not explctly be part of the general rule/model. Why s a ernel method capable of resolvng nonlnear structures at a low computatonal cost? Frst, the capablty of dealng wth nonlnear structures s due to the use of ϕ(x), whch flattens a curved space. Specfcally, flattenng s acheved by mappng x X to ϕ(x) n a hgh-dmensonal feature space such that the nonlnear structure embedded n {x } becomes a lnear structure n the feature space. For example, a nonlnear surface n X becomes a lnear hyperplane n the feature space after applyng map ϕ(x) (see Fg. 1). Second, low computatonal complexty s due to the dual representaton of pattern functon f (x), also nown as ernel trc,.e., ernel κ(x,x) = ϕ(x ),ϕ(x) can be evaluated wthout computng ϕ(x ) and ϕ(x). Ths s because a ernel can be drectly gven as a functon of x and x wthout explctly defnng ϕ( ). Avodng computng ϕ(x ) and ϕ(x) sgnfcantly reduces the computatonal complexty. For feature extracton problems, f (x) s a feature vector n a feature space. For clusterng problems, f (x) s a cluster ndex. For classfcaton problems, f (x) s a class ndex. For regresson problems, y = f (x) s a regresson functon. For nonlnear system dentfcaton problems, x(t + 1) = f (x(t)) s a system state equaton that governs the dynamcs of a gven nonlnear system. We wll descrbe the frst two pattern functons n the followng sectons. KERNEL-BASED FEATURE EXTRACTION Transformng the nput data x nto a feature vector ϕ(x) n a feature space s called feature extracton. The purpose of feature extracton s to extract relevant nformaton from the nput data. Dmensonalty reducton (.e., removng rrelevant feature dmensons) s usually nvolved n feature extracton. In ths secton, we descrbe two nown technques n the lterature: ernel prncple component analyss (KPCA) and a dscrmnant-learnng-based ernel feature extracton method. KPCA [1] utlzes a dual representaton of an egenvector u j of the covarance matrx of x, so that the projecton P of ϕ(x) onto the drecton u j n the feature space s gven by P u j (ϕ(x)) = 2

3 N =1 α, jκ(x,x). Ths dual representaton enables us to avod computng ϕ(x). KPCA s summarzed below. Input: {x : x R L, = 1,2,,N}, ernel κ(, ), and (the dmenson of the output feature space). 1. Calculate, j = κ(x,x j ) for = 1,2,,N and j = 1,2,,N, and obtan the ernel matrx K R N N, whose -th row and j-th column s, j. 2. Fnd the largest egenvalues {λ j : j = 1,,} and the correspondng egenvectors {v j : v j R N, j = 1,,} of matrx K. 3. Let α, j = v, j / λ j ( = 1,,N and j = 1,,), where v, j s the -th element of v j. 4. Compute x j,m = N =1 α, jκ(x,x m ) (m = 1,,N and j = 1,,), where x j,m s the j-th element of x m. Output: transformed data { x : x R, = 1,,N}. ernel lnear feature extracton (KLFE) [2] s a dscrmnant-learnng-based ernel feature extracton method for supervsed learnng. Dscrmnant-learnng-based feature extracton sees a feature space that maxmzes the dfference between data of dfference classes. Consder supervsed learnng for two classes and assume that the nput data set D conssts of {(x,y ) : = 1,2,,N}, where x R L and the class label y { 1,+1}. LFE [3, 4] sees a lnear transformaton matrx W that maxmzes the dfference between transformed data ponts Wx of dfferent classes. Ths dfference s called a margn, smlar to that n a support vector machne (SVM) [1]. The margn ρ (W) of x under W s defned by ρ (W) = m T Wm h T Wh, where m T Wm s the Mahalanobs dstance between x and ts nearest neghbor n a dfferent class, and h T Wh s the Mahalanobs dstance between x and ts nearest neghbor n the same class. Let m x NM(x,y ), where the nearest mss functon NM(, ) s gven by NM(x,y) argmn x x x p, (1) s.t. (x,y ) D, (2) y y, (3) where x p s l p norm of x. Let h x NH(x,y ), where the nearest ht functon NH(, ) s gven by NH(x,y) argmn x x x p, (4) s.t. (x,y ) D, (5) y = y. (6) The margn-maxmzng W can be found by solvng the followng optmzaton problem: max W N =1 ρ (W), (7) s.t. W 2 F = 1,W 0, 3

4 LFE KLFE PCA Fgure 2: Classfcaton error rate (y-axs) vs. dmenson L (x-axs) of Swss roll data where W F s the Frobenus norm of W. W 0 means that matrx W has to be postve semdefnte. KLFE s a ernel extenson of LFE. Usng a nonlnear mappng ϕ(x) that maps x R L to ϕ(x) R L ( L > L), we can defne m ϕ(x ) ϕ(nm(x,y )) and h ϕ(x ) ϕ(nh(x,y )). Under KLFE, the margn-maxmzng W R L L can be found by solvng max W N =1 ( mt W m h T W h ), (8) s.t. W 2 F = 1, W 0. In [2], a KPCA-based method was proposed to effcently compute nonlnearly transformed ponts x = Wϕ(x ). KLFE can also acheve dmensonalty reducton by choosng the dmensons wth largest varance n the feature space. To compare the performance of KLFE, LFE, and PCA, we use an experment wth synthetc data, a Swss roll (Fg. 1(a)). To generate 3-dmensonal sample ponts x = [x (1),x (2) a Swss roll, we let x (1) = θ cos(θ) and x (2) dstrbuted n [0,4π]; x (3),x (3) ] T ( = 1,,N) on = θ sn(θ), where θ s a random varable unformly s a random varable unformly dstrbuted n [0, 2]; then the 3D sample ponts x ( = 1,,N) are on a 3-D helx surface (Swss roll). To evaluate pattern classfcaton performance under varous feature extracton schemes, we label sample ponts generated by θ [0,2π] wth y = 1 and label sample ponts generated by θ (2π,4π] wth y = 1. The 3-dmensonal vector x s further mapped to a L-dmensonal vector z by matrx R,.e., z = Rx, where matrx R s randomly generated and has dmenson L 3. The purpose of mappng x to z s to add some rrelevant features and test whether a feature extracton scheme s able to perform well under rrelevant features. In ths way, we obtan the nput data set D = {(z,y ) : = 1,2,,N}, where z R L and the class label y { 1,+1}. For each feature extracton scheme, we use K-Nearest-Neghbor (K=1) as the classfer so that we can evaluate the performance of feature extracton n terms of classfcaton error rate. The classfcaton error rates are averaged over 10 smulaton runs, each wth a dfferent matrx R. Fg. 2 shows the classfcaton error rate vs. dmenson L. We can see that KLFE and LFE acheve comparable performance, and both KLFE and LFE outperform PCA. When dmenson L = 1, KLFE acheves better performance than LFE. In addton, KLFE s robust aganst the change of dmenson L, because KLFE has an explct mechansm to elmnate rrelevant features. 4

5 KERNEL-BASED CLUSTERING Clusterng parttons a set of objects nto groups (clusters) so that objects n the same cluster are more smlar (n some sense) to each other than to objects n other clusters. In ths secton, we descrbe three nown clusterng algorthms: ernel K-means, spectral clusterng, and Self-Organzng-Queue (SOQ) based clusterng [5]. The K-means algorthm s a wdely used teratve clusterng algorthm. In each teraton, the K- means algorthm computes a new centrod 1 µ for each cluster and then updates the cluster members usng the new centrods based on the nearest neghbor rule. Kernel K-means [6] s a ernel extenson of K-means algorthm, whch s summarzed below. Input: {x : = 1,,N}, ernel κ(, ), and the number of clusters K. 1. Intalze the K clusters and obtan {C (0) : = 1,,K}, where C (t) denotes the set contanng all the members of Cluster at Stage t. 2. Let t = For each x ( = 1,,N), update ts new cluster ndex by (x ) = argmn ϕ(x ) µ 2 2, where ϕ(x ) µ 2 2 can be computed by ϕ(x ) µ 2 2 = ϕ(x ) 1 C (t) = κ(x,x ) 2 C (t) x C (t) x C (t) ϕ(x) 2 2 (9) κ(x,x) + 1 C (t) 2 x C (t) z C (t) κ(x,z) (10) Snce we assume ponts {ϕ(x ) : = 1,,N} form a lnear geometrc structure n the feature space due to flattenng capablty of ϕ( ), we use the Eucldean dstance n (9) nstead of a geodesc dstance used n a curved space. Agan, n (10), the ernel trc bypasses drect computaton of ϕ(x). 4. Update the membershp of each cluster ( = 1,,K) by C (t+1) = {x : (x ) =, {1,,N}}. 5. If the termnaton crtera are not satsfed, let t = t + 1 and go to Step 3; otherwse, stop. Output: {C (t+1) : = 1,,K}. Spectral clusterng technques are wdely used for graph clusterng [7] or communty detecton [8],.e., fndng sets of related vertces (called communtes) n a graph. Spectral clusterng utlzes the spectrum of the Laplacan matrx L of a gven graph for groupng the nodes, snce the multplcty K of the egenvalue 0 of Laplacan L equals the number of connected components n the graph (denote these connected components by A 1,,A K ), and the egenspace of egenvalue 0 s spanned by the ndcator vectors 1 A1,,1 AK of those components, where ndcator vector 1 A R N, the -th entry 1 The centrod of a cluster s the arthmetc mean poston of all the ponts/members n the cluster 5

6 of whch s 1 f Node belongs to A, and s 0 otherwse. Hence we can use the egenvectors of egenvalue 0 to obtan the ndcator vectors 1 A1,,1 AK, whch s exactly a partton of the graph nto K connected components. Spectral clusterng technques can be categorzed nto unnormalzed and normalzed technques. An unnormalzed spectral clusterng algorthm leverages the spectrum of the unnormalzed Laplacan matrx of a gven graph, whle a normalzed spectral clusterng algorthm leverages the spectrum of the normalzed Laplacan matrx. Spectral clusterng can be regarded as a specal type of weghted ernel K-means [6] snce a weghted ernel K-means scheme can be reduced to an unnormalzed/normalzed spectral clusterng scheme by choosng approprate weght matrx and lettng ernel matrx K = S, where S s an affnty matrx used n spectral clusterng. The followng shows an unnormalzed spectral clusterng algorthm. Input: Affnty matrx S (where S R N N ), and the number of clusters K. 1. Compute the unnormalzed Laplacan matrx L = D S, where D s a dagonal matrx whose dagonal entres are row-sum of S. 2. Compute the K smallest egenvalues and the correspondng egenvectors u 1,,u K of L. 3. Let U (U R N K ) be a matrx contanng vectors u 1,,u K as columns. 4. For = 1,,N, let y (y R K ) be the vector correspondng to the -th row of U. 5. Use the K-means algorthm to partton {y : = 1,,N} nto clusters C 1,,C K. 6. Let A = { j : y j C } ( = 1,,K), where A contans the ndces of nodes that belong to Cluster. Output: {A : = 1,,K}. The performance of exstng spectral clusterng technques s not satsfactory for many applcatons. To mprove the performance, a bo-nspred approach called Self-Organzng-Queue (SOQ) [5] was proposed for the graph clusterng problem. The ey dea of SOQ s to enable fcttous queues of ntellgent nodes wth self-organzng decson capablty to choose a queue wth most frends to jon so that closely-related nodes are grouped nto the same cluster/queue. The SOQ clusterng algorthm s gven as below. Input: a set of N nodes, affnty matrx S (where S R N N ), and the number of clusters K. 1. Intalzaton: dvde the set of N nodes nto K queues; assgn a queue to Current Queue; Flag=1. 2. Whle (Flag) 3. WHO: Choose who n Current Queue as Current Person. 4. HOW: (How to) select a queue as Next Queue for Current Person to jon. 5. WHERE: (Where to) place Current Person n Next Queue. 6

7 6. Assgn Next Queue to Current Queue. 7. WHEN: (When to) let Flag=0,.e., stop the loop. 8. Endwhle Output: the resultng K queues/clusters. The ey features of SOQ are: 1) self-organzaton,.e., each node has the ablty to decde where t wants to jon; 2) the smlarty matrx S can be asymmetrc, and the entres n S can tae any real value, ncludng negatve values. Note that none of the exstng spectral clusterng algorthms allows asymmetrc smlarty matrx and smlarty matrx wth negatve entres. There are many varatons of SOQ, dependng on how Step 1, 3, 4, 5, and 7 are mplemented. For example, n Step 3 (WHO), we can choose the head of Current Queue as Current Person; n Step 4 (HOW), Current Person can use the followng crteron (called Most Frends) to choose Queue ˆ as the Next Queue to jon: ˆ = argmax j C (s, j + s j, ) C where s, j s the entry of S at row and column j, and C s the set of ndces of members n Queue ; n Step 5 (WHERE), we can place Current Person at the tal of Next Queue. Due to the selforganzng decson capablty, SOQ clusterng scheme acheves better clusterng performance than exstng spectral clusterng technques and K-means algorthm for many applcatons [5]. To compare the performance of representatve ernel based clusterng schemes,.e., unnormalzed spectral clusterng (USC) [6], normalzed cut (ncut) [6], and SOQ, as well as K-means, we conduct two experments. Snce unnormalzed and normalzed spectral clusterng algorthms can be regarded as specal types of weghted ernel K-means, we use spectral clusterng to represent ernel K-means as well. The frst experment uses synthetc data consstng of 2-D Gaussan-dstrbuted sample ponts. To smulate four clusters, we use four 2-D Gaussan dstrbutons wth the same standard devaton of 0.05 and mean (-0.3, 0), (0, 0), (0.3, 0), and (0.6, 0), respectvely, and each 2-D Gaussan dstrbuton corresponds to one cluster; the two dmensons of the 2-D Gaussan are ndependent and dentcally dstrbuted. The number of samples for the four clusters are 105, 15, 15, and 15, respectvely, and the total number of ponts N s 150. Fg. 3 shows the 2-D postons of the 150 sample ponts. For spectral clusterng algorthms, an affnty matrx s needed. Let the generated 2-D ponts be p 1,p 2,...,p N, wth p n = (x n,y n ) for 1 n N. We generate the affnty measure s, j between any two ponts p and p j by s, j = exp( p p j 2 2 /(2σ 2 )), where σ = 1 n ths experment. In ths way, we obtan an affnty matrx S. We run the algorthms 20 tmes, each wth a dfferent randomly permuted nput, to obtan 20 clusterng results. By comparng to the ground truth n Fg. 3, we obtan the error rate for each experment. For the 20 experments, we calculate the mean clusterng error rate and 95% confdence nterval, whch are lsted n the second column of Table 1, where µ error denotes the mean clusterng error rate and µ error ± ν denotes upper/lower bound of the confdence nterval, respectvely. Table 1 demonstrates that SOQ sgnfcantly outperforms K-means, USC, and ncut for ths synthetc data set. The second experment uses real-world data, consstng of mages of handwrtten dgts, whch are descrbed n [9] and are downloadable from [10]. The 10 dgts data set s used n our experment. (11) 7

8 Fgure 3: Postons of the two-dmensonal sample ponts n a 2D plane; ponts wth the same color belong to the same cluster (.e., generated by the same dstrbuton). Table 1: Error rate µ error ± ν Synthetc Data Handwrtten Dgts K-means ± ± USC ± ± ncut ± ± SOQ 0 ± ± There are 10 clusters n the data set, wth 100 members n each cluster. Agan, we run the algorthms 20 tmes, each wth dfferent randomly permuted nput. For the 20 experments, we calculate the mean clusterng error rate and 95% confdence nterval, whch are lsted n the thrd column of Table 1. Table 1 demonstrates that SOQ sgnfcantly outperforms K-means, USC, and ncut for ths set of handwrtten dgts. CONCLUSION In ths column, we have dscussed ernel methods as pattern analyss tools, and provded nsghts n two mportant pattern analyss problems, namely, feature extracton and clusterng. Kernel methods have been wdely appled to computer vson, mage processng, nformaton retreval, text mnng, handwrtng recognton, geostatstcs, rgng, bonformatcs, chemonformatcs, nformaton extracton, among others. It s expected that ernel methods wll provde valuable pattern analyss tools for emergng bg data applcatons. AUTHORS Quyuan Huang (dfree@ufl.edu) s a Ph.D. canddate n Department of Electrcal and Computer Engneerng at Unversty of Florda, Ganesvlle, FL. She receved B.S. n Computer Scence and 8

9 Engneerng from Unversty of Scence and Technology of Chna, Hefe, Chna, n Her research nterests are sgnal processng, machne learnng, networng, socal networs, and smart grd. Dapeng Olver Wu (wu@ece.ufl.edu) s a professor wth Electrcal and Computer Engneerng Department at Unversty of Florda, Ganesvlle, FL. Hs research nterests are n the areas of networng, communcatons, sgnal processng, computer vson, and machne learnng. He receved Unversty of Florda Research Foundaton Professorshp Award n 2009, AFOSR Young Investgator Program (YIP) Award n 2009, ONR Young Investgator Program (YIP) Award n 2008, NSF CA- REER award n 2007, the IEEE Crcuts and Systems for Vdeo Technology (CSVT) Transactons Best Paper Award for Year 2001, and the Best Paper Awards n IEEE GLOBECOM 2011 and Internatonal Conference on Qualty of Servce n Heterogeneous Wred/Wreless Networs (QShne) Currently, he serves as an Assocate Edtor for IEEE Transactons on Crcuts and Systems for Vdeo Technology, Journal of Vsual Communcaton and Image Representaton, and Internatonal Journal of Ad Hoc and Ubqutous Computng. He s the founder of IEEE Transactons on Networ Scence and Engneerng. He was the foundng Edtor-n-Chef of Journal of Advances n Multmeda between 2006 and 2008, and an Assocate Edtor for IEEE Transactons on Wreless Communcatons and IEEE Transactons on Vehcular Technology between 2004 and He s also a guest-edtor for IEEE Journal on Selected Areas n Communcatons (JSAC), Specal Issue on Cross-layer Optmzed Wreless Multmeda Communcatons. He has served as Techncal Program Commttee (TPC) Char for IEEE INFOCOM 2012, and TPC char for IEEE Internatonal Conference on Communcatons (ICC 2008), Sgnal Processng for Communcatons Symposum, and as a member of executve commttee and/or techncal program commttee of over 80 conferences. He has served as Char for the Award Commttee, and Char of Moble and wreless multmeda Interest Group (MobIG), Techncal Commttee on Multmeda Communcatons, IEEE Communcatons Socety. He was a member of Multmeda Sgnal Processng Techncal Commttee, IEEE Sgnal Processng Socety from Jan. 1, 2009 to Dec. 31, He s an IEEE Fellow. References [1] J. Shawe-Taylor and N. Crstann, Kernel methods for pattern analyss. Cambrdge unversty press, [2] J. Wang, J. Fan, H. L, and D. Wu, Kernel-based feature extracton under maxmum margn crteron, Journal of Vsual Communcaton and Image Representaton, vol. 23, no. 1, pp , [3] Y. Sun and D. Wu, A relef based feature extracton algorthm, n Proceedngs of SIAM Internatonal Conference on Data Mnng, 2008, pp [4], Feature extracton through local learnng, Statstcal Analyss and Data Mnng, vol. 2, no. 1, pp , [5] B. Sun and D. Wu, Self-organzng-queue based clusterng, IEEE Sgnal Processng Letters, vol. 19, no. 12, pp , [6] I. Dhllon, Y. Guan, and B. Kuls, Kernel -means, spectral clusterng and normalzed cuts, n Proceedngs of the Tenth ACM SIGKDD Internatonal Conference on Knowledge Dscovery and Data Mnng, 2004, pp

10 [7] S. Schaeffer, Graph clusterng, Computer Scence Revew, vol. 1, no. 1, pp , [8] S. Fortunato, Communty detecton n graphs, Physcs Reports, vol. 486, no. 3, pp , [9] D. Verma and M. Mela, A comparson of spectral clusterng algorthms, Unversty of Washngton, Tech. Rep. UW-CSE , [10], dgt1000.mat, [Onlne]. Avalable: 10