A Semi- Non-Negative Matrix Factorization and Principal Component Analysis Unified Framework for Data Clustering

Transcription

1 A Semi- No-Negative Matrix Factorizatio ad Pricipal Compoet Aalysis Uified Framework for Data Clusterig V.Yuvaraj, N.SivaKumar Assistat Professor, Departmet of Computer Sciece, K.S.G college of Arts ad Sciece, (Affiliated to Bharathiar Uiversity), Coimbatore Tamiladu, Idia. Assistat Professor/HOD, Departmet of Iformatio Techology, K.S.G college of Arts ad Sciece, (Affiliated to Bharathiar Uiversity), Coimbatore Tamiladu, Idia. ABSTRACT: Clusterig has received a sigificat amout of attetio as a importat problem with may applicatios, ad a umber of differet algorithms have emerged over the years. Recetly, the use of No-Negative Matrix Factorizatio (NMF) for partitioal clusterig has attracted much iterest. However, the popularity of NMF has sigificatly icreased the proposed multiplicative NMF algorithms which they applied to image data. At preset, NMF ad its variats have already foud a wide spectrum of applicatios i several areas such as patter recogitio ad feature extractio, dimesioality reductio, segmetatio ad clusterig, text miig ad eurobiology. Noegative Matrix Factorizatio (NMF) is a popular matrix decompositio method with various applicatios i e.g. machie learig, data miig, patter recogitio, ad sigal processig. The o egativity costraits have bee show to result i parts-based represetatio of the data, ad such additive property ca lead to the discovery of data s hidde structures that have meaigful iterpretatios. KEYWORDS: Data miig,, o-egative matrix factor, Clusterig, applicatios of NMF I.INTRODUCTION Clusterig has received a sigificat amout of attetio as a importat problem with may applicatios, ad a umber of differet algorithms have emerged over the years. Recetly, the use of No-Negative Matrix Factorizatio (NMF) for partitioal clusterig has attracted much iterest. However, the popularity of NMF has sigificatly icreased the proposed multiplicative NMF algorithms which they applied to image data. At preset, NMF ad its variats have already foud a wide spectrum of applicatios i several areas such as patter recogitio ad feature extractio, dimesioality reductio, segmetatio ad clusterig, text miig ad eurobiology. Noegative Matrix Factorizatio (NMF) is a popular matrix decompositio method with various applicatios i e.g. machie learig, data miig, patter recogitio, ad sigal processig. The o egativity costraits have bee show to result i partsbased represetatio of the data, ad such additive property ca lead to the discovery of data s hidde structures that have meaigful iterpretatios. A. NON-NEGATIVE MATRIX FACTOR I liear algebra, a Matrix Factorizatio (MF) is a decompositio of a matrix ito a product of matrices. Let the iput data matrix X = x 1,. x cotai data vectors of dimesioality m, W = w 1,. w r, ad H = 1,.. To factorize matrix X ito the product of matrices W ad H, oe ca write: X = WH (1) I covetioal MF, both the iput matrix X ad the factorized matrices Wad H ca cotai either positive or egative etries. The idea of Noegative Matrix Factorizatio (NMF) i which they itroduced a factor aalysis method called Positive Matrix Factorizatio (PMF). Give a observed positive data matrix X, PMF solves the followig weighted factorizatio problem with oegativity costraits: mi W 0,H 0 A X WH F (2) Copyright to IJARSET

2 where F deotes Frobeius orm, deotes Hadamard (elemet-wise) product, A is the weightig matrix, ad W, H are factor matrices that are costraied to be oegative. NMF attract more research attetios ad gai more applicatios i various fields. Give a oegative iput m data matrix X R m +, NMF fids two oegative W R r + ad H R r + such that X WH (3) The rak r is ofte chose so that r < mi m,, A appropriate selectio of the value r is critical i practice, but its choice is usually problem depedet. Let us write X i W i = w k ki. Oe ca see that NMF approximates each oegative iput data vector i X by a additive liear combiatio of r oegative basis colums i W, with oegative coefficiets i the correspodig colum i H. Therefore, the matrix factor W is usually regarded as the basis matrix, the factor H as the coefficiet matrix, ad the product term WH is called the compressed versio of the X or the approximatig matrix of X. The additive ature of NMF ca ofte geerate parts-based data represetatio that coveys physical meaigs. II CLUSTERING Clusterig is a combiatorial problem whose aim is to fid the cluster assigmet of data that optimizes certai objective. The aim of clusterig is to group a set of objects i such a way that the objects i the same cluster are more similar to each other tha to the objects i other clusters, accordig to a particular objective. Clusterig belogs to the usupervised learig scope that ivolves ulabeled data oly, which makes it a more difficult ad challegig problem tha classificatio because o labeled data or groud truth ca be used for traiig. Cluster aalysis is prevalet i may scietific fields with a variety of applicatios. For example, image segmetatio, a importat research area of computer visio, ca be formulated as a clusterig problem. A. PRINCIPAL COMPONENT ANALYSIS AND NON-NEGATIVE DATA There has bee published several papers where NMF outperforms PCA. Aalyze the outcome of Pricipal Compoet Aalysis (PCA) whe the observatio is oegative. This aalysis shows that PCA will output oly oe purely positive compoet ad the remaiig compoets will cotai both positive ad egative elemets. I PCA a set of vectors v 1,. v m R is projected to a r-dimesioal space such that most variace is obtaied. I other words, PCA fids a matrix P PCA R r with orthoormal row vectors that fulfils P PCA = arg max P R r PV F 2 (4) Note, there are may solutios to the maximizatio problem. Therefore, argmax meas that P PCA is just oe of the optimal matrices. B. EVALUATION MEASURES Effective evaluatio measures are crucial for quatifyig ad comparig the performace of clusterig methods. Geerally, there are three differet categories of evaluatio criteria: iteral, relative, ad exteral. Iteral criteria examie the resultig clusters directly from the origial iput data. Relative criteria compare several clusterig structures, which ca be produced by differet algorithms, ad decide which oe may best characterize the data to certai extet. Exteral criteria have bee commoly used; they measure the clusterig performace by usig the kow iformatio (ofte referred to as groud truth). Two widely used exteral criteria are Purity, defied as purity = 1 2 max 1 l q Copyright to IJARSET r k=1 r k=1 k l (4) l where k is the umber of vertices i the partitio k that belog to the groud-truth class l. Purity is easy to uderstad, as it ca be iterpreted i a similar way as the classificatio accuracy i supervised learig. However, purity has a drawback i that it teds to emphasize the large clusters. Normalized Mutual Iformatio, defied as NMI = K K k j =1 i log i i,j log i, j i m j K j =1 m j log m j (5)

3 where K ad K respectively deote the umber of clusters ad classes; i,,j is the umber of data poits agreed by cluster i ad class j; i, ad m i, deote the umber of data poits i cluster i ad class j respectively; ad is the total umber of data poits i the dataset. NMI examies the quality of clusters from a iformatio-theoretic perspective. Compared with purity, NMI teds to be less affected by the cluster sizes due to the ormalizatio step give by its deomiator, but it is ot that ituitive as purity for people to iterpret. III. SIGNIFICANCE OF THE SYSTEM The cocept of matrix factorizatio is used i a wide rage of importat applicatios ad each matrix factorizatio relies o a assumptio about its compoets ad its uderlyig structures, it is a essetial process i each applicatio domai. Very ofte, the data sets to be aalyzed are o-egative, ad sometimes they also have a sparse represetatio. The spectral clusterig, ormalized cuts, ad Kerel k-meas are particular cases of clusterig with NMF uder a doubly stochastic costrait. They also cosidered the symmetric matrix decompositio uder o-egativity costraits similar to those formulated. IV. LITERATURE SURVEY The propose system to improve the multi-view poit algorithm i two actual ways. First, the curret PVC algorithm is cosidered specifically for two-view datasets. We exted this algorithm for the k multi-view poit. Secod, exted our k multiple-view algorithm to iclude view-specific graph laplacia regularizatio. This eables the proposed algorithm to exploit the itrisic geometry of the data distributio i each view. The compare propose method agaist existig clusterig algorithm o both text ad image datasets. The baselie methods ad datasets used i this method are more exhaustive tha what is used i the PVC work. The experimets show that the proposed GPMVC method outperforms PVC ad other competitive baselie methods o all the differet datasets. It also provides isights ito how our algorithm performs whe there is a skew i the distributio of partial examples across views. A. NON-NEGATIVE MATRIX FACTORIZATION NMF with the sum of formed error cost meaig is equal to a comfortable K-meas clusterig, the most this algorithm used for usupervised dataset learig. The NMF with the I-divergece cost fuctio is matched ito probabilistic latet sematic idexig aalysis, usupervised learig method mostly used for text aalysis. May existig data miig ad machie learig algorithm ca be used to solve the NMF problem. Here cosider the iput data row ad colum matrix deoted by X = X 1, X 2, X cotai the o ofdata colum vectors. factorizex ito two matrices, X FG T (1) wherex R p, F R p k ad G R k. Geerally, p < ad the rak of matricesf,gis much lower tha the rak of X, i.e., k mi(p;). F;Gare obtaied by miimizig a cost fuctio. The most commo cost fuctio is the sum of squared errors, Mi F,G 0 J sse = X FG T 2 (2) the matrix orm is idirectly assumed to be the Frobeius orm. A rak o-deficiecy disorder is assumed for F, G. the cost fuctio is the so-called I-divergece: m Mi F,G 0 J ID = X ij log X ij FG T X ij + FG T ij (3) ij j =1 It s easy to show that the dissimilarity I(x) = x log x x+1 0 holds whe x 0; the equivalece holds whe x = 1. The quatity I(u;v) = (u=v) log(u=v) u=v+1 is called I-divergece, NMF ad K-meas Clusterig Here proposed K-meas based clusterig algorithm is oe of the best clusterig algorithm for high dimesio dataset. Let cosider X= X 1, X 2, X be data poits. The divider them ito K equally disjoit clusters. The K-meas clusterig objective ca be writte as J kmeas = mi 1 K k x i f k 2 = x i f k 2 (4) k=1 i c k Copyright to IJARSET k

4 The followig theorem shows that NMF is iheretly related to K-meas clusterig algorithm mi F 0,G 0 X FG T 2 st G T G = 1 (5) is equivalet to K-meas clusterig. The k-meas if X ad F have mixed-sig accesses. appreciate this relatioship i this similarity. Here c is the o of cluster to be cosider the C = C 1, C 2, C k be the cluster cetroids foud via base cluster K-meas clusterig. Let H be the cluster idicators: i.e., ki = 1 if x i belogs to cluster c k ; ik = 0 otherwise. We ca write the K-meas cluster objective as J = ik x i c 2 k = X CH T 2 From this aalogy, i NMF F has the meaig of cluster cetroids ad G is the cluster idicator. Thus K-meas ad NMF have similar objective fuctio but with differet costraits. The origially K-meas objective fuctio ca be expressed to igore the oegativity costrait while keepig the orthogoality restrictio, the pricipal compoet is the solutio. O the other had, if igore the orthogoality while keepig the oegativity, NMF is the solutio. B. MULTI-VIEW CLUSTERING The multi-view poit clusterig algorithm to aalysis based o data poit i multi views would be assiged to the same cluster with high likelihood. To apply this istict i a NMF settig the coefficiet matrices (V i ) leart from dissimilar views are softly regularized towards a commo agreemet matrix (V ). This agreemet matrix is cosidered to reflect the latet structure shared by differet views. The multi-poit NMF clusterig setup divergece betwee the ith coefficiet matrix ad agreemet matrix i.e. k V i V is miimized. The V i from multi views might ot be similar at the same scale oe eeds to adopt a ormalizatio policy. Each coefficiet matrix V is ormalized usig matrix Q (where, Q is a diagoal matrix, Q k,k = i U i,k. This gives us the followig multi- view NMF-based clusterig problem, S.t U i 0, V i 0, i s, t 1 i v v K k=1 Mi ui,vi,v X i U i V i T F 2 + μ i V i Q i V F 2 This algorithm to learig a joit represetatio (V ) of the data totally igores the itrisic geometrical structure of each sigle view. The existig work show that respectig the geometrical/low-dimesioal maifold iformatio ca improve clusterig quality. The propose system maily cosider multi-view pit i multi dimesio datasets. N i Pealty GR = 1 V 2 i j V 2 i l W jl j,l=1 = Tr V T i D i V i Tr V T i W i V i = Tr V T i L i V i (7) To eable this, itroduce a added graph regularizatio pealty. Give a similarity matrix W 2 oe ca defie a smoothess cosequece. V. METHODOLOGY A. DATASET ORL: This is image dataset cotais a set of 400 face images. Here cocept two views oe based o raw pixel values ad the other comprisig of HOG features. 3Sources: This three-view text dataset is collected from three olie ews sources. I total there are 948 ews articles coverig 416 distict ews stories. Of these stories, 169 were reported i all three sources. For our multi-view experimets the dataset cotaiig 169 articles was used. BBC Sports: This text dataset is a collectio of sports ews articles from the BBC Sport web site. For our Multiview experimets choose the 3-view dataset which cotaiig 282 reports. Digit: This image dataset is from the UCI repository ad cosists of 2000 had-writte digits (0-9). This is a 5-view dataset. Similar to two-view experimets it cosiders the followig two views: 216 profile correlatios, 240-pixel averages i 2 x 3 widows. Cora: This dataset cosists of 2708 scietific publicatios. It cosiders the followig two views for experimets: umber of citatios betwee documets ad the term-documet matrix. (6) Copyright to IJARSET

5 B. PERFORMANCE EVALUATION To measure the clusterig performace of the proposed algorithms we use the commoly adopted metrics, the accuracy, the Normalized Mutual Iformatio ad the Adjusted Rad Idex. The clusterig accuracy oted (Acc) discovers the oe-to-oe relatioship betwee two partitios ad measures the extet to which each cluster cotais data poits from the correspodig class. It is defied as follows: Acc = 1 δ (C i, map P i (13) where is the total umber of samples, Pi is the ith obtaied cluster ad Ci is the true ith class provided by the data set. (x; y) is the delta fuctio that equals oe if x = y ad equals zero otherwise, ad map(pi) is the permutatio mappig fuctio that maps the obtaied label Pi to the equivalet label from the data set. The secod measure employed is the Normalized Mutual Iformatio (NMI); it is estimated by NMI = k kl kl k,l log k l k log k l log l l (14) where k deotes the umber of data cotaied i cluster C k 1 k K, l is the umber of data belogig to the class L l 1 l K ad kl deotes the umber of data that are i the itersectio betwee cluster C k ad class L l. C. COMPARISON GRAPHS Table 4.1: Average computatio time vs differet algorithms Datasets Semi-NMF- PCA F-Semi-NMF- PCA RF-Semi-NMF PCA Coil Orl ,584 1, Webkb NMF- multi-view graph clusterig Ru Time(ms) Semi-NMF-PCA F-Semi-NMF-PCA RF-Semi-NMF-PCA NMF-multi-view graph cluster Algorithms Figure 4.1 Compariso of ru time Vs differet datasets Table 4.2 Results obtaied by the compared methods o differet data sets i terms Acc, NMI ad ARI. Coil Orl Copyright to IJARSET

6 Performa 0.75 ce A cc N M I Semi-NMF-PCA F-Semi-NMF-PCA RF-Semi-NMF-PCA NMF-multi-view graph clusteri Figure 4.2 Results obtaied by the compared methods o Coil data sets i terms Acc, NMI ad ARI Performa 0.4 ce Semi-NMF-PCA F-Semi-NMF-PCA RF-Semi-NMF-PCA NMF-multi-view graph cluster A c c Figure 4.3 Results obtaied by the compared methods o Orl data sets i terms Acc, NMI ad ARI Performac e Acc NMI ARI Figure 4.4 Results obtaied by the compared methods o Webkb data sets i terms Acc, NMI ad ARI Copyright to IJARSET

7 D. SCREENSHOTS Coil Dataset Oral Dataset Copyright to IJARSET

8 VI. CONCLUSION AND FUTURE WORK The proposed Multi View Clusterig (MVC) algorithm is show to be effective i states with icomplete or missig iformatio i the high dimesio dataset. The propose system to improve the MVC algorithm to support multi-views ad view-specific based graph Laplacia regularizatio. The MVC algorithm is cosidered specifically for two-view datasets. The first view k-multi-view dimesio. The secod view k multiple-view algorithm to iclude viewspecific graph Laplacia regularizatio. The propose algorithm to activity the iheret geometry of the data distributio i each view. The experimets show that the propose multi-view graph clusterig (MVGC) algorithm outperforms MVC ad other competitive baselie clusterig method methods o all the differet datasets. FUTURE WORK The future work observatios show that our algorithm is a good cadidate to apply it to image segmetatio ad text dataset, that will be ext task. The applyspectral clusterig algorithms usig statistical alarm method. The alarm boud discloses that the clusterig rate is closely related to the amout of data alarm oe ca make the clusterig rate small by reducig the amout of alarm. The future work shows the that clusterig rate coverges to zero as the umber of represetative poits produces. These results provide a theoretical foudatio for algorithms ad also have potetially wider applicability. I particular, a atural directio to pursue i future work is the use of other local data reductio methods (e.g., data squashig ad codesatio methods) for pre-processig; this boud ca be exteded to these methods. The future pla to explore other methods for assigig clusterig membership to the origial data accordig to the membership of the represetative data based o local optimizatio ad edge-swappig methods. REFERENCES [1] A. Baerjee, S. Merugu, I. S. Dhillo, ad J. Ghosh, Clusterig with bregma divergeces. Joural of Machie Learig Research, vol. 6, pp , [2] M. Belki ad P. Niyogi. Laplacia eigemaps ad spectral techiques for embeddig ad clusterig. I NIPS, [3] P. Carmoa-Saez, R. D. Pascual-Marqui ad A. Pascual-Motao, Biclusterig of gee expressio data by o-smooth oegative matrix factorizatio. BMC Bioiformatics, vol. 7(78), pp. 1 18, [4] H. Cho, I. Dhillo, Y. Gua, ad S. Sra, Miimum sum squared residue based co-clusterig of gee expressio data. SIAM SDM, pp , [5] I. S. Dhillo. Co-clusterig documets ad words usig bipartite spectral graph partitioig. ACM SIGKDD, pages , [6] C. Dig, X. He, ad H. D. Simo, O the equivalece of oegative matrix factorizatio ad spectral clusterig. SIAM SDM, pp , [7] C. Dig, T. Li, ad W. Peg., Noegative matrix factorizatio ad probabilistic latet sematic idexig: Equivalece,chi-square statistic, ad a hybrid method. AAAI, vol. 42, pp , [8] C. HQ, Dig, T. Li ad M. Jorda. I. Covex ad semioegative matrix factorizatios. TPAMI, 32(1), pp , [9] Q. Gu ad J. Zhou, Co-clusterig o maifolds. ACM SIGKDD, [10] J. V. D. Guillamet ad B. Schiele, Itroducig a weighted oegative matrix factorizatio for image classificatio. Patter Recogitio Letters, vol. 24(14), pp , [11] T. Hofma, Usupervised learig by probabilistic latet sematic aalysis. Machie Learig, vol. 42, pp , [12] J. Kim ad H. Park., Sparse oegative matrix factorizatio for clusterig. Techical report, Georgia Istitute of Techology, [13] D. D. Lee ad H. S. Seug, Algorithms for o-egative matrix factorizatio. NIPS, volume 13, pp , [14] T. Li ad C. Dig., The relatioships amog various oegative matrix factorizatio methods for clusterig. IEEE ICDM, pp , [15] W. Liu ad N. Zheg, No-egative matrix factorizatio based methods for object recogitio. Patter Recogitio Letters, vol. 25(8), pp , [16] F. Shahaz, M. Berry, P. Pauca, ad R. Plemmos, Documet clusterig usig o-egative matrix factorizatio. Iformatio Processig ad Maagemet, vol. 42, pp , [17] A. Strehl ad J. Ghosh, Cluster esembles - a kowledge reuse framework for combiig multiple partitios. Machie Learig Research, pp , [18] H. Wag, F. Nie, H. Huag ad F. Makedo, Fast oegative matrix tri-factorizatio for large-scale data co-clusterig. IJCAI, [19] R. Zass ad A. Shashua, A uifyig approach to hard ad probabilistic clusterig. IEEE ICCV, pp , [20] D. Cai ad X. He ad J. Ha, SRDA: A efficiet algorithm for large-scale discrimiat aalysis. TKDE, vol. 20, pp. 1 12, , Copyright to IJARSET