New Clustering Algorithm for Multidimensional Data

Transcription

1 New Clusterig Algorithm for Multidimesioal Data Liyua Fa 1, Jigyag Zhog 1 School of Statistics, Capital Uiversity of Ecoomics ad Busiess Beijig , Chia Departmet of Mathematics, Uiversity of Califoria Sata Cruz, CA 95064, U.S.A faliyua@cueb.edu.c Joural of Digital Iformatio Maagemet ABSTRACT: Calculatig similarity for multidimesioal data is oe of the key problems that must be addressed i order to promote the developmet of data clusterig algorithms. I this study, we developed ad tested a ew similarity calculatio idex to improve the accuracy of multidimesioal data clusterig. First, the iformatio divergece (ID) ad geeralized gradiet agle (GGZ) were explored i detail. Secod, the ID ad GGZ were combied to calculate the similarity of multidimesioal data, thus eablig a ew algorithm for data clusterig. Fially, two experimets were coducted to evaluate the performace of the proposed algorithm. The results of the experimets demostrate that our proposed similarity calculatio idex for multidimesioal data is both accurate ad effective, providig better performace as measured by the metrics of accuracy (ACC), ormalized mutual iformatio (NMI), ad purity (PUR). Based o this research, we coclude that the applicatio of the proposed similarity calculatio idex is coducive to the improvemet of data clusterig for multidimesioal data. Subject Categories ad Descriptors K..8 [Database Applicatios]: Data clusterig; B..4 [High- Speed Arithmetic]: Algorithms Geeral Terms Data clusterig, Algorithm Keywords: Data clusterig, data clusterig algorithm, iformatio divergece, gradiet agle, multidimesioal data, kerel problem, similarity calculatio Received: 1 March 015, Revised Jue 016, Accepted 1 Jue Itroductio I recet years, clusterig algorithms have attracted a icreasig amout of attetio. These algorithms have bee used widely i data miig, patter recogitio, image processig, parallel computig, ad other fields [1-4]. To eable data clusterig, first the similarity betwee various data poits must be calculated. With the developmet of techology, clusterig problems ow essetially deal with multidimesioal clusterig i which the data structure is highly complex. Most traditioal clusterig approaches do ot work well for complex data because they pay attetio oly to the global features [5]. For this reaso, a ew algorithm is eeded that takes ito cosideratio both global ad local similarity. Such a algorithm would help distiguish data more comprehesively ad improve clusterig accuracy of multidimesioal data.. State of the Art K-meas [6] ad K-medoids [7] are amog the traditioal clusterig algorithms that are used frequetly. The K- meas algorithm is based maily o the Euclidea distace criterio. While this metric is effective whe used with certai types of data, the K-meas algorithm chooses the iitial cluster ceter radomly, which ca ifluece the clusterig result greatly. The K-medoids algorithm takes the data at the ceter of the class as the cluster ceter, which allows this method to hadle isolated data effectively. However, K-medoids also uses a iitial cluster ceter chose radomly, which ca impact the clusterig result as well. Although some improvemets [8 9] have bee adopted to optimize these classical clusterig Joural of Digital Iformatio Maagemet Volume 14 Number 4 August

2 algorithms, the overall effect has ot bee optimal. Curretly, ew clusterig algorithms are begiig to attract more attetio. Xuesog Yi [10] developed a ovel semi-supervised, metric-based fuzzy clusterig algorithm called SMUC by itroducig metric learig ad etropy regularizatio simultaeously ito the covetioal fuzzy clusterig method (FCM). SMUC itroduces maximum etropy as a regularized term i its objective fuctio. The resultig formulas have clear physical meaig compared with the other semi-supervised FCMs. Experimets o real-world data sets have show the feasibility ad effectiveess of SMUC with ecouragig results. H.Vekateswara Reddy [11] proposed a method for data labelig usig Relative Rough Etropy (RRE) for clusterig categorical data. I this method, data labelig was performed by itegratig etropy with rough sets. The experimetal results showed that the efficiecy ad clusterig quality of this algorithm are better tha demostrated by previous algorithms. Lei Zhag [1] proposed a ovel at-based clusterig algorithm usig Reyi Etropy (NAC-RE). The theoretical aalysis coducted by the kerel method showed that the Reyi etropy metric was feasible ad superior to a distace metric, demostratig that NAC-RE ca get good results. M. Zaribal [13], cocered with ucertaities, proposed the Iterval Type- Relative Etropy Fuzzy C-Meas (IT REFCM) clusterig method. The study s results demostrated that this proposed method has a strog ability to detect oise ad assig suitable membership degrees to observatios. M. Zaribal [14] also proposed a ovel collaborative fuzzy clusterig method. I this approach, relative etropy is used as the commuicatio method, which prompts it to have the highest quality of collaboratio ad eables it to classify data more efficietly. Thus, the cocept of etropy itroduced by Shao with particular referece to iformatio theory proved to be a powerful mechaism for the measuremet of ucertaity iformatio ad effectively improved the clusterig result. This type of method, however, pays more attetio to the global similarity betwee various data with much less focus o local similarity, thereby causig the method to fail for some data sets. To address this limitatio, Niladri Shekhar Mishra [15] studied how fuzzy clusterig algorithms icorporate local iformatio to ehace the performace of the algorithms. Tests showed that the proposed techique is less time cosumig ad does ot require ay a priori kowledge of distributios of chaged ad uchaged pixels. Based o a o-euclidea metric, Xiao-bi Zhi [16] preseted a robust local feature weightig hard c-meas (RLWHCM) clusterig algorithm. The experimetal results idicated that this algorithm has a high level of effectiveess. Ruoche Liu [17] proposed a dyamic local search-based immue automatic clusterig algorithm (DLSIAC) to automatically evolve the umber of clusters as well as a proper partitio of data sets. Experimetal results showed that this algorithm provides good image segmetatio, demostratig that local iformatio ca improve the performace of the origial algorithm effectively. Based o the aalysis above, our study proposed a ew clusterig algorithm that cosiders both global ad local features. Ispired by the etropy theory, first we calculated the iformatio divergece (ID) betwee the two groups of data to reflect the global feature. Our secod step was to characterize the local feature usig the taget of the geeralized gradiet agle (GGZ). At the same time, we proposed a coversio to chage the scope of the agle, which cotributes to the algorithm s ability to fid subtle differeces betwee the two groups of data. A similarity idex was the developed by combiig the iformatio divergece ad the geeralized gradiet agle. This idex ca reflect both the global ad local similarity ad ca be used for data clusterig. Fially, two experimets were coducted to show the performace of this algorithm. The remaider of this paper provides details of our study. Sectio 3 describes the iformatio divergece ad cosie similarity, ad explais how to use them to calculate the similarity of multidimesioal data. Sectio 4 itroduces the evaluatio metrics ad discusses the performace of the proposed clusterig algorithm through some case studies. Sectio 5 summarizes our coclusios. 3. Methodology Sice the proposed data clusterig algorithm is based o our calculatio of similarity, we will itroduce iformatio divergece ad cosie similarity briefly. 3.1 Iformatio divergece Iformatio divergece is a importat measuremet i iformatio theory because it ca reflect the differeces betwee various sources. Iformatio divergece has bee used i various fields i recet years [18] maily for iformatio idetificatio ad fusio. For example, if there are two possible probability desity fuctios p ( z ) ad p ( ) 1 z of the radom variable z, the the iformatio divergece of p 1 (z) to p (z) is [ D(p 1 (z),p (z))= p 1 (z)i p 1 (z) dz = E I p 1 p (z) p (z) (z)] (1) Iformatio divergece has the followig properties a) Noegative. D(p 1 (z), p (z)) 0 the equality holds if ad oly if p 1 (z)= p (z); b) Asymmetry. D(p 1 (z), p (z)) D(p (z), p 1 (z)). Iformatio divergece evaluates the similarity betwee 56 Joural of Digital Iformatio Maagemet Volume 14 Number 4 August 016

3 two probability distributios o the side of the amplitude: the higher the similarity of probability desity fuctio, the smaller the iformatio divergece. I order to overcome the asymmetry of the iformatio divergece, some improved iformatio divergeces have bee adopted. I this paper, the J iformatio divergece was used, defied as follows D ( p ( z), p ( z)) = D( p ( z), p ( z)) + D( p ( z), p ( z)) J p ( z) p ( z) = p ( z)l dz + p ( z)l dz p ( z) p ( z) ( 1 ) = p ( z) p ( z) p ( z) p ( z) l dz For discrete data, the J iformatio divergece was calculated as follows p1 ( z() i ) DJ ( p1( z), p( z)) = p1( z( i) ) p( z( i) ) l i p ( z() i ) (3) where pz ( ) = z/ z. i i i i As see from this defiitio, the iformatio divergece ca reflect the global similarity betwee two groups of multidimesioal data effectively, but it lacks a descriptio of the local iformatio. 3. Cosie similarity The mai priciple of cosie similarity is to take two groups of multidimesioal data as a two-dimesioal vector space, ad the to characterize the degree of similarity by calculatig their geeralized agle. The smaller the agle, the higher the similarity [19]. If we suppose two groups of multidimesioal data x R ad y R, respectively, the the iverse cosie form of their geeralized agle is xy, i= 1 (, ) = arccos = arccos x y xi S x y 1 i x y i x i i= 1 i= 1 () (4) The smaller the value of S, the closer the cosie value of the two groups of multidimesioal data is to 1. Obviously, the method metioed above evaluates the similarity betwee two groups of multidimesioal data from the vector geometry feature, which ca reflect the local feature chage of the data effectively. 3.3 Similarity calculatio based o ID ad GGZ It ca be see from the descriptios of iformatio divergece ad cosie similarity that iformatio divergece ca reflect the global similarity betwee two groups of multidimesioal data, while the geeralized agle ca characterize local features. By usig a calculatio that combies ID ad GGZ, similarity ca be described comprehesively, which should eable more effective clusterig. Although the geeralized agle ca characterize local features, it is ot sufficiet. Therefore, a improved form called the geeralized gradiet agle was adopted for this research. We first eeded to perform a derivatio of the two groups of multi- dimesioal data, ad the calculate the geeralized gradiet agle usig the same method as the geeralized agle. For the data offered above x R ad y R, the gradiet vector was ( ), (5) SG( x) = x x1, x3 x,, x x 1. SG (y)=(y - y 1, y 3 - y,...y - y -1 ) (6) The geeralized gradiet agle was calculated as. SG(x,y) = arc cos SG(x), SG(y) (7) SG(x). SG(y) The formula (7) showed that the geeralized gradiet agle could reflect the local features of the data i a maer especially sesitive to the chages of the geometric slope. I order to desig a effective similarity calculatio idex, we first calculated separately the iformatio divergece ad geeralized gradiet agle, ad the performed a product operatio for them. It should be oted that the iformatio divergece caot multiply the geeralized gradiet agle directly, so the taget of the geeralized gradiet agle was calculated here. I theory, the scope of the geeralized agle betwee the two groups of multidimesioal data is [0, π/], ad the scope of the geeralized gradiet agle is the same. Obviously, the taget of the geeralized gradiet agle is a mootoe icreasig fuctio. Its demarcatio poit is π/4, with the value icreasig slowly whe the agle smaller tha π/4. Otherwise, it will icrease quickly if the agle is larger tha π/4. Accordig to this characteristic, a coversio was used to chage the scope of the agle to [π/4, π/4]. Thus, if the agle betwee two groups of multidimesioal data chages slightly, the taget value will chage obviously, which is beeficial for fidig the similarity chages betwee them. The fial similarity calculatio formula was. SIM (x,y)=d (x,y) SG (x,y)+ π/ ta[ ] (8) 3.4 Implemetatio of the proposed algorithm For our proposed algorithm for multidimesioal data clusterig, our approach is to calculate first the iformatio divergece ad the geeralized gradiet agle, ad the calculate the SIM. If SIM θ, the two groups of data belog to the same class; otherwise, they belog to two differet classes separately. For SIM, θ is a threshold for which the iitial value is determied by experiece or some Joural of Digital Iformatio Maagemet Volume 14 Number 4 August

4 criterio ad will gradually chage i the algorithm ruig process. The implemetatio steps for the algorithm were as follows NMI (C,C ) = MI (C,C ) max (H(C),H(C )) (10) Step 1: Iitializatio: Icludes the data X = { x1, x,, xm} to be clustered, where x j = {x j1,x j,...x j }, the umber N of the class, the cluster ceter c i, i = 1,,...,N, ad the threshold θ [1, ). Step : Calculate the iformatio divergece betwee the cluster ceter C i ad other data by the formula (3). Step 3: Calculate the geeralized gradiet agle betwee the cluster ceter C i ad other data by the formula (7). Step 4: Calculate the similarity betwee the cluster ceter C i ad other data by the formula (8). Step 5: For all data, if SIM (C i, X j ) θ, j =1,,...,m holds, the the data x j belogs to the class X ci. Step 6: Set i = i+1, θ = θ + Δθ, if i < N, ad jump to the step. Otherwise, the algorithm stops. 4. Aalysis of Results ad Discussio I this sectio, we preset two experimets: the first experimet used -dimesioal Gaussia radom data sets with added oise poits, ad the secod used four well-kow real-world data sets. The proposed ID-GGZ algorithm was applied i these experimets, ad the results were the compared with four well-kow clusterig methods icludig K-meas, K-medoids, fuzzy clusterig method (FCM), ad the RRE method proposed i [11]. 4.1 Brief descriptio of evaluatio metrics Three metrics were used to evaluate the clusterig performace of all methods: accuracy (ACC), ormalized mutual iformatio (NMI), ad purity (PUR). ACC is the percetage of correctly predicted labels. If the real label of each cluster was ukow, the Hugaria algorithm [0] was used to get the best map to the real label. If C deotes the groud truth label, ad C deotes the label obtaied from a clusterig algorithm, the mutual iformatio (MI) is defied as MI (C,C ) = p (c i,c j ) log p (c i,c j ) c i C j C p(c i ) p(c j ) I the above formula, p(c i ) ad p(c j ) are the probability of a arbitrarily selected sample belogig to cluster c i ad c j, respectively. p (c i,c j ) is the probability of a arbitrarily selected sample belogig to both cluster c i ad c j. NMI is the ormalized MI as follows (9) where H(C) ad H(C ) are the etropies of c ad C respectively. PUR is computed by assigig the label of a cluster to the most frequet class. More formally, it is defied as PUR(C,C ) = 1 Σ max j i 4. Case study A: Toy datasets This experimet ivolved three clusters of 100 poits each with 100 oise poits added to this oise-free data set. The first 100 poits were geerated based o the multivariate ormal distributio with parameters μ 1 =[1,], 0.1. The parameters used for geeratig Σ 1 = [ 0.1 ] the secod 100 poits were μ =[-1,-], Σ = [ 0 0 ]. The third cluster s parameters were μ 3 =[-3,4], Σ These three clusters are demostrated 3 =[ 0.1 1]. i Fig.1(a). The 100 oise poits were the geerated based o μ Ν =[-3,5], Σ Ν = [ 0 (c j c i ) 0 1] to the oise-free data set (Fig.1(b)). (11). ad were added Usig these five methods, we tried to cluster the data ad elimiate the oise poits from the data set. Note that a data poit was cosidered as a oise poit if it had a low membership degree i all clusters. The obtaied ew deoised data sets based o these clusterig methods were plotted i Fig.(a) (e), respectively. It is clear that amog these methods, ID-GGZ was the most effective i clusterig data ad detectig oise poits. 4.3 Case study B: Real-world data sets I this experimet, four real-world data sets from the UCI database were used to validate the effectiveess of these methods. Importat statistics for the data sets are summarized i Table 1. For each data set, we ra differet methods 40 times, with the compariso based o the average performace. Experimets were performed o the whole real-world data set. Tables, 3, ad 4 show the clusterig performace of the aforemetioed methods evaluated by the ACC, NMI, ad PUR measures. The mea ad stadard deviatio values were both provided as percets. The bold digit i each lie idicates that the correspodig best mea value was obtaied from the correspodig method. The average ruig times (ART) of all the methods are 58 Joural of Digital Iformatio Maagemet Volume 14 Number 4 August 016

5 (a) without oise poits (b) with oise poits Figure 1. Two-dimesioal data set. (a) K-meas (b) K-medoids (c) FCM (d) RRE Joural of Digital Iformatio Maagemet Volume 14 Number 4 August

6 (e) ID-GGZ Figure. The clusterig results Data sets sample feature cluster Wie Soar Vehicle Pe digits Table 1. Data set descriptios Data sets K-meas K-medoids FCM RRE ID-GGZ Wie 56.47± ± ± ± ±1.14 Soar 51.3± ± ± ± ±3.37 Vehicle 46.9± ± ± ± ±4.17 Pe digits 58.79± ± ± ± ±3.4 Table. Compariso of ACC (%) Data sets K-meas K-medoids FCM RRE ID-GGZ Wie 53.64± ± ± ± ±4.5 Soar 58.61± ± ± ± ±5.69 Vehicle 43.53± ± ± ± ±6.30 Pe digits 4.56± ± ± ± ±7.43 Table 3. Compariso of NMI (%) 60 Joural of Digital Iformatio Maagemet Volume 14 Number 4 August 016

7 Data sets K-meas K-medoids FCM RRE ID-GGZ Wie 64.76± ± ± ± ±3.94 Soar 53.59± ± ± ± ±5.6 Vehicle 45.88± ± ± ± ±6.15 Pe digits 46.8± ± ± ± ±6.09 Table 4. Compariso of PUR (%) Data sets K-meas K-medoids FCM RRE ID-GGZ Wie Soar Vehicle Pe digits Table 5. Compariso of ART (s) reported i Table 5. The results evaluated respectively by three differet evaluatio metrics are icosistet o these data sets. For istace, accordig to the ACC metric, for the data set Vehicle the RRE method performs slightly better tha ID-GGZ. However, accordig to the NMI ad PUR metrics, the RRE ad ID-GGZ methods perform comparably. This fidig idicates that usig more tha oe evaluatio metric is importat for evaluatig the performace of a clusterig algorithm. We ca see from Tables, 3 ad 4 that ID-GGZ ca give relatively high ad cosistet clusterig accuracies whe evaluated by ACC, NMI ad PUR as compared with the other clusterig methods. I additio, it ca be see from Table 5 that the ART of the proposed ID-GGZ method is much higher tha the others. This higher ruig time maily reflects the eed for this method to calculate the iformatio divergece ad geeralized gradiet agle simultaeously. Noetheless, the global level of the ART still satisfies the requiremets of practical applicatios. 5. Coclusio I this study, based o our aalysis of iformatio divergece ad geeralized gradiet agle, we desiged a ew similarity idex, which we used to establish a ew data clusterig algorithm. Our study employed Matlab laguage to compile the correspodig calculatio program. Simulatio aalyses were coducted to evaluate the ACC, NMI, ad PUR of the proposed algorithm. The effect of the ew algorithm o ART was assessed as well. The mai coclusios of this research are the followig: (1) The ID ad GGZ were combied for the first time to calculate the similarity betwee various data. Accordig to the results of the field dyamic test, this proposed method for calculatig similarity is both accurate ad effective. Local features are also reflected very well, demostratig that the ew idex is more sesitive to data chages ad its adjusted scope is larger tha foud i previous methods. () Our ew clusterig algorithm for multidimesioal data based o the proposed similarity idex is show to be simple ad easy to use. It is well suited for multidimesioal data clusterig with good performace. (3) The case studies cosidered i this research verified that the proposed data clusterig algorithm ca reflect both the global ad local similarity of the multidimesioal data. Compared with some other commoly used clusterig algorithms, the average ACC, NMI, ad PUR of this ew algorithm was 75.8%, 68.7%, ad 71.19% respectively. These results agree with the test coclusio that the clusterig performace is much better tha the commoly used algorithms. This proposed similarity calculatio idex is highly suitable for the demad of multidimesioal data clusterig with high ACC. However, the ART of the proposed algorithm is poor. Moreover, real-world vagaries may cause ucertaities i the algorithm s parameters, makig calculatio of a exact degree of membership for each data poit impossible. These fidigs suggest that improvig the proposed method usig fuzzy logic would be the focus of potetial future work resultig from this study. Ackowledgemets This study was supported by the Natural Sciece Joural of Digital Iformatio Maagemet Volume 14 Number 4 August

8 Foudatio of Chia (NSFC) (No ) ad the research fudig by Capital Uiversity of Ecoomics ad Busiess (CUEB)(No. 014XJQ011). Refereces [1] Tarel Elguebalya, Nizar Bouguila (015). Model-based approach for high-dimesioal o Gaussia clusterig ad feature weightig. Digital Sigal Processig, 40(3) [] Efedi Nasibov, Ca Atilga, Murat Erse Berberler, et al (015). Fuzzy joit poits based clusterig algorithms for large data sets. Fuzzy Sets ad Systems. 70(7) [3] Yag Tag, Rya P. Browe, Paul D. McNicholas (015). Model based clusterig of high-dimesioal biary data. Coputatioal Statistics ad Data Aalysis, 87(7) [4] Huag Z (1998). Extesios to K-meas algorithm for clusterig large data sets with categorical values, Data Mig ad Kowledge Discovery, (3) [5] Caimig Zhoga, Mikko Malieb,Duoqia Miao (015). A fast miimum spaig tree algorithm based o K- meas. Iformatio Scieces, 95() [6] Mohamed Walid Ayech, Djemel Ziou (015). Segmetatio of Terahertz imagig usig k-meas clusterig based o raked set samplig, Expert systems with Applicatios, 4(6) [7] Hae-Sag Park, Chi-Hyuck Ju (009). A simple ad fast algorithm for K-medoids clusterig, Expert Systems with Applicatios, 36() [8] Abdolreza Hatamloua, Salwai Abdullahb, Hossei Nezamabadi-pour (01). Abombied approach for clusterig based o K-meas ad gravitatioal search algorithms, Swarm ad Evolutioary Computatio, 6(10) [9] B. Bemabe-Loraca, R. Gozalez-Velazquez, E. Olivares-Beitez, et al (014). Extesios to K-Medoids with Balace Restrictios over the Cardiality of thepartitios, Joural of Applied Research ad Techology, 1(3) [10] Xuesog Yi, Tig Shu, Qi Huag (01). Semisupervised fuzzy clusterig with metric learig ad etropy regularizatio, Kowledge-Based Systems, 35(1) [11] H. Vekateswara Reddy, S. Viswaadha Raju, Pratibha Agrawal (013). Data Labelig method based o Cluster Puity usig Relative Rough Etropy for Categorical Data Clusterig. I: IEEE Iteratioal Coferece o Advaces i Computig, Commuicatios ad Iformatics Idia: IEEE, pages [1] Lei Zhag, Qixi Cao, Jay Lee (013). A ovel at-based clusterig algorithm usig Reyi etropy, Applied Soft Computig, 13(5) [13] M.Zaribal, M.H. Fazel Zaradi, I.B. Turkse (014), Iterval Type- Relative Etropy Fuzzy C-meas Clusterig, Iformatio Scieces, 7(7) [14] M.Zaribal, M.H. Fazel Zaradi, I.B. Turkse (015), Relative Etropy collaborative fuzzy clusterig method, Patter Recogitio, 48(3) [15] Niladri Shekhar Mishra, Susmita Ghosh, Ashish Ghosh (01), Fuzzy clusterig algorithms icorporatig local iformatio for chage detectio i remotely sesed images, Applied Soft Computig, 1(8) [16] Xiao-bi Zhi, Jiu-lu Fa, Feg Zhao (014), Robust local feature weightig hard c-meas clusterig algorithm, Neurocomputig, 134(6) 0-9. [17] Ruoche Liu, Beibi Zhu, Rey Bia (015). Dyamic local search based immue automatic clusterig algorithm ad its applicatios, Applied Soft Computig, 7() [18] Chagki Lee, Gary Geubae Lee (006). Iformatio gai ad divergece based feature selectio for machie learig-based text categorizatio, Iformatio Processig & Maagemet, 4(1) [19] Peipei Xia, Lia Zhag, Fazhag Li (015). Learig similarity with cosie similarity esemble, Iformatio Scieces, 307(6) [0] Kuh, H.W (1955). The Hugaria method for the assigmet problem, Naval research logistics quarterly, (1-) Joural of Digital Iformatio Maagemet Volume 14 Number 4 August 016