Application of Clustering Algorithm in Big Data Sample Set Optimization

Transcription

1 Applcaton of Clusterng Algorthm n Bg Data Sample Set Optmzaton Yutang Lu 1, Qn Zhang 2 1 Department of Basc Subjects, Henan Insttute of Technology, Xnxang , Chna 2 School of Mathematcs and Informaton Scence, Xnxang Unversty, Xnxang , Chna Abstract In order to solve the problem of poor clusterng accuracy and slow convergence speed of K-means clusterng algorthm n bg data envronment, A K-means algorthm based on optmzed samplng clusterng are proposed. The algorthm not only ensures the ratonalty of the ntal value of k-means algorthm but also makes the algorthm clusterng n a smaller sample set to mprove effcency. Fnally, the clusterng centers of the orgnal samples are obtaned by re-clusterng usng a bottom-up herarchcal clusterng method. The algorthm combnes the advantages of herarchcal method, partton method and densty method. The theoretcal analyss and expermental results show that the samplng clusterng algorthm has better clusterng accuracy than other algorthms, and has strong robustness and scalablty. Keywords: Bg data, K-means, Probablty samplng, Clusterng accuracy, Mult-cluster, Evdence theory. 1. INTRODUCTION As nformaton, technology contnues to evolve, many large enterprses, nsttutons and organzatons contnue to have access to a vast array of dverse and heterogeneous data, as well as the techncal ssues of effcently storng, processng and analyzng such valuable data. It s of great mportance to mne useful nformaton effcently from bg data sets (Jang, 2006). The clusterng method can reveal the ntrnsc relatonshp between data wthout pror knowledge, and can cluster valuable data of the same attrbute nto a sngle category, whch can excavate the valuable nformaton of campus network bg data for storage and further analyss of nformaton. However, the tme and space complexty of the tradtonal methods s nadequate under the campus network bg data (Bruno and For, 2013). The k-means algorthm mantans a lnear relatonshp wth the data sze n and satsfes the large-scale data processng However, there s an NP problem n the algorthm (Marchett and Zhou, 2014). It can be seen that the sze of the sample set and the degree of coverage of the orgnal large dataset category play a decsve role n mprovng the applcaton of the k-means algorthm n large dataset clusterng (Almaksour and Anquetl, 2014). 2. BIG DATA SAMPLING BASED ON LEADERS ALGORITHM 2.1 Leaders algorthm Leaders method based on the energy densty, by selectng each type of Leader pont to complete the clusterng, the algorthm s as follows. Leader algorthm does not need a pror to specfy the number of categores, and only needs to scan a large data set once to cluster (Chen and Lu, 2016). The advantages of Leader algorthm n dealng wth large data sets are obvous, but the algorthm s senstve to the nput sequence of data ponts. Appear wthn the class smlarty s greater than the smlartes between classes. 520

2 Clusterng result T1 Bg data set Improved collecton Clusterng result Tk Clustered successfully deleted data Clustered successfully deleted data New collecton Local clusterng results Fgure 1. Bg data set local ncremental clusterng method clusterng process 2.2 Samplng Whle mantanng the correlaton between the samples, the data ponts whose ntersectons n FIG. 1 are not correctly classfed are dstrbuted to dfferent sngle samples by random samplng Concentraton, so through a sngle sample set of clusterng to acheve ths part of the data category re-clusterng (Chen et al., 2014; Peddnt and Saxena, 2014). The amount of data for a sngle sample set. n 1 n 1 1 s f n f n n (1) 2 log( ) log( ) 2 log( ) n Accordng to the formula (1), the sze of the samplng sample set s calculated. In ths paper, m tmes of largescale data sets of sze n are randomly sampled for m tmes. The samplng condtons are as follows: C C (2) j n n (3) j n m n (4) In the formula, = 1, 2,..., m, j = 1,2,..., m, m Z and j, there s no ntersecton of each sample. 3. SAMPLE SET CLUSTERING CENTER CALCULATION METHOD At each samplng, the data n each ntal cluster center area s randomly sampled. In ths way, samplng can make the unon of sample sets reach the maxmum possble coverage of all categores n the orgnal data set, so that the category of samplng unon ts center s close to the orgnal bg data set. The expected samplng s that each sngle samplng set contans all the classes of the orgnal bg data set. 3.1 Determnaton of sngle sample cluster center Because of the small amount of data n a sngle sample set, a varety of classcal clusterng methods could be used for clusterng each sngle sample set. In ths paper, k-means algorthm s used to cluster each sngle sample set, n whch the ntal We choose the ntal cluster centers determned by the leaders preprocessng stage to solve the problem that the number of teratons of the algorthm s easly affected by the ntal cluster center settng (Langone et al., 2016). At the same tme, fast sample clusterng can be acheved due to the small amount of data n a sngle sample set. 521

3 Snce each sample set has the same sze, the sngle sample set clusterng process s performed ndependently, whch can acheve parallel processng and further reduce the processng tme of the algorthm. At the same tme, n order to synthesze the advantages of dfferent clusterng methods, when clusterng sngle sample sets, other algorthms such as K-medods can be used for dfferent sample sets to enhance the clusterng effect of sample sets. 3.2 Mult-sample cluster cluster fuson Suppose the number of categores n a bg data set s k, and the category covered by the th sample set s k (1 k k). After clusterng the sample set, n order to solve the stuaton shown n Fgure 1, some Natural clusters wll nevtably be subdvded. At the same tme, due to the naccuracy of the leaders clusterng centers, the number of clusters n some sample sets wll be greater than the ntal cluster centers, so the clusterng results of each sngle sample set need to be carred out Merge mergers to pnpont the orgnal bg data-clusterng center. Map: Generate samples randomly Reduce: wrte data to the correspondng sample Map: Fern sample gets cluster center Reduce: Consoldaton of all sample clusters Tradtonal parallel k- means clusterng bg data Probablty samplng Sample clusterng and result ntegraton Bg data clusterng Fgure 2. Sample Clusterng K-means Process When the dstance between two classes' mean s less than the preset threshold, the two classes are combned nto a sngle class and the mean of the new class s recalculated. 4. EXPERIMENTAL RESULTS AND ANALYSIS 4.1 Expermental envronment The expermental envronment cluster contans a total of 6 nodes, ncludng 1 master node and 5 slave nodes. Each node s connected va a 100Mb / s Fast Ethernet Swtch. Cluster conssts of 6 ordnary PC, each PC nstalled operatng system are Ubuntu14.04, memory sze s 2GB, hard drve capacty of 350GB. 4.2 Data set Two types of data sets are used n the experment: the frst s a Gaussan dstrbuton dataset, whch generates a 1-ggabyte Gaussan dstrbuton data set. The second s from a standard test set-real-world UCIrvne machne learnng lbrary Data Set Bag of Words Data Set (BoW), whch conssts of 3-D data ponts and represents 3 categores of features (doc ID, word ID, count). 4.3 Performance evaluaton In order to llustrate the robustness, scalablty and good clusterng performance of the algorthm, experments were carred out to compare the results of the two data sets. In ths experment, the parallel K-means, K-means n Map Reduce, K-means n Map Reduce, K-means n sample clusterng and K-means n optmzed sample clusterng were comparatvely analyzed. The expermental clusterng category k = 3, the number of samples s = 24, the number of data ponts for each sample accordng to the sze of the orgnal data set n the set V = {30000, 20000, 15000, 10000, 8000, 5000, 3000} value. 522

4 error/10 9 tme/10 3 s Revsta de la Facultad de Ingenería U.C.V., Vol. 32, N 14, pp , 2017 When the amount of data s small, the data n a stand-alone envronment only operates n memory and the calculaton speed s very fast. In a cluster envronment, a small amount of data occupes a large part of tme n the mport and export of varous compute nodes and dsks. Large amount of data, stand-alone memory s very dffcult to deal wth ths cluster wll show the advantages of dstrbuted computng, data transfer between nodes and dsk mport and export tme relatve to the entre task a small percentage of tme. Sample Samplng Clusterng K-means Gettng approxmate clusterng centers for large datasets wth representatve samples can dramatcally reduce the number of teratons requred for large data sets, reducng the amount of tme requred Data sze / MB Fgure 3. Tme performance curve Then compare the performance of each algorthm clusterng performance, as shown n Fgure 3. As can be seen from Fgure 3, the OSCK algorthm has the best clusterng performance and s obvously superor to the samplng clusterng K-means. Compared wth the clusterng K-means algorthm, the clusterng performance of the optmzed parallel K-means algorthm s better than that of the clusterng algorthm, but the clusterng performance s worse than that of the OSCK algorthm. By removng the sub-optmal clusterng centers of the samples, OSCK algorthm makes the fnal clusterng center more representatve, so t can mprove the clusterng accuracy to a certan extent Data sze / MB Fgure 4. Clusterng performance curve 523

5 Speedup rato tme/10 3 s Revsta de la Facultad de Ingenería U.C.V., Vol. 32, N 14, pp , 2017 Fnally, the algorthm s compared from the pont of vew of the nfluence of the number of nodes n the cluster on the performance of the algorthm. As can be seen from Fgure 5, the sample clusterng K-means algorthm and the OSCK algorthm are stll optmal n runtme when the number of nodes s changed. In addton, as the number of compute nodes ncreases, the runnng tme of each algorthm gradually decreases. However, Wll to some extent mprove performance. If T represents a sngle node calculaton tme, TP s a mult-node calculaton tme, the acceleraton rato Sp = T / TP Number of nodes Fgure 5. Algorthm tme-consumng contrast In order to verfy the robustness of the OSCK algorthm n processng large data sets, the experment uses Gaussan dstrbuton data set to run each algorthm 10 tmes, and statstcs the clusterng results of each run. The sample clusterng K-means and OSCK algorthms have good robustness compared wth the other two algorthms Number of nodes Fgure 6. Algorthm speedup comparson 5. CONCLUSIONS Wth the rapd development of network applcatons, computer networks have penetrated nto every area of socal lfe. Whle brngng strong mpetus to socal development, the ssue of network nformaton securty has 524

6 become an mportant ssue that affects the development of the Internet. Due to the dversty of connecton forms, termnal dstrbuton unevenness, network openness, and nterconnectvty, the network s vulnerable to attacks by hackers and other msdeeds. As t brngs great convenence and speed to people, but also brought huge rsks to people. Ths artcle focuses on the nformaton securty model, securty mechansms, and vrtual prvate network and ntruson detecton systems and so on. Among them, ntruson detecton system s the hot pont of network securty research n the future. The future technology wll be herarchcal and ntellgent development. Future IDSs should be able to detect and alert on ntruson at dfferent levels of the network protocol. REFERENCES Almaksour A., Anquetl E. (2014). Ilclass: error-drven antecedent learnng for evolvng takag-sugeno classfcaton systems, Appled Soft Computng, 19(2), Bruno G., For A. (2013). Mcroclan: mcroarray clusterng analyss, Journal of Parallel & Dstrbuted Computng, 73(3), Chen M.C., Kong X.S., Chen K. (2014). Applcaton of statstcal analyss software n food scentfc modelng, Advance Journal of Food Scence and Technology, 6(10), Chen M.C., Lu Q.L. (2016). Blow-up crtera of smooth solutons to a 3D model of electro-knetc fluds n a bounded doman, Electronc Journal of Dfferental Equatons, 128, 1-8. Jang W. (2006). Nonparametrc densty estmaton and clusterng n astronomcal sky surveys, Computatonal Statstcs & Data Analyss, 50(3), Langone R., Van B.M., Suykens J. (2016). Entropy-based ncomplete cholesky decomposton for a scalable spectral clusterng algorthm: computatonal studes and senstvty analyss, Entropy, 18(5), 182. Marchett Y., Zhou Q. (2014). Iteratve subsamplng n soluton path clusterng of nosy bg data, Statstcs, 9(4), 1-9. Peddnt S.T., Saxena N. (2014). Web search query prvacy: evaluatng query obfuscaton and anonymzng networks, Journal of Computer Securty, 22(1),