Research on K-Means Algorithm Based on Parallel Improving and Applying

Transcription

1 Sed Orders for Reprits to 288 The Ope Cyberetics & Systemics Joural, 2015, 9, Ope Access Research o K-Meas Algorithm Based o Parallel Improvig ad Applyig Deg Zherog 1,2,*, Deg Xig 1, Zhag Chua 1, Xu Liag 1 ad Huag Wemig 1,2 1 School of Computer Sciece ad Egieerig, Guili Uiversity of Electroic Techology, Guili, , P.R. Chia; 2 Guagxi Key Laboratory of Trusted Software, Guili, , P.R. Chia Abstract: The capacity of sigle server or CPU is uable to fiish the task of the miig of mass data. I cosideratio of this bottleeck problem, a combied algorithm which is used by geetic ad MR-based parallel clusterig algorithm is proposed. To make up for the defects of clusterig aalysis i screeig the clusterig ceter, the clusters are used by geetic algorithm, relyig o M-R parallel computig model to accelerate the covergece of the clusterig aalysis. To verify reasoableess of algorithm, this algorithm which is applied to aalysis of the actual log is based o buildig of Hadoop platform. Experimetal results show that relyig o performace of distributed cluster computig ad geetic clusterig aalysis to process log files ca get better clusterig results, ad the efficiecy of miig of massive log ca be greatly improved. Keywords: Cloud computig, Clusterig aalysis, Geetic algorithm, Map-reduce, Mass data processig. 1. INTRODUCTION Nowadays, the arrival of the era of big data is ot i doubt. The data showed the tred of rapid growth i scietific research (Astroomy, biology, high-eergy physics), the Iteret, e-commerce, ad computer simulatio applicatio. Especially, the ew amout of data geerated i the aual scietific research is about 15PB; the era of big data has two major treds: the data expasio ad the depth aalysis of data. Eighty percet data is ustructured data i some large eterprises. This grows expoetially every year. Large data challege ot oly ivolves the costructio of eterprise storage architecture, data ceter ifrastructure, but ca also produce chai reactios for cloud computig, data miig, busiess itelligece, data warehouse, etc. Eterprises ivest more eergy i busiess aalysis ad for miig the TB level data i the future, for example, study of miig i massive log ad log depth aalysis will be a essetial part of [1]. The log is a very broad cocept term i the computig world; ay program may have a log file, such as the applicatio of computer kerel system, computer server, all kids of busiesses ad social etworkig sites. The log file cotais the iformatio of iterest, as the busiess eterprise develops the potetial value of commercial iformatio by aalysis of user lik ad statistics such as the umber of access i the logs. Therefore, aalysis ad miig of a log have become a hot topic i computer research field. Various characteristics of large data such as outstadig?, have made traditioal log processig, log miig methods ad algorithms o loger applicable. Facig the itesive, complex hybrid log file eeds to have a more efficiet method of computig ad date miig algorithm. *Address correspodece to this author at the School of Computer Sciece ad Egieerig, Guili Uiversity of Electroic Techology, Guili, , P.R. Chia; Tel: ; zhrdeg@guet.edu.c X/15 This paper is based o parallel computig model of MapReduce that has caused widespread attetio [2, 3]. The combied algorithm [4-6], which is used by geetic ad MRbased parallel clusterig algorithm is proposed i view of the advatages of geetic algorithm. The algorithm follows the thought of sample global optimizatio, task of divide ad rule, ad results of summary ad aalysis. O oe had, it solves the defect of cluster ceter iitializatio istability for k-meas algorithm, ad o the other had, accelerates the covergece of K-Meas algorithm. The algorithm is deployed o the Hadoop experimetal platform, verifyig the effectiveess of the algorithm. 2. ALGORITHM ANALYSIS 2.1. Geetic Algorithm Geetic algorithm is a geeral learig method based o evolutio. Geetic algorithm is o loger as other algorithms searchig hypothesis ragig from geeral to specific ad from simple to complex, but selects the parets sample through a certai probability from the iitial sample groups. The geetic algorithm ca cotrol the global search process, fially obtaiig the search optimal solutio set or ear optimal solutio set. Compared with the traditioal heuristic search, geetic algorithm advatage is the coloy search strategy. The mai work is the choice of geetic operatio parameters which ca be completed from frequet humacomputer iteractio process i the search process K-Meas Clusterig aalysis is that the data set is divided ito family, the family data are similar, ad differet data as far as possible. It ca mie valuable iformatio distributio patters i the data ad is a importat meas of mass iformatio processig. It also plays a vital role i machie learig, data miig, text aalysis ad other fields. Classi Betham Ope

2 Research o K-Meas Algorithm Based o Parallel Improvig ad Applyig The Ope Cyberetics & Systemics Joural, 2015, Volume cal clusterig algorithm is compared to the K-Meas clusterig, fuzzy clusterig ad spectral clusterig. the log of servers ad custom trasactio data, similar customer groups, web pages clusterig, ad clusterig algorithm of frequet access path are aalyzed [7] based o couplig matrix processig. Distributed affiity propagatio clusterig algorithm based o MapReduce is further discussed [8]; this algorithm overcomes the limitatio of the dese dataot Sparse. [9, 10] Three clusterig methods are clustered: clusterig method based o group, clusterig method based o graularity, ad clusterig method based o fuzzy. These algorithms caot reflect the advatage i big data aalysis. K-Meas is a clusterig algorithm based o iterative distace. Its realizatio is simple, the efficiecy is higher for large data processig, especially, it gets better whe miig high dimesio ad agglomerate data. The K-Meas algorithm observes istace classificatio to K clusterig where it is less tha the other clusterig ceter distace. The K- meas algorithm cosists of three steps: (1) Fid the cluster ceter iitializatio, K is defied as the umber of clusters; (2) Calculate the distace for each observatio istaces to the cluster ceter, while puttig istaces to the earest cluster. Distace uses Euclidea distace criterio. The calculatio formula is as follows: D( X i,y j ) = "( X ij! Y ij ) 2,i = 1,2,..., ; j = 1,2,..., k (1) j=1 (3) Calculate the average distace of each cluster for all observed istaces, ad the average value as a ew clusterig ceter. The calculatio formula is as follows: ceter i = 1 " x s i j (2) x j!s i Repeat the third step util the clusterig ceter o loger chages. Termiate the clusterig process whe achievig the optimal objective fuctio or a maximum umber of iteratios. Whe usig the Euclidea distace as the metric, the calculatio formula is as follows: K mi!! dist(c i, x) 2 (3) i=1 x"c i 2.3. MapReduce Map-Reduce is a excellet model of distributed computig. It is widely applied to log aalysis, mass data sortig, ad also i massive data search. Busiess logic is abstracted ito two fuctios by Map-Reduce i the large scale cluster system: Map ad Reduce. Usually, the iput data is divided ito several idepedet data blocks by Map-Reduce. Processig the data block is parallel to Map fuctio, ad calculatio model of frame is to sort the results of the Map task. The the results are outputted to the Reduce fuctio. Its core idea is to work (through the mappig) ad Reduce (simplificatio). Each phase of the Map-Reduce process is as follows: (1) Iput: The iput data is divided ito several idepedet data blocks i this phase. (2) Map: The user iput data is regarded as several groups such as <key, value> key by Map-Reduce. I this phase, the Map fuctio calls a user-defied model to hadle the <key, value> key, ad the geerates the middle <key, value> key. (3) Shuffle: the middle <key, value> key is obtaied by Map i shuffle phase, accordig to the value of key to sort the iput data. (4) Combie: Whe the map operatio outputs its pairs, they are already available i memory. For efficiecy reasos, sometimes it makes sese to take advatage of this fact by supplyig a combier class to perform a reduce-type fuctio. (5) Reduce: The framework calls the applicatio's Reduce fuctio oce for each uique key i the sorted order. The Reduce ca iterate through the values that are associated with that key ad produce zero or more outputs. (6) Output: The Output Writer writes the output of the Reduce to the stable storage, usually a distributed file system. As show i Fig. (1), Map-Reduce is efficiet i makig cocise model, good scalability, fault tolerace ad parallelism HDFS HDFS is a storage structure of a distributed computig. It provides iput, output data for Map-Reduce parallel computig stage. A HDFS cluster is composed of a umber of NameNode ad DataNode; these two types of odes are Master ad Worker, respectively. The NameNode is resposible for maiteace tasks of amespace directory ad idex file ad participates i the cluster eviromet schedulig i a cluster system. DataNode is maily resposible for the odes for data storage ad task executio, ad at the same time, uiterrupted implemetatio ad trasmissio data report is trasmitted through the heart (HeartBeat) mode to NameNode Clusterig Geetic Parallel Algorithm based o M-R Model Clusterig ceters obtaied are optimized by geetic algorithm to avoid the ureasoable clusterig aalysis i screeig the iitial cluster ceter of clusterig results ad covergece rate of the problem. The mai work of algorithm is to calculate the distace of sample to the cluster ceter ad redistributio of the cluster, ad the calculatio of differet clusters is idepedet. Accordig to the idepedet characteristics of each group, this paper presets a parallel K-Meas algorithm by the computig model of the MapReduce. The desig of algorithm maily icludes the clusterig ceter optimizatio fuctio, Map, Combie, Reduce fuctio. The algorithm process is show i Fig. (2). 3. THE IMPLEMENTATION OF IMPROVED K- MEANS ALGORITHM 3.1. Screeig the Optimal Clusterig Ceter K-Meas clusterig algorithm has the characteristics of local search, but its clusterig covergece speed ad its

3 290 The Ope Cyberetics & Systemics Joural, 2015, Volume 9 Zherog et al. Fig. (1). The computig model of map-reduce. effective clusterig results are affected by the selectio of iitial clusterig ceter. Oce the iitial value selectio is ot good, it is difficult to get ideal results. Geetic algorithm is a adaptive global optimizatio algorithm. This paper itroduces geetic algorithm i K-Meas. The ifluece of the clusterig result is o loger affected by the iitial clusterig ceter through pretreatmet of the clusterig ceter selectio. The global optimal cluster ceters are foud by chromosome codig, fitess fuctio selectio ad geetic operator of geetic algorithm. The specific steps are as follows. Step 1: Chromosome code selectio. Chromosome code selectio directly iflueces the efficiecy of the geetic algorithm ad the fial result. The biary coded mode is regarded as the solutio set of the problem which is metioed i the literature [11], but the accuracy is ot high eough i the process of large-scale umerical optimizatio. Accordig to the characteristics of large amout of data ad complex iformatio, this paper adopts real codig. The code will eed the chromosome legth as the umber of cluster ceters, assumig the chromosome legth is Li, the codig populatio for X(k), where k is the first geeratio of populatio, the the i-geeratio populatio is X(k)i={X(k)1,X (k)2,,x(k)i}, Ki is the umber of cluster ceters i the chromosome. Step 2: Populatio iitializatio. Sample K for evolutio is radomly selected from the date sample, i order to avoid the premature ed of geetic operatio; the desired effect of optimal solutio caot be achieved as K caot choose too small. Step 3: The appropriate fuctio selectio. Sort criteria of cadidate hypotheses are defied by the fitess fuctio, ad the ext geeratio populatio criterio is chose with a certai probability. I this paper, the fitess fuctio is costructed by referecig the K-Meas criterio fuctio. The K- Meas criterio fuctio ad fitess fuctio are as follows: k E = # # ( X j! u i ) 2 (4) i=1 f = 1 1+ E x j "s i E represets the square error of all objects; The Ui represets the average of the Si value, amely, the clusterig ceter; Xj represets the J class sample space; ad K is the umber of cluster ceters. Accordig to clusterig criterio F, the more less the E value is, the more excellet is the clusterig quality. Step 4: Geetic operator. Selectio: The idividual probability is obtaied by idividual to idividual ad group fitess ratio. The choice of idividual method is called fitess proportioal selectio or rotary table selectio. Calculatio formula of idividual is selected as follows: (5)

4 Research o K-Meas Algorithm Based o Parallel Improvig ad Applyig The Ope Cyberetics & Systemics Joural, 2015, Volume Fig. (2). Flow chart of the algorithm. p(h i ) = Fitess(h i )! j=1 Fitess(h j ) This method ca guaratee the radomess, higher fitess is selected ad the smaller is elimiated. Itersectio: Accordig to the real umber codig of the log miig, this paper uses uiform crossover method. The method is to radomly select two idividuals, takig two itersectios i a radom way. Whe the radom umber is 0, the frot part of the idividual is cross; whe the radom umber is 1, the middle part of the idividual is cross; whe the radom umber is 2, the tail part of the idividual is cross. Variatio: Accordig to the characteristics of mass data, each gee selected sample chromosome represets a cluster ceter. I order to meet the geetic algorithm based o assurace of global search which has also local search performace ad the diversity of the group, a mutatio operator is itroduced. Accordig to the variatio of the probability, (6) algorithm selects variat of the idividual i the group, the the samples of gee segmet have radom variatio (Here is a selectio of uiform probability as the mutatio probability). Note that variatio rage should be i the rage of gee segmets. Step 5: Output the optimal iitial clusterig ceter ad the clusterig descriptio iformatio. Iput: The iitializatio of populatio N, the probability of crossover ad mutatio. Output: The best cluster ceter, the clusterig ceter descriptio. // Calculatio of each idividual fitess, selectio probability, the expected probability for each i rage do CalAllGes(i); while maxges do // Iitial populatio

5 292 The Ope Cyberetics & Systemics Joural, 2015, Volume 9 Zherog et al. Iitial(i); // Selectio of high fitess to eter ext geeratio selectges(i); // Itersectio Crossover(); // Variatio Mutate(); maxges--; // Output the optimal clusterig ceter set pritbestresult(); 3.2. Desig of the Map ad Reduce fuctios The parallel computig task i each MapReduce task is iitially a Job. Each Job lik ca be split ito two parts: Map ad Reduce stage. The clusterig ceter durig the implemetatio of K-Meas is temporarily stored i HDFS; data exchage format uses the TextIputFomat fuctio to regulate. Key represets the clusterig ceter; value represets Euclidea distace of cluster sample to the cluster ceter. Iput: The clusterig ceter poit optimizatio ad clustered sample. Output: Output clusterig is set whe the coditios of covergece are satisfied. Iformatio is maily described o the map stage ad record belogs to the ew cluster. The calculatio of the shortest distace is carried out betwee the iput sample ad the clusterig ceter. 1. Class Mapper 2. method Map(LogWritable K,Text V,Cotext C) // radom sample of iput segmetatio 3. for each logfiles f V do // Traversig the clusterig ceter, gets the short iformatio of the cluster ceter 4. for each ceter i ceters do 5. if (mi more-tha distace) // record miimum 6. Save distace,pos 7. ed for // Record category ad attribute vector set of clusterig 8. emitresult(text(pos), Text(distace) I the Combie stage, the ecoded data files have duplicate data, i order to reduce the etwork flow, while the itermediate results are local mergers. With the same key value of itermediate results, amely clusterig ceter subscripts the same <key, value> key to merge ito a group. 9. Class Combie 10. Method Combie(LogWritable K,Text V,Cotext C) 11. EmitLocalResult(Text(pos), Text(distace) At the etrace Reduce phase, data is derived from the Map phase of <key, value> character key queue, where key is the idex of cluster ceter ad value is the Euclidea distace of the cluster family. The key of the same sample value is accumulated ad calculates the average value, ad the the mea value is regarded as the clusterig ceter at the ext iteratio, ad stored i the HDFS. 12. Class Reducer 13. Method Reduce(LogWritable K,Iterable V,Cotext C) 14. sum 0, ave for each V i Iterable<rs> 16. sum+=v(value) 17. Ave = sum/cout // Calculate the average value ad save it to the iitial clusterig ceter set i the ext roud 18. EmitNextResult(K,ave) 19. Class MRkmeas // Implemetatio of the algorithm ad output results 20. Job(cofigratio,alg) The algorithm termiates coditios: The last roud of the results is compared with the results of this clusterig. If the clusterig results do ot chage or are less tha the threshold value, the coditios for covergece are satisfied, clusterig algorithm is termiated, whereas the results would serve as the ext roud of the K-Meas iput. 4. EXPERIMENT AND ANALYSIS OF RESULTS 4.1. Set up the Experimetal Eviromet By buildig the Hadoop cloud computig platform to validate the effectiveess of the algorithm, the architecture is show i the figure. The hardware icludes five PC machies, oe machie as a master ode, loadig schedulig ad real-time moitorig of task, the remaiig four machies as a slave ode, loadig distributed processig of the task. Each ode cofiguratio of The Hadoop platform is show i Table 1. Nameode is regarded as the mai cotrol (JobTracker), which is task schedulig ad maagemet for distributed clusters; Dataode is regarded as TaskTracker, resposible for performig the task Aalysis of Results Accordig to the differet allocatios of hardware ad software for the two class statuses, this paper maily studied several experimets. Data source collectio is from log files of compay server. Experimet 1: Pseudo distributed cluster mode Firstly, build a good pseudo distributed platform, ru the traditioal clusterig algorithm (K-Meas), a parallel clusterig algorithm (CPA) ad geetic clusterig algorithm (M-R CPGA), to process the size scale of log file as 2MB, 4MB, 6MB, 8MB, 10MB, the compare the executio time, as show i Fig. (3).

6 Research o K-Meas Algorithm Based o Parallel Improvig ad Applyig The Ope Cyberetics & Systemics Joural, 2015, Volume Table 1. Cofiguratio of Hadoop platform. Name IP Fuctio Operatio System hadoop-master ameode dataode Ubutu Server hadoop-slave dataode Ubutu hadoop-slave dataode Ubutu hadoop-slave dataode Ubutu hadoop-slave dataode Ubutu I Experimet 2, the parallel clusterig algorithm (CPA) ad geetic clusterig algorithm (M-R CGPA) were compared. The experimetal results show that i the case of M-R cluster distributed CGPA, algorithm shows good speedup. As the data size icreases, computig clusters distributed advatages become more promiet, ad the scalability of Hadoop platform esures high availability program algorithm as show i Fig. (4). Fig. (3). Executio time of pseudo distributed cluster algorithm. I Experimet 1, three algorithms are used i differet sizes of the log file. The experimets of umerical comparisos show that i a sigle pseudo distributed case, whe a log file is of a very small scale, three kids of algorithm executio time have a obvious drop; however, data acquisitio is very large, because the performace of sigle cofiguratio caot reach the fully distributed parallel geetic clusterig algorithm. This reflects its disadvatage. Experimet 2: Distributed cluster mode I order to validate the algorithm i dealig with more data, thus complete cluster distributed eviromet eeds to be set up i which four machies work as the task processig ode while a machie as the task schedulig ode. Due to the hardware task, call ode cofiguratio is higher but also works as a task ode, as show i Fig. (3). Fig. (4). Executio time of distributed cluster algorithm. CONCLUSION This paper combied the global optimizatio of geetic algorithm ad K-Meas algorithm of local searchig characteristic ad proposed a geetic clusterig based o the calculatio model of M-R parallel algorithm. Through the deploymet of Hadoop cluster eviromet, the algorithm ra i HDFS ad the massive log files were processed. The experimetal results show that with the cluster odes ad the amout of data icrease, the efficiecy of the algorithm is higher to achieve effect of miig o massive log, layig the foudatio for future study of combiatio i the massive data off-lie calculatio ad multidimesioal data miig. CONFLICT OF INTEREST The authors cofirm that this article cotet has o coflict of iterest. ACKNOWLEDGEMENTS This work is supported by Guagxi key Laboratory of Trusted Software (No: kx201317), by the Postgraduate s Iovatio Project of Guili Uiversity of Electroic Techology uder (No: GDYCSZ201470), by the 2014 Guagxi Uiversity of Sciece ad Techology Research Projects (NO: LX ), by the Nature Sciece Foudatio of Guagxi (No: 2013GXNSFAA019350). REFERENCES [1] H. Yu, ad D. Wag, Mass log data processig ad miig based o Hadoop ad cloud Computig, I: Computer Sciece & Educatio (ICCSE), [2] Y. Liu, N. Cao, W. Pa, ad G. Qiao, System aomaly detectio i distributed systems through MapReduce-Based log aalysis, Advaced Computer Theory ad Egieerig (ICACTE), vol. 6, pp , [3] S. Badyopadhyay, Geetic algorithms for clusterig ad fuzzy clusterig, Wiley Iterdiscipliary Reviews: Data Miig ad Kowledge Discovery, vol. 3, p. 285, [4] X. Qi, ad H. Wag, Big Data Aalysis:Competitio ad symbiosis of RDBMS ad MapReduce, Joural of Software, vol. 23, pp , 2012.

7 294 The Ope Cyberetics & Systemics Joural, 2015, Volume 9 Zherog et al. [5] X. Meg, ad X. Ci, Big data maagemet: coceptio, techology ad challege, Research ad Developmet of Computer, vol. 50, pp , [6] S. Wag, H. Wag, ad X. Qi, Big data structure: challege, preset situatio ad prospect, Joural of Computer, vol. 10, pp , [7] Q. Sog, ad J. She, Efficiet miig algorithm of web log, Research ad Developmet of Computer, vol. 3, pp , [8] C. Zhag, ad H. Yig, Uiform block Crossover Geetic Algorithm, Techology ad Applicatio of Automatio, vol. 24, pp , [9] Y. Wu, Discussio o custerig aalysis method, Sciece of Computer, vol. 39, pp , [10] Y. Ma, ad W. Yu, Research Progress o geetic algorithm, The Research ad Applicatio of Computer, vol. 29, pp , [11] R. C Taylor, A overview of the Hadoop/MapReduce/HBase framework ad its curret applicatios i bioiformatics, I: 11 th Aual Bioiformatics Ope Source Coferece (BOSC), Bosto, MA, USA, Received: September 16, 2014 Revised: December 23, 2014 Accepted: December 31, 2014 Zherog et al.; Licesee Betham Ope. This is a ope access article licesed uder the terms of the Creative Commos Attributio No-Commercial Licese ( liceses/by-c/4.0/) which permits urestricted, o-commercial use, distributio ad reproductio i ay medium, provided the work is properly cited.