DATA CLUSTERING: APPLICATIONS IN ENGINEERING

Transcription

1 DATA CLUSTERING: APPLICATIONS IN ENGINEERING Zdravo Krpć Faculty of Electrcal Engneerng, Unversty of Ose Kneza Trpmra 2B, HR-3000 Ose E-mal: Goran Martnovć Faculty of Electrcal Engneerng, Unversty of Ose Kneza Trpmra 2B, HR-3000 Ose Phone: ; E-mal: goran.martnovc@etfos.hr Ivan Vazler Department of Mathematcs, Unversty of Ose Gaev trg 6, HR-3000 Ose Phone: ; E-mal: vazler@mathos.hr Abstract T n Dvdng a set S x ( x, x ) :, m dsunct subsets,, R (a set of vectors from a vector space n, m, such that S, 0,,,,, determnes a partton of the set S. The elements of such partton,, are called clusters. n R ) nto For practcal clusterng applcatons the number of all clusters s too bg and the problem of determnng the optmal partton n the least-squares sense s an NP-hard problem. In ths paper we wll consder some well-nown algorthms for searchng for an optmal LS-partton, lst some of the numerous applcatons of cluster analyss n engneerng and gve some practcal applcatons. Key words: data clusterng, engneerng, least squares. INTRODUCTION In short, clusterng problems are problems of dentfyng groups of ndvduals or obects that are smlar to each other but dfferent from those n other groups. Many web portals and nternet busnesses trac consumer 80

2 habts and tae advantage of these smlartes to target specfc offers to subgroups that are most lely to be receptve to them. Many search engnes cluster ther databases so they can offer smlar results (le boostores suggestng other boos by the same author, or boos wth smlar topcs, or boos from the same publsher, and so on). T n Dvdng a set S x ( x, x ) :, m dsunct subsets,, R (a set of vectors from a vector space n, m, such that S, 0,,,,, determnes a partton of the set S, whch wll be denoted by,, S n R ) nto. The elements of such partton,, are called clusters. The set of all parttons of the set S contanng clusters whch satsfy the propertes above wll be denoted by S, The number of all -parttons s.! S, ( ) and the goal of clusterng s to fnd the optmal partton n some sense. For practcal clusterng applcatons that number s too bg. The problem of clusterng can be dvded n several subproblems. Obects beng clustered need to be represented n a way that the clusterng algorthms can easly measure ther smlarty or dssmlarty (or dstance). Determnng the goal functon and an algorthm for clusterng (whch n most cases fnds only the approxmaton of the optmal partton) s another problem. Dependng on the algorthm, the problem of determnng the number of clusters can also arse. Clusterng algorthms can be classfed n several categores: - Herarchcal clusterng - Based on a tree model of data, t can be ether agglomeratve or dvsve. Agglomeratve clusterng begns wth each obect n ts own cluster and n each step clusters are oned based on smlarty. Its bad sde s that once two obects are n the same cluster, they stay n t tll the algorthm ends. Dvsve clusterng wors smlarly n the opposte drecton. m 8

3 - Parttonal clusterng - These are methods that teratvely mprove the parttonng by movng elements from one cluster to another, usually startng from a random partton. K-means and - medods are the most nown such algorthms. - Neural networ-based clusterng - Hgh dmensonal and large-scale data clusterng - These are methods based on reducng the dmensonalty of the problem. They nclude random samplng methods, densty-based methods and grd-based methods. If we defne a crtera functon on the set S, f : P F : then we can defne a partton S 0, f of all parttons of the set S contanng clusters by x S, 0, F f, x c (o) whch s optmal n the least-squares sense,.e. F ( o) mn S, F. The problem of determnng the optmal partton n the least-squares sense s an NP-hard problem. The -means algorthm can be used for searchng the optmal partton n the LS sense. 2 Algorthm : K-Means Input: Arbtrary -partton (0) T n of the set S x ( x, xn ) R :, m Output: Locally LS-optmal -partton t 0 (l) t) ( t ) ( t) ( t ) 2 calculate cluster centres c c,, c c 3 repeat 4 t t ( t ) t 5 ),, ( t ) ( ( x ( t ) ( t ) x ( t) ( t) such that x S : arg mn x c t) ( t ) ( t) ( t) 6 calculate new cluster centres c c,, c c ) 7 untl c t c (t ) 8 return ( ( t ) ( x ( t) ( t ) x It s easly shown that the -means algorthm monotonously reduces the crtera functon F. The algorthm often stops before reachng the optmal partton n the LS sense. The partton on whch the algorthm stops p p 82

4 depends on the choce of the ntal partton, and snce the algorthm s usually very quc, t s very common to run t multple tmes wth dfferent startng parttons to ncrease the chance of obtanng a better resultng partton. 2. APPLICATIONS 2.. Text clusterng There are many uses of clusterng n text analyss. Clusterng can be used to derve eywords, group artcles wth smlar topcs (ntally unnown), fnd possble synonyms n monolngual dctonares and n many other areas. To cluster textual data one must frst transform the textual data to a format that can be used n clusterng algorthms. One way to represent artcles would be to use ther references and represent ther connectons wth a graph. Ths representaton s sutable for clusterng wth herarchcal clusterng algorthms. Another way to pre-process textual data would be to represent t n the form of vectors (whch can be done n many ways dependng of our goal). The method of representaton used here s the vectorbased model descrbed n Berry (2004) and Srvastava and Saham (2009). The most common way of creatng a vector space model can be dvded n two stages. The frst stage s the extracton of content bearng terms (words or short phrases) and settng ther weght proportonal to the count of the correspondng term n the document. The second stage s to modfy the weghts so that the mportant terms get more emphass. The set of m documents would be represented by a set of vectors T n S x ( x, x ) R :, m where dmenson n of the vectors s equal to the number of n terms n the whole document collecton. The frst tas n stage one s to determne all the terms. Some terms n a document don t descrbe any mportant content (e.g. pronouns,...). Other terms may appear n all (or most) documents or only several documents. These words are usually fltered from the documents and do not appear n the vector representaton. In many languages some terms can be condensed to one due to conugaton or declenson. In the frst stage we create vectors f,, m of frequences of terms. The value of () f s set to the frequency of the th term n the th document. Note that the vectors are very sparse, snce many terms appear only n several documents. In the second stage, the term frequences are multpled by the nverse document frequency of a term n the document collecton. If we denote by W daglnm / w lnm / w n a dagonal matrx where w s the total number of documents contanng the th term, then the vector representaton of document s x' W f,, m. 83

5 Ths s done so that terms occurrng n almost all documents don t nfluence the clusterng results as much. 3.0 few occurences many occurences Fgure : Example of nverse document frequency weghts The last thng left to do s to normalze the vectors so that we observe the relatve frequency of terms. Normalzaton can be done n dfferent norms (,2, ) resultng n data on a correspondng n -dmensonal sphere. After the preprocessng step we have x x' x',, m. Although the -means algorthm s not very good for clusterng hgh dmensonal data the clusterng of normalzed vectors usng the -means should be done wth an extra step so that the centres of clusters also le on a unt sphere. If the Eucldan norm s used for normalzaton, a smple normalzaton of the centrods yelds the centres of clusters constraned on a unt sphere. Ths modfed algorthm s often called sphercal -means algorthm and s obtaned by addng the normalzaton step after steps 2 and 6 n Algorthm. ( t) c c ( t) Example. We tred out ths method of groupng of 0 short news artcles n Croatan found on the nternet. There were 290 dfferent terms. We dd not combne terms based on conugaton and declenson and therefore the results are not as good as they could be. Also, many of the artcles are too short to have enough groupng terms. Despte those shortcomngs, the results are satsfactory. These are the most frequent terms by clusters: ) utamca, ugovor, Zagreb, Masmr, postgao, gol, prva, navač,... 2) azna, sedala, prodaa, orsn, pad, automobl, vozla,... 3) porenuta, odvetn, bvš, uzeo, zbrsao, bane, mto,... 4) aptal, rebalans, povećat, cgareta, bane, pdv, porez, proračun, Terms n those clusters are characterstc terms for football, automobles, banng and poltcs. / c ( t) 84

6 2.2. Image analyss Cameras and other magng equpment are cheap, avalable and used n many areas. Wth that, the need for an automatc mage analyss has arsen. Often the goal s to dstngush smlar or dssmlar areas of an mage. In medcal scences clusterng can be appled to varous body scans for tumor dagnostcs. On satellte mages, clusterng can be used to dscern urban areas, felds, forests... Clusterng can also be used to fnd dfferently textured areas of an mage. To do any of these thngs, mages must often be pre-processed to show the dstngushng features. The most common mage attrbute used for mage clusterng s colour. Other smple mage characterstcs applcable for clusterng parameters are hue and saturaton. More complex mage propertes nclude pxel dstances and patterns. Some pattern recognton applcatons also requre large mage databases aganst whch canddate mages are compared. Example 2. In ths example -means clusterng method s used on a 256 colour greyscale mage, as proposed n Saha and Bandyopadhyay (2008). The mage s 256 pxels n wdth and heght. Dfferent areas of nterest are extracted from t based on the pxel colour value. Ths applcaton s common n satellte mage analyss, but t has some other uses, such as those descrbed n Tatrau and Mehta (2008). The number of clusters represent granularty of segments needed for the extracton from the mage. a) b) c) d) Fgure 2: Clusterng of a greyscale mage based on pxel colour value: a) Orgnal mage, b) =2, c) =3, d) =8. 85

7 Fgures 2 and 3 show two mages segmented nto dfferent numbers of clusters. As the number of cluster ncreases, more detaled mage analyss s done, but ncreasng the number of clusters beyond a certan threshold can mae clusterng meanngless. a) b) c) d) Fgure 3: -means clustered satellte mage: a). Orgnal mage, b) Separaton of heavy clouds wth =2, c) Separaton of lght and heavy clouds wth =4, d) Wth 8 clusters, there s no gan n enhancng cloud separaton. In order to dscover mportant areas of a greyscale mage we need to now how many clusters to loo for. There are many crtera for determnng that number, and many of them requre clusterng of the data for each. One of the easer ways s to use hstogram analyss of the mage. The procedure s as follows: Let G denote the set of all grey levels. For every shade of grey G we fnd ts frequency f, the number of pxels wth that colour. We detect the set of local maxmums n the mage hstogram S f f f f f, & We remove the local maxmums wth frequences below some emprcal threshold (for example f / 00 f max ). thr S 2, f S f f thr 86

8 From S 2 we remove the elements havng close peas (ther dfference n grey levels s below some emprcally determned threshold t ). S 3 S 2 f S, f S such that & t& f f, 2 2 The number S3 s the number of clusters to loo for, and the elements remanng n S 3 are good ntal centres for clusterng. a) b) Fgure 4: Grey level hstograms of mages from a) Fgure 2. b) Fgure 3. In Fgure 4 we can see that the frst mage has fluctuatng grey level frequences so we would use the frequency threshold f thr to remove low level peas. The grey level frequences of the second mage are more unform so we would use the threshold t to reduce the number of peas Applcaton n computer scence A possble applcaton of clusterng n computer scence can be n applcaton mappng n heterogeneous envronments. Advances of ths research can be found n Segel (2009). The heterogeneous computng envronment comprses of dfferent computers under dfferent loads. The goal s to fnd optmal groups of computers (computer clusters) whch are capable of performng varous applcaton tass. These systems can be found n varous computer cluster nstallatons, computer grds and cloud computng systems. Fndng the optmal computer(s) for solvng an applcaton-gven problem can often be NP hard, as there are many parameters whch descrbe each computer. Another dffculty arses due to the dfferent nature of these parameters (processor speed n MHz, RAM capacty n megabytes, networ throughput n megabts per second, etc.). Ths mples that normalzaton of parameters has to be done frst. After that, snce the mappng system uses dfferent preferences on dfferent parameters for every applcaton tas, analyss has to be performed whch evaluates parameter mpact on canddate sutablty. Ths s done by usng weghts, and multplyng normalzed parameter values wth them. The metrc used for measurng the dstance and for calculatng the smlarty matrx depends on a number of parameters for each mappng canddate. In Table, 87

9 ten parameters whch descrbe mappng canddates are presented. The range of values (Mnmum value and Maxmum value) used durng calculaton s also gven for each canddate. After normalzaton, statc and dynamc data are combned together to form current computer mappng canddate (MC) state. Table : Canddate parameters Parameter Measurng unt Mnmum value Maxmum value Processor speed MHz memory capacty MB hard ds capacty GB networ throughput Mbt/s 000 Operatng system -3 3 Processor load % 0 00 Memory load % 0 00 Ds space usage % 0 00 Networ traffc % 0 00 St at c Dynamc Example 3. For the purpose of vsualzaton smplcty, greatest weghts were gven to the frst two parameters (avalable CPU speed and avalable RAM memory), meanng that these are mostly requred by the applcaton tas. Runnng the mappng n ths envronment, wth -means clusterng ( 4 ) gves selectons shown on Fgure 5. Fgure 5: Results of a mappng system, usng -means clusterng method It s obvous that only computer cluster has the benefts from both parameters, formng the most powerful computer cluster. Computer cluster 2 comprses computers wth great processng power, but they lac RAM memory. However, ths cluster has ts uses. Many applcatons depend almost exclusvely on processor 88

10 speed. The thrd computer cluster holds most of the computers, whch have lower performance. Ths cluster contans unwanted canddates, whch are ether heavly loaded already, or ther hardware s nsuffcent. Last computer cluster holds computers wth large amount of RAM memory. These are the approprate canddates for applcaton tass whch are hungry memory-wse. In concluson, there are many dfferent applcatons of clusterng. They dffer n data representaton, goal functons or method of clusterng and that s the reason behnd the ncreasng number of artcles n ths feld. REFERENCES Bandyopadhyay, S. and Saha, S. (2008), Unsupervsed pxel classfcaton n satellte magery usng a new multobectve symmetry based clusterng approach, TENCON, Inda, pp. -6. Berry, M. W. (2004), Survey of text mnng: Clusterng, classfcaton, and retreval, Sprnger, Berln Dubes, R.C. and Jan, A. K. (988), Algorthms for clusterng data, Prentce Hall, New Jersey Evertt, B. S., Landau, S. and Leese, M. (200), Cluster analyss, Wley, London Fran, E. and Wtten, I. H. (2005), Data mnng: Practcal machne learnng tools and technques, Morgan Kaufmann Gan, G., Ma, C. and Wu, J. (2007), Data clusterng: theory, algorthms, and applcatons, SIAM, Phladelpha Han, J. and Kamber, M. (2006), Data mnng: concepts and technques, Morgan Kaufmann Hartgan, J. A. (975) Clusterng algorthms, Wley Jauga, K., Soolows, A. and Boc, H. H. (2002), Classfcaton, clusterng and data analyss, Sprnger, Berln Jng, T., Oscar, C. A., Ruobng, Z., Weyu, Y. and Zhdng, Y. (2008), An adaptve unsupervsed approach toward pxel clusterng and color mage segmentaton, Elsever Kaufman, L. and Rousseeuw, P. J. (2005), Fndng groups n data: an ntroducton to cluster analyss, Jonh Wley & Sons, Hoboen Kogan, J. (2007), Introducton to clusterng large and hgh-dmensonal data, Cambrdge Unversty Press Mehta, A. and Tatrau, S. (2008), Image segmentaton usng -means clusterng, EM and Normalzed Cuts, Department of EECS report, Unversty Of Calforna Segel H. J. (2009), Stochastcally robust resource management n heterogeneous parallel computng systems, ISPAN, USA, pp. -2. Srvastava, A. and Saham, M. (2009), Text mnng: Classfcaton, clusterng, and applcatons, Chapman & Hall 89