CHAPTER 8. Clustering Algorithm for Outlier Detection in. Data Mining

Size: px

Start display at page:

Download "CHAPTER 8. Clustering Algorithm for Outlier Detection in. Data Mining"

Alice Clarke
5 years ago
Views:

1 CHAPTER 8 Clustering Algrithm fr Outlier Detectin in Data Mining 8.1 Intrductin In many data mining applicatins, the primary step is detecting utliers in a dataset. Outlier detectin fr data mining is nrmally based n distance, clustering and spatial methds. This chapter deals with lcating utliers in large, multidimensinal datasets. This chapter deals with k-means, k-medids and Fuzzy c-means clustering methds are cmputed with identifying utliers. The k means clustering algrithm partitins a dataset int a number f clusters, and then the results are used t find ut the utliers frm each cluster, using any ne f the utliers detectin methds. The k means clustering algrithm is enhanced in three manners. The first is by using a different distance metric. The secnd and third enhancements are brught frward 168

2 Chapter 8. Cluster Based Outlier Detectin in Data Mining 169 by autmating the prcess f estimating k value and initial seed selectin using the enhanced clustering algrithm. Anther tw clustering methds k-medids and Fuzzy c-means are cmputed. The results shw that classificatin accuracy, speeds are imprved and nrmalized rt mean square errr is reduced. This Chapter main bjective is aimed t a cmparative review f three f the varius partitining based clustering methds. A divisin f data bjects int grups f similar bjects are called clusters. Objects pssessed by same cluster tend t be similar, while dissimilar bjects are pssessed by different clusters. These clusters represent grups f data and prvide simplificatin by representing many data bjects by fewer clusters. And, this helps t mdel data by its clusters. Clustering is a methd f unsupervised learning and a well knwn technique fr statistical data analysis. It is used in many fields such as machine learning, image analysis, pattern recgnitin, utlier detectin, and biinfrmatics t name a few. Many researchers have prpsed different methds t achieve clustering. Alng with managing a very large dataset, a rbust clustering methd must satisfy sme requirements such as scalability, dealing different types f attributes, discvering clusters f arbitrary shape, high dimensinality, ability t deal with nise and utliers, interpretability and usability. With clustering, time cmplexity increases with dealing large number f dimensins and large set f data bjects. Als the effectiveness depends upn the definitin f similarity (r dissimilarity) amng bjects. Alng with this, the utput f clustering can be interpreted in different ways.

3 Chapter 8. Cluster Based Outlier Detectin in Data Mining Methdlgy A partitining methd creates k partitins, called clusters, frm given set f n data bjects. Initially, each data bjects are assigned t sme f the partitins. An iterative relcatin technique is used t imprve the partitining by mving bjects frm ne grup t anther. Here, each partitin is represented by either a centrid r a medid. A centrid is an average f all data bjects in a partitin, while the medid is the mst representative pint f a cluster. The fundamental requirements f the partitining based methds are each cluster must cntain at least ne data bject, and each data bjects must belng t exactly ne cluster. In this categry f clustering, varius methds have been develped. In this sectin describes three types f clustering methds such as k-means, k-medids and Fuzzy c-means k-means k-means which is firstly prpsed by MacQueen (1967), is a well-knwn and widely used clustering algrithm. k-means is ne f the simplest clustering algrithms in machine learning which can be used t autmatically recgnize grups f similar instances/items/bjects/pints in data training. The algrithm classifies instances t a pre-defined number f clusters specified by the use (e.g. assume k clusters). The first imprtant step is t chse a set f k instances as centrids (centres f the clusters) randmly, usually chse ne fr each cluster as far as pssible frm each ther. Next, the algrithm cntinues t read each instance frm the data set and assigns it t the

4 Chapter 8. Cluster Based Outlier Detectin in Data Mining 171 nearest cluster. There are sme methds t measure the distance between instance and the centrid but the mst ppular ne is Euclidian distance. The cluster centrids are always recalculated after every instance insertin. This prcess is iterated until n mre changes are made. The k-means algrithm is explained in this fllwing pseud cde. Finally, this algrithm aims at minimizing an bjective functin, in this case a squared errr functin. The bjective functin k n 2 J = c i j x (j ) j=1 i=1 (8.1) where x (j ) 2 (j ) c i j is a chsen distance measure between a data pint x and i the cluster centre c j is an indicatr f the distance f the n data pints frm their respective cluster centres. The algrithm is cmpsed f the fllwing steps: 1. Place K pints int the space represented by the bjects that are being clustered. These pints represent initial grup centrids. 2. Assign each bject t the grup that has the clsest centrid. 3. When all bjects have been assigned, recalculate the psitins f the K centrids. 4. Repeat Steps 2 and 3 until the centrids n lnger mve. This prduces a separatin f the bjects int grups frm which the metric t be minimized can be calculated. Althugh it can be prved that the prcedure will always terminate, the k-means

5 Chapter 8. Cluster Based Outlier Detectin in Data Mining 172 algrithm des nt necessarily find the mst ptimal cnfiguratin, crrespnding t the glbal bjective functin minimum. The algrithm is als significantly sensitive t the initial randmly selected cluster centres. The k-means algrithm can be run multiple times t reduce this effect k-medids k-medids clustering is a partitining methd cmmnly used in dmains that require rbustness t utlier data, arbitrary distance metrics, r nes fr which the mean r median des nt have a clear definitin. It is similar t k-means, and the gal f bth methds is t divide a set f measurements r bservatins int k subsets r clusters s that the subsets minimize the sum f distances between a measurement and a center f the measurement s cluster. In the k-means algrithm, the center f the subset is the mean f measurements in the subset, ften called a centrid. In the k-medids algrithm, the center f the subset is a member f the subset, called a medid. Suppse that we have n bjects having p variables that will be classified int k(k < n) clusters (Assume that is given). Let us define j th variable f bject i as X i j (i = 1,...,n; j = 1,..., p). The prpsed algrithm is cmpsed f the fllwing three steps. Step 1 : (Select initial medids) 1. Using Euclidean distance as a dissimilarity measure, cmpute the distance between every pair f all bjects as fllws:

6 Chapter 8. Cluster Based Outlier Detectin in Data Mining 173 d i j = p ( ) 2 Xiα X j α i = 1,...,n; j = 1,...,n (8.2) α=1 2. Calculate p i j t make an initial guess at the centers f the clusters. p i j = d i j i = 1,...,n; j = 1,...,n (8.3) n d i j i=1 Step 2 (Find new medids) 3. Replace the current medid in each cluster by the bject which minimizes the ttal distance t ther bjects in its cluster. The k-medids algrithm returns medids which are the actual data pints in the data set. This allws yu t use the algrithm in situatins where the mean f the data des nt exist within the data set. This is the main difference between k-medids and k-means where the centrids returned by k-means may nt be within the data set. Hence k-medids is useful fr clustering categrical data where a mean is impssible t define r interpret Fuzzy C- Means Fuzzy Clustering is a pwerful apprach used in machine learning. There is variety f Fuzzy Clustering methds available which utilizes different types f distances fr cluster membership calculatin. Fuzzy C Means (FCM) was pineered by Bezdek (1981) knwn as ne f the ppular appraches f Fuzzy Clustering which uses fuzzy weights based n reciprcal distance. Fuzzy weights decrease the ttal weighted mean-square errr.

7 Chapter 8. Cluster Based Outlier Detectin in Data Mining 174 Fuzzy c-means (FCM) is a methd f clustering which allws ne piece f data t belng t tw r mre clusters. This methd (develped by Dunn (1973) and imprved by Bezdek (1981) is frequently used in pattern recgnitin. It is based n minimizatin f the fllwing bjective functin: N C J m = u m i j i=1 j=1 xi c j 2 (8.4) The algrithm is very similar t k-means, except that a matrix (rw is each data pint, clumn is each centrid, and each cell is the degree f membership) is used. 1. Initialize the membership matrix U () 2. At k-step: calculate the centers vectrs C (k) = [c j ] with U (k) (Repeat step (3), (4) until cnverge) 3. Cmpute lcatin f each centrid based n the weighted fractin f its member data pint s lcatin. c j = N u m i j x i i=1 N u m i j i=1 4. Update each cell as fllws U (k),u (k+1) 5. If U (k+1) U (k) <ǫ then STOP; therwise return t step 2. u i j = c k=1 1 ( xi c j x i c k ) 2 m 1 (8.5) This iteratin will stp when max i j U (k+1) U (k) <ǫ, where ǫ is a terminatin i j i j criterin between 0 and 1, whereas k are the iteratin steps. This prcedure cnverges

8 Chapter 8. Cluster Based Outlier Detectin in Data Mining 175 t a lcal minimum r a saddle pint f J m. Ntice that the parameter m is the degree f fuzziness. The utput is the matrix with each data pint assigned a degree f membership t each centrids. Membership Functins A membership functin (MF) is a curve that defines hw each pint in the input space is mapped t a membership value (r degree f membership) between 0 and 1. The input space is smetimes referred t as the universe f discurse, a fancy name fr a simple cncept. The nly cnditin a membership functin must really satisfy is that it must vary between 0 and 1. The functin itself can be an arbitrary curve whse shape we can define as a functin that suits us frm the pint f view f simplicity, cnvenience, speed, and efficiency. A classical set might be expressed as A=x x > 6 A fuzzy set is an extensin f a classical set. If X is the universe f discurse and its elements are dented by x, then a fuzzy set A in X is defined as a set f rdered pairs. A=x,µA(x) x X

9 Chapter 8. Cluster Based Outlier Detectin in Data Mining 176 µa(x) is called the membership functin (r MF) f x in A. The membership functin maps each element f X t a membership value between 0 and 1. Definitin Fr any set X, a membership functin n X is any functin frm X t the real unit interval [0,1]. Membership functins n X represent fuzzy subsets f X. The FIGURE 8.1: Membership functin f a fuzzy set membership functin which represents a fuzzy set Ã is usually dented by µ A. Fr an element x f X, the value µ A (x) is called the membership degree f x in the fuzzy set Ã. The membership degree µ A (x) quantifies the grade f membership f the element x t the fuzzy set Ã. The value 0 means that x is nt a member f the fuzzy set; the value 1 means that x is fully a member f the fuzzy set. The values between 0 and 1 characterize fuzzy members, which belng t the fuzzy set nly partially.

10 Chapter 8. Cluster Based Outlier Detectin in Data Mining 177 Summary f Membership Functins 1. Fuzzy sets describe vague cncepts (e.g., fast runner, ht weather, weekend days). 2. A fuzzy set admits the pssibility f partial membership in it. (e.g., Friday is srt f a weekend day, the weather is rather ht). 3. The degree an bject belngs t a fuzzy set is dented by a membership value between 0 and 1. (e.g., Friday is a weekend day t the degree 0.8). A membership functin assciated with a given fuzzy set maps an input value t its apprpriate membership value. 8.3 Selectin f the Number f Clusters (K) This sectin reviews existing methds fr selecting K fr the K-means algrithm and the crrespnding clustering validatin techniques. The selectin f the number f clusters is an imprtant subprblem f clustering. Since a priri knwledge is generally nt available and the vectrs dimensins are ften higher than tw, which d nt have visually apparent clusters. The slutin f this prblem directly affects the quality f the result. If the number f clusters is t small, different bjects in data will nt be separated. Mrever, if this estimated number is t large, relatively regins may be separated int a number f smaller regins Zhang et al. (2003). Bth f these situatins are t be avided. This prblem is knwn as the cluster validatin prblem. The aim is t estimate the number f clusters during the clustering prcess. The basic idea is the evaluatin f a clustering structure by generating several clustering fr

11 Chapter 8. Cluster Based Outlier Detectin in Data Mining 178 varius number f clusters and cmpare them against sme evaluatin criteria Values f K specified within a range r set The perfrmance f a clustering algrithm may be affected by the chsen value f K. Therefre, instead f using a single predefined K, a set f values might be adpted. It is imprtant fr the number f values cnsidered t be reasnably large, t reflect the specific characteristics f the data sets. At the same time, the selected values have t be significantly smaller than the number f bjects in the data sets, which is the main mtivatin fr perfrming data clustering Outlier Detectin Algrithms Outlier detectin is a technique t find patterns in data that d nt cnfrm t expected behaviur. Mst f the clustering algrithms d nt assign all pints t clusters but accunt fr nise bjects, in ther wrds clustering algrithms are ptimized t find clusters rather than utliers. Outlier detectin algrithms lk fr utliers by applying ne f the clustering algrithms and retrieve the nise set, therefre the perfrmance f utlier detectin algrithms depends n hw gd the clustering algrithm captures the structure f clusters. Outlier detectin can be divided int tw appraches: distance-based utlier detectin and density-based utlier detectin. The first methd, distance-based utlier detectin algrithms, wrks n the assumptins that the nrmal data bjects have a dense neighbrhd and the utliers are far apart frm their neighbrs. The secnd methd wrks n assumptins that the density arund a nrmal data bject

12 Chapter 8. Cluster Based Outlier Detectin in Data Mining 179 is similar t the density arund its neighbrs while the density arund an utlier is cnsiderably different t the density arund its neighbrs. The density-based apprach cmpares the density arund a pint with the density arund its lcal neighbrs by cmputing an utlier scre. This paper fcuses n distance-based utlier detectin algrithm nly. 8.4 Result and Discussins In this chapter, three clustering based methds such as k-means, k-medids and Fuzzy c-means are used fr identifying the utliers in the iris data set. The data sets are extracted frm the surce: tt s r s t s ts r s. This data sets are given in the Appendix D K-means TABLE 8.1: Centrids fr k-means clustering Number f clusters: 3 Ttal number f training cases: 150 sepal length sepal width petal length petal width Number f cases Percentage (%) In table 8.1 represents the centrids fr k-means clustering number f clusters: 3 and ttal number f training cases: 150. The table 8.2 represents k-means clustering and find the utlier pint cmputed using distances frm the iris data set. In figure 8.2 visualised the number f clusters and utliers pint. The + symbl indicate utlier pints and symbl indicates number f clusters.

13 Chapter 8. Cluster Based Outlier Detectin in Data Mining 180 TABLE 8.2: Number f utliers detected frm iris data set Obs.N Sepal.Length Sepal.Width Petal.Length Petal.Width Sepal.Length Sepal.Width FIGURE 8.2: utliers with k-means clustering K-Medids TABLE 8.3: k-medids with number f clusters: 2 ttal number f training cases: 150 ID Sepal.Length Sepal.Width Petal.Length Petal.Width In table 8.3 represents the centrids fr k-means clustering number f clusters: 3 and ttal number f training cases: 150.

14 Chapter 8. Cluster Based Outlier Detectin in Data Mining 181 clusplt(pam(x = sdata, k = k, diss = diss Silhuette plt f pam(x = sdata, k n = clusters C j j : n j ave i Cj s i Cmpnent : : : Cmpnent 1 These tw cmpnents explain % Silhuette width s i Average silhuette width : 0.55 FIGURE 8.3: Cluster plt and Silhuette Plt fr k-medids In figure 8.3 is cluster plt and pam is a plt knwn as a silhuette plt. First, a measure is calculated fr each bservatin t see hw well it fits int the cluster that it s been assigned t. This is dne by cmparing hw clse the bject is t ther bjects in its wn cluster with hw clse it is t bjects in ther clusters. (A cmplete descriptin can be fund in the help page fr silhuette.) Values near ne mean that the bservatin is well placed in its cluster; values near 0.55 mean that it s likely that an bservatin might really belng in sme ther cluster. Within each cluster, the value fr this measure is displayed frm smallest t largest. If the silhuette plt shws values clse t ne fr each bservatin, the fit was gd; if there are many bservatins clser t zer, it s an indicatin that the fit was nt gd. The sihuette plt is very useful in lcating grups in a cluster analysis that may nt be ding a gd jb; in turn this infrmatin can be used t help select the prper number f clusters. The plt indicates that there is a reasnable structure has been fund, with mst bservatins seeming t belng t the cluster that they re in. There is a summary

15 Chapter 8. Cluster Based Outlier Detectin in Data Mining 182 measure at the bttm f the plt labeled "Average Silhuette Width". This table shws hw t use the value: Range f SC Interpretatin A strng structure has been fund A reasnable structure has been fund The structure is weak and culd be artificial < 0.25 N substantial structure has been fund TABLE 8.4: utlier detectin with k-medids Outliers Sepal.Length Sepal.Width Petal.Length Petal.Width In table 8.4 represents the utliers detected in the data sets using k-medids algrithm. Fuzzy C-means TABLE 8.5: Fuzzy c-means clustering with 3 clusters S.N Sepal.Length Sepal.Width Petal.Length Petal.Width The table 8.5 represents k-means clustering and find the utlier pint cmputed using distances frm the iris data set. The cmputed results f membership functin is given in Appendix B. In figure 8.4 visualised the number f clusters and utliers pint. The + symbl indicate utlier pints and symbl indicates number f clusters.

16 Chapter 8. Cluster Based Outlier Detectin in Data Mining 183 TABLE 8.6: utlier detectin with Fuzzy c-means Outliers Sepal.Length Sepal.Width Petal.Length Petal.Width iris2[, 2] iris2[, 1] FIGURE 8.4: Plt fr Fuzzy c-means and utliers TABLE 8.7: Cmparative methd f k-means, k-medids and Fuzzy c-means Features / Settings k-means k-medids Fuzzy c-means cmplexity O(ikn) O(i k(n k) 2 ) O(ndc2i) Efficiency Cmparatively Cmparatively Cmparatively mre less less Implementatin Easy Cmplicated Cmplicated cmputatin time Less Mre Mre Sensitive fr utliers Yes N N Necessity f Cnvex Shape Yes Yes Yes Nt s much Nt s much Nt s much Specificatin f number f clusters k Required Required Required Initial patrician affects results and runtime Yes Yes Yes Optimized fr Separated clusters Separated clusters Small Dataset Small Dataset

17 Chapter 8. Cluster Based Outlier Detectin in Data Mining Cnclusins Frm the abve study, it can be cncluded that partitining based clustering methds are suitable fr spherical shaped clusters in small t medium sized data sets. K-means k-medids and Fuzzy c-means are the methds find ut clusters frm the given database. Result and runtime depends upn initial partitin fr bth f these methds. The advantage f k-means is its lw cmputatin time, while drawback is sensitivity t nisy data and utliers. Cmpared t this, k-medid and Fuzzy c means are nt sensitive t nisy data and utliers, but it has high cmputatin time. The time cmplexity f the K-Means algrithm is O(ncd i ) and the time cmplexity f Fuzzy C Means algrithm is O(ndc2i ). Frm the btained results we may cnclude that K-Means algrithm is better than Fuzzy C Means algrithm. Fuzzy C Means prduces clse results t K-Means clustering but it still requires mre cmputatin time than K-Means because f the fuzzy measures calculatins invlvement in the algrithm. Infact, Fuzzy C Means clustering which cnstitute the ldest cmpnent f sftware cmputing, are really suitable fr handling the issues related t understand ability f patterns, incmplete/nisy data, mixed media infrmatin, human interactin and it can prvide apprximate slutins faster. They have been mainly used fr discvering assciatin rules and functinal dependencies as well as image retrieval. S, verall cnclusin is that K-Means algrithm seems t be superir than k-medids and Fuzzy C-Means algrithm.

Using SPLAY Tree s for state-full packet classification

Using SPLAY Tree s for state-full packet classification Curse Prject Using SPLAY Tree s fr state-full packet classificatin 1- What is a Splay Tree? These ntes discuss the splay tree, a frm f self-adjusting search tree in which the amrtized time fr an access,