Detection of Outliers and Reduction of their Undesirable Effects for Improving the Accuracy of K-means Clustering Algorithm

Size: px

Start display at page:

Download "Detection of Outliers and Reduction of their Undesirable Effects for Improving the Accuracy of K-means Clustering Algorithm"

Theodore Ford
5 years ago
Views:

1 Detection of Outliers and Reduction of their Undesirable Effects for Iproving the Accuracy of K-eans Clustering Algorith Bahan Askari Departent of Coputer Science and Research Branch, Islaic Azad University, Khouzestan, Iran Sattar Hashei Departent of Coputer Science & Engineering, Shiraz University, Shiraz, Iran Mohaad Hossein Yektaei Departent of Coputer Abadan Branch, Islaic Azad University, Abadan, Iran Abstract: Clustering is an unsupervised categorization technique and also a highly used operation in data ining, in which, the data sets are divided into certain clusters according to siilarity or dissiilarity criterions so that the assigned objects to each cluster would be ore siilar to each other coparing to the objects of other clusters. The k-eans algorith is one of the ost well-known algoriths in clustering that is used in various odels of data ining. The k-eans categorizes a set of objects into certain nuber of clusters. One of the ost iportant probles of this algorith occurs when encountering to outliers. The outliers in the data set lead to getting away fro the real cluster centers and consequently a reduction in the clustering algorith accuracy. In this paper, we separate outliers fro noral objects using a echanis based on dissiilarity of objects. Then, the noral objects are clustered using k- eans algorith process and finally, the outliers are assigned to the closest cluster. The experiental results show the accuracy and efficiency of the proposed ethod. Keywords: clustering, k-eans, outliers, outlier detection 1. INTRODUCTION Clustering is one of the highly used ethods in data ining [12], wireless sensor networks [8,13], pattern recognition [14] and achine learning [7], which is used to detect the groups that are different enough fro each other and contain siilar objects [4,5]. The iportance of clustering in various fields and also the type of data being used, clustering speed, accuracy and lots of other paraeters, leads to introduce various ethods and algoriths in data clustering. Clustering is an unsupervised technique in which the data sets which are K-eans algorith Input: Data set D = { d 1, d 2, d n }, where d i =data points, n= nuber of data points K = nuber of cluster centers Output: Clusters : K clusters with their centers Step 1: usually vectors in ulti-diension space are divided into a Randoly select k data object fro dataset D as initial cluster certain nuber of clusters based on a siilarity or centers. dissiilarity criterions. For exaple, if the nuber of clusters Step 2: is K, and there exist n nuber of -diension data, the Repeat step 3 to step 4 till no new cluster centers are found clustering algorith will assign each one of these data to a Step 3: cluster. This assignent takes place according to this rule that Calculate the distance between each data object di (1<=i<=n) the assigned data to a certain cluster are ore siilar to each and all K cluster centers Cj (1<=j<=K) and assign data object other rather than the other clusters. The k-eans algorith is di to the nearest cluster. one of the ost well-known clustering algoriths and is being Step 4: used in various types of data ining. The k-eans categorizes For each cluster j (1<=j<=K), recalculate the cluster center. data set objects in certain nubers of clusters [10,3]. This ethod is one of the ost attractive and highly used operations in clustering techniques, because it is siple and Figure 1. The pseudo-code of k-eans algorith understandable and its tie coplexity is linear. In general, this algorith consists of two phases. In the first phase, k The distance to the cluster center is calculated in the following nubers of objects are selected fro data set in a rando ethod: anner and are considered as the initial centers of clusters. In the second phase, the distance between objects and the clusters centers is deterined and each object is placed in the (1) Dis tan( ce d,) i C j d ip C ip p 1 nearest cluster. To deterine the distance between objects, the Euclidean distance criterion is used, generally. When all of Where, d i denotes the i-th vector of data, C j is the center of j- the objects have been placed in the corresponding clusters, the th cluster and denotes the nuber of attributes and cluster clusters centers are calculated using repetitive averaging of centers. objects of each cluster. The second phase continues until satisfying the algorith ending condition. The Pseudo-code of The cluster centers are updated according to the following k-eans algorith is shown in figure 1. equation: 552

2 (2) 1 C j d n i j d i Cluster j Where n j denotes the nuber of vectors in the j-th cluster and cluster j is a subset of all vectors which for the j-th cluster. Figure 2 illustrates the clustering steps of a anual data set. This dataset is divided into two clusters using the k-eans algorith. At first, two clusters are fored by rando selection of objects (figure 2 -a), then the clusters centers are deterined by averaging of objects of each cluster (figure 2- b) and the clustering process continues using the new cluster centers (figure 2-c). For this dataset, the clustering is finished after two iterations of the algorith (figure 2-d). neighbors based that are not suitable for outlier detection in data streas due to their high tie coplexity. He et al in [6] presented new definition of outlier which they naed as cluster-based local outlier, which provides iportance to the local data behavior. They defined Cluster-Based Local Outlier Factor (CBLOF), a easure for identifying the physical significance of an outlier and an algorith for discovering outliers is also proposed by the. After that Duan et al in [9] proposed a cluster based outlier detection algorith which can detect both single point outliers and cluster-based outliers, and can assign each outlier a degree of being an outlier. Zhuo et al in [15] presented an outlier ining algorith based on dissiilarity (OMABD), which detected outliers by coparing the dissiilarity degree with dissiilarity threshold. A dissiilarity based ethod is used for iproving the efficiency of k-eans algorith in this study. Figure 2. clustering of anual data using the k-eans algorith The k-eans algorith has soe pitfalls. For instance, it stops in local optius and is sensitive to the initial values of clusters centers and outliers in dataset. In each dataset, an outlier is an object which its distance is not noral coparing to the other objects. In other words, an outlier is an object that has less siilarity than the other objects. The existence of these objects in dataset has an undesirable effect on the efficiency and accuracy of the clustering algoriths such as the k-eans. 2. RELATED WORK Outlier detection has been a very interesting topic for research counity [11,1,6,2,9]. Raasway et al proposed a distance based outlier detection ethod in [11]. According to which, given paraeters k and n, an object is an outlier if no ore than n-1 other object in the dataset have higher value for D k than object o, where D k (o) denotes the distance of k-th nearest neighbor of object o. This idea is further extended in [14], where each data point is ranked by the su of distance fro its k-th nearest neighbors. Breunig et al introduced the notion of the Local Outlier Factor (LOF) in [1] which captures the relative degree of outlierness of an object. It is local in sense that the degree of outlierness depends on how isolated an object is with respect to surrounding neighborhood. Above described ethods are either distance based or nearest 3. presented odel To iprove the efficiency of the K-eans algorith, at first the dataset should be investigated for specifying and detecting the outliers. After that, the dataset should be divided into two subsets of noral object and outliers. Then, the clustering process should be perfored separately for noral object and outliers. The noral object are clustered using the entioned steps of the k-eans algorith shown in figure 1, and finally, the outliers are assigned to the nearest cluster according to the Euclidean distance criterion and calculated cluster centers fro the previous steps. 3.1 Outlier detection in dataset In this paper, the ODBD algorith is presented for specifying and detecting the outliers which is based on the dissiilarity of objects. According to this algorith, a value is calculated for all of the dataset objects which is called the dissiilarity degree. The objects that their degree is higher than the threshold value are considered as the outlier objects. Assue that a data set DS is defined in the for of: DS= (D,A) in which D={d 1,d 2, d n } is the set of n objects and A={a 1,a 2, a } is the set of attributes with the order of. The dissiilarity degree of two objects d i, d j D on the attribute of fa is calculated as following: (3) d if d f d jf d f ad f ij d ax f d in f Where dif and d jf are the values of attribute f in the objects i and j, respectively. d f, d ax and d in are the average, the axiu and the iniu value of attribute f on all objects of the dataset, respectively. The dissiilarity degree of two objects can be obtained fro the average of these objects dissiilarity on each of attributes, as following: (4) od ( i,) j ak 1 Where ak ad ij the a k -th attribute. ak ad ij 2 is the dissiilarity value of the objects i, j on

3 According to the relations (3) and (4), the dissiilarity atrix d, which is an order N square atrix, is created to calculate the dissiilarity of each object with respect to the other objects. By adding the row eleents of this atrix, the objects synergic dissiilarity atrix is ade which shows the dissiilarity degree of each object with respect to all other objects of the data set. For the sake of siplicity and siplification of coparisons, the synergic dissiilarity degree atrix is noralized and then, the average of eleents of this atrix with an ipact factor value in the range of [0,1] is considered as the threshold siilarity value. The ODBD pseudo-code algorith is shown in figure 3. noral process of k-eans algorith is used to cluster the noral object. In this phase, because of data set being pruned by the ODBD algorith and the use of noral object, the centers and the objects are deterined in a ore accurate way. In the second phase, we use the centers obtained fro the previous phase and with iteration, we calculate the distance between each of outliers and these centers. Then, each outlier is assigned to the nearest cluster. The Euclidean distance criterion is used for the calculation of this distance. The presented algorith pseudo-code is illustrated in figure 4. ODBD-K-Means algorith ODBD algorith input: data set D = { d 1, d 2, d n }, where d i =data points, n= nuber of data points ipact factor value IFV [0,1] output: outlier and noral objects step 1: step 1-1: for each data object d i fro D for each data object d j fro D calculate od(i,j) by using equation (3) and (4) d(i,j)=od(i,j) step 1-2: for each row r i in d calculate su of eleents and assign in sd i calculate d ax = axiu value in sd step 1-3: for each data value sd i in sd dd i = (d ax - sd i ) / d ax td=ean(dd) *IFV step 2: for each data object d i fro D if dd i < td assign d i to outlier objects else assign d i to noral objects end if Figure 3. The pseudo-code of ODBD algorith 3.2 Iproving the Clustering process with the ODBD-k-eans algorith Selecting the initial values in the noral k-eans algorith is copletely rando. Thus, the existence of the outliers results in the cluster centers getting distance fro real position and consequently decreasing the accuracy of this algorith. In the presented algorith it is attepted that the outliers do not affect the process of selecting the clusters center. For this purpose, after separating the data set objects using ODBD, the clustering is done during two phases. In the first phase, the input: data set D = { d 1, d 2, d n }, where d i =data points, n= nuber of data points k = nuber of cluster centers output: k clusters with their centers step 1: find outliers and noral objects by using ODBD algorith step2: step 2-1: randoly select k data object fro noral objects as initial cluster centers. step 2-2: repeat step 2-3 to step 2-4 till no new cluster centers are found step 2-3: calculate the distance between each data object d i (1<=i<=size(noral object)) and all k cluster centers C j (1<=j<=k) and assign data object d i to the nearest cluster. step 2-4: for each cluster j (1<=j<=k), recalculate the cluster center. step 3: for each data object d i fro outliers step 3-1 calculate the distance of d i to all k final cluster centers C fro step 2 by using euclidean distance step 3-2 find the closest center c j and assign d i to the cluster with nearest center C j Figure 4. The pseudo-code of ODBD-k-eans algorith 4. Results and discussion To evaluate the algorith presented in this study, the algorith is ipleented using MATLAB 2010 prograing software and the results are copared with the k-eans algorith. The Iris, Bupa and Glass data sets fro UCI are used in the experients. The Iris data set is a categorization of iris flowers in which, there exists three different classes of iris and each class contains 50 objects. Each object has 4 attributes. In the Bupa data set, 345 objects exist each having 6 attributes. The attributes are gathered fro blood tests concerning the diagnosis of liver hapering caused by irregular drink of alcohol. Each object of this data set is the record of a ale person. In the glass data set, 214 objects exist and each object has 9 attributes and this set has 6 classes. The properties of these data sets are listed in table

4 Table 1. The datasets and their properties dataset #objects #features #clusters Iris Bupa Glass Selecting the ipact factor and consequently the siilarity threshold value is very iportant to detect and deterine the nuber of outliers. This factor value ight be different for each data set. According to the ODBD algorith, if the ipact factor assued to be zero, the nuber of outliers would be zero. This eans that the ODBD-k-eans algorith changes to the noral k-eans algorith. For calculating the algorith accuracy, we can use accuracy indicator which is obtained using to the following relation: According to this table, it is obvious that the algorith accuracy is dependent to the nuber of outliers. If we select zero as the value of the ipact factor, the presented algorith would be equivalent to the noral k-eans algorith. The real outliers of the Iris data set are the points that are separated with an ipact factor of 0.4. By running the ODBD algorith and selecting the ost suitable ipact factor for data set, the nuber of outliers in each data set is obtained. The results are shown in table 3. Table 3. The nuber of outliers in data sets dataset Ipact factor #outlier objects Iris Bupa (5) Accuracy True Positive TruePositive False Postive Glass Because the nuber of noral object affects the selection of cluster centers and also the proposed algorith like the k- eans algorith calculates the initial centers by rando selection of objects, the results of each execution of this algorith ay not the sae. For different values of ipact factor in the range [0,1], the presented algorith is executed for 100 ties and according to the relevant nuber of outliers, the average value of results is considered as the algorith accuracy. These results are presented in table 2, for Iris data set. Results of algorith accuracy on the data sets are shown in table 4. Table 4. The results of algoriths on datasets Dataset Iris Bupa K-eans ODBD-K-eans Table 2. Results of ODBD-k-eans algorith on the iris Glass Ipact factor #outlier Accuracy Figure 5 depicts the algorith accuracy results on a diagra. According to this diagra, effect of the presented algorith on the standard data sets can be observed Figure 5. The results of algoriths on standard datasets The k-eans algorith has a good accuracy on the Iris data set, because of a suitable distribution and structure of objects in data set. The existence of the outliers in the Bupa and Glass 555

5 data sets, has an undesirable effect on the selection of centers and objects and it consequently leads to reduction in the accuracy of the k-eans algorith. After teporarily reoving this objects and using the presented algorith, the retrieval accuracy on these three data sets have been iproved. 5. Conclusion In this paper, a new algorith is presented based on the k- eans algorith and dissiilarity of the objects. In the presented odel, the data sets were analyzed to specify the outliers. By detecting these objects, the data set pruned and the outliers and noral objects were separated fro each other. Then, the noral objects were grouped using the k- eans algorith. Finally, using of the final cluster centers of previous stage and by calculating the distances between the outliers and the final cluster centers, these objects were assigned to the nearest cluster center, separately. The experiental results on the standard data sets showed that the presented algorith probes the data sets with a better accuracy for finding better results. Despite the good results of the presented algorith, different results were observed at different executions of the algorith and that is due to the rando selection of the initial points. In the future studies, it is possible to present better solutions for this challenge. 6. REFERENCES [1] Breuing, M., Kriegel, H., Ng, R., and Sander J LOF: identifying density-based local outlier factor. In Proceedings of ACM SIGMOD International Conference on anageent of Data., 29(2), pp [2] Fabrizio, A., Stefano B., and Clara P Distancebased detection and prediction of outliers. IEEE Transaction on Knowledge. And data Engineering., 18(2), pp [3] Frogy, E Cluster analysis of ultivariate data: efficiency vs interpretability of classification. Bioetrics., 21(3), pp [4] Guha, S., Rastogi, R., and Shi, K An efficient clustring algorith for large database. In Proceedings of the ACM SIGMOD Conference., 27(2), pp [5] Guha, S., Rastogi R., and Shi K A Robust clustring algorith for categorical attributes. In Proceedings of the third IEEE International Conference on Data Mining., 25(5), pp [6] He, Z., Xu X., and Deng, S Discovering clusterbased local outlier. Pattern Recognization Letters., 24(9), pp [7] Kao, Y., and Lee S.Y Cobining k-eans and particle swar optiization for dyneic data clustring probles. IEEE, International Conference on Intelligent Coputing and Intelligent Systes., Shanghai, pp [8] Kuar, M., and Vera, S Data clustring in sensor network using art. 4th International Conference on Wireless Counication and Sensor Network, Allahabad, pp [9] Lian D., Lida, X., Ying, L., and Jun, L Clusterbased otlier detection. Annals of Operation Research, 68(11), pp [10] Pelleg, D., and Moore, A X-eans: extending k- eans with efficient estiation of the nuber of clusters. In Proceedings of ICML the Seventeenth International Conference on Machine Learning., San Francisco, pp [11] Roaswany, S., Rastogi R., and Shi K Efficient algorith for ining outliers fro large data set. In Proceedings of the ACM SIGMOD International Conference on Manageent of Data., Texas, ACM press, pp [12] Tsai, C. F., Tsai, C. W., Wu, H. C. and Yang, T A new data clustering approach for data ining in large databases. The 6th IEEE International Syposiu on Parallel Architectures, Algoriths, and Networks., pp [13] Wang, T., and Yang, Z. A location-aware-based data clustring algorith in wireless sensor networks IEEE, 11th International Conference on Counication Systes., pp [14] Wong, A. K. C., and Li, G. C. L siultaneous pattern and data clustring for pattern cluster analysis. IEEE Transaction on Knowledge and Data Engineering., 20(8), pp [15] Zhou, M., and Chen, X an outlier ining algorith based on dissiilarity. International Conference on Environental Science and Engineering., pp

Clustering. Cluster Analysis of Microarray Data. Microarray Data for Clustering. Data for Clustering

Clustering. Cluster Analysis of Microarray Data. Microarray Data for Clustering. Data for Clustering Clustering Cluster Analysis of Microarray Data 4/3/009 Copyright 009 Dan Nettleton Group obects that are siilar to one another together in a cluster. Separate obects that are dissiilar fro each other into