arxiv: v1 [cs.db] 13 Sep 2017

Transcription

1 An effiient lustering algorithm from the measure of loal Gaussian distribution Yuan-Yen Tai (Dated: May 27, 2018) In this paper, I will introdue a fast and novel lustering algorithm based on Gaussian distribution and it an guarantee the separation of eah luster entroid as a given parameter,. The worst run time omplexity of this algorithm is approximately O(T N log(n)) where T is the iteration steps and N is the number of features. arxiv: v1 [s.db] 13 Sep 2017 I. INTRODUCTION Clustering algorithms have many appliations 1,2 to the unsupervised anemisupervised learning, whih they help lassify data sets into lusters within few given parameters. Potentially, it an be applied to many different domains suh as image analysis, objet traking, data ompression, physial/hemial struture optimization problems, to name a few. Some luster algorithms (e.g. K-means 3 and Gaussian- Mixture 4 ) require the knowledge of number of lusters as input parameters, anome others (e.g. Dbsan 5, Hdbsan 6 ) require a distane ut-off threshold to separate lusters. Eah lustering algorithm has its own pros and ons and its own domain for different appliations. My goal in this paper is to invent a new lustering algorithm whih has the following properties: Require eah luster to be a Gaussian-like distribution. Free from the luster number parameter. Able to deal with large number of data set effiiently. Over all, this new lustering algorithm is able to automatially searh for luster numbers with a given separation threshold in a fast and robust way, whih was diffiult for K-means and Gaussian-Mixture. While, if all lusters are gaussianlike, it is also more effiient than Hdbsan in finding major lusters automatially. The reason for suh effiient gain is based on the design of the new algorithm whih is using loal data information instead of ompare eah data pair in a global fashion. Nevertheless, this algorithm an be regarded as an effiient improvement between Gaussian-Mixture model and K-means, and an improvement of Hdbsan an be found in here 7. II. THE ALGORITHM This setion will walk through the onepts of this algorithm, and I will analyze more in-depth details in the next setion. Over all, the entire algorithm is going through the following analysis pipeline: a. Indexing all data points by the R-tree struture. b. Seeding entroids aross K-dimensional features.. Converge entroids and delete exessive ones to find the definitive luster entroid. weighted loal data points. e. Re-assign data points to eah luster id via the Gaussian distribution, P(x µ ). Where steps b. and. are designed for finding the definitive entroid of eah luster. One the definitive luster entroid has been found, the next step is to alulate o-variane matrix of the Gaussian distribution via a mean-field iterative alulation in step D. Finally, re-assign all the data points to the orrespond luster id in step e. The following paragraphs desribe more details of the entire algorithm. a. Indexing all data points by the R-tree struture. The very first step is to onstrut the K-dimensional R-tree with all the data points, x i, whih generally gives average O(log N) omplexity of multi-dimensional range searh. Here, the index i indiated the i-th data point. During the onstrution of R-tree, the minimum (x a min ) and maximum (x a max) value of feature, a, an also be found without loosing too muh alulation resoure. b. Seeding entroids aross K-dimensional features. Now, the entroids, µ t=0, are seeded with a given separation distane,, aross all features within [x a min, xa max], where t = 0 means the zeroth iteration of the entroid. Fig. 1.a shows the seeding of entroids, where µ 1 is the first entroid inside the red ube(square) ano on so forth. During the seeding proess, the algorithm also searh for the data points in the K-dimensional ubi of volume, d K s, around eah entroid µ, and thus find out the temporary loal data sets, x i, to the orrespond entroid. Note that, eah µ possess a loal ount of data points, N, inside eah d K s volume. By this nature, several entroids an be exluded if they possess too few ounts of data point by a given ounting threshold, N < L.. Converge entroids and delete exessive ones to find the definitive luster enters. The previous step an be regarded as the zeroth iteration of the onverge proess. In order to find the best luster enters, entroid need to be updated, µ, by alulate the means of x i, µ t+1 = 1 x i. (1) N i

2 2 (a) ~µ 2 ~µ 1 ~µ 3 ~µ 4 (b) 2 generator mn X i2 w i, (x m i µ m )(x n i µ n ), w i, P [ Input ] Pj2 P [ Input ]. () (d) Input = 1 2 ( t + t 1 ) t+1 Chek onvergene max element{ t+1,mn Input,mn } < FIG. 1. The illustration of the iterative onvergene of four entroid seed into two lusters in a K=2 (2 dimensional) data set. In (a), the ubes (squares) indiated the loal searhing d K=2 s volume of eah entroid. In (b), the ubes (squares) indiated the d 2 ollision box of eah entroid. While, the position of entroids are moved a little bit from (b) to (a) to illustrate the iterations of step. in the algorithm. After finding the definitive entroids, the loal data points has been olleted as shown in (), and used for the alulation of o-variant matrixs, Σ, to fit the distribution, P ( x, µ ), as shown in (d). After words, the iterations of eah entroid an be stopped by the riterion, µ t+1 µ t < ɛ, (2) where denotes the Eulidean distane under K- dimension and ɛ is the onvergene threshold. In Fig. 1.a, the dotted lines show the iteration paths of eah entroid, µ. During the iteration of entroids, two or more entroid seeds may ends up in the same luster enter. Aording to this fator, the iteration proess an be further speed up by delete the entroid µ with lower data point ounting, N, within a ollision detetion. The ollision detetion is defined in a K-dimensional ubi box that spanned by the ollision distane, d 2. Fig. 1.b shows µ 1 (red) and µ 3 (orange) to be deleted as they possess less ount (N ) than µ 2 (green) and µ 4 (blue). Note that, it is possible to set a different value of d instead of using d 2. However, it would be effiient and aurate by setting d 2. In prinipal, there ould be several different ways to implement the entroid deletion algorithm. However, I will not dive into too muh details in FIG. 2. The self-onsistent loop alulation for Σ for the proess of ii. and iii. in step d. of the algorithm. here. weighted loal data points. After finding all the definitive luster entroid, µ, the o-variant matrix Σ of eah luster are ready to be alulated via the loal data points, x i. Firstly, the Gaussian distribution is generally written, P ( x µ ) = 1 2π Σ e ( x µ)t Σ 1 ( x µ), (3) where, x and µ are olumn vetors and the T operation is the transpose of them. The following formula is the usual way to alulate the o-variant matrix from a given data set, = 1 (x m i µ m )(x n i µ n ). (4) N Σ mn i While, sine x i is not a omplete data set from eah luster entroid µ, it might indue inauraies. Therefore, Eq. 4 an be modified into the following equations, Σ mn i w i, (x m i µ m )(x n i µ n ), w i, P ( x i µ ) j P ( x j µ ). It means, some data points of x i get more important if they are loser to the luster entroid, µ, aording to the weight fator, w i,. Overall, Eq. 5 meant to keep good quality of the (5)

3 3 o-variant matrix, Σ, that even it is only alulated via the loal data set, x i. However, Eq. 5 also indiated a funtional form, P ( x i µ ) P i [ Σ ], and the following variational ondition an be arried out to yield the optimization of P [Σ ], δ d x P [Σ ] = 0. (6) A mathematial rigorous solution for Eq. 6 might be hard to get. While, Eq. 5 and 6 an be approximately solved by meanfield method, iteratively, (a) (b) i. Calulate Σ 0 from Eq. 4, anet Σ 1 = Σ 0, from Eq. 5 with the input of the o- ], where Σ Input = 1 2 (Σt + ii. Calulate Σ t+1 variant matrix for P [Σ Input Σ t 1 ), iii. Iterate proess ii. until, max element{ Σ t+1,mn Σ Input,mn } < ɛ, (7) where ɛ is the onvergene threshold. In above, the form of the seontep is to ensure a smooth iteration proess for the o-variant matrix. The entire proess of the self-onsistene loop alulation is shown in Fig. 2. e. Re-assign data points to eah luster enter via the Gaussian distribution, P( x µ ). Finally, after all of the ovariant matrix, Σ, for eah entroid are alulated, the assignment of eah data point, x i, beome very easy. x i µ max, where µ max max P-value of {P ( x i µ 0 ),..., P ( x i µ n )}. The entire proess of the lustering algorithm ends here. III. DISCUSSIONS TO THE ALGORITHM This setion is divided into three sub-setions, where I will disuss: A. how to set proper parameters for the lustering algorithm, B. analyze the run time omplexities of the analysis pipeline, and C. establish some post lustering proess to ensure the read out data quality. A. Parameter settings Three parameters was mentioned in ther previous setion that desribe the algorithm: The entroieparation distane,. The loal data ounting threshold, L. The onvergene threshold, ɛ. (8) FIG. 3. Two different settings for the entroieparation distane, (a) larger separation, (b) smaller separation. The red irle presenting the entroieeds, and the blue ube (square) indiated the ollision box. The onvergene threshold, ɛ 0.01, is a small number, and it is not sensitive in general. The loal data ounting threshold, L, is an empirial parameter whih depends on the amount of data points and the separation distane,. L only served the purpose to boost the initialization and iteration of the algorithm during in steps b. and., and the results will not be hanged that even L = 0. A good hoie of L an boost the alulation speed as well as eliminate some small lusters in the beginning. While the entroieparation distane is an important parameter, if it is wrongly set, the finding of lusters ould be hanged. If a larger value of is given, illustrated in Fig. 4 (a), the alulation ould be faster due to fewer ount of entroieeds in the initial stage. But it is possible that only a single luster entroid to be survived due to a large ollision distane, d 2. On the other hand, if a smaller value of is given, illustrated in Fig. 4 (b), the alulation will be slower a bit due to more ount of entroieeds, yet two luster an be found. However, if an extremely small value of has been set, it is possible to found eah luster that only possess a single data point, whih is undesirable. A simple rule of thub an be applied to make a good hoie of, where an be set lose to but smaller than one-half of the smallest distane from the atual luster enters. B. Run time omplexities It is ompliate to establish a preise analysis for the run time omplexity due to three inter-related variables: (a) number of features, (b) number of data points and () number of lusters. Therefore, I will fous on the speial situation for small number of feature with few lusters. Here I list the worst run time omplexity of eah step for this situation, a. Indexing all data points by the R-tree struture: O(N log(n)), b. Seeding entroids aross K-dimensional features: = M =0 N log(n) O(N log(n)),

4 P-value 4 L p L % Sorted data points, ~x i2 Gaussian distribution, the P-value an be easily traed with a good model fit, and Gaussian-Mixture model (GM) was invented to serve this purpose. However, GM is not able to deal with large number of data points due to its time omplexity, O(T N 2 ), and this is one of the reasons for the reation of this paper. Here, I define three ut-off threshold values to ensure the read out of data quality, P-value ut-off threshold, L p : For any luster of entroid-, x i, drop the data if P ( x i µ ) < L p. FIG. 4. Illustration of the definition of L p and L % for a given luster, x i. The red arrow indiated the portion of data to be dropped.. Converge entroids and delete exessive ones to find the definitive luster entroid: O(T µ N log(n)), weighted loal data points: O(T Σ N K 3 ), e. Re-assign data points to eah luster id via the Gaussian distribution, P(x µ ): O(N ), where N is the total number of data points, M is the number of entroieeds, T µ is the entroid iteration steps, N is the number of loal data points that belong to µ, T Σ is the iteration steps in d., and K is the dimension (feature) of the data. Note that, a matrix-inversion operation is required for Eq. 3, and it takes O(K 3 ) run time omplexity. If there are only few features (small number of K) to be onsidered, this operation ould be regarded as onstant run time omplexity. Finally, it is easy to know the bottlenek of the entire algorithm is in either. or d. whih depends on the types of data sets. Therefore, the upper limit run time omplexity ould be roughly estimated, O(T max N log(n)), where the maximum of iterations, T max, is less then 20 in most of the ases. C. Post lustering proess Clustering algorithms serve in many different purpose of usage. For example, in a loth store, the salesman an apply some lustering algorithms to suggest a ustomer to buy whih size of loth based on their hight and weight. In this situation, almost all data points (ustomers) should be onsidered, and thus to be assigned to the orrespond luster entroids (the size labels). However, it is not a good idea to inlude all data points in some other situations, where some falsely lassified data points need to be avoid based on experimental fats. In this situation, it is more prefferable to take data points whih are loser to the luster entroid. If we are dealing with the seonenario, the benefit of the Gaussian distribution beome obvious. Due to the nature of Perentage ut-off threshold, L % : For any luster of entroid-, x i, sort the data points aording to the P-value in asending order, sort{p ( x i µ )}, (9) and drop L % of data from the begin of the sorted data list. Separation ut-off threshold, L s : For any data point, x i, alulate first aneond maximum value of P ( x i µ ) denoted as P ( x i 1 st ) and P ( x i 2 nd ). Finally, drop data x i if the followig riterion mathed, P ( x i 1 st ) P ( x i 1 st ) + P ( x i 2 nd ) < L s. (10) In general, L p itself an ensure the data quality and exlude the false lassified data points. However, the simple definition of L p may ause the imbalane loading of eah luster due to the variane of shapes of eah luster, P ( x µ ). Therefore, L % an better ensure a balaned loading of data points in eah luster. After all, L s an ensure the separations of two lusters. IV. CONCLUSION In this paper, I introdued an effiient multi-dimensional lustering algorithm based on the multivariate Gaussian funtion. The run time omplexity of this new algorithm is muh better then the Gaussian mixture model due to the lusters are alulated with loally weighted data points. On the other hand, similar to Hdbsan algorithms, it automatially find out the loations of eah luster with a better run time omplexity (the run time omplexity for Hdbsan is roughly O(N 2 log N)). It would be worth to mention, sine most of the operations of this new algorithm are just vetor summations, whih means it an be easily aelerated with a multithread parallel sheme. While, one an perform a similar alulation by the mixture of Dbsan/Hdbsan and Gaussian mixture model, 1. run Dbsan/Hdbsan to find out the optimal luster number and approximate loations of entroids,

5 5 2. run Gaussian Mixture model based on the input of luster number and loations of entroids. However, due to the run time omplexity of Dbsan/Hdbsan and Gaussian mixture model and the diffiulties of parallel the algorithm for minimal-spanning-tree, it is not pratiall to take this mixed two step algorithm rather then our new algorithm. I have testeome randomly generated luster data with 150,000 features, over all, the run time is less than 20 seonds under a 2.5GHz single threaded CPU. The ode is written in C++ with boost library for the need of R-tree data struture. 1 C. C. Aggarwal and C. K. Reddy, Data Clustering: Algorithms and Appliations, CRC Press, (2013). 2 A. A. Abbasi and M. Younis, ACM Computer Communiations 30 (1415) , (2007). 3 M. Inaba, N. Katoh and H. Imai, Proeedings of 10th ACM Symposium on Computational Geometry. pp , (1994). 4 R. O. Duda and P. E. Hart, Pattern lassifiation anene analysis, John Wiley & Sons, (1973). 5 M. Ester, H.-P. Kriegel, J. Sander and X. Xu, AAAI Press. pp , (1996). 6 R. J. G. B. Campello, D. MoulavI, J. Sander, Density-based lustering based on hierarhial density estimates., ACM 10 Issue 1, Artile No. 5., (2015), 7 L. MInnes and J. Healy, arxiv: , (2017).