Micro-Batching Growing Neural Gas for Clustering Data Streams using Spark Streaming

Transcription

1 Proedia Computer Siene Volume 53, 2015, Pages INNS Conferene on Big Data Miro-Bathing Growing Neural Gas for Clustering Data Streams using Spark Streaming Mohammed Ghesmoune, Mustapha Lebbah, Hanene Azzag University of Paris 13, Sorbonne Paris City LIPN-UMR CNRS 99, av. J-B Clément F Villetaneuse, Frane firstname.seondname@lipn.univ-paris13.fr Abstrat In reent years, the data stream lustering problem has gained onsiderable attention in the literature. Clustering data streams requires a proess apable of partitioning observations ontinuously while taking into aount restritions of memory and time. In this paper we present MBG-Stream, a Miro-Bathing version of the growing neural gas approah, aimed to lustering data streams by making one pass over the data. MBG-Stream allows us to disover lusters of arbitrary shapes without any assumptions on the number of lusters. The proposed algorithm is implemented on a distributed streaming platform, the Spark Streaming API, and its performane is evaluated on publi data sets. Keywords: Data Stream Clustering, GNG, Topologial Struture, Miro-Bath Streaming 1 Introdution When applying data mining tehniques, or more speifially lustering algorithms to data streams, restritions in exeution time and memory have to be onsidered arefully. Currently, distributed streaming systems are based on two proessing models, reord-at-a-time and miro-bathing. On a reord-at-a-time proessing model, long-running stateful operators proess reords as they arrive, update internal state, and emit new reords. On the other side, the miro-bathing proessing model runs eah streaming omputation as a series of deterministi bath omputations on small time intervals. Among the available frameworks that implements the miro-bathing proessing model, we an find Spark Streaming [1]. It is an extension of the ore Spark API that enables high-throughput, reliable proessing of live data streams. In a previous work, G-Stream [2] was proposed as a data stream lustering approah based on the Growing Neural Gas algorithm. G-Stream uses a stohasti approah to update the prototypes, and it was implemented on a entralized platform. In this paper, we propose MBG-Stream, a novel algorithm for disovering lusters of arbitrary shape in an evolving data 158 Seletion and peer-review under responsibility of the Sientifi Programme Committee of INNS-BigData2015 The Authors. Published by Elsevier B.V. doi: /j.pros

2 stream. MBG-Stream is implemented on a distributed streaming platform based on the mirobathing proessing model, i.e., the Spark Streaming API. In the proposed algorithm, the topologial struture is represented by a graph wherein eah node represents a luster, whih is a set of lose data points and neighboring nodes (lusters) are onneted by edges. Starting with only two nodes, the graph size is not fixed but may also evolve as several nodes (lusters) are reated in eah iteration. We use an exponential fading funtion to redue the impat of old data whose relevane diminishes over time. For the same reason, links between nodes are also weighted by an exponential funtion. The data reeived in eah interval is stored reliably aross the luster to form an input dataset for that interval. One the time interval is ompleted, this dataset is proessed via deterministi parallel operations, suh as map and redue to produe new datasets representing either program outputs or intermediate states [1]. The input data is split and the master assigns the splits to Map workers. Eah worker proesses the orresponding input split, generates key/value pairs and writes them to intermediate files (on disk or in memory). The Redue funtion is responsible for aggregating information reeived from Map funtions. The remainder of this paper is organized as follows: Setion 2 is dediated to related works. Setion 3 desribes the MBG-Stream algorithm. Setion 4 reports the experimental evaluation on both syntheti and real-world data sets. Setion 5 onludes this paper. 2 Related Works This setion disusses previous works on data stream lustering problems, and highlights the most relevant algorithms proposed in the literature to deal with this problem. Most of the existing algorithms divide the lustering proess in two phases: (a) Online, the data will be summarized; (b) Offline, the final lusters will be generated. Both CluStream [3] and Den- Stream [4] use a temporal extension of the Clustering Feature vetor(alled miro-lusters) to maintain statistial summaries about data loality and timestamps during the online phase. By reating two kinds of miro-lusters (potential and outlier miro-lusters), DenStream overomes one of the drawbaks of CluStream, its sensitivity to noise. In the offline phase, the miro-lusters found during the online phase are onsidered as pseudo-points and will be passed to a variant of k-means in the CluStream algorithm (resp. to a variant of DBSan in the DenStream algorithm) in order to determine the final lusters. ClusTree [5] is an anytime algorithm that organizes miro-lusters in a tree struture for faster aess and automatially adapts miro-luster sizes based on the variane of the assigned data points. Any lustering algorithm, e.g. k-means or DBSan, an be used in its offline phase. The merge step is performed by a means of a data struture, named the buket set. The redue step is performed by a signifiantly different summary data struture, the oreset tree. [6], [7] and [8] give approximations of the streaming k-means algorithm. G-Stream [2] is an extension of the GNG algorithm to data streams. Whereas all the previous algorithms are implemented on entralized platforms, we propose in this paper a new approah for lustering data streams implemented on a distributed platform. 3 Miro-Bathing Clustering In this setion we introdue Miro-Bathing Growing Neural Gas for Clustering Data Streams (MBG-Stream) and highlight some of its novel features. MBG-Stream is based on Growing Neural Gas (GNG), whih is an inremental self-organizing approah that belongs to the family of topologial maps suh as Self-Organizing Maps (SOM) [9] or Neural Gas (NG) [10]. It is 159

3 an unsupervised algorithm apable of representing a high dimensional input spae in a low dimensional feature map. Typially, it is used for finding topologial strutures that losely reflet the struture of the input distribution. We assume that the data stream onsists of a sequene DS = {x 1, x 2,..., x n } of n (potentially infinite) elements of a data stream arriving at times t 1,t 2,..., t n,wherex i =(x 1 i,x2 i,..., xd i ) is a vetor in Rd.WedenotebyX 1 = {x 1,..., x p } where p is the size of the window, thus DS = {X 1, X 2,..., X L }. At eah time, MBG-Stream is represented by a graph C where eah node represents a luster. Eah node Chas (a) aprototypew = (w,w 1, 2..., w d ) representing its position; (b) π representing the weight of this node; () error() an error variable representing the distane between this node and the assigned data-point. When data arrive in a stream, we may want to estimate lusters dynamially, updating them as new data arrive. An implementation of a Growing Neural Gas algorithm over Data Stream on a entralized platform would be as follows [2]: Starting with two nodes, and as a new data point is reahed, the nearest and the seond-nearest nodes are identified, linked by an edge, and the nearest node with its topologial neighbors are moved toward the data point. Eah node has an aumulated error variable and a weight, whih varies over time using a fading funtion. Using an edge management proedure, one, two or three nodes are inserted into the graph between the nodes with the largest error values. Nodes an also be removed if they are identified as being superfluous. However, the design of a distributed version of G-Stream [2] would raise diffiulties. MBG- Stream an disover lusters of arbitrary shape in an evolving data stream. It operates with parameters to ontrol the deay (or forgetfulness ) of the estimates. The algorithm uses a generalization of the mini-bath GNG update rule. In the adaptation step of the GNG algorithm, the nearest node and all of its neighbors are moved in the diretion of the data point. However, in MBG-Stream (see Algorithm 1 for detail), for eah bath of data X p,we assign all points x i to their best math unit, ompute new luster enters, then update eah luster. The update rule, i.e., the adaptation step, in a mini-bath version without taking into aount the neighbors of the referent would be as desribed in Equation 1: w (t+1) = w(t) n (t) α + z (t) m (t) n (t) α + m (t) (1) whereas Equation 2 updates the number of points assigned to the luster, where w (t) previous enter for the luster, n (t) is the new luster enter from the urrent bath, and m (t) luster in the urrent bath. is the is the number of points assigned to the luster thus far, z (t) is the number of points added to the n (t+1) = n (t) + m (t) (2) In most data stream senarios, more reent data an reflet the emergene of new trends or hanges in the data distribution [11]. There are three window models ommonly studied in data streams: landmark, sliding and damped. We onsider the damped window model, in whih the weight of eah data point dereases exponentially with time via a fading funtion. The weight of eah node dereases exponentially with time t via a deay fator parameter 0 <α<1, i.e., π (t+1) = π (t) α (3) If the weight of a node is less than a threshold value then this node is onsidered as outdated and then deleted (with its links). The deay fator an be used to ignore the past: with α = 1 all data will be used from the beginning; with α = 0 only the most reent data will be used. This is analogous to the fading funtion [11] whih is defined as follows : f(t) =2 λt,whereλ>0. 160

4 In a general ase, when the referent moves toward a data-point, it also moves its neighborhood toward this point [9]. In our model, we use Equation 4 to arry out the adaptation step: w (t+1) = w(t) n (t) α + r C K(r, )z(t) r m (t) r n (t) α + r C K(r, )m(t) r where z (t) r is the previous enter for the luster r (whih is a neighbor of the onsidered referent node), K is alled the neighborhood funtion defined in Equation 5, where δ(r, ) is the length of the shortest path between nodes r and : K(r, ) =exp (4) ( ) δ(r, ). (5) T The funtion updaterule performs operations related to updating graph edges. The way to inrease the age of edges is inspired by the fading funtion in the sense that the reation time of a link is taken into aount. Contrary to the fading funtion, the age of the links will be strengthened by the exponential funtion 2 λage(t t0),whereλ age > 0, defines the rate of growth of the age over time, t denotes the urrent time and t 0 is the reation time of the edge. The next step is to add a new edge that onnets the two losest nodes. The last step is to remove eah link exeeding a maximum age, sine these links are no longer useful beause they were replaed by younger and shorter edges that were reated during the graph refinement in step 9. The input data is split and the master assigns splits to Map workers. Eah worker proesses Algorithm 1: MBG-Stream Input: DS = {x 1, x 2,..., x n }, α, λ age, the number of nodes to add at eah iteration, π min, age max Output: set of nodes C = { 1, 2,...} and their prototypes W = {w 1, w 2,...} 1 Initialize of the model by reating a graph of two nodes (the first 2 data-points) 2 while there is a miro-bath to proeed do 3 D t get the miro-bath of data points arrived at time interval t 4 Apply the mapping step as desribed in Funtion map 5 Apply the redue step as desribed in Funtion redue 6 Adaptation step: updaterule(pointstats, α, λ age, age max ) 7 Update the variable error of eah node 8 Apply fading, delete isolated nodes 9 Add new nodes as desribed in Funtion addnewnodes 10 Derease the error of all units 11 end the orresponding input split, generates key/value pairs and writes them to intermediate files (ondiskorinmemory). Thekey orresponds to the bmu whereas its value represents a tuple of (bmu 2, error, point, 1). Then the master will launh redue tasks that take as input both the results of the maps and the results of the previous interval s redues. The Redue funtion is responsible for aggregating information reeived from Map funtions. For eah key, the Redue funtion works on the list of values, losest. To ompute the entroid of eah node, the Redue funtion groups by bmu and sums the values reeived in the losest list. The final output is the list pointstats. Eah element of pointstats ontains a bmu, askey, with the seond nearest node, the sum of errors, the sum and the ount of points assigned to eah node, as the value. 161

5 Funtion map(d t :thet-th miro-bath of data points) 1 foreah x ti D t do 2 Key bmu 1, the nearest node 3 Value (bmu 2,error,x ti, 1) suh as: bmu 2 is the seond nearest node, and error = x ti w bmu1 2 4 Emit (Key, Value) 5 end Funtion redue(key t,listlosest) Output: entroid t :entroidofthet-th miro-bath, ount t : number of data points in the t-th miro-bath 1 bmu 2 0; error t 0; sum t 0; ount t 0; 2 foreah value t losest do // suh as value t is the orresponding value of the pair (key t, Value) 3 bmu 2 bmu 2 + the 1-st value of tuple value t 4 error t error t + the 2-nd value of tuple value t 5 sum t sum t + the 3-th value of tuple value t 6 ount t ount t + the 4-th value of tuple value t 7 end 8 entroid t sumt ount t 4 Experimental Evaluations In this setion, we present an experimental evaluation of the MBG-Stream algorithm. We ompared our algorithm with several well-known and relevant data stream lustering algorithms, inluding ClusTree, DenStream, and the MLlib implementation of Streaming-KMeans. Our experiments were performed on Spark Streaming platform using real-world and syntheti data sets. All the experiments are onduted on a PC with Core(TM)i7-4800MQ with two 2.70 GHz proessors, and 8GB of RAM, whih runs Ubuntu operating system. Funtion updaterule(list pointstats, α, λ age, age max ) // Derease the weight of nodes 1 foreah Cdo π α.π ; 2 foreah ps pointstats do // ps is a tuple: (bmu, (bmu 2, error, sum, ount)) 3 Calulate the new entroid as desribed in Equation 4 4 Inrement the age of all edges emanating from bmu and weight them 5 if bmu and bmu 2 are onneted by an edge then set the age of this edge to zero ; 6 else reate an edge between bmu and bmu 2, and mark its time stamp; 7 end 8 Remove the edges whose age is greater than age max. If this results in nodes having no emanating edges, remove them as well 162

6 Funtion addnewnodes(η : number of nodes to add) 1 for j 1 to η do 2 Find the node with the largest error 3 Find the neighbor f with the largest aumulated error 4 Add the new node r half-way between nodes q and f: w r 0.5(w q + w f ) 5 Insert edges onneting the new unit r with units q and f, and remove the original edge between q and f. Remove the original edge between q and f 6 Initialize the weight of r and the age of edges emanating from r to zero 7 Derease the error variables of q and f by multiplying them with a onstant ɛ where: 0 <ɛ<1 8 Initialize the error variable of r with the new value of the error variable of q 9 end Table 1: Overview of all data sets. Datasets #reords #features #lasses DS1 9, letter4 9, Sea 60, HyperPlan 100, KddCup99 494, CoverType 581, Sensor 2,219, Data Sets and Quality riteria To evaluate the lustering quality and salability of the MBG-Stream algorithm both real and syntheti data sets are used. The syntheti data sets used are DS1 and letter4. All the others are real-world publily available data sets. Table 1 overviews all the data sets used. DS1 is generated by bz2. The letter4 data set is generated by a Java ode Token-Cluster-Generator. The Sea data set was taken from kdus/produts/datasets-for-onept-drift. The HyperPlan data set was taken from [12]. The real-world databases were taken from the UCI repository [13], whih are the KDD-CUP 99 Network Intrusion Detetion stream data set (KddCup99) and the Forest CoverType data set (CoverType) respetively. The algorithms are evaluated using three performane measures: Auray (Purity), Normalized Mutual Information (NMI) and Rand index. The value of eah measure lies between 0 and 1. A higher value indiates better lustering results. The Auray (Purity) averages the fration of items belonging to the majority lass of in eah luster. A = K 100%, where K denotes the number of lusters, Ni d denotes the number of points with the dominant lass label in luster i, andn i denotes the number of points in luster i. Intuitively, the auray (purity) measures the purity of the lusters with respet to the true luster (lass) labels that are known for our data sets [4]. Normalized mutual information provides a measure that is independent of the number of lusters as ompared to purity [14]. It reahes its maximum value of 1 only when the two sets of labels have a perfet one-to-one orrespondene. The Rand index measures how aurately a lusterer an lassify data elements by omparing luster labels with K i=1 N d i N i 163

7 Table 2: Comparing G-Stream with different algorithms. Datasets MBG-Stream Streaming-KMeans DenStream ClusTree A DS1 NMI Rand A letter4 NMI Rand A Sea NMI Rand A HyperPlan NMI Rand A KddCup99 NMI Rand A CoverType NMI Rand A Sensor NMI Rand the underlying lass labels. 4.2 Evaluation and performane omparison This setion aims to evaluate the lustering quality of the MBG-Stream and ompare it to well-known data stream lustering algorithms. As explained in setion 3, MBG-Stream algorithms start with two nodes. For omparison purposes, we used the MLlib implementation of Streaming-KMeans (this latter algorithm was also oded in the Spark Streaming platform) 1. Comparison is also performed with DenStream [4] and ClusTree [5] from the stream R pakage [15]. Streaming-KMeans was evaluated by setting the k parameter to the right number of lasses of eah dataset. DenStream was evaluated by performing a variant of the DBSan algorithm in the offline step. ClusTree was evaluated by performing the k-means algorithm in the offline step by setting the k parameter to 10. All experiments were repeated 10 times and the results (the average value) are reported in Table 2. In this Table, it is notieable that G-Stream s Auraies (A) are higher for all data sets as ompared to Streaming-KMeans, DenStream and CluStree, exept for ClusTree for the HyperPlan data set and for Streaming- KMeans for the KddCup99 data set. Its NMI values are higher than the other algorithms exept for Streaming-KMeans for DS1 and KddCup99 data sets. Its Rand index values are higher than the other algorithms exept for Streaming-KMeans for Sea and DS1 data sets. We reall that MBG-Stream proeeds in one single phase whereas Streaming-KMeans, DenStream and ClusTree proeed in two phases (online and offline phase)

8 MBG-Stream Figure 1: Evolution of graph reation of MBG-Stream on DS1 (data set and topologial result). The intermediate graph after seeing the first window s data points; the 1/3 of all windows; the 2/3 of all windows; and the final graph. 4.3 Visual Validation Figure 1 shows the evolution of the node reation by applying MBG-Stream on the DS1 data set (olored points represent data points of the data stream and red points are nodes of the graph with edges in blue lines; eah olor of the data points orrespond to lass of labels and the size of the nodes of the graph are proportional to their weight). It illustrates that MBG-Stream manages to reognize the strutures of the data stream and an separate these strutures with the best visualization. 5 Conlusion In this paper, we have proposed MBG-Stream, an effiient method for topologial lustering an evolving data stream in an online manner. In MBG-Stream, the nodes are weighted by a fading funtion and the edges by an exponential funtion. MBG-Stream is implemented on a distributed streaming platform based on the miro-bathing proessing model. Experimental evaluation over a number of real and syntheti data sets demonstrates the effetiveness and effiieny of MBG-Stream in disovering lusters of arbitrary shape. The performane of MBGStream, in terms of lustering quality as ompared to three relevant data stream algorithms are promising. We plan in the future to test the speed-up of MBG-Stream on large lusters, to extend it to deal with binary, ategorial, and mixed data streams, and also to make our algorithm as autonomous as possible. Aknowledgments: This researh has been supported by the Frenh Foundation FSN, PIA Grant Big data-investissements d Avenir. The projet is titled Square Predit ( We thank anonymous reviewers for their insightful remarks. Referenes [1] M. Zaharia, T. Das, H. Li, S. Shenker, I. Stoia, Disretized streams: An effiient and fault-tolerant model for stream proessing on large lusters, in: Proeedings of the 4th USENIX Conferene on Hot Topis in Cloud Computing, HotCloud 12, USENIX Assoiation, Berkeley, CA, USA, 2012, pp URL [2] M. Ghesmoune, H. Azzag, M. Lebbah, G-stream: Growing neural gas over data stream, in: Neural Information Proessing - 21st International Conferene, ICONIP 2014, Kuhing, Malaysia, Novem- 165

9 ber 3-6, Proeedings, Part I, 2014, pp doi: / _26. URL [3] C. C. Aggarwal, T. J. Watson, R. Ctr, J. Han, J. Wang, P. S. Yu, A framework for lustering evolving data streams, in: In VLDB, 2003, pp [4] F. Cao, M. Ester, W. Qian, A. Zhou, Density-based lustering over an evolving data stream with noise, in: SDM, 2006, pp [5] P. Kranen, I. Assent, C. Baldauf, T. Seidl, The ClusTree: indexing miro-lusters for anytime stream mining, Knowledge and information systems 29 (2) (2011) [6] N. Ailon, R. Jaiswal, C. Monteleoni, Streaming k-means approximation, in: Advanes in Neural Information Proessing Systems 22: 23rd Annual Conferene on Neural Information Proessing Systems Proeedings of a meeting held 7-10 Deember 2009, Vanouver, British Columbia, Canada., 2009, pp URL [7] V. Braverman, A. Meyerson, R. Ostrovsky, A. Roytman, M. Shindler, B. Tagiku, Streaming k-means on well-lusterable data, in: Proeedings of the Twenty-Seond Annual ACM-SIAM Symposium on Disrete Algorithms, SODA 2011, San Franiso, California, USA, January 23-25, 2011, 2011, pp doi: / URL [8] M. Shindler, A. Wong, A. Meyerson, Fast and aurate k-means for large datasets, in: Advanes in Neural Information Proessing Systems 24: 25th Annual Conferene on Neural Information Proessing Systems Proeedings of a meeting held Deember 2011, Granada, Spain., 2011, pp URL [9] T. Kohonen, M. R. Shroeder, T. S. Huang (Eds.), Self-Organizing Maps, 3rd Edition, Springer- Verlag New York, In., Seauus, NJ, USA, [10] T. Martinetz, K. Shulten, A Neural-Gas Network Learns Topologies, Artifiial Neural Networks I (1991) [11] J. de Andrade Silva, E. R. Faria, R. C. Barros, E. R. Hrushka, A. C. P. L. F. de Carvalho, J. Gama, Data stream lustering: A survey, ACM Comput. Surv. 46 (1) (2013) 13. [12] X. H. Zhu, Stream data mining repository (web site) (2010). URL [13] K. Bahe, M. Lihman, UCI mahine learning repository (2013). URL [14] A. Strehl, J. Ghosh, Cluster ensembles a knowledge reuse framework for ombining multiple partitions, Journal of Mahine Learning Researh 3 (2002) [15] M. Bolanos, J. Forrest, M. Hahsler, stream: Infrastruture for Data Stream Mining, r pakage version (2014). URL 166