A MPAA-Based Iterative Clustering Algorithm Augmented by Nearest Neighbors Search for Time-Series Data Streams

Transcription

1 A MPAA-Based Iteratve Clusterng Algorthm Augmented by Nearest Neghbors Searh for Tme-Seres Data Streams Jessa Ln 1, Mha Vlahos 1, Eamonn Keogh 1, Dmtros Gunopulos 1, Janwe Lu 2, Shouan Yu 2, and Jan Le 2 1 Department of Computer Sene and Engneerng Unversty of Calforna, Rversde Rversde, CA {essa, mvlahos, eamonn, dg}@s.ur.edu 2 College of Computer Sene & Tehnology, Donghua Unversty luw@mal.dhu.edu.n Abstrat. In streamng tme seres the Clusterng problem s more omplex, sne the dynam nature of streamng data makes prevous lusterng methods napproprate. In ths paper, we propose frstly a new method to evaluate Clusterng n streamng tme seres databases. Frst, we ntrodue a novel multresoluton PAA (MPAA) transform to aheve our teratve lusterng algorthm. The method s based on the use of a mult-resoluton peewse aggregate approxmaton representaton, whh s used to extrat features of tme seres. Then, we propose our teratve lusterng approah for streamng tme seres. We take advantage of the multresoluton property of MPPA and equp a stoppng rtera based on Hoeffdng bound n order to aheve fast response tme. Our streamng tme-seres lusterng algorthm also works by leveragng off the nearest neghbors of the nomng streamng tme seres datasets and fulfll nremental lusterng approah. The omprehensve experments based on several publly avalable real data sets shows that sgnfant performane mprovement s aheved and produe hgh-qualty lusters n omparson to the prevous methods. 1 Introduton Numerous lusterng algorthms of tme seres have been proposed, the maorty of them work n relatvely stat model, whle many urrent and emergng applatons requre support for on-lne analyss of rapdly hangng streamng tme seres. In ths paper, we present a new approah for luster streamng tme seres datasets. Our work s motvated by the reent work by Jessa Ln and Eamonn Keogh on teratve nremental lusterng of tme seres [1]. Whle we speed up lusterng proess by examnng the tme seres at nreasngly fner levels of approxmaton usng mult-soluton peewse aggregate approxmaton (MPAA). We argue that MPAA has all the prunng power of Wavelet transform dmensonalty reduton, but s also able to handle arbtrary length queres, s muh faster to ompute and an support a T.B. Ho, D. Cheung, and H. Lu (Eds.): PAKDD 2005, LNAI 3518, pp , Sprnger-Verlag Berln Hedelberg 2005

2 334 J. Ln et al. more general dstane measures. Although there has been a lot of work on more flexble dstane measures usng Wavelet [2, 3], none of these tehnques are ndexable. Whle tme seres databases are often extremely large, any dmensonalty reduton tehnque should support ndex method. For the task of ndexng MPAA has all the advantages of Wavelet wth none of the drawbaks. Our work addresses four maor hallenges n applyng ther deas for lusterng tme seres n a streamng envronment. Spefally, our work has fourfold man ontrbuton: Clusterng Tme Seres n Streamng Envronment: Streamng tme-seres are ommon n many reent applatons, e.g., stok quotes, e-ommere data, system logs, network traff management, et [4]. Compared wth tradtonal datasets, streamng tme-seres pose new hallenges for query proessng due to the streamng nature of data whh onstantly hanges over tme. Clusterng s perhaps the most frequently used data mnng algorthm. Surprsngly, lusterng streamng tme-seres stll have not explored thoroughly, to the best of our knowledge, no prevous work has addressed ths problem. MPAA-based Iteratve Tme Seres Clusterng: PAA (Peewse Aggregate Approxmaton) [5] transformaton produes a peewse onstant approxmaton of the orgnal sequene. In ths paper, we ntrodue a novel mult-resoluton PAA (MPAA) transform to aheve our teratve lusterng algorthm. Proposed stoppng rtera for mult-level teratve lusterng: We solve the dffult problem of dedng exatly how many levels are neessary at eah node n teratve lusterng algorthm by usng a statstal result known as the Hoeffdng bound [6]. Tme Seres Clusterng augmented Nearest Neghbor: Our proposed nlne lusterng algorthm explots haraterst of a neghborhood and sgnfantly redue lusterng onstruton tme and mprove lusterng qualty. The rest of the paper s organzed as follows. In seton 2, we develop enhaned teratve lusterng and streamng lusterng algorthm. Seton 4 presents the expermental evaluaton of our proposed algorthms both n offlne and onlne form. We onlude n Seton 5 wth some summary remarks and future researh dretons. 2 Streamng Iteratve Clusterng Method 2.1 MPAA -Based Dmensonalty Reduton Our MPAA-based tme seres representaton work s derved from the reent work by Eamonn Keogh [5] and Y and Faloutsos [7] on segmentng tme seres representatons of dmensonalty reduton. The bas dea on whh ther work develops s as follows. Suppose, we denote the set of tme seres whh onsttute the database as X = { X1,!, X n }. A tme seres

3 A MPAA-Based Iteratve Clusterng Algorthm 335!. The th ele- X of length n s represented n N spae by a vetor ment of X s alulated by the followng equaton: Our MPPA method dvdes tme seres X n N n n = ( 1) + 1 N X = x 1,, x N =! x (1) X of length n nto a seres of lowerdmensonal sgnal wth dfferent resoluton N. where N {1,!, n}. Smply stated, n frst level, the data s dvded nto N "frames", whose szes need not be ontguous and equal. The mean value of the data fallng wthn a frame s alulated and a vetor of these values beomes the data redued representaton. Then Reursvely applyng the above parwse averagng proess on the lower-resoluton array ontanng the averages, we get a mult-resoluton representaton of tme seres. We gve a smple example to llustrate the MPAA deomposton proedure n Table 1. Suppose we are gven a tme seres ontanng the followng eght values A= [3, 5, 2, 6, 4, 8, 7, 1] and we ntate dvde t nto 4 segments. The MPPA transform of A an be omputed as follows. We frst average the values together parwse to get a new lower-resoluton representaton of the data wth the followng average values [4, 4, 6, 4]. In other words, the average of the frst two values (that s, 3 and 5) s 4 and that of the next two values (that s, 6 and 4) s 5, and so on. Reursvely applyng the above parwse averagng proess on the lower-resoluton array ontanng the averages, we get the followng full deomposton: N Table 1. A smple example to llustrate the MPAA deomposton proedure Resoluton MPAA Values 8 3,5,2,6,4,8,7,1 4 4,4,6,4 2 4, The MPAA approxmaton sheme has some desrable propertes that allow nremental omputaton of the soluton. These propertes are neessary n order for the algorthm to be able to operate effently on large datasets and streamng envronment. 2.2 Enhaned Iteratve Clusterng Methods Our teratve lusterng method s smlar to [1]. The algorthm works by leveragng off the multresoluton property of MPPA. Note that an open problem that arse wth ths sort of teratve models s the defnton of a mnmum number of observatons,.e., devsng an obetve funtons that determne the qualty of lusterng results from the prevous stages to elmnate the need to re-ompute all the dstanes.

4 336 J. Ln et al. Due to us perform the k-means lusterng algorthm, startng at the seond level and gradually progress to fner levels, n order to fnd the stoppng resolutons as low as possble to omplete a good k-means lusterng, t may be suffent to onsder only a small subset of the mult-level lusterng examples that pass through the level of deomposton tree. We solve the dffult problem of dedng exatly how many levels are neessary at eah node by usng a statstal result known as the Hoeffdng bound or addtve Chernoff bound [6], whh have n fat be suessfully used n onlne deson trees [8][9]. After n ndependent observatons of a real-valued random varable r wth range R, the Hoeffdng bound ensures that, wth onfdene 1 δ, the true mean of r s at least r ε, where r s the observed mean of the samples and 2 RIn(1 δ ) ε = (2) 2n Ths s true rrespetve of the probablty dstrbuton that generated the observatons. Table 2. The enhaned teratve lusterng algorthms Algorthm SI-kMeans 1 Dede on a value for k. 2 Perform MPAA deomposton on raw data 3 Intalze the k luster enters (randomly, f neessary). 4 Compute the hoeffdng bound(ε ) 5 Run the k-means algorthm on the level of MPAA representaton of the data 6 Use fnal enters from level as ntal enters for level +1. Ths s aheved by proetng the k enters returned by k- Means algorthm for the 2 1 spae n the 2 + spae. 7 Compute the dstane D enter between ntal enters of level and ntal enters for level +1 8 Compute respetvely maxmum values of the sum of squared ntra-luster errors n th teratve lusterng and (+1) th teratve lusterng,.e. E max( ) and E max( + 1) 9 If E max( + 1) E max( ) > ε, ext. 10 If D enter > ε, goto 3. We all the new teratve lusterng algorthm supportng stoppng rtera SIkMeans, where S stands for stoppng rtera., and I stands for nteratve. Table 2 gves a skeleton of ths dea. 2.3 Proposed Streamng Clusterng Algorthm A key hallengng ssue wth streamng tme seres lusterng algorthm s the hgh rate of nput sequenes nserton.

5 A MPAA-Based Iteratve Clusterng Algorthm 337 To llustrate our applaton, onsder the followng ssue. Most streamng tme seres are related to prevously arrved tme seres or future ones, hene, ths strong temporal dependeny between the streamng tme seres should not be gnored when lusterng streamng data olleton. Ths ssue an be addressed by onsderng the nearest neghbor. A smple dstane metr between two new arrvng tme seres and the lusterng enter wll show how muh they are related to eah other. Hene, the nearest neghbor analyss allows us to automatally dentfy related luster. Below, we gve a more formally defnton n order to dept our Streamng Clusterng algorthm. Defnton 1. Smlarty measure: To measure loseness between two sequenes, we use orrelaton between tme seres as a smlarty measure. Supposed that tme-seres T and T n a sldng wndow whh length s w s represented respetvely by { < u 1, t 1 >,!, < un, tn > } and { < v1, t1 >,!, < vn, tn > } The Smlarty between two tme seres T andt s defned by s mlarty( T, T ) =!!! w u 1 k vk wuv k = w 2 2 w 2 2 u k 1 k wu v k 1 k wv = = Defnton 2. Smlar: If s mlarty( T, T) ς, then a tme seres T s referred to as smlar to a tme seres T. Based on the defnton of smlar n Defnton 1, we an defne the ζ - neghborhood Nζ ( T ) as follows: Defnton 3. ζ -neghborhood Nζ ( T ) defned as a set of sequenes{ X :s mlarty( X, T ) ζ } (3) : ζ -neghborhood for a tme seres T s. Our proposed lusterng algorthm explots haraterst of a neghborhood. It s based on the observaton that a property of a tme-seres would be nfluened by ts neghbors. Examples of suh propertes are the propertes of the neghbors, or the perentage of neghbors that fulfll a ertan onstrant. The above dea an be translated nto lusterng perspetve as follows: a luster label of a tme-seres depends on the luster labels of ts neghbors. The ntuton behnd ths algorthm orgnates from the observaton that the luster of tme seres sequenes an often be approxmately aptured by performng nearest neghbor searh. In what follows, our dea s explaned n detal. Intally, we assume that only tme seres n now wndow s avalable. Thus, we mplement SI-kMeans lusterng on these sequenes tself and form k lusters. Addng new sequenes to exstng luster struture proeeds n three phases: neghborhood searh, dentfaton of an approprate luster for a new sequenes, and relusterng based on loal nformaton. The proposed nremental lusterng algorthm STSI-kMeans (streamng tme seres teratve K-means lusterng algorthm) an be dsrbed as follow:

6 338 J. Ln et al. Step 1. Intalzaton. Get next new sequenes { T w+ 1,!, T} n now wndow. Step 2. Neghborhood searh. Gven a new nomng sequenes{, T } and let K T w+ 1,! C be the set of lusters ontanng any tme seres belongng to N ( T ) ζ, ζ ζ ζ obtan { N ( T w+ 1), N ( T w+ 2),!, N ( T )} by performng a neghborhood searh on { T w+ 1,!, T}, and fnd the anddate luster new sequene T { T w+ 1,!, T}, that mean to dentfy K CK whh an host a C N ( T ) ζ. Step 3. Identfyng an approprate luster. Cluster If there exsts a luster C K that an host a sequenet, and then add T to the luster C K. Otherwse, reate a new luster C new fort. To dentfy a luster C whh an absorb the new tme-serest from the set of anddate lusters C K K, we employ a smple but effetve approah, whh measures the Euldean dstane between the enter of anddate lusters and the new tmeserest, the luster whh returns the mnmum dstane s seleted as a lusterc K whh an absorb the new tme-serest. Step 4. Re-lusterng over affeted luster. If T s assgned to CK or reate a new luster C new fort, then a merge operaton needs to be trggered. Ths s based on a loalty assumpton [10]. Instead of re-lusterng the whole dataset, we only need to fous on the lusters that are affeted by the new tme-seres. That s, a new tmeseres s plaed n the luster, and a sequene of luster re-struturng proesses s performed only n regons that have been affeted by the new tme-seres,.e., lusters that ontan any tme-seres belongng to the neghborhood of a new tme-seres need to be onsdered. Note that based on SI-kMeans re-lusterng, the number of lusters, k value Dede by the number of affeted lusters k by absorbng the new tme-seres. Where k = k. Step 5. Repetton. Repeat Step 2-4 whenever new sequenes avalable n the next wndow. 3 Expermental Evaluaton In ths seton, we mplemented our algorthms SI-kMeans and STSI-kMeans, and onduted a seres of experments to evaluate ther effeny. We also mplemented the I-kMeans algorthm, to ompare aganst our tehnques. When not expltly mentoned, the results reported are averages over 100 tests.

7 A MPAA-Based Iteratve Clusterng Algorthm Datasets The data usng n our experment s smlar to [1]. We tested on two publly avalable, real datasets: JPL datasets and heterogeneous datasets [11]. The dataset ardnaltes range from 1,000 to 8,000. The length of eah tme seres has been set to 512 on one dataset, and 1024 on the other. Eah tme seres s z-normalzed to have mean value of 0 and standard devaton of Offlne Clusterng Comparson To show that our SI-kMeans approah s superor to the I-kMeans algorthm for lusterng tme seres n offlne form, n the frst set of experments, we performed a seres of experments on publly avalable real datasets. After eah exeuton, we ompute the error and the exeuton tme on the lusterng results. Fg. 1. Comparson of the lusterng approxmaton error between SI-kMeans and I-kMeans. (a) Error of SI-kMeans algorthm on the Heterogeneous dataset, presented as fraton of the error from the I-kMeans algorthm. (b) Obetve funtons of SI-kMeans algorthm on the JPL dataset, presented as fraton of error from the I-kMeans algorthm Fgure 1 llustrates the results of lusterng approxmaton error. As t an be seen, our algorthm aheves better lusterng auray. Fgure 2 shows Speedup of SI-kMeans aganst I-kMeans. the SI-kMeans algorthm fnds the best result n relatvely early stage and does not need to run through all levels. 3.3 Onlne Clusterng Comparson In the next set of experments, we ompare the nlne performane of STSI-kMeans to I-kMeans, whh s essentally a omparson between an onlne and the orrespondng offlne algorthm. Sne orgnal I-kMeans algorthm s not sutable for onlne lusterng streamng tme seres, we revse t and adapt t to onlne lusterng.

8 340 J. Ln et al. Fg. 2. Speedup of SI-kMeans aganst I-kMeans. (a) SI-kMeans vs. I-kMeans algorthms n terms of lusterng error and runnng tme for n the Heterogeneous dataset. (b) SI-kMeans vs. I-kMeans algorthms n terms of obetve funton and runnng tme for JPL dataset We quantfy frstly the dfferenes n the performane of the two algorthms. We report the umulatve relatve error over ount-based or sequene-based wndows, whh measure the relatve nrease n the umulatve error when usng STSI-kMeans and I-kMeans.!! q q 1 STSI-kMeans = = 1 q! Error = 1 I-kMeans Error ( w ) ErrorI-kMeans ( w) CRE = 100 ( w ) Where, q s the number of elapsed wndows. In Fgure 3, we dept CRE as a funton of q and k. In the experment of Fgure 5, the length of streamng tme seres 1000,2000,4000,8000 ponts, through, for nreasng q we observe a very slow buld-up of the relatve error. Our algorthm performs better as the number of q nreases. (4) Fg. 3. Comparson of the lusterng approxmaton error between STSI-kMeans and I-kMeans

9 A MPAA-Based Iteratve Clusterng Algorthm 341 The seond measure of nterest s the speedup, whh measures how many tmes faster STSI-kMeans s when ompared to I-kMeans. speedup q! = 1 = q! = 1 tme tme STSI-kMeans I-kMeans ( w ) ( w ) (5) Fgure 4 shows the speedup that our algorthm aheves, whh translates to one or two orders of magntude faster exeuton than the offlne I-kMeans algorthm (for the experments we ran). The STSI-kMeans algorthm s tmes faster than I- kmeans. We observe that the speedup nreases sgnfantly for dereasng k. Ths s beause the amount of work that STSI-kMeans does remans almost onstant, whle I- kmeans requres lots of extra effort for smaller values of k. As expeted, the speedup gets larger when we nrease q. Fg. 4. Speedup of STSI-kMeans aganst I-kMeans 4 Conlusons In ths paper, we have presented frstly an approah to perform nremental lusterng of tme-seres at varous resolutons usng the mult-resoluton peewse aggregate transform. The algorthm equppng a stoppng rtera based on Hoeffdng bound stablzes at very early stages, elmnatng the needs to operate on hgh resolutons. Ths approah resolves the dlemma assoated wth the hoes of ntal enters for k-means and at whh stage termnate the program for I-kMeans. Ths allows our algorthm to termnate the program at early stage wth qualty guarantee, thus elmnate the need to re-ompute all the dstanes and sgnfantly mproves the exeuton tme and lusterng qualty. We also expend our method to streamng tme seres envronment. Our streamng tme-seres lusterng algorthm works by leveragng off the nearest neghbors of the nomng streamng tme seres datasets and fulfll nremental lusterng approah. Our expermental results based on several publly avalable real data sets shows that sgnfant performane mprovement s aheved and produe hgh-qualty lusters n omparson to the prevous methods.

10 342 J. Ln et al. Referenes 1. Ln, J., Vlahos, M., Keogh, E., & Gunopulos, D.: Iteratve Inremental Clusterng of Tme Seres. In proeedngs of the IX Conferene on Extendng Database Tehnology (EDBT 2004). Crete, Greee. (2004) Huhtala, Y., Kärkkänen, J., & Tovonen. H.: Mnng for Smlartes n Algned Tme Seres Usng Wavelets. In Data Mnng and Knowledge Dsovery: Theory, Tools, and Tehnology. SPIE Proeedngs Seres Vol Orlando, Florda. (1999) Struzk, Z., Sebes, A.: The Haar Wavelet Transform n The Tme Seres Smlarty Paradgm. In Pro 3rd European Conferene on Prnples and Prate of Knowledge Dsovery n Databases. (1999) D. Carney, U. Cetnternel, M. Chernak, C. Convey, S. Lee, G. Sedman, M. Stonebraker, N. Tatbul, S. Zdonk.: Montorng streams: A New Class of Data Management Applatons. In Pro. 28th Int. Conf. on Very Large Data Bases, (2002) Keogh, E., Chakrabart, K. Pazzan, M, Mehrotra, S.: Dmensonalty Reduton for Fast Smlarty Searh n Large Tme Seres Databases. Journal of Knowledge and Informaton Systems. Vol. 3, No. 3. (2001) Hoeffdng, W.: Probablty nequaltes for sums of bounded random varables. Journal of the Ameran Statstal Assoaton (1963) Y, B., Faloutsos, C.: Fast Tme Sequene Indexng for Arbtrary Lp Norms. In proeedngs of the 26th Int'l Conferene on Very Large Databases. Caro, Egypt, Sept pp l Database Management. Berln, Germany, Jul (2000) Domngos, P., Hulten, G.: Mnng Hgh-Speed Data Streams. In: Proeedngs of the Sxth Internatonal Conferene on Knowledge Dsovery and Data Mnng, Boston, MA, ACM Press (2000) Gama, J., Medas, P., Rodrgues, P.: Conept Drft n Deson-Tree Learnng for Data Streams. In: Proeedngs of the Fourth European Symposum on Intellgent Tehnologes and ther mplementaton on Smart Adaptve Systems, Aahen, Germany, Verlag Manz (2004) L. Ralavola, F. dalhe-bu.: Inremental Support Vetor Mahne Learnng: A Loal Approah. In Proeedngs of the Annual Conferene of the European Neural Network Soety. (2001) Bay, S. D.: The UCI KDD Arhve [ Irvne, CA: Unversty of Calforna, Department of Informaton and Computer Sene. (1999)