A Symbolic Representation of Time Series, with Implications for Streaming Algorithms

Transcription

1 A Symbol Representaton of Tme Seres, th Implatons for Streamng Algorthms Jessa Ln Eamonn Keogh Stefano Lonard Bll Chu Unversty of Calforna - Rversde Computer Sene & Engneerng Department Rversde, CA 9252, USA {jessa, eamonn, stelo, bll}@s.ur.edu ABSTRACT The parallel explosons of nterest n streamng data, and data mnng of tme seres have had surprsngly lttle nterseton. Ths s n spte of the fat that tme seres data are typally streamng data. The man reason for ths apparent paradox s the fat that the vast majorty of ork on streamng data expltly assumes that the data s dsrete, hereas the vast majorty of tme seres data s real valued. Many researhers have also onsdered transformng real valued tme seres nto symbol representatons, notng that suh representatons ould potentally allo researhers to aval of the ealth of data strutures and algorthms from the text proessng and bonformats ommuntes, n addton to allong formerly bath-only problems to be takled by the streamng ommunty. Whle many symbol representatons of tme seres have been ntrodued over the past deades, they all suffer from three fatal flas. Frstly, the dmensonalty of the symbol representaton s the same as the orgnal data, and vrtually all data mnng algorthms sale poorly th dmensonalty. Seondly, although dstane measures an be defned on the symbol approahes, these dstane measures have lttle orrelaton th dstane measures defned on the orgnal tme seres. Fnally, most of these symbol approahes requre one to have aess to all the data, before reatng the symbol representaton. Ths last feature expltly tharts efforts to use the representatons th streamng algorthms. In ths ork e ntrodue a ne symbol representaton of tme seres. Our representaton s unque n that t allos dmensonalty/numerosty reduton, and t also allos dstane measures to be defned on the symbol approah that loer bound orrespondng dstane measures defned on the orgnal seres. As e shall demonstrate, ths latter feature s partularly extng beause t allos one to run ertan data mnng algorthms on the effently manpulated symbol representaton, hle produng dental results to the algorthms that operate on the orgnal data. Fnally, our representaton allos the real valued data to be onverted n a streamng fashon, th only an nfntesmal tme and spae overhead. We ll demonstrate the utlty of our representaton on the lass data mnng tasks of lusterng, lassfaton, query by ontent and anomaly deteton. Keyords Tme Seres, Data Mnng, Data Streams, Symbol, Dsretze. INTRODUCTION The parallel explosons of nterest n streamng data [4, 8,, 8], and data mnng of tme seres [6, 7, 9, 2, 2, 24, 26, 34] have had surprsngly lttle nterseton. Ths s n spte of the fat that tme seres data are typally streamng data, for example, stok value, medal and meteorologal data [3]. The man reason for ths apparent paradox s the fat that the vast majorty of ork on streamng data expltly assumes that the data s dsrete, hereas the vast majorty of tme seres data s real valued [23]. Many hgh level representatons of tme seres have been proposed for data mnng. Fgure llustrates a herarhy of all the varous tme seres representatons n the lterature [2, 7, 4, 6, 2, 22, 25, 3, 3, 35]. One representaton that the data mnng ommunty has not onsdered n detal s the dsretzaton of the orgnal data nto symbol strngs. At frst glane ths seems a surprsng oversght. In addton to allong the framng of tme seres problems as streamng problems, there s an enormous ealth of exstng algorthms and data strutures that allo the effent manpulatons of symbol representatons. Suh algorthms have reeved deades of attenton n the text retreval ommunty, and more reent attenton from the bonformats ommunty [3, 3, 7, 29, 32, 33]. Some smple examples of tools that are not defned for real-valued sequenes but are defned for symbol approahes nlude hashng, Markov models, suffx trees, deson trees et. As a more onrete example, onsder the Jaard oeffent [3], a dstane measure beloved by streamng researhers. The Jaard oeffent s only ell defned for dsrete data (suh as eb lks or ndvdual keystrokes) as thus annot be used th real-valued tme seres. There s a smple explanaton for the data mnng ommunty s lak of nterest n symbol manpulaton as a supportng tehnque for mnng tme seres. If the data are transformed nto vrtually any of the other representatons depted n Fgure, then t s possble to measure the smlarty of to tme seres n that representaton spae, suh that the dstane s guaranteed to loer bound the true dstane beteen the tme seres n the orgnal spae. Ths smple fat s at the ore of almost all algorthms n tme seres data mnng and ndexng [4]. Hoever, n spte of the fat that there are dozens of tehnques for produng dfferent varants of the symbol representaton [2,, 2], there s no knon method for alulatng the dstane n the symbol spae, hle provdng the loer boundng guarantee. In addton to allong the reaton of loer boundng dstane measures, there s one other hghly desrable property of any tme seres representaton, nludng a symbol one. Almost all tme seres datasets are very hgh dmensonal. Ths s a hallengng fat beause all non-trval data mnng and ndexng algorthms degrade exponentally th dmensonalty. For example, above 6-2 dmensons, ndex strutures degrade to sequental sannng [9].

2 Data Adaptve Tme Seres Representatons Non Data Adaptve Sorted Coeffents Peese Lnear Approxmaton Interpolaton Regresson Peese Polynomal Sngular Value Deomposton Adaptve Peese Constant Approxma ton Symbol Natural Language Loer Boundng Strngs Trees Non - Loer Boundng Haar Wavelets Random Mappngs Orthonormal B - Orthonormal Daubehes dbn n > Coflets Symlets Spetral Dsrete Fourer Transform Peese Aggregate Approxmaton Dsrete Cosne Transform Fgure : A herarhy of all the varous tme seres representatons n the lterature. The leaf nodes refer to the atual representaton, and the nternal nodes refer to the lassfaton of the approah. The ontrbuton of ths paper s to ntrodue a ne representaton, the loer boundng symbol approah None of the symbol representatons that e are aare of allo dmensonalty reduton [2,, 2]. There s some reduton n the storage spae requred, sne feer bts are requred for eah value; hoever, the ntrns dmensonalty of the symbol representaton s the same as the orgnal data. In [4], Babok et. al ask f there s a need for database researhers to develop fundamental and general-purpose models for data streams. The opnon of the authors s affrmatve. In ths ork e take a step toards ths goal by ntrodung a representaton of tme seres that s sutable for streamng algorthms. It s dmensonalty redung, loer boundng and an be obtaned n a streamng fashon. As e shall demonstrate, the loer boundng feature s partularly extng beause t allos one to run ertan data mnng algorthms on the effently manpulated symbol representaton, hle produng dental results to the algorthms that operate on the orgnal data. In partular, e ll demonstrate the utlty of our representaton on the lass data mnng tasks of lusterng [2], lassfaton [6], ndexng [, 4, 22, 35], and anomaly deteton [9, 24, 3]. The rest of ths paper s organzed as follos. Seton 2 brefly dsusses bakground materal on tme seres data mnng and related ork. Seton 3 ntrodues our novel symbol approah, and dsusses ts dmensonalty reduton, numerosty reduton and loer boundng abltes. Seton 4 ontans an expermental evaluaton of the symbol approah on a varety of data mnng tasks. Fnally, Seton 5 offers some onlusons and suggestons for future ork. 2. BACKGROUND AND RELATED WORK Tme seres data mnng has attrated enormous attenton n the last deade. The reve belo s neessarly bref; e refer nterested readers to [3, 23] for a more n depth reve. 2. Tme Seres Data Mnng Tasks Whle makng no pretene to be exhaustve, the follong lst summarzes the areas that have seen the majorty of researh nterest n tme seres data mnng. Indexng: Gven a query tme seres Q, and some smlarty/dssmlarty measure D(Q,C), fnd the most smlar tme seres n database DB [, 7, 4,22, 35]. Clusterng: Fnd natural groupngs of the tme seres n database DB under some smlarty/dssmlarty measure D(Q,C) [2,25]. Classfaton: Gven an unlabeled tme seres Q, assgn t to one of to or more predefned lasses [6]. Summarzaton: Gven a tme seres Q ontanng n dataponts here n s an extremely large number, reate a (possbly graph) approxmaton of Q hh retans ts essental features but fts on a sngle page, omputer sreen, exeutve summary, et [26]. Anomaly Deteton: Gven a tme seres Q, and some model of normal behavor, fnd all setons of Q hh ontan anomales, or surprsng/nterestng/unexpeted/novel behavor [9, 24, 3]. Sne the datasets enountered by data mners typally don t ft n man memory, and dsk I/O tends to be the bottlenek for any data mnng task, a smple gener frameork for tme seres data mnng has emerged [4]. The bas approah s outlned n Table.. Create an approxmaton of the data, hh ll ft n man memory, yet retans the essental features of nterest. 2. Approxmately solve the task at hand n man memory. Make (hopefully very fe) aesses to the orgnal data on dsk to onfrm the soluton obtaned n Step 2, or to 3. modfy the soluton so t agrees th the soluton e ould have obtaned on the orgnal data. Table : A gener tme seres data mnng approah It should be lear that the utlty of ths frameork depends heavly on the qualty of the approxmaton reated n Step. If the approxmaton s very fathful to the orgnal data, then the soluton obtaned n man memory s lkely to be the same as, or very lose to, the soluton e ould have obtaned on the orgnal data. The handful of dsk aesses made n Step 3 to onfrm or slghtly modfy the soluton ll be nonsequental ompared to the number of dsk aesses requred had e orked on the orgnal data. Wth ths n mnd, there has been great nterest n approxmate representatons of tme seres, hh e onsder belo. 2.2 Tme Seres Representatons As th most problems n omputer sene, the sutable hoe of representaton greatly affets the ease and effeny of tme seres data mnng. Wth ths n mnd, a great number of tme seres representatons have been ntrodued, nludng the Dsrete Fourer Transform (DFT) [4], the Dsrete Wavelet Transform (DWT) [7], Peese Lnear, and Peese Constant models (PAA) [22], (APCA) [6, 22], and Sngular Value Deomposton (SVD) [22]. Fgure 2 llustrates the most ommonly used representatons.

3 Fgure 2: The most ommon representatons for tme seres data mnng. Eah an be vsualzed as an attempt to approxmate the sgnal th a lnear ombnaton of bass funtons Reent ork suggests that there s lttle to hoose beteen the above n terms of ndexng poer [23]; hoever, the representatons have other features that may at as strengths or eaknesses. As a smple example, avelets have the useful multresoluton property, but are only defned for tme seres that are an nteger poer of to n length [7]. One mportant feature of all the above representatons s that they are real valued. Ths lmts the algorthms, data strutures and defntons avalable for them. For example, n anomaly deteton e annot meanngfully defne the probablty of observng any partular set of avelet oeffents, sne the probablty of observng any real number s zero [27]. Suh lmtatons have lead researhers to onsder usng a symbol representaton of tme seres. Whle there are lterally hundreds of papers on dsretzng (symbolzng, tokenzng, quantzng) tme seres [2, 2] (see [] for an extensve survey), none of the tehnques allos a dstane measure that loer bounds a dstane measure defned on the orgnal tme seres. For ths reason, the gener tme seres data mnng approah llustrated n Table s of lttle utlty, sne the approxmate soluton to problem reated n man memory may be arbtrarly dssmlar to the true soluton that ould have been obtaned on the orgnal data. If, hoever, one had a symbol approah that alloed loer boundng of the true dstane, one ould take advantage of the gener tme seres data mnng model, and of a host of other algorthms, defntons and data strutures hh are only defned for dsrete data, nludng hashng, Markov models, and suffx trees. Ths s exatly the ontrbuton of ths paper. We all our symbol representaton of tme seres SAX (Symbol Aggregate approxmaton), and defne t n the next seton. 3. SAX: OUR SYMBOLIC APPROACH SAX allos a tme seres of arbtrary length n to be redued to a strng of arbtrary length, ( < n, typally << n). The alphabet sze s also an arbtrary nteger a, here a > 2. Table 2 summarzes the major notaton used n ths and subsequent setons. C A tme seres C =,, n A Peese Aggregate Approxmaton of a tme seres C C =,..., Ĉ A symbol representaton of a tme seres Cˆ = ˆ,..., ˆ Dsrete Fourer Transform Peese Lnear Approxmaton Haar Wavelet Adaptve Peese Constant Approxmaton The number of PAA segments representng tme seres C a Alphabet sze (e.g., for the alphabet = {a,b,}, a = 3) Table 2: A summarzaton of the notaton used n ths paper Our dsretzaton proedure s unque n that t uses an ntermedate representaton beteen the ra tme seres and the symbol strngs. We frst transform the data nto the Peese Aggregate Approxmaton (PAA) representaton and then symbolze the PAA representaton nto a dsrete strng. There are to mportant advantages to dong ths: Dmensonalty Reduton: We an use the ell-defned and ell-doumented dmensonalty reduton poer of PAA [22, 35], and the reduton s automatally arred over to the symbol representaton. Loer Boundng: Provng that a dstane measure beteen to symbol strngs loer bounds the true dstane beteen the orgnal tme seres s non-trval. The key observaton that allos us to prove loer bounds s to onentrate on provng that the symbol dstane measure loer bounds the PAA dstane measure. Then e an prove the desred result by transtvty by smply pontng to the exstng proofs for the PAA representaton tself [35]. We ll brefly reve the PAA tehnque before onsderng the symbol extenson. 3. Dmensonalty Reduton Va PAA A tme seres C of length n an be represented n a -dmensonal spae by a vetor C =,,. The th element of C s K alulated by the follong equaton: = n n j j= n ( ) + Smply stated, to redue the tme seres from n dmensons to dmensons, the data s dvded nto equal szed frames. The mean value of the data fallng thn a frame s alulated and a vetor of these values beomes the data-redued representaton. The representaton an be vsualzed as an attempt to approxmate the orgnal tme seres th a lnear ombnaton of box bass funtons as shon n Fgure C Fgure 3: The PAA representaton an be vsualzed as an attempt to model a tme seres th a lnear ombnaton of box bass funtons. In ths ase, a sequene of length 28 s redued to 8 dmensons C The PAA dmensonalty reduton s ntutve and smple, yet has been shon to rval more sophstated dmensonalty reduton tehnques lke Fourer transforms and avelets [22, 23, 35]. We normalze eah tme seres to have a mean of zero and a standard devaton of one before onvertng t to the PAA ()

4 representaton, sne t s ell understood that t s meanngless to ompare tme seres th dfferent offsets and ampltudes [23]. 3.2 Dsretzaton Havng transformed a tme seres database nto PAA, e an apply a further transformaton to obtan a dsrete representaton. It s desrable to have a dsretzaton tehnque that ll produe symbols th equprobablty [3, 28]. Ths s easly aheved sne normalzed tme seres have a Gaussan dstrbuton [27]. To llustrate ths, e extrated subsequenes of length 28 from 8 dfferent tme seres and plotted a normal probablty plot of the data as shon n Fgure 4. - Fgure 4: A normal probablty plot of the umulatve dstrbuton of values from subsequenes of length 28 from 8 dfferent datasets. The hghly lnear nature of the plot strongly suggests that the data ame from a Gaussan dstrbuton Gven that the normalzed tme seres have hghly Gaussan dstrbuton, e an smply determne the breakponts that ll produe a equal-szed areas under Gaussan urve [27]. Defnton. Breakponts: breakponts are a sorted lst of numbers Β = β,,β a- suh that the area under a N(,) Gaussan urve from β to β + = /a (β and β a are defned as - and, respetvely). These breakponts may be determned by lookng them up n a statstal table. For example, Table 3 gves the breakponts for values of a from 3 to. β a Probablty β β β β β β β β β 9.28 Table 3: A lookup table that ontans the breakponts that dvde a Gaussan dstrbuton n an arbtrary number (from 3 to ) of equprobable regons One the breakponts have been obtaned e an dsretze a tme seres n the follong manner. We frst obtan a PAA of the tme seres. All PAA oeffents that are belo the smallest breakpont are mapped to the symbol a, all oeffents greater than or equal to the smallest breakpont and less than the seond smallest breakpont are mapped to the symbol b, et. Fgure 5 llustrates the dea b Fgure 5: A tme seres s dsretzed by frst obtanng a PAA approxmaton and then usng predetermned breakponts to map the PAA oeffents nto SAX symbols. In the example above, th n = 28, = 8 and a = 3, the tme seres s mapped to the ord baabb Note that n ths example the 3 symbols, a, b, and are approxmately equprobable as e desred. We all the onatenaton of symbols that represent a subsequene a ord. Defnton 2. Word: A subsequene C of length n an be represented as a ord Cˆ = ˆ,, ˆ K as follos. Let alpha denote the th element of the alphabet,.e., alpha = a and alpha 2 = b. Then the mappng from a PAA approxmaton C to a ord Ĉ s obtaned as follos: a a ˆ β < β (2) = alpha j, f j We have no defned SAX, our symbol representaton (the PAA representaton s merely an ntermedate step requred to obtan the symbol representaton). 3.3 Dstane Measures Havng ntrodued the ne representaton of tme seres, e an no defne a dstane measure on t. By far the most ommon dstane measure for tme seres s the Euldean dstane [23, 29]. Gven to tme seres Q and C of the same length n, Eq. 3 defnes ther Euldean dstane, and Fgure 6.A llustrates a vsual ntuton of the measure. n ( C) ( ) D Q, q (3) = If e transform the orgnal subsequenes nto PAA representatons, Q and C, usng Eq., e an then obtan a loer boundng approxmaton of the Euldean dstane beteen the orgnal subsequenes by: = ( q ) 2 n DR( Q, C ) (4) Ths measure s llustrated n Fgure 6.B. A proof that DR(Q, C ) loer bounds the true Euldean dstane appears n [22] (an alteratve proof appears n [35] ). If e further transform the data nto the symbol representaton, e an defne a MINDIST funton that returns the mnmum dstane beteen the orgnal tme seres of to ords: MINDIST Qˆ, Cˆ) n = 2 ( dst( qˆ, ˆ )) ( (5) The funton resembles Eq. 4 exept for the fat that the dstane beteen the to PAA oeffents has been replaed th the subfunton dst(). The dst() funton an be mplemented usng a table lookup as llustrated n Table 4. b b 2 j

5 a b d a b d Table 4: A lookup table used by the MINDIST funton. Ths table s for an alphabet of ardnalty of 4,.e. a=4. The dstane beteen to symbols an be read off by examnng the orrespondng ro and olumn. For example, dst(a,b) = and dst(a,) =.67. The value n ell (r,) for any lookup table an be alulated by the follong expresson., f r ellr, = (6) β max( r, ) β mn( r, ), otherse For a gven value of the alphabet sze a, the table needs only be alulated one, then stored for fast lookup. The MINDIST funton an be vsualzed n Fgure 6.C C C Cˆ Qˆ Fgure 6: A vsual ntuton of the three representatons dsussed n ths ork, and the dstane measures defned on them. A) The Euldean dstane beteen to tme seres an be vsualzed as the square root of the sum of the squared dfferenes of eah par of orrespondng ponts. B) The dstane measure defned for the PAA approxmaton an be seen as the square root of the sum of the squared dfferenes beteen eah par of orrespondng PAA oeffents, multpled by the square root of the ompresson rate. C) The dstane beteen to SAX representatons of a tme seres requres lookng up the dstanes beteen eah par of symbols, squarng them, summng them, takng the square root and fnally multplyng by the square root of the ompresson rate There s one ssue e must address f e are to use a symbol representaton of tme seres. If e sh to approxmate a massve dataset n man memory, the parameters and a have to be hosen n suh a ay that the approxmaton makes the best use of the prmary memory avalable. There s a lear tradeoff beteen the parameter ontrollng the number of approxmatng Q Q = b a a b b = b a b a a (A ) (B ) (C ) elements, and the value a ontrollng the granularty of eah approxmatng element. It s nfeasble to determne the best tradeoff analytally, sne t s hghly data dependent. We an, hoever, emprally determne the best values th a smple experment. Sne e sh to aheve the tghtest possble loer bounds, e an smply estmate the loer bounds over all possble feasble parameters, and hoose the best settngs. MINDIST( Qˆ, Cˆ) Tghtness of Loer Bound = (7) D( Q, C) We performed suh a test th a onatenaton of 5 tme seres databases taken from the UCR tme seres data mnng arhve. For every ombnaton of parameters e averaged the result of, experments on subsequenes of length 256. Fgure 7 shos the results. Tghtness of loer bound Word Sze Fgure 7: The emprally estmated tghtness of loer bounds over the ross produt of a = [3 ] and = [2 8]. The darker hstogram bars llustrate ombnatons of parameters that requre approxmately equal spae to store every possble ord (approxmately 4 megabytes) The results suggest that usng a lo value for a results n eak bounds, but that there are dmnshng returns for large values of a. The results also suggest that the parameters are not too rtal; an alphabet sze n the range of 5 to 8 seems to be a good hoe. 3.4 Numerosty Reduton We have seen that, gven a sngle tme seres, our approah an sgnfantly redue ts dmensonalty. In addton, our approah an redue the numerosty of the data for some applatons. Most applatons assume that e have one very long tme seres T, and that manageable subsequenes of length n are extrated by use of a sldng ndo, then stored n a matrx for further manpulaton [7, 4, 22, 35]. Fgure 8 llustrates the dea. Fgure 8: An llustraton of the notaton ntrodued n ths seton: A tme seres T of length 28, the subsequene C 67 of length n = 6, and the frst 8 subsequenes extrated by a sldng ndo. Note that the sldng ndos are overlappng Alphabet sze a C p C 6 7 p = T

6 When performng sldng ndos subsequene extraton, th any of the real-valued representatons, e must store all T - n + extrated subsequenes (n dmensonalty redued form). Hoever, magne for a moment that e are usng our proposed approah. If the frst ord extrated s aabb, and the ndo s shfted to dsover that the seond ord s also aabb, e an reasonably dede not to nlude the seond ourrene of the ord n sldng ndos matrx. If e ever need to retreve all ourrenes of aabb, e an go to the loaton ponted to by the frst ourrenes, and remember to slde to the rght, testng to see f the next ndo s also mapped to the same ord. We an stop testng as soon as the ord hanges. Ths smple dea s very smlar to the run-length-enodng data ompresson algorthm. The utlty of ths optmzaton depends on the parameters used and the data tself, but t typally yelds a numerosty reduton fator of to or three. Hoever, many datasets are haraterzed by long perods of lttle or no movement, folloed by bursts of atvty (sesmologal data s an obvous example). On these datasets the numerosty reduton fator an be huge. Consder the example shon n Fgure 9. Spae Shuttle STS-57 Telemetry aabb aabb Fgure 9: Sldng ndo extraton on Spae Shuttle Telemetry data, th n = 32. At tme pont 6, the extrated ord s aabb, and the next 4 subsequenes also map to ths ord. Only a ponter to the frst ourrene must be reorded, thus produng a large reduton n numerosty There s only one speal ase e must onsder. As e noted n Seton 3., e normalze eah tme seres (nludng subsequenes) to have a mean of zero and a standard devaton of one. Hoever, f the subsequene ontans only one value, the standard devaton s not defned. More troublesome s the ase here the subsequene s almost onstant, perhaps 3 zeros and a sngle.. If e normalze ths subsequene, the sngle dfferng element ll have ts value exploded to Ths stuaton ours qute frequently. For example, the last 2 tme unts of the data n Fgure 9 appear to be onstant, but atually ontan tny amounts of nose. If e ere to normalze subsequenes extrated from ths area, the normalzaton ould magnfy the nose to large meanngless patterns. We an easly deal th ths problem, f the standard devaton of the sequene before normalzaton s belo an epslon ε, e smply assgn the entre ord to the mddle-ranged alphabet (e.g. f a = 5). We end ths seton th a vsual omparson beteen SAX and the four most used representatons n the lterature (Fgure ) Fgure : A vsual omparson of SAX and the four most ommon tme seres data mnng representatons. A ra tme seres of length 28 s transformed nto the ord ffffffeeeddbaabeedbaaaaaddee. Ths s a far omparson sne the number of bts n eah representaton s the same 4. EXPERIMENTAL VALIDATION OF SAX We performed varous data mnng tasks usng SAX. For lusterng and lassfaton, e ompared the results th the lass Euldean dstane, and th other prevously proposed symbol approahes. Note that none of these other approahes use dmensonalty reduton. In the next paragraph e summarze the stramen representatons that e ompare SAX to. We hoose these to approahes sne they are typal representatves of symbol approahes n the lterature. André-Jönsson, and Badal [2] proposed the SDA algorthm that omputes the hanges beteen values from one nstane to the next, and dvde the range nto user-predefned regons. The dsadvantages of ths approah are obvous: pror knoledge of the data dstrbuton of the tme seres s requred n order to set the breakponts; and the dsretzed tme seres does not onserve the general shape or dstrbuton of the data values. Huang and Yu proposed the IMPACTS algorthm, hh uses hange rato beteen one tme pont to the next tme pont to dsretze the tme seres [2]. The range of hange ratos are then dvded nto equal-szed setons and mapped nto symbols. The tme seres s onverted to a dsretzed olleton of hange ratos. As th SAX, the user needs to defne the ardnalty of symbols. 4. Clusterng Clusterng s one of the most ommon data mnng tasks, beng useful n ts on rght as an exploratory tool, and also as a subroutne n more omplex algorthms [2,5, 2]. 4.. Herarhal Clusterng Comparng herarhal lusterngs s a very good ay to ompare and ontrast smlarty measures, sne a dendrogram of sze N summarzes O(N 2 ) dstane alulatons [23]. The evaluaton s typally subjetve; e smply adjudge hh dstane measure appears to reate the most natural groupngs of the data. Hoever, f e kno the data labels n advane e an also make objetve statements of the qualty of the lusterng. In Fgure e lustered nne tme seres from the Control Chart dataset, three eah from the dereasng trend, upard shft and normal lasses. f e d b a DFT PLA Haar APCA

7 Euldean SAX 265 IMPACTS (alphabet=8) SDA Objetve Funton Ra Ra data data Our Symbol SAX Approah Number of Iteratons Fgure : A omparson of the four dstane measures ablty to luster members of the Control Chart dataset. Complete lnkage as used as the agglomeraton tehnque In ths ase e an objetvely state that SAX s superor, sne t orretly assgns eah lass to ts on subtree. Ths s smply a sde effet due to the smoothng effet of dmensonalty reduton. More generally, e observed SAX losely mms Euldean dstane on varous datasets Parttonal Clusterng Although herarhal lusterng s a good santy hek for any proposed dstane measure, t has lmted utlty for data mnng beause of ts poor salablty. The most ommonly used data mnng lusterng algorthm s k-means [5], so for ompleteness e ll onsder t here. We performed k-means on both the orgnal ra data, and our symbol representaton. Fgure 2 shos a typal run of k-means on a spae telemetry dataset. Both algorthms onverge after teratons on average. The results here are qute unntutve and surprsng; orkng th an approxmaton of the data gves better results than orkng th the orgnal data. Fortunately, a reent paper offers a suggeston as to hy ths mght be so. It has been shon that ntalzng the lusters enters on a lo dmenson approxmaton of the data an mprove the qualty [2], ths s hat lusterng th SAX mpltly does. Fgure 2: A omparson of the k-means lusterng algorthm usng SAX and the ra data. The dataset as Spae Shuttle telemetry,, subsequenes of length 52. Surprsngly, orkng th the symbol approxmaton produes better results than orkng th the orgnal data 4.2 Classfaton Classfaton of tme seres has attrated muh nterest from the data mnng ommunty. The hgh dmensonalty, hgh feature orrelaton, and typally hgh levels of nose found n tme seres provde an nterestng researh problem [23]. Although spealpurpose algorthms have been proposed [25], e ll onsder only the to most ommon lassfaton algorthms for brevty, larty of presentatons and to faltate ndependent onfrmaton of our fndngs Nearest Neghbor Classfaton To ompare dfferent dstane measures on -nearest-neghbor lassfaton, e used leavng-one-out ross valdaton. We ompare SAX th Euldean dstane, IMPACTS, SDA, and LP. To lass synthet datasets are used: the Cylnder-Bell- Funnel (CBF) dataset has 5 nstanes of tme seres for eah of the three lusters, and the Control Chart (CC) has nstanes for eah of the sx lusters [23]. Sne SAX allos dmensonalty and alphabet sze as user nput, and the IMPACTS allos varable alphabet sze, e ran the experments on dfferent ombnatons of dmensonalty reduton and alphabet sze. For the other approahes e appled the smple dmensonalty reduton tehnque of skppng data ponts at a fxed nterval. In Fgure 3, e sho the results th a dmensonalty reduton of 4 to. Smlar results ere observed for other levels of dmensonalty reduton. One agan, SAX s ablty to beat Euldean dstane s probably due to the smoothng effet of dmensonalty reduton; nevertheless, ths experment does sho the superorty of SAX over the other approahes proposed n the lterature Deson Tree Classfaton Due to ts poor salablty, Nearest Neghbor s unsutable for most data mnng applatons; nstead, deson trees are the most ommon hoe of lassfer. Whle deson trees are defned for real data, attemptng to lassfy tme seres usng the ra data ould learly be a mstake, sne the hgh dmensonalty and nose levels ould result n a deep, bushy tree th poor auray.

8 Error Rate Cylnder - Bell - Funnel Control Chart Impats SDA Euldean LP max SAX To ompare performane, e measure the perentage of dsk I/Os requred n order to retreve the one-nearest neghbor to a randomly extrated query, relatve to the number of dsk I/Os requred for sequental san. Sne t has been forbly shon that the hoe of dataset an make a sgnfant dfferene n the relatve ndexng ablty of a representaton, e tested on more than 5 datasets from the UCR Tme Seres Data Mnng Arhve. In Fgure 4 e sho 4 representatve examples Alphabet Sze Alphabet Sze Fgure 3: A omparson of fve dstane measures utlty for nearest neghbor lassfaton. We tested dfferent alphabet szes for SAX and IMPACTS. SDA s alphabet sze s fxed at 5. In an attempt to overome ths problem, Geurts [6] suggests representng the tme seres as a Regresson Tree (RT) (ths representaton s essentally the same as APCA [22], see Fgure 2), and tranng the deson tree dretly on ths representaton. The tehnque shos great promse. We ompared SAX to the Regresson Tree (RT) on to datasets; the results are n Table 5. Dataset SAX Regresson Tree CC 3.4 ± ± 2. CBF.97 ±.4.4 ±.2 Table 5: A omparson of SAX th the spealzed Regresson Tree approah for deson tree lassfaton. Our approah used an alphabet sze of 6; both approahes used a dmensonalty of 8 Note that hle our results are ompettve th the RT approah, The RT representaton s undoubtedly superor n terms of nterpretablty [6]. One agan, our pont s smply that our blak box approah an be ompettve th spealzed solutons. 4.3 Query by Content (Indexng) The majorty of ork on tme seres data mnng appearng n the lterature has addressed the problem of ndexng tme seres for fast retreval [3]. Indeed, t s n ths ontext that most of the representatons enumerated n Fgure ere ntrodued [7, 4, 22, 35]. Dozens of papers have ntrodued tehnques to do ndexng th a symbol approah [2, 2], but thout exepton, the anser set retreved by these tehnques an be very dfferent to the anser set that ould be retreved by the true Euldean dstane. It s only by usng a loer boundng tehnque that one an guarantee retrevng the full anser set, th no false dsmssals [4]. To perform query by ontent, e bult an ndex usng SAX, and ompared t to an ndex bult usng the Haar avelet approah [7]. Sne the datasets e use are large and dsk-resdent, and the redued dmensonalty ould stll be potentally hgh (or at least hgh enough suh that the performane degenerates to sequental san f R-tree ere used [9]), e use Vetor Approxmaton (VA) fle as our ndexng algorthm. We note, hoever, that SAX ould also be ndexed by lass strng ndexng tehnques suh as suffx trees DWT Haar SAX Ballbeam Chaot Memory Wndng Dataset Fgure 4: A omparson of ndexng ablty of avelets versus SAX. The Y-axs s the perentage of the data that must be retreved from dsk to anser a -NN query of length 256, hen the dmensonalty reduton rato s 32 to for both approahes One agan e fnd our representaton ompettve th exstng approahes. 4.4 Takng Advantage of the Dsrete Nature of our Representaton In the prevous setons e shoed examples of ho our proposed representaton an ompete th real-valued representatons and the orgnal data. In ths seton e llustrate examples of data mnng algorthms that take explt advantage of the dsrete nature of our representaton Detetng Novel/Surprsng/Anomalous Behavor A smple dea for detetng anomalous behavor n tme seres s to examne prevously observed normal data and buld a model of t. Data obtaned n the future an be ompared to ths model and any lak of onformty an sgnal an anomaly [9]. In order to aheve ths, n [24] e ombned a statstally sound sheme th an effent ombnatoral approah. The statstal sheme s based on Markov hans and normalzaton. Markov hans are used to model the normal behavor, hh s nferred from the prevously observed data. The tme- and spae-effeny of the algorthm omes from the use of suffx tree as the man data struture. Eah node of the suffx tree represents a pattern. The tree s annotated th a sore obtaned omparng the support of a pattern observed n the ne data th the support reorded n the Markov model. Ths apparently smple strategy turns out to be very effetve n dsoverng surprsng patterns. In the orgnal ork e use a smple symbol approah, smlar to IMPACTS [2]; here e revst the ork usng SAX. For ompleteness, e ll ompare SAX to to hghly referened anomaly deteton algorthms that are defned on real valued representatons, the TSA-tree Wavelet based approah of Shahab et al. [3] and the Immunology (IMM) nspred ork of Dasgupta and Forrest [9]. We also nlude the Markov tehnque usng IMPACTS and SDA n order to dsover ho muh of the dfferene an be attrbuted dretly to the representaton. Fgure 5 ontans an experment omparng all 5 tehnques.

9 5 I) -5 5 II) shos an example of a motf dsovered n an ndustral dataset [5] usng ths tehnque. Wndng Dataset (Angular speed of reel ) A B III) IIII) V) VI) A B VII) Fgure 5: A omparson of fve anomaly deteton algorthms on the same task. I) The tranng data, a slghtly nosy sne ave of length,. II) The tme seres to be examned for anomales s a nosy sne ave that as reated th the same parameters as the tranng sequene, then an assortment of anomales ere ntrodued at tme perods 25, 5 and 75. III) and IIII) The Markov Model tehnque usng the IMPACTS and SDA representaton dd not learly dsover the anomales, and reported some false alarms. V) The IMM anomaly deteton algorthm appears to have dsovered the frst anomaly, but t also reported many false alarms. VI) The TSA-Tree approah s unable to detet the anomales. VII) The Markov model-based tehnque usng SAX learly fnds the anomales, th no false alarms The results on ths smple experment are mpressve. Sne suffx trees and Markov models an be used only on dsrete data, ths offers a motvaton for our symbol approah Motf dsovery It s ell understood n bonformats that overrepresented DNA sequenes often have bologal sgnfane [3, 3, 29]. A substantal body of lterature has been devoted to tehnques to dsover suh patterns [7, 32, 33]. In a prevous ork, e defned the related onept of tme seres motf [26]. Tme seres motfs are lose analogues of ther dsrete ousns, although the defntons must be augmented to prevent ertan degenerate solutons. The naïve algorthm to dsover the motfs s quadrat n the length of the tme seres. In [26], e demonstrated a smple tehnque to mtgate the quadrat omplexty by a large onstant fator; nevertheless, ths tme omplexty s learly untenable for most real datasets. The symbol nature of SAX offers a unque opportunty to aval of the ealth of bonformats researh n ths area. In partular, reent ork by Tompa and Buhler holds great promse [33]. The authors sho that many prevously unsolvable motf dsovery problems an be solved by hashng subsequenes nto bukets usng a random subset of ther features as a key, then dong some post-proessng searh on the hash bukets. They all ther algorthm PROJECTION. We arefully remplemented the random projeton algorthm of Tompa and Buhler, makng mnor hanges n the post-proessng step to allo for the fat that although e are hashng random projetons of our symbol representaton, e atually sh to dsover motfs defned on the orgnal ra data. Fgure 6 Of ourse, ths desrpton greatly understates the ontrbutons of ths ork. We urge the reader to onsult the orgnal paper Fgure 6: Above, a motf dsovered n a omplex dataset by the modfed PROJECTION algorthm. Belo, the motf s best vsualzed by algnng the to subsequenes and zoomng n. The smlarty of the to subsequenes s strkng, and hnts at unexpeted regularty Apart from the attratve salablty of the algorthm, there s another mportant advantage over other approahes. The PROJECTION algorthm s able to dsover motfs even n the presene of nose. Our extenson of the algorthm nherts ths robustness to nose. 5. CONCLUSIONS AND FUTURE DIRECTIONS In ths ork e ntrodued the frst dmensonalty reduton, loer boundng, streamng symbol approah n the lterature. We have shon that our representaton s ompettve th, or superor to, other representatons on a de varety of lass data mnng problems, and that ts dsrete nature allos us to takle emergng tasks suh as anomaly deteton and motf dsovery. A host of future dretons suggest themselves. In addton to use th streamng algorthms, there s an enormous ealth of useful defntons, algorthms and data strutures n the bonformats lterature that an be exploted by our representaton [3, 3, 7, 28, 29, 32, 33]. It may be possble to reate a loer boundng approxmaton of Dynam Tme Warpng [6], by slghtly modfyng the lass strng edt dstane. Fnally, there may be utlty n extendng our ork to multdmensonal tme seres [34]. 6. REFERENCES [] Agraal, R., Psala, G., Wmmers, E. L. & Zat, M. (995). Queryng Shapes of Hstores. In proeedngs of the 2 st Int'l Conferene on Very Large Databases. Zurh, Stzerland, Sept - 5. pp [2] André-Jönsson, H. & Badal. D. (997). Usng Sgnature Fles for Queryng Tme-Seres Data. In proeedngs of Prnples of Data Mnng and Knoledge Dsovery, st European Symposum. Trondhem, Noray, Jun pp [3] Apostolo, A., Bok, M. E. & Lonard, S. (22). Monotony of Surprse and Large-Sale Quest for Unusual Words. In proeedngs of the 6 th Int l Conferene on Researh n Computatonal Moleular Bology. Washngton, DC, Aprl 8-2. pp [4] Babok, B, Babu, S., Datar, M., Motan, R. & Wdom, J. (22). Models and Issues n Data Stream Systems. Invted Paper n proeedngs of the 22 ACM Symp. On Prnples of Database Systems. June 3-5, Madson, WI.

10 [5] Bastogne, T., Noura, H., Rhard A. & Httnger, J..M. (22). Applaton of Subspae Methods to the Identfaton of a Wndng Proess. In proeedngs of the 4 th European Control Conferene, Vol. 5, Brussels. [6] Berndt, D. & Clfford, J. (994) Usng Dynam Tme Warpng to Fnd Patterns n Tme Seres. In proeedngs of the Workshop on Knoledge Dsovery n Databases, at the 2 th Int l Conferene on Artfal Intellgene. July 3-Aug 4, Seattle, WA. pp [7] Chan, K. & Fu, A. W. (999). Effent Tme Seres Mathng by Wavelets. In proeedngs of the 5 th IEEE Int'l Conferene on Data Engneerng. Sydney, Australa, Mar pp [8] Cortes, C., Fsher, K., Pregbon, D., Rogers, A. & Smth, F. (2). Hanok: a Language for Extratng Sgnatures from Data Streams. In proeedngs of the 6 th ACM SIGKDD Int l Conferene on Knoledge Dsovery and Data Mnng. Aug 2-23, Boston, MA. pp 9-7. [9] Dasgupta, D. & Forrest, S. (996) Novelty Deteton n Tme Seres Data usng Ideas from Immunology. In proeedngs of The Internatonal Conferene on Intellgent Systems. June 9-2. [] Datar, M. & Muthukrshnan, S. (22). Estmatng Rarty and Smlarty over Data Stream Wndos. In proeedngs of the th European Symposum on Algorthms. Sep 7-2, Rome, Italy. [] Da, C. S., Fnney, C. E. A. & Tray, E. R. (2). Symbol Analyss of Expermental Data. Reve of Sentf Instruments. ( ). [2] Dng, C., He, X., Zha, & Smon., H. (22). Adaptve Dmenson Reduton for Clusterng Hgh Dmensonal Data. In proeedngs of the 2 nd IEEE Internatonal Conferene on Data Mnng. De 9-2. Maebash, Japan. pp [3] Durbn, R., Eddy, S., Krogh, A. & Mthson, G. (998). Bologal Sequene Analyss: Probablst Models of Protens and Nule Ads. Cambrdge Unversty Press. [4] Faloutsos, C., Ranganathan, M., & Manolopoulos, Y. (994). Fast Subsequene Mathng n Tme-Seres Databases. In proeedngs of the ACM SIGMOD Int l Conferene on Management of Data. May 24-27, Mnneapols, MN. pp [5] Fayyad, U., Rena, C. &. Bradley, P. (998). Intalzaton of Iteratve Refnement Clusterng Algorthms. In proeedngs of the 4 th Internatonal Conferene on Knoledge Dsovery and Data Mnng. Ne York, NY, Aug pp [6] Geurts, P. (2). Pattern Extraton for Tme Seres Classfaton. In proeedngs of the 5th European Conferene on Prnples of Data Mnng and Knoledge Dsovery. Sep 3-7, Freburg, Germany. pp [7] Gons, A. & Mannla, H. (23). Fndng Reurrent Soures n Sequenes. In proeedngs of the 7 th Internatonal Conferene on Researh n Computatonal Moleular Bology. Apr -3, Berln, Germany. To Appear. [8] Guha, S., Mshra, N., Motan, R. & O'Callaghan, L. (2). Clusterng Data Streams. In proeedngs of the 4 st Symposum on Foundatons of Computer Sene. Nov 2-4, Redondo Beah, CA. pp [9] Hellersten, J. M., Papadmtrou, C. H. & Koutsoupas, E. (997). Toards an Analyss of Indexng Shemes. In proeedngs of the 6 th ACM Symposum on Prnples of Database Systems. May 2-4, Tuson, AZ. pp [2] Huang, Y. & Yu, P. S. (999). Adaptve Query Proessng for Tme- Seres Data. In proeedngs of the 5 th Int'l Conferene on Knoledge Dsovery and Data Mnng. San Dego, CA, Aug 5-8. pp [2] Kalpaks, K., Gada, D. & Puttagunta, V. (2). Dstane Measures for Effetve Clusterng of ARIMA Tme-Seres. In proeedngs of the 2 IEEE Internatonal Conferene on Data Mnng, San Jose, CA, Nov 29-De 2. pp [22] Keogh, E., Chakrabart, K., Pazzan, M. & Mehrotra, S. (2). Loally Adaptve Dmensonalty Reduton for Indexng Large Tme Seres Databases. In proeedngs of ACM SIGMOD Conferene on Management of Data. Santa Barbara, CA, May pp [23] Keogh, E. & Kasetty, S. (22). On the Need for Tme Seres Data Mnng Benhmarks: A Survey and Empral Demonstraton. In proeedngs of the 8 th ACM SIGKDD Internatonal Conferene on Knoledge Dsovery and Data Mnng. July 23-26, 22. Edmonton, Alberta, Canada. pp 2-. [24] Keogh, E., Lonard, S. & Chu, W. (22). Fndng Surprsng Patterns n a Tme Seres Database n Lnear Tme and Spae. In the 8 th ACM SIGKDD Internatonal Conferene on Knoledge Dsovery and Data Mnng. July 23-26, 22. Edmonton, Alberta, Canada. pp [25] Keogh, E. & Pazzan, M. (998). An Enhaned Representaton of Tme Seres Whh Allos Fast and Aurate Classfaton, Clusterng and Relevane Feedbak. In proeedngs of the 4 th Int'l Conferene on Knoledge Dsovery and Data Mnng. Ne York, NY, Aug pp [26] Ln, J., Keogh, E., Lonard, S. & Patel, P. (22). Fndng Motfs n Tme Seres. In proeedngs of the 2 nd Workshop on Temporal Data Mnng, at the 8 th ACM SIGKDD Int l Conferene on Knoledge Dsovery and Data Mnng. Edmonton, Alberta, Canada, July pp [27] Larsen, R. J. & Marx, M. L. (986). An Introduton to Mathematal Statsts and Its Applatons. Prente Hall, Engleood, Clffs, N.J. 2 nd Edton. [28] Lonard, S. (2). Global Detetors of Unusual Words: Desgn, Implementaton, and Applatons to Pattern Dsovery n Bosequenes. PhD thess, Department of Computer Senes, Purdue Unversty, August, 2. [29] Renert, G., Shbath, S. & Waterman, M. S. (2). Probablst and Statstal Propertes of Words: An Overve. Journal of Computatonal. Bology. Vol. 7, pp -46. [3] Roddk, J. F., Hornsby, K. & Splopoulou, M. (2). An Updated Bblography of Temporal, Spatal and Spato-Temporal Data Mnng Researh. In Post-Workshop Proeedngs of the Internatonal Workshop on Temporal, Spatal and Spato-Temporal Data Mnng. Berln, Sprnger. Leture Notes n Artfal Intellgene. Roddk, J. F. and Hornsby, K., Eds [3] Shahab, C., Tan, X. & Zhao, W. (2). TSA-tree: A Wavelet- Based Approah to Improve the Effeny of Mult-Level Surprse and Trend Queres In proeedngs of the 2 th Int l Conferene on Sentf and Statstal Database Management. pp [32] Staden, R. (989). Methods for Dsoverng Novel Motfs n Nule Ad Sequenes. Computer Applatons n Bosenes. Vol. 5(5). pp [33] Tompa, M. & Buhler, J. (2). Fndng Motfs Usng Random Projetons. In proeedngs of the 5 th Int l Conferene on Computatonal Moleular Bology. Montreal, Canada, Apr pp [34] Vlahos, M., Kollos, G. & Gunopulos, G. (22). Dsoverng Smlar Multdmensonal Trajetores. In proeedngs of the 8 th Internatonal Conferene on Data Engneerng. Feb 26-Mar, San Jose, CA. [35] Y, B, K., & Faloutsos, C. (2). Fast Tme Sequene Indexng for Arbtrary Lp Norms. In proeedngs of the 26 st Int l Conferene on Very Large Databases. Sep -4, Caro, Egypt. pp