A Novel Bit Level Time Series Representation with Implication of Similarity Search and Clustering

Transcription

1 A Novel Bit Level Time Series Representation with Impliation of Similarity Searh and lustering hotirat Ratanamahatana, Eamonn Keogh, Anthony J. Bagnall 2, and Stefano Lonardi Dept. of omputer Siene & Engineering, Univ. of alifornia, Riverside, A 9252 USA {ratana, eamonn, stelo}@s.ur.edu 2 Shool of omputing Sienes, University of East Anglia, Norwih, UK ajb@mp.uea.a.uk Abstrat. Beause time series are a ubiquitous and inreasingly prevalent type of data, there has been muh researh effort devoted to time series data mining reently. As with all data mining problems, the key to effetive and salable algorithms is hoosing the right representation of the data. Many high level representations of time series have been proposed for data mining. In this work, we introdue a new tehnique based on a bit level approximation of the data. The representation has several important advantages over existing tehniques. One unique advantage is that it allows raw data to be diretly ompared to the redued representation, while still guaranteeing lower bounds to Eulidean distane. This fat an be exploited to produe faster exat algorithms for similarly searh. In addition, we demonstrate that our new representation allows time series lustering to sale to muh larger datasets. Introdution Time series are a ubiquitous and inreasingly prevalent type of data. Beause of this fat, there has been muh researh effort devoted to time series data mining in the last deade [ ],[ 2],[ 3],[ 4]. As with all data mining problems, the key to effetive and salable algorithms is hoosing a suitable representation of the data. Many high level representations of time series have been proposed for data mining. In this work, we introdue a novel tehnique based on a bit level approximation of the data. As we will show, our lipped representation has several important advantages over existing tehniques. The proposed approah is not only a new representation; it is a new type of representation. For data adaptive, non-data adaptive, and model-based approahes, the user has a hoie (impliit or expliit) of the ompression ratio. This allows the user to fine tune the parameters to ahieve the ideal ompression/ fidelity tradeoff for their partiular appliation. In ontrast, with the lipped representation, the data itself ditates the ompression ratio; the user has no hoie to make. This may be seen as somewhat of a disadvantage (although removing parameters from a data mining task is often a good thing T.B. Ho, D. heung, and H. Liu (Eds.): PAKDD 2005, LNAI 358, pp , Springer-Verlag Berlin Heidelberg 2005

2 772. Ratanamahatana et al. [ 5]). However, this lak of flexibility is ounterbalaned by another unique property of the lipped representation. For all other dimensionality redution approahes, we must transform the query into the same representation as the dimensionality redued database, i.e. having a loss of fidelity for the andidate mathes stored in the index and a loss of fidelity for the query. This in turn produes weak lower bounds, and thus weak pruning power. In ontrast, the lipped representation is unique in that the original raw query an be ompared diretly to the lipped andidate sequenes, thus produing tighter lower bounds, greater pruning power and faster query by ontent. 2 The lipped Representation Our proposed representation works by replaing eah real valued data point with a single bit. gives the visual intuition Fig.. A time series,, of length 64, is onverted to the lipped representation,, by observing eah element of ; if its value is stritly above zero, the orresponding bit is set to, and to 0 otherwise More formally, we an define, the lipped representation of as: if ( i) > µ ( i) = 0 otherwise where µ is the mean of. Sine the importane of normalizing the data before attempting any lustering, lassifiation or indexing [ 3] is well-established, we an simply assume µ = 0, without loss of generality for the rest of this work. Note that this representation has been onsidered before in the statistial ommunity [ 6], but its utility for data mining, namely, the ability to lower bound distane funtions, is first doumented here. 2. Lower Bounding Eulidean Distane Suppose we have 2 time series, a query Q = Q,Q 2,,Q i,,q n, and a andidate math =, 2,, j,, n. The Eulidean Distane an simply be used to ompare the two time series. However, if we have a lipped time series, and a raw time series Q, we an also lower bound the squared Eulidean distane between and Q, using equation 2) below. Due to spae limitations, the proof of this LB_lipped is omitted and an be found in [ 7]. However, gives its visual intuition. ()

3 A Novel Bit Level Time Series Representation n Q > = = ( ) i if ( Qi 0 and i 0) or( Qi 0 and i ) LB _ lipped Q, (2) i= 0 Otherwise Q Q Q D(Q,) LB_lipped (Q,) Fig. 2. The distane returned by both LB_lipped(Q, ) and D(Q,) is the sum of squared lengths of the gray hath lines. Beause every hath line for LB_lipped(Q,) is mathed with orresponding line in D(Q,) whih is at least as long, we must have LB_lipped(Q,) D(Q,) 2.2 Run Length Enoding onsider the lipped sequene, whih we have been using as a running example. Its value is Note that we ould write this as 22#0, #, 2#0, #, 3#0, 24#, whih we an interpret as 22 zeros followed by 2 ones, et. The shorter format allows us to fit more data in main memory. In fat, we an be even terser; beause we always toggle from zero to one or vie versa, so we only need to reord the parity of the first bit, giving us 22#0,,2,,3,24. This lassi lossless ompression tehnique is known as Run Length Enoding (RLE). To make the representation even shorter, we an represent the parity bits of 0 and with two speial haraters, and!, respetively; our run length enoding now an be represented We an use this to further redue the lipped representation of the data. Note that while the example above illustrates the idea with ASII haraters, we atually do RLE at the bit level. 2.3 Numerosity Redution Even though the run length-enoding sheme itself gives an impressive ompression ratio, we an improve it by numerosity redution on sliding windows. This step is motivated by observing that while applying a sliding window on the streaming data, time series in onseutive sliding windows are very often idential in the lipped representation, exept for the first and the last values that are omitted and added, respetively. If the time series in eah sliding window has this property, we an exploit this fat and just reord the maximum amount of time this property has onseutively been observed, along with a speial harater, $, that represents this redution. onsider the run length enoding from our example in the previous setion and let the enoding of the next five sliding windows 7,2. We an readily see that the first four windows are very similar and an be redued to one sine the only values differ from eah other are the first and the last (italiized

4 774. Ratanamahatana et al. for larity). However, the 5 th window annot be ombined with the previous one sine the last bit has hanged from to 0, but it an be ombined with its next window. As a result, the final enoding with numerosity redution As before, although we demonstrate the idea with ASII text, we atually enode everything at the bit level. With the Power Demand dataset of size 0,000 data points, numerosity redution together with Huffman oding yields a huge ompression ratio of 057:. Note that while the fator of 32 to ahieved by lipping is lossy, the remaining fator of approximately 33 to is lossless with respet to the lipped data. 3 Empirial Evaluations In this setion, we will provide an extensive empirial omparison among the raw and various representations of ompressed data in two major data mining tasks, time series indexing and lustering. Twelve datasets were used in our indexing experiments, and two were used for lustering experiments (only subsets of results are shown here due to spae limitations). We also tested on a wide range of both real and syntheti datasets. The datasets range from 66 Kilobytes to 2 Gigabytes in size (see [ 7] for omplete details). 3. Experimental Methodology For indexing, we will demonstrate the superiority of our lipped representation in terms of number of disk aesses. We ompare our proposed method with the lassi Pieewise Aggregate Approximation (PAA) and Disrete Fourier Transform (DFT), all preserving similar ompression ratio. We then demonstrate that lipped series an produe lusters similar to those obtained with the raw data when lustering a very large real world database introdued in setion 3.3. We show that lipping performs favorably when ompared to lustering with unlipped data sine lustering an be done faster and with muh less memory requirement. For similarity searh, we performed all experiments over a range of query lengths. Sine we want to inlude PAA in our experiments, the query length is somewhat limited. We therefore onsider query lengths of 256 and 52 data points. We tested our approah on a variety of twelve datasets with various properties within the data, obtained from the UR Time Series Data Mining Arhive [ 8]. The sizes of the datasets range from 6,875 data points to 98,400 data points. Leaving-one-out ross validation is used; on eah run, we randomly pik a query from a database, reate a runlength enoding with numerosity redution for the rest of the data, and determine the resultant ompression ratio. We then reate PAA and DFT on the same data and with the same ompression ratio (or with smaller ompression ratio, in favor of PAA and DFT) then measure the number of random disk aesses for the nearest neighbor queries of all methods. To determine the number of dimensionality redution (m) in PAA and DFT in these ases, we assume that eah value in PAA and DFT an be represented by only two bytes (instead of 4 or 8 bytes) to demonstrate that our results are still ompetitive among all the approahes. In addition, to avoid any possibility of implementation bias, the number of I/O disk aesses of eah method is measured

5 A Novel Bit Level Time Series Representation 775 instead of reording the atual running time. This is done by first omputing the lower bound distanes using LB_lipped and Eulidean distane, between a query and all the sequenes in the dataset. Then to retrieve the nearest neighbor, eah sequene is visited in the order aording to the lower bound values. We ount the number of times the real disk aesses must be made. These numbers also indiates the tightness of the lower bounds for eah representation. The results are averaged over 00 separate runs for eah dataset. For simpliity, we only report results for one-nearest neighbor queries. 3.2 Indexing Results As noted above, the amount of ompression is ditated by the data itself. For the twelve datasets onsidered the ompression ratios range between 60.2: to,089.5:. We ompare different representations in terms of I/O random disk aesses during the proess of the -nearest neighbor retrieval of a query time series. In partiular, in eah run, we redue the dimensionality of the data from n to m using lipped, PAA, and DFT representations, and build their indies on the redued spaes based on their lower bounds between eah subsetion (sliding window) of the time series and the query. To allow a visual omparison, we normalize eah experiment on eah dataset by the worst performing algorithm; the raw numbers are available in [ 7]. Fig. 3 shows the number of disk aesses with lower bounding the Eulidean distane, using the three dimensionality-redution tehniques over the range of query lengths of 256 and 52 data points. In general, the results show that the lipped representation greatly outperforms or at least is omparable to the other approahes, expressing the superiority in its tightness of the lower bounds. Again, we would like to emphasize that our results here are obtained by onservatively assuming only two-byte requirement to represent eah number in PAA and DFT. If we assume 4 or 8 bytes or without the parameter m adjusted, the results will be muh improved Anngun Burst str ERP_data Foetal_eg Infrasound Koski_eg Memory Network Power_data Po wer_italy Winding DFT_256 PAA_256 li pped_256 0 Anngun Burst st r ERP_data Foetal_eg Infrasound Koski_eg Memory Network Power_data Po wer _ita ly Wind ing DFT_52 PAA_52 li pped_52 Fig. 3. Number of disk aesses with lower bounding of Eulidean distane, normalized by the worst performing approah, using the 3 representations for query lengths of 256 and 52 points 3.3 General ompression-based lustering We examine a lass of problems where a DFT approah should produe good results, and show that lipping is better than the most ommonly used DFT approah desribed in [ 2].

6 776. Ratanamahatana et al. To demonstrate how lipping an help with a real world large dataset, we luster optial reording data from a bee's olfatory system [ 9]. The data onsists of 980 images, eah image ontaining of 688x520 measurements. If we onsider eah position in the image as a time series, the data onsists of 357,760 time series of length 980. Preliminary analysis has shown that lustering the series based on similarity in time produes results that have a sensible physiologial interpretation [ 9]. We luster with k-means (with k set to 6) restarted 50 times from random initial entroids, and take as the best lustering the one with the lowest within-luster variation A B D Fig. 4. A) 6 lusters produed using all 2GB of raw data. B) lusters formed using the lipped data with 32: ompression ratio. The spatial luster o-ourrenes between this plot and A) shows its effetiveness in the lipped data redution tehnique. ) lusters formed using PAA with 59 oeffiients, giving 20: ompression ratio. D) lusters formed using first 7 DFT oeffiients, giving 29.7: ompression ratio 4 onlusions In this paper, we have shown that a simple dimensionality redution tehnique, i.e. the lipped representation, an outperform more sophistiated tehniques by a few orders of magnitude. We have shown that our proposed lipped representation an improve the ompression ratio by a wide margin, while being able to maintain or inrease the tightness of its lower bound, whih allows even faster nearest neighbor queries, espeially in ones that require Dynami Time Warping distane measure. Other than produing faster exat algorithms for similarity searh, we have also demonstrated that our lipped representation approah an support lustering and sale to muh larger datasets. Aknowledgements. This researh was partly funded by the NSF under grant IIS

7 A Novel Bit Level Time Series Representation 777 Referenes. Aah, J. & hurh, G. Aligning gene expression time series with time warping algorithms. Bioinformatis(7) (200) Berndt, D., lifford, J. Using dynami time warping to find patterns in time series. AAAI- 94 Workshop on Knowledge Disovery in Databases. (994) Keogh E., Kasetty, S. On the Need for Time Series Data Mining Benhmarks: A Survey and Empirial Demonstration. In the 8 th AM SIGKDD (2002) Yi B K., Faloutsos,. Fast time sequene indexing for arbitrary Lp norms. VLDB (2000) Keogh, E., Lonardi, S., Ratanamahatana, A. Towards Parameter-Free Data Mining. In proeedings of SIGKDD (2004). 6. Kedem B., Slud, E. On Goodness of Fit of Time Series Models: An Appliation of Higher Order rossings, Biometrika, Vol 68, (98) Ratanamahatana,.A., Keogh, E., Bagnall, A.J., Lonardi, S. A Novel Bit Level Time Series Representation with Impliation of Similarity Searh and lustering. (2004) [ 8. Keogh E., Folias, T. The UR time Series Data Mining arhive. (2002) [ 9. Galan, R.F., Sahse, S., Galizia,.G., Herz, A.V.M. "Odor-driven attrator dynamis in the antennal lobe allow for simple and rapid olfatory pattern lassifiation." Neural omputation (2004) 0. Bagnall, A. J., Janaek, G. lustering time series from ARMA models with lipped data, SIGKDD (2004).