Novel pruning based hierarchical agglomerative clustering for mining outliers in financial time series

Transcription

1 Computatioal Fiace ad its Applicatios III 33 Novel pruig based hierarchical agglomerative clusterig for miig outliers i fiacial time series D. Wag, P. J. Fortier & H. E. Michel Wester Asset Maagemet, USA Departmet of Electrical ad Computer Egieerig, Uiversity of Massachusetts, Dartmouth, USA Abstract Ivestors must make iformed decisios usig partial ad imperfect iformatio. As accuracy ad completeess of iformatio held by the ivestor rise, the probability for better decisio makig also rises. Similarity search based outlier detectio i fiacial time series is key to makig better decisios for may ivestmet strategies ad portfolio maagemet techiques. This motivates people to utilize umerous data miig techiques to discover similarities from massive fiacial time series data pools. The research itroduces a ovel pruig based Hierarchical Agglomerative Clusterig (HAC) algorithm to search for similarity amog fiacial time series i high dimesioal space usig securities i the S&P500 idex as experimetal data. The algorithm is based o vertical ad horizotal dimesio reductio algorithms [] ad a uique similarity measuremet defiitio [] with the time value cocept. This paper discloses a series of experimet results that illustrate the effectiveess of the algorithm. Keywords: outlier, data miig, computatioal fiace, fiacial time series, similarity search, high dimesio, clusterig. Itroductio We propose a ovel similarity search i high dimesioal fiacial time series by usig a pruig based HAC algorithm. The similarity search is performed after dimesioality reductio, which composes of a Attributes Selectio (AS) algorithm [] ad a Piecewise Liear Represetatio (PLR) based Segmetatio doi:0.495/cf08004

2 34 Computatioal Fiace ad its Applicatios III ad Pruig (SP) algorithm as preseted i Wag [], ad a uique similarity measuremet defiitio [] based o PLR represeted fiacial time series objects. The paper is orgaized as follows: Sectio presets the problem statemet to be addressed ad summarizes the major cotributio of this work toward academia. The backgroud ad related work of this topic is reviewed i Sectio 3. Sectio 4 states the busiess similarity. Similarity defiitio ad measuremet i Wag [] is re-preseted i Sectio 5. Sectio 6 describes the pruig based HAC algorithm to search similarity. The experimets ad result aalysis are preseted i Sectio 7. The coclusio ad future work are discussed i Sectio 8. Problem statemet ad cotributio Ivestors usually have differet ivestmet ad portfolio maagemet strategies [0]. Some strategies iclude, cash surrogate passive strategy, passive strategy, outlier active strategy, ad portfolio diversificatio [7] across segmets etc. The key is to measure the similarity of the fiacial time series based o historical data before further miig. Existig similarity search algorithms [4, 6, 3] look for similar sequece based o give whole time series sequece or subsequece i o-high dimesioal space ad assume all historical data has equivalet weight for similarity. I real fiacial markets, the fiacial time series are drive by a high umber of dimesios ad historical data has time value. More recet data is more relevat to decisio makig based our research. Sophisticated usupervised similarity search algorithms are eeded to help ivestors compute similarity from hudreds or eve thousads of securities. The goal is to develop a method that ot oly improves the efficiecy i idetifyig similarity, but also provides a very high level of accuracy. If that is achieved, ivestors ca expect to reduce the risk compoet i their ivestmet decisio makig process. The developed similarity search algorithm cotributes to the academic research fields i may ways. The first to study applyig data objects time value fuctio to high dimesioal fiacial time series. Developed a ovel pruig based HAC algorithm based o dimesioal reductio algorithms. Developed a uique similarity measuremet defiitio ad calculatio. Provided a similarity measuremet foudatio for developig potetial models i fiacial markets. 3 Backgroud ad related work 3. Time series ad sequece data miig research A time series is a sequece of real umbers, represetig the measuremets of a real variable at equal time itervals. A time series database is a large collectio of time series data such as that foud i the NYSE, ad NASDAQ stock

3 Computatioal Fiace ad its Applicatios III 35 databases. Classical time series aalysis ca be categorized ito idetifyig patters [5] ad forecastig. Similarity search is oe of the major research topics withi time series data ad is used i subsequece similarity, clusterig, idexig, ad rule discovery. Existig similar sequece matchig algorithms, map a data sequece of legth to a poit i a -dimesioal space. Most of the algorithms defie similarity betwee two data sequeces usig the Euclidea distace (ED) betwee two correspodig poits. ED is easy to compute ad scales to other problems such as clusterig, ad idexig. However, it does t allow for differet baselies (i.e. stock A fluctuates at $80 ad stock B at $0) ad differet scales (i.e. Stock A fluctuates betwee $80 ad $0, stock B betwee $5 ad $5). The challege is to desig solutios that attempt to strike a balace betwee accuracy ad efficiecy. Sice search performace degrades expoetially as the dimesioality of the idex structures icreases, most of the existig algorithms reduce the dimesioality by mappig the -dimesioal poits ito the f-dimesioal oes (f<). Wu, Salzberg ad Zhag [3] proposed a subsequece based similarity search method particularly for stock market predictio aalysis. However, the paper employs a b% idicator, a popular idicator derived from Bolliger Bads, ad utterly relies o zigzag price movemet to do predictio ad therefore is ot a high dimesioality approach. I reality, most sophisticated ivestors will cosider ot oly the price, but also other dimesios (EPS, daily tradig volume, 5 weeks low ad highs, etc.) i makig ivestmet decisios. 3. Hierarchical agglomerative clusterig Hierarchical clusterig is a determiistic algorithm. The hierarchical based algorithms use a distace matrix as clusterig criteria ad do ot require the umber of clusters k as a iput, but require a termiatio coditio. Two wellkow heuristic methods are k-meas ad k-medoids algorithms [6]. I our study we use HAC to build a hierarchical classificatio of objects through a series of biary merges. The algorithm starts by mergig the idividual objects followed by mergig these iitial clusters. A slice across the hierarchy ca the be take at ay level to provide the desired umber of clusters. We experieced differet rules for agglomeratio, i.e. merge two clusters with the smallest distace betwee two iter-cluster elemets. 4 Busiess similarity The busiess defiitio of similarity is the foudatio of our algorithm ad implemetatio. Fiacial Aalysts believe that idividual stocks with a similar price chage, a similar slope chage, a similar legth of chage period ad with other attributes such as similar tradig volume, solid earig per share ad price per earig are cosidered to be similar objects. Fiacial Aalysts believe the reflect characteristics at the curret time. Price is the most importat attribute i fiacial markets ad is the foudatio of the similarity defiitio i this study. The properties of subsequece of the

4 36 Computatioal Fiace ad its Applicatios III price attributes movemet over time are typically of more cocer by fiacial aalysts. For example, i the Figure, S ad S3, S ad S4 have same amplitudes with differet time differeces. S ad S, S3 ad S4 have the same time differece with differet amplitude differeces. Fiacial aalysts cosider S ad S3, S ad S4 more similar while S ad S, S3 ad S4 are less similar. So we apply differet weights for ormalized amplitude chage ad time differece. Based o similarity priciples above from the busiess world, we proposed our similarity defiitio ad measuremets [] i sectio 5. Price p8 p7 S S4 p6 p5 p4 p3 S S3 p p t0 t t t3 t4 t5 t6 t7 Time Figure : Price dimesio similarity amplitude vs. time differece. 5 Similarity defiitio ad measuremet Various idexig based similarity searchig techiques have bee explored for large databases. The biggest challege i our research was how to implemet similarity search i high dimesioal data space across all time series data objects. The geeral idea is outlied below. Based o characteristics aalysis of time series ad data trasformatio, the followig measuremets are proposed: First, the zigzag shape of the Price dimesio must be similar. We use a method similar to that proposed i Wu et al [3] based o permutatio of the chagig poits ad Euclidea distace betwee the objects based o the PLR. Secod, the tred is always very importat i a time series ad should be part of the similarity search. We cosider the Price attribute tred oly for approvig the cocept. Third, for the remaiig selected attributes, we calculate weighted distace based o the ormalized values at chagig poits time stamps. This is based o the correlatio coefficiet value of the attributes calculated durig the Attribute Selectio algorithm [] agaist the Price dimesio. Fourth, a time value fuctio is applied to all data poits. Based o the above aalysis ad observatio, we proposed a similarity defiitio betwee two time series objects i Wag []: Defiitio : Similarity measuremet Give two time series object S ad S, S= ((t, D, T ), (t, D, T ) (t, D, T ), t 0, t c ),

5 Computatioal Fiace ad its Applicatios III 37 S = ((t, D, T ), (t, D, T ) (t, D, T ), t 0, t c ), where D = (D, D, D 3,, D k ), T = (T, T, T 3,, T k ) ad k the umber of fial dimesios. S ad S are similar if S ad S s satisfy the followig properties: S ad S have the same umber of segmets ad permutatio of the Price dimesio. This is the pre-coditio for ay of the followig properties. Distace(S, S ) < ε, where ε is the give error threshold ad Distace(S, k S ) = γ i * f ( t) * d( S. Ait, S. Ait ), γ i is the aggregated correlatio i= t= 0 coefficiet value of i th dimesio A i. f(t) is the Time Fuctio. The distace refers to the amplitude chage differece of Price ad the other selected dimesios. Distace(T,T ) < ε, where ε is give a error threshold ad Distace(T,T ) = t= f ( t) * d( S. T, S. T ). T i this study is oly for the Price dimesio. f(t) is the Time Fuctio ad λ t c t f ( t) = i this study where λ is a experimet value i.e. λ =0.95. The similarity calculatio uses backward order to calculate the value of each segmet based o a time value fuctio. Accordig to the time value fuctio, older data poits withi a selected time widow cotribute less to the fial similarity value. The segmet is the base uit for whole calculatio. The similarity value is the average of the sum of the similarity of each segmet. Each pair of segmet similarities is calculated based o the price dimesio, tred of the price dimesio ad other values from the fial selected attributes with the time value fuctio. Equatio : Segmet similarity betwee two objects: [] Sim ( seg = ( λ = ( λ ( ( h h, seg t + t + t t ) ) α P P + β T ~ ) α P P + β T ) ) α P P P P ~ + α + + α m j = m ( γ j A j A j ) j = m ( γ j A j A j ) j = ( γ ( ) j A j A j A j A j + β T ~ T ~ t ( + t h ) where ( λ ) is the time value fuctio ad h is the umber of busiess days withi the user specified time period. t is the date for edig chagig poit ad t is the date for startig chagig poit of seg, where seg stads for segmet. P ad P stad for the edig ad startig plast value. α ad β

6 38 Computatioal Fiace ad its Applicatios III stad for the relative cotributio scale of the attributes amplitude ad tred to the similarity respectively. T ~ stads for the tred degree value for seg with ed poit as ad start poit as. 6 Hierarchical agglomerative clusterig 6. Pruig based HAC algorithm We customized the traditioal HAC algorithm through applicatio of a pruig coditio. After our PLR based Segmetatio ad Pruig algorithm [], the iitial data set is divided ito groups based o the umber of segmets. Moreover, the groups are divided ito subgroups based o a permutatio. The data objects with the same permutatio are grouped ito sub groups. We cosider these objects as havig more similar characteristics. First, the similarity values oly eed to be computed withi subgroups. Secod, the clusters formatio ca ot be across sub groups. With these two coditios, we prue the cluster formatio based o the traditioal HAC algorithm. The followig is our customized HAC algorithm pseudo code. /*************customized HAC algorithm pseudo-code****************/ Begi; Read i all PLRed stock file ames; Iitialize HAC istace hac; //step : groupig by umber of segmets hac.groupbysegnumber(); // step : Iside each group, filter stock by permutatio algorithm hac.groupbypermutatio(){ Perform permutatio iside each segmet group; For every two stocks iside each segmet group sort two stocks low poits by plast attribute; if all low poit idex are same; cotiue to high poits permutatio; Else Skip these two stocks because they ca ot be permuted; sort two stocks high poits by plast attribute; If all high poit idex are same; Put i oe sub group uder the segmet group; Else Skip these two stocks because they ca ot be permuted; } //step 3: HAC clusterig based o segmet groupig ad permutatio sub groupig Load each sub group withi segmet group as idividual group; Create HACMatrix for each idividual group; Perform HAC clusterig based o HACMatrix(){ //Compute Similarity For every two stocks iside HACMatrix group{ Compute similarity value; Store the similarity value alog with two stocks idex i a Array; Compute similarity calculatio; } //customized clusterig based o miimum distace betwee clusters Fid miimum stocks similarity value; Produce a cluster by joiig the stocks with this miimum similarity value; Update HACMatrix; Recompute similarity value betwee ew grouped stocks ad other stocks; } /*************customized HAC algorithm pseudo-code****************/ As depicted i the pseudo code above, first objects are grouped by the umber of segmets. Secod further group objects to produce subgroups iside each group through the permutatio process. The permutatio process divides all chagig poits of ay two objects ito upper chagig poits ad lower chagig poits. If the idex is the same for each poit, the two objects belog to

7 Computatioal Fiace ad its Applicatios III 39 the same subgroup. The third step performs HAC based o the produced subgroups. The algorithm creates a HAC matrix to store the similarity values for each idividual subgroup. After the cluster is geerated at each step, the algorithm updates the matrix ad re-computes the similarity values betwee the ew cluster ad other clusters or objects. 7 Experimet ad result aalysis 7. Raw data ad attribute set The Stadard & Poor 500 Composite (S&P 500) stocks from BLOOMBERG were selected as our raw data. We obtaied 5 years of historical data with 95 attributes ragig from the pricig attributes to fudametal fiacial attributes. I order to adapt the evirometal effect, also collected were key ecoomic idicators for the Uited States over those 5 years from DATASTREAM. 7. Attribute selectio ad costructio The selectio of the iitial attribute set was coducted usig iputs from fiacial market domai experts. Three categories of attributes: market attributes, fiacial attributes, ad evirometal attributes were selected. Besides these directly accessible attributes, others were created to better measure Price l Price t stock performace. For istace, γ : which reflects the ratio of retur betwee the curret day ad last day. 7.3 Sample clusters aalysis t = t Figure below represets groups of detected similar objects based o pxlast pruig thresholds equal to.5% with the data ragig from Jauary, 00 to December 30, 003. We explai each figure i detail accordig to our framework but will oly focus o Price amplitude chage ad tred degree. All three objects CVX, CSX, ad UCL have 6 segmets. All three objects have the same upper ad lower poits which esure that the basic shapes of all three objects are the same. CVX ad UCL joied as oe cluster first. CSX is joied to the cluster i secod step. As you ca otice from figure above, the reaso why CVX ad UCL are joied as oe cluster first is due to the time value fuctio. The first segmet (backward) of CVX ad UCL is more similar sice both segmets are very short with similar amplitude chage. The differece betwee these three objects is mostly see i segmet #3 ad #4. Segmet #3 s objects have very close amplitude chage which is times the weighted tred chage i our experimet. The amplitude chage plays a more importat role. Also, ote that the tred chage is ot too far away.

8 J a 0 J a 0 3 Ja04 Computatioal Fiace ad its Applicatios III C l u s t e r # C V X, C S X, U C L p x l a s t r a w d a t a s t r e a m s C V X C S X U C L T i m e (a) Raw data streams 60 CSX CSX 0 Ja0 Apr0 Jul0 Oct0 Ja03 Apr03 Jul03 Oct03 Ja04 Tim e Ja0 Apr0 Jul0 Oct0 Ja03 Apr03 Jul03 Oct03 Ja04 Tim e 30 CVX UCL 0 Ja0 Apr0 Jul0 Oct0 Ja03 Apr03 Jul03 Oct03 Ja04 Tim e CVX UCL (b) PLR segmets represetatio of raw data stream before pruig 50 CSX 30 Ja0 Apr0 Jul0 Oct0 Ja03 Apr03 Jul03 Oct03 Ja04 Time 60 CVX 0 Ja0 Apr0 Jul0 Oct0 Ja03 Apr03 Jul03 Oct03 Ja04 Time UCL 30 0 Ja0 Apr0 Jul0 Oct0 Ja03 Apr03 Jul03 Oct03 Ja04 Time (c) Pruig result used to compute the similarity Figure : Cluster # CVX, CSX, ad UCL with.5% puig threshold.

9 Computatioal Fiace ad its Applicatios III 4 Therefore they are cosidered to be very close by the algorithm. There are also other selected attributes deoted as o i the figure ad their amplitude chages cotribute to the fial similarity value as well. You ca see they are similar over the Price dimesio i fact, especially the CVX ad UCL stock objects. 8 Coclusio I this study, we itroduced a pruig based Hierarchical Agglomerative Clusterig algorithm over high dimesioal fiacial time series. The pruig based Hierarchical Agglomerative Clusterig algorithm is based o a piecewise liear represeted multi-dimesioal data object, produced by our Segmetatio ad Pruig algorithm over high dimesioal fiacial data alog with our Attribute Selectio algorithm. Our experimets show that our vertical (Attribute Selectio algorithm) ad horizotal (PLR based Segmetatio ad Pruig algorithm) dimesio reductio algorithms [] are able to preset the objects i cocise ad accurate form to be further processed by our HAC algorithm. We also proposed a uique similarity measuremet [] which is composed of three parts, the Price dimesio amplitude similarity, Price dimesio tred similarity, ad other selected attributes amplitude similarity. Our similarity measuremet properties were derived from fiace idustry aalysts, givig this research a realistic method for idetifyig similarity. Moreover, we itroduced a ew time value cocept to the similarity measuremet. We fid that more recet data cotributes more sigificatly to the similarity measuremet. Cosideratio of a etire time series history is ot meaigful to curret decisio makig accordig to fiacial aalysts i the idustry. The experimets illustrate that our algorithms are able to fid similarity efficietly withi a large set of high dimesioal data. I this study, we focused o static time series data. We ca improve the algorithms preseted i this research to dyamic (icremetally updated) time series data by dealig with the movig perspective of the data. I additio, our high dimesioal dimesio reductio algorithms ca be utilized to predict the depedet variable data characteristics i ext time iterval based o classificatio. Refereces [] Adam Blazejewski, Richard Coggis. Applicatio of Self-orgaizig Maps to Clusterig of High-frequecy Fiacial Data. The secod workshop o Australasia iformatio security, Data Miig ad Web Itelligece, ad Software Iteratioalisatio - Volume 3, Pages: 85 90, 004. [] Steve Craighead ad Bruce Klemesrud. Stock Selectio Based o Cluster ad Outlier Aalysis. Fifteeth Iteratioal Symposium o Mathematical Theory of Networks ad Systems Uiversity of Notre Dame, August -6, 00.

10 4 Computatioal Fiace ad its Applicatios III [3] Marti Gavrilov, Dragomir Aguelov, Piotr Idyk, Rajeev Motwai. Miig the Stock Market: Which Measure Is Best? The sixth ACM SIGKDD Iteratioal Coferece o Kowledge Discovery ad Data Miig, Pages: , 000. [4] Dimitrios Guopulos(UC, Riverside), Gauta Das(Microsoft Research). Tutorial PM- Time Series Similarity Measures. The sixth ACM SIGKDD Iteratioal Coferece o Kowledge Discovery ad Data Miig, Pages: , 000. [5] Jiawei Ha, Guozhu Dog, Yiwe Yi. Efficiet Miig of Partial Periodic Patters i Time Series Database. The fifteeth Iteratioal Coferece o Data Egieerig, pages 06 5, 999. [6] Jiawei Ha ad Michelie Kamber. Data miig: Cocepts ad Techiques. 00 by Academic Press. [7] Mehmed Katardzic, Pedram Sadeghia, Chu She. The Time Diversificatio Moitorig of a Stock Portfolio: a Approach Based o the Fractal Dimesio. The 004 ACM symposium o Applied computig, Pages: , 004. [8] Boris Kovalerchuk ad Evgeii Vityaev. Data Miig i Fiace: Advaces i Relatioal ad Hybrid Methods. 000 by Kluwer Academic Publisher. [9] Jessica Li, Eamo Keogh, Wager Truppel. Clusterig of Streamig Time Series Is Meaigless. The 8th ACM SIGMOD workshop o Research Issues i Data Miig ad Kowledge Discovery, Pages: 56 65, 003. [0] Robert R. Trippi, Jae K. Lee. Artificial Itelligece i Fiace & Ivestig: State-Of-The-Art Techologies for Securities Selectio ad Portfolio Maagemet. Irwi Professioal Publishig; Revised Editio (Jauary, 996). [] Daju Wag, Paul J. Fortier, Howard E. Michel, ad Theophao Mitsa. T- outlier ad a ovel dimesioality reductio framework i high dimesioal time series fiacial data. The Secod Iteratioal Coferece o Computatioal Fiace ad its Applicatios, Jue 006 [] Daju Wag, Paul J. Fortier, Howard E. Michel, ad Theophao Mitsa. Hierarchical Agglomerative Clusterig Based T-outlier Detectio. The Sixth IEEE Iteratioal Coferece o Data Miig, Risk Miig Workshop Page(s): [3] Huamei Wu, Betty Salzberg, ad Doghui Zhag. Olie Evet-drive Subsequece Matchig over Fiacial Data Streams. ACM SIGMOD Iteratioal Coferece o Maagemet of Data, Pages: 3 34, 004. [4] Keji Yamaishi ad Ju-ichi Takeuchi. A Uifyig Framework for Detectig Outliers ad Chage Poits from No-Statioary Time series Data. The eighth ACM SIGKDD Iteratioal Coferece o Kowledge Discovery ad Data Miig, Pages: , 00.