Time Series Data Mining (TSDM): A New Temporal Pattern Identification Method for Characterization and Prediction of Complex Time Series Events

Transcription

1 Time Series Daa Mining (TSDM): A New Temoral Paern Idenificaion Mehod for Characerizaion and Predicion of Comlex Time Series Evens Xin Feng, Ph.D. Dearmen of Elecrical and Comuer Engineering Marquee Universiy Milwaukee, Wisconsin 53233, USA (44) xin.feng@mu.edu November 23 An Ancien Chinese Say You are no grown-us ill you reach he age of 3. You are no free from doubs ill he age of 4. You do no know God s Will ill he age of 5. The boom line: Learning is a life-long rocess /4/23 2 Page, 3/9/99

2 Overview of Presenaion Problem Saemen Grahical Problem Saemen Time Series Analysis Lieraure Innovaive New Aroach Algorihm Phase Sace Mahemaical Formulaion Algorihm Resuls Alicaions Progression of Time Series Engineering Financial /4/23 3 Collaboraors/Credis Dr. Noveen Bansal, MSCS Dr. Richard Povinelli, EECE Hai Huang, Microsof, Inc. Odilon K. Senyana, FAA Dr. Wenjing Zhang, Discover, Inc. Shaobo Wang, MU Mining Mulile Temoral Paerns 4 Page 2, 3/9/99

3 Grahical Problem Saemen 2.5 nonsaionary, non-eriodic ime series 2. X Can we auomaically characerize hese sequences (emoral aerns)? Can we use such emoral aerns for redicion? /4/23 5 Temoral Paern Find emoral aerns P Q, a vecor of lengh Q emoral aern Q = x /4/23 6 Page 3, 3/9/99

4 Evens Temoral aerns ha characerize and redic evens Chosen a riori Algorihm no resriced by even definiion Even definiion is roblem secific Time insances wih high even value /4/23 7 Time Series Analysis Lieraure Box-Jenkins, ARMA Pandi and Wu (983) Bowerman and O Connell (993) Chaoic deerminisic Takens (98) Sauer (99) Casdagli (989) Abarbanel (99, 994) Ghoshray (996) /4/23 8 Page 4, 3/9/99

5 An Innovaive Aroach Time Series Daa Mining Time Series, X x,,, N Find emoral aerns ha are characerisic and redicive of an evengx Q P Nonsaionary, non-eriodic ime series Chaoic deerminisic ime series whose aracors are non-saionary Local model, local redicion No concerned wih characerizing and redicing everywhere (every ime) Characerize and redic evens Alying Geneic Algorihm (GA) o find he oimal local model Exciing, new resuls wih difficul real world ime series /4/23 9 Inroducion o Daa Mining A se in he knowledge discovery rocess Alicaion of algorihms o exrac meaningful aerns /4/23 Page 5, 3/9/99

6 Daa Mining and Knowledge Discovery "The nonrivial exracion of imlici, reviously unknown, and oenially useful informaion from daa"[] Uses arificial inelligence, saisical and visualizaion echniques o discover and resen knowledge in a form which is easily comrehensible o humans. [] W. Frawley and G. Piaesky-Shairo and C. Maheus, Knowledge Discovery in Daabases: An Overview. AI Magazine, Fall 992, gs /4/23 Daa Mining and Knowledge Discovery More recenly (already obsolee): The huge amoun of racking daa available from web sies as an "informaion gold mine. Business Week 74% of large comanies exec o mine web daa o increase rofis by 22. Forreser Research (998) /4/23 2 Page 6, 3/9/99

7 Poular Daa Mining Problems Associaing Idenifies aerns or grous of iems men who buy red ies also ofen buy cigars. Classifying Idenifies clusers of iems wih common aribues men who buy red ies and cigars also usually have wine a lunch and ay by credi card. /4/23 3 Poular Daa Mining Problems Sequencing Idenifies he order of evens men end o buy red ies before lunch and cigars afer lunch. Predicive Modeling Idenifies a likely oucome from iem clusers. men who buy red ies and cigars and have wine a lunch are very likely o buy a silver Benz wihin wo years and finance heir urchase by borrowing money from bank /4/23 4 Page 7, 3/9/99

8 Gold Mining Analogy Where do rosecors search for he gold? Geological formaion Quarz and ironsone Srucures such as banded iron formaions Daa Mining Define formaions ha oin o nugges of informaion Define aerns ha idenify an informaion srike Definiion of gold For Gold Mining Size of nugges makes a difference Mining for oil or silver is differen Daa Mining Definiion of knowledge Clearly define he desired nugges of informaion /4/23 5 Knowledge Discovery in Daabases Cleaning Inegraion Selecion Transformaion Daa Mining Evaluaion Visualizaion Daa Warehouse Preared daa Daa Knowledge Base Paerns Knowledge /4/23 6 Page 8, 3/9/99

9 Feaures of Daa Mining Inend o discover knowledge which are reviously unknown A Muli-discilinary field growing ou of many areas Mahemaical modeling and saisics Paern recogniion Comuaional inelligence /4/23 7 Daa Mining Tools Tradiional aern recogniion algorihms: Saisics, Mahemaical Modeling, Algorihms. Daa Visualizaion, Grahics Inelligen comuing: Arificial Neural neworks, Fuzzy Logic, Geneic Algorihms/Evoluionary Comuing, Exer Sysems, Naural Language Processing, ec. They all use mechanical comuing ower: Daabase Sysems; Clien-Server; Inerne/WWW; Sofware/rogramming; ec. /4/23 8 Page 9, 3/9/99

10 Chaoic Deerminisic Time Series No universal definiion Characerized by he following crieria Sensiiviy o iniial condiions Posiive Lyaunov exonen Broadband Fourier secra Finie, ossibly fracional aracors /4/23 9 Aracors Given a manifold M and a ma f: M M, an invarian se S is defined as follows S x x S f n x S n M :,, A osiively invarian se requires ha n. A se A M is an aracing se if A is a closed invarian se and here exiss a neighborhood U of A such ha U is a osiively invarian se and n f x A x U An aracor is defined as an aracing se ha conains a dense orbi. N. B. Tufillaro, T. Abbo, and J. Reilly, An Exerimenal Aroach o Nonlinear Dynamics and Chaos. Redwood Ciy, CA: Addison-Wesley, 992. /4/23 2 Page, 3/9/99

11 Gold Mining Analogy Where do rosecors search for he gold? Geological formaions Quarz and ironsone Srucures such as banded iron formaions Daa Mining Define formaions ha oin o nugges of informaion (evens) Define emoral aerns ha idenify an informaion srike. Definiion of gold Size of nugges makes a difference in how he mining is aroached. Mining for oil or silver is differen Daa Mining Definiion of knowledge Clearly define he desired nugges of informaion (evens) Define even funcion g( ) /4/23 2 Two ideas arise from he analogy Paerns (Temoral Paerns) Paerns ha guide and direc o he nugges of informaion need o be undersood and idenified Temoral aerns and ime series embedding Gold (Evens) Nugges of informaion require clear definiion Time series evens Relaionshi beween a emoral aern and even Find emoral aerns ha indicae a high even value Examle A sequence of charge flow values in he brain ha recede a hysical resonse A series of volage values ha recede he release of a drole of meal from a welder A sequence of sock rices ha recede a raid increase in he sock rice /4/23 22 Page, 3/9/99

12 Presenaion Saus Problem Saemen Grahical Problem Saemen Time Series Analysis Lieraure Innovaive New Aroach Algorihm Phase Sace Mahemaical Formulaion Algorihm Resuls Alicaions Progression of Time Series Engineering Financial /4/23 23 Algorihm Search for he oimal emoral aern ha yields he maximal even values Choose he suor, Q, of he emoral aern Define he even funcion, g Embed he ime series ino a hase sace Define a meric o allow comarison beween he emoral aern and embedded ime series Find he oimal aern cluser Paern neighborhood ha conains he maximum average evenness Geneic Algorihm /4/23 24 Page 2, 3/9/99

13 Grahical Maing X Phase Sace, Q = /4/ Phase Sace wih Even Values g /4/23 26 x Q = Page 3, 3/9/99

14 Mahemaical Formulaion I Non-saionary raining ime series N is he lengh of he series X x,,, N Tesing ime series Y x, R,, S, R N Define Even Funcion Inus mus chosen carefully Relaionshi o emoral aern is imoran Examle g The ercenage change in he ime series beween imes +Q and +Q+ Q is he lengh of he emoral aern xq xq gx xq /4/23 27 Mahemaical Formulaion II Define P Q A Q dimensional real sace Phase sace Embed X ino P, likewise for Y creae y Subses,, of X of cardinaliy Q Chosen by any consisen rule T x x, x,, x,,, N 2 Q Define meric d for P d(, ) Comare he embedded ime series and he emoral aern - emoral aern, a vecor of lengh Q - embedded ime series Q Q /4/23 28 Page 4, 3/9/99

15 Proery Find a emoral aern P and a hreshold ha have he following wo roeries Proery Given he following definiions Tha and he se gx : M is saisically differen from he se rain M rain : d x,,,, N Q M gx rain cm X N NQ Q Mrain X Mrain gx /4/23 29 rain gx :,, N Q Proery 2 Proery 2 Given he following definiions Tha es and ha he se gx : M is saisically differen from he se M es : d y,, R,, S Q M gx es cm Y S Q R Mes SQ 2 R Mes Y gx es gx : R,, S Q /4/23 3 Page 5, 3/9/99

16 Mahemaical Formulaion III Define hreshold In erms of normalized hreshold d Use he mean and sandard deviaion saisics of d(, ) 2 d N Q N Q d dx, N Q N Q d x, Oimizaion formulaion Consrain forces cluser o be greaer han in size maximize, subjec o d d d f,, X, g M rain cm cm N,. rain /4/23 3 d 2 Mrain gx Algorihm Resuls highes even value ime insances Training ime series + aern neighborhood emoral aern radius, /4/23 32 Page 6, 3/9/99

17 Time Series View of Resuls Training ime series x even /4/23 33 Tes Series Phase Sace X nex oins of 2.5 nonsaionary, noneriodic ime series /4/23 34 x Page 7, 3/9/99

18 Tes Series Resuls highes gold value ime insances + aern neighborhood from raining emoral aern from raining /4/23 35 Tesing Phase Sace Even Values and Training Paern Cluser g /4/23 36 Page 8, 3/9/99

19 Time Series View of Tesing Paern,, is from raining rocess x even /4/23 37 Presenaion Saus Problem Saemen Grahical Problem Saemen Time Series Analysis Lieraure Innovaive New Aroach Algorihm Phase Sace Mahemaical Formulaion Algorihm Resuls Alicaions Progression of Time Series Engineering Financial Proosed Work /4/23 38 Page 9, 3/9/99

20 Alicaions Se of rogressively more comleime series Show how each is embedded ino he hase sace Show resuls of algorihm Consan Sinusoidal Chir Random noise Engineering Alicaion Welding Time Series Financial Alicaion Sock Oen Price /4/23 39 Progression of Time Series Choose he suor of he aern, Q = 2, o allow grahical resenaion Define he gold funcion The +Qh value in he ime series Q is he lengh of he aern Q gx B x /4/23 4 Page 2, 3/9/99

21 Consan Train Phase Sace Training orion of ime series Q = /4/23 4 Consan Tes Phase Sace Tesing orion of ime series Q = /4/23 42 Page 2, 3/9/99

22 Linearly Increasing Train Training orion of ime series Q = /4/23 43 Linearly Increasing Tes Tesing orion of ime series Q = /4/23 44 Page 22, 3/9/99

23 Sinusoidal Train Phase Sace Training orion of ime series Q = 2 /4/23 45 Sinusoidal Tes Phase Sace Tesing orion of ime series Q = 2 /4/23 46 Page 23, 3/9/99

24 Chir Train Phase Sace Q = Training orion of ime series /4/23 47 Chir Tes Phase Sace Q = 2 Tesing orion of ime series /4/23 48 Page 24, 3/9/99

25 Noisy Sinusoidal Train Phase Sace Training orion of ime series Q = 2 /4/23 49 Noisy Sinusoidal Tes Phase Sace Tesing orion of ime series Q = 2 /4/23 5 Page 25, 3/9/99

26 Noise Train Phase Sace Uniform Densiy Random Variable Q = /4/23 5 Noise Tes Phase Sace Tesing orion of ime series Q = /4/23 52 Page 26, 3/9/99

27 Consan Paern Cluser cluser Training Paern Cluser cluser Paern cluser alied o es ime series /4/23 53 Linearly Increasing Paern Cluser cluser.5 + Training orion of ime series cluser Paern cluser alied o es ime series /4/23 54 Page 27, 3/9/99

28 Sinusoidal Paern Cluser.5 cluser + - Training orion of ime series cluser Paern cluser alied o es ime series /4/23 55 Time Series View of Sinusoidal Tesing orion of ime series x even Paern,, is from raining rocess /4/23 56 Page 28, 3/9/99

29 Sinusoidal Saisical Tess Runs Tes H : There is no difference beween he mached ime series and he whole ime series. H A : There is significan difference beween he mached ime series and he whole ime series. Using a % robabiliy of Tye I error ( =.). =.87e-8 which means he null hyohesis can easily be rejeced. Difference of wo indeenden means Alhough he wo oulaions are robably deenden, his can be ignored because i makes he saisics more conservaive, i.e., i will end o overesimae he Tye I error. H : M - g(x) =. H A : M - g(x) >. Using a % robabiliy of Tye I error ( =.). = e-43 shows ha he null hyohesis can be rejeced. /4/23 57 Chir Paern Cluser.5 cluser + - Training orion of ime series cluser + - Paern cluser alied o es ime series /4/23 58 Page 29, 3/9/99

30 Time Series View of Chir Tesing orion of ime series x even Paern,, is from raining rocess /4/23 59 Noisy Sinusoidal Paern Cluser Training orion of ime series - cluser cluser Paern cluser alied o es ime series /4/23 6 Page 3, 3/9/99

31 Time Series View of Noisy Sinusoidal Tesing orion of ime series x even Paern,, is from raining rocess /4/23 6 Noise Paern Cluser cluser Training orion of ime series cluser Paern cluser alied o es ime series /4/23 62 Page 3, 3/9/99

32 Welding Alicaion Welding Process Two ieces of meal joined ino one by making a join beween hem Arcing curren is creaed beween welder and meal o be joined Wire is ushed ou of welder Ti of wire mels, forming a drole of meal ha elongaes (sicks ou) unil i releases Goal: Predic when drole will release Can be done by radiional mehods /4/23 63 Welding Daa Se Goal: Predic when a drole will release Four ime series Release (even), khz samling rae (~5 daa oins) Sickou, khz samling rae (~5 daa oins) Curren, 5kHz samling rae (~35, daa oins) Volage, 5kHz samling rae (~35, daa oins) Daa no originally synchronized Firs Pass Used release ime series as even funcion Used sickou as ime series no oo reliable Used a se of hase saces (Q) from dimension o 6 Generaed 6 emoral aerns /4/23 64 Page 32, 3/9/99

33 Welding Time Series sickou deachmen curren volage /4/23 65 Welding Iniial Resuls Training Se Runs = Means = 3.2 x -49 rue osiives: 25 (5.6%), false osiives: 99 (4.4%) rue negaives: 25 (89.3%), false negaives: 7 (.8%) 94.8% accuracy Tesing Se Runs = Means =.34 x -5 rue osiives: 97 (3.5%), false osiives: 52 (.9%) rue negaives: 247 (89.%), false negaives: 55 (2.%) 95.% accuracy /4/23 66 Page 33, 3/9/99

34 Saisical Tess Runs Tes H : There is no difference beween he mached ime series and he whole ime series. H A : There is significan difference beween he mached ime series and he whole ime series. Using a % robabiliy of Tye I error ( =.). Difference of wo indeenden means Alhough he wo oulaions are robably deenden, his can be ignored because i makes he saisics more conservaive, i.e., i will end o overesimae he Tye I error. H : M - g(x) =. H A : M - g(x) >. Using a % robabiliy of Tye I error ( =.). /4/23 67 ICN Time Series ICN is a harmaceuical comany whose sock rades on he NYSE Training Time Series Daily oen rice for s 26 rading days of 99 Tesing Time Series Daily oen rice for he 2nd 27 rading days of 99 Sock rices end o grow exonenially A filer is alied o he ime series B Z z x B /4/23 68 Page 34, 3/9/99

35 ICN Train Phase Sace Filer cluser.2 z +. Q = z /4/23 69 ICN Tes Phase Sace Filer cluser Paern,, is from raining rocess /4/23 7 Page 35, 3/9/99

36 Time Series View of ICN.3.2 z even.2. z Tesing orion of ime series, filered Paern,, is from raining rocess /4/23 7 ICN Iniial Resuls Training Se Using Buy and Hold Sraegy: -26.2% (25 days) Using Temoral Paerns: 5.% (8 days) Runs = 2.2 x -3 Means = 2.42 x -2 Tesing Se Using Buy and Hold Sraegy: -24.% (26 days) Using Temoral Paerns: 7.3% (22 days) Runs = 3.63 x -3 Means = 2.54 x - /4/23 72 Page 36, 3/9/99

37 Presenaion Saus Problem Saemen Grahical Problem Saemen Time Series Analysis Lieraure Innovaive New Aroach Algorihm Phase Sace Mahemaical Formulaion Algorihm Resuls Alicaions Progression of Time Series Engineering Financial /4/23 73 Thank You References Povinelli, R. J., and Feng, X. (998). Temoral Paern Idenificaion of Time Series Daa using Paern Waveles and Geneic Algorihms Proceedings, 8 h Inernaional Conference on Arificial Neural Neworks in Engineering (ANNIE 99), Vol. 8, , November 998. Povinelli and Xin Feng. (999). "Daa Mining of Mulile Non-saionary Time Series" Proceedings, 9h Inernaional Conference on Arificial Neural Neworks in Engineering (ANNIE 99), Vol. 9,. 5-56, November 999. Richard J. Povinelli and Xin Feng. "Imroving Geneic Algorihms Performance By Hashing Finess Values" Proceedings, 9h Inernaional Conference on Arificial Neural Neworks in Engineering (ANNIE 99), Vol. 9, , November 999. Richard J. Povinelli and Xin Feng. "Characerizaion And Predicion of Welding Drole Release Using Time Series Daa Mining" Proceedings, h Inernaional Conference on Arificial Neural Neworks in Engineering (ANNIE2), Vol. 9, , November, 2 Richard J. Povinelli and Xin Feng, A New Temoral Paern Idenificaion Mehod for Characerizaion and Predicion of Comlex Time Series Evens, IEEE Transacions On Knowledge And Daa Engineering, Vol. 5, No. 2, March/Aril 23. /4/23 74 Page 37, 3/9/99

38 THANK YOU! /4/23 76 Page 38, 3/9/99