A Double-Window-based Classification Algorithm for Concept Drifting Data Streams

Transcription

1 00 IEEE Iteratoal Coferece o Graular Computg A Double-Wdow-based Classfcato Algorthm for Cocept Drftg Data Streams Qu Zhu, Xuegag Hu, Yuhog Zhag, Pepe L, Xdog Wu, School of Computer Scece ad Iformato Egeerg, Hefe Uversty of Techology, Cha, Departmet of Computer Scece, Uversty of Vermot, USA, {quzhu.hfut, jsjxhuxg, yuhog.hfut, pepel.hfut}@gmal.com, xwu@cems.uvm.edu Abstract Trackg cocept drfts data streams has recetly become a hot topc data mg. Most of the exstg work s bult o a sgle-wdow-based mechasm to detect cocept drfts. Due to the heret lmtato of the sgle-wdow-based mechasm, t s a challege to hadle dfferet types of drfts. Motvated by ths, a ew classfcato algorthm based o a doublewdow mechasm for hadlg varous cocept drftg data streams (amed DWCDS) s proposed ths paper. I terms of a esemble classfer radom decso trees, a double-wdow-based mechasm s preseted to detect cocept drfts perodcally, ad the model s updated dyamcally to adapt to cocept drfts. Extesve studes o both sythetc ad real-word data demostrate that DWCDS could quckly ad effcetly detect cocept drfts from streamg data, ad the performace o the robustess to ose ad the accuracy of classfcato s also mproved sgfcatly. Keywords: data stream, cocept drft, classfcato. Itroducto Huge amouts of data are emergg the felds of etwork securty, stock aalyss, e-commerce ad weather motorg etc [, ]. These data are called data streams, whch preset ew characterstcs, such as fast, cotuous, hgh-volume, ope-eded, ad cocept drftg [3]. It s challegg for most of tradtoal algorthms for streamg data classfcato. Actually, may classfcato algorthms for cocept drftg data streams have bee proposed. For example, a classfcato algorthm of CVFDT [4] s based o a sgle tree, utlzg sgle wdow of tme ad alteratve sub-trees mechasms to hadle cocept drfts; several classfcato algorthms lke UFFT [5] ad MSRT [6] are based o esemble classfers for cocept drftg data streams. However, algorthms of CVFDT ad UFFT adopt a sgle sldg wdow mechasm to detect cocept drfts the local structure of the classfer. Ths mechasm has the followg lmtato: f the sze of the wdow s large, the model has slower adaptato to a ew target, ad f the value of the wdow s small, the staces are suffcet for learg [7]. The algorthm of MSRT takes two sldg wdows for trag ad testg, but t s actually a dyamc mechasm of stll a sgle wdow. It s also dffcult to determe the wdow s rate of dyamc chage. To overcome the problem metoed above, ths paper, we propose a ew classfcato algorthm for data streams usg a double wdow mechasm (called DWCDS). It s based o the chages of the orgal data dstrbuto the wdow to detect cocept drfts. Correspodgly, the wdow szes are adjusted dyamcally to ehace the adaptablty to cocept drfts. Expermets show that DWCDS performs better o the cocept drft detecto, the ablty of robustess to ose ad the accuracy of classfcato compared to other state-of-the-art algorthms. The rest of ths paper s orgazed as follows. Our Double-Wdow-based Classfcato algorthm for cocept drftg Data Streams (DWCDS) s descrbed Secto. Secto 3 provdes our expermetal study ad Secto 4 summarzes our results ad future work.. The DWCDS algorthm. Algorthm descrpto Our algorthm to be proposed ths secto s based o a double-wdow mechasm to hadle cocept drftg data streams. There are three steps. At the frst step, we geerate N-basc classfers usg the data a N-cell sldg wdow SW (each classfer /0 $ IEEE DOI 0.09/GrC

2 cossts of K-radom decso trees). The, whe SW s full, we check the dstrbuto chages of streamg data the wdow to detect cocept drfts. If a cocept drft occurs, we wll take some operatos to update the model of DWCDS. Fally, we classfy the test data a votg mechasm usg a esemble classfer CT. The ma process of DWCDS s as follows. Iput: Trag set DSTR; Test set DSTE; Attrbute set A; Maxmum heght of trees h 0 ; Basc classfer cout N; Capacty of each basc classfer K; Sldg wdow SW; Mmum threshold of wdow MSW; Maxmum threshold of wdow MaxSW; Basc wdow w; Mmum umber of splt staces m ; Coeffcet of drft warg ; Coeffcet of drft ; Output: Error rate of classfcato Procedure: DWCDS {DSTR, DSTE, A, h 0, N, K, SW, MSW, MaxSW, w, m,, } { for (=; <N; ++) Geerate K-radom decso trees as a basc classfer CT usg the data SW; whle (a ew stace arrves) { f (SW s full) Detect cocept drfts usg the data dstrbuto chages of curret streamg data the sldg wdow ad a basc wdow; f (a cocept drft occurs) { Delete the classfer CT wth the worst performace o classfcato from CT; Buld a ew basc classfer usg the data that has a ew cocept SW ad put t to CT; Adjust the sze of the sldg wdow; }} for (each test stace DSTE ) Classfy t a votg mechasm usg esemble classfer {CT }; retur the error rate of classfcato; } The growg strategy of trees s as follows. A dscrete attrbutes should ot be chose aga a decso path of the tree whle umercal attrbutes ca be chose multple tmes. If a umercal attrbute s selected, we wll mata the attrbute formato of staces ths ode ad dscretze the umercal attrbute wth the method MSRT. If the umber of staces reaches the value of m, we wll select a dex of the dscretzato tervals radomly ad the set the average value of staces ths terval to a cut-pot. If a dscrete attrbute s chose, we wll geerate a chld brach wth the value of ths attrbute ad a ull poter. Ths geerates ew braches whe the ew values of the attrbute come. I ths paper, we use the methods of Majorty Class ad Naïve Bayes. Majorty Class s sutable for ay databases whle Naïve Bayes s ot sutable for databases that have strog depedeces amog data attrbutes, but the accuracy of classfcato ad the at-ose performace Majorty Class are ot as good as Naïve Bayes. Therefore, we choose dfferet methods to classfy accordg to the characterstcs of each database our expermets..3 A double-wdow-based mechasm for cocept drftg detecto A ew mechasm of double-wdow proposed ths paper ams to detect cocept drfts by testg the dstrbuto chages of streamg data perodcally. Double-wdow s a sldg wdow composed of several basc wdows as show Fgure, deoted SW { w, w,, w, w }. as. Basc classfcato model Because radom decso trees have strog ose resstace ad perform better o the overheads of space ad tme [0, ], our algorthm of DWCDS selects ths model as the basc classfers. I our model, we geerate N-esemble classfers cremetally ad each classfer cotas K-radom decso trees wth the maxmum heght of h 0. Due to the fewer attrbute dmeso several databases, we defe the umber of radom decso trees each basc classfer Attrs / as K m{ C Attrs,0} to avod geeratg trees redudatly ad to mprove the performace o the overhead of space ad tme, where refers to cel(), Attrs specfes the total dmesos of attrbutes a gve database, ad Attrs / [, 3] meas the maxmum heght h 0 of trees. Fgure : A double-wdow The detecto mechasm our algorthm also depeds o the chage of error rates a sldg wdow. If the data dstrbuto chages, the exstg model wll o loger ft the curret data dstrbuto ad the error rate of classfcato wll crease; f the data dstrbuto does ot chage, as the umber of staces creases, the error rate of classfcato wll decrease [8, 4]. I addto, to avod the mpact from osy data, the Beroull dstrbuto method used [5] s troduced ad dfferet thresholds are adopted to dstgush cocept drfts from ose. The detals of Beroull dstrbuto are descrbed below. For a set of examples the error s a radom varable from Beroull trals. The correspodg error rate s p error / ad 640

3 the stadard devato s s p*( p)/, where s the total umber of staces ad the error meas the umber of msclassfed staces. Now, we gve the detals of cocept drftg detecto as follows. Frst of all, calculate the error rate p ad the stadard devato s of staces the basc wdow w ( the curret sldg wdow SW ); the cosder w as a ew cocept ad search drftg pots from back to frot as show Fgure. Because the cocept drfts may occur wth the sldg wdow or the frst basc wdow w, two cases are take to accout respectvely: Fgure -(a): Drft wth the sldg wdow wdow, therefore, the search strategy from frot to back or from back to frot has o devato the cocept drftg detecto. I ths paper, we select the latter as the search strategy. Also, the dvso ths double-wdow mechasm could solve the problem that the larger sze of a sgle-wdow has slower adaptato to a ew target fucto. If the value of a sgle-wdow s small ad the staces are suffcet, there wll be the problem of suffcet learg. The mergg faclty the double-wdow mechasm could solve ths problem. I addto, cosderg the ope-eded characterstc of data streams, we remove a basc classfer f t performs worst o the classfcato the esemble classfer CT, ad create a ew classfer usg the merged data after the drft pot (ad load t to CT). Meawhle, we adjust the wdow sze dyamcally accordg to the detecto results ad specfy the maxmum ad mmum bouds to lmt the umber of basc wdows. Ths s because a large wdow sze s sutable for gradual drfts whle a small wdow sze s adapt to abrupt drfts [5]..4 Complextes of tme ad space Fgure -(b): Drft at the begg of sldg wdow ) As show Fgure -(a), a cocept drft occurs wth the sldg wdow, ad our algorthm begs to search the suspcous drftg pot forward from w. If the detecto values o wj ( j ) meet the equalty (), we beleve that the drft warg status begs from wj, the merge w ~ w j ad wj ~ w, ad calculate ther ew error rates P ad P ', ad the stadard devato S of w ~ w j. If they satsfy the equalty (), we cosder that the cocept drft s occurrg; otherwse, t s possbly mpacted by ose (where ad are costat coeffcet ad ). p s p () * P * S P' () ) As show Fgure -(b), a cocept drft occurs at the begg of the sldg wdow. To reduce the possblty of mssg cocepts ths case, we tegrate w ~ w as SW ad calculate the error rate P ' correspodgly. I terms of the values of P ' SW, ad P ad S SW, we further detect whether cocept drfts occur. If the codto equalty () s met, we cosder that a cocept drft s occurrg; otherwse, there s o cocept drft. Our algorthm s bult o the followg assumpto that the cocept drft occurs oly oce each sldg Tme complexty: regardg the tme cosumpto of DWCDS, t cossts of the cremetal geerato of trees, the detecto o cocept drfts ad the classfcato o test data. The tme cost cremetal geerato of trees s O( co t ds ), where Ot () refers to the average tme cost a splt test at odes wth cotuous splt attrbutes whle t s oly O () at a dscrete ode; the umber of odes wth cotuous splt-attrbutes s co ad the cout of odes wth dscrete splt-attrbutes s ds o average. The tme cost cocept drftg detecto s O ( ), where specfes the umber of basc wdows w a sldg wdow. Wth respect to the tme complexty of classfcato, f the Majorty Class method s selected, the tme complexty s O( classes ), whle the tme complexty of Naïve Bayes s O( leaf classes Attrs ), where classes meas the umber of class labels of the test staces classfed Naïve Bayes or Majorty Class; leaf dcates the cout of staces at the leaf ode ad Attrs specfes the total dmesos of attrbutes a gve database. Space complexty: the space cost maly depeds o the costructo of classfers ad the storage of the sldg wdow. Amog them, the space cost of classfers s O d v classes l Attrs v classes, where d ad l refer to the couts of o-leaves 64

4 ad leaf odes respectvely ad v dcates the maxmum umber of attrbute-values a attrbute. The space cosumpto of the sldg wdow s Ok ( w e ), where k specfes the umber of basc wdows w the sldg wdow, w meas the sze of each basc wdow, ad Oe () refers to the storage space of each stace. 3. Expermetal evaluatos To valdate the effcecy ad effectveess of our proposed drft detecto mechasm DWCDS, a great umber of expermets are coducted o bechmark cocept drftg databases ad real-world databases to compare DWCDS wth SWCDS (whch s based o the same esemble classfer as DWCDS, but adopts a sgle-wdow-based mechasm for cocept drftg detecto). Meawhle, we also compare DWCDS wth the state-of-the-art algorthms of MSRT, CVFDT, HT-DDM [8] (a sgle Hoeffdg Tree wth Drft Detecto Method) ad HT-EDDM [9] (a sgle Hoeffdg Tree wth Early Drft Detecto Method) o the robustess to ose ad the predctve accuracy. The databases used our expermets are from two bechmark databases SEA (whch has four cocepts, ad each test set correspods to a cocept, deoted as SEA-~SEA-4 respectvely) ad HyperPlae, two real databases KDDCup99 [7] ad Yahoo shoppg data [8], ad a osy database LED. All the data of SEA, HyperPlae ad LED are geerated by the data geerator from MOA [6] (a tool for Massve Ole Aalyss). Values of parameters DWCDS are gve below: / h0 Attrs /, N=0, K m{ C, 0}, the tal value of sldg wdow SW=k, MSW=0.4k, MaxSW=k, w=0.k, m =0.k, =.5, ad =3, where the values of h 0, N ad m follow the expermetal results [, 3]; w s specfed correspodg to our expermetal results; ad are chose accordg to the values of parameters used [5] ad the coclusos of our expermets. All expermets are performed o a P4,.5GHz PC wth G ma memory, rug Wdows XP Professoal. The algorthms of HT-DDM, HT- EDDM are coded Java from the ope source of MOA whle others are developed C Detecto o cocept drft Some deotatos used ths secto are frst troduced below. Each database ame s composed of the ame of the database, the sze of the trag set, the sze of the test set, the type of database (C: cotuous, D: dscrete, CD: hybrd) ad the umber of attrbute dmesos. Detecto refers to the umber of cocept drftg detectos. False Alarm specfes the umber of false alarms occurrg the drftg detecto. Mssg meas the umber of cocepts mssg the drftg detecto. Fgures 3~5 preset the cases of trackg cocept drfts by DWCDS o the three databases of SEA, HyperPlae ad KDDCup99, ad ther relevat statstcs are gve Table. We ca see that our algorthm could track cocept drfts, though there are several mssg cocepts ad false alarms occurrg. For example, SEA (cotag 50k-szed staces wth 0% ose), all of the three real cocept drfts are correctly detected, but t also has three false alarms. Ths s due to the osy data. I HyperPlae (cotag 00k-szed staces wth 5% ose), DWCDS correctly detects e cocept drfts, but t also has three msses ad two false alarms. Ths s because of the detecto threshold whch s selected to trade off the effect dfferet types of cocept drftg detecto results. I KDDCup99 (cotag 490kszed staces), DWCDS correctly detect twety-three cocept drfts, but has thrtee mssg cocepts ad e false alarms. Ths s due to the skewed dstrbuto of class labels the KDDCup99 database. Fgure 3: Drft trackg o SEA Fgure 4: Drft trackg o HyperPlae To verfy the valdty of the double-wdow mechasm cocept drftg detecto, we have compared DWCDS wth SWCDS whose wdow sze s set to 0.k, k ad k respectvely, ad the relevat statstcs are show Table. We ca fd that the capablty of SWCDS cocept drftg detecto s worse tha DWCDS. Ths s because the sze of the wdow SWCDS s dffcult to determe. A small wdow sze s easy to cause false alarms. For example, SEA, f the sze of the wdow s set to 0.k, the umber of false alarms s up to thrty-three. But a large wdow sze would crease the probablty of Mssg. 64

5 Fgure 5: Drft trackg o KDDCup99 Table Statstcs of drftg detecto Algorthm #Detecto #False Alarm #Mss -g SEA wth 3 cocept drfts SWCDS-0.k Bayes SWCDS-k SWCDS-k 5 DWCDS HyperPlae wth cocept drfts SWCDS-0.k Max SWCDS-k SWCDS-k 40 7 DWCDS 47 3 KDDCup99 wth 36 cocept drfts SWCDS-0.k Bayes SWCDS-k SWCDS-k DWCDS For example, f the sze of the wdow s set to k, oe cocept s mssed. However, eve f we select a optmal wdow sze, the rate of false alarms SWCDS s stll hgher tha our algorthm of DWCDS. For example, f the sze of the wdow s set to k, the cout of false alarms SWCDS s up to fve whle t s oly three DWCDS. These results cofrm the better performace of DWCDS drftg detecto. As the sgle-wdow-based algorthms of CVFDT, HT-DDM ad HT-EDDM ad MSRT based o two sldg wdows do ot gve specfc drftg detecto results cludg Detecto, False Alarm ad Mssg, we compare them wth DWCDS terms of reslece to ose ad the error rate of classfcato as follows. 3. Robustess Fgure 6 Robustess performace o LED I ths secto, we compare the robustess of all the algorthms o LED ad HyperPlae wth the ose rates varyg from 5% to 30%. The expermetal results show Fgure 6~7 preset that the robustess to ose DWCDS s superor to the other four algorthms. More specfcally, comparso wth the algorthms of MSRT, CVFDT, HT-DDM ad HT- EDDM, the classfcato accuracy o HyperPlae DWCDS s mproved by a rage from 3.5% to 6.9%. Fgure 7: Robustess performace o HyperPlae 3.3 Predctve accuracy I ths secto, we aalyze the accuracy of classfcato DWCDS ad the other four classfcato algorthms o the databases of SEA, KDDCup99 ad Yahoo Shoppg data. Expermetal results show Table reveal that the accuracy of classfcato DWCDS s superor to the others o KDDCup99 ad Yahoo Shoppg data. The mprovemet of accuracy rages from.8% to 35.7% ad from 3.9% to 7.8% respectvely. Meawhle, o the SEA database, the classfcato accuracy of DWCDS s mproved by the rage from.6% to 4.5% compared to MSRT. As compared wth other three algorthms, the predctve accuracy of DWCDS s slghtly worse o the test sets of SEA- ad SEA-3. The reaso s that there are oly three-dmesoal features SEA, ad the heght of trees h 0 s DWCDS ad MSRT. The heght determes the predctve accuracy of models. A lower heght wll result a worse performace o classfcato accuracy. However, f the heght of trees s the same, the classfcato accuracy of models wll be mproved wth the creasg of the umber of decso trees. 4. Coclusos Ths paper preseted a classfcato algorthm for data streams wth a double-wdow mechasm cocept drftg detecto called DWCDS. It s based 643

6 Table Error Rate of Classfcato (%) Database Algorthm DWCDS- Bayes MSRT- Bayes CVFDT HT-DDM HT-EDDM SEA SEA-50k-.5k-C-3 SEA SEA SEA KDDCup99-490K-30K-CD Yahoo shoppg data-84k-8k-cd o radom decso trees, usg the chage of data dstrbuto basc wdows ad the sldg wdow to detect potetal cocept drfts. Expermetal results demostrated that DWCDS has better performace o detecto cocept drfts as compared wth the sglewdow-based method. Meawhle, comparso wth the algorthms of MSRT, CVFDT, HT-EDDM ad HT-EDDM, our algorthm of DWCDS outperforms o the accuracy of classfcato ad the robustess to ose. However, DWCDS stll eeds to create multple decso trees, thus t has a dsadvatage stad-aloe evromets. It s more sutable for mult-cpu ad mult-pc platforms. How to further decrease the rutme ad mprove the teractve votg mechasm the classfcato phase amog multple trees, ad how to model the osy data to dscer cocept drfts from ose are our future work. Ackowledgemets Ths paper s supported by the 973 Program of Cha uder grat 009CB3603, the Natoal Natural Scece Foudato of Cha uder grat , ad the Natural Scece Foudato of Ahu Provce of Cha uder grat Refereces [] L. Golab, ad M.T. Ozsu, Issues data stream maagemet, SIGMOD Record, 003, pp [] Y.Y. Zhu, ad D. Shasha, Effcet elastc burst detecto data streams, Proc. of the 9th ACM SIGKDD Iteratoal Coferece o Kowledge Dscovery ad Data mg, 003, pp [3] A. Tsymbal, The problem of cocept drft: deftos ad related work, Techcal Report TCD-CS-004-5, Computer Scece Departmet, Techcal Report, 004. [4] G. Hulte, L. Specer, ad P. Domgos, Mg tmechagg data streams, Proc. of the 7th ACM SIGKDD Iteratoal Coferece o Kowledge Dscovery ad Data Mg, Sa Fracsco: ACM Press, 00, pp [5] J. Gama, P. Medas, ad P.P. Rodrgues, Cocept drft decso trees learg from data streams, Proc. of the 4th Europea Symposum o Itellget Techologes ad ther mplemetato o Smart Adaptve Systems, 004, pp [6] P. P. L, X.Q. Hu, ad X.D. Wu, Mg coceptdrftg data streams wth multple sem-radom decso trees, Proc. of the 4th Iteratoal Coferece o Advaced Data Mg ad Applcatos, 008, pp [7] D. Wdyatoro, Cocept drft learg ad ts applcato to adaptve formato flterg, PhD thess, Texas A&M Uversty, 003. [8] J. Gama, P. Medas, G. Castllo, ad P.P. Rodrgues, Learg wth drft detecto, SBIA Brazla Symposum o Artfcal Itellgece, 004, pp [9] M. Baea-Garca, J. D. Campo-Avla, R. Fdalgo, A. Bfet, R. Gavalda, ad R. Morales-Bueo, Early drft detecto method, Proc. of the 4th Iteratoal Workshop o Kowledge Dscovery from Data Streams, 006, pp [0] W. Fa, H. Wag, P.S. Yu, ad S. Ma, Is radom model better? O ts accuracy ad effcecy, Proc. of 3rd IEEE Iteratoal Coferece o Data Mg (ICDM 03), 003, pp [] X.G. Hu, P.P. L, X.D. Wu, ad G.Q. Wu, A semradom multple decso-tree algorthm for mg data streams, Joural of Computer Scece ad Techology, 007, (5): pp [] W. Fa, O the optmalty of probablty estmato by radom decso trees, Proc. of the 9th Natoal Coferece o Artfcal Itellgece (AAAI'04), AAAI Press, 004, pp [3] P.P. L, Q.H. Lag, X.D. Wu, ad X.G. Hu, Parameter estmato sem-radom decso tree esemblg o streamg data, Proc. of the 3th Pacfc-Asa Coferece o Kowledge Dscovery ad Data Mg (PAKDD'09), Sprger-Verlag Berl Hedelberg, 009, pp [4] G. Wdmer, ad M. Kubat, Learg the presece of cocept drft ad hdde cotexts, Mache Learg, 996, 3: pp [5] G. Wdmer ad M. Kubat, Learg flexble cocepts from streams of examples: Flora, Proc. of the 0th Europea Coferece o Artfcal Itellgece (ECAI 9), 99, pp [6] G. Holmes, R. Krkby, ad B. Pfahrger, MOA: massve ole aalyss, projecys/moa-datastream, 007. [7] KDDCUP99 data set, ACM Specal Iterest Group o Kowledge Dscovery ad Data Mg, 999. [8] Yahoo! Shoppg Web Servces, yahoo.com/everythg.html, Yahoo! Ic.. 644