Model Selection with Cross-Validations and Bootstraps Application to Time Series Prediction with RBFN Models

Transcription

1 Model Selecton wth Cross-Valdatons and Bootstraps Applcaton to Tme Seres Predcton wth RBF Models Amaury Lendasse Vncent Wertz and Mchel Verleysen Unversté catholque de Louvan CESAME av. G. Lemaître 3 B-348 Louvan-la-euve Belgum {lendasse wertz}@auto.ucl.ac.be DCE pl. du Levant 3 B-348 Louvan-la-euve Belgum verleysen@dce.ucl.ac.be Abstract. Ths paper compares several model selecton methods based on expermental estmates of ther generalzaton errors. Experments n the context of nonlnear tme seres predcton by Radal-Bass Functon etworks show the superorty of the bootstrap methodology over classcal cross-datons and leave-one-out. ntroducton onlnear modelng has rased a consderable research effort snce decades ncludng n the feld of artfcal neural networks. onlnear modelng ncludes the necessty to compare models (for example of dfferent complextes) n order to select the best model among several ones. For ths purpose t s necessary to obtan a good approxmaton of the generalzaton error (or expected loss ) of each model (the generalzaton error beng the average error that the model would make on an nfntesze and unknown test set). n ths paper the terms model selecton wll be used when several models must be compared based on estmatons of ther generalzaton errors n order to select one of them. owadays there exst some well-known and wdely used methods able to fulfll ths task. Among them we can cte: œ the hold-out (HO) whch consst n removng data from the ng set and keepng them for daton; HO s also called daton [] or external daton for example n chemometrcs etc. œ the Monte-Carlo cross-daton (or smply cross-daton CV) where several HO daton sets are randomly and sequentally drawn; œ the k-fold cross-daton where the ntal set s randomly splt nto k roughly equal parts each one beng used successvely as a daton set; œ Leave-One-Out (LOO) s a k-fold cross-daton where the sze of the daton set s ; œ the bootstrap [ 3] whch conssts n drawng sets wth replacement from the orgnal sample and usng these sets to estmate the generalzaton errors (boostrap 63 and 63+ are mproved versons of the orgnal bootstrap): O. aynak et al. (Eds.): CA/COP 003 LCS 74 pp Sprnger-Verlag Berln Hedelberg 003

2 574 A. Lendasse V. Wertz and M. Verleysen All these methods of estmatng generalzaton errors have been shown to be asymptotcally roughly equent (see for example [4]) wth some exceptons and lmtatons: œ LOO s less based [5 ] but ts varance s unacceptable [6]; œ cross-daton s consstent (.e. converges to the generalzaton error when the sze of the sample ncreases) f the sze of the daton set grows nfntely faster than the sze of the ng set (whch s counter-ntutve!) [7]; œ cross-daton s almost unbased []; œ bootstrap s downward based but has a very low varance []; œ most recent bootstrap methods (63 and 63+) are almost unbased and also have a low varance []. Gven ths lst of possble model selecton methods and crtera the purpose of ths paper s to compare expermentally these methods n the context of () hghly nonlnear regresson (makng complex model structures unavodable) and () a small number of data ponts or nput-output pars (whch s often the case n real-world applcatons n medum- or hgh-dmensonal spaces). An mportant argument n favor of smple relable model selecton methods s the followng. n the context of nonlnear regresson or model dentfcaton the ng tme (or computatonal complexty) may be far from neglgble. n some cases for example when one tres a sngle model structure and has a large daton set at dsposal ths mght not be a concern. However n most stuatons frst one has to try many dfferent model structures (for example of dfferent complextes dfferent number of neurons n the hdden layers of mult-layer perceptrons etc.); and secondly the use of resamplng methods (CV LOO bootstrap ) s necessary because of the small sze of the orgnal sample at dsposal. Ths mght multply the ng tme by several hundreds or thousands makng computaton tme a real concern even on upto-date powerful machnes. Therefore ths paper wll consder the varous model selecton methods n the context of nonlnear regresson wth relatvely small sample from the pont of vew of performances and computatonal complexty. Ths wll be llustrated on a standard tme seres predcton benchmark the Santa Fe A seres [8]. Ths partcular example has been chosen for llustraton because [8] () the deal regressor s known avodng the supplementary problem of choosng ts sze () f the rules of the Santa Fe competton are followed we have a small set of data (000) (3) the relaton to s hghly nonlnear and (4) makng the hypothess of a statonary process we have a very large test set at dsposal (8000 data) used to obtan a very good approxmaton of the generalzaton error (ndependently from the model selecton method) for comparson purposes. Accordng to our prevous experence [9] we chose to approxmate ths seres by a Radal-Bass Functon network wth 00 Gaussan kernels the ng beng acheved by the method presented n [0]. Model Selecton Methods Ths secton presents the dfferent methods enumerated n the ntroducton. The ntal data used for the ng phase are the pars (x ) wth x representng the d-

3 Model Selecton wth Cross-Valdatons and Bootstraps 575 dmensonal nput vectors and y the scalar outputs. Theses pars form the ng dataset X. Each of the methods below computes an approxmaton of the generalzaton error obtaned wth a model g. The generalzaton error s defned by: E gen ( ( ) y) lm () gx wth (x ) the elements of an nfnte test dataset and g(x ) the approxmaton of y obtaned wth the model g. The selected model wll be the model mnmzng ths estmate of the generalzaton error.. Monte-Carlo Cross-Valdaton The consecutve steps of the Monte-Carlo cross-daton method are:. One randomly draws wthout replacement some elements of the dataset X; these elements form a new ng dataset X. The remanng elements of X form the daton set X. Usually two thrd of the elements of X are used n X and one thrd n X []; ths rule wll be used n the followng.. The tranng of the model g s done usng X and the error E k (g) s calculated accordng to: E k /3 ( gx ( ) y ) /3 () wth ( x ) the elements of X and gx ( ) the approxmaton of y by model g. 3. Steps and are repeated tmes wth as large as possble. The error E k (g) s computed for each repetton k. The average error s defned by: Eˆ gen k Ek. A partcular case of the Monte-Carlo cross-daton method s the Hold-Out method where the number of repettons s equal to. (3). k-fold Cross-Valdaton The k-fold cross-daton method s a varant of the Monte-Carlo cross-daton method. The consecutve steps of ths method are:. One dvdes the elements of the dataset X nto sets of roughly equal sze. The elements of the k th set form the daton set X. The other sets form a new ng dataset X.. The tranng of the model g s done usng X and the error E k (g) s calculated accordng to:

4 576 A. Lendasse V. Wertz and M. Verleysen E k / ( gx ( ) y ) / (4) wth ( x ) the elements of X and gx ( ) the approxmaton of y by model g. 3. Steps and are repeated for k varyng from to. The average error s computed accordng to (3). A partcular case of the k-fold cross-daton method s the Leave-One-Out where s equal to..3 Bootstrap The consecutve steps of the bootstrap method developed by Efron [] are:. n the dataset X one draws randomly samples wth replacement. These new samples form a new ng set X wth the same sze as the orgnal one. The daton set X s the orgnal ng set X. Ths process s called re-samplng.. The tranng of the model g s done usng X and the errors Ek ( g ) and Ek obtaned wth ths model are calculated accordng to the followng equatons: E k ( gx ( ) ) y (5) wth ( x ) the elements of X and gx ( ) the approxmaton of y obtaned by model g; E k ( gx ( ) ) y (6) wth ( x ) the elements of X and gx ( ) the approxmaton of y by model g. 3. The optmsm D k (g) a measure of performance degradaton (for the same model) between a ng and a daton set s computed accordng to: D E E. (7) k k k 4. Steps and 3 are repeated tmes wth as large as possble. The average optmsm Dg ˆ ( ) s computed by: Dg ˆ ( ) k Dk. (8)

5 Model Selecton wth Cross-Valdatons and Bootstraps Once ths average optmsm s computed a new tranng of the model g s done usng the ntal dataset X; the ng error E ( g ) s calculated accordng to: m E m ( gx ( ) ) y (9) wth ( x ) the elements of X and gx ( ) the approxmaton of y obtaned usng the model g. 6. Step 5 s repeated repeated M tmes wth M as large as possble. For each repetton m the ng error Em ( g ) s computed. The apparent error ˆ E ( g ) s defned as the average of errors Em ( g ) over the M repettons. n the case of a lnear model g ths repetton s not necessary; ng of a lnear model gves a unque set of parameters makng all ng errors Em ( g ) equal. Wth nonlnear models ths repetton performs a (Monte-Carlo) estmate of the most probable apparent error obtaned after tranng of g. 7. ow we have an estmate of the apparent error and of the optmsm ther sum gves an estmate of the generalzaton error: ˆ ˆ E E + Dˆ. (0) gen A partcular case of bootstrap method s the bootstrap 63 [3]. n ths method Ê gen (g) s calculated accordng to: ˆ ˆ ˆ 63 E.368 E +.63 D () gen 63 wth Dˆ ( g ) an optmsm estmated only on the data that are not selected durng the re-samplng (see [3] for detals). 3 Radal-Bass Functon etworks and Tme Seres Predcton The nonlnear models that have been chosen n ths paper are the RBF []. These approxmators have the well-known property of unversal approxmaton. Other approxmators as Mult-Layers Perceptrons could have been chosen. However the goal of ths paper s not to compare dfferent famles of approxmators: RBF have been chosen for the smplcty of ther tranng phase. ndeed a possble ng algorthm for RBF [0] conssts n a three-folded strategy (separated computaton of kernel centers wdths and weghts). As a consequence the weghts are smply calculated as the soluton of a lnear system. The problem of tme seres predcton s a common problem n many dfferent felds as fnance electrcty producton hydrology etc. A set of successve ponts (the tme seres) s known; the goal s to predct the next unknown ue. To perform ths task a model s needed that assocates some prevous ues of the tme seres to the next one. Ths model has the followng form:

6 578 A. Lendasse V. Wertz and M. Verleysen yt ( ) g( yt ( )t ( )...t ( n)) () where n s the lag order []. Model g can be lnear or nonlnear. n ths paper the class of models we have chosen s a set of RBF wth dfferent numbers C of Gaussan kernels but wth a gven sze for the regressor. The problem of model selecton s then to choose one of the models n the set.e.: how many Gaussan kernels must be used? 4 Expermental Results We llustrate the methods descrbed n the prevous secton on a standard benchmark n tme-seres predcton. The Santa Fe A tme seres [8] has been chosen manly for the large number of data aable for the tranng stage (000) as well as for the test stage (8000). These two numbers correspond to the rules of the Santa Fe competton as detaled n [8]. Accordng to [8] we also choose n 6. We traned 7 RBFs on the Santa Fe ng dataset for C and 40 respectvely. Ths number of models s kept small to avod a too large computatonal tme; ndeed the goal s not to fnd the best among an nfnte set of models but to compare the performances of the dfferent model selecton methods. Havng a huge (8000 samples) test set makes t possble to have a very good approxmaton of the true generalzaton error: 8000 ( gx ( ) y) ( gx ( ) y) Egen lm. (3) n practce for smlar reasons as those argued n favour of a Monte-Carlo euaton (step 6) of the apparent error ˆ E ( g ) n the bootstrap (use of nonlnear models) a Monte-Carlo method (wth M 00 repettons) s used to euate E ( ) gen g for each model. Then accordng to Fg. (a) the best RBF model has a number C of Gaussan kernels equal to Error 40 Error umber of Gaussan ernels (a) umber of Gaussan ernels Fg.. (a) Estmaton of the true generalzaton error; (b) Estmaton of the generalzaton error by a Monte-Carlo cross-daton method. (b) Fg. (b) shows the estmaton of the generalzaton error by a Monte-Carlo crossdaton method. The optmal number of Gaussan kernel s 80 (another number of

7 Model Selecton wth Cross-Valdatons and Bootstraps 579 kernels that could be chosen -accordng to the parsmony prncple- s 40 correspondng to a local mnmum). Fgure (a) shows the estmaton of the generalzaton error by a Leave-One-Out method. The optmal number of Gaussan kernels s 80. Fg.. (a) Estmaton of the generalzaton error by Leave-One-Out; (b) Estmaton of the generalzaton error by the classcal bootstrap. Fg. (b) shows the estmaton of the generalzaton error by the classcal bootstrap. The optmal number of Gaussan kernels s between 80 and 0. n ths case agan accordng to the parsmony prncple t s advsed to choose C 80. Fgure 3 shows the estmaton of the generalzaton error by the bootstrap 63. The optmal number of Gaussan kernels s 00. Error umber of Gaussan ernels Fg. 3. Estmaton of the generalzaton error by the bootstrap Concluson A number of smlar experments have been conducted wth the purpose of comparng model selecton methods based on emprcal estmatons of the generalzaton error: Monte-Carlo cross-daton Leave-One-Out bootstrap bootstrap 63. They all lead to smlar conclusons: bootstrap methods and n partcular the bootstrap 63 gve the best estmatons of the generalzaton error. Although theoretcal results (brefly summarzed n the ntroducton) show that all these methods are roughly asymptotcally equent ths s no more the case n real applcatons wth lmted number of data. Concernng the computaton tme the number of repettons n the cross-daton and bootstraps may be tuned whch s not the case wth the Leave-One-Out. Ths

8 580 A. Lendasse V. Wertz and M. Verleysen allows n our 000 ng data experments to strongly reduce the computatonal costs assocated wth cross-daton and Bootstraps. However n accordance wth theoretcal results cross-daton has a much hgher varance than bootstraps. Usng the same number of repettons n crossdaton and bootstraps hghlghts the advantages of the bootstraps. Such results have also been publshed n dfferent contexts namely classfcaton [5]. Ths paper extends these results to regresson and tme seres predcton n partcular. Acknowledgements. Mchel Verleysen s Senor research assocate at the Belgan atonal Fund for Scentfc Research (FRS). The work of A. Lendasse and V. Wertz s supported by the nterunversty Attracton Poles (AP) ntated by the Belgan Federal State Mnstry of Scences Technologes and Culture. The scentfc responsblty rests wth the authors. References. Ljung L.: System dentfcaton - Theory for the user nd ed Prentce Hall Efron B. Tbshran R. J. An ntroducton to the bootstrap Chapman & Hall Efron B Tbshran R.: mprovements on cross-daton: The.63+ bootstrap method. J. Amer. Statst. Assoc. (997) 9: Stone M.: An asymptotc equence of choce of model by cross-daton and Akake s crteron J. Royal. Statst. Soc. B ohav R.: A study of Cross-Valdaton and Bootstrap for Accuracy Estmaton and Model Selecton Proc. of the 4th nt. Jont Conf. on A.. Vol. Canada Efron B.: Estmatng the error rate of a predcton rule: mprovements on cross-daton. Journal of Amercan Statstcal Assocaton (983) 78(38): Shao J. Tu D.: The Jackknfe and Bootstrap Sprnger Seres n Statstcs Sprnger- Verlag ew York (995). 8. Wegend A. S. Gershenfeld. A.: Tmes Seres Predcton: Forecastng the future and Understandng the Past Addson-Wesley Publshng Company (994). 9. Smon G. Lendasse A. Wertz V. Verleysen M.: Fast approxmaton of the bootstrap for model selecton submtted to ESA03 Bruges (Belgum) Aprl Benoudjt. Archambeau C. Lendasse A. Lee J. Verleysen M.: Wdth optmzaton of the Gaussan kernels n Radal Bass Functon etworks ESA 00 European Symposum on Artfcal eural etworks Bruges (00) Moody J. Darken C.: Learnng wth localsed receptve felds Proceedngs of the 988 Connectonst Models Summer School San Mateo CA (989).