A Fusion of Stacking with Dynamic Integration

Transcription

1 A Fuson of Stackng wth Dynamc Integraton all Rooney, Davd Patterson orthern Ireland Knowledge Engneerng Laboratory Faculty of Engneerng, Unversty of Ulster Jordanstown, ewtownabbey, BT37 OQB, U.K. {nf.rooney, Abstract In ths paper we present a novel method that fuses the ensemble meta-technques of Stackng and Dynamc Integraton () for regresson problems, wthout addng any major computatonal overhead. The ntenton of the technque s to beneft from the varyng performance of Stackng and for dfferent data sets, n order to provde a more robust technque. We detal an emprcal analyss of the technque referred to as weghted Meta Combner (wmetacomb) and compare ts performance to Stackng and the technque of Dynamc Weghtng wth Selecton. The emprcal analyss conssted of four sets of experments where each experment recorded the cross-fold evaluaton of each technque for a large number of dverse data sets, where each base model s created usng random feature selecton and the same base learnng algorthm. Each experment dffered n terms of the latter base learnng algorthm used. We demonstrate that for each evaluaton, wmeta- Comb was able to outperform and Stackng for each experment and as such fuses the two underlyng mechansms successfully. 1 Introducton The objectve of ensemble learnng s to ntegrate a number of base learnng models n an ensemble so that the generalzaton performance of the ensemble s better than any of the ndvdual base learnng models. If the base learnng models are created usng the same learnng algorthm, the ensemble learnng schema s homogeneous otherwse t s heterogeneous. Clearly n the former case, t s mportant that the base learnng models are not dentcal, and theoretcally t has been shown that for such an ensemble to be effectve, t s mportant that the base learnng models are suffcently accurate and dverse n ther predctons [Krogh & Vedelsby, 1995], where dversty n terms of regresson s usually measured by the ambguty or varance n ther predctons. To acheve ths n terms of homogeneous learnng, methods exst that ether manpulate, for each model, the tranng data through nstance or feature samplng methods or manpulate the learnng parameters of the learnng algorthm. Brown et al.[2004] provde a recent survey of such ensemble generaton methods. Havng generated an approprate set of dverse and suffcently accurate models, an ensemble ntegraton method s requred to create a successful ensemble. One well known approach to ensemble ntegraton s to combne the models usng a learnng model tself. Ths process, referred to as Stackng [Wolpert, 1992], forms a meta-model created from a meta-data set. The meta-data set s formed from the predctons of the base models and the output varable of each tranng nstance. As a consequence, for each tranng nstance I =< xy, > belongng to tranng set D, metanstance MI =< f ˆ ˆ 1( x),.. f ( x), s formed, where f ˆ ( x) s the predcton of base model, = 1.. where s the sze of the ensemble. To ensure that the predctons are unbased, the tranng data s dvded by a J -fold cross-valdaton process nto J subsets, so that for each teraton of a crossvaldaton process, one of the subsets s removed from the tranng data and the models are re-traned on the remanng data. Predctons are then made by each base model on the held out sub-set. The meta-model s traned on the accumulated meta-data after the completon of the cross valdaton process and each of the base models s traned on the whole tranng data. Breman[1996b] showed the successful applcaton of ths method for regresson problems usng a lnear regresson method for the meta-model where the coeffcents of the lnear regresson model were constraned to be non-negatve. However there s no specfc requrement to use lnear regresson as the meta-model [Haste et al., 2001] and other lnear algorthms have been used partcularly n the area of classfcaton [Seewald, 2002, Džerosk, & Ženko, 2004]. A technque whch appears at frst very smlar n scope to Stackng (and can be seen as a varant thereof) s that proposed by Tsymbal et al.[2003] referred to as Dynamc Integraton. Ths technque also runs a J-fold cross valdaton 2844

2 process for each base model, however the process by whch t creates and uses meta-data s dfferent. Each metanstance conssts of the followng format < x, Err1 ( x),.., Err ( x), where Err ( x ) s the predcton error made by each base model for each tranng nstance < x,. Rather than formng a meta-model based on the derved meta-data, Dynamc Integraton uses a k- nearest neghbour (k-) lazy model to fnd the nearest neghbours based on the orgnal nstance feature space (whch s a subspace n the meta-data) for a gven test nstance. It then determnes the total tranng error of the base models recorded for ths meta-nearest neghbour set. The level of error s used to allow the dynamc selecton of one of the base models or an approprate weghted combnaton of all or a select number of base models to make a predcton for the gven test nstance. Tsymbal et al. [2003] descrbes three Dynamc Integraton technques for classfcaton whch have been adapted for regresson problems [Rooney et al., 2004]. In ths paper we consder one varant of these technques referred to as Dynamc Weghtng wth Selecton (DWS) whch apples a weghtng combnaton to the base models based on ther localzed error, to make a predcton, but excludes frst those base models that are deemed too naccurate to be added to the weghted combnaton. A method of njectng dversty nto a homogeneous ensemble s to use random subspacng [Ho, 1998a, 1998b]. Random subspacng s based on the process whereby each base model s bult wth data wth randomly subspaced features. Tsymbal et al. [2003] revsed ths mechansm so that a varable length of features are randomly subspaced. They shown ths method to be effectve n creatng ensembles that were ntegrated usng methods for classfcaton. It has been shown emprcally that Stackng and Dynamc Integraton based on ntegratng models, generated usng random subspacng, are suffcently dfferent from each other n that they can gve varyng generalzaton performance on the same data sets [Rooney et al., 2004]. The man computatonal requrement n both Stackng and Dynamc Integraton, s the use of cross-valdaton of the base models to form meta-data. It requres very lttle addtonal computatonal overhead to propose an ensemble meta-learner that does both n one cross valdaton of the ensemble base models and as a consequence, s able to create both a Stackng meta-model and a Dynamc Integraton meta-model, wth the requred knowledge for the ensemble meta-learner to use both models to form predctons for test nstances. The focus of ths paper s on the formulaton and evaluaton of such an ensemble meta-learner, referred to as weghted Meta- Combner (wmetacomb). We nvestgated whether wmeta- Comb wll on average outperform both Stackng for regresson () and DWS for a range of data sets and a range of base learnng algorthms used to generate homogeneous randomly subspaced models. 2 Methodology In ths secton, we descrbe n detal the process of wmeta- Comb. The tranng phase of wmetacomb s shown n Fgure 1.. Algorthm : wmetacomb: Tranng Phase Input: D Output: f ˆ, f ˆ, f ˆ, = 1.. (* step 1 *) partton D nto J fold parttons D = 1.. ntalse meta-data set = ntalse meta-data set = For = 1..J-2 Dtran = D\ D D = D test buld learnng models f ˆ, = 1.. usng Dtran For each nstance < x, n Dtest Form meta-nstance MI : < f ˆ ˆ 1( x),.., f ( x), EndFor Endfor J Form meta-nstance MI : < x, Err1 ( x),.., Err ( x), Add Add MI MI to to Buld meta-model fˆ usng the meta-data set Buld meta-model ˆ f (* step 2 *) =J D = D test usng the meta-data set Determne TotalErr and TotalErr usng Dtest (* step 3 *) = J-1 Dtran = D\ D Dtest = D buld learnng models f ˆ, = 1.. usng Dtran For each nstance < x, n Dtest Form meta-nstance MI : < f ˆ ˆ 1( x),.., f ( x), Form meta-nstance MI : < x, Err1 ( x),.., Err ( x), Add MI to Add MI to EndFor Retran the f, = 1.. on D ˆ Buld meta-model fˆ usng the meta-data set Buld meta-model fˆ usng the meta-data set Fgure 1. Tranng Phase of wmetacomb The frst step n the tranng phase of wmetacomb conssts of buldng and meta-models on J-1 folds of the tranng data (step 1). The second step nvolves testng the meta-models usng the remanng tranng fold, as an ndcator of ther potental generalzaton performance. It s mportant to do ths as even though models are not created 2845

3 usng the complete meta-data, they are tested usng data that they are not traned on, n order to gve a relable estmate of ther test performance. The performance of the meta-models s recorded by ther total absolute error, TotalError and TotalError wthn step 2. The meta-models are then rebult usng the entre meta-data completed by step 3. The computatonal overhead of wmetacomb n comparson to by tself centres only on the cost of tranng the addtonal meta-model. However as the addtonal meta-model s based on a lazy nearest neghbour learner, ths cost s trval. The predcton made by wmetacomb s made by a smple weghted combnaton of ts meta-models based on ther total error performance determned n step 2. Denote TotalErr =< TotalError, TotalError >. The ndvdual weght for each meta-model s determned by fndng the normalzed error for each meta-model: normerror = TotalError / TotalError = 1..2 A normalzed weghtng for each meta-model s gven by: mw normerror T e = (1) norm _ mw mw = (2) mw Equaton 1 s nfluenced by a weghtng mechansm that Wolpert & Macready [1996] proposed for combnng base models generated by stochastc process such as used n Baggng [Breman,1996a]. The weghtng parameter T determnes how much relatve nfluence to gve to each meta-model based on ther normalzed error e.g. suppose normerror 1 = 0.4 and normerror 2 = 0.6, then Table 1 gves the normalzed weght values for dfferent values of T. Ths ndcates that a low value of T wll gve large mbalance to the weghtng, whereas a hgh value of T almost equally weghts the metalearners regardless of ther dfference n normalzed errors. T norm _ mw 1 norm _ mw Table 1. The effect of the weghtng parameter Based on these consderatons, we appled the followng heurstc functon for the settng of T, whch ncreases the mbalance n the weghtng dependent on how great the normalzed errors dffer. T normerror normerror = (3) wmetacomb forms a predcton for a test nstance x by the followng combnaton of the predctons made by the metamodels: _ * ˆ ( ) _. ˆ norm mw f x + norm mw f ( x) Expermental Evaluaton and Analyss In these experments, we nvestgated the performance of wmetacomb n comparson to and DWS. The work was carred out based on approprate extensons to the WEKA envronment, such as the mplementaton of DWS. The ensemble sets conssted of 25 base homogeneous models where the data for each base model was randomly subspaced [Tsymbal et al., 2003]. ote that random subspacng was a pseudo-random process so that a feature sub-set generated for each model n an ensemble set s the same for all ensemble sets created usng dfferent base learnng algorthms and the same data set. The meta-learner for and the Stackng component wthn wmetacomb was based on the model tree M5 [Qunlan, 1992], as ths has a larger hypothess space than Lnear Regresson. The number of nearest neghbours for DWS k- meta-learner was set to 15. The performance of DWS can be affected by the sze of k for partcular data sets, but for ths partcular choce of data sets, prevous emprcal analyss had shown that the value of 15, gave overall a relatvely strong performance. The value of J durng the tranng phase of each metatechnque was set to 10. In order to evaluate the performance of the dfferent ensemble technques, we chose 30 data sets avalable from the WEKA envronment [Wtten & Frank, 1999]. These data sets represent a number of synthetc and real-world problems from a varety of dfferent domans, have a varyng range n ther attrbute szes and many contan a mxture of both dscrete and contnuous attrbutes. Data sets whch had more than 500 nstances, were sampled wthout replacement to reduce the sze of the data set to a maxmum of 500 nstances, n order to make the evaluaton computatonally tractable. Any mssng values n the data set were replaced usng a mean or modal technque. The characterstcs of data sets are shown n Table 2. We carred out a set of four experments, where each experment dffered purely n the base learnng algorthm deployed to generate base models n the ensemble. Each experment calculated the relatve root mean squared error (RRMSE) [Setono et al., 2002] of each ensemble technque for each data set based on a 10 fold cross valdaton measure. 2846

4 Data set Orgnal Data set sze Total Data set sze n experments umber of nput Attrbutes Con- Ds- tnuous crete abalone autohorse autompg autoprce auto bank8fm bank32nh bodyfat breasttumor cal_housng cloud cpu cpu_act cpu_small echomonths elevators fshcatch fredman housng house_8l house_16h lowbwt machne_cpu polluton pyrm sensory servo sleep strke veteran Table 2. The data sets characterstcs The RRMSE s a percentage measure. By way of comparson of the ensemble approaches n general, we also determned the RRMSE for a sngle model bult on the whole unsampled data, usng the same learnng algorthm as used n the ensemble base models. We repeated ths evaluaton four tmes, each tme usng a dfferent learnng algorthm to generate base models. The choce of learnng algorthms was based on a choce of smple and effcent methods n order to make the study computatonally tractable, but also to have representatve range of methods wth dfferent learnng hypothess spaces n order to determne whether wmetacomb was a robust method ndependent of the base learnng algorthm deployed. The base learnng algorthms used for the four experments were 5 nearest neghbours (5- ), Lnear Regresson (LR), Locally weghted regresson [Atkeson et al., 1997] and the model tree learner M5. The results of the evaluaton were assessed by performng a 2 taled pared t test (p=0.05) on the cross fold RRMSE for each technque n comparson to each other for each data set. The results of the comparson between one technque A and another B were then summarsed n the form wns:losses:tes (ASD) where wns s a count of the number of data sets where technque A outperformed technque B sgnfcantly, losses s a count of the number of data sets where technque B outperformed technque A sgnfcantly and tes a count where there was no sgnfcant dfference. The sgnfcance gan s the dfference n the number of wns and the number of losses. If ths zero or less, no gan was shown. If the gan s less than 3 ths s ndcatve of only a slght mprovement. We determned the average sgnfcance dfference (ASD) between the RRMSE of the two technques, averaged over wns + losses data sets only where a sgnfcant dfference was shown. If technque A outperformed B as shown by the sgnfcance rato, the ASD gave us a percentage measure of the degree by whch A outperformed B (f technque A was outperformed by B, then the ASD value s negatve). The ASD by tself of course has no drect ndcaton of the performance of the technques and s only a supplementary measure. A large ASD does not ndcate that technque A performed consderably better than B f the dfference n the number of wns to losses was small. Conversely even f the number of wns was much larger than the number of losses but the ASD was small, ths reduces the mpact of the result consderably. Ths secton s dvded nto a dscusson of the summary of results for each ndvdual base learnng algorthm then a dscusson of the overall results. 3.1 Base Learner: k- Table 3 shows a summary of the results for the evaluaton when the base learnng algorthm was 5-. Clearly all the ensemble technques strongly outperformed sngle 5- whereby n terms of sgnfcance alone, wmetacomb shown the largest mprovement wth a gan of 22. Although showed fewer data sets wth sgnfcant mprovement, t had a greater ASD than wmetacomb. wmetacomb outperformed DWS wth a gan of 7, and showed a small mprovement over, wth a gan of 3. showed a sgnfcance gan of only 1 over DWS. Technque 5- DWS wmetacomb 5-0:11:19 (-16.07) 0:20:10 (-12.28) 0:22:8 (-14.47) 11:0:19 (16.07) 3:2:25 (3.82) 0:3:27 (-6.8) DWS 20:0:10 (12.28) 2:3:25 (-3.82) 1:8:21 (-5.28) wmetacomb 22:0:8 (14.47) 3:0:27 (6.8) 8:1:21 (5.28) Table 3. Comparson of methods when base learnng algorthm was

5 3.2 Base Learner: LR Table 4 shows that all ensembles technques outperformed sngle LR, wth wmetacomb showng the largest mprovement wth a gan of 13 and an ASD of wmetacomb outperformed and DWS wth a larger mprovement shown n comparson to DWS than to both n terms of sgnfcance and ASD. outperformed DWS wth a sgnfcance gan of 7 and ASD of 5.1. Technque LR DWS wmetacomb LR 2:10:18 (-11.38) 1:11:18 (-5.56) 0:13:17 (-11.99) 10:2:18 (11.38) 11:4:15 (5.1) 0:5:25 (-5.58) DWS 11:1:18 (5.56) 4:11:15 (-5.1) 3:13:14 (-6.02) wmetacomb 13:0:17 (11.99) 5:0:25 (5.58) 13:3:14 (6.02) Table 4. Comparson of methods when base learnng algorthm was LR 3.3 Base Learner: LWR Table 5 shows that all ensembles technques outperformed sngle LWR, wth wmetacomb showng the largest mprovement wth a sgnfcance gan of 16. wmetacomb outperformed and DWS wth a larger mprovement shown over DWS than n terms of sgnfcance and ASD. outperformed DWS wth a sgnfcance gan of 4 and an ASD of Technque LWR DWS wmetacomb LWR 1:9:20 (-13.81) 0:16:14 (-12.28) 0:16:14 (-12.28) 9:1:20 (13.81) 6:2:22 (6.24) 0:5:25 (-10.48) DWS 16:0:14 (12.28) 2:6:22 (-6.24) 0:9:21 (-8.28) wmetacomb 16:0:14 (15.91) 5:0:25 (10.48) 9:0:21 (8.28) Table 5. Comparson of methods when base learnng algorthm was LWR Table 5 shows that all ensembles technques outperformed sngle LWR, wth wmetacomb showng the largest mprovement wth a sgnfcance gan of 16. wmetacomb outperformed and DWS wth a larger mprovement shown over DWS than n terms of sgnfcance and ASD. outperformed DWS wth a sgnfcance gan of 4 and ASD of Base Learner: M5 Technque M5 DWS wmetacomb M5 3:1:26 (4.4) 1:6:23 (-3.45) 0:8:22 (-4.51) 1:3:26 (-4.4) 2:6:22 (-2.62) 0:11:19 (-4.48) DWS 6:1:23 (3.45) 6:2:22 (2.62) 0:6:24 (-3.53) wmetacomb 8:0:22 (4.51) 11:0:19 (4.48) 6:0:24 (3.53) Table 6. Comparson of methods when base learnng algorthm was M5 Table 6 shows a much reduced performance for the ensembles n comparson to the sngle model, when the base learnng algorthm was M5. In fact, showed no mprovement over M5, and wmetacomb and DWS although they mproved on M5 n terms of sgnfcance dd not show an ASD as large as for the prevous learnng algorthms. The reduced performance wth M5 s ndcatve that n the case of certan learnng algorthms, random subspacng by tself s not able to generate suffcently accurate base models, and needs to be enhanced usng search approaches that mprove the qualty of ensemble members such as consdered n [Tsymbal et al., 2005]. wmetacomb showed mprovements over and to a lesser degree both n terms of sgnfcance and ASD values. Overall, t can be seen that wmetacomb gave the strongest performance of the ensembles for 3 out of 4 base learnng algorthms (5-, LR, M5) n comparson to the sngle model, n terms of ts sgnfcance gan, and ted n comparson to DWS for LWR. For 3 out of the 4 learnng algorthms (LR, LWR, M5), wmetacomb also had the hghest ASD compared to the sngle model. wmetacomb outperformed for all learnng algorthms. The largest sgnfcant mprovement over was recorded wth M5 wth a sgnfcance gan of 11, and the largest ASD n comparson to was recorded wth LWR of The lowest sgnfcant mprovement over was recorded wth 5-, LR, and LWR wth a sgnfcant gan of 5, and lowest ASD of 4.48 wth M5. wmetacomb outperformed DWS for all four base learnng algorthms. The degree of mprovement was larger n comparson to DWS than. The largest sgnfcant mprovement over DWS was shown wth LR wth a sgnfcance gan of 10 and the largest ASD of 8.28 wth LWR. The lowest mprovement was shown wth M5 both n terms of gan and ASD whch were 6 and 3.53 respectvely. In effect, wmetacomb was able to fuse effectvely the overlappng expertse of the two meta-technques. Ths was seen to be case ndependent of the learnng algorthm employed. In addton, wmetacomb provded beneft whether overall for a gven experment, was stronger than DWS or not (as 2848

6 was the case wth LR and LWR, but not M5), ndcatng the general robustness of the technque. 4 Conclusons We have presented a technque called wmetacomb that merges the technque of Stackng for regresson and Dynamc ntegraton for regresson, and showed that t was successfully able to beneft from the ndvdual metatechnques often complementary expertse for dfferent data sets. Ths was demonstrated based on 4 sets of experments each usng dfferent base learnng algorthms to generate homogeneous ensemble sets. It was shown that for each experment, wmetacomb outperformed Stackng and the Dynamc Integraton technque of Dynamc Weghtng wth Selecton. In future, we ntend to generalze the technque so as to allow the combnaton of more than two metaensemble technques. In ths, we may consder combnatons of Stackng usng dfferent learnng algorthms or other Dynamc ntegraton combnaton methods. References [Atkeson et al., 1997] Atkeson, C. G., Moore, A. W., and Schaal, S.. Locally weghted learnng. Artfcal Intellgence Revew, 11:11-73, Kluwer, [Breman, L., 1996a]. Baggng Predctors. Machne learnng, 24: , Kluwer, [Breman, L., 1996b] Stacked Regressons. Machne Learnng, 24:49-64, Kluwer, [Brown et al., 2004] Brown, G., Wyatt, J., Harrs, R. amd Yao, X. Dversty Creaton Methods: A Survey and Categorsaton. Informaton Fuson 6(1):5-20, Elsever,2004. [Džerosk, & Ženko, 2004] Džerosk, S. & Ženko, B. Is Combnng Classfers wth Stackng Better than Selectng the Best One? Machne Learnng 54(3): , Kluwer [Haste et al., 2001] Haste, T., Tbshran, R., Fredman, J. The Elements of Statstcal Learnng:Data mnng, Inference and Predcton. Sprnger seres n Statstcs, [Ho, 1998a] Ho, T.K. The random subspace method for constructng decson forests. IEEE Transactons on Pattern Analyss and Machne Intellgence, 20(8): , [Ho, 1998b] Ho, T.K. earest eghbors n Random Subspaces. In Proceedngs of the Jont IAPR Workshops on Advances n Pattern Recognton, LCS, Vol. 1451, pp , Sprnger-Verlag, [Krogh & Vedelsby, 1995] Krogh, A. and Vedelsby, J. eural etworks Ensembles, Cross valdaton, and Actve Learnng. In Advances n eural Informaton Processng Systems, pp , MIT Press, [Qunlan, 1992] Qunlan, R. Learnng wth contnuous classes, In Proceedngs of the 5 th Australan Jont Conference on Artfcal Intellgence, pp , World Scentfc, [Rooney et al., 2004] Rooney,., Patterson, D., Anand S, & Tsymbal, A. Dynamc Integraton of Regresson Models. In Proceedngs of the 5 th Internatonal Multple Classfer Systems Workshop. LCS Vol. 3077, pp , Sprnger, [Seewald, 2002] Seewald, A. Explorng the Parameter State Space of Stackng. In Proceedngs of the 2002 IEEE Internatonal Conference on Data Mnng, pp , [Setono et al., 2002] Setono, R., Leow, W.K., & Zurada, J. Extracton of Rules from Artfcal eural etworks for onlnear Regresson. IEEE Transactons on eural etworks, 13(3): , [Tsymbal et al., 2003] Tsymbal, A., Puuronen, S., Patterson, D. Ensemble feature selecton wth the smple Bayesan classfcaton, Informaton Fuson, 4:87-100, Elsever, [Tsymbal et al., 2005] Tysmbal, A. Pechenzky, M., Cunnngham, P. Sequental Genetc Search for Ensemble Feature Selecton. In Proc. of the 19 th Internatonal Jont Conference on Artfcal Intellgence, pp , [Wtten & Frank, 1999] Wtten, I. and Frank, E. Data Mnng: Practcal Machne Learnng Tools and Technques wth Java Implementatons, Morgan Kaufmann, [Wolpert, 1992] Wolpert, D. Stacked Generalzaton. eural etworks 5, pp , [Wolpert & Macready, 1996] Wolpert, D. & Macready, W. Combnng Stackng wth Baggng to Improve a Learnng Algorthm, Santa Fe Insttute Techncal Report ,