Application of Additive Groves Ensemble with Multiple Counts Feature Evaluation to KDD Cup 09 Small Data Set

Transcription

1 Applcaton of Addtve Groves Ensemble wth Multple Counts Feature Evaluaton to KDD Cup 09 Small Data Set Dara Sorokna School of Computer Scence, Carnege Mellon Unversty, Pttsburgh PA Edtor: Gdeon Dror, Marc Boullé, Isabelle Guyon, Vncent Lemare, Davd Vogel Abstract Ths paper descrbes a feld tral for a recently developed ensemble called Addtve Groves on KDD Cup 09 competton. Addtve Groves were appled to three tasks provded at the competton usng the small data set. On one of the three tasks, appetency, we acheved the best result among partcpants who smlarly worked wth the small dataset only. Postcompetton analyss showed that less successfull result on another task, churn, was partally due to nsuffcent preprocessng of nomnal attrbutes. Code for Addtve Groves s publcly avalable as a part of TreeExtra package. Another part of ths package provdes an mportant preprocessng technque also used for ths competton entry, feature evaluaton through baggng wth multple counts. Keywords: Addtve Groves, feature evaluaton, KDD Cup Introducton Addtve Groves s a powerful ensemble of regresson trees (Sorokna et al. (2007)). It s orgnally developed for regresson problems and s shown to outperform state-of-the-art methods on medum-sze regresson data sets. The data sets for the KDD Cup competton presented new challenges for the algorthm due to the followng dfferences from the data sets used n the orgnal paper: Bnary classfcaton problems ROC performance metrc Large sze of the tran set (50000 data ponts) Hgh-dmensonal data (260 features n the small, n the large data set) Presence of nomnal features wth extremely hgh arty Bnary classfcaton and ROC. Several modfcatons were attempted to talor orgnal regresson algorthm to classfcaton problem and ROC performance metrc. However, t turned out that bnary classfcaton combned wth ROC s a favorable settng for the orgnal Addtve Groves. In the end, only mnor changes were appled to create the classfcaton verson of the algorthm. Large hgh-dmensonal data set. The sze of data was too large to be processed by Addtve Groves n reasonable tme. Therefore, a feature selecton technque had to be 1

2 Sorokna appled. From our experence, whte-box feature evaluaton technques based on analyzng a structure of bagged trees provde good sets of features for tree-based ensembles n general and therefore are useful as a preprocessng step for Addtve Groves. We used multple counts, a smple technque descrbed n (Caruana et al. (2006)). Nomnal features. We have converted each nomnal feature nto a sngle nteger feature, wth dfferent nteger values correspondng to dfferent nomnal values. We dscovered later that such preprocessng was not suffcent to prevent overfttng n at least one of the three competton tasks. In the end, Addtve Groves proved ts applcablty to large classfcaton problems by producng the absolute best score on one of the three tasks n the small challenge. 2. Data preprocessng There were two versons of the data set at the competton, large and small. Both data sets contan data ponts n the tran set. The large data set ncludes features and due to the lack of computng resources was not used n ths entry. The small data set conssts of 260 features: 190 numercal and 40 nomnal. Labels were provded for three tasks on the same data: churn, appetency and upsellng. 2.1 Nomnal features Our mplementaton of decson trees does not have a bult-n technque to deal wth nomnal (also called categorcal or non-numercal) features. We had to convert them to bnary or nteger features durng the data preprocessng stage. As some nomnal features contaned thousands of values, each nomnal feature was converted nto a sngle nteger feature, where each nteger value corresponded to a sngle nomnal value. The values were ordered by the number of data ponts takng on ths value. See Secton 6 on useful mprovements for ths type of preprocessng. 2.2 Intal parameter tunng Our feature selecton technque descrbed n the next secton requres tranng an ensemble of bagged trees. Baggng s a well-known ensemble method ntroduced by Breman (1996). It creates a set of dverse models by samplng from the tranng set, and then decreases varance by averagng the predctons of these models. Decson tree s a common choce for the model n the baggng ensemble. We would lke to calculate feature scores from the verson of ensemble that produces the best performance. Bagged trees have an mportant parameter: sze of tree. Our mplementaton controls the sze of the tree through an external parameter α. α defnes the percentage of tranng data n a leaf, so n some sense t s reverse to the sze of the tree. We have tred the followng set of values of α: 1, 0.5, 0.2, 0.1, 0.05, 0.02, 0.01, 0.005, 0. See (Sorokna et al. (2007)) for the table of approxmate correspondence between dfferent values of α and the number of nodes n the resultng trees. Another parameter s the number of trees n the ensemble. We have opted for a fxed value of 100 trees. 2

3 task α tran set sze churn appetency upsellng Table 1: Parameters producng the best result for bagged trees In the competton framework we have a lmted labeled data set, so we have to decde on another mportant parameter: tran/valdaton set rato. Our data contans data ponts. We have consdered tran sets of 1, 2, 5, 10, 20, 30, 40, 45, 48 and 49 thousands of data ponts. Table 1 shows the combnaton of parameters that produced the best performance on valdaton set for bagged trees on each task. 2.3 Feature evaluaton: multple counts technque The smple dea behnd the whte-box feature mportance evaluaton s the followng: let us run a fast tree-based ensemble - baggng - and see whch features are actually used by the trees. We dscuss a number of such technques n our earler work (Caruana et al. (2006)). For ths entry we used one of the methods producng good results - multple counts. We defne a score of a tree node as the number of the data ponts n the tran set passng through ths node durng tranng. Then, the score of the feature s defned as the sum of the scores of the nternal tree nodes that use ths feature. For convenence the score s scaled by the number of data ponts n the tran set as well as the number of trees. Multple n the name of ths method refers to the fact that a data pont can be counted several tmes towards the same score f the same feature s used several tmes along one branch. For example, the tree on Fg. 1 wll produce scores of 1.6, 0.8 and 0.2 for features A, B and C respectvely. For every task we created ensembles of bagged trees usng the whole tran set and the best values of α from Table 1. These ensembles were used to calculate the multple A (100) C (20) B (80) A (60) Fgure 1: Sample decson tree splts the data usng features A, B and C. Node labels show the percentage of the tran set data ponts passng through each node. 3

4 Sorokna churn appetency upsellng Var Var Var Var Var Var Var Var Var Var Var Var Var Var Var Var Var Var Var Var Var Var Var Var Var Var Var Var Var Var Var Var Var Var Var Var Var Var Var Var Var Var Var Var Var Var Var Var Var Var Var Var Var Var Var Var Var Var Var Var Table 2: Top 20 features and ther mportance scores countng scores. In each case, top 20 features were selected to compose a data set for applyng Addtve Groves. The lst of selected features and ther scores s shown n Table Addtve Groves 3.1 Regresson ensemble Addtve Groves s a tree ensemble algorthm based on regresson trees, addtve models and baggng and s capable of both fttng addtve structure of the problem and modellng nonlnear components wth large trees at the same tme. Combnaton of these propertes makes t superor n performance to other exstng tree ensemble methods lke baggng, boostng and Random Forests. Addtve Groves conssts of bagged addtve models, where every element of an addtve model s a tree. A sngle grove s traned smlar to an addtve model: each tree s traned on the resduals of the sum of the predctons of the other trees. Trees are dscarded and retraned n turn untl the overall predctons converge to a stable functon. In addton, each grove of a non-trval sze s teratvely bult from smaller models: ether a tree s added to a grove wth fewer trees or the sze of trees s ncreased n a grove wth smaller trees; whch of the two steps happen n each partcular case s defned by comparng models on out-of-bag data. 4

5 Algorthm 1 Tranng Addtve Groves // Tree (α j,n) denotes an th tree n the grove wth the parameters (α j, n) functon TranGrove(α,N,tranSet) α 0 = 0.5, α 1 = 0.2, α 2 = 0.1,...,α max = α for j = 0 to max do for n = 1 to N do for = 1 to n 1 do Tree attempt1, = Tree (α j,n 1) Tree attempt1,n = 0 Converge(α j,n,tran,tree attempt1,1,...,tree attempt1,n ) f j > 0 then for = 1 to n do Tree attempt2, = Tree (α j 1,n) Converge(α j,n,tran,tree attempt2,1,...,tree attempt2,n ) wnner = Compare Tree attempt1, and Tree attempt2, on out-of-bag data for = 1 to n do = Tree wnner, Tree (α j,n) functon Converge(α,N,{x,y},Tree (α,n) 1,...,Tree (α,n) repeat for = 1 to N do newtranset = {x, y k Tree(α,N) k (x)} Tree (α,n) = TranTree(α, newtranset) untl (change from the last teraton s small) N ) A sze of a sngle grove (sngle addtve model) s defned by two parameters: the sze of tree and the number of trees. A sngle grove consstng of large trees can and wll overft heavly to the tranng set, therefore baggng s appled on top n order to reduce varance. Algorthm 1 gves pseudo-code for tranng a grd of groves of dfferent szes durng one baggng teraton. We refer the reader to the paper where Addtve Groves were ntroduced (Sorokna et al. (2007)) for more detaled descrpton of the algorthm. 3.2 Modfcaton for classfcaton problems In Addtve Groves, models of dfferent szes are naturally produced durng the tranng phase. These models are evaluated on the valdaton set n order to determne the combnaton of parameters producng the best model. The only modfcaton we ended up usng for the classfcaton problems s the use of ROC metrc nsted of orgnal RMSE (root mean squared error) durng ths fnal model evaluaton. We stll use RMSE for fttng sngle trees and buldng addtve models. 5

6 Sorokna Several modfcatons were tred n an attempt to tune Addtve Groves better for classfcaton problems evaluated by ROC metrc. None of them demonstrated any mprovements and eventually these modfcatons were dscarded. They nclude: Usng ROC nstead of RMSE when comparng dfferent models of the same sze durng tranng Clppng predctons of each addtve model at 0 and 1 1 whle fttng models Clppng predctons of the whole bagged ensemble at 0 and 1 after the tranng s completed The last two modfcatons were attempted because addtve model-based algorthms sometmes can predct a value greater than 1 or smaller than 0, whch does not make much sense n bnary 0 1 classfcaton. Such predcton wll generate a negatve resdual durng tranng even f t s techncally correct (that s, actual true values are 1 or 0 respectvely). Some performance metrcs, for example RMSE, wll count such predctons as errors. However, ROC does not have ths problem, because t does not evaluate the actual predcton numbers. ROC s nterested only n the orderng of predctons, and t does not matter f the true postve predcton s greater than 1 as long as t s larger than predctons for negatve cases. In ths sense ROC s a very convenent metrc for evaluatng models such as Addtve Groves on classfcaton problems. 4. Fnal models Addtve Groves have 3 crucal parameters: α (controls sze of tree), N (number of trees n each grove) and number of baggng teratons. The last parameter s conservatve - the more baggng teratons the better, whle for the other two parameters there usually exst some optmal level of complexty. In order to evaluate the models and to fnd the optmal values of α and N, we reused the tran/valdaton splts of data from the earler experments (Table 1). Table 3 provdes best performance of Addtve Groves on our valdaton sets and parameters of correspondent models. task α N # of bagged models ROC on valdaton set churn appetency upsellng Table 3: Performance on valdaton set We have traned fnal models usng the whole labeled data set, (α, N) values from the Table 3 and 1000 baggng teratons. It s worth notng that our mplementaton allows to tran the models much faster once the desred values of N and α are fxed. 1. Predctons greater than 1 become 1, predctons less than 0 become 0 6

7 team churn appetency upsellng average creon LosKallos FEG BOSS Lenca M FEG ATeam pavel Addtve Groves nkhop m java.lang.outofmemoryerror Lajkonk FEG CTeam Celne Theeuws zlm FEG B CSN TonyM Sundance Mner FEG D TEAM DKW dg homehome Ma Dang parramnng.blogspot.com bob muckl C.A.Wang KDD@PT decaff StatConsultng Dr. Bunsen Honeydew Raymond Falk vodafone K Clamner rw Persstent Lous Duclos-Gosseln Chy Abo-Al sduzx MT Shraz Unversty Klmma thes Table 4: Competton results for small data set wthout unscramblng 5. Competton results KDD Cup 09 competton conssted of several separate challenges: fast, slow(large) and slow(small). More, there were two types of submsson for small challenge: wth unscramblng and wthout. Submssons of the frst type made use of both large and small data set and therefore are not comparable to the submssons that used the small data set only. Addtve Groves competed n the small challenge wthout unscramblng. Table 4 shows 47 entres from the offcal results table (Guyon (2009)) that fall nto ths category. 7

8 Sorokna The good news s that Addtve Groves defeated every other entry on the appetency task wth the score of The second best score n ths category s Ths result shows that Addtve Groves can be as superor for classfcaton as t s for regresson at least on some tasks. On the upsellng task Addtve Groves got the score of The best result was Unfortunately, the result for churn was much worse - Addtve Groves acheved ROC of whle the best result was Wth more than half compettors ahead, t was clear that we faled to extract some mportant nformaton from the data n ths case. In the next secton we dscuss one of the possble reasons. 6. Post-competton mprovements We have reconsdered several aspects of preprocessng of nomnal values after the competton. Frst, we modfed the way of orderng the values whle convertng a nomnal feature to a numercal. Durng the competton we have been orderng values by the number of data ponts takng on ths value. Later we have adopted a potentally more useful technque: orderng the values by the rato of postve responses among the data ponts takng on each value (Haste et al., 2001, chapter 9.2.4). Second, n order to understand the reasons for falng on churn problem, the author has contacted Hugh Mller, the leader of Un Melb team. Ths team acheved much better results on churn n the small challenge. We learned that they also used 20 features, however, the exact set of features they were usng dd not help to mprove the results n our case. Further, we learned that they used a more elaborate preprocessng of the nomnal features. For those categorcal varables wth more than 25 categores, levels wth fewer than 100 nstances n the tranng data were aggregated nto a small category, those wth nstances were aggregated nto a medum category, and those wth nstances aggregated nto a large category. - Un Melb factsheet Such preprocessng decreases the chance for the trees to overft by makng too many splts on a nomnal feature. We have ncorporated the technques mentoned above nto preprocessng and repeated experments for churn wth Addtve Groves on the mproved verson of the data. Ths helped to boost our best performance on the offcal test set from to Therefore, t seems that at least a part of our problem wth churn task was caused by nsuffcent data preprocessng. 7. TreeExtra package Both Addtve Groves and bagged trees wth multple counts feature evaluaton are avalable as parts of TreeExtra package. TreeExtra s a set of command lne tools wrtten n C++/STL. You can fnd bnares, code and manuals at dara/treeextra.htm 8

9 Acknowledgments The author would lke to acknowledge her collaborators who helped to develop algorthms descrbed n ths paper: Rch Caruana, Mrek Redewald and Art Munson. References Leo Breman. Baggng Predctors. Machne Learnng, 24: , Rch Caruana, Mohamed Elhawary, Art Munson, Mrek Redewald, Dara Sorokna, Danel Fnk, Wesley M. Hochachka, and Steve Kellng. Mnng ctzen scence data to predct prevalence of wld brd speces. In Proceedngs of ACM KDD, I. Guyon. KDD Cup results T. Haste, R. Tbshran, and J.H. Fredman. The Elements of Statstcal Learnng. Sprnger, Dara Sorokna, Rch Caruana, and Mrek Redewald. Addtve Groves of Regresson Trees. In Proceedngs of ECML,