Boosting Weighted Linear Discriminant Analysis

Transcription

1 . Okada et al. / Internatonal Journal of Advaned Statsts and I&C for Eonoms and Lfe Senes Boostng Weghted Lnear Dsrmnant Analyss azunor Okada, Arturo Flores 2, Marus George Lnguraru 3 Computer Sene Department, San Franso State Unversty,600 Holloway Avenue, San Franso, CA 9432, USA, kazokada@sfsu.edu 2 Department of Computer Sene and Engneerng, Unversty of Calforna San Dego, 9500 Glman Drve, La Jolla, CA 92093, USA 3 Radology and Imagng Senes, Clnal Center, atonal Insttutes of Heath, 0 Center Drve, Bethesda, MD 20892, USA, lngurarum@mal.nh.gov Abstrat We propose a novel approah to boostng weghted lnear dsrmnant analyss (LDA) as a weak lassfer. Combnng Adaboost wth LDA allows to selet the most relevant features for lassfaton at eah boostng teraton, thus beneftng from feature orrelaton. he advantages of ths approah nlude the use of a smaller number of weak learners to aheve a low error rate, mproved lassfaton performane due to the robustness and stable nature of LDA, and omputatonal effeny. he performane of the proposed method was evaluated on artfal data and addtonally on two popular ndependent data sets: the Irs Data Set and the Breast Caner Wsonsn Dagnost Data Set, both publly avalable at the Unversty of Calforna at Irvne Mahne Learnng Repostory. Expermental results showed the superor auray of the proposed method over LDA and AdaBoost ombned wth other types of weak lassfers. he weghted LDA algorthm was proven to be equvalent to the tradtonal LDA n the ase of unform weght dstrbutons. eywords: Classfaton, Adaboost, weghted lnear dsrmnant analyss... Introduton Adaboost [5] s one of the most popular meta/ensemble learnng algorthms that buld an aurate (strong) lassfer from a group of naurate (weak) lassfers. In ts lass formulaton, the weak lassfer of Adaboost s defned by a smple funton that performs bas bnary thresholdng on a sngle feature extrated from eah tranng sample. Hene, eah weak lassfer s traned to mnmze a weghted error on a sngle feature. Postve-valued weghts are ntrodued to eah tranng sample and adapted durng the ourse of tranng n order to fous on samples that are dffult to lassfy. At eah teraton, Adaboost selets the feature that mnmzes the weghted error. In ths respet, Adaboost s essentally a feature seleton algorthm. Fnally, the strong lassfer of Adaboost ombnes n a lnear fashon multple weak lassfers seleted aordng to the sample weghts, resultng n a jont sngle deson rule. Adaboost has been suessfully appled to varous medal and non- 200 Luan Blaga Unversty of Sbu

2 . Okada et al. / Internatonal Journal of Advaned Statsts and I&C for Eonoms and Lfe Senes medal applatons suh as lung tumor deteton [], Alzhemer s dsease deteton [0] and pedestran deteton [5] to name a few. Alternatvely, Lnear Dsrmnant Analyss (LDA) [2] s one of the lass statstal pattern lassfaton tehnques. he goal n LDA s to fnd an optmal subspae from a feature vetor spae, whh maxmzes the rato of between-lass satter to wthn-lass satter [2]. As a result, LDA benefts from feature ombnatons that produe the hghest separaton between lasses. hs paper presents a new approah to ombne the Adaboost and LDA algorthms to mprove lassfaton performane by explotng the LDA lassfers as weak lassfers of Adaboost. By ombnng these two algorthms, Adaboost an selet the best feature ombnaton at eah boostng teraton nstead of a sngle feature, therefore takng advantage of feature orrelatons. he lass LDA theory, however, does not nlude sample weghts that are neessary for the boostng framework. In order to adopt LDA as weak learner of Adaboost, we ntrodue sample weghts n the LDA formulaton. Several prevous studes relate to our work. An outler-lass resstant approah to estmate the wthn-lass ovarane matrx n LDA for lnear dmensonalty reduton was presented by ang et al. [3]. Several methods were proposed to estmate the LDA weghts and tests were performed on medal and non-medal data. Although ths work explored smlar sample weghts n ther LDA formulaton, they dd not nvestgate ts applaton n the boostng ontext of our nterest. he adaptaton of other lassfaton models to Adaboost has reently been explored [8,4], however there are only very few reports that nvestgated LDA n ths ontext, despte the popular usage of dsrmnant analyss [9,]. An nterestng approah was proposed by Lu et al. [9] who adapted LDA to norporate sample weghts n an applaton related to ndoor/outdoor dgtal mage lassfaton. Whle the authors ntrodued a weghted LDA - Adaboost ombnaton, ther approah was not equvalent to a tradtonal LDA n the ase of unform weght dstrbutons. As AdaBoost ntalzes the sample weghts to a unform dstrbuton, t s mportant that a true LDA lassfer be appled n the frst teraton. hs ondton an be seen as a guarantee that the ntal lassfer represents an adequate startng pont. Skurhna and Dun [] also nvestgated boostng LDA lassfers; however ther sheme dd not am to ombne the two algorthms by ntrodung weghts n LDA formulaton. o evaluate the effetveness of the proposed approah, we apply the method frst to artfal data and then to two ndependent data sets and ompare our method wth AdaBoost ombned wth other types of weak lassfers. he expermental results reflet the superor auray of the proposed method over LDA and AdaBoost. 2. Data and Methods he followng setons desrbe the proposed method for boostng weghted LDA lassfers. Frst the expermental data are presented followed by some bas methodologal onepts used n ths study. 2. Data For the evaluaton of our method, we use two publly avalable data sets, the Irs Data Set [3] and Breast Caner Wsonsn Dagnost Data Set (BCWDDS) [2], both from the Unversty of Calforna Irvne Mahne Learnng Repostory [7]. he rs data ontan 50 nstanes wth 4 attrbutes, whh are grouped n 3 lasses orrespondng to a type of rs plant. One lass s lnearly separable from the other Luan Blaga Unversty of Sbu

3 . Okada et al. / Internatonal Journal of Advaned Statsts and I&C for Eonoms and Lfe Senes two; the latter are not lnearly separable from eah other; we employed the 00 ases that are not lnearly separable n our experments. More presely, a par of labels for versolor and vrgna are used for our bnary lassfaton task. he BCWDDS data are 569 nstaned from bengn (357) and malgnant (22) ases eah wth 32 attrbutes. Features were omputed from a dgtzed mage of a fne needle asprate of a breast mass. hey desrbe haratersts of the ell nule present n the mage. ypally, the raw nput to a lassfaton system s frst subjeted to a feature extraton stage n order to transform the nformaton nto a more dsrmnant representaton of the data. Our study employs the entre feature spae of the two data sets: 4 features for the rs data [3] and 32 features for BCWDDS [2]. Addtonally, a two dmensonal artfal/smulated data set was used to valdate the algorthm n a very low feature spae (Fgure ). he artfal data set onssts of 500 samples evenly dvded between two lasses. Samples from one lass are lustered on a entral pont and are ompletely surrounded by samples of the seond lass. he data of the two lasses are learly not lnearly separable. 2.2 Adaboost Adaboost [5] s an ensemble learner where the jont deson rule of multple weak lassfers forms an overall strong lassfer. Adaboost assgns an ntal unform weght W ()/n to eah tranng sample x, where n s the number of tranng samples. At teraton k, Adaboost fnds the lassfer h k traned usng samples x that mnmzes the weghted error E k aordng to W k (). he weghts are updated by W W ( ) exp( α y h ( x )) ( ) k k k k+, Z k () Ek where α k log, y are the ground truth labels, and Z k s a normalzaton 2 Ek fator. Aordng to these formulae, weghts for orretly lassfed samples are nreased and weghts for norretly lassfed samples are dereased. hs allows Adaboost to fous on the nformatve and dffult tranng samples. he resultng lassfer of the Adaboost algorthm beomes H ( x) sgnα tht ( x), t (2) where h t s the t-th weak lassfer hypothess, and H(x) s the strong lassfer hypothess. Adaboost s typally ombned wth a smple threshold lassfer that mnmzes the weghted error on a sngle feature. In other words, at teraton k, Adaboost selets the feature whh mnmzes the weghted error. W (k+) s nreased when x s lassfed orretly by h k, and dereased otherwse. In ths respet, Adaboost s essentally a feature seleton algorthm. 2.3 Lnear Dsrmnant Analyss Lnear Dsrmnant Analyss (LDA) [2] s desgned to maxmze the rato of between-lass satter to wthn-lass satter. Between-lass satter measures the 200 Luan Blaga Unversty of Sbu 3

4 . Okada et al. / Internatonal Journal of Advaned Statsts and I&C for Eonoms and Lfe Senes varane of the projetons for eah ndvdual lass. Wthn-lass measures the varane of the projetons from all data samples. LDA employs feature ombnatons that produe the hghest separaton between lasses. he tradtonal LDA mplementaton does not take nto onsderaton weghts. Supposng that we are gven tranng samples that are parttoned nto lasses, S,, S,, S. hen the tradtonal LDA formulae for between-lass satter S B and wthn-lass satter S W are S S B W µ x ( µ x)( µ x) ( x µ )( x µ ) S S x x µ, (3) (4) (5) (6) where s the number of ases n lass. he LDA rteron funton an be wrtten as w S Bw J ( w). w S w w (7) LDA fnds the vetor w where the J(w) s maxmzed. hs vetor s used to apply a lnear transformaton to the data,.e. projetng the data onto the vetor w by yw x. After ths transformaton, the separaton between lasses s maxmzed. In order to have a omplete lassfer, a threshold w 0 s neessary. For a bnary lassfer, a typal hoe of threshold s to ompute the projetons of the lass means, µ, and then ompute the mean of these two projetons. A tranng sample an now be lassfed as postve f w x > w 0, and negatve otherwse. 2.4 Weghted LDA In order to use LDA wth Adaboost, the LDA formulae are modfed to take nto aount weghts. By ombnng Adaboost wth LDA, Adaboost an selet the best feature ombnaton at eah boostng teraton nstead of a sngle feature. We propose the reformulaton of the satter matres for LDA wth sample weghts as follows ( µ ' x' )( ' x' ) S' B P( ) µ S S ( x µ ' )( x ' ) S' P( ) µ W (8) (9) Luan Blaga Unversty of Sbu

5 . Okada et al. / Internatonal Journal of Advaned Statsts and I&C for Eonoms and Lfe Senes µ ' P( ) S S S P( ) x x' P( ) x, () (0) where P() s the weght assgned to eah -th tranng sample wth P( ), as n the Adaboost algorthm. Instead of seletng the sngle feature that mnmzes the lassfaton error, wth weghted LDA (wlda) the best feature ombnaton of features an be seleted at eah teraton, therefore takng advantage of feature orrelaton. In the ase of unform weght dstrbuton, namely P() /, the wlda formulae beome equvalent to the tradtonal LDA formulae. Another related formulaton was proposed by Lu et al [9], but t dd not norporate sample weghts nto S B and S W. Hene, the formulaton n [9] does not guarantee the equvalene to the tradtonal LDA, whh an lead to nonsstent results, espeally n a low dmensonal feature spae. 2.5 Classfaton va Adaboost wth Weghted LDA Reall that the tradtonal Adaboost mplementaton searhes aross smple threshold lassfers traned on a sngle feature. o ombne Adaboost wth wlda, we tran a wlda lassfer wth all possble ombnatons of two and three features at eah teraton. he feature ombnaton that mnmzes the weghted error s then seleted. he sample weghts are updated at eah teraton aordng to equaton (). ote that the threshold w 0 also norporates the weghts, sne t s omputed usng the projeton of the weghted lass means µ. 2.6 Valdaton A set of features were avalable for eah nstane n a data set. hese features and ground truth labels were used to tran the lassfers. We ompare our proposed lassfaton method ombnng Adaboost wth the weghted LDA (AB+wLDA) aganst two related boostng methods: the orgnal Adaboost wth smple threshold weak lassfers (AB+S) and another adaptaton of Adaboost wth weghted LDA (AB+wLDA_Lu) proposed by Lu [9]. AB+S uses the weak lassfer based on the smple threshold that mnmzes the weghted error on a sngle feature. AB+wLDA_Lu offers a soluton for boostng wlda that does not norporate sample weghts n S B, resultng n unbalaned satter matres that do not onverge to the orgnal LDA when usng the unform weghts. Several lassfaton performane metrs were used nludng test/tranng error at eah boostng teraton and Leave Out Cross Valdaton (LOCV) [7]. For the rs and BCWDDS data, the LOCV fold sze was k 30 and 20, respetvely, and averaged over four test. Intal tests are performed on the artfal data set to ompare the ablty of the dsussed lassfers to dstngush nstanes from data wth a low feature spae. Results on the rs and BCWDDS data ompare the same lassfers on nstanes wth hgher feature spae. 200 Luan Blaga Unversty of Sbu 5

6 . Okada et al. / Internatonal Journal of Advaned Statsts and I&C for Eonoms and Lfe Senes 3. Results 3. Artfal Data A omparatve example of lassfaton from artfal data an be seen n Fgure. he smple threshold lassfer an only use one feature at a tme. When Adaboost s ombned wth ths type of weak learner (AB+S), Adaboost selets the sngle feature that mnmzes the weghted error at eah teraton, whh an be seen as the vertal and horzontal lnes n Fgure. On the other hand, LDA uses feature ombnatons nstead of sngle features, thus takng advantage of feature orrelatons to separate between lasses. In both tranng and test error (Fgure ), the results for AB+S and AB+wLDA are smlar. A hgh number of smlar lassfaton rules an be seen n a loalzed area for AB+wLDA_Lu. In ths low feature spae AB+wLDA_Lu s equvalent to random guessng. Fgure. Comparson of lassfaton tehnques on artfal data. he top two rows show the lassfaton results of AB+S, AB+wLDA, and AB+wLDA_Lu along the average auray (ross-valdaton) of the tehnques. he bottom row presents tranng and test errors for eah tral run. Lght olors n the data represent one lass entered on a pont; dark olors embody the seond lass surroundng the frst lass. Squares represent tranng samples and rles show test samples. Eah lne represents the deson boundary of a learned weak lassfer. he ross-valdaton results for AB+wLDA_Lu show that n ths low feature spae, the method s equvalent to random guessng Luan Blaga Unversty of Sbu

7 . Okada et al. / Internatonal Journal of Advaned Statsts and I&C for Eonoms and Lfe Senes 3.2 Irs Data Set he wlda mplementaton was further tested on the two non-lnearly separable lasses of the rs data set and results are presented n Fgure 2. he rs data have a hgher feature spae (sze 4), and we tested AB+wLDA by searhng for the best three feature ombnaton. AB+wLDA outperforms the other two lassfers (AB+S and AB+wLDA_Lu) n the ross-valdaton shown n the bottom of Fgure 2. In partular, AB+wLDA has a lear advantage over AB+S on the hgher feature spae. Wth AB+S, the test error nreases slghtly as the number of boostng teratons nreases, whh ndates that AB+S s over-fttng the data. Wth AB+wLDA there s a slght ntal nrease n the test error, but then t drops down the startng test error at the frst teraton and remans onstantly low. he strength and robustness of Adaboost ombned wth our wlda mplementaton are also noteable n the tranng set. Fgure 2. Expermental results on the rs data. he top row presents tranng and test errors for eah tral run of LOCV, whle the bottom row shows the average auray (rossvaldaton). 200 Luan Blaga Unversty of Sbu 7

8 . Okada et al. / Internatonal Journal of Advaned Statsts and I&C for Eonoms and Lfe Senes 3.3 Breast Caner Wsonsn Dagnost Data Set For the BCWDDS data, the results usng AB+wLDA are omparable to AB+wLDA_Lu n both tranng and testng sets, as shown n Fgure 3. Moreover, AB+S performed lose to the other two methods at tests, as seen n the ross valdaton at the bottom of Fgure 3, but underperformed on the tranng set. he BCWDDS data have a feature vetor of sze 32. For AB+wLDA and AB+wLDA_Lu, the best possble ombnaton of three features s seleted at eah teraton. Whle n the rs data AB+wLDA and AB+S had the lowest tranng error, on BCWDDS data AB+wLDA and AB+wLDA_Lu performed wth muh lower tranng error. he dfferene on performane ths data s lkely related to the dfferene n features spae. Fgure 3. Expermental results on the BCWDDS data. he top row presents tranng and test errors for eah tral run of LOCV, whle the bottom row shows the average auray 4. Dsusson and Conluson hs paper proposes a novel method to boostng weghted LDA as weak lassfers that ombnes the strength and robustness of AdaBoost wth LDA. he expermental results demonstrated the advantage of our method over the orgnal Adaboost and a Luan Blaga Unversty of Sbu

9 . Okada et al. / Internatonal Journal of Advaned Statsts and I&C for Eonoms and Lfe Senes smlarly weghted LDA-based Adaboost wth unbalaned satter matres, proposed prevously by Lu [9]. Our method outperformed both baselne methods usng the rs data set whle performng omparably wth other lassfers when tested on the BCWDDS data set. he weghted LDA algorthm presented here was proven to be equvalent to the tradtonal LDA n the ase of unform weght dstrbutons. hs observaton s mportant beause AdaBoost ntalzes sample weghts unformly and t s rtal to have a true LDA lassfer as the ntal startng pont. By fully norporatng weghts nto all aspets of the LDA formulaton, AB+wLDA was effetve n produng an ensemble of rules that an aheve hgh auray, even n low feature spae data, suh as seen n Fgure. Combnng Adaboost wth LDA allows seletng the most relevant features for lassfaton at eah boostng teraton, thus beneftng from feature orrelaton. he advantages of ths approah nlude the use of a smaller number of weak learners to aheve a low error rate, mproved lassfaton performane due to the robustness and stable nature of LDA, and omputatonal effeny. AB+wLDA_Lu faled to produe a good ensemble of rules on the artfal data, whle AB+S and AB+wLDA performed well and aheved good separaton. hs data set had a small feature spae, so AB+S aheved lower tranng/test error faster than AB+wLDA. AB+wLDA. he results on the Irs Data Set (non-medal) and the Breast Caner Wsonsn Dagnost Data Set (medal), both wth hgher feature spae, showed that the ombnaton of Adaboost and weghted lnear dsrmnant analyss outperforms the other lassfers n all the performane metrs used. hs onfrms the hypothess that AB+wLDA an aheve low lassfaton error wth fewer number of rules, resultng n a more ompat and effent lassfaton model. he lassfaton method ombnng Adaboost and weghted LDA s lkely to be reproduble on other medal (and non-medal) data sets, makng t a valuable tool for lnal dagnoss. For future work, the lassfaton method wll be extended for mult-lass problems and evaluated on addtonal data sets n omparson wth other lassfers, suh as support vetor mahnes. Referenes. B J, Peraswamy S, Okada, ubota, Fung G, Salganoff M, Rao RB. Computer aded deteton va asymmetr asade of sparse hyperplane lassfers, Proeedngs SIGDD 2006; Duda R O, Hart P E., Stork D G. Pattern Classfaton. Wley-Intersene Publaton Fsher R A. he use of multple measurements n taxonom problems. Annual Eugens 936; 7(II): Flores A, Rysavy S, Enso R, Okada. onnvasve dfferental dagnoss of dental perapal lesons n one-beam C. In Proeedngs IEEE ISBI 2009; Freund Y, Shapre R E. A deson-theoret generalzaton of on-lne learnng and an applaton to boostng. In European Conferene on Computatonal Learnng heory 995; Huang Y L, Wang L. Dagnoss of breast tumors wth ultrason texture analyss usng support vetor mahnes. eural Computng and Applatons 2006; 5(2): ohav R. A study of ross-valdaton and bootstrap for auray estmaton and model seleton. In Proeedngs IJCAI 995; L X, Wang L, Sung E. Adaboost wth SVM-based omponent lassfers, Engneerng Applatons of Artfal Intellgene 2008: 2(5): Lu X, Zhang L, L M, Zhang H, Wang D. Boostng mage lassfaton wth LDA-based feature ombnaton for dgtal photograph management. Pattern Reognton 2005; 38(6): Luan Blaga Unversty of Sbu 9

10 . Okada et al. / Internatonal Journal of Advaned Statsts and I&C for Eonoms and Lfe Senes 0. Morra JH, u Z, Apostolova JG, Green AE, oga AW, hompson PM. Comparson of AdaBoost and Support vetor mahnes for detetng Alzhemer d dsease through automated hppoampal segmentaton, IEEE ransatons on Medal Imagng 2009; 29(): Skurhna M, Dun RPW. Boostng n Lnear Dsrmnant Analyses, Proeedngs Int Workshop on Multple Classfer Systems 2000: MCall J O, Wald S S. Clnal Dental Radology. Phladelpha: Saunders 954; 4th ed. 3. ang E, Suganthan P, Yao X, Qn A. Lnear dmensonalty reduton usng relevane weghted LDA. Pattern Reognton 2005; 38: ruyen, Phung DQ, Venkatesh S, Bu HH. AdaBoost.MRF: Boosted Markov Random Forests and Applaton to Multlevel Atvty Reognton, Proeedngs CVPR 2006: Vola P, Jones MJ, Snow D. Detetng Pedestrans Usng Patterns of Moton and Appearane, Internatonal Journal of Computer Vson 2005; 63(2): Wolberg WH, Street W, Mangasaran OL. Mahne learnng tehnques to dagnose breast aner from fne-needle asprates. Caner Letters 994; 77: Luan Blaga Unversty of Sbu