Low training strength high capacity classifiers for accurate ensembles using Walsh Coefficients

Transcription

1 Low tranng strength hgh capacty classfers for accurate ensebles usng Walsh Coeffcents Terry Wndeatt, Cere Zor Unv Surrey, Guldford, Surrey, Gu2 7H t.wndeatt surrey.ac.uk Abstract. If a bnary decson s taken for each classfer n an enseble, tranng patterns ay be represented as bnary vectors. For a two-class supervsed learnng proble ths leads to a partally specfed Boolean functon that ay be analysed n ters of spectral coeffcents. In ths paper t s shown that a vote whch s weghted by the coeffcents enables a fast enseble classfer that acheves perforance close to Bayes rate. Experental evdence shows that effectve classfer perforance ay be acheved wth one epoch of tranng of an MLP usng Levenberg-Marquardt wth 64 hdden nodes. Keywords: Ensebles, Multlayer Perceptrons, Boolean Functon, Walsh Coeffcents INTRODUCTION For an enseble of classfers t s often useful to thnk of each base classfer as beng controlled by two an paraeters, the capacty and the tranng strength of the learnng algorth []. The ter capacty refers to the flexblty of the classfer boundary. By tranng strength we ean the effort that s put nto tranng the classfer. For an MLP, the capacty s the nuber of hdden nodes, and tranng strength s the nuber of epochs. In ths paper we consder the trade-off between these two paraeters, and what cobnaton s sutable for a weghted ajorty vote. The weghted vote s coputed usng Walsh coeffcents. If each base classfer n an enseble s gven a bnary decson, and f the proble s two-class, a Boolean appng s defned. Ths appng ay be analysed usng Walsh spectral coeffcents. Frst order Walsh coeffcents were shown to provde a easure of class separablty for selectng optal base classfers n [2], n whch t s also shown that ths does not ply optalty of the enseble. In contrast, n [3] t was shown that second order Walsh coeffcents ay be used to deterne optal enseble perforance. The otvaton for usng Walsh coeffcents n enseble desgn s fully explored n [4] and [2]. For further understandng of the eanng and applcatons of Walsh coeffcents see [5] and [6]. To understand the coputaton of the weghted vote, the Tuer-Ghosh odel [7] for enseble classfers wll be descrbed. Ths odel defnes Added Classfcaton Error as the dfference between classfer error and Bayes error, and provdes a fraework for understandng the reducton n error due to cobnng. Secton 2 explans the coputaton of the Walsh coeffcents, and Secton 3 dscusses the relatonshp wth the odel of Added Classfcaton Error. In Secton 4, the weghted vote usng Walsh coeffcents s copared as the nuber of nodes and tranng epochs of MLP base classfers are systeatcally vared.

2 2 WALSH COEFFICIENTS Consder an enseble fraework, n whch there are N parallel base classfers, and s the N-denson vector representng the th tranng pattern, fored fro the decsons of the N classfers. For a two-class supervsed learnng proble of tranng patterns, the target label gven to each pattern s denoted by ) ( where =, {, } and s the unknown Boolean functon that aps to. Thus the bnary vector represents the th orgnal tranng pattern (, 2,, N ) where {, } s a vertex n the N-densonal bnary hypercube. The Walsh transfor of s derved fro the appng T n and defned recursvely as follows () T n T T n n T n T n T (2) The frst and second order spectral coeffcents s and s j derved fro (2) are defned n [5] as s ( ) (3) s j s In (3) j ( j ) s represents the correlaton between ) ( and ( (, j N, j) n (4) represents correlaton between ) and j, where s logc exclusve-or. Realstc learnng probles are ll-posed [8], and therefore ay be partally specfed, nosy and possbly contradctory. Relatonshps for coputng spectral coeffcents for partally specfed Boolean functons, are proved n [9], for whch the context s logc crcut desgn. The relevant deas are presented here usng dfferent ternology, specfcally nters nterpreted as patterns. In [9], the concept of a standard trval functon s ntroduced. Each spectral coeffcent gves a correlaton value between the Boolean functon and. For frst order coeffcents, s the Boolean varable n (3) whle for second order (4) and

3 3 27/8/22 :4 coeffcents j s n (4). Note n (4) ples par of j classfers and j dsagree for pattern and ples classfers agree. For thrd order coeffcents, j j jk s j k and hgher order follows, but n ths paper we restrct ourselves to frst and second order spectral coeffcents. The equatons (3) and (4) requre bnary varables {, } but for coputng coeffcents t s notatonally ore convenent to use {,}. For p, q {, } defne n pq to be the nuber of class p patterns of Boolean functon for whch both and have the logcal value q. Then n s the nuber of class patterns (true nters n [9]) for whch both and that have the logcal value. Slarly n s the nuber of class patterns (false nters n [9]) for whch both and have the logcal value. Correspondng defntons follow for n and n. Now defne d and d to be the nuber of unspecfed patterns (don t care nters) for whch has the logcal value and respectvely. It s clear that the su of all patterns of an N-densonal Boolean functon s gven by n n n n d d 2 N (5) Accordng to [9], all spectral coeffcents s l ay be coputed as s l ( n n) ( n n) (6) where l ay be or j. Substtuton of (5) nto (6) gves varous equvalent forulae, but the advantage of (6) s that t s not necessary to nclude unspecfed patterns d explctly n the coputaton.,d 3 ADDED CLASSIFICATION ERROR MODEL, Fgure shows the two class ( ) odel of Added Classfcaton Error ( darkly shaded regon) accordng to [7], whch for splcty s restrcted to one denson (x). The optu (Bayes) boundary n Fgure s the loc of all ponts ~ x : P( ~ ) ( ~ x P x ). The output of the classfer representng class s gven by Pˆ ( P( ( ) where P, P ˆ are the actual and estated x a posteror probablty dstrbutons as shown n Fgure, and ( x ) s the dfference between the. A slar equaton s obtaned for class wth

4 P, Pˆ( ) and error ( ). In Fgure b s the aount that the kth ( x x classfer boundary (x b ) dffers fro the deal Bayes boundary ( x ~ ), and assung that b s a Gaussan rando varable wth ean β and varance σ b, n [7] t s shown that 2 2 Added Classfcaton Error for kth classfer s gven by E P( ) where P.5( P( ~ x ) P( ~ x )) p( ~ ) x k and P ndcates dfferentaton. Fgure 2 shows decson boundares of (,j)th classfers for whch t s assued that the coplexty s not suffcent to approxate the Bayes boundary, so that both classfers under-ft. Note n Fgure 2 that estated probabltes Pˆ( and Pˆ( are otted for clarty. Mutually exclusve areas under the probablty dstrbuton are labelled 8 n Fgure 2, and denotng the nuber of patterns n area y by a y, the contrbuton fro classfers,j accordng to area s gven n Table. The odel assuptons are dscussed n [3], n whch the expresson for the dfference n Added Classfcaton Error of th and jth classfers E E E s derved j b j E j E E j.5( s ) j (7) where 2p and p s the pror probablty of class. Averagng over all pars of classfers n (7) the ean dfference n added error s gven by E E j j, j, (8) Therefore fro (7) and (8) we can approxate ean Added Error by subtractng and averagng over all par-wse second order coeffcents, call t S2M. In [3] t s shown that S2M s a good predctor of enseble perforance as nuber of epochs s ncreased. For the datasets n Secton 4, optal perforance for ajorty vote occurs on average around 2-3 epochs. The usual dea n weghted votng s to reward ndvdual classfers that perfor accurately []. In ths paper, a dfferent approach s taken. For classfers wth lower tranng strength, t s expected that classfers aybe unevenly spread around the optal boundary. The dea s to gve larger weght to pars of classfers wth low Added Error. The classfers are chosen based on the product of frst order coeffcents as follows. The frst order coeffcents n (3) are decoposed nto the contrbutons fro the two classes ) ( n ( n n and n ) and the weght s proportonal to ther product. The weght of the th classfer s gven by w l ( n n)( n n) (9)

5 5 27/8/22 :4 wth negatve weghts n (9) set to zero. Consderng Fgure, classfers close to the Bayes boundary wll receve larger weght, but as they ove further away, weght s decreased and becoes zero as n approaches n or as n approaches n. When classes are unbalanced, (9) tends to favour classfers on ether sde of the Bayes boundary, n contrast to a weghtng schee based on tranng error. The weghtng schee usng (9) s referred to as WP n Secton 4, and shown to reduce the ean Added Error gven by (8). 4 EPERIMENTAL EVIDENCE Natural two-class benchark probles selected fro [] and [2] are shown n Table 2.The orgnal features are noralsed to ean std, and for datasets wth ssng values the schee suggested n [] s used. Rando perturbaton of the MLP base classfers s caused by dfferent startng weghts on each run. The nuber of hdden nodes and tranng epochs of hoogenous (sae nuber of nodes and epochs) MLP base classfers are systeatcally vared over -5 epochs and 2-64 nodes. The experents are perfored wth two hundred sngle hdden-layer MLP base classfers, usng the Levenberg-Marquardt tranng algorth wth default paraeters. Cobnng uses ajorty (MAJ) or weghted vote. The rando tran/test splt s 2/8 and experents are repeated twenty tes and averaged. Note that, for each dataset the class wth ost patterns s assgned to gve the sae sgn to γ n (7). Bas/Varance wll refer to / loss functon usng Brean s decoposton [3], for whch Bas plus Varance plus Bayes equals the base classfer error rate. Bas s ntended to capture the systeatc dfference wth Bayes, and requres Bayes probablty. Patterns are dvded nto two sets, the Bas set contanng patterns for whch the Bayes classfcaton dsagrees wth the enseble classfer and the Unbas set contanng the reander. Bas s coputed usng the Bas Set and Varance s coputed usng the Unbas Set, but both Bas and Varance are defned as the dfference between the probabltes that the Bayes and base classfer predct the correct class label. The Bayes estaton s perfored for 9/ splt usng orgnal features, and a Support Vector Classfer (SVC) wth polynoal kernel run tes. The polynoal degree and regularsaton constant are vared, and lowest test error s gven n Table 2. Fgure 3 gves ean results over seven datasets, whch clearly ndcates the overall trend. Fgure 3 (a) (b) shows base and enseble (MAJ) test error rates. Fgure 3 (c)- (f) shows dfference between MAJ and varous weghted vote schees. Fgure 3 (c) uses the frst order Walsh coeffcent (WD) n (3), Fgure 3 (d) s the proposed schee (WP) usng (9), Fgure 3 (e) uses the logarthc weghtng schee used n Adaboost (ADA) [4]. Fgure 3 (f) uses a traned lnear perceptron (LIN) to learn the appng. All the weghtng schees gve a large proveent over MAJ at epoch, the best beng WP, wth a 3 percent proveent at 64 nodes. The best MAJ error occurs at 3-4 epochs, and here there s a sall proveent WP over MAJ of between.3 percent at 64 nodes and percent at 4 nodes. Fg. 4 shows varous easures to help explan the results. Fg 4 (a) shows the ean second order coeffcents (S2M), noralsed by the total nuber of tranng patterns,

6 and whch s an estate of the ean added error n (8). Fgure 4 (b) s slar to (a) but shows coeffcents weghted by (9) (for classfer and j, weght s gven by ( w w j ) 2 ). Fgure 4 (c) (f) show bas and varance for MAJ (Bas, Var) and WP (BasW, VarW). By coparng Fgure 4 (a) and (b) the weghted coeffcents (S2W) shows that weghted classfers have reduced the Added Error. The Weghted bas (BasW) n (d) s reduced n coparson wth Bas n (c). For 64 nodes, the best weghted error rate s at epoch, shown n (d), whch s wthn percent of Bayes rate. On the other hand, at epoch Fgure 4 (e) (f) show that weghted varance has ncreased, ndcatng that ore dverse classfers are weghted. Optu Boundary kth Classfer Boundary E k Pˆ( P( P( Pˆ( Class class x~ b x b x Fgure : Model of error regon assocated wth a posteror probabltes showng optu (Bayes) boundary, kth classfer boundary wth Added Classfcaton Error (E k ) th Classfer jth Classfer E j P( P( class class Fgure 2: Model showng par of classfer boundares and the dfference n Added Classfcaton Error between th and jth classfers E (area 2) j

7 7 27/8/22 :4 Fgure 3: Mean test errors over 2-class datasets for [4,8,6,32,64] nodes -5 epochs (a) Base Classfer (b) Majorty Vote (c) (f) Weghted votes wth MAJ subtracted CONCLUSION For two-class supervsed learnng probles, the spectral representaton of the appng between bnary base classfer decsons and target class has been analysed wth the help of the Tuer-Ghosh odel of Added Classfcaton Error. If the ajorty vote s weghted by the product of the class-dependent frst-order coeffcents, the enseble has error rate that s close to optal, even wth fast naccurate base classfers.

8 Fgure 4: (a) Mean easures over 2-class datasets for [4,8,6,32,64] nodes -5 epochs (a) Second order coeffcents (b) Weghted Second order coeffcents (c) Bas (d) Bas WP (e) Varance (f) Varance WP Table : Areas under Dstrbuton defned n Fg. 2, showng correspondng nuber of class ω, ω patterns ( st subscrpt) for whch the par of classfers agree or dsagree (2 nd subscrpt) a a 2 a 3 a 4 a 5 a 6 a 7 a 8 ω n n n n n n n ω n n n n n

9 9 27/8/22 :4 Table 2: Datasets showng # patterns, pror probablty ω, #contnuous and dscrete features and estated Bayes error DATASET #pat p #con #ds %Bay cancer card credta dabetes heart on vote References [] R. S. Sth and T. Wndeatt, A Bas-Varance Analyss of Bootstrapped Class- Separablty Weghtng for ECOC Ensebles, n Proceedngs of the 22 nd Internatonal Conference on Pattern Recognton, Istanbul, Turkey, Aug. 2. [2] T. Wndeatt, Vote Countng Measures for Enseble Classfers, Pattern Recognton, vol. 36, no. 2, pp , 23. [3] T. Wndeatt and C. Zor, Mnsng Added Classfcaton Error usng Walsh Coeffcents, IEEE Trans Neural Networks, vol. 22 no. 8, pp , 2. [4] T. Wndeatt, Accuracy/ Dversty and enseble classfer desgn, IEEE Trans Neural Networks, vol. 7, pp. 94-2, Sept. 26. [5] L. Hurst, D. M. Mller, and J. Muzo, Spectral Technques n Dgtal Logc, Acadec Press, 985. [6] K. G. Beauchap, Walsh Functons and ther Applcatons, Acadec Press, 975. [7] K. Tuer and J. Ghosh, Error correlaton and error reducton n enseble classfers, Connecton Scence, vol. 8, no. 3, pp , 996. [8] A. N. Tkhonov and V. A. Arsenn, Solutons of Ill-posed Probles, Wnston & Sons, Washngton, 977. [9] B. J. Falkowsk, M.A.Perkowsk, Effectve Coputer Methods for the Calculaton of Radeacher-Walsh Spectru for Copletely and Incopletely Specfed Boolean Functons. IEEE Trans. on Coputer-Aded Desgn (), , 992. [] L. I. Kuncheva, Cobnng Pattern Classfers, Wley, 24. [] L. Prechelt, Proben: A set of neural network Benchark Probles and Bencharkng Rules, Tech Report 2/94, Unv. Karlsruhe, Gerany, Sept., 994. [2] C. J. Merz and P. M. Murphy, UCI repostory of ML databases, [Onlne], [3] L. Brean, Arcng Classfers, The Annals of Statstcs, vol. 26, no. 3, pp , 998. [4]Y Freund and R. E. Shapre, A decson-theoretc generalzaton of on-lne learnng and ts applcaton to Boostng, LNCS vol. 94, pp 23-37, 995.