A Classifier Ensemble of Binary Classifier Ensembles

Transcription

1 Internatonal Journal of Electroncs Communcaton and Comuter Technology (IJECCT) Volume 1 Issue 1 Setember 2011 A Classfer Ensemble of Bnary Classfer Ensembles Hamd Parvn Comuter Engneerng, Islamc Azad Unversty, Nourabad Mamasan Branch, Nourabad, Iran hamdarvn@mamasanauacr Sajad Parvn Comuter Engneerng Islamc Azad Unversty, Nourabad Mamasan Branch, Nourabad, Iran sarvn@mamasanauacr Abstract Ths aer rooses an nnovatve combnatonal algorthm to mrove the erformance n multclass classfcaton domans Because the more accurate classfer the better erformance of classfcaton, the researchers n comuter communtes have been tended to mrove the accuraces of classfers Although a better erformance for classfer s defned the more accurate classfer, but turnng to the best classfer s not always the best oton to obtan the best ualty n classfcaton It means to reach the best classfcaton there s another alternatve to use many naccurate or weak classfers each of them s secalzed for a sub-sace n the roblem sace and usng ther consensus vote as the fnal classfer So ths aer rooses a heurstc classfer ensemble to mrove the erformance of classfcaton learnng It s secally deal wth multclass roblems whch ther am s to learn the boundares of each class from many other classes Based on the concet of multclass roblems classfers are dvded nto two dfferent categores: arwse classfers and multclass classfers The am of a arwse classfer s to searate one class from another one Because of arwse classfers just tran for dscrmnaton between two classes, decson boundares of them are smler and more effectve than those of multclass classfers The man dea behnd the roosed method s to focus classfer n the erroneous saces of roblem and use of arwse classfcaton concet nstead of multclass classfcaton concet Indeed although usage of arwse classfcaton concet nstead of multclass classfcaton concet s not new, we roose a new arwse classfer ensemble wth a very lower order In ths aer, frst the most confused classes are determned and then some ensembles of classfers are created The classfers of each of these ensembles jontly work usng majorty weghtng votes The results of these ensembles are combned to decde the fnal vote n a weghted manner Fnally the oututs of these ensembles are heurstcally aggregated The roosed framework s evaluated on a very large scale Persan dgt handwrtten dataset and the exermental results show the effectveness of the algorthm Keywords-Genetc Algorthm; Otcal Character Recognton; Parwse Classfer; Multclass Classfcaton I INTRODUCTION Usage of recognton systems has found many alcatons n almost all felds However, most of classfcaton algorthms have obtaned good erformance for secfc roblems; but they have not enough robustness for other roblems Combnaton of multle classfers can be consdered as a general soluton method for any attern recognton roblems It has been shown that combnaton of classfers can usually oerate better than sngle classfer rovded that ts comonents are ndeendent or they have dverse oututs It has shown that the necessary dversty of an ensemble can be acheved by manulaton of data set features Parvn et al have roosed some methods of creatng ths dversty [12]- [13] In ractce, there may be roblems that one sngle classfer can t delver a satsfactory erformance [7]-[9] In such stuatons, emloyng an ensemble of classfyng models nstead of a sngle classfer can reach the model to a better learnng [6] Although obtanng the more accurate classfer s often targeted, there s an alternatve way to reach for t Indeed one can use many naccurate or weak classfers each of whch s secalzed for a few data tems n the roblem sace and then he can emloy ther consensus vote as the classfcaton Ths can lead to better erformance due to renforcement of the classfer n error-rone roblem saces Based on the concet of multclass roblem, classfers are dvded nto two dfferent categores: arwse classfers and multclass classfers Whle the am of multclass roblems s to learn the boundares of each class from many other classes, the am of a arwse classfer s to searate one class from another one Because arwse classfers are just traned to learn the boundary between two classes, decson boundares roduced by them are smler and more effectve than those roduced by multclass classfers Parwse dscrmnaton between classes has been suggested n [16]-[18] In ths model there are c*(c-1)/2 ossble arwse classfcatons, one for each ar of classes The class label for an nut x s nferred from the smlarty between the code words and the oututs of the classfers The code word for class wll contan don t care symbols to denote the classfers that are not concerned wth ths class label Ths method s mractcal for a large c as the number of classfers becomes rohbtve ISSN: IJECCT wwwjecctorg 1

2 Internatonal Journal of Electroncs Communcaton and Comuter Technology (IJECCT) Volume 1 Issue 1 Setember 2011 In General, t s ever-true sentence that "combnng the dverse classfers any of whch erforms better than a random results n a better classfcaton erformance" [2], [6] and [10] Dversty s always consdered as a very mortant concet n classfer ensemble methodology It s consdered as the most effectve factor n succeedng an ensemble The dversty n an ensemble refers to the amount of dfferences n the oututs of ts comonents (classfers) n decdng for a gven samle Assume an examle dataset wth two classes Indeed the dversty concet for an ensemble of two classfers refers to the robablty that they may roduce two dssmlar results for an arbtrary nut samle The dversty concet for an ensemble of three classfers refers to the robablty that one of them roduces dssmlar result from the two others for an arbtrary nut samle It s worthy to menton that the dversty can converge to 05 and 066 n the ensembles of two and three classfers resectvely Although reachng the more dverse ensemble of classfers s generally handful, t s harmful n boundary lmt It s very mortant dlemma n classfer ensemble feld: the ensemble of accurate/dverse classfers can be the best It means that although the more dverse classfers, the better ensemble, t s rovded that the classfers are better than random An Artfcal Neural Network (ANN) s a model whch s to be confgured to be able to roduce the desred set of oututs, gven an arbtrary set of nuts An ANN generally comosed of two basc elements: (a) neurons and (b) connectons Indeed each ANN s a set of neurons wth some connectons between them From another ersectve an ANN contans two dstnct vews: (a) toology and (b) learnng The toology of an ANN s about the exstence or nonexstence of a connecton The learnng n an ANN s to determne the strengths of the toology connectons One of the most reresentatves of ANNs s MultLayer Percetron Varous methods of settng the strength of connectons n an MLP exst One way s to set the weghts exlctly, usng a ror knowledge Another way s to 'tran' the MLP, feedng t by teachng atterns and then lettng t change ts weghts accordng to some learnng rule In ths aer the MLP s used as one of the base classfers Decson Tree (DT) s consdered as one of the most versatle classfers n the machne learnng feld DT s consdered as one of unstable classfers It means that t can converge to dfferent solutons n successve tranngs on same dataset wth same ntalzatons It uses a tree-lke grah or model of decsons The knd of ts knowledge reresentaton s arorate for exerts to understand what t does [11] Its ntrnsc nstablty can be emloyed as a source of the dversty whch s needed n classfer ensemble The ensemble of a number of DTs s a well-known algorthm called Random Forest (RF) whch s consdered as one of the most owerful ensemble algorthms The algorthm of RF was frst develoed by Breman [1] In a revous work, Parvn et al have only dealt wth the reducng the sze of classfer ensemble [9] They have shown that one can reduce the sze of an ensemble of arwse classfers Indeed they roose a method for reducng the ensemble sze n the best meanngful manner Here we nsre from ther method, we roose a framework based on that a set of classfer ensembles are roduced that ts sze order s not mortant Indeed we roose an ensemble of bnary classfer ensembles that has the order of c, where c s number of classes Ths aer rooses a framework to develo combnatonal classfers In ths new aradgm, a multclass classfer n addton to a few ensembles of arwse classfers creates a classfer ensemble At last, to roduce fnal consensus vote, dfferent votes (or oututs) are gathered, after that a heurstc classfer ensemble algorthm s emloyed to aggregate them We focus on Persan handwrtten dgt recognton (PHDR), esecally on Hoda dataset [4] Although there are well works on PHDR, t s not ratonal to comare them wth each other, because there was no standard dataset n the PHDR feld untl 2006 [4] The contrbuton s only comared wth those used the same dataset used n ths aer, e Hoda dataset II ARTIFICIAL NEURAL NETWORK A frst wave of nterest n ANN (also known as 'connectonst models' or 'arallel dstrbuted rocessng') emerged after the ntroducton of smlfed neurons by McCulloch and Ptts n 1943 These neurons were resented as models of bologcal neurons and as concetual comonents for crcuts that could erform comutatonal tasks Each unt of an ANN erforms a relatvely smle job: receve nut from neghbors or external sources and use ths to comute an outut sgnal whch s roagated to other unts Aart from ths rocessng, a second task s the adjustment of the weghts The system s nherently arallel n the sense that many unts can carry out ther comutatons at the same tme Wthn neural systems t s useful to dstngush three tyes of unts: nut unts (ndcated by an ndex ) whch receve data from outsde the ANN, outut unts (ndcated by an ndex o) whch send data out of the ANN, and hdden unts (ndcated by an ndex h) whose nut and outut sgnals reman wthn the ANN Durng oeraton, unts can be udated ether synchronously or asynchronously Wth synchronous udatng, all unts udate ther actvaton smultaneously; wth asynchronous udatng, each unt has a (usually fxed) robablty of udatng ts actvaton at a tme t, and usually only one unt wll be able to do ths at a tme In some cases the latter model has some advantages An ANN has to be confgured such that the alcaton of a set of nuts roduces the desred set of oututs Varous methods to set the strengths of the connectons exst One way s to set the weghts exlctly, usng a ror knowledge Another way s to 'tran' the ANN by feedng t teachng atterns and lettng t change ts weghts accordng to some learnng rule For examle, the weghts are udated accordng to the gradent of the error functon For further study the reader must refer to an ANN book such as Haykn's book on theory of ANN [3] ISSN: IJECCT wwwjecctorg 2

3 Internatonal Journal of Electroncs Communcaton and Comuter Technology (IJECCT) Volume 1 Issue 1 Setember 2011 III DECISION TREE LEARNING DT as a machne learnng tool uses a tree-lke grah or model to oerate decdng on a secfc goal DT learnng s a data mnng technue whch creates a model to redct the value of the goal or class based on nut varables Interor nodes are the reresentatve of the nut varables and the leaves are the reresentatve of the target value By slttng the source set nto subsets based on ther values, DT can be learned Learnng rocess s done for each subset by recursve arttonng Ths rocess contnues untl all reman features n subset has the same value for our goal or untl there s no mrovement n Entroy Entroy s a measure of the uncertanty assocated wth a random varable Fgure 1 An exemlary raw data Data comes n records of the form: (x,y) = (x 1, x 2, x 3,, x n,y) The deendent varable, Y, s the target varable that we are tryng to understand, classfy or generalze The vector x s comosed of the nut varables, x 1, x 2, x 3 etc, that are used for that task To clarfy that what the DT learnng s, consder Fgure 1 Fgure 1 has 3 attrbutes Refund, Martal Status and Taxable Income and our goal s cheat status We should recognze f someone cheats by the hel of our 3 attrbutes To do learn rocess, attrbutes slt nto subsets Fgure 2 shows the rocess tendency Frst, we slt our source by the Refund and then MarSt and TaxInc For makng rules from a decson tree, we must go uward from leaves as our antecedent to root as our conseuent For examle consder Fgure 2 Rules such as followng are arehensble We can use these rules such as what we have n Assocaton Rule Mnng Fgure 2 The rocess tendency for Fgure 1 Refund=Yescheat=No TaxInc<80, MarSt= (Sngle or Dvorce), Refund=Nocheat=No TaxInc>80, MarSt= (Sngle or Dvorce), Refund=Nocheat=Yes Refund=No, MarSt=Marredcheat=No IV K-NEAREST NEIGHBOR ALGORITHM k-nearest neghbor algorthm (k-nn) s a method for classfyng objects based on closest tranng examles n the feature sace k-nn s a tye of nstance-based learnng, or lazy learnng where the functon s only aroxmated locally and all comutaton s deferred untl classfcaton The k- nearest neghbor algorthm s amongst the smlest of all machne learnng algorthms: an object s classfed by a majorty vote of ts neghbors, wth the object beng assgned to the class most common amongst ts k nearest neghbors (k s a ostve nteger, tycally small) If k = 1, then the object s smly assgned to the class of ts nearest neghbor As t s obvous, the k-nn classfer s a stable classfer A stable classfer s the one converge to an dentcal classfer aart from ts tranng ntalzaton It means the 2 consecutve tranngs of the k-nn algorthm wth dentcal k value, results n two classfers wth the same erformance Ths s not vald for the MLP and DT classfers We use 3-NN as a base classfer n the aer It s then nferred that usng a k-nn classfer n an ensemble s not a good oton V PROPOSED ALGORITHM The man dea behnd the roosed method s to use a number of arwse classfers to renforce the man classfer n the error-rone regons of the roblem sace Fgure 3 dects the tranng hase of the roosed method schematcally In the roosed algorthm, a multclass classfer s frst traned Its duty s to roduce a confuson matrx over the valdaton set Note that ths classfer s traned over the total tran set At next ste, the ar-classes whch are mostly confused wth each other and are also mostly error-rone are detected After that, a number of arwse classfers are emloyed to renforce the drawbacks of the man classfer n those error-rone regons A smle heurstc s used to aggregate ther oututs At the frst ste, a multclass classfer s traned on all tran data Then, usng the results of ths classfer on the valdaton data, confuson matrx s obtaned Ths matrx contans mortant nformaton about the functonaltes of classfers n the dataset localtes The close and Error-Prone Par-Classes (EPPC) can be detected usng ths matrx Indeed, confuson matrx determnes the between-class error dstrbutons Assume that ths matrx s denoted by a Item a j of ths matrx determnes how many nstances of class c j have been msclassfed as class c ISSN: IJECCT wwwjecctorg 3

4 Internatonal Journal of Electroncs Communcaton and Comuter Technology (IJECCT) Volume 1 Issue 1 Setember 2011 Fgure 4 shows the confuson matrx obtaned from the base multclass classfer As you can see, dgt 5 (or euvalently class 6) s ncorrectly recognzed as dgt 0 ffteen tmes (or euvalently class 1), and also dgt 0 s ncorrectly recognzed as dgt 5 fourteen tmes It means 29 msclassfcatons have totally occurred n recognton of these two dgts (classes) The mostly erroneous ar-classes are resectvely (2, 3), (0, 5), (3, 4), (1, 4), (6, 9) and so on accordng to ths matrx Assume that the -th mostly EPPC s denoted by EPPC So EPPC 1 wll be (2, 3) Also assume that the number of selected EPPC s denoted by k tran Tranng Dataset Multclass Classfer Test Confuson Matrx Selecton of EPP Error-Prone Parclasses Valdaton Dataset b% selecton Data Bag 1 PWC1,1 on 1 st EPP P 1,1 Data Bag m PWC1,m on 1 st EPP P 1,m PWC1 Data Bag 1 Data Bag m PWCk,1 on k st EPP PWCk,m on k st EPP P k,1 P k,m PWC,j: jth classfer of th arwse classfer ensemble secalzed for th error-rone arclass P,j: accuracy of jth classfer n PWC ensembles PWCk Fgure 3 The frst tranng hase of the roosed method After determnng the mostly erroneous ar-classes, or EPPCs, a set of m ensembles of bnary classfers s to be traned to jontly, as an ensemble of bnary classfers, renforce the man multclass classfer n the regon of each EPPC So as t can be nferred, t s necessary to tran k ensembles of m bnary classfers Assume that the ensemble whch s to renforce the man multclass classfer n the regon of EPPC s denoted by PWC Each bnary classfer contaned n PWC, s traned over a bag of tran data lke RF The bags of tran data contan only b ercet of the randomly selected of tran data It s worthy to be mentoned that arwse classfers whch are to artcate n PWC are traned only on those nstances whch belongs to EPPC Assume that the j-th classfer bnary classfer of PWC s denoted by PWC,j Because there exsts m classfers n each of PWC and also there exsts k EPPC, so there wll be k*m bnary classfers totally For examle n Fgure 4 the EPPC (2, 3) can be consdered as an erroneous ar-class So a classfer s necessary to be traned for that EPPC usng those datatems of tran data that belongs to class 2 or class 3 As mentoned before, ths method s flexble, so we can add arbtrary number of PWC to the base rmary classfers It s exected that the erformance of the roosed framework outerforms the rmary base classfer It s worthy to note that the accuraces of PWC,j can easly be aroxmated usng the tran set Because PWC,j s traned only on b ercet of the tran set wth labels belong to EPPC, rovded that b s very small rate, then the accuracy of PWC,j on the tran set wth labels belong to EPPC can be consdered as ts aroxmated accuracy Assume that the mentoned aroxmated accuracy of PWC,j s denoted by P,j Fgure 4 The rocess tendency for Fgure 1 ISSN: IJECCT wwwjecctorg 4

5 Internatonal Journal of Electroncs Communcaton and Comuter Technology (IJECCT) Volume 1 Issue 1 Setember 2011 It s mortant to note that each of PWC acts as a bnary classfer As t mentoned each PWC contans m bnary classfers wth an accuracy vector, P It means of these bnary ensemble can take a decson wth weghed sum algorthm llustrated n [5] So we can combne ther results accordng to weghs comuted by (1) where w,j s the accuracy of j-th classfer n the -th bnary ensemble It s roved that the weghts obtaned accordng to the (1) are otmal weghts n theory Now the two oututs of each PWC are comuted as (2), j w, j log( ) 1, j where x s a test data m PW C ( x h) w j1, j * PW C, j ( x h), h EPPC Test nstance Multclass Classfer Multclass Classfer decdes PWC1 PWC1,1 on 1 st EPP w 1,1 w 1,m Mean MPWC1 NO PWC1,m on 1 st EPP Max Abs(Maxval) > thr PWCk,1 on 1 st EPP PWCk,m on 1 st EPP w k,1 w k,m Mean MPWCk YES Max decdes P,j: accuracy of jth classfer n PWC ensembles w,j=log(,j/(1-,j)) thr s threshold for decson PWCk Fgure 5 Heurstc test hase of the roosed method test aggregators The Outut of aggregator s the fnal jont outut for class Here, the aggregaton s done usng a secal, j w, j log( ) heurstc method Ths rocess s done usng a heurstc based 1, j ensemble whch s llustrated n the Fgure 5 As the Fgure 5 shows, after roducng the ntermedate sace, the oututs of - where w,j s the accuracy The last ste of the roosed th ensemble of bnary classfer are multled n a number framework s to combne the results of the man multclass Ths number s eual to the sum of the man multclass classfer and those of PWC It s worthy to note that there are classfer's confdences for the classes belong to EPPC 2*k oututs from the bnary ensembles lus c oututs of the Assume that the results of the multlcaton of by the man multclass classfer So the roblem s to ma a 2*k+c oututs of PWC are denoted by MPWC It s mortant to ntermedate sace to a c sace each of whch corresonds to a note that MPWC s a vector of two confdences; the class The results of all these classfers are fed as nuts n the ISSN: IJECCT wwwjecctorg 5

6 Internatonal Journal of Electroncs Communcaton and Comuter Technology (IJECCT) Volume 1 Issue 1 Setember 2011 confdences of the classfer framework to the classes belongng to PWC After calculatng the MPWC, the value s selected between all of them If the framework's confdence for the most confdent class s satsfactory for a test data, then t s selected for fnal decson of framework, else the man multclass classfer decdes for the data It means that the fnal decson s taken by (3) MaxDecson ( x) Decson( x) ( MCC( h x)) h{1,, c} ( EPPC (, ) x ) ( MCC( x) MCC( x)) * ( MPW Csc ( h x)) thr ( PWC( x), PWC( x)) otherwse where w,j s the accuracy heppcsc where MCC(h x) s the confdence of the man multclass classfer for the class h gven a test data x MPWC sc (h x) s the confdence of the sc-th ensemble of bnary classfers for the class h gven a test data x MaxDecson s calculated accordng to (4) where s a fxed value and then we have: MaxDecson ( x) arg ( MPWC ( h x)) where sc s comuted as (5) heppcsc sc dstngush two classes: r and They can be s obtaned by (7) and (8) resectvely r ( EPPC (, r) x ) ( MCC( x) MCC( r x)) * ( PWC( x), PWC( r x)) We can assume (9) wthout losng generalty r ( PWC( x ), PWC( r x )) ( PWC( x ), PWC( x )) r ( EPPC (, r) x ) ( MCC( x) MCC( r x)) ( b b ), r, sc( x) arg ( ( MPW C( h x))) heppc Because of the renforcement of the man classfer by some ensembles n erroneous regons, t s exected that the accuracy of ths method outerforms a smle MLP or unweghted ensemble Fgure 3 along wth Fgure 5 stands as the structure of the ensemble framework ( EPPC (, ) x ) ( MCC( x) MCC( x)) ( b b ),, As t s nferred from the algorthm n the same condton, ts error can be formulated as follow VI WHY PROPOSED METHOD WORKS As we resume n the aer, t s amed to add as many as arwse classfers to comensate a redefned error rate, PDER*EF(MCL,DValdaton), where PDER s a redefned error rate and EF(MCL,DValdaton) s error freuency of multclass classfer, MCL, over the valdaton data, DValdaton Assume we add EPS arwse classfers to the man MLC It s as n the euaton below es 1 ( ( wˆ EPPC x w EPPC y, x) ( wˆ EPPC y w EPPC x, x)) PDER * EF( MCL, DValdaton, DTran) Now assume that a data nstance x whch belongs really to class s to be classfed by the roosed algorthm; t has the error rate whch can be obtan by (12) Frst assume s robablty for the roosed classfer ensemble to take decson by one of ts bnary classfers that s able to dstngush two classes: and Also assume r s robablty for the roosed classfer ensemble to take decson by one of ts bnary classfers that s able to error ( x w ) r EPPC(, r ) EPPC(, ) ( EPPC x) (1 ( EPPC x)* )(1 b ( x) confuson j, bj, c confuson r ar, ) where ar s robablty of takng correct decson by bnary classfer and b j, s defned as follow 1, So we can reformulate (12) as follow ISSN: IJECCT wwwjecctorg 6

7 Internatonal Journal of Electroncs Communcaton and Comuter Technology (IJECCT) Volume 1 Issue 1 Setember 2011 error( x w ) r EPPC (, r) EPPC (, ) (1 ( EPPC x) (1 ( EPPC x)* r EPPC (, ) )(1 b, ) ar ( EPPC x)* ( x) r ar )(1 b ( x), ) Note that n (14) f r and r are zero for an exemlary nut the error of classfcaton wll be stll eual to the man multclass classfer If they are not zero for an exemlary nut the msclassfcaton rate wll stll be reduced because of reducton n second art of (14) Although the frst art ncreases the error n (14), but f we assume that the bnary classfers are more accurate than the multclass classfer, then the ncrease s nullfed by the decrease art 20 ercet of the tran set wth the corresondng classes The cardnalty of ths set s calculated by (15) Car tran * 2* b / c 60000*2*02 / It means that each bnary classfer s traned on 2400 dataonts wth 2 class labels Table 1 shows the exermental results comaratvely As t s nferred the framework s outerforms the revous works and the smle classfers n the case of emloyng decson tree as the base classfer VII EXPERIMENTAL RESULTS Ths secton evaluates the results of alyng the roosed framework on a Persan handwrtten dgt dataset named Hoda [4] Ths dataset contans 102,364 nstances of dgts 0-9 Dataset s dvded nto 3 arts: tran, evaluaton and test sets Tran set contans 60,000 nstances Evaluaton and test datasets are contaned 20,000 and 22,364 nstances The 106 features from each of them have been extracted whch are descrbed n [4] Some nstances of ths dataset are dected n Fgure 6 In ths aer, MLP, 3-NN and DT are used as base rmary classfer We use an MLPs wth 2 hdden layers ncludng resectvely 10 and 5 neurons n the hdden layer 1 and 2, as the base Multclass classfer Confuson matrx s obtaned from ts outut Also DT s measure of decson s taken as Gn measure The classfers arameters are ket fxed durng all of ther exerments It s mortant to take a note that all classfers n the algorthm are ket unchanged It means that all classfers are consdered as MLP n the frst exerments After that the same exerments are taken by substtutng all MLPs wht DTs The arameter k s set to 11 So, the number of arwse ensembles of bnary classfers added euals to 11 n the exerments The arameter m s also set to 9 So, the number of bnary classfers er each EPPC euals to 9 n the exerments It means that 99 bnary classfers are traned for the ar-classes that have consderable error rates Assume that the error number of each ar-class s avalable For choosng the most erroneous ar-classes, t s suffcent to sort error numbers of ar-classes Then we can select an arbtrary number of them Ths arbtrary number can be determned by try and error whch t s set to 11 n the exerments As mentoned 9*11=99 arwse classfers are added to man multclass classfer As the arameter b s selected 20, so each of these classfers s traned on only b recets of corresondng tran data It means each of them s traned over Fgure 6 Some nstances of Persan OCR data set, wth dfferent ualtes It s nferred from Table 1 that the roosed framework affects sgnfcantly n mrovng the classfcaton recson secally when emloyng DT as base classfer Takng a look at Table 1 shows that usng DT as base classfer n ensemble almost always roduces a better erformng classfcaton It may be due to nherent nstablty of DT It means that because a DT s unstable classfer, so t s better to use t as a base classfer n an ensemble A stable classfer s the one converge to an dentcal classfer aart from ts tranng ntalzaton It means the 2 consecutve tranngs of the classfer wth dentcal ntalzatons, results n two classfers wth the same erformance Ths s not vald for the MLP and DT classfers Although MLP s not a stable classfer, t s more stable than DT So t s also exected that usng DT classfer as base classfer has the most mact n mrovng the recognton rato As another ont to be mentoned, reader can nfer that usng the framework can outerforms Unweghted Full Ensemble, Unweghted Statc Classfer Selecton and Unweghted Statc Classfer Selecton methods exlaned n [14] Ths can be n conseuence of emloyng bnary classfers nstead of multclass classfers It s nferred from the Table 1 that the roosed framework affects sgnfcantly n mrovng the classfcaton recson secally when emloyng DT and MLP as base classfer It s also obvous that usng DT classfer as base classfer has the most mact n mrovng the recognton rato It s may be due to ts nherent nstablty ISSN: IJECCT wwwjecctorg 7

8 Internatonal Journal of Electroncs Communcaton and Comuter Technology (IJECCT) Volume 1 Issue 1 Setember 2011 As t s exected usng a stable classfer lke k-nn n an ensemble s not a good oton and unstable classfers lke DT and MLP are better otons VIII CONCLUSION Although the more accurate classfer leads to a better erformance, there s another oton to use many naccurate classfers whle each one s secalzed for a few data n the roblem sace and usng ther consensus vote as the classfer So ths aer rooses a heurstc classfer ensemble to mrove the erformance of learnng n multclass roblems The man dea behnd the roosed method s to focus classfer n the erroneous saces of the roblem The new roosed method tres to mrove the erformance of multclass classfcaton system We also roose a framework based on that a set of classfer ensembles are roduced that ts sze order s not mortant It means that we roose a new arwse classfer ensemble wth a very lower order than usage of all ossble arwse classfers Indeed aer rooses an ensemble of bnary classfer ensembles that has the order of c, where c s number of classes So frst an arbtrary number of bnary classfer ensembles are added to man classfer Then results of all these bnary classfer ensembles are gven to a set of a heurstc based ensemble The results of these bnary ensembles ndeed are combned to decde the fnal vote n a weghted manner The roosed framework s evaluated on a very large scale Persan dgt handwrtten dataset and the exermental results show the effectveness of the algorthm Usage of confuson matrx make roosed method a flexble one The number of all ossble arwse classfers s c*(c-1)/2 that t s O(c^2) Usng ths method wthout gvng u a consderable accuracy, we decrease ts order to O(1) Ths feature of our roosed method makes t alcable for roblems wth a large number of classes The exerments show the effectveness of ths method Also we reached to very good results n Persan handwrtten dgt recognton whch s a very large dataset TABLE I Methods A smle multclass classfer Method Proosed n [8] Method Proosed n [7] Method Proosed n [15] Unweghted Full Ensemble n [14] Unweghted Statc Classfer Selecton n [14] THE ACCURACIES OF DIFFERENT SETTINGS OF THE PROPOSED FRAMEWORK Base Classfer DT MLP 3-NN Weghted Statc Classfer Selecton n [14] Proosed Method It s concluded that usng a stable classfer lke k-nn n an ensemble s not a good oton and unstable classfers lke DT and MLP are better otons REFERENCES [1] L Breman, "Baggng Predctors," Journal of Machne Learnng, Vol 24, no 2, , 1996 [2] S Gunter, and H Bunke, "Creaton of classfer ensembles for handwrtten word recognton usng feature selecton algorthms," IWFHR 2002 on January 15, 2002 [3] S Haykn, Neural Networks, a comrehensve foundaton, second edton, Prentce Hall Internatonal, 1999 [4] H Khosrav, and E Kabr, "Introducng a very large dataset of handwrtten Fars dgts and a study on the varety of handwrtng styles," Pattern Recognton Letters, vol 28 ssue , 2007 [5] LI Kuncheva, Combnng Pattern Classfers, Methods and Algorthms, New York: Wley, 2005 [6] B Mnae-Bdgol, and WF Punch, "Usng Genetc Algorthms for Data Mnng Otmzaton n an Educatonal Web-based System," GECCO, 2003 [7] H Parvn, H Alzadeh and B Mnae-Bdgol, "A New Aroach to Imrove the Vote-Based Classfer Selecton," Internatonal Conference on Networked Comutng and advanced Informaton Management, 2008 [8] H Parvn, H Alzadeh, M Fath, B Mnae-Bdgol, "Imroved Face Detecton Usng Satal Hstogram Features," Int Conf on Image Processng, Comuter Vson, and Pattern Recognton, , 2008 [9] H Parvn, H Alzadeh, B Mnae-Bdgol, M Analou, "An Scalable Method for Imrovng the Performance of Classfers n Multclass Alcatons by Parwse Classfers and GA," Internatonal Conference on Networked Comutng and advanced Informaton Management, , 2008 [10] A Saber, M Vahd, B Mnae-Bdgol, "Learn to Detect Phshng Scams Usng Learnng and Ensemble Methods," IEEE/WIC/ACM Internatonal Conference on Intellgent Agent Technology, Workshos, , 2007 [11] T Yang, "Comutatonal Verb Decson Trees," Internatonal Journal of Comutatonal Cognton, 34 46, 2006 [12] H Parvn, H Alzadeh, B Mnae-Bdgol, "Usng Clusterng for Generatng Dversty n Classfer Ensemble," JDCTA Vol 3, no 1, 51-57, 2009 [13] H Parvn, H Alzadeh, B Mnae-Bdgol, "A New Method for Constructng Classfer Ensembles," JDCTA Vol 3, no 2, 62-66, 2009 [14] H Parvn, H Alzadeh, "Classfer Ensemble Based Class Weghtng," Amercan Journal of Scentfc Research, 84-90, 2011 [15] H Parvn, H Alzadeh, M Moshk, B Mnae-Bdgol, N Mozayan, "Dvde & Conuer Classfcaton and Otmzaton by Genetc Algorthm," Internatonal Conference on Convergence and hybrd Informaton Technology, , 2008 [16] F Masull, and G Valentn, "Comarng decomoston methods for classfcaton," In Proc Internatonal Conference on Knowledge-Based Intellgent Engneerng Systems and Aled Technologes, , 2000 ISSN: IJECCT wwwjecctorg 8

9 Internatonal Journal of Electroncs Communcaton and Comuter Technology (IJECCT) Volume 1 Issue 1 Setember 2011 [17] F Cutzu, "Polychotomous classfcaton wth arwse classfers: A new votng rncle," In Proc Internatonal Worksho on Multle Classfer Systems, Lecture Notes n Comuter Scence, , 2003 [18] A Jozwk, and G Vernazza "Recognton of leucocytes by a arallel k- nn classfer," In Proc Internatonal Conference on Comuter-Aded Medcal Dagnoss, , 1987 ISSN: IJECCT wwwjecctorg 9