Improving Combination Methods of Neural Classifiers Using NCL

Transcription

1 Internatonal Journal of Computer Informaton Systems and Industral Management Applcatons. ISS Volume 4 (0) pp MIR Labs, Improvng Combnaton Methods of eural Classfers Usng CL Arash Iranzad, Saeed Masoudna, Fatemeh Cheraghch, Abbas owzar-daln and Reza Ebrahmpour Department of Computer Scence, School of Mathematcs, Statstcs and Computer Scence, Unversty of Tehran, Tehran, Iran {ranzad}@sad.ut.r, {masoudna; xeraghch; nowzar}@ut.ac.r Electrcal and Computer Engneerng Department Shahd Rajaee Unversty, Tehran, Iran ebrahmpour@pm.r Abstract: In ths paper the effect of dversty caused by egatve Correlaton Learnng (CL) n the combnaton of neural classfer s nvestgated and an effcent way to mprove combnng performance s presented. Decson Templates and Averagng, as two non-tranable combnng methods and Stacked Generalzaton as a tranable combner are selected as base ensemble learner and CL verson of them are compared wth them n our experments. Utlzng CL for dversfyng the base classfers leads to sgnfcantly better results n all employed combnng methods. Expermental results on fve datasets from UCI Machne Learnng Repostory ndcate that by employng CL, the performance of the ensemble structure can be more favorable compared to that of an ensemble use ndependent base classfers. Keywords: eural etworks, Combnng Classfers, egatve Correlaton Learnng. I. Introducton Combnng multple classfers s a machne learnng method that uses a set of base classfers and combne the output of them for producng the fnal output. Ths method s known under dfferent names n the lterature such as: classfer ensembles, fuson of learners, mxtures of experts, multple classfer systems, etc. In general pont of vew, classfer ensembles system s a data classfcaton method made up of an ensemble of base classfers whose outputs on an nput sample data are combned n some way to get a fnal output on ts classfcaton task [,, 3]. Classfer ensembles system may generate more accurate output than each of the base classfers. It has been proved that Classfer ensembles system s a sutable way to mprove the data classfcaton performance, partcularly for complex problems such as those nvolvng lmted number of sample patterns, hgh-dmensonal feature sets, and hghly overlapped classes [4-8]. Combnaton of multple classfers decsons s a promsng soluton to gan an acceptable classfcaton performance [9-]. By utlzng a proper strategy for the constructon of an ensemble network, t can be successfully appled to classfcaton problems wth mprecse and uncertan nformaton. An artfcal neural network (A), commonly called neural network (), s one of the most common classfers whch are used n multple classfer systems. A s an nformaton processng system that s nspred by bologcal nervous systems, and approxmates the operaton of the human bran,.e. a s a mathematcal and computatonal model that tres to smulate the structure, functonal aspects, and behavor of bologcal neural networks of the human. eural network s the major technque used n decson makng and classfcaton phase for problem solvng n the recent years [3-6]. A s able to learnng and generalzng from some samples and tres to produce meanngful solutons to data whch s never seen by t. It can solve the problems even when nput data of problem contan errors and are ncomplete. In most cases, a s an adaptve system that changes ts topology based on external or nternal nformaton that flows through the network durng the tranng phase. eural networks are n the class of non-lnear statstcal system modelng machnes. They can be used to model complex relatonshps between nputs data and outputs data or to fnd patterns and sgnals n nput data. A neural network may be traned to perform classfcaton, estmaton, approxmaton, smulaton, and predcton,.e. a can be confgure for a specfc applcaton, such as pattern fndng, data classfcaton, functon approxmaton through the learnng process. eural network ensembles methods have two major components, a method to create base experts and a method for combnng output of base experts [7]. Both theoretcal and expermental studes [8, 9] are shown that combnng procedure s the most effectve when the experts MIR Labs, USA

2 Iranzad, Masoudna, Cheraghch, owzar-daln and Ebrahmpour estmates are negatvely correlated; but ths method s moderately effectve when the experts are uncorrelated and only mldly effectve when the experts are postvely correlated. Therefore, more mproved generalzaton ablty can be obtaned by combnng the outputs of experts whch are accurate and ther errors are negatvely correlated [0]. A large number of combnng schemes for ensembles classfer exsts. As a general vewpont, there are two types of combnaton strategy: classfer selecton and classfer fuson []. The presumpton n classfer selecton s that each classfer s an expert n some local area of feature space; and one or more classfer are selected to assgn the label of the nput sample n ths area. Recall from [], classfer fuson assumes that all classfers traned over the whole feature space, and are thereby consders as compettve rather than complementary. Classfer selecton has not attracted as much attenton as classfer fuson. However, classfer selecton s probably the better of the two strateges, f traned well. There also exst combnatonal schemes between the two pure strateges []. Such a scheme; for example, s takng the average of outputs wth coeffcents whch depend on the nput x. Thus, the local competence of the classfers, wth respect to x, s measured by the weghts, and, more than one classfer s responsble for x and the outputs of all responsble classfers are fused. The mxture of experts combnng method s an example of a method between selecton and fuson [, 3]. On the other pont of vew, some combnng methods do not need tranng after the classfers n the ensemble have been traned ndvdually. The Averagng combner [4] and Decson Template (DT) [5] are two examples of ths group. Other combners need addtonal tranng, for example, the weghted average combner and Stacked Generalzaton (SG) [6]. The frst group s called non-tranable combnng and the second one s named tranable [7, ]. The Averagng method s a relatvely smple method of combnng models n non-tranable group. In ths method, each classfer s traned and outputs for each class are combned by averagng. Commonly, the fnal result s obtaned by just averagng of the estmated posteror probabltes of each base classfer. Ths smple method gves very good results for some problems [4]. Ths result s slghtly surprsng, especally consderng ths fact that the posteror probabltes averagng are not based on some theoretcal support. DTs are another non-tranable combnng technque n fuson category. A DT s a robust classfer ensemble scheme that the output of classfers s combned by comparng them to a characterstc template for each class. DT ensemble method uses outputs of all classfer to calculate the fnal output for each class, that s n hgh contrast to most other ensemble methods whch use only the support for that partcular class to make ther decson [7]. SG s tranable combner n fuson category. SG s a way of combnng multple classfers that have been learned for a classfcaton task. In ths method, the output pattern of an ensemble of traned experts serves as an nput to a second-level expert. The frst step s to collect the output of each frst level classfer, nto a new set. For each nstance n the orgnal tranng set, ths data set represents every classfer's predcton. The new data s treated as the nput for another classfer and n the second step a learnng algorthm s employed to solve ths problem [6]. In ensemble research, t s wdely beleved that the success of ensemble algorthms depends on both the accuracy and dversty among ndvdual learners n the ensemble, demonstrated by theoretcal [8, 9] and emprcal studes [30]. In general, the component learners n an ensemble are desgned to be accurate yet dverse. The emprcal results show that the performance of an ensemble algorthm s related wth the dversty among base classfers n the ensemble, and better performance may be acheved wth more dversty [3-36]. Many related research on analyss and applcatons of dversty have been conducted [37-40]. There are varous approaches to construct dverse base classfers. Dfferent learnng algorthms, dfferent representaton of patterns and parttonng the tranng set are some of these approaches. egatve Correlaton Learnng (CL) s a method based on dfferent learnng approach to make dversty between base neural classfers [30, 39, 4]. Ths method adds a penalty term to the error functon whch helps n makng the base classfers as dfferent from each other as possble whle encouragng the accuracy of base classfers. In ths paper, the effect of dversty made by CL on the combnaton methods of neural networks s nvestgated. Some fuson methods whch nclude DTs, SG and average are employed as combnng method. The expermental results show that dversfed base classfers by CL enhanced the performance of all mentoned combnng methods. II. Combnng Methods Learnng s a process, that dfferent procedures exceeds to dfferent performance also feature extracton methods nfluence on performance. There s no standard algorthm to do best performance wth a few numbers of data. Each of classfcaton method dependent on dfferent bases performs dfferent from the other. The combnaton of multple classfers was shown to be sutable way to mprove the performance of predcton n dffcult class determnaton problems [4]. It has become clear that for more datasets that are complcated, varous types of combnng rules can mprove the fnal classfcaton. Complexty n classfcaton can be represented from lmtaton n number of data, classes overlappng and credble nose n data. Results from [8] show that when classfers have small error rate and are ndependent n decson-makng, combnaton of classfers s useful. There are many ways to combne the results of a set of classfers, dependng on the type of the classfers' output [5]. Classfers are dfferent by one of these methods. In ths secton some famous fuson methods whch are used n ths paper, are descrbed. A. Decson Templates DT s a fuson approach taken from fuzzy templates [4]. The general schema of a DT ensemble s shown n Fgure. Assume that {D, D,,D L } s a set of L classfers that classfy samples set S = {( x, t ),( x, t ),...,( x, t )}, nto C classes. The outputs of these classfers for nput x can be arranged as a L C matrx called decson profle (DP). The structure of ths matrx s defned as follows: d, d, C DP( x) = () d L, dl, C

3 Improvng Combnaton Methods of eural Classfers Usng CL 3 Fgure. Decson Template schema. In ths matrx, the row shows the output of classfer D and column j shows support of classfers D,..., D L for class j, therefore entry d,j ndcates the support of classfer for class j. Decson template matrx of a class s the average of decson profles obtaned from the tranng samples belongng to that class. The (k, s)-th element of the DT matrx for class j s calculated by: Ind( x, ) ( ) q= q j dk, s xq dt j ( k, s)( x) =, () Ind( x, j) q= where x q s the q-th element of tranng dataset, Ind(x q,j) s an ndcator functon and ts value s f x q belongs to class j, and 0 otherwse. Durng the testng phase, DT scheme compares the DP of the nput sample wth all of the DTs and suggest a support for each class equal to the smlarty between ts DT and the DP. There are varous smlarty measures that can be appled to estmate the smlarty. The below functon s used as smlarty measure n ths paper. q the level-network. The tranng algorthm of ths modular ensemble can be summarzed as follows. ) From samples of dataset S, leave out one test sample, and tran each base classfers of the level-0 on the remanng - samples ) Produce a predcton output for the test sample. The output pattern y=[y, y,,y K ] wth the target t of the base classfers of the level-0, for the test sample, becomes a tranng sample for the classfer of level-. 3) Repeat the process of Step and, n a leave-one-out methodology. Ths process produces a tranng set D wth samples, whch s used to tran the classfer of level-. 4) To create fnal learner system, all of the base classfers of the level-0 are traned one more tme usng samples n S. L C H( DT, DP( x)) = DT ( k, s) d( k, s) ( x). (3) LC k= s= B. Stacked Generalzaton In stacked generalzaton method, the output patterns of some ensemble of traned base experts serve as an nput to a second-level expert. The general framework of ths method conssts of two levels (see Fgure ). The frst level, level-0, s formed by base classfers whch are traned usng the nput data and the target output data. The output of level-0 s then used as the nput data to level-. As s shown n Fgure, for tranng set S = {( x, t ),( x, t ),...,( x, t )}, a set of K 0 0 level-0 base neural networks from to s arranged as the frst layer, and s traned by S. Then, the outputs of frst layer networks are combned usng a level- network. In the other words, frst, the level-0 networks are traned usng the nput data and the target outputs. Then the outputs of the frst layer wth the correspondng target class are used to tran Fgure. Archtecture of Stacked Generalzaton. C. Averagng The Averagng method s a relatvely smple method of combnng models. In ths method, each classfer s traned and outputs for each class are combned by averagng. Once the classfers n the ensemble are traned they do not requre any further tranng. The Averagng method s used when each classfer produces a confdence estmate (e.g., a posteror). In ths case, the wnner class s the class wth the hghest average posteror probablty across the ensemble. Let

4 4 Iranzad, Masoudna, Cheraghch, owzar-daln and Ebrahmpour S = {( x, t),( x, t ),...,( x, t )} be the tranng set. At frst, the set of classfers are traned by ths set. After that, n the Averagng combnaton, class label of test data x s obtaned by µ x) = f ( y,, y ), (4) ( L E = = = = e ( n) ( y ( n) t ( n)) + λ. (6) where y the output of classfer for test data x, f s the average operaton, and µ (x) s the class label of pattern x. III. egatve Correlaton Learnng n combnng methods In ths secton, at frst, a short explanaton of the egatve Correlaton Learnng (CL) s gven. Later ensemble schema wth egatve Correlaton Learnng for desgnng neural network ensembles s presented. A. egatve Correlaton Learnng Most of the methods for desgnng neural network ensembles tran the ndvdual neural networks ndependent of each other or n sequental. One of dsadvantage of such methods s the loss of nteracton among ndvdual neural networks durng learnng process. It s desred to encourage dfferent ndvdual neural networks to learn dfferent parts of tranng data set so that the neural network ensemble can learn the whole parts of tranng data set better. CL ntroduced n Secton I s a method that supports ths am and trans the ensemble smultaneously and nteractvely. Ths method presents a correlaton penalty term whch has been added to the error functon of each ndvdual neural network. Durng the tranng process, each ndvdual neural network n the ensemble nteracts wth other neural networks through ther penalty terms n the error functons [30, 43]. Suppose that there s a tranng set denoted by S = {( x, t ),( x, t ),...,( x, t where x R n s the n -dmensonal pattern, t s a scalar and the target output and s the sze of the tranng set. The assumpton that the output t s a scalar has been made nearly to smplfy exposton of dea wthout loss of generalty. CL consders estmatng output by formng an ensemble, whose fnal output s a smple averagng on outputs of the set of ndvdual neural networks, L y( n) = y ( n), (5) L = where L s the number of ndvdual neural networks n the ensemble and y (n) s the output of network on the n -th tranng pattern, and y (n) s the output of ensemble on the same tranng pattern. The general Archtecture of CL s shown n Fgure 3. The error functon E for neural network n negatve correlaton learnng s defned by )}, Smlarly, t (n) and P (n) are the desred output and correlaton penalty functon for the n -th pattern respectvely. The penalty term P (n) s the error functon of each ndvdual neural network, and λ s a weghtng parameter on the penalty functon. The λ parameter controls a trade-off between objectve and penalty functons; when λ=0, the penalty functon s omtted and we have an ensemble that each neural network s traned ndependently of the other, usng backpropagaton algorthm. All networks can tran smultaneously and nteractvely on the same tranng dataset, and the penalty term has the followng form: P ( n ) = ( y ( n ) y ( n )) ( y k ( n ) y ( n)), (7) P ( n ) = ( y ( n ) y ( n)) Above equaton follows from the fact that L = k. ( y ( n) y( n)) = 0, (8) whch s based on the fact that the sum of devatons around a mean s equal to zero. Mnmzaton of (7) mples that each neural network has to mnmze the dfference between the target output and ts actual output, lke the penalty term. Mnmzaton of the penalty term n (7) results n maxmzaton of the dstance between ndvdual neural networks output and the average values. It s clear that from the second part of (7) snce there s a negatve sgn before the dstance term n ths equaton. Therefore the penalty term causes each neural network acts functonally dfferent, and can be expected to get useful dversty among the neural networks. Snce the CL ncorporates a dversty measurement drectly nto the error functon, therefore t s consdered as an explct ensemble method [43, 44]. Brown et al. [44] showed that CL can be vewed as a technque derved from ambguty decomposton. The ambguty decomposton, gven by Eq. 9, has been wdely recognzed as one of the most mportant theoretcal results obtaned for ensemble learnng. ( y t) = w ( y t) w ( y y). Ths equaton states that the Mean-Square-Error (MSE) of the ensemble s guaranteed to be less than or equal to the average MSE of the ensemble members. In ths equaton t s the target value of an arbtrary data pont and w =, 0 and y w s the convex combnaton of the L ensemble members as: (9) y = w y. (0)

5 Improvng Combnaton Methods of eural Classfers Usng CL 5 Fgure 3. Archtecture of egatve Correlaton Learnng. The ambguty decomposton n CL ensemble method provdes a smple expresson for the effect of error correlatons. The decomposton s composed of two terms. The frst term w ( y t), s the weghted average error of the ndvdual neural networks. The second term, w ( y y) referred to as the ambguty, measures the amount of varablty among the ensemble members and can be consdered to be a correlaton of the ndvdual neural networks. Utlzng the coeffcent λ allows us to vary the emphass on the correlaton components to yeld a near-optmum balance n the trade-off between these two terms; ths can be regarded as an accuracy-dversty trade-off [39, 44]. Hansen [8, 45] showed that there s a relatonshp between bas-varance-covarance decomposton and ambguty decomposton, n whch portons of the frst decomposton terms correspond to the portons of the ambguty decomposton terms. Therefore, any attempt to strke a balance between the two ambguty decomposton terms leads to three components of the other decomposton that can be balanced aganst each other, and the MSE tends to a near-mnmum condton. B. Usng CL to dverse base classfers of combnaton Combnng classfers s used as a common way for ncreasng the classfcaton performance. Ths technque s helpful, but ts effects can be ncreased by usng basc classfers whch have been dversfed by a dversty method. Ths proposed method makes classfers dverse by CL and then combne them by some combnng methods. It wll be shown that ths method s successful when Decson Template, Stacked Generalzaton and Averagng are used as combnng methods. IV. Expermental Results To evaluate the proposed method, we expermentally compare the Decson Template, Stacked Generalzaton and Averagng to CL verson of these ensembles. We test the combnaton rules usng real data sets. For ease of comparson, fve man benchmark classfcaton datasets are used. These datasets are Sat-mage, Vehcle, Pma, Breast and Sonar from the UCI Machne Learnng Repostory [46]. Informaton of these data sets s shown n Table. Also, a bref revew of each data set s gven n the followng. Dataset name umber of umber of attrbutes Tranng sze Test sze Classes Sat-mage Vehcle Pma Breast Sonar Table. Datasets Informaton. Sat-mage Ths data was generated from Land sat Mult-Spectral Scanner mage data. It conssts of 6435 pxels wth 36 attrbutes. The pxels are crsply classfed n 6 classes. The classes are: red sol (3.8 %), cotton crop (0.9 %), grey sol (.0 %), damp grey sol (9.73 %), sol wth vegetaton stubble (0.99 %), and very damp grey sol (3.43 %). we used only features # 7 to # 0, as recommended by the database desgners. Vehcle Ths data was generated from D mages of varous types of vehcles by applcaton of an ensemble of shape feature extractors to the D slhouettes of the objects. It conssts of 946 eghteen-dmensonal vectors as the extracted features for four types of vehcle: OPEL, SAAB, BUS and VA. The four classes of ths dataset almost have same sze. Pma Ths data was from atonal Insttute of Dabetes and Dgestve and Kdney Dseases. All consdered patents are females at least years old of Pma Indan hertage. The feature vector of each patent nformaton s an eght-dmensonal vector, ncludng both the nteger and real

6 6 Iranzad, Masoudna, Cheraghch, owzar-daln and Ebrahmpour valued numbers. The dataset conssts of 768 cases for two types: healthy people aganst who have dabetes. Breast Cancer Wsconsn Ths data was computed from a dgtzed D mage of a fne needle asprate of a breast mass. It descrbes characterstcs of the cell nucle present n the mage. The dataset conssts of 569 thrty-two dmensonal vectors as the real-valued features are computed for each cell nucleus. The two classes of ths dataset present two types of dagnoss: malgnant and bengn tumors. Sonar Ths dataset obtaned by bouncng sonar sgnals off cylnders made up from two dfferent materals, at varous angles and under varous condtons. It contans patterns obtaned by bouncng sonar sgnals off a metal cylnder and 97 patterns obtaned from rocks under smlar condtons. The transmtted sonar sgnal s a frequency-modulated chrp, rsng n frequency. The data set contans sgnals obtaned from a varety of dfferent aspect angles, spannng 90 degrees for the cylnder and 80 degrees for the rock. Each pattern s a set of 60 numbers n the range 0.0 to.0. Each number represents the energy wthn a partcular frequency band, ntegrated over a certan perod of tme. Therefore, ths dataset conssts of 08 sxty dmensonal real value vectors for two groups of bouncng sonar sgnals off metal and rocks. In all experments there are fve MLPs n the ensemble. The strength parameter of CL tested n the nterval of [0:0.:] and 0.8 has been chosen as the best one and 50 epochs has been done to tran the ensemble. The weak classfers have been chosen as the base classfers whch are obtaned by usng neural networks wth low complexty. The small sub-sample of each dataset s selected randomly as the tranng set. It s appled to show generalzaton ablty of proposed method. The small part of dataset (about 0%) s selected for tranng, but for some datasets ncludng: Vehcle and Pma, followng ths approach was not practcal because they have many classes that t s mpossble to tran well by that sze of tranng data and t have to be chosen more data as tranng set (around 0%). Also, snce Sonar dataset has small nstances, classfers cannot tran by 0 percentages of dataset and t has to be chosen more tranng data (around 0%). It s expected that ths proposed method would have thrvng results n the case of small sample sze problems, and ths s a way for solvng that problems whch are dffcult to classfy by smple neural networks lke MLP. The related tables of the results are llustrated below. The results of proposed method n Decson Template as a combnng method are presented n Table. Dataset Average percentages of 5 base DT CL-DT classfers Sat-mage (±3.9) Vehcle 6.07(±.9) Pma 63.4(±.34) Breast (±.) Sonar 44.70(±.46) Table. Results for combnng based on Decson Template. The results of proposed method n Stacked Generalzaton method are presented n Table 3. Dataset Average percentages of 5 base SG CL-SG classfers Sat-mage (±3.9) Vehcle 6.07(±.9) Pma 63.4(±.34) Breast (±.) Sonar 44.70(±.46) Table 3. Results of Stacked Generalzaton. The results of proposed method n Averagng combnng method are presented n Table 4. Dataset Average percentages of 5 base classfers Averagng CL- Averagng Sat-mage (±3.9) Vehcle 6.07(±.9) Pma 63.4(±.34) Breast (±.) Sonar 44.70(±.46) Table 4. Results of combnng based on averagng. It s obvous that CL mproved the performance of all nvestgated methods consderng to the results shown n the tables. The examples of the above tables show that the classfers on the ndependent data sets classfy a large fracton of the data correctly when the CL method uses for dversty. The results also show that the ndependent vews of the dfferent classfers can contrbute sgnfcantly to correct the output for some of the more dffcult data n CL verson of each combnng classfers. V. Concluson The approach ntroduced n ths paper s usng dverse neural network classfers for combnng,.e. the man goal of ths work was to nvestgate the relatve effect of CL over classfer outputs. The proposed method has two steps: at frst, the base classfers are dversfed by CL and then combnng methods are appled. Two non-tranable combnng methods ncludng: averagng and DTs and a tranable combner SG are nvestgated. Expermental results show that dverse base classfers mprove the performance of combnng n both tranable and non-tranable combnng methods. References [] L. Rokach, 00, Ensemble-based classfers, Artfcal Intellgence Revew, 33(), pp [] T. Wlk, and M. Woznak, 0, Soft computng methods appled to combnaton of one class classfers, eurocomputng, 75(), pp

7 Improvng Combnaton Methods of eural Classfers Usng CL 7 [3] S. Bhardwaj, S. Srvastava, J. Gupta, and A. Srvastava, 0, Chaotc tme seres predcton usng combnaton of hdden markov model and neural nets, Internatonal Journal of Computer Informaton Systems and Industral Management Applcatons, 4, pp [4] T. P. Tran, T. T. S. guyen, P. Tsa et al., 0, BSP: boosted subspace probablstc neural network for emal securty, Artfcal Intellgence Revew, 35(4), pp [5] S. Kotsants, 0, An ncremental ensemble of classfers, Artfcal Intellgence Revew, 36(4), pp [6] S. Kotsants, 0, Combnng baggng, boostng, rotaton forest and random subspace methods, Artfcal Intellgence Revew, 35(3), pp [7] L. Rokach, 00, Pattern Classfcaton Usng Ensemble Methods: World Scentfc Pub. Co. Inc., Hackensack, J. [8] Z. Yu-Quan, O. J-Shun, C. Geng, and Y. Ha-Png, 0, Dynamc weghtng ensemble classfers based on cross-valdaton, eural Computng & Applcaton, 0(3), pp [9] M. Tabassan, R. Ghader, and R. Ebrahmpour, 0, Combnaton of multple dverse classfers usng functons for handlng data wth mperfect labels, Expert Systems wth Applcatons, 39, pp [0] M. Tabassan, R. Ghader, and R. Ebrahmpour, 0, Kntted fabrc defect classfcaton for uncertan labels based on Dempster-Shafer theory of evdence, Expert Systems wth Applcatons, 38, pp [] R. Ebrahmpour, E. Kabr, and M. Yousef, 0, Improvng mxture of experts for vew-ndependent face recognton usng teacher-drected learnng, Machne Vson & Applcatons,, pp [] F. Behjat-Ardakan, F. Khademan, A. owzar-daln, and R. Ebrahmpour, 0, Low resoluton face recognton usng mxture of experts, World Academy of Scence, Engneerng and Technology, 80, pp [3] R. Ghazal,. M. aw, and M. Z. M. Salkon, 009, Forecastng the UK/EU and JP/UK tradng sgnals usng polynomal neural networks, Internatonal Journal of Computer Informaton Systems and Industral Management Applcatons,, pp [4] P. Klement and V. Snasel, 00, SOM neural network - a pece of ntellgence n dsaster management, Internatonal Journal of Computer Informaton Systems and Industral Management Applcatons,, pp [5] P. M. Carell, E. Olvera, C. Badue, and A. F. D. SouzaKlement, 009, Mult-Label text categorzaton usng a probablstc neural etwork, Internatonal Journal of Computer Informaton Systems and Industral Management Applcatons,, pp [6] M. Barakata, F. Druauxa, D. Lefebvrea, M. Khallb, O. Mustaphac, 0, Self adaptve growng neural network classfer for faults detecton and dagnoss, eurocomputng, 74(8), pp [7] R. Polkar, 007, Bootstrap-nspred technques n computatonal ntellgence, IEEE Sgnal Processng Magazne, 4(4), pp [8] K. Tumer and J. Ghosh, 996, Error correlaton and error reducton n ensemble classfers, Connecton Scence, 8(3), pp [9] R. Polkar, 006, Ensemble based systems n decson makng, IEEE Crcuts and Systems Magazn, 6(3), pp [0] R. A. Jacobs, 997, Bas/varance analyses of mxtures-of-experts archtectures, eural Computaton, 9(), pp [] L. I. Kuncheva, 004, Combnng Pattern Classfers, Methods and Algorthms: John Wley, Hoboken, J. [] L. I. Kuncheva, 00, Swtchng between selecton and fuson n combnng classfers: An experment, IEEE Transactons on Systems Man and Cybernetcs Part B-Cybernetcs, 3(), pp [3] R. Ebrahmpour, H. koo, S. Masoudna et al., 00, Mxture of MLP-experts for trend forecastng of tme seres: A case study of the Tehran stock exchange, Internatonal Journal of Forecastng, 7(3), pp [4] J. Kttler, 998, Combnng classfers: A theoretcal framework, Pattern Analyss and Applcatons, (), pp [5] L. I. Kuncheva, J. C. Bezdek, and R. P. W. Dun, 00, Decson templates for multple classfer fuson: an expermental comparson, Pattern Recognton, 34(), pp [6] D. H. Wolpert, 99, Stacked generalzaton, eural etworks, 5(), pp [7] L. I. Kuncheva, 00, Usng measures of smlarty and ncluson for multple classfer fuson by decson templates, Fuzzy Sets and Systems, (3), pp [8] L. K. Hansen and P. Salamon, 990, eural network ensembles, IEEE Transactons on Pattern Analyss and Machne Intellgence, (0), pp [9] A. Krogh and J. Vedelsby, 995, eural network ensembles, cross valdaton and actve learnng, In Proceedngs of Advances n eural Informaton Processng Systems 7, G. Teasuro, D. S. Touretzky, and T. K. Leen (eds.), MIT Press, Cambrdge, MA, pp [30] Y. Lu and X. Yao, 999, Ensemble learnng va negatve correlaton, eural etworks, (0), pp [3] P. Cunnngham and J. Carney, 000, Dversty versus qualty n classfcaton ensembles based on feature selecton, In Proceedngs of th European Conference on Machne Learnng, pp [3] L. I. Kuncheva and C. J. Whtaker, 003, Measures of dversty n classfer ensembles and ther relatonshp wth the ensemble accuracy, Machne Learnng, 5(), pp [33] T. Wndeatt, 006, Accuracy/dversty and ensemble MLP classfer desgn, IEEE Transactons on eural etworks, 7(5), pp [34] G. I. Webb and Z. Zheng, 004, Multstrategy ensemble learnng: Reducng error by combnng ensemble learnng technques, IEEE Transactons on Knowledge and Data Engneerng, 6(8), pp [35] Z. H. Zhou and Y. Yu, 005, Ensemblng local learners through multmodal perturbaton, IEEE Transactons

8 8 Iranzad, Masoudna, Cheraghch, owzar-daln and Ebrahmpour on Systems Man and Cybernetcs Part B-Cybernetcs, 35(4), pp [36] L. I. Kuncheva, 005, Usng dversty measures for generatng error-correctng output codes n classfer ensembles, Pattern Recognton Letters, 6(), pp [37] R. Kohav and D. H. Wolpert, 996, Bas plus varance decomposton for zero-one loss functons, In Proceedngs of the 3th Internatonal Conference on Machne Learnng, pp [38] D. Partrdge and W. Krzanowsk, 997, Software dversty: practcal statstcs for ts measurement and explotaton, Informaton and software technology, 39(0), pp [39] H. Chen and Xn Yao, 008, Regularzed negatve correlaton learnng for neural network ensembles, IEEE Transactons on eural etworks, 0(), pp [40] J. J. Rodrguez and L. I. Kuncheva, 006, Rotaton forest: A new classfer ensemble method, IEEE Transactons on Pattern Analyss and Machne Intellgence, 8(0), pp [4] Y. Lu and X. Yao, 999, Smultaneous tranng of negatvely correlated neural networks n an ensemble, IEEE Transactons on Systems Man and Cybernetcs Part B-Cybernetcs, 9(6), pp [4] K. Woods, W. P. Kegelmeyer, and K. Bowyer, 997, Combnaton of multple classfers usng local accuracy estmates, IEEE Transactons on Pattern Analyss and Machne Intellgence, 9(4), pp [43] M. F. Amn, M. M. Islam, and K. Murase, 009, Ensemble of sngle-layered complex-valued neural networks for classfcaton tasks, eurocomputng, 7(0), pp [44] G. Brown, and J. M. Wyatt, 003, egatve correlaton learnng and the ambguty famly of ensemble methods, Lecture ote n Computer Scence, 709, pp [45] J. V. Hansen, 999, Combnng predctors: comparson of fve meta machne learnng methods, Informaton Scences, 9(), pp [46] A. Asuncon and D. J. ewman, 00, UCI machne learnng repostory [ MLRepostory.html], Unversty of Calforna, School of Informaton and Computer Scence, Irvne, CA. Arash Iranzad receved the BS degree n computer scence from School of Mathematcs, Statstcs, and Computer Scence, Unversty of Tehran n 0. He s currently a MS student n Computer Scence Department, Brock Unversty, Ontaro, CA. Hs research nterests are neural networks, pattern recognton, multple classfer systems, and machne vson. Saeed Masoudna receved the MS degree n computer scence from School of Mathematcs, Statstcs, and Computer Scence, Unversty of Tehran n 0. Hs research nterests are neural networks, pattern recognton, machne learnng, computatonal neuro-scence and models of human vson. Fatemeh Cheraghch receved the MS degree n computer scence from School of Mathematcs, Statstcs, and Computer Scence, Unversty of Tehran n 00. Hs research nterests are neural networks, pattern recognton, multple classfer systems, quantum computng and analyss of algorthms. Author Bographes Abbas owzar-daln receved hs PhD degree from the School of Mathematcs, Statstcs, and Computer Scence n 003. He s an assocate professor of School of Mathematcs and Computer Scence, Unversty of Tehran, and s currently the drector of Graduate studes n ths school. Hs research nterests nclude neural networks, bonformatcs, combnatoral algorthm, parallel algorthms, DA computng, and computer networks. Reza Ebrahmpour receved the BS degree n electroncs engneerng from Mazandaran Unversty, Mazandaran and the MS degree n bomedcal engneerng from Tarbat Modarres Unversty, Tehran, Iran, n 999 and 00, respectvely. He receved hs PhD degree n July 007 from the School of Cogntve Scence, Insttute for Studes on Theoretcal Physcs and Mathematcs, where he worked on vew-ndependent face recognton wth Mxture of Experts. He s an assstant professor of Shahd Rajaee Unversty. Hs research nterests nclude human and machne vson, neural networks, and pattern recognton.