A new algorithm to build feed forward neural networks.

Transcription

1 A ew algorithm to build feed forward eural etworks. Amit Thombre Cetre of Excellece, Software Techologies ad Kowledge Maagemet, Tech Mahidra, Pue, Idia Abstract The paper presets a ew algorithm to build a feedforward eural etwork with a sigle hidde layer. The algorithm starts with 1 hidde uit ad the ew hidde uits are added to the etwork oly if they improve the classificatio accuracy of the etwork o the cross-validatio samples. The iitializatio of the weights ad bias is doe usig Nguye-Widrow method ad the etwork is traied without freezig the weights. The results show that the traiig accuracies obtaied usig this algorithm are better tha that obtaied usig N2C2S algorithm. The crossvalidatio accuracies ad the test predictio accuracies obtaied by usig both the algorithms are ot statistically sigificatly differet. Due to this ad also sice it is easy to uderstad ad implemet tha N2C2S algorithm, the proposed algorithm should be preferred tha the N2C2S algorithm. Alog with this 3 differet methods of obtaiig weights for eural etworks are also compared. The classificatio results obtaied usig this algorithm are compared to the predictio accuracies obtaied usig logistic regressio, C5.0 ad M5 classificatio techiques o 5 freely available data sets. The classificatio results show that NN is better tha logistic regressio over 2 data sets, equivalet i performace over 2 data sets ad has low performace tha logistic regressio i case of 1 data set. NN is better tha C5.0 over 1 data set ad equivalet i performace o the remaiig data sets. It is observed that M5 is a better classificatio techique tha other techiques over 1 dataset. M5 is better tha logistic regressio over 2 data sets ad equivalet i performace for the remaiig datasets. C5.0 is better tha logistic regressio over 1 data set ad equivalet i performace o the remaiig data sets. Keywords-classificatio, logistic regerssio, feedforward eural etwork, backpropagatio. I. INTRODUCTION Classificatio is a data miig (machie learig) techique which divides up data istaces such that each is assiged to oe of a umber of classes. The data istaces are assiged to precisely oe class ad ever to more tha oe class or ever to o class at all. Classificatio problems ca be foud i busiess, sciece, idustry, ad medicie. Some of the examples iclude bakruptcy predictio, customer chur predictio, credit scorig, medical diagosis (like predictig cacer), quality cotrol, hadwritte character recogitio, speech recogitio etc. Some of the widely used classificatio techiques are decisio trees, rule based classificatio, eural etworks (NN), Bayesia etworks, logistic regressio, k- earest eighbor classifier, support vector machies etc. I this study, the computatioally powerful techique based o Artificial Neural Net-works (ANN) is built usig a ew algorithm ad the the classificatio results are compared with those from logistic regressio, the model tree techique based o M5 ad the classical decisio tree based techique based o C5.0. NN have gaied popularity over a few years ad are beig successfully applied across wide problem domais such as fiace, medicie, egieerig, geology etc. NN ca adapt to the data without ay explicit specificatio of fuctioal or distributioal form for the uderlyig model [1]. NN ca approximate ay fuctio thus makig them flexible i modelig real world complex relatioships. Neural etworks have successfully bee applied for classificatio problems i bakruptcy predictio [2], [3], medical diagosis [4], [5], hadwritig recogitio [6] ad speech recogitio [7]. Artificial eural etworks are computig systems made up of large umber of simple, highly itercoected processig elemets called odes or artificial euros. The focus of this work is o the feedforward multilayer etworks or multilayer perceptros (MLPs) with 1 iput layer, 1 output layer ad 1 hidde layer as oe hidde layer is sufficiet to map a arbitrary fuctio to ay degree of accuracy. The umber of euros i the hidde layer eeds to be fixed to arrive at the correct architecture of the etwork. There are may algorithms which costruct the etworks with sigle hidde layer. I the Dyamic ode creatio (DNC) algorithm [8] the odes are added to the hidde layer oe at a time till a desired accuracy is obtaied. I Feed-forward Neural etwork Creatio Algorithm (FNNCA) [9] ad Costructive Algorithm for Real- Value Examples (CARVE)[10] the hidde uits are added to the hidde layer oe at a time util a etwork that completely recogizes all its iput patters is costructed. Usig these algorithms ca lead to overfittig of the traiig data ad do ot geeralize well with ukow data. I Neural etwork Costructio with Cross-Validatio Samples algorithm (N2C2S) [11] the odes are added to the hidde layer oly if they improve the accuracy of the etwork o the traiig ad the cross-validatio data. This algorithm uses freezig of weights which ca lead to icrease i umber of hidde uits [12]. I the proposed algorithm, the weights are ot froze ad the odes are added to the hidde layer oly if they improve the accuracy of the etwork o the cross-validatio data oly.

2 The cross-validatio data accuracy is selected because it is true measure of the performace of the model. Logistic regressio (sometimes called the logistic model or logit model) is used for predictio of the probability of occurrece of a evet by fittig data to a logit fuctio. The logistic fuctio which is as give below takes z as the iput ad f (z) is the output which is always betwee 0 ad 1. f (z)= e z /e z +1 =1/1+e z (1) The variable z comprises of other idepedet variables ad is give as z=β 0 +β 1 x 1 +β 2 x 2 +β 3 x 3 +β 4 x 4 +β 5 x β k x k... (2) where β0 is the itercept ad β1, β2, β3 are the regressio coefficiets of idepedet variables x1, x2, x3 respectively. Each of the regressio coefficiets describes the size of the cotributig idepedet variable. Logistic regressio is widely used i medical field [1], [13]. C5.0 is a algorithm to build decisio trees usig the cocept of iformatio etropy. The decisio tree is used as a predictive model which maps observatios about a item to coclusios about the item s target value. I these tree structures the leaves represet the class labels ad braches represet the rules of features that lead to those class labels. M5 builds tree based models i which the trees costructed have multivariate liear models. Thus these trees are aalogous to piecewise liear fuctios. They are applied to classificatio problems by applyig a stadard method of trasformig a classificatio problem ito a problem of fuctio approximatio [14]. M5 is based o M5 developed by Quila [15] but icludes techiques to hadle eumerated data ad missig values effectively [16]. The orgaizatio of the paper is as follows. After the itroductio i sectio I, sectio II presets the algorithm used for buildig the NN. Sectio III describes the experimetal setup. Sectio IV presets the results from the experimets ad sectio V gives the discussio ad coclusio from the work carried out. The paper cocludes with future work i Sectio VI. II. THE PROPOSED ALGORITHM The steps of the algorithm used to build the eural etwork are as follows:- 1. Let N 1 deote the etwork with I iput uits ad O output uits ad H hidde uits. 2. The iitializatio of the weights ad bias is doe usig Nguye-Widrow method [17]. This algorithm chooses values i order to distribute the active regio of each euro i the layer approximately evely across the layer s iput space. The weight values cotai a degree of radomess, so they are ot the same each time this fuctio is called. Also the wastage of euros by this method is less as compared to radom iitializatio [18]. 3. The accuracy of this etwork o the traiig ad validatio data set is A T1 ad A V1. 4. The umber of hidde uits is the icreased by 1 uit ad the weights are iitialized with the Nguye-Widrow method. I case of N2C2S the weights of the first H hidde uits of N 2 is obtaied from the optimal weights of N 1 ad the remaiig coectio weights are set radomly. Thus i the proposed algorithm the weights are ot froze ad the whole etwork is retraied as freezig requires large umber of hidde uits to achieve the same performace as that obtaied without freezig [12]. 5. Let the accuracy of this etwork N 2 be A T2 ad A V2 o the traiig ad the validatio set respectively. If A V2 > A V1 the N 2 etwork is better tha N 1 else the etwork is ru with icreasig the hidde uits by 1. Thus the steps 4 ad 5 are repeated till we get the etwork with highest accuracy over validatio data set. III. 1) Neural Networks EXPERIMENTAL SETUP a. The eural etwork was costructed usig Matlab. b. As metioed above i sectio I the etwork cosists of iput layer, output layer ad hidde layer. The umber of euros i the iput layer correspods to the umber of idepedet variables. The umber of euros i the hidde layer is obtaied by usig the algorithm metioed i sectio II. Ad the umber of euros i the output layer is correspodig to the category of classes i the depedet data mius oe for programmig purpose. c. The back propagatio traiig fuctio is scaled cojugate gradiet algorithm. d. Each ru of the etwork was carried out for 200 epochs with the goal for error set as 0. e. The Nguye-Widrow algorithm chooses the weight values i order to distribute the active regio of each euro i the layer approximately evely across the layer s iput space. These values cotai a degree of radomess, so they are ot the same each time this fuctio is called. That s why 30 rus were carried out for each fold of cross-validatio ad for each cofiguratio of the etwork to cosider as may as radom weight values ad the average of 30 rus is take. The 10 fold cross-validatio was ru 10 times ad the the predictio accuracy was averaged over these 10 rus. 2) Logistic Regressio The programmig for logistic regressio is doe usig R [19], a free statistical software. 10 times, 10 fold cross-validatio was used to carry out the rus. 3) C5.0 ad M5 The results usig these algorithms are obtaied from [14]. The experimets usig these algorithms were carried out usig 10 rus of te fold cross-validatio.

3 4) Data The Chapma data set was obtaied from [20] ad the rest of the data sets were obtaied from the UCI Machie Learig Laboratory [21]. The data sets were chose i such a way that they are publicly available ad the results of classificatio usig eural ets, C5.0 ad M5, o these data sets (except oe data set) is already available for compariso purpose. Also the data sets are small i size ad cotaied oly cotiuous data ad biary data. The missig cotiuous attribute value was replaced by the average of the o-missig values. The data was ormalized before carryig out the experimets o them. TABLE I. DETAILS OF THE DATA SETS Data sets Size Missig Attributes values Cotiuous Biary Nomial Glass(G2) Chapma Ioospher e Votig ) Experimets All the data sets used were split up ito 3 sets, viz traiig, validatio ad test data sets. Let them be referred as T R, V ad T E data sets respectively. Iitially the eural etwork is built usig the algorithm metioed i Sectio II. The umber of euros i the hidde layer varied from 1 to 20 maximum. These steps are commo to all the experimets metioed below. 1. Experimet A a. The weights i the commo step metioed above were obtaied correspodig to lowest geeralizatio error over validatio data set. b. The the etwork with weights fixed as obtaied from step a was ru o the traiig set comprisig of the traiig set T R ad validatio set V ad its accuracy was tested o the testig data set T E. This predictio accuracy is reported i the results. 2. Experimet B I this experimet, the steps a ad b remai the same as experimet A except that the criteria for obtaiig weights i step a is correspodig to the highest predictio accuracy over the validatio data set. 3. Experimet C I this experimet there was o exteral criteria laid dow as doe i experimets A ad B for gettig the weights. Thus the weights were ot fixed. The weights which were obtaied by each ru were used as it is for gettig the predictio accuracy over the traiig set. 1 * idicates that the data is ot available. IV. EXPERIMENTAL RESULTS The predictio accuracies obtaied by 2 differet methods were compared by usig the Welch's t test [22]. Welch s test is a adaptatio of Studet s t-test with the 2 samples havig uequal variaces. The ull hypothesis that the two meas are equal was rejected at the sigificace value of Followig are the results from various experimets tried out:- 1. The umber of hidde uits ad the accuracy rates of the eural etworks costructed by N2C2S ad the proposed algorithm are give i the tables II ad III below. Usig N2C2S Algorithm Data set Hidde uits Traiig Accuracy Crossvalidatio Accuracy Glass Chapma * 1 * * Ioosphe re Votig TABLE II. TABLE III. RESULTS USING N2C2S ALGORITHM Usig Metioed Algorithm Data set Hidde uits Traiig Accuracy Cross-validatio Accuracy Glass Chapma Ioosphe re Votig RESULTS USING MENTIONED ALGORITHM 2. The predictio accuracy over test data set obtaied usig experimets A, B ad C is give i the table below. This predictio accuracy is also compared with that obtaied from logistic regressio. TABLE IV PREDICTION ACCURACY BY NEURAL NETWORKS USING EXPERIMENTS A, B AND C Data Logistic Exp A Exp B Exp C sets Regressio Glass Chapma Ioosph ere Votig idicates that the predictio accuracy is less compared with other methods 3 idicates that the predictio accuracy is high compared with other methods

4 3. From table IV, the summary of results showig the compariso of eural etworks with logistic regressio is give i the table below. The wis ad losses are decided as per the Welch s test described at the start of this sectio. TABLE V. SUMMARY OF RESULTS NN versus Wi Ties Losses Logistic Regressio 4. The test predictio accuracies usig the N2C2S algorithm ad the proposed algorithm are give i the table below. TABLE VI. Data PREDICTION ACCURACIES USING N2C2S AND PROPOSED ALGORITHM Usig Exp A Exp B Exp C sets N2C2S Glass Chapma * Ioosphere Votig Referece [15] gives the predictio accuracies usig C5.0 ad M5 o the same data sets used i this study. The predictio accuracies of logistic regressio ad NN are compared with that obtaied by usig C5.0 ad M5 algorithms i the table below. The results of C5.0 ad M5 are the averages ad stadard deviatios from 10 rus of te fold cross-validatio experimets. The predictio accuracies usig NN from experimet A are used here. TABLE VII. PREDICTION ACCURACIES USING NN, LOGISTIC REGRESSION, C5.0 AND M5 Data sets Predictio Accuracy usig Neural Logistic C5.0 M5 Networks Regressio Glass Chapma * * Ioosphere Votig V. DISCUSSION AND CONCLUSIONS 1. Table VI shows that the predictio accuracies usig N2C2S is same as that obtaied usig the proposed algorithm. The proposed method of buildig the etwork used 30 iteratios for each ru, thus takig more combiatios of weights usig Nguye-Widrow iitializatio method ito cosideratio to arrive at the results as compared to ruig the N2C2S algorithm [11] which uses radom weight iitializatio oly oce. Also the results usig N2C2S algorithm are based o a smaller traiig set tha the oe used for the experimets carried out. Thus the results usig N2C2S algorithm are likely to chage if more iteratios ad a larger traiig set is cosidered. 2. The results i table II ad III show that the hidde uits obtaied usig proposed algorithm was less tha that obtaied usig N2C2S over 2 data sets ad greater i case of 2 other data sets. Thus it caot be cocluded if the ofreezig of weights has ay advatage over freezig of weights o the umber of hidde uits ad some more experimets eed to be performed over differet data sets to arrive at some coclusio. 3. The predictio accuracy over traiig data sets usig proposed algorithm was statistically better tha that obtaied usig N2C2S algorithm over 3 data sets. From tables II, III ad VI, it is observed that the cross-validatio accuracies ad test accuracies obtaied usig both the algorithms are ot sigificatly differet. Thus the proposed algorithm should be preferred as the chages are easy to uderstad ad implemet as compared to the N2C2S algorithm. 4. From table IV it is observed there is o sigle experimet which has performace better tha other 2 experimets. 5. From tables V it is observed that NN gives better performace tha logistic regressio o 2 data sets; gives the same results o 2 data sets ad lower tha that of logistic regressio o 1 data set. But it is observed that NN cosumes more executio time tha logistic regressio. Also logistic regressio techique is a white-box techique which allows the iterpretatio of the model parameters whereas NN is a black box techique which does ot allow the iterpretatio of the model. 6. NN is better tha C5.0 over 1 data set ad equivalet i performace o the remaiig data sets. 7. Table VII shows that M5 is better tha NN, logistic regressio ad C5.0 over 1 dataset. M5 is better tha logistic regressio over 2 data sets ad equivalet i performace for the remaiig datasets. M5 ad NN do ot have sigificatly differet accuracies except for 1 data set as metioed at the startig of this poit. M5 ad C5.0 do ot have sigificatly differet accuracies o the data sets except for 1as metioed at the startig of this poit. 8. From table VII, C5.0 is better tha logistic regressio over 1 data set ad equivalet i performace o the remaiig data sets. VI. FUTURE WORK From tables II, III ad VI, the proposed algorithm for buildig the eural etworks has give cross-validatio predictio accuracies ad test accuracies which are ot sigificatly differet with that obtaied by usig the N2C2S algorithm. The further work will be to improve this algorithm so that the

5 cross-validatio accuracies are better tha that obtaied usig N2C2S algorithm ad the predictio accuracies over test data are better tha that obtaied by usig N2C2S algorithm ad logistic regressio. Also some more experimets eed to be performed usig N2C2S algorithm ad the chages suggested i poit 1 of sectio V. ACKNOWLEDGEMENT I am grateful to Tech Mahidra Cetre of Excellece group for givig me the support to write this paper. REFERENCES [1] Guoqiag Peter Zhag, Neural Networks for classificatio: A Survey. IEEE Trasactios o Systems, Ma, ad Cyberetics- Part C: Applicatios ad reviews, vol. 30, No. 4, pp (2000) [2] Amir F. Atiya, Bakruptcy Predictio for Credit Risk Usig Neural Networks: A Survey ad New Results. IEEE Trasactios o Neural Networks, vol. 12, No. 4, pp (2000) [3] R. C. Lacher, P. K. Coats, S. C. Sharma, ad L. F. Fat, A eural etwork for classifyig the fiacial health of a firm. Eur. J. Oper. Res., vol. 85, pp (1995). [4] W. G. Baxt, Use of a artificial eural etwork for data aalysis i cliical decisio-makig: The diagosis of acute coroary occlusio. Neural Computig., vol. 2, pp (1990) [5] M. A. Mazurowski, P. A. Habas, J. M. Zurada, J.Y. Lo, J.A. Baker ad G. D. Tourassi, Traiig eural etwork classifiers for medical decisio makig: The effects of imbalaced datasets o classificatio performace. Neural Networks. vol. 21, pp (2007) [6] I. Guyo, Applicatios of eural etworks to character recogitio, Iteratioal Joural of Patter Recogitio ad Artificial. Itelligece. vol. 5, pp (1991) [7] H. Bourlard ad N. Morga, Cotiuous speech recogitio by coectioist statistical methods. IEEE Trasactios o Neural Networks. vol. 4, pp (1993) [8] Ash, T, Dyamic ode creatio i backpropagatio etworks. Coectio Sciece. 1(4), (2002) [9] Rudy Setioo, Feedforward Neural Network Costructio Usig Cross Validatio. Neural Computatio. 13:12, (2001) [10] Youg, S. ad Dows, T, CARVE -a costructive algorithm for realvalued examples. IEEE Trasactio o Neural Networks 9(6), (1998) [11] Setioo R.: A Neural Network Costructio Algorithm which Maximizes the Likelihood Fuctio. Coectio Sciece. vol 7, Number 2, pp (1996) [12] Ti-Yau Kwok ad Dit-Ya Yeug, Experimetal aalysis of iput weight freezig i costructig eural etworks, IEEE Iteratioal Coferece o Neural Networks. vol 1, pp (1993) [13] J.G. Liao ad Chi, Logistic regressio for disease classificatio usig microarray data: model selectio i a large p ad small case. Bioiformatics, vol 23, pp (2007) [14] Frak, E., Wag, Y, Iglis, S., Holmes, G. ad Witte, I.H, Usig Model trees for classificatio. Machie Learig 32(1), pp (1997) [15] Quila, J.R. Learig with cotiuous classes, Proceedigs Australia Joit Coferece o Artificial Itelligece (pp ). World Scietific, Sigapore(1992). [16] Wag, Y. & Witte, I.H. (1997). Iductio of model trees for predictig cotiuous classes, Proceedigs of the poster papers of the Europea Coferece o Machie Learig, Uiversity of Ecoomics, Faculty of Iformatics ad Statistics, Prague. [17] Derrick Nguye ad Berard Widrow, Improvig the learig speed of 2-layer eural etworks by choosig iitial values of the adaptive weights. Proceedigs of the Iteratioal Joit Coferece o Neural Networks, 3: (1990) [18] Howard Demuth, Mark Beale ad Marti T. Haga, Neural Network Toolbox 7, User s Guide, The MathWorks, Ic., Natick, MA, Revised for Versio 7.0 (Release 2010b), September [19] [20] [21] UCI Machie learig repository data sets, [22] Welch, B. L, The geeralizatio of "Studet's" problem whe several differet populatio variaces are ivolved. Biometrika 34 (1 2): (1947)