Pattern Classification of Back-Propagation Algorithm Using Exclusive Connecting Network

Transcription

1 World Acade of Scence, Engneerng and Technolog 36 7 Pattern Classfcaton of Bac-Propagaton Algorth Usng Eclusve Connectng Networ Insung Jung, and G-Na Wang Abstract The obectve of ths paper s to a desgn of pattern classfcaton odel based on the bac-propagaton (BP algorth for decson support sste. Standard BP odel has done full connecton of each node n the s fro nput to output s. Therefore, t taes a lot of coputng te and teraton coputng for good perforance and less accepted error rate when we are dong soe pattern generaton or tranng the networ. owever, ths odel s usng eclusve connecton n between hdden nodes and output nodes. The advantage of ths odel s less nuber of teraton and better perforance copare wth standard bac-propagaton odel. We sulated soe cases of classfcaton data and dfferent settng of networ factors (e.g. hdden nuber and nodes, nuber of classfcaton and teraton. Durng our sulaton, we found that ost of sulatons cases were satsfed b BP based usng eclusve connecton networ odel copared to standard BP. We epect that ths algorth can be avalable to dentfcaton of user face, analss of data, appng data n between envronent data and nforaton. Kewords Neural networ, Bac-propagaton, classfcaton. I. INTRODUCTION ANY people and ndustres are nterested n the decson M support sste, and predcton sstes for the better choce and reducton of rs based on ntellgence ethod. Especall, artfcal neural networ based decson ang and predcton sstes. These ethods are seeed to be successful to solve dffcult and dverse probles b supervsed tranng ethods such as bac-propagaton algorth. Ths algorth s the ost popular neural networ archtecture for supervsed learnng, because t s based on the weght error correcton rules. Although bac-propagaton algorth could correct weghts, t stll got error and taes uch of pattern generaton coputng te. The odel structure of BP (bac-propagaton classfcaton algorth use full connecton each s and nodes fro nput to output. Consequentl t needs uch of calculaton. owever, we are not stll satsfed wth standard neural networ or bac-propagaton odel based decson support sste because we want to get better qualt of decson Insung Jung, s wth Departent of Industral Engneerng Aou Unverst , Korea (e-al: nsung998@gal.co. G-Na Wang, s wth Departent of Industral Engneerng Aou Unverst , Korea (e-al: gnwang@aou.ac.r. perforance and less coputng teraton when we want to develop n the specfc doan area. In ths paper, we proposed dvdend eclusve connecton n between hdden nodes and output nodes. Ths ethod s alost sae as BP odel. owever, between last hdden nodes and output later nodes weghts are not full connected. Before output node, t has got eclusve nodes for appng hdden nodes. It could be preventon of gven wrong nforaton. We thn soetes full connecton of weght could be gven no effect or wrong calculaton value and nforaton. The advantage of ths odel s less nuber of teraton and better perforance copare wth standard bac-propagaton odel. To evaluate of ths algorth, we sulated soe cases of classfcaton data and dfferent settng of networ factors (e.g. hdden nuber and nodes, nuber of classfcaton and teraton. We found that proposed dvdend eclusve connecton based BP odel s farl better perforance than standard BP odel. Organzaton of paper s as follows. Secton s a revew of the related wor; Secton 3 descrbes ethodolog; Secton 4 s sulaton results and fnall t suarzed the concluson. II. RELATED WORK In 958, Fran Rosenblatt ntroduced a tranng algorth that provded the frst procedure for tranng a sple artfcal neural networ (ANN: a perceptron [. The perceptron s the splest for of an ANN. It conssts of a sngle neuron wth adustable snaptc weghts and a hard lter. The perceptron learnng rule was frst proposed b Rosenblatt n 96 [. It s base odel of the perceptron tranng algorth for classfcaton tass. Mns and Papert (969 showed that a two- feed-forward networ can overcoe an restrctons of sngle ANN [3. But t dd not present a soluton to the proble of how to adust the weghts fro nput to hdden unts. An answer to ths queston was presented b Ruelhart, nton, and Wllas n 986. Slar solutons appear to have been publshed earler (Werbos, 974; Parer, 985; Cun, 985 [4-[7. The central dea behnd ths soluton s that the errors for the unts of the hdden are deterned b bac-propagatng the errors of the unts of the output. For ths reason the ethod s often called the bac-propagaton learnng rule. Bac-propagaton can also be consdered a generalzaton of the delta rule for nonlnear actvaton functons and ult networs. 89

2 World Acade of Scence, Engneerng and Technolog 36 7 owever, when we use data pattern generaton, t needs uch of teraton nuber and coputng te. Therefore, we proposed the Pattern classfcaton of bac-propagaton algorth usng eclusve connectng networ. III. METODOLOGY A. Bacground Method Neural Mult perceptron l & BP (Bac-propagaton odel Standard ult perceptron (MLP archtecture conssts ore than s; A MLP can have an nuber of s, unts per, networ nputs, and networ outputs such as fg odels. Ths networ has 3 Laers; frst s called nput and last s called output ; n between frst and last s whch are called hdden s. Fnall, ths networ has three networ nputs, one networ output and hdden networ. OUTPUT SECOND IDDEN FIRST IDDEN n OUTPUT INPUT Fg. Standard Mult perceptron archtecture n Input w Input sgnals dden Error sgnals w Fg. Bac-propagaton neural networ l Output owever, ths research s copared wth Bac-propagaton (BP odel. Ths odel s the ost popular n the supervsed learnng archtecture because of the weght error correct rules. It s consdered a generalzaton of the delta rule for nonlnear actvaton functons and ult networs. In a bac-propagaton neural networ, the learnng algorth has two phases. Frst, a tranng nput pattern s presented to the networ nput. The networ propagates the nput pattern fro to untl the output pattern s generated b the output. If ths pattern s dfferent fro the desred output, an error s calculated and then propagated bacward through the networ fro the output to the nput. The l weghts are odfed as the error s propagated. Accordng to the Rchard P. Lppann [8, he represents step of the bac-propagaton tranng algorth and eplanaton. The bac-propagaton tranng algorth s an teratve gradent desgned to nze the ean square error between the actual output of ult- feed forward perceptron and the desred output. It requres contnuous dfferentable non-lneart. The followng assues a sgod logstc nonlneart. Step: Intalze weghts and offsets Set all weghts and node offsets to sall rando values. Step: Present nput and desred outputs Present a contnuous valued nput vector X, X..X N- and specf the desred output d,d,.d M-. If the net s used as a classfer the all desred outputs are tpcall set to zero ecept for that correspondng to the class the nput s fro. That desred output s. The nput could be new on each tral or saples fro a tranng set could be presented cclcall untl stablze. Step 3: Calculate Actual Output Use the sgod non lneart fro above and forulas as n fg 3 to calculate output,.m-. Step 4: Adapt weghts Use a recursve algorth startng at the output nodes and worng bac to the frst hdden. Adust weghts b w ( t + = w ( t + nδ (3 In ths equaton w (t s the weght fro hdden node or fro an nput to node at te t, w, s ether the output of node or s an nput, η s a gan ter, and δ, s an error ter for node, f node s an output node, then δ = ( d (4 ( where d s the desred output of node and s the actual output. If node s an nternal hdden node, then δ = ( δ w (5 where s over all nodes n the s above node. Internal node thresholds are adapted n a slar anner b assung the are connecton weghts on lns fro aular constant-valued nputs. Convergence s soetes faster f a oentu ter s added and weght change are soothed b w( t + = w( t + nδ + α( w( t w( t,where< α <. (6 Step 5: Repeat b gong to step B. Dvdend Eclusve Connecton based Bac-Propagaton In ths paper, we proposed dvdend eclusve connecton n between hdden nodes and output nodes. Ths ethod s alost sae as BP odel. owever, between last hdden nodes and output later nodes weghts are not full connected. 9

3 World Acade of Scence, Engneerng and Technolog 36 7 Before output node, t has got eclusve nodes for appng hdden nodes. It could be preventon of gven wrong nforaton. We thn soetes full connecton of weght could be gven no effect or wrong calculaton value and nforaton. Input dden s Dvdend eclusve connecton Fg. 3 Dvdend eclusve connecton based BP Output Step. Modelng of dvdend eclusve connecton based BP To Select each nodes fro nput to output (nput nodes, frst hdden nodes, second hdden nodes and output nodes. It has to be sae second hdden nodes nuber wth output nodes. Frst nodes of hdden connected wth second hdden nodes n the couple wa (Dvdend eclusve connecton; t has to be ore than double nodes than dvded eclusve nodes. N : the nuber of nput node. M : the nuber of output node. Z : the nuber of frst hdden node. ( I, w s the weght between nput th node and frst hdden th node where =,,, N and =,,, Z. Frst hdden nodes ust be grouped nto the nuber of dvdend eclusve nodes, M. (, w s the weght between frst hdden, th node and dvdend eclusve th node where =,,, M. s the hdden th node n group. (, O w s the weght between dvdend eclusve th node and output th node o where =,,, M. Step: Intalze weghts and offsets Set all weghts and node offsets to sall rando values. Step3: Present nput and desred outputs Present a contnuous valued nput vector,,, N and specf the desred output d, d,, d. If the net s used as a classfer the M all desred outputs are tpcall set to zero ecept for that correspondng to the class the nput s fro. That desred output s. The nput could be new on each tral or saples fro a tranng set could be presented cclcall untl stablze. Step 4: Proceedng Forward. Calculate hdden vector Y (,,, Z wth nput vector X (,,, N and weght vector ( I, W ( I, Y = F ( XW and dvdend eclusve node = f ( M = w (,, where f ( s the sgod functon. Step 5: Calculate Actual Output Output th node, O = f ( M = ( w (, O Step 6: Adapt weghts Use a recursve algorth startng at the output nodes and worng bac to the frst hdden. Adust weghts b w ( t + = w ( t + nδ (3 In ths equaton w (t s the weght fro hdden node or fro an nput to node at te t, w, s ether the output of node or s an nput, η s a gan ter, and δ, s an error ter for node, f node s an output node, then δ = ( d (4 ( where d s the desred output of node and s the actual output. If node s an nternal hdden node, then δ = ( δ w (5 where s over all nodes n the s above node. Internal node thresholds are adapted n a slar anner b assung the are connecton weghts on lns fro aular constant-valued nputs. Convergence s soetes faster f a oentu ter s added and weght change are soothed b w( t + = w( t + nδ + α( w( t w( t,where< α <. (6 Step 7: Repeat b gong to step 3 The advantage of ths odel s less nuber of teraton and better perforance copare wth standard bac-propagaton odel. To evaluate ths algorth, we sulated soe cases of classfcaton data and dfferent settng of networ factors (e.g. hdden nuber and nodes, nuber of classfcaton and teraton. We found that proposed dvdend eclusve connecton based BP odel has farl better perforance than standard BP odel. 9

4 World Acade of Scence, Engneerng and Technolog 36 7 IV. SIMULATION RESULT In ths sulaton, wth rando generaton data between and usng neural networ algorths, standard BP and dvdend eclusve connecton are based on BP odel. The evolutonal condton was eactl sae at each odels. owever, the coparng of each odels s a lttle bt dffcult stuaton because standard BP needs uch of teraton nuber. Tables I and II present the perforance of copared odels. Frst test of our evaluaton case was sall sze of output nodes tpe classfcaton. In ths te both odels have ver successful results. The perforance values are ore than 95% (standard odel: 97.8%, proposed odel: 98.4%. Our proposed odel was a lttle bt better than standard odel. Other sulaton case s 6 output nodes classfcaton. Soe case of sulaton b standard odel of BP was not satsfed because of beng lac of nodes and teraton nuber. But when we setup enough teraton nuber, t s also qute reasonable. On the other hand, our odel of dvdend eclusve connecton based BP odel was proved to be better perforance than standard BP results. Nuber of cluster TABLE I BACK-PROPAGATION MODEL PERFORMANCE Data dden Iteraton Perforance nuber / node nuber 4 5 (8, % (489/5 6 (8, % (379 / 5 6 (8, 4 8% (4/5 6 5 (6, % (49/5 6 (6, 6 9.4% (45/5 6 (3, 6 9.4% (457/5 6 (56, % (474/5 6 (56, % (48/5 TABLE II DIVIDEND EXCLUSIVE CONNECTION BASED BP PERFORMANCE Nuber of Data dden cluster nuber / node nuber Iteraton Perforance 4 5 (8, % (49/5 6 5 (3, 6 5 9%(455/5 6 5 (3, 6 93.% (466/5 6 5 (56, 6 3 9% (455/5 6 5 (56, % (47/5 6 5 (56, % (476/5 6 5 (56, % (483/5 Fg. 5 Dvdend eclusve connecton based BP odel (4output, 5 teraton Fg. 6 BP odel result (6output, teraton Fg. 7 Dvdend eclusve connecton based BP odel (6output, 5 teraton Fg. 8 Dvdend eclusve connecton based BP odel (6output, teraton Fg. 4 BP odel result (4output, 5 teraton Fg. 9 BP odel result (6output, 5 teraton 9

5 World Acade of Scence, Engneerng and Technolog 36 7 Fg. Dvdend eclusve connecton based BP odel (6output, 5 teraton [4 D. E. Ruelhart, nton, G. E., & Wllas, R. J., " Learnng representatons b bacpropagatng errors," Nature, vol. 33, pp , 986. [5 P. J. Werbos, "Beond Regresson: New Tools for Predcton and Analss n the Behavoral Scences," Unpublshed doctoral dssertaton arvard Unverst., 976. [6 D. B. Parer, "Learnng-Logc (Tech. Rep. Nos. TR{47." 985. [7 Y. L. Cun, "Une procedure dapprentssage pour reseau a seul assetrque," Proceedngs of Cogntva, vol. 85, pp [8 Rchard P. Lppann,: An ntroducton to coputng wth neural networ, IEEE ASSP agazne, 987, pp. 4-. V. CONCLUSION The obectve of ths paper s to desgn of pattern classfcaton odel based on the bac-propagaton (BP algorth for decson support sste. Standard BP odel has done full connecton of each node n the s fro nput to output s. Therefore, t taes a lot of coputng te and teraton coputng for good perforance and less accepted error rate when we are dong soe pattern generaton or tranng the networ. owever, our proposed odel s usng eclusve connecton n between hdden nodes and output nodes. The advantage of ths odel s less nuber of teraton and better perforance copare wth standard bac-propagaton odel. To evaluate of ths algorth, we sulated soe cases of classfcaton data and dfferent settng of networ factors (e.g. hdden nuber and nodes, nuber of classfcaton and teraton wth sae condtons. Sall sze of output nodes tpe classfcaton perforance results are ore than 95% (standard odel: 97.8%, proposed odel: 98.4%. owever, standard odel s a lttle bt hard to use coplcated case because of beng lac of nodes and teraton nuber. On the other hand, our odel of dvdend eclusve connecton based BP odel was proved to be better perforance (96% than standard BP results. We found that proposed dvdend eclusve connecton based BP odel s farl better perforance than standard BP odel. The proposed odel could be useful for dentfcaton of user face, analss of data, appng data n between envronent data and nforaton. The ltaton of ths research s not use real feld data. ACKNOWLEDGMENT Ths research s supported b the ubqutous Autonoc Coputng and Networ Proect, the Mnstr of Inforaton and Councaton (MIC st Centur Fronter R&D Progra n Korea. REFERENCES [ F. Rosenblatt, "A Probablstc Model for Inforaton Storage and Organzaton n the Bran," Cornell Aeronautcal Laborator, vol. 65, pp , 958. [ R.. Rosenblatt, "The Atlantc speces of the blennod fsh genus", 96. [3 M. Mns, & Papert, S., "Perceptrons: An Introducton to Coputatonal Geoetr," The MIT Press, pp. 3, 6, 3, 33, (