Input Layer f = 2 f = 0 f = f = 3 1,16 1,1 1,2 1,3 2, ,2 3,3 3,16. f = 1. f = Output Layer

Transcription

1 Using the Gow-And-Pune Netwok to Solve Poblems of Lage Dimensionality B.J. Biedis and T.D. Gedeon School of Compute Science & Engineeing The Univesity of New South Wales Sydney NSW 2052 AUSTRALIA ABSTRACT This pape investigates a technique fo ceating spasely connected feed-fowad neual netwoks which may be capable of poducing netwoks that have vey lage input and output layes. The achitectue appeas to be paticulaly suited to tasks that involve spase taining data as it is able to take advantage of the spaseness to futhe educe taining time. Some initial esults ae pesented based on tests on the bit compession poblem. 1. Intoduction Feed-fowad neual netwoks ae geneally esticted to elatively few neuons. Vey lage netwoks take a pohibitively long time to tain and contain so many connections that a huge taining set is often equied in ode to obtain good genealisation. This estiction on netwok size pevents the use of feed-fowad neual netwoks in applications which equie lage numbes of inputs and outputs. The taining time of a feed-fowad netwok may be educed by esticting the numbe of its inteconnections. One way of ensuing a small numbe of connections is to have a vey small numbe of hidden units which ae fully connected to thei suounding layes. Unfotunately netwoks of this sot ae seveely esticted in the amount of infomation they may contain. Bette use may be made of the limited numbe of connections by allowing nodes to instead be spasely connected (fo example see [8]). A lage numbe of nodes may be pesent in a netwok, with the aveage fan-in and fan-out of the units being small. If in addition to the netwok being spase the taining data is spase, it is possible to futhe decease taining time as not all nodes ae necessaily involved in the taining pocess fo each given patten. This pape will hencefoth only conside feed-fowad neual netwoks with one hidden laye, although the gowand-pune (GAP) technique pesented hee may also be applied to netwoks with multiple hidden layes. 2. Backgound While thee has been a vast amount of eseach into the constuction of neual netwok achitectues, most of this wok has been concened with obtaining esults fo poblems of compaatively small dimensionality. One appoach which has been adopted to deal with poblems of lage dimensionality is to subdivide the poblem into a numbe of subpoblems and then to use sepaate fully connected netwoks to addess each one. This division may be done in a numbe of ways: by input patten, by input node o by output node (see [11] in these confeence poceedings fo moe details). In addition hidden units may be gouped in vaious ways so as to educe the numbe of inteconnections. One inteesting suggestion is to use a factal stuctue as a netwok achitectue [9]. It is in geneal not obvious how to best subdivide a high dimensional poblem. This is a concen as inappopiately dividing of a poblem can advesely aect the pefomance of a system. Designing a seies of netwoks by hand to deal with a high dimensional poblem is an ad hoc pocess which is likely to be time consuming and poblematic. It is possible to use clusteing techniques to help identify goupings in the data, although this is an added complication. It also should be noted that thee will not always exist a good way of subdividing a high dimensional poblem. Spase netwoks oe anothe method fo dealing with high dimensional data. Instead of subdividing lage poblems they seek to stoe only the most signicant infomation. Spase netwoks ae often ceated by taining a fully connected netwok and then puning some

2 of its connections. This is commonly done eithe to impove netwok genealisation o to extact ules [3, 6]. Thee ae two poblems with this appoach. Fistly, taining a lage fully connected netwok is a lengthy pocess. Secondly, thee is no guaantee that the spase netwok that esults is close to having an optimal stuctue. Thee has been little eseach into taining netwoks which ae spase thoughout the couse of thei taining. A classication of netwoks whose stuctues do adapt duing the couse of thei taining is to be found in a pape by Fiesle [1]. Few of these netwoks, howeve, appea to be designed in ode to allow supevised leaning to be applied to vey high-dimensional poblems. One inteesting case of a spase neual netwok being applied to a highdimensional poblem is the aea of phoneme pobability estimation [8]. In this pape the autho found a andomly connected netwok pefomed bette than a fully connected with a lage numbe of connections. One technique which is often used to impove netwok pefomance is to add hidden nodes duing the taining of a neual netwok. This educes the likelihood of taining becoming tapped in local minima as the eo suface is fequently changed. It often impoves genealisation and can also educe taining time [2, 10]. The GAP method combines the techniques of gowing and puning to fom spase netwoks. 3. The GAP netwok The GAP netwok stats with a small numbe of fully connected units which ae tained using some easonable taining method (e.g. back-popagation o RPROP [7]). Once taining has levelled out, a numbe of netwok connections ae emoved simultaneously using a puning algoithm. The puning educes the numbe of inteconnections to a oo (call it n) that stays xed fo the duation of the entie taining. Duing the puning stage it may also be desiable to check fo and emove any neuons which have had all thei inputs and/o all thei outputs emoved, as these neuons do not fulll any useful function. Afte puning, a new neuon which is connected to all the input and output neuons is added with andom weights, and taining is ecommenced. It is desiable to use a test set to detemine when the netwok stuctue is to be adjusted and to decide when to nally stop adding hidden nodes. This epetition of puning connections and adding new fully connected nodes causes a spase netwok to develop which can potentially contain a vey lage numbe of nodes. Thoughout the pocess the numbe of connections emains capped, the maximum numbe of nodes being equal to n plus the numbe of links to one fully connected node. The nal numbe of nodes in the netwok is equal to n Using spaseness to limit activation spead It is possible to avoid many calculations in spase netwoks (such as GAP netwoks) when they ae tained with spase input pattens. Units that have only small fan-ins often fail to be activated when thee is spase input. When this occus thee is no need to pefom calculations involving these neuons fo a given patten. In RPROP and back popagation this benet is felt both duing the fowad popagation phase and when eos ae popagated backwads. Theshold values may be placed on neuons to educe the fequency of them ing. If this is not done, two equiements must be met in ode to limit the spead of activation. Fistly, biases must be puned in the same way as odinay weights. If this is not done, all the neuons (except those with 0 biases) ae activated duing the fowad popagation phase. Secondly, neuons must have zeo activity when the sum of thei inputs is zeo. The anti-symmetic sigmoid cuve cented on (0,0) and anging between -1 and 1 is one cuve that satis- es this equiement, although othes may be used. The fomula fo this cuve is: f(x) = 1? e?x 1 + e?x (1) The taining time that is saved by limiting activation spead dies in accodance to the spasity of the netwok and the taining set Limiting weights While testing the GAP netwok it became evident that it is necessay to limit the gowth of the connection weights in some way. If this is not done, the ecently added connections geneally fail to each the magnitude of the moe established connections and ae thus often pematuely puned. One way to educe this poblem is to stop taining just as the oot mean squae (RMS) is stabilising. Taining is stopped when the RMS fails to impove by at least some constant k between consecutive epochs. This esults in each stage of taining being vey quick, and gives the weights little time in which to gow. Two othe methods fo limiting the gowth of weights wee also tested. The second method tested involved decaying each weight by an amount of kw 2 pe epoch. A poblem, howeve, is evealed with this fom of decay when it is epesented as a tansfomation of weights, as below: w 0 = w? kw 2 sign(w) (2)

3 In the equation above w is the oiginal weight and w 0 the tansfomed weight. This tansfomation is illustated in Figue 1. As can be seen, it contains w w Fig. 1: Two tansfomations fo limiting weights. The full line is w 0 = w? kw 2, and is equivalent to using kw 2 decay. k in this case is The dotted line is w 0 = k(1?e?w=k ), which is an exponential method fo limiting weights. k hee is 11. a tuning point and this implies that given two weights w1 > w2 > 0, thee is no guaantee that w 0 1 > w0 2. It thus seems that lage weights ae ovepenalised in some cases. A thid appoach fo limiting the gowth of weights is to tansfom the weights afte each epoch accoding to the following function: w 0 = k(1? e?jwj=k ) sign(w) (3) This function (also shown in Figue 1) places a limit on the magnitude of the weights yet peseves the odeing of the weights. These thee methods fo limiting the gowth of weights will be efeed to espectively as abbeviated taining, kw 2 decay and exponential limitation Specic netwok details Fo the puposes of testing, the GAP netwoks used wee kept as simple as possible fo ease of analysis and eplication. The puning algoithm used simply eliminates the connections whose weights have the smallest absolute values (this is sometimes efeed to as magnitude-based puning). Seveal connections wee emoved at the same time with no taining being done between deletions. No neuons wee emoved duing taining. The RPROP taining method was selected as it geneally pefoms bette than back popagation and it lacks the leaning ate paamete [7]. The weights and update values (used in RPROP) wee not changed when taining was ecommenced afte an adjustment of the netwok stuctue. The activation cuve used was that shown in Equation 1. No noise was employed, no was thesholding. Each of the thee methods mentioned above fo limiting the gowth of weights was tested. Except fo in the abbeviated taining case, taining was stopped at each stage of netwok gowth when the RMS had failed to impove by at least 0.1% ove 20 epochs. 4. Tests 4.1. Measuing success The GAP netwok was tested on the bit compession poblem, which although not a eal-wold poblem does allow fo an easy visual intepetation of the esults. It is also small enough to be un many times. The bit compession poblem has pattens, with each patten having one input set a dieent input in each patten. The output is the same as the input, thus ceating an auto-associative netwok. Fo a netwok with one hidden laye, the optimal spase solution to the poblem has 32 weights and hidden units, with each input connecting to its coesponding output via some hidden unit. Each hidden unit connects one and only one set of matching inputs and outputs. It is possible to compae the eal esults with this ideal. In ode to make this compaison two numbes ae used: the numbe of input-to-output connections, which measues how many coesponding input and output nodes ae connected, and the imbalance of the netwok stuctue, which eectively measues how well the hidden laye is used. In the optimal solution evey input and hidden node has one output link and evey hidden and output node has one input link. That is, evey fan-in and fan-out in the netwok is of size one. The imbalance is the sum of the absolute dieences of 1 and each fan-in o fan-out (see Equation 4 and Figue 2). 4X X imbalance = j1? f ij j (4) i=1 j=1 Input Laye f = 2 f = 0 f = 1 1,1 1,2 1,3 f 2,1= 1 f 2,2= 1 f 2,3= f 3,1= 1 f = 1 f = 0 3,2 3,3 f 4,1= 1 f 4,2= 0 f 4,3= Output Laye Fig. 2: Calculating the imbalance measue f = 3 1, f = 1 2, f = 2 3, f 4,= 1

4 Input Laye Output Laye Fig. 3: Example of a tained netwok An example of a tained netwok is given in Figue 3. The biases have been omitted as all the bias weights wee emoved by puning. The numbe of connections in this example is 15, the imbalance 4 and the nal RMS It would only be necessay to move one connection to give this netwok an ideal stuctue: the link between hidden neuon 14 and output neuon 12 would need to be moved to connect input neuon 15 and hidden neuon 5. It is inteesting to note that thee is aleady a connection between the 5 th hidden neuon and 15 th output unit. Clealy the eo is less having a connection to a hidden neuon that is neve activated than it would be to have a connection to a neuon that is activated by the wong taining pattens Results The GAP netwok was tested using each of the thee weight limitation methods descibed above with seveal values of k fo each. The best values tested fo abbeviated taining, kw 2 decay, and exponential limitation wee , and 0.11 espectively. In all cases the pefomance was faily sensitive to changes in k. This is not, howeve, necessaily a poblem as the best choice of paamete could pove to be elatively independent of the taining set used, at least fo the last two methods. Also tested was a netwok with fully connected hidden nodes which was tained, puned so as to contain 32 connections, and etained. The esults of the the fully connected netwok and the GAP netwoks (using the best values of k) ae given in Table 1. As can be seen the exponential limitation technique out-pefomed the othe methods in all but taining time. The abbeviated taining method caused taining to poceed vey quickly, although the savings have pobably been exaggeated by the simplicity of the taining data. The peiod of attening that was avoided in the case of the abbeviated taining technique made up an unusually high popotion of the taining time of the othe two techniques. It should be noted that the pefomance of the netwok when using the abbeviated taining technique was fa bette than when the RMS was allowed to stabilise futhe, assuming that no othe method was used to limit the gowth of weights. In all cases the GAP netwoks pefomed bette than the fully connected netwok and wee quicke to tain. It may be that the pefomance of fully connected netwok could be impoved by puning the netwok moe gadually, with taining being done between punings. Figue 4 shows the aveage RMS eached befoe each puning occued when using the exponential limitation method with k set to 11. The nal point shown was obtained afte the nal pune and a subsequent taining session. The nal netwok stuctue contained 32 weights wheeas while in the pocess of taining it had Conclusion The fact that the RMS dops as exta nodes ae added is encouaging as it conms that adding exta hidden nodes allows moe infomation to be

5 Taining Input-to-output Final Numbe of Pefect Best k Imbalance method connections RMS multiplications solutions Fully connected, afte puning N/A : % and etaining GAP: Abbeviated taining : % GAP: kw 2 decay : % GAP: Exponential limitation : % Table: 1: The esults fom fou sets of spase netwoks. The st ow was obtained fom a netwok with fully connected hidden nodes which was tained, puned to 32 connections and etained. The emaining thee ows show the GAP esults obtained when using thee methods of weight limitation. Except whee othewise indicated, all values ae the mean of 50 tests. RMS Numbe of hidden neuons Fig. 4: The RMS is shown plotted against the numbe of hidden units. The RMS is measued afte each peiod of taining, but befoe puning. The last point (at position 17) is the RMS afte the nal puning and a subsequent taining. stoed in the netwok even though the numbe of connections emains constant. The numbe of input-to-output connections and the imbalance gues also indicate the achitectue tends to move towads that of the optimal netwok. The bit compession poblem is a useful benchmak poblem fo evaluating GAP achitectues, as it is quick to un and the esults ae easy to intepet. 6. Futhe Wok Puning methods moe sophisticated than magnitude-based puning, such as optimal bain damage [5] and Kanin's method [4], might yield signicant impovements in GAP netwoks, as they ae subject to continuous heavy puning. The adoption of a cascade of neuons could also be wothwhile [10]. A futhe possibility is to eintoduce a few deleted connections thoughout taining, pehaps on a andom basis. The most impotant eseach still to be done is to test the GAP method on a ange of lage ealwold poblems. The authos ae cuently testing the GAP netwok in the context of infomation etieval using standad infomation etieval benchmak tests. Any applications which equie lage numbes of inputs and outputs could in time benet fom the use of the GAP achitectue. Acknowledgements The authos ae gateful to Nicholas Teadgold fo his advice concening gowing neual netwoks and use of the RPROP algoithm. Refeences [1] E. Fiesle. Compaative bibliogaphy of ontogenic neual netwoks. In Poceedings of the Intenational Confeence on Aticial Neual Netwoks (ICANN 94), pages 793{796, [2] D. Hais and T. D. Gedeon. Adaptive insetion of units in feed-fowad netwoks. In Poceedings 4th Intenational Confeence on Neual Netwoks and thei Applications, [3] D. Hais and T. D. Gedeon. Netwok eduction techniques. In Poceedings Intenational Confeence Neual Netwoks Methodologies and Applications, volume 1, pages 119{ 126, [4] E. Kanin. A simple pocedue fo puning back-popagation tained neual netwoks. IEEE Tansactions on Neual Netwoks, 1(2):239{242, [5] Y. Le Cun, J. Denke, and S. Solla. Optimal bain damage. In D. Touetzky, edito, Advances in Neual Infomation Pocessing, volume 2, pages 598{605. Mogan Kauman, [6] R. Reed. Puning algoithms, a suvey. IEEE Tansactions on Neual Netwoks, 4(5):740{ 747, [7] M. Riedmille and H. Baun. A diect adaptive method fo faste backpopagation leaning: The RPROP algoithm. In Poceedings of

6 the IEEE Intenational Confeence on Neual Netwoks (ICNN), pages 586{591, [8] N. Stom. Phoneme pobability estimation with dynamic spasely connected aticial neual netwoks. The Fee Speech Jounal, 1(5), [9] J. P. Sutton. Neuobiological and computational aspects of modulaity. In Poceedings of the Ninth Austalian Confeence on Neual Netwoks (ACNN'98), Febuay [10] N. Teadgold and T. D. Gedeon. A cascade netwok algoithm employing pogessive RPROP. In Biological and Aticial Computation: Fom Neuoscience to Technology. Intenational Wok-Confeence on Aticial and Natual Neual Netwoks (IWANN'97), pages 733{742, [11] W. X. Wen. SGNNN: Self-geneating netwok of neual netwoks. In Poceedings of the Ninth Austalian Confeence on Neual Netwoks (ACNN'98), Febuay 1998.