Shortcomings with Tree-Structured Edge Encodings for Neural Networks

Transcription

1 Shortcomings with Tree-Structured Edge Encodings for Neurl Networks Gregory S. Horny QSS Group Inc., NASA Ames Reserch Center Mil Stop 269-3, Moffett Field, CA Astrct. In evolutionry lgorithms common method for encoding neurl networks is to use tree-structured ssemly procedure for constructing them. Since node opertors hve difficulties in specifying edge weights nd these opertors re execution-order dependent, n lterntive is to use edge opertors. Here we identify three prolems with edge opertors: in the initiliztion phse most rndomly creted genotypes produce n incorrect numer of inputs nd outputs; vrition opertors cn esily chnge the numer of input/output (I/O) units; nd units hve connectivity is sed on their order of cretion. Insted of creting I/O nodes s prt of the construction process we propose using prmeterized opertors to connect to pre-existing I/O units. Results from experiments show tht these prmeterized opertors gretly improve the proility of creting nd mintining networks with the correct numer of I/O units, remove the connectivity is with I/O units nd produce etter controllers for gol-scoring tsk. 1 Introduction Neurl networks re one of the more common types of controllers used for rtificil cretures nd evolutionry rootics [1]. Since representtions tht directly encode the weights nd connections of network hve sclility prolems indirect representtions must e used for lrger networks lthough to chieve etter sclility the indirect representtion must llow for reuse of the genotype in creting the phenotype [2]. One type of indirect representtion tht is ecoming incresingly populr for encoding neurl networks is to use tree-structured genotype which specifies how to construct them. Advntges of indirect, treestructured representtions re tht they etter llow for vrile sized networks thn directly using weight mtrix, nd Genetic Progrmming style recomintion etween two trees is esier nd more meningful thn trying to swp su-networks with grph-structured representtion. One of the originl systems for encoding neurl networks in tree-structured ssemly procedures is cellulr encoding [3]. Yet cellulr encoding hs een found to hve shortcomings due to its use of node opertors: sutrees swpped through recomintion do not produce the sme sugrphs ecuse node opertors re execution-order dependent nd specifying connection weights is pro- K. De et l. (Eds.): GECCO 2004, LNCS 3103, pp , c Springer-Verlg Berlin Heidelerg 2004

2 496 G.S. Horny lemtic since node opertors cn crete n ritrry numer of edges [4]. Consequently, of growing interest is the use of edge-encoding commnds in which opertors ct on edges insted of nodes [5,6,7]. In this pper we point out three different shortcomings of edge-encoding lnguges. First, regrdless of whether the first N nodes re tken s input/output (I/O) units or if specil node-construction commnds re used for creting I/O units, when creting n initil popultion it is difficult to ensure tht rndomly creted genotypes hve the correct numer of them. A second prolem is tht s evolution proceeds the vrition opertors hve high proility of chnging the genotype so tht it produces networks with incorrect numers of I/O units. Finlly, more serious prolem with tree-structured ssemly procedures is the node cretion-order connectivity is (NCOCB). The NCOCB prolem is tht nodes creted from edge opertors t the ottom of the genotype will hve only single input nd output, wheres nodes creted from opertors higher up in the genotype will hve connectivity proportionl to 2 height. One wy to ddress the prolems of producing the correct numer of I/O nodes nd the NCOCB with I/O nodes is y chnging the construction lnguge. Rther thn hving commnds in the lnguge for creting new I/O unit, or ssigning the Nth creted unit s the ith I/O unit, we propose strting network construction with the desired numer of I/O units nd then using prmeterized-connection opertors for dding edges to these units. Prolems in creting nd mintining networks with the correct numer of I/O units re reduced since ll networks strt with the desired numer nd no commnds exist for creting/removing them. Also, prmeterized connection commnds men tht the expected numer of connections for ll I/O units is equl for rndomly creted genotypes nd does not suffer from the NCOCB. In the following sections we first descrie cnonicl method for using edge encoding opertors to represent neurl networks s well s our prmeterized opertors for connecting to I/O units. Next we present our experiments which show the different ises with stndrd edge-encoding opertors nd demonstrte tht evolution with the prmeterized opertors for connecting to I/O units produces etter controllers on gol-scoring tsk. Finlly we close with discussion on the underlying prolem with edge opertors nd tree-structured representtions nd conclusion in which we restte our findings. 2 Encoding Neurl-Networks In this section fter descriing the type of neurl networks tht we wnt to evolve we then descrie tree-structured representtion for encoding them, followed y two different methods for hndling input nd output (I/O) units. The first method for hndling I/O units uses stndrd edge-encoding lnguge (SEEL) nd hs specil commnds for creting I/O nodes. Since this method hs prolems in creting the correct numer of I/O nodes nd lso hs node cretion-order connectivity is (NCOCB) we then descrie second method for hndling I/O units. In this second method the initil network strts with

3 Shortcomings with Tree-Structured Edge Encodings for Neurl Networks 497 the desired numer of I/O units nd opertors in the lnguge connect to them using opertor prmeters to specify which unit to connect to (PEEL, for prmeterized edge-encoding lnguge). 2.1 Neurl Network Architecture The neurl networks used in these experiments re continuous-time, recurrent networks similr to those of Beer nd Gllgher [8], nd of our previous work [9,10]. Ech non-input neuron hs n input is, θ, nd time constnt, τ. The ctivtion vlue of non-input neuron i t time t is: ( ) i,t = τ i i,t 1 +(1 τ)tnh W ji j,t 1 + θ i (1) For input neurons, their ctivtion vlue is the vlue of the corresponding sensor. j 2.2 Creting Network from Tree The different methods for encoding neurl networks or grphs with treestructured ssemly procedure ll strt with single node nd edge nd then new nodes/edges re dded y executing the opertors in the ssemly procedure. Using n edge-encoding lnguge in which grph-construction opertors ct on the edge connecting from unit A to unit B, typicl set of commnds re s follows. dd reverse cretes link from B to A. dd split(n) cretes new neuron, C, with is of θ = n, nd dds new link from A to C nd cretes new edge connecting from neuron C to neuron B. The is of this node is set to θ = n, nd the time constnt is set to zero. dd split cont(m, n) cts the sme s dd split(), only it cretes continuous time neuron with is of θ = m nd time constnt of τ = n. connect cretes new link from neuron A to neuron B. dest to next chnges the to-neuron in the current link to its next siling. loop cretes new link from neuron B to itself. The no-op commnd performs no opertion. set weight(n) sets the weight of the current link to n. source to next chnges the from-neuron in the current link to its next siling. source to prent chnges the from-neuron in the current link to the inputneuron of the current from-neuron. Of these commnds dd split(n) nd dd split cont(m, n) hve exctly three children commnds since fter their execution the edge they ct on ecomes three edges. The set weight(n) commnd hs no children, consequently it is lwys lef node nd the no-op hs either zero or one children so it cn e either lef node nd hlt the development of the grph on the current edge,

4 498 G.S. Horny or it cn e used to dely execution on the current edge for round llowing time for the rest of the grph to develop more. The rest of the commnds result in the ddition of single new edge to the grph so they hve exctly two children commnds: one to continue grph construction long the originl edge one commnd to construction long the new edge. Using the commnds descried ove the sequence of grphs from figure 1-i shows the construction of network from the genotype in figure 1.. Grphs re constructed from this genotype y strting with single neuron linked to itself, figure 1., nd executing the commnds in the ssemly procedure in redthfirst order. First, executing split(0.01) dds node with is of 0.01 nd pir of links, figure 1.c. The commnd set-weight() sets the weight of the link to, no-op performs no opertion, nd then split(0.02) results in the cretion of neuron c with is of 0.02 nd two more links, figure 1.d. Source-to-prent cretes second link, nd set-weight(0.2) sets the weight of the link to 0.2, figure 1.e. The second source-to-prent commnd cretes the link c, executing set-weight(0.3) sets the weight of the link c to 0.3 nd set-weight(-0.2) results in weight of -0.2 ssigned to the link, figure 1.f. The source-to-next commnd results in the link eing creted, figure 1.g. The commnd set-weight(0.4) sets the weight of link c to 0.4 nd then executing connect cretes n dditionl link c, figure 1.h. Executing set-weight(0.5) sets the weight of link to 0.5, set-weight(0.6) sets the weight of link to 0.6, no-op sets the weight of link to 0.0, nd set-weight(0.7) sets the weight of link c to 0.7, figure 1.i. In ddition, fter ll tree-construction opertors hve een executed, there is pruning phse tht consolidtes the weights of links with the sme source nd destintion neurons, figure 1.j, nd removes hidden neurons tht re not on directed pth to n output neuron. 2.3 Stndrd Edge Encoding Lnguge The grph-construction lnguge of the previous su-section cn e used to crete neurl networks either y ssigning the first n units s I/O units or y dding commnds specificlly for creting I/O units. Assigning ritrry units to e I/O units hs the drwck tht chnges in the genotype cn dd/delete units in the network so tht units shift position nd wht ws the ith input unit in the prent ecomes the i + 1 input unit in the child. To void this disruption the SEEL we use hs specilized commnds for creting I/O units. I/O units re creted through the use of the dd input nd output split(n) commnds. Since these re edge opertors, we lel the edge they re ssocited with to e from the vertex A to the vertex B. Executing the dd input commnd cretes new input unit nd n edge connecting from this unit to A. Output units re creted with the output split(n) commnd, which performs split on the existing edge nd the newly creted neuron is set s n output unit with is of θ = n.

5 Shortcomings with Tree-Structured Edge Encodings for Neurl Networks 499 split(0.01) set weight() source to prent no op split(0.02) set weight(0.2) set weight(0.3) source to prent set weight( 0.2) source next set weight(0.4) connect set weight(0.5) set weight(0.6) () no op set weight(0.7) () (c) (d) c c c (e) (f) (g) 0.3 c c c c (h) (i) (j) Fig. 1. A tree-structured encoding of network (), with dshed-lines to seprte the lyers, nd (-j) construction of the network it encodes.

6 500 G.S. Horny 2.4 Prmetric Edge Encoding Lnguge A method to remove the connectivity is with I/O nodes is y hving these nodes exist in the initil grph nd then dding connections to them, such s with the commnds connect input(i) nd connect output(i). Leling the current edge s connecting from unit A to unit B, connect input(i) cretes link from the ith input neuron to B nd connect output(i) cretes link from B to the ith output neuron. Since ech of these commnds cretes new edge, oth commnds hve exctly two children opertors: one to continue network construction long the originl edge nd one to construct long the new edge. 3 Experiments In this section we present our experiments compring SEEL with PEEL. First we show tht rndomly creted genotypes using SEEL hve prolems producing networks with the desired numer of I/O neurons wheres this prolem is gretly reduced when using PEEL. Next we show tht networks encoded with PEEL re more roust to mintining the correct numer of I/O units under muttion nd recomintion thn re networks encoded with SEEL. In our third set of experiments we demonstrte the existence of the node cretion-order connectivity is. Finlly, we demonstrte tht using PEEL results in etter neurl-controllers for the evolution of gol-scoring ehvior. 3.1 Initiliztion Comprison One enefit of strting with the desired numer of I/O neurons is tht rndomly creted, network-constructing, ssemly procedures re more likely to hve the correct numer of I/O neurons. This cn e shown y compring the numer of vlid networks creted using oth network construction lnguges. A network is considered vlid if it hs four input neurons nd four output neurons (ritrry vlues selected for this experiment) nd for ech input neuron there is pth to t lest one output neuron nd ech output neuron is on pth from t lest one input neuron. Tle 1 shows the numer of vlid networks creted from ten thousnd rndomly creted ssemly procedures for vrious tree depths. From this tle it cn e seen tht vlid networks re significntly more likely to e creted with PEEL thn with SEEL. The reson PEEL does not score 100% even though it strts with the correct numer of I/O neurons is ecuse some input neurons my not e on pth to n output neuron. 3.2 Vrition Comprison In ddition to the prolem of creting initil individuls with the correct numer of I/O units, SEELs hve difficulty mintining these numers under muttion nd recomintion. To show tht PEELs etter mintin vlid networks we compre the numer of networks tht still hve four inputs nd four outputs fter muttion nd recomintion from vlid prents.

7 Shortcomings with Tree-Structured Edge Encodings for Neurl Networks 501 Tle 1. Numer of vlid networks generted out of ten thousnd rndomly creted tree-structured ssemly procedures. Depth SEEL PEEL For this comprison the muttion opertor modifies n individul y chnging one symol with nother, perturing the prmeter vlue of symol, dding/deleting some symols, or recomining n individul with itself. Two types of recomintion re used, with equl proility of using one or the other. The first recomintion opertor is the stndrd GP recomintion tht swps rndom sutrees etween prents [11]. The second recomintion opertor is similr to one-point crossover [12] nd we cll it mtched recomintion (MR). MR works y lining up two trees nd, strting t the root, mtches up the children nodes y type nd rgument vlues, finds the loctions t which sutrees differ nd then picks one of these plces t rndom to swp. Since rndom trees of depth seven produced the most vlid networks with SEEL, we compred ten thousnd muttions nd recomintions etween SEEL nd PEEL on vlid, rndomly creted individuls. With SEEL muttion hd success rte of 84.8% nd recomintion hd success rte of 79.2%. In comprison, with PEEL muttion produced vlid children 93.5% of the time nd recomintion did so 89.5% of them. These results show tht networks encoded with PEEL re more roust to vrition opertors. 3.3 Node Cretion Order Connectivity Bis A more serious prolem with tree-structured ssemly procedures is the node cretion-order connectivity is (NCOCB). Nodes creted from commnds erly in the construction process tend to hve greter numer of edges into nd out of them then nodes creted lter in the the process. One consequence of this is is tht I/O neurons tht re creted erly in the construction process will hve significntly higher numer of outputs/inputs thn those I/O neurons creted t the end of the construction process. The grph in figure 2. shows the verge connectivity (sum of inputs nd outputs) of node plotted ginst its cretion order. From this grph it cn e seen tht nodes creted erlier in the construction process hve more connections thn those creted lter nd most nodes only hve two connections: one input nd one output link. Thus if I/O nodes re creted y the tree-structured ssemly procedure, the first I/O nodes will hve significntly more inputs/outputs from/to them thn those creted lter in the construction process. For input neurons, this suggests tht the first inputs re likely to hve greter influence on the ehvior of the network thn the ltter inputs nd for output neurons this suggests tht more processing of inputs is hppening for the ctivtion vlues of the first output neurons thn for the ltter output neurons.

8 502 G.S. Horny connectivity depth 25 depth 20 depth 15 depth 10 depth neuron cretion numer () numer of nodes connectivity () depth 25 depth 20 depth 15 depth 10 depth 5 Fig. 2. Grphs of () the verge node connectivity y order of cretion, nd () the numer of nodes with given connectivity for rndomly creted individuls. Becuse the connectivity of node is strongly ised y its height in the treestructured ssemly procedure nd since most commnds re ner the leves in the tree this results in is in the distriution of the numer of nodes with given connectivity. Most nodes in the network will hve connectivity of two one input nd one output nd the numer of nodes with given connectivity decreses exponentilly (figure 2.). 3.4 Comprison on Evolving Gol-Scoring Behvior While PEEL hs een shown to e etter thn SEEL for removing vrious ises, of prcticl importnce is whether evolution with PEEL produces etter controllers thn evolution with SEEL. To test this we evolve neurl-controllers for gol-scoring tsk. Gol-scoring tkes plce in computer-simulted, 275x152.5 wlled, soccer field with gols t ech end (figure 3.). Inside the soccer field is two-wheeled, soccer plyer which hs seven sensor inputs (three sensors to detect distnce to the wll (one pointing directly in front nd the other two t 30 to the left nd right), nd four sensors tht return ngle to the ll, distnce to the ll, ngle to the gol nd distnce to the gol) nd two outputs (desired wheel-speed for the left nd right wheels) (figure 3.). Evluting n individul consists of summing the score from eight trils, two ech with the ll initilly plced in ech of the four corners of the field, nd the soccer-plyer plced in the middle of the field. Initil loctions for oth the plyer nd ll re pertured y smll rndom mount nd then the plyer is given 60 seconds (t 30fps this results in 1800 network updtes) to score s mny gols s it cn. For ech gol scored the distnce from the vehicle s strting position to the ll plus the distnce from the ll s initil loction to the gol is dded to the network s score. After gol is scored the ll is rndomly locted t the center of the field (±30, ±30), the minimum distnces to the ll nd to

9 Shortcomings with Tree-Structured Edge Encodings for Neurl Networks 503 Bll sensor Gol sensor Wll sensors () () Fig. 3. () The soccer field nd () the soccer plyer nd its sensors. the gol re reset, nd the network is llowed to try to score nother gol. Once time runs out, network s score is incresed y how much closer it moved the plyer to the ll nd how much closer it moved the ll to the gol. In ddition, if the plyer scores n own-gol, its score is reduced y the distnce it moved the ll from its strting position to the gol. To perform these experiments the EA ws run on Linux-PC with evlutions frmed out to five PlySttion R 2 1 development systems. Ech experiment consisted of evolving fifty individuls for fifty genertions. A genertionl EA ws used nd new individuls were creted with equl proility of either muttion or recomintion nd n elitism of three. Evluting one genertion of fifty individuls took pproximtely four minutes. The results of experiments re shown in figure 4 nd show tht evolution with PEEL produces soccer plyers lmost twice s fit s with SEEL SEEL PEEL 2000 fitness genertion Fig. 4. Fitness of the est evolved gol-scores verged over four trils. 1 PlySttion is registered trdemrk of Sony Computer Entertinment Inc.

10 504 G.S. Horny The higher fitness of networks encoded with PEEL is reflected in the ehviors produced y these networks. Networks encoded with SEEL produced soccer plyers tht tended to spin in plce nd move wkwrdly or in looping pttern. These networks only moved towrd the ll somewht hphzrdly nd generlly did not pper to e iming their shots. In contrst, networks encoded with PEEL would move to position themselves on the other side of the ll from the gol nd then either push the ll towrd the gol or spin to kick it towrd the gol. The est of these networks seldom missed in its shots nd n exmple of its ehvior is shown in the sequence of imges in figure 5. 4 Discussion While the results of the experiments section show tht vrious ises hold for the edge-encoding lnguges presented here, of interest is the degree to which these ises exist in other edge-encoding lnguges. The edge-encoding lnguge of section 2 differs from Luke s [4] in tht edges re not explicitly deleted, rther they dispper if they re not ssigned weight, nd the split commnd does not delete the link when it cretes the new neuron c nd links c nd c. A commnd for explicitly deleting links would not necessrily chnge the ises in resulting networks since the no-op commnd with no children hs the sme effect. In contrst, since the split opertor used here dds links to existing neurons without removing ny, it should produce lrger is thn Luke s split opertor. Although the differences in opertors etween different edge encoding lnguges ffect the degree of connectivity is tht cn e expected, the min cuse of the ises is the tree-structure of the representtion. When neuron is creted it hs single input nd output edge. Since edge opertors cn dd one input or output to n existing neuron (except for the loop commnd, which dds oth n input nd n output) the expected connectivity of neuron is on the order of 2 height. Since PEEL only ddresses the NCOCB for I/O units nd does not scle for lrge networks the direction to go for ddressing the vrious shortcomings of edge encodings is not cler. One wy to remove the NCOCB is to chnge from tree-structured to grph-structured genotypes, ut then there re difficulties in creting meningful recomintion opertors. Another wy is to switch to opertors in which the connectivity of new node is not dependent on its depth in the genotype; ut these would e node opertors which hve their own shortcomings [4]. 5 Conclusion In this pper we identified three shortcomings with typicl edge encoding opertors for representing neurl networks: individuls creted t rndom in the initiliztion phse do not usully hve the correct numer of inputs/outputs; vrition opertors cn esily chnge the numer input/output neurons; nd the node cretion-order connectivity is (NCOCB). To ddress these prolems we

11 Shortcomings with Tree-Structured Edge Encodings for Neurl Networks 505 () () (c) (d) (e) (f) Fig. 5. An evolved gol-scorer in ction: ()-(c) the soccer-plyer circles round the ll; (d) it pushes the ll towrd the gol; (e)-(f), while the ll is going into the gol the plyer moves to the center of the field where the ll will re-pper fter the gol is scored. proposed using prmeterized opertors for connecting to input/output units nd demonstrted tht evolution with these opertors produces etter neurl networks on gol-scoring tsk. While these prmeterized opertors gretly improve the proility of creting nd mintining networks with the correct numer of input/output units it does not ddress the NCOCB prolem for hidden units. Consequently the contriution of this pper is more n oservtion tht these shortcomings exist. Future work with edge encoding opertors will

12 506 G.S. Horny need to ddress more generl solutions to these prolems tht scle with the size of the network nd work for hidden units. Acknowledgements. Most of this reserch ws conducted while the uthor ws t Sony Computer Entertinment Americ, Reserch nd Development nd then t Brndeis University. The soccer gme nd simultor ws developed y Eric Lrsen t SCEA R&D. References 1. Nolfi, S., Floreno, D., eds.: Evolutionry Rootics. MIT Press, Cmridge, MA (2000) 2. Horny, G.S., Pollck, J.B.: Creting high-level components with genertive representtion for ody-rin evolution. Artificil Life 8 (2002) Gruu, F.: Neurl Network Synthesis Using Cellulr Encoding nd the Genetic Algorithm. PhD thesis, Ecole Normle Supérieure de Lyon (1994) 4. Luke, S., Spector, L.: Evolving grphs nd networks with edge encoding: Preliminry report. In Koz, J., ed.: Lte-reking Ppers of Genetic Progrmming 96, Stnford Bookstore (1996) Brve, S.: Evolving deterministic finite utomt using cellulr encoding. In Koz, J.R., Golderg, D.E., Fogel, D.B., Riolo, R.L., eds.: Genetic Progrmming 1996: Proceedings of the First Annul Conference, Stnford University, CA, USA, MIT Press (1996) Horny, G.S., Pollck, J.B.: Body-rin coevolution using L-systems s genertive encoding. In: Genetic nd Evolutionry Computtion Conference. (2001) Koz, J., Bennett, F., Andre, D., Kene, M.: Genetic Progrmming III: Drwinin Invention nd Prolem Solving. Morgn Kufmnn (1999) 8. Beer, R.D., Gllgher, J.G.: Evolving dynmicl neurl networks for dptive ehvior. Adptive Behvior 1 (1992) Horny, G.S., Mirtich, B.: Diffuse versus true coevolution in physics-sed world. In Bnzhf, W., et l., eds.: Proc. of the Genetic nd Evolutionry Computtion Conference, Morgn Kufmnn (1999) Horny, G.S., Tkmur, S., Hngt, O., Fujit, M., Pollck, J.: Evolution of controllers from high-level simultor to high dof root. In Miller, J., ed.: Evolvle Systems: from iology to hrdwre; Proc. of the Third Intl. Conf. Lecture Notes in Computer Science; Vol. 1801, Springer (2000) Koz, J.R.: Genetic Progrmming: on the progrmming of computers y mens of nturl selection. MIT Press, Cmridge, Mss. (1992) 12. Poli, R., Lngdon, W.B.: Schem theory for genetic progrmming with one-point crossover nd point muttion. Evolutionry Computtion 6 (1998)