Parallel and Distributed Systems for Constructive Neural Network Learning*

Transcription

1 Parallel and Disribued Sysems for Consrucive Neural Nework Learning* J. Flecher Z. Obradovi School of Elecrical Engineering and Compuer Science Washingon Sae Universiy Pullman WA Absrac A consrucive learning algorihm dynamically creaes a problem-specific neural nework archiecure raher han learning on a pre-specified archiecure. We propose a parallel version of our recenly presened consrucive neural nework learning algorihm. Parallelizaion provides a compuaional speedup by a facor of O() where is he number of raining ezamples. Disribued and parallel implemenaions under p4 using a nework of Worksaions and a Touchsone DELTA are examined. Ezperimenal resuls indicae ha algorihm parallelizaion may resul no only in improved compuaional ime, bu also in beer predicion qualiy. 1 nroducion A neural nework is a weighed graph of simple processing unis (or neurons). The inerconnecion graph of a feed-forward nework is acyclic wih processing unis arranged in muliple layers consising of inpu, zero or more hidden, and oupu layers. All unis in any layer are fully conneced o he succeeding layer. Unis compue an acivaion funcion of heir weighed inpu sum. Here we consider binary neural neworks where he acivaion funcion of each uni is of he form g(z) : R (, l}, if z< 1 if z 2. Tradiional neural neworks learning (e.g. backpropagaion [12]) involves modificaion of he inerconnecion weighs beween neurons on a pre-specified *Research sponsored in par by he NSF ndusry / Universiy Cooperaive Cener for he Design of Analog-Digial ASCs (CDADC) under gran NSF-CDADC-9-1 and by Washingon Sae Universiy Research Gran 1OC Also affiliaed wih he Mahemaical nsiue, Belgrade, Yugoslavia. nework. Deermining he nework archiecure is a challenging problem which currenly requires an expensive rial-and-error process. n selecing an appropriae neural nework opology for a classificaion problem, here are wo opposing objecives. The nework mus be large enough o be able o adequaely define he separaing surface and should be small enough o generalize well [7]. Raher han learning on a pre-specified nework opology, a consrucive algorihm also learns he opology in a manner specific o he problem. The advanage of such consrucive learning is ha i auomaically fis nework size o he daa wihou overspecializing which ofen yields beer generalizaion. Examples include he iling algorihm of Mizard and Nadal [9] and he cascade-correlaion algorihm of Fahlman and Lebiere [4]. Our goal is o explore he use of disribued and parallel sysems in consrucively learning a single hidden layer binary neural nework archiecure. We argue ha a parallel approach improves compuaional efficiency and generalizaion qualiy. n a single hidden layer feed-forward binary neural nework, each hidden uni wih fan-in k is a represenaion of a k-1 dimensional hyperplane. The hyperplane corresponding o he hidden uni may be deermined hrough soluion of he equaion sysem defined by k poins on he hyperplane. Our work is inspired by a consrucive algorihm proposed by Baum [l] where a sequence of oracle queries are used in conjuncion wih raining examples o find hese k poins. Here he learner is allowed o ask an oracle for he correc class associaed wih arbirary poins in he problem domain in addiion o using he raining examples provided. The hyperplanes are sequenially deermined by pariioning he problem domain space using raining examples and queries. The hidden unis of a single hidden layer feed-forward binary neural nework and corresponding connecions are hen creaed from he hyperplanes. The connecion weighs from he hidden layer o he oupu layer are deermined by an algorihm which separaes he hidden layer represen /93 $ EEE 174 _ ~ ~~~

2 fin i * o Figure 1: Firs unknown region aion of he problem by a single hyperplane (e.g. he percepron algorihm [ll]). n Secion 2 we describe our consrucive learning algorihm which does no require oracle queries. n Secion 3 a new parallel approach o his algorihm is explored wih analysis and experimenal resuls following in Secion 4. 2 Sequenial hidden layer consrucion While Baum s algorihm is applicable where an oracle for he classificaion of any given poin exiss, in many cases such an oracle is no available or may be oo expensive for pracical use. n [5] we proposed a modificaion of Baum s algorihm which does no assume he availabiliy of such an oracle and incremenally consrucs he neural nework from examples alone. n his modificaion, approximaions of he poins on he hyperplane are found by repeaedly inerpolaing beween example poins of he various classes T and Tz in he raining se T. The inerpolaion begins by selecing posiive and negaive exam- ples m E T, n E T2. The unknown region beween m and n is hen searched for he neares poin q E T o he midpoin of m and n. The unknown region is defined as he he circle cenered a he midpoin of m and n wih a diameer of he disance beween m and n, as shown in Figure 1. f q is found, he search is hen repeaed in he smaller unknown region beween q and m or q and n respecively depending on wheher q is posiive or negaive (Figure 2). f no poin from T is found in he curren unknown region, is midpoin p1 is he closes approximaion o a poin on he separaing hyperplane. f p1 is deermined o be wihin a specified olerance of an exising Figure 2: Nex unknown region hyperplane, a new pair of poins is seleced and he search is repeaed. The remaining poins p2 hrough pk ha define he hyperplane are found by aking a random vecor from p1 o a poin v E T (Figure 3) and inerpolaing beween eiher m and v or n and v o pi based on he class of U. The inerpolaed poins from T and he generaed hyperplane are shown in Figure 4. As in Baum s algorihm, he connecion weighs from he hidden layer o he oupu layer unis mus be compued once he hidden uni layer has been generaed. n he modified algorihm, he hidden layer unis are generaed from examples alone, and so may no correspond o he opimal separaing hyperplanes. As such, he hidden layer problem represenaion of he generaed nework wih he same number of hidden unis as in he minimal nework may no be linearly separable. n order o accoun for his possibiliy, hidden unis coninue o be generaed beyond he minimal archiecure; for example, unil he daa is exhaused, a number of daa poins have been examined wihou generaing a new hidden uni or a predeermined number of unis have been creaed. The pocke algorihm [6] is a single-layer neural nework learning algorihm ha finds he opimal separaion under a given opology for problems ha are no linearly separable. The algorihm keeps he bes se of weighs in he pocke while he percepron is rained incremenally. A pracical modificaion of he pocke learning algorihm is proposed in [lo] which is faser and sill has he same guaranee for convergence o he opimal separaing hyperplane. This parallel dynamic algorihm is used o deermine he oupu layer weighs in he consruced nework.

3 i! Figure 3: Random vecor n Figure 4: Separaing hyperplane Oupu Uni Hidden Unis (Consruced From Separaing Hyperplanes) npu Unis Figure 5: Hidden layer consrucion 3 Parallel hidden layer consrucion While he sequenial algorihm provides good generalizaion, significan compuaional resources are required. Here we propose a speedup by a parallelizaion ha disribues he compuaional load across a number of processors. n order for parallelizaion o be efficien, an appropriae pariioning of he inpu space is required. This is accomplished by assigning he example poins of one class evenly across he available processors. Given raining se T of examples belonging o classes 2'1 and 2'2 le l and z (l 2 z) be he number of examples in each class respecively. n a sysem wih P 1 processors each of P slave processors (1 5 P 5 l) is assigned rl/p1 examples of class T. Each slave processor examines he inpu subspace formed by pairing is assigned examples of T wih all examples in 2'2. A sysem wih a balanced compuaional load is obained by his pariioning of he iniial pairs. Figure 6 shows he proposed parallel archiecure. Each processor may be eiher a worksaion in a disribued environmen or a processor on a parallel machine. One processor is responsible for he maser process. This maser process disribues he raining daa a iniializaion and creaes neural nework hidden layer unis from he deermined separaing hyperplanes. All slave processors search for separaing hyperplanes as described in Secion 2 saring from he iniial pairs in heir assigned daa pariions. When such a separaing hyperplane is found, i is communicaed o he maser process. The maser process hen compares he hyperplane o hose ha currenly exis. f i is no sufficienly similar o an exising hyperplane, a new hidden uni corresponding o he hyperplane is generaed (Figure 5). Hidden layer consrucion is compleed when a predeermined number of hidden unis have been generaed, he inpu space has been exhausively searched, or a number of iniial pairs have been examined wih- 176

4 Maser Processor.. Slave Processors Figure 6: Parallel archiecure ou deermining a new separaing hyperplane. Finally, he maser process performs he relaively simple ask ofraining he oupu layer weighs as in he algorihm of Secion 2. 4 Analysis and Experimenal Resuls The oal running ime of our algorihm depends primarily upon he ime required o deermine if a separaing hyperplane can be consruced saring from a given pair of raining examples. Search for a poin on he hyperplane akes O(1og) inerpolaion seps since each inerpolaion removes a leas half of he raining examples. n each inerpolaion sep, finding he neares raining example o he cener of an unknown region can be deermined in ime O(1og) hrough use of he k-d ree of Benley [2]. Thus, he wors case ime required o search for one poin on he hyperplane is O(log2 ). A hyperplane is defined by k poins, and so he oal ime o deermine if a hyperplane can be found saring from a given pair of raining examples is O(~C log2 ). n he sequenial algorihm an exhausive daa pariioning saring from all l2 raining pairs of examples can be performed in wors case ime of O(kl2 log2 ). n he parallel algorihm an iniial overhead of O() is required for daa disribuion. A minimal overhead of O(k) is incurred for ransfer of generaed hyperplane daa from he slave o he maser processor. Since l 2 = he wors case parallel ime for an exhausive daa pariioning is hus O((kl21g2)/P) where P is he number of slave processors. n learning problems w.1.o.g. we can assume ha boh l and 2 are of order O() as boh classes have o be well represened in he raining se. Wih ha assumpion he wors case parallel compuing ime of he maximal disribued sysem (P = max(1, 2)) is O(k log2 ) compared o a sequenial ime of (k2 log2 ). Algorihm parallelizaion hus provides a compuaional speedup by a facor of O(). The algorihm was implemened using p4 [3]. Developed a Argonne Naional Laboraory, p4 suppors parallel programming for boh disribued environmens and highly parallel compuers. Two implemenaion plaforms were used: a disribued sysem of 19 DECSaions and a Touchsone DELTA. The Touchsone [8] is an nel high-speed concurren mulicompuer, consising of 576 nodes in a 19 x 36 mesh. Of hese, 64 nodes were allocaed for our experimens. mplemenaion under p4 allowed he same code o be used for he Touchsone as for he DECSaion nework. Experimens were performed using he MONK S problems [13] o compare he qualiy of generalizaion beween he sequenial and parallel implemenaions. The MONK S problems consis of hree six-feaure binary classificaion problems which represen specific challenges for sandard machine-learning algorihms, such as he abiliy o learn daa in disjuncive normal form, pariy problems and performance in he presence of noise. To allow he random vecor o search equally in each dimension, he inpu daa is normalized o poins on a hypersphere.

5 Processors Accuracy Problem 1 1. Problem Problem Sequenial Disribued Parallel Train Tes Train Tes Train Tes Table 1: Percenage Accuracy on he MONK s Problems The generalizaion abiliy of he sequenial and parallel implemenaions is compared in Table 1. is ineresing o noe ha in he more complex problems 2 and 3 he generalizaion of he parallel algorihm exceeds ha of he sequenial algorihm. This improvemen may be due o he fac ha as he number of processors increases, a greaer diversiy in he inpu space will be searched. While hese resuls are promising, he principles described here are being furher evaluaed on he largescale problem of predicing proein srucure. 5 Conclusions Neural neworks efficiency and predicion qualiy depends significanly on how we selec nework archiecure, learning algorihm and iniial se of weighs. The consrucive learning algorihm of Secion 2 efficienly learns no jus connecion weighs bu also creaes he required archiecure. A parallel version proposed in Secion 3 provides a significan speed-up in he consrucion of he hidden layer and a greaer diversiy in he inpu space searched, also resuling in improved generalizaion qualiy. References [l] E. B. Baum. Neural ne algorihms ha learn in polynomial ime from examples and queries. EEE Transacions on Neural Neworks, 2( 1):5-19, January [2] J. L. Benley. Mulidimensional binary search ree used for associaive searching. Communicaions of he ACM, 18(9):59-517, Sepember [3] R. Buler and E. Lusk. User s Guide o he p4 Parallel Programming Sysem, November [4] S. Fahlman and C. Lebiere. The cascadecorrelaion learning archiecure. n D. Toure- zky, edior, Advances in Neural nformaion PTOcessing Sysems, volume 2, pages , Denver 1989, 199. Morgan Kaufmann, San Maeo. [5] J. Flecher and Z. Obradovid. Creaion of neural neworks by hyperplane generaion from examples alone. n Neural Neworks for Learning, Recogniion and Conrol, page 23, Boson, [6] S.. Gallan. Percepron-based learning algorihms. EEE Dansacions on Neural Neworks, 1(2): , June 199. [7] S. Geman, E. Biensock, and R. Doursa. Neural neworks and he bias / variance dilemma. Neural Compuaion, 4(1):1-58, [8] nel Supercompuer Sysems Division, Beaveron, OR. Touchsone Dela Sysem User s Guide, Ocober [9] M. Mizard and J.-P. Nadal. Learning in feedforward layered neworks: The iling algorihm. Journal of Physics A, 22: , [lo] Z. Obradovid and R. Srikumar. Dynamic evaluaion of a backup hypohesis. n Neural Neworks for Learning, Recogniion and Conrol, page 71, Boson, [ll] F. Rosenbla. Principles of Neurodynamics. Sparan, New York, D. Rumelhar, G. Hinon, and R. Williams. Learning inernal represenaions by error propagaion. n D. Rumelhar and J. McClelland, ediors, Parallel Disribued Processing, volume 1, chaper 8, pages MT Press, Cambridge, [13] S. B. Thrun e al. The MONK s problems: A performance comparison of differen learning algorihms. Technical Repor CMU-CS , Deparmen of Compuer Science, Carnegie Mellon Universiy, Pisburgh, PA, 1991.