A SCALABLE DIGITAL ARCHITECTURE OF A KOHONEN NEURAL NETWORK

Transcription

1 A SCALABLE DIGITAL ARCHITECTURE OF A KOHONEN NEURAL NETWORK Andrés E. Valenca, Jorge A. Peña and Maurco Vanegas. Unversdad Pontfca Bolvarana, Medellín, Colomba andrez_valenca@yahoo.com, jorge.pena@alar.ch, mvanegas@upb.edu.co ABSTRACT Kohonen self-organzng feature maps are unsupervsed learnng neural networks that categorze or classfy data. Effcent hardware mplementaton of such neural networks requres the defnton of a certan number of smplfcatons to the orgnal algorthm. In partcular, multplcatons should be avoded by means of smplfcatons n the dstance metrc, the neghborhood functon and the learnng parameter values. In many applcatons, a scalable soluton becomes necessary due to the lmted memory resources avalable n many embedded platforms. In ths paper, a scalable Kohonen map called Local Wnner-Take- All (LWTA) desgn wth Mnkowsk norm L and an exponental neghborng functon s defned, and ts hardware archtecture s presented. The scalablty of the net s acheved va a Local Wnner Take- All approach. Results of VHDL smulatons as well as synthess on an FPGA of the proposed archtecture demonstrate satsfactory functonalty of such archtecture. 1. INTRODUCTION Artfcal neural networks are parallel computatonal models comprsed of densely nterconnected adaptve processng unts (neurons), able to learn a statc map (supervsed learnng) or to classfy and categorze the nput space (unsupervsed learnng) from an nput data [1]. A Kohonen s self-organzng feature map (Kohonen s SOFM) s an artfcal neural network of unsupervsed learnng that captures the topology and probablty dstrbuton of nput data [2]. Most of the neural networks applcatons have used smulatons on conventonal sngle-processor machnes. The possblty of parallel processng and short operaton tmes has encouraged the mplementaton of hardware neural networks [3], [4], [5], [6]. Kohonen s SOFM has not been the excepton, and there are several mplementatons n analog [7] as well as dgtal crcuts [8] [9]. Generally, dgtal mplementatons have been more successful because of ther lesser vulnerablty to nose and ther hgher scale of ntegraton when s compared wth ther analog counterparts, addtonally several laboratores currently work strongly on reconfgurable crcuts n whch ths knd of dgtal archtectures wll be sutable[10]. In many applcatons, the data load can ncrease at an ncredble rate and overcome the resources of the system. Moreover due to the lmted memory resources avalable n many embedded platforms, such as moble termnals [11], a scalable soluton becomes necessary. In [12] we proposed a very smple but effcent dgtal archtecture of a Kohonen s SOFM. In ths work, we add a new feature to ths desgn n order to make t more easly scalable. The dea s to be able to construct a larger map by connectng smlar chps (e.g. FPGAs mplementng a certan number of nterconnected neurons) wthout the necessty to change the whole archtecture. 2. KOHONEN S SELF ORGANIZING FEATURE MAPS A Kohonen s self-organzng feature map s an unsupervsed learnng neural net that captures the topology and probablty dstrbuton of nput data. Its archtecture conssts of an array of unts or neurons wth a fxed poston R wthn the map and a varable n- dmensonal weght W, where n s the dmensonalty of nput patterns. 1

2 The weghts of the neurons are updated each tme a new nput pattern s presented. The magntude of the change of the weght of the neuron depends on the topologcal proxmty to the wnner neuron, whose ndex * s gven by: = arg mn d ( w, x * k When the wnner neuron s found, the weght of the -th neuron s updated accordng to: k w = w + w = w + ρ Φ( r, r * )( x wo) o o where ρ s the learnng rate and Ф(r, r * ) s a neghborhood functon. 3. THE PROPOSED KOHONEN S SOFM As descrbed n [12], the LWTA desgn s based on the dea to smplfy the algorthm of self organzng feature maps due to hardware aspects n order to mnmze the necessary chp area and thus to maxmze the number of processng unt per FPGA. Frst the Chessboard dstance s used nstead of the Eucldean dstance. L ( x, y) = max x y where L (x,y) s the dstance metrc between vectors x and y and x (y ) s the -th component of the vector x (y). ) 4. HARDWARE DESIGN A one-dmensonal Kohonen map of 10 unts s desgned; the system s capable of handlng a two-dmensonal nput patter wth 8-bt resoluton The Network The network s desgned to work n parallel and to optmze executon speed. Each neuron s defned as a buldng block that processes data n an ndependently fashon. The proper behavor of the net as an aggregaton of these buldng blocks s guaranteed by one global control block, the parameter scheduler. Consequently, when several FPGAs are connected n the LWTA desgn, the learnng rate sgnal, α, the neghborhood wdth sgnal, β, and the reset sgnal must be connected as well as the sgnals downout and downn wth the sgnals upn and upout of the new FPGA (Fg. 1) The Neuron A Kohonen s neuron s composed of a Chessboard dstance calculaton block and a weght update block (Fg. 2). Each neuron ncludes four 8-bt and three 4-bt adders, one comparator and two shfters. In order to assure a proper synchronzaton of the dgtal neuron, the Chessboard dstance calculaton block operates on fallng edges and the weght updatng block on rsng edges of a clock sgnal. Therefore no multplers are necessary to calculate the dstance. Secondly the values of the learnng rate are restrcted to the set {1, ½, ¼... (½} α }. Fnally a dscrete exponental neghborhood functon s defned. Wth the proposed smplfcatons we can reformulate the weght update formula: w α 1 ( x 2 = 1 2 k w ) α+ d ( r, r *) + β o ( x k w ) o s = s * * Therefore, w s obtaned smply by multplyng (x k -w 0 ) by a power of ½, whch can be easly mplemented wth a shfter, avodng the necessty of multplers (expensve logc) n the crcut. 2

3 The Parameter Scheduler The parameter scheduler updates the learnng rate parameter α and the neghborhood wdth parameter β ncrementng them from an ntal value of 0 to a fnal value determned beforehand. Bascally, ths block conssts of two counters (one for each parameter) regsters and a combnatoral logc to determne the set of tme steps at whch the parameters wll be ncremented The Wnner -Take- All (WTA) Block The WTA blocks n the LWTA desgn has two tasks. Frst, when reset s pressed, the neuron 1 sends a 1, the neuron s poston n the map, to the neuron 2 throughout sgnal downout. Afterwards, the neuron 2 sends a 2 throughout sgnal downout to neuron 3, and so on, untl all the neurons know ther poston n the map (Fg. 4a). Fg. 1. Network archtecture of the LWTA desgn. The second task of the WTA block s to fnd the mnmum of the dstances between all the neurons and the nput pattern. Each neuron calculates the dstance, D j, on a fallng edge, afterward the neuron s j WTA block send the dstance D j and the poston j to the adjacent neurons throughout sgnals upout and downout, then the WTA block of the neurons j+1 and j-1 receve the dstance and the poston throughout the sgnals upn and downn, and compares the dstance wth the proper value (Fg. 3). Fg. 2. Neuron s archtecture 3

4 For example, f neuron 4 has a dstance of 100, neuron 5 has a dstance of 176 and neuron 3 has a dstance of 65 (Fg. 4b), then the output of the WTA blocks are 4 and 100 (100 < 176) and 3 and 65 (65 < 100) (Fg. 4c). Note that the sgnal downn s the concatenaton of j-1 and D j-1 and sgnal upn s the concatenaton of j+1 and D j+1 (Fg. 3). Fg. 3. Wnner -take- all block of LWTA desgn. Fg. 4. Behavor of the LWTA desgn (a) n reset (b) at the begnnng of the comparson (c) at the end of the comparson. 5. EXPERIMENTAL SETUP AND RESULTS The expermental setup conssts of two parts: the VHDL smulaton of a Kohonen map for a smple clusterng task and the synthess of the net on an FPGA Behavoral Smulaton The proposed hardware was descrbed n VHDL and then smulated usng the Modelsm Smulator XE II 5.7g of Mentor Graphcs [13] for two smple clusterng tasks. The frst nput data conssted of 1000 ponts randomly generated n the xy plane between lnes wth slopes 1 and ntersectons of +25 and -25. The net was traned for teratons (tme steps). The parameters α and β were both ntalzed at zero, whle the neuron weghts were ntalzed at (125, 125). The parameter α was ncremented by one at tme steps 4000, 8000 and 12000, and the parameter β at tme steps 2000, 6000, and Fg. 5 shows graphs of the nput data and the weght vectors for dfferent phases of the algorthm. It can be seen how the unts arrange themselves so as to follow the probablty dstrbuton of nput vectors. Furthermore, the weght vectors are fnally ordered accordng to ther mutual smlarty, so that neurons close to each other n the lnear array correspond to neurons wth close weghts n the nput pattern space, as must be the case of a well traned Kohonen net. 4

5 Where r s the radal dstance, θ s the polar angle and n s the constant whch determnes how tghtly the spral s wrapped. The nput data conssted of 1000 ponts generated wth a radal dstance of 30 and a constant of 2. The net was traned for teratons. The parameters α and β were both ntalzed at zero, whle the neuron weghts were ntalzed at (125, 125). The parameter α was ncremented by one at tme steps 4000, 8000 and 12000, and the parameter β at tme steps 2000, 6000, and Fg. 7 shows graphs of the nput data and the weght vectors for dfferent phases of the algorthm. Fg. 5. Input data for a smple clusterng task (small dots) and weght vectors (crcles) n the nput space: (a) Iteraton 0, (b) Iteraton 1000, (c) Iteraton 2000, (d) Iteraton 3000, (e) Iteraton 12000, and (f) Iteraton The hstogram of Fg. 6 shows for each neuron the number of tmes t became the wnner unt of the net. At the end of the run, each neuron has almost the same chance to wn, whch assures that the neurons have covered the nput space followng the probablty dstrbuton of the nput data. Fg. 7. Input data for a smple clusterng task (small dots) and weght vectors (crcles) n the nput space: (a) Iteraton 0, (b) Iteraton 1000, (c) Iteraton 2000, (d) Iteraton 3000, (e) Iteraton 12000, and (f) Iteraton The hstogram of Fg. 8 shows for each neuron the number of tmes t became the wnner unt of the net. Fg. 6. Hstogram of wnnng events: (a) Iteraton 0, (b) Iteraton 1000, (c) Iteraton 2000, (d) Iteraton 3000, (e) Iteraton 12000, (f) Iteraton Note that a common property n the SOFMs s a border aberraton effect that causes a slght contracton of the map and a hgher densty of weghts at the borders. Ths aberraton s cause by the pullng by the unts n the map thus the asymmetrc behavor of the hstogram. The second nput data s an Archmedean spral wth a polar equaton gven by: r = αθ 1 n 5

6 Desgns GWTA LWTA Slces F.F. Slces LUTs Maxmum pn delay (nseg) Although both desgns process nput data n one clock cycle, the LWTA desgn s not as effcent as the GTWA desgn n term of slcon area and speed executon. Ths dfference s due to the concepton of the WTA block. Fg. 8. Hstogram of wnnng events: (a) Iteraton 0, (b) Iteraton 1000, (c) Iteraton 2000, (d) Iteraton 3000, (e) Iteraton 12000, (f) Iteraton Synthess The descrpton of a one-dmensonal LWTA desgn composed of 10 unts wth chessboard metrc and exponental neghborng functon was syntheszed on an FPGA n order to determne the sutablty of a real mplementaton of the proposed hardware. An FPGA s an array of logc cells whose functonalty and nterconnecton can be programmed by a confguraton bt stream [14]. We used a Spartan III xc3s200ft256 from Xlnx Corp. [15] whch has a maxmum capacty of mplementng logc gates. Ths FPGA has 24 x 20 confgurable logc blocks (CLBs). The Kohonen map used 1357 slces (70% of the whole FPGA). It must be sad, however, that no attempt was made to optmze the synthess. The maxmal xc3s200 s pn-to-pn delay was 5.5 ns. Gven that the processng tme per nput vector s one clock cycle and that both, the rsng and the fallng edge of the clock sgnal are beng used, the system could process a new nput vector every 11 ns. The GWTA desgn fnds the mnmum dstance between the weghts of the neurons and the nput pattern by defnng a global block, the WTA block (Fg, 9). However, when several FPGAs are connected, each FPGA has a part of the global map thus the exstence of ths block creates a problem. Therefore, n a Kohonen s SOFM of n neurons, the WTA block of the GWTA desgn has n-1 multplexors and n-1 comparators (Fg. 9), however n the LWTA desgn, the WTA block has 2 multplexors and 2 comparators per neuron (Fg. 4) for a total of 2*n multplexors and 2*n comparators. In the GWTA desgn, the WTA global s composed of comparators and multplexors arranged n a cascade confguraton (Fg. 9). 6. A Comparson of resources of both desgns The LWTA desgn and the GWTA desgn proposed n [12] are compared by measurng the chp area (slces, flp flop slces and look up tables) and speed executon (maxmum pn delay). Fg. 9. Wnner take all block of GWTA desgn. For the sake of smplcty, a 4-neuron map s assumed. Nevertheless The Spartan x3c400 can have 30 neurons of the GWTA desgn but the LWTA desgn can handle the same amount of unts n sx Spartan x3c50 and can be easly scale by addng more FPGAs. 6

7 7. Conclusons and Future Work A dgtal scalable hardware desgn of a Kohonen map has been presented. Though very smple when compared wth the orgnal algorthm, the proposed model succeeds n retanng the man features of the network, allowng data clusterng wth topology preservaton. To acheve scalablty, the LWTA desgn dstrbutes the Wnner -Take- All block by spreadng the global nformaton locally, wthout the need of a central control unt, makng plausble the nterconnecton of several FPGAs, each one confgure wth a part of the map, to mplement a large map that otherwse would be mpossble to mplement n a sngle chp. Consequently, n the LWTA desgn, each neuron s connected only wth the neghbor neurons and all the neurons work ndependently. 12. A. E. Valenca A., J. A. Peña J. and M. Vanegas, Dgtal Hardware Desgn of a One-Dmensonal Kohonen's SOFM wth Chessboard Norm and Exponental Neghborng Functon, Internatonal Congress on Computatonal Intellgence, Montera, Colomba, Mentor Graphcs, ModelSm, n S. M. Trmberger, Feld-Programmable Gate Array Technology, Kluwer Academc Publshers, Xlnx Corp, Xlnx: The Programmable Logc Company, n The GWTA desgn has been used successfully on mage segmentaton problem and the LWTA desgn s taken nto consderaton for the mage segmentaton as well as percepton of moble robot. 8. References 1. M. Hassoun, Fundamental of artfcal neural networks, MIT Press / Bradford Book, T. Kohonen, Self-Organzaton and Assocatve Memory, Sprnger-Verlag, Berln, Y. Lao, Neural Networks n Hardware: A Survey, Department of Computer Scence, Unversty of Calforna. 4. R. Newcom, J. Lohn. Analog VLSI for Neural Networks, MIT Press / Bradford Book, T. Schönauer, A. et. Al, Dgtal Neurohardware, Techncal Unversty of Berln, Berln. 6. M. Skrbek, Neural Networks-Hardware Implementaton, Department of Computer Scence and Engneerg, FEE CTU, Prague 7. D. Macq et. Al, Analog Implementaton of a Kohonen Map wth On-Chp Learnng, IEEE Transactons on Neural Networks, Vol 4, No 3, May D. Ghosh, A. P. Shvaprasad, Possblstc Clusterng n Kohonen Networks for Vector Quantzaton, Insttute of Scence, Bangalore, Inda. 9. S. Rupng et. al. Hardware Desgn for Self Organzng Feature Maps wth Bnary Input Vectors, Lecture notes n Computer Scence, Sprnger Verlag, pp , A. Upegu, E. Sanchez, Evolvng Hardware by Dynamcally Reconfgurng Xlnx FPGAs. Internatonal Conference on Evolvable Systems, J. Tan, J. Suontausta, Scalable Neural Network Based Language Identfcaton from Wrtten Text, Noka Research Center, Tampere, Fnland. 7