Dynamically Reconfigurable Neuron Architecture for the Implementation of Self- Organizing Learning Array

Transcription

1 Dynamically Reconfigurable Neuron Architecture for the Implementation of Self- Organizing Learning Array Januz A. Starzyk,Yongtao Guo, and Zhineng Zhu School of Electrical Engineering & Computer Science Ohio Univerity, Athen, OH {tarzyk, Abtract: In thi paper, we decribe a new dynamically reconfigurable neuron hardware architecture baed on the modified Xilinx Picoblaze microcontroller and elforganizing learning array (SOLAR) algorithm reported earlier. Thi architecture i aiming at uing hundred of traditional reconfigurable field programmable gate array (FPGA) to build the SOLAR learning machine. SOLAR ha many advantage over traditional neural network hardware implementation. Neuron are optimized for area and peed, and the whole ytem i dynamically elfreconfigurable during the runtime. The ytem architecture i expandable to a large multiple-chip ytem. 1. Introduction The immene computing power of the brain i believed to be the reult of the parallel and ditributed computing performed by approximately neuron, each with an average of connection. To prototype network that mimic the biological neuron uing hardware, FPGA technology can be ued with it flexible and programmable interconnect and computing reource. During the lat decade, number of approache have been attempted to apply reconfigurable hardware to the neural network [1,2]. However, thee implementation are focued on denely interconnected, weight dominated architecture which are not eaily expandable to multiple FPGA with identical cell architecture. Due to the additional hardware cot of adjuting interconnection weight of traditional neural network, often their connection and functionalitie are fixed and predefined by off-line imulation. The gradient method i difficult to implement and therefore, it i hard to expand the network to a large-cale ytem. One exception here are FPGA baed array of identical cell to implement evolutionary computing and genetic algorithm [3,4,5]. New FPGA tructure, adopting pule denity [6], pule poition and time difference imultaneou perturbation are preented to avoid the hardware gradient evaluation in NN. Alo, recently, cellular neural network ued in image proceing [7] and their hardware implementation have attracted a lot of attention. In thi paper, we preent a dynamically reconfigurable (i.e. during runtime), data-baed and expandable neuron architecture to implement a elf-organizing learning array (SOLAR). The neuron functionality i repreented in the form of flexible connection and organization of an array of evolvable ignal proceing block. Each neuron can implement variou arithmetic and logic function. The neuron can elf-reconfigure and ue local interconnect for maximum performance. The ret of the paper i organized a follow. Section 2 talk about our perviou work on SOLAR. Section 3 deal with the architecture of the dynamically reconfigurable neuron. Section 4 report the imulation and experimental reult. Finally, Section 5 conclude thi paper. 2. Previou Work Self-organization i important in artificial neural network (ANN) and machine learning. Previouly, a elf-organizing learning algorithm that combine neural network and information theory wa preented in [8]. Thi entropy-baed neural network learning algorithm wa imulated on tandard benchmark and proved to be advantageou over many exiting neural network and machine learning algorithm in a wide range of technical application [9], in particular while dealing with noiy or incomplete data. Baed on thi algorithm, we propoed SOLAR ytem which i different from claical ANN in the way it i organized and how it learn. SOLAR elforganize it hardware reource to perform claification and recognition tak. Similar to the tructure of cellular neural network, SOLAR ha a fixed array of elemental proceing unit acting a ingle neuron, and programmable interconnection between them. Initially, SOLAR neuron are randomly connected to previouly generated neuron. They learn adaptively uing both primary input and input from other neuron. Controlled by ignal from other neuron, they perform baic tranformation of their input ignal. A neuron parameter and connection are re-configured a a reult of training, and effectively, SOLAR tructure elf-organize etablihing it final wiring. Earlier work mainly focued on SOLAR algorithm imulation, and it hardware implementation wa limited to a ingle FPGA chip with the cooperation from oftware [10]. Thi hardware implementation explored the adaptability of SOLAR to FPGA implementation,

2 although, the cellular neuron architecture wa not fully explored ince a hared block memory wa ued for all neuron. In SOLAR imulation, we adopted feed forward network tructure for it tability and fat learning. SOLAR architecuture i divided into three main layer a hown in Fig. 1 input, proceing and output layer. The interconnection are randomly initialized at the beginning of the network training. The baic building block of the architecture are the mall neuron, trained uing the entropy-baed algorithm. The implified learning algorithm i introduced a follow: Let S {( ν, c) } = be a training et of N vector, where n v R i a feature vector and c Ζ i it cla label from an index et Ζ. A claifier i a mapping C : R n Ζ, which aign a cla label in Ζ to each n vector in R. A training pair ( ν, c) S i miclaified if C( ν ) c. The performance meaure of the claifier i the probability of error, i.e. the fraction of the training et that it miclaifie. Our goal i to minimize thi miclaification probability, and it i achieved through a imple threhold earching baed on a maximum information index calculated from etimated ubpace probability and local cla probabilitie. The et of training data i earched equentially cla by cla to etablih the optimum threhold, which bet eparate the ignal of variou training clae. A partition quality i meaured by the entropy baed information index defined by: E I = 1 E where and Fig. 1 Baic SOLAR tructure max = Pclog( Pc) + E P log( P) c E max = P c log( P c ), c where P, c P, P c repreent the probabilitie of each cla, attribute probability, and joint probability, repectively. The ummation i performed with repect to all clae and ubpace. The information index hould be maximized to provide an optimum eparation of the input training data. Entropy baed information index i alo ued to evolve the neural interconnect network tructure and to elect neuron tranformation function. The neuron learning i realized by maximizing the information index. If a neuron reache pecified information index threhold, then it become a voting neuron. Voting neuron output are weighted together to olve a given problem. 3. Neuron Architecture The hardware architecture preented in thi paper i baed on the identical neuron module. The contant Coded Programmable State Machine (KCPSM)[11], an 8- bit micro-controller developed by Xilinx Corp., ha been modified and embedded into the neuron module. A block level of ingle neuron architecture i hown on Fig.2. Each neuron module contain circuit to be reprogrammed dynamically and to execute it program without affecting other neuron operation. An aembler i ued to implement neuron function and it operation i controlled by locally tored parameter value. addre intruction<15:0> addr_bu<7:0> data_bu<15:0> Dual clk port enable memory 30 input x R intruction <15:0> addre in_port <7:0> out_port <7:0> clk port_id<7:0> interrupt read_trobe reet write_trobe Fig.2 Single neuron chematic After tranlation from the aembler to binary code, thee binary code are written into the dual port memory via calling the Peripheral Component Interconnect (PCI) function. The dynamical programming i implemented by a dual-port 256x16-bit memory. The KCPSM read the current program on one port while the other port can be ued to tore the new program. The two port of the dualport Random Acce Memory (RAM) operate independently, and the operation i via hared programming bu among all neuron. Therefore, the elfreconfiguration proce can be performed affecting only the current neuron. The configuration time and content can be controlled by the oftware outide the chip or by the R Neural Controller (ue KCPSM)

3 configuration data that are located in the ytem ditributed memory cell. The neuron input are obtained either from the primary input or other neuron output via a 30 to 1 multiplexer (MUX). The election ignal are decided by the content of the programming dual-port memory via the execution of the programming command for the particular neuron. The ingle neuron architecture i expanded to multiple neuron in a ingle FPGA chip. In thi work, we ue a Nallatech board with Xilinx Virtex XCV800 FPGA [12]. It can contain an array of up to 28 neuron organized a hown in Fig.3 (The 480 Virtex XCV1000 FPGA we are uing to build 3D learning machine will contain 64 neuron on each chip). hardware implementation. If the reource ued to implement a ingle ub-network can be reued to implement multiple ub-network equentially, then many reource will be aved. Thi of coure will be accomplihed at the cot of additional proceing time for equential operation of neural network block; however, thi i not a problem, ince a neural network repone can be obtained in a fraction of time required by a practical application. For intance, in NTSC video tandard, one full frame i diplayed about every 1/30 of a econd; in PAL video tandard, the video image are diplayed 25 frame per econd, leaving more than 40 m for recognition of each frame. A practical time limit for recognition may be even larger. Since different network could have different connection, a new dynamically reconfigurable routing and addreing cheme i propoed. Thi routing and addreing cheme i uitable for general artificial neuron network, whether or not their connection are local or global. Fig. 4 give an example where 4 ub-network are replaced by a ingle ub-network baed on the propoed routing and addreing cheme. Fig. 3 Array neuron organization Thee neuron are fully interconnected via the connection bu and a 30 to 1 MUX thu forming a local cluter of neuron. The full connection implementation mimic the dene connection cheme in the neighborhood neuron. A 3D expanion of thee chip repreent the pare connection between remote neuron. Thee interchip connection cale linearly with the number of neuron added. The neuron connection are decided by each neuron baed on it learning reult. The programming content can be dynamically updated via the configuration bu or et locally by a neuron. The configuration bu ued to configure every ingle neuron i divided into 16-bit data, 8-bit addre and 5-bit neuron election bue. To demontrate functionality of 28 neuron cluter, we integrate a PCI interface controller to tranfer the data/configuration via the PCI bu to neuron. 4. Routing Scheme A traditional neural network ytem may conit of many ub-network, which take up a lot of reource in the Fig. 4 One network replacing multiple network Fig. 5 give an example of a ingle ub-network with more detailed interconnect tructure. Fig. 5 Structure of a ingle network It conit of an array of 4*4 neuron. The column order of the neuron in thi ingle network i not natural: the original order of 1, 2, 3 and 4 i replaced by 1, 4, 2 and 3. Thi kind of routing cheme take advantage of the

4 reconfigurable reource in FPGA that can eaily be configured into a long hift regiter Scheme of the routing channel The neural network tructure hown in Fig. 5 ha two baic element: the routing channel and the neuron cell. The routing channel i implemented by RAM in Configurable Logic Block (CLB) of FPGA chip. Thee torage RAM cell are configured into a cloed loop hift regiter, hifting data among neuron. Thi hift regiter create a routing channel carrying neuron input data and their operation reult. Each ub-network ha it unique connection and information about thee connection i tored in individual neuron. The input data for a pecific neuron in thi ub-network will appear on neuron input port after a fixed number of clock cycle. The tak of addreing thee data i equivalent to counting the clock cycle and comparing them againt the pre-tored addree (pre-tored number of clock cycle). When the clock cycle matche one of it pre-tored value, thi neuron will read data from the routing channel. So, each neuron require ome memory to tore the addree of neuron which connect to thi neuron. For example, conider an original network ytem that conit of 2 ub-network with the ize of 4 by 5 neuron each (Fig. 6). Aume, that thi network ytem i implemented by a ingle ub-network. Conider a neuron in row 3 and column 4 for analyi. Aume that thi neuron ha 4 input in the firt ub-network of the original network ytem, and 5 input in the econd ub-network. When thee 2 ub-network are replaced by a ingle ubnetwork, then thi neuron will have two group of addree: the firt group of 4 addree toring the number of clock cycle correponding to the 4 input in the firt ub-network, and the other group of 5 addree toring the number of clock cycle correponding to the 5 input in the econd network. Output data of each neuron i handled in a imilar way. reconfigurability and data tranfer between ub-network can be explained a follow. The firt column of the routing channel i connected to the ytem input. So, the ytem data i applied directly to the routing channel. For the network in Fig. 5, after the firt row finihe proceing it input data, the routing channel correponding to the firt column of neuron tore the reult of thoe neuron operation. Thee reult are hifted into the routing channel and mixed with the input ignal value which are already in the channel, are connected to the neuron in the econd column, then into the third and forth column. After the neuron in the fourth column proceed their input data, the routing channel connecting to the neuron in forth column tore the output of the firt ub-network. By now, the firt ub-network in the original network ytem ha finihed it operation, but at thi time, it i not prepared to receive the next et of input data becaue thi ingle network will change it connection to implement the original econd ub-network. So, the output of the firt ub-network, which are tored in the fourth column of the routing channel, will be hifted back to the routing channel in the firt column, which now repreent the firt column of the econd ub-network of the original network ytem. The data i proceed in the ame way a wa decribed above. If the original network ytem ha four ubnetwork, then thoe data will be iterated for four time before thi ingle ub-network receive next et of input data (The feedback network implementation in thi iterative cheme will iterate more than 4 iteration). Changing connection for different network can be implemented by comparing clock cycle value againt different group of pre-tored addree. Structure of routing channel In the SOLAR hardware implementation, the neuron function i implemented by the PicoBlaze [Xilinx]. The PicoBlaze i a micro-core with 8-bit parallel input and output. A diagram of the routing channel connected to the input port of a PicoBlaze i given in Fig. 7. Fig. 6 Hardware reue to implement 2 ub-network. In order to tranfer data from one ub-network to the other one, the ame routing channel i ued. Thi dynamic Fig.7 Routing channel with a micro controller

5 The RAM in each CLB i configured into a hift regiter, with length configured from 1 to 16 - here we chooe 8. There are eight uch hift regiter connecting to the correponding bu line of the PicoBlaze input bu to provide the input data in parallel. In the routing cheme preented in thi paper, neuron are directly connected to the local routing channel, and don t need direct phyical connection among the neuron, no matter local or global. The connection among the neuron are et up by tored addree. For the network with random connection, thoe tored addree can be generated on-line randomly. For other type of learning network with fixed connection, the tored addree are baed on pecified network interconnect tructure. The bet advantage of thi kind of routing and addreing cheme i that it make on-line learning and reconfiguration eay to implement. For the on-line network reconfiguration, we don't need to change the phyical connection between network component. What we need to do i only to change pre-tored communication channel addree according to the learning rule. At preent, the implemented neural network conit of 28 fully interconnected neuron on a ingle chip. In a neural network ytem, compoed of many chip, ome neuron in one chip need to be connected to the neuron in other chip. In order to tranfer data from one chip to the other, the ame concept of routing channel a hown in Fig. 5 i ued. Thi routing channel i laid around perimeter of each chip. Channel in different chip can be connected erially to form a long chain of hift regiter. Neuron in one chip can be connected to thi routing channel and output their operation reult. Thee reult will be hifted to the channel, and after hifting they will be picked up by neuron in other chip. Thi way, remote connection between neuron in different chip are et up. If the neuron are connected to the neuron in the ubnetwork of ame neighboring row or column, then thoe neuron can be directly connected, without paing through the routing channel (a illutrated in Fig. 3). One advantage of thi cheme i that, with the introduction of the routing channel, the neuron can be connected in 3 dimenion. The neuron output their reult to the routing channel, and then thoe reult can be hifted to the routing channel in different layer of the neural ytem. 5. Simulation and Reult In thi ection, reult from prototyping a imple SOLAR architecture onto a ingle VIRTEX FPGA chip are dicued. The neuron elf-organizing learning proce can be configured during chip programming or dynamically updated while running. To demontrate thi proce, a imple example i given to illutrate the implementation tep of the SOLAR algorithm. The implementation of the whole SOLAR array i organized in a imilar way. In thi imulation example, we have ix out of twenty-eight neuron configured a hown in Fig Fig.8 Experimental network and reult The initial connection are hown in olid line. Every neuron imply add two input together, for intance, neuron 1 add input 1 and 2 to it content; neuron 3 add input 2 and the output of neuron 1; neuron 5 add the output of neuron 2 and 3, etc. Later, we dynamically reconfigure the connection of neuron 3 and neuron 5. So neuron 3 ha input from the output of neuron 1 and 2, and neuron 5 ha input from the output of neuron 3 and 4 hown by the dotted line. The reult read out from the chip via PCI bu are hown in the Matlab conole. In the inerted Matlab command conole in Fig.8, initial value how primary input value (6 and 2) and neuron output for two row of neuron, while updated value how input and neuron output after dynamical reconfiguration tep. We developed the Matlab DLL to implement the I/O function including read and write. We can ee the reult from the real experiment correponding to VHDL imulation reult a hown in Fig. 9. In thi plot, Enable_bu repreenting the neuron election ignal, elect a particular neuron to be configured. Once the configuration proce for all neuron i over, the output from neuron are table and ready to be read out. Thi way we can update any neuron configuration information without affecting the other neuron. In thi example, we only update neuron 3 and 4 connection repreented by Enable_bu content 4 and 5. The imulation reult after updating correpond to the real experiment read back from hardware a the updated value in Fig.8. neuron output initial updated Fig. 9 VHDL imulation for partial configuration

6 The deign ha been decribed in VHDL and yntheized uing the Xilinx XST and the Xilinx Alliance tool for place and route. The implementation reult are ummed up in Fig. 10. Fig. 10 Implementation report The implementation reult how that thi neural network architecture realize a maximum parallel intruction throughput of 23.16x28 MIP with 28 fully connected neuron. The neuron number i limited to 28 ince there are only 28 BRAM module on the chip although we till have ome other reource unued. If we further lower the neuron program memory ize, we can put more neuron on a ingle chip to overcome the BRAM bottleneck. Due to the parallel proceing in hardware and the ditributed network memory, the peed improvement uing FPGA implementation i ignificant comparing to the oftware imulation. The mapping reult i hown in Fig. 11. We can ee how the neuron are ditributed inide the chip after the mapping proce. Every ingle neuron occupie a compact and concentrated logic area. The compactne depend on the tructure-oriented hardware deign, for intance, we adopt LUT and dedicated multiplexer module in addition to the optimized Picoblaze tructure to build every ingle neuron. 6. 3D SOLAR The propoed neuron tructure can be expanded to hundred of FPGA chip to form a large 3D architecture. Thi architecture i baed on many printed circuit board (PCB) with 4 interconnected VIRTEX XCV1000 FPGA chip on every ingle PCB board. Four PCB board are interconnected vertically through the connector on upper and lower ide of each board to build up the rack SOLAR architecture a hown in the left ide of Fig. 12. The rack SOLAR will be expanded to a rack array a hown in the right ide of Fig.12 through the flat connector on both the left and the right ide of the board. In thi rack array, one PCB board in a pecific rack will be choen to be the firt board from which all ignal will be broadcat to other board in the whole rack array - the identical configuration bit are ent to every PCB board and FPGA chip are programmed through JTAG port in a daiy chain or through Slave Serial mode in parallel. Thee broadcat ignal contain the configuration ignal and data ignal. Thee ignal are buffered to aure ignal integrity. Thee chip can communicate with each other through the data bu via variou connector. All of the chip have identical neuron ditribution a hown in Fig. 3. Every neuron can be dynamically updated without affecting other neuron. Finally, 3D SOLAR ytem, containing 5x5 SOLAR rack with cloe to 400 million gate, a hown in Fig. 12 will be built The 3D SOLAR learning machine will have a tremendou elf-organizing learning ability baed on numerou computing cell (neuron) working in parallel to proce the incoming data. Fig.12 RACK and CUBE SOLAR Fig. 11 Mapping reult

7 7. Concluion Thi paper ha decribed the architecture and chip implementation of an array of neuron aimed at implementation of the SOLAR architecture. The elforganizing learning i baed on a new machine-learning algorithm that combine knowledge from NN and information theory. SOLAR repreent a new idea in hardware deign of neural network. It i modular and expandable ytem. It alo define a new breed of dynamically reconfigurable architecture that can dynamically reconfigure themelve baed on information included in the input data. Thi preented architecture i a novel dynamically reconfigurable (via dual-port memory) neural network implementation that ued imple generalpurpoe proceor (KCPSM) architecture. Firtly, it ha a regular expandable parallel architecture. Therefore, it peed and learning abilitie can be greatly improved comparing to the oftware imulation. Secondly, it ha data-driven elf-organizing learning tructure baed on a new elf-organizing learning algorithm. Furthermore, deign flexibility i attained by exploiting the feature of elf-reconfigurable neuron unit. Finally, hardware reconfigurability i achieved in thi elf-organizing learning array by involving reconfigurable routing module. According to the implementation reult, thi neuron architecture realize a maximum parallel intruction throughput of 648 MIP with 28 fully connected neuron. Sytem performance increae a more neuron are connected. The PicoBlaze for Virtex-II Serie FPGA reache performance level of up to 55 MIPS. With up to 336 PicoBlaze node on the chip SOLAR will reach the performance of 18.5 GIPS on a ingle chip. So it parallel proceing ability can be further improved. With the popular PowerPC on VIRTEX II Pro FPGA ued a the main CPU, the PicoBlaze neuron can be ued a lave peripheral to further improve the throughput for ytem on a chip implementation of neural network. Hence it can be of a practical ue for the embedded hardware application in ignal proceing, wirele communication, multimedia ytem, data network, and o forth. In our deign, the number of neuron/chip i limited by the number of BRAM, o the total logic utilization i comparatively low. The remaining LUT RAM will be ued for the communication between neuron on other chip and to reue the ame Picoblaze for everal neuron, to fully utilize the hardware reource and accommodate more neuron in a ingle FPGA chip. In thi way, our deign eem device-dependent, ince we need to carefully arrange additional reource for different FPGA. Each device type ha to be individually optimized. In caling our deign to 3D ytem, thi will be a neceary work to be computed in order to accommodate more neuron in a ingle elected FPGA chip. Reference [1] Mira, M., 1997, Parallel Environment for Implementing Neural Network. Neural Computing Survey, Vol.1, 48-60,1997. [2] Glener, M. and Pochmuller, W., 1994, An Overview of Neural Network in VLSI, Chapman & Hall, London, [3] G. Tempeti, D. Mange, A. Stauffer, C. Teucher The BioWall: an Electronic Tiue for Prototyping Bio-Inpired Sytem, Proceeding, NASA/DoD Conf. on Evolvable Hardware, Lo Alamito, Calif., pp , [4] L. Sekanina, Toward Evolvable IP Core for FPGA, In: Proc. NASA/DoD Conf. on Evolvable Hardware, Lo Alamito, US, ICSP, 2003, pp , [5] Hugo de Gari, Michael Korkin, "The CAM-Brain Machine for Real Time Robot Control", J. Neurocomputing, Elevier, Vol. 42, Iue 1-4, Feb., [6] Maeda, Y., Tada, T., FPGA implementation of a pule denity neural network with learning ability uing imultaneou perturbation, IEEE Tranaction on Neural Network, Vol. 14 Iue3, May [7] T.Roka et. al., "The Computational Infratructure of Analogic CNN Computing Part I.: the CNN-UM Prototyping Sytem, IEEE Tran. Circuit and Sytem I, vol. 46, pp , [8] J. A. Starzyk and T-H.Liu, Deign of a elf-organizing learning array ytem, IEEE Int. Sympoium on Circuit and Sytem (ISCAS), Bangkok, Thailand, May [9] J. A. Starzyk and Z. Zhu, Software imulation of a elforganizing learning array ytem, The 6th IASTED Int. Conf. Artificial Intelligence & Soft Comp (ASC 2002), Canada. [10] J. A. Starzyk, and Y. Guo, Dynamically Self- Reconfigurable Machine Learning Structure for FPGA Implementation Proc. Int. Conf. on Engineering of Reconfigurable Sytem and Algorithm (ERSA) La Vega, Nevada, USA, June,2003. [11] K. Chapman, 8-Bit Microcontroller for Virtex Device. Xilinx XAPP213, Online Oct [12] Nallatech Ltd, Ballynuey 2 VIRTEX PCI card uer guide,