Technology, Covenant University, Ota, Ogun State, Nigeria. 2

Transcription

1 Iteratioal Joural of Egieerig & Techology IJET-IJENS Vol:4 No: 5 Implemetatio of Efficiet Multilayer Perceptro ANN Neuros o Field Programmable Gate Array Chip Emmauel ADETIBA*, F.A. IBIKUNLE 2, S.A. DARAMOLA 3, A.T. OLAJIDE 4,3 Departmet of Electrical & Iformatio Egieerig, School of Egieerig ad Techology, College of Sciece ad Techology, Coveat Uiversity, Ota, Ogu State, Nigeria. 2 Departmet of Computer, Iformatio ad Telecommuicatios Egieerig, College of Sciece ad Techology, Botswaa Iteratioal Uiversity of Sciece ad Techology, Gaboroe, Botswaa. 4 Departmet of Computer Sciece, Kwara State Polytechics, Ilori, Kwara State, Nigeria. *Correspodece Author: emmauel.adetiba@coveatuiversity.edu.g Abstract-- Artificial Neural Network is widely used to lear data from systems for differet types of applicatios. The capability of differet types of Itegrated Circuit (IC) based ANN structures also depeds o the hardware backboe used for their implemetatio. I this work, Field Programmable Gate Array (FPGA) based Multilayer Perceptro Artificial Neural Network (MLP-ANN) euro is developed. Eperimets were carried out to demostrate the hardware realizatio of the artificial euro usig FPGA. Two differet activatio fuctios (i.e. ta-sigmoid ad log-sigmoid) were tested for the implemetatio of the proposed euro. Simulatio result shows that ta-sigmoid with a high ide (i.e. k >= 4) is a better choice of sigmoid activatio fuctio for the harware implemetatio of a MLP-ANN euro. Ide Term-- ANN, ASIC, DSP, FPGA, MLP. INTRODUCTION A artificial euro was ispired pricipally from the structure ad fuctios of the biological euro. It lears through a iterative process of adjustmet of its syaptic weights ad a euro becomes more kowledgeable after each iteratio of the learig process. The ultimate aim of learig by the X Iput layer w jk euro is to adjust the weights ad update the output for a ew actual output which coicides with the desired output. However, the capability of a sigle artificial euro is very limited. For istace, the Perceptro (a threshold euro) caot lear o-liearly separable fuctio []. To lear fuctios that caot be leared by a sigle euro, a itercoectio of multiple euros called Neural Network (NN) or Artificial Neural Network (ANN) must be employed. Apart from the artificial euro which is the basic processig uits i ANN, there are patters of coectios betwee the euros ad the propagatio of data called etwork topology. There are two mai types of ANN topology which are; feedforward ad recurret etwork topologies. I feed-forward etworks, the data flow from iput to output strictly i a forward directio ad there is o feedback of coectios while i recurret etworks, there are feedback coectios. A commoly used feed-forward etwork topology is Multi- Layer Perceptro (MLP). MLP caters for learig of oliear fuctios ad Figure. shows its architectural represetatio. Hidde layer w kj Output layer y X 2 y X y m The MLP etworks are typically traied with the traiig algorithm called the Backpropagatio (BP) algorithm which is a supervised learig method that maps the process iputs to Fig... Multi-Layer Perceptro (MLP) topology [2]. the desired outputs by miimizig the errors betwee the desired outputs ad the calculated outputs [2]. BP is a applicatio of the gradiet method or other umerical

2 Accumulator Iteratioal Joural of Egieerig & Techology IJET-IJENS Vol:4 No: 52 optimizatio methods to a ANN with feed-forward architecture i order to miimize the error fuctio. The algorithm is the most popular method for performig supervised learig [3]. There are differet variats of BP algorithm which iclude; cojugate gradiet, Leveberg Marquardt (LM), gradiet descet, quasi-newto ad etc. I order to fully beefit from the massive parallelism that is iheret i ANN, it is essetial to implemet it i hardware. ANNs ca be implemeted i hardware usig either aalog or digital electroics [4]. Aalog electroics implemetatio of ANN is always very efficiet with respect to space ad processig speed, however, these are achieved by tradig off the accuracy of the computatio elemets of the etwork. Digital electroics implemetatio of ANN ca be classified ito three groups; i.) DSP-based implemetatio ii.) ASIC based implemetatio ad iii.) FPGA-based implemetatio [5]. DSP-based implemetatios are sequetial ad do ot preserve the parallel architecture of ANNs ad ASIC implemetatios do ot support recofigurability after deploymet. However, FPGA based implemetatio is very suitable for hardware realizatio of ANN. It ot oly preserves the parallel architecture of eural etworks, but also, it offers fleibility i recofiguratio, modularity ad dyamic adaptatio for eural computatio elemets. FPGA which is a acroym for Field Programmable Gate Array is described by Stephe ad Joatha [6] as a itegrated circuit cotaiig gate matri which ca be programmed by the user i the field without usig epesive equipmet. The maufacturers of FPGA iclude; Xili, Altera, Actel, Lattice, QuickLogic ad Atmel. Majority of FPGAs are based o SRAM (Static RAM) ad they store logic cells cofiguratio data i the static memory orgaized as a array of latches. This class of FPGA must be programmed upo start because SRAM is volatile. Eamples of SRAM based FPGAs are Virte ad Sparta families (from Xili) ad Cycloe ad Strati (from Altera). SRAM based Altera Cycloe FPGA is the adopted techology for hardware implemetatio of the artificial euro i this work. 2. MATERIALS AND METHODS Geerally, ANN implemetatio usually starts with the euro because it is the basic uit of ay eural etwork. Meawhile, the hardware implemetatio of a euro has two major parts. The first part is the basic fuctioal uits that realise the ier product ad the secod part is the implemetatio of the activatio fuctio. The architecture for the hardware implemetatio of a artificial euro is show i Figure 2. multiplier adder Weight Register X + Activatio Fuctio Output Iput Register Multiply-Accumulate (MAC) Uit Fig. 2.. Hardware architecture of a artificial euro 2. Basic Fuctioal Uits The basic fuctioal uits of a hardware euro compute the ier product for the euro ad it is made up of the etities show i Figure 2.. The iput register was implemeted with a shift register for iterative eterig of the iput values ito the euro. The weights register was realized usig a shift register ad it serves the purpose of eterig the correspodig weight of the curret iput value ito the euro. The multiply accumulate (MAC) uit of the euro was realized with combiatioal circuits for full adder ad multiplier. Appropriate umber of bits were used for the iput ad output sigals i the code so as to cater for the epected data rage. These uits were implemeted with Very High- Level Descriptio Laguage (VHDL) i Quartus II 9. Web Editio ad the target was a Altera s DE2 board. This board cotais a Altera Cycloe II 2C35 FPGA with a wide rage of eteral memory, embedded multiplier, iterfaces, I/O protocols ad parameterizable IP cores [7]. The VHDL codes for the basic fuctioal uits are show i Figure 3..

3 Iteratioal Joural of Egieerig & Techology IJET-IJENS Vol:4 No: 53 library ieee; use ieee.std_logic_64.all; use ieee.std_logic_arith.all; use ieee.std_logic_siged.all; -- declare the etity etity mac is geeric (r : iteger := 3; b : iteger := 32); port( p : i siged (b- dowto ); w : i siged(b- dowto ); clk : std_logic; --w_out : out siged(b- dowto ); a : out siged (2* b- dowto )); ed mac; architecture Behavioral of mac is type weights is array ( to r) of siged (b- dowto ); type iputs is array( to r) of siged (b- dowto ); begi process(clk,w,p) variable weight : weights; variable iput : iputs; variable prod, acc : siged (2 * b- dowto ); begi if (clk'evet ad clk='') the weight := w & weight( to r-); -- weights shift register iput := p & iput( to r-); ed if; --iput():= p; iput(2) := p2; iput(3) := p3; acc :=(Others =>''); --output weights --multiply-accumulate(mac) L: for j i to r loop prod := iput(j) * weight(j); acc := acc + prod; ed loop L; a <= acc; --liear output of the euro ed process; ed Behavioral; Fig. 3.. VHDL codes for the basic fuctioal uits of a euro 2.2 Neuro Activatio Fuctio The commoly used activatio fuctios i artificial euros are liear, sigmoid ad radial fuctios. The liear activatio has the form;. (.) The sigmoid activatio fuctios are S shaped ad the oes that are mostly used are the logistic ad the hyperbolic taget (equatios (2.) ad (3.) respectively); (2.). (3.) There are differet types of radial activatio fuctios, but the oe that is usually adopted uses Gaussia fuctio; However, for the hardware implemetatio of the euro i this work, the taylor series of the sigmoid activatio fuctios (i.e. log-sigmoid ad ta-sigmoid) which are the most commoly used activatio fuctios i ANNs were aalyzed sice they caot be implemeted directly i hardware (4.) because they both cotai epoetial fuctios. With proper aalysis, we were able to make a iformed decisio o the appropriate choice of sigmoid activatio fuctio for hardware euro implemetatio. The aalysis is reported i the subsequet sub-sectios.

4 Iteratioal Joural of Egieerig & Techology IJET-IJENS Vol:4 No: Taylor Series Approimatios for Log-Sigmoid ad Ta-Sigmoid Restatig equatio (2.), we have; f( ) e, (5.) The power series of e - is; e ( ),! R. (6.) k For let z ( ) (7.)! Puttig (7.) ito (5.), Taylor s series is obtaied for (5.) as: f( z) z. (8.) Equatio (8.) is the Taylor s series represetatio of logsigmoid activatio fuctio. Restatig equatio (3.) for ta-sigmoid activatio fuctio gives; f( ) e e e e The power series for e is; e For!, (9.), R. (.) k, (.) becomes; y. (.)! Substitutig equatios (7.) ad (.) ito (9.) produces; y z f ( y, z). (2.) y z Therefore, equatio (2.) is the Taylor series for ta-sigmoid activatio fuctio. The pseudocodes from these aalysis are show i figures 4. ad 5.. LogSigmoid(X) /* Iitialize variables*/ y = : Prod = : LogSig = Read k For = To k Prod = ((-) ^ ) * ((X ^ )/Fact()) y = y + Prod EdFor /* Compute the Taylor series LogSig */ LogSig = /( + y) DISPLAY LogSig Ed Fact() /* Iitialize variable(s) */ Factorial = For i = To 2 Factorial = Factorial * (i-) EdFor Retur Factorial Fig. 4.. Pseudocode for log-sigmoid Taylor series

5 Iteratioal Joural of Egieerig & Techology IJET-IJENS Vol:4 No: 55 TaSigmoid(X) /* Iitialize variables*/ y = : z = : Prod = : Prod2 = : TaSig = Read k For = To k Prod = ((-) ^ ) * ((X ^ )/ LogSigmoid.Fact()) y = y + Prod Prod2 = (X ^ )/LogSigmoid.Fact() z = z + Prod2 EdFor /* Compute the Taylor series Ta-Sig */ TaSig = (y-z)/(y+z) DISPLAY TaSig Ed Figure 5.: Pseudocode for ta-sigmoid Taylor series 3. Eperimetal Results ad Discussio The VHDL codes show i Figure 3. were simulated fuctioally i Quartus II 9. Web Editio eviromet o Altera s DE2 board that cotais Altera Cycloe II 2C35 FPGA. The Register Trasfer Logic (RTL) of the VHDL code for the basic fuctioal uits with 3 iputs ad 3 weights is show i Figure 6. ad the simulatio output is show i Fig. 7.. Fig. 6.. RTL of the basic fuctioal uits of the artificial euro

6 Computed Log-sigmoid ad Taylor's Series Iteratioal Joural of Egieerig & Techology IJET-IJENS Vol:4 No: 56 Fig. 7.. Simulatio output of the basic fuctioal uits of the artificial euro The simulatio output i Figure 7. which shows the implemetatio result of multiply ad accumulate (MAC) operatio o the cotet of the iput ad weight registers illustrates a perfect output for the iputs. This is a attestatio to the correctess of our VHDL codes (Figure 3.) for the basic fuctioal uits part of the artificial euro implemeted i this work. Also, eperimets were carried out so as to ascertai the appropriate sigmoid activatio fuctio betwee log-sigmoid ad ta-siged for the hardware realizatio of artificial euros. The pseudocode for log-sigmoid ad ta-sigmoid ad their respective Taylor series approimatios (Figures 4. ad 5.) were implemeted i MATLAB R28a. Eperimetal trials for k=, 2 ad 4 ad for values of X ragig from -2 to +2 (i Equatios 7. ad.) for the two activatio fuctios were performed. The results obtaied from these eperimets were graphically plotted i order to aid our comparative aalysis. These plots are show i Figures 8., 8., 8.2, 9., 9. ad log-sigmoid Taylor series Values of Fig. 8.. Log-sigmoid ad it s Taylor series approimatio for k

7 Computed values of Log-sigmoid ad Taylor's series Computed Log-sigmoid ad Taylor's series Iteratioal Joural of Egieerig & Techology IJET-IJENS Vol:4 No: 57.9 log-sigmoid Taylor's series Values of Fig. 8.. Log-sigmoid ad it s Taylor series approimatio for k 2.9 log-sigmoid Taylor's series Values of Fig Log-sigmoid ad it s Taylor series approimatio for k 4

8 Computed Ta-sigmoid ad Taylor Series Computed Ta-sigmoid ad Taylor Series Iteratioal Joural of Egieerig & Techology IJET-IJENS Vol:4 No: ta-sigmoid Taylor series Values of Fig. 9.. Ta-sigmoid ad it s Taylor series approimatio for k.6 ta-sigmoid Taylor series Values of Fig. 9.. Ta-sigmoid ad it s Taylor series approimatio for k 2

9 Computed Ta-sigmoid ad Taylor Series Iteratioal Joural of Egieerig & Techology IJET-IJENS Vol:4 No: 59.6 ta-sigmoid Taylor series Values of Fig Ta-sigmoid ad it s Taylor series approimatio for k 4 Figures 8., 8. ad 8.2 show that for the various values of k (i.e., 2 ad 4) for, there are little covergeces betwee the actual log-sigmoid fuctio ad its Taylor series approimatios. Meawhile, k = 4 was aticipated to give a good covergece but the plot i Figure 8. shows that for higher positive values of X, the deviatio betwee the actual fuctio ad the Taylors series approimatio was gettig more proouced. However, from Figures 9., 9. ad 9.2, the covergeces betwee tasigmoid fuctio ad its Taylor series approimatios improve as the values of k rage from to 2 to 4 for. Ifact, as show i Figure 9.2, at k = 4, there is a perfect covergece betwee the actual ta-sigmoid fuctio ad its Taylor series approimatio for the rage. 4. CONCLUSION The result i Figure 7. shows that our VHDL code i this work is very accurate ad ca be reliably loaded ito a FPGA (i.e. Altera s DE2 board that cotais Altera Cycloe II 2C35 FPGA) to realize the basic fuctioal uits of ay artificial euro. Also, the plot i Figure 9.2 shows that ta-sigmoid with a high ide (i.e. k >= 4) is a better choice of sigmoid activatio fuctio for the hardware implemetatio of a artificial euro. A Multi-Layer Perceptro (MLP) eural etwork ca therefore be implemeted o FPGA by aggregatig several of the hardware euros i this work based o the required MLP cofiguratio for a give area of applicatio. Our et directio for this work is to adapt a FPGA-based MLP eural etwork to realize the classifier submodule of a geomics-based diagostic system for lug cacer. However, FPGA-based MLP eural etwork hardware ca be applied i other areas such as commuicatios, cotrol, et-geeratio sequecig, biometrics ad biomedical devices. REFERENCES [] Tredeick, N., (996). Microprocessor-based computers, IEEE Computer: 5 years of computig, [2] Huag, Y. (29). Advaces i Artificial Neural Networks: Methodological Developmet ad Applicatio, Algorithms, 2: [3] Werbos, P. J., (994). The Roots of Backpropagatio: From ordered derivatives to Neural Networks ad Political Forecastig, Joh Wiley ad Sos, New York. [4] Fiesler E., ad Beale, R., (997). Hadbook of Neural Computatio, E.2:-3, Istitute of Physics Publishig ad Oford Uiversity Publishig, New York. [5] Pedro F., Pedro, R., Aa A., ad Ferado M. D., (27). A high bit resolutio FPGA implemetatio of a FNN with a ew algorithm for the activatio fuctio, Neurocomputig, 7:7 77. [6] Stephe B., ad Joatha R., (2). Architecture of FPGAs ad CPLDs: A Tutorial, Departmet of Electrical ad Computer Egieerig, Uiversity of Toroto. [7] Altera (22). DE2 Developmet ad Educatio Board User Maual, Versio.6, Altera Corporatio, 4-6.