ANN WHICH COVERS MLP AND RBF

Transcription

1 ANN WHICH COVERS MLP AND RBF Josef Boští, Jaromír Kual Faculty of Nuclear Scieces ad Physical Egieerig, CTU i Prague Departmet of Software Egieerig Abstract Two basic types of artificial eural etwors Multi Layer Perceptro (MLP) ad Radial Basis Fuctio etwor (RBF) are frequetly discussed. The MLP is built of oe type of euro decomposable ito liear ad sigmoid part. The secod type (RBF) cosists of radial ad liear euros. The ew Multi layer Radial Basis Fuctio (M-RBF) cosists of two types of euros: liear ad exteded sigmoid oes. Four layer M-RBF etwor should approximate ay RBF etwor while five layer M-RBF etwor should replace ay MLP etwor wi ree layers. The ew M-RBF etwor a geeralize abilities of bo basic types of ANN. The etwor ad its learig are demostrated o umerical example. The results are compared wi RBF ad MLP. Keywords: ANN, MLP, RBF, sigmoid fuctio, decompositio. Itroductio. Basic characteristics of MLP Geeral Multi Layer Perceptro (MLP) [] [] etwor cosists of sigle iput layer, at least oe hidde layer ad sigle output layer. The sigal processig i every hidde ad output euro is described by formula f w x () y iput value ad its weight, where Ν is umber of euro iputs, x, w R are y a; b is euro output, x ad f : R a;b is a o-decreasig cotiuous sigmoid fuctio. Traditioal example of sigmoid fuctio is logistic fuctio Oer examples of sigmoid fuctios are f f s s f 3 s s tah s () arcta π s π (3) s (4) s mi,max, s f 4 (5)

2 f 5 s erf πs (6) where erf x π x e t dt. Basic characteristics of RBF Geeral Radial Basis Fuctio (RBF) [] [3] etwor cosists of ree layers: iput, hidde ad output oes. The sigal processig i every output euro is described by liear formula y w x (7) where N is umber of euro iputs, x, w R are iput value ad its weight, y R is euro output ad x. The sigal processig i every hidde euro is described by formula y exp x w (8) where is space factor, y (; ] is euro output ad e oer quatities have e same meaig. Problem Formulatio. Uiversal o liear elemet for MLP ad ANN Havig our favorite sigmoid fuctio we ca costruct base fuctio for RBF hidde layer as a product of f s ad f s but alas e radial property is ot guarateed. Base fuctio ca be used for e approximatio of RBF euro usig formula G s,..., s f s f s (9) Let f s s exp sig () be a ew id of sigmoid fuctio. I is case g s f s f s sig s e e () is a Gaussia erel fuctio. After e substitutio () ito (9) we obtai a fucio which is proportioal to traditioal radial basis fuctio. The existece of ew sigmoid fuctio 4 4

3 f s 6 s exp is i cotradictio wi traditioal logistic fuctio f s ca be deoted as D s f 6 s f s meawhile D D' D" limds limds sig () tah s. The absolute differece betwee em ad reach absolute maximum D for s , s s. So e differece betwee ew ad traditioal perceptro characteristics is raer symbolic e dramatic ad e way is ope for e desig of ew artificial eural etwor.. New structure of ANN which covers MLP ad RBF The defiitio of ew sigmoid fuctio f 6 s is a motivatio to build up a ew id of artificial eural etwor wi two types of processig euros. Defiitio Let N, x, R for,..., w is called liear euro., x. The e fuctio x,w Defiitio Let N, s R for,...,. The e fuctio y w x (3) sigs exp G s (4) is called multiplicative perceptro. Now we ca defie M-RBF ANN as hierarchical artificial eural etwor wi processig layers of two ids. Defiitio 3 Let L N be umber of layers. Let N N be umber of euros i,..., L. Let e liear euros for,..., L / L / st layer cosist of iput euros. Let / L layer be output layer. Let layer of hierarchical ANN for j layer cosists of j. Let j layer cosists of multiplicative perceptros for j,...,. The e etwor is called Multi layer Radial Basis Fuctio ANN ad deoted as M RBF N N N L or M RBF L. Now, it is easy to recogize at ay liear ANN ca be realized as M RBF, ay RBF etwor ca be realized as M RBF 4, ay two layer perceptro etwor ca be approximated as M RBF 3, ay ree layer perceptro etwor wi liear output (MLL) ca be approximated as M RBF 4 ad ay ree layer perceptro etwor (MLP) ca be approximated as M RBF 5. The M RBF is a hierarchical ANN wi icomplete coectivity of eighbor layers. Ay multiplicative perceptro eed ot operate o complete set of euros of previous liear layer. Formally, e ovel etwor is a fuctio L y ANN x, W,..., W. Supposig e traiig set of patters x, y for,..., M, we ca use least square meod for learig of M-RBF. The e sum of squares

4 SSQ M L L W,..., W y ANN x, W,..., W (5) is subject of miimizatio, where... is euclidea orm. The relatioships betwee ew ANN, MLL, MLP, RBF ad OLAM will help to fid iitial structure ad weights of M RBF ad e cotiue to e earest local optimum of SSQ..3 Testig ad compariso The ew testig eviromet i Matlab was desiged. It allows addig ew ANN models ad compare em wi aoer oes. The weights of ANN are optimized by e applicatio of fmico fuctio i Matlab which employs e LSQ meod to global miimum searchig via repeated local optimizatio from radom iitial poits. Followig values are recorded: i umber of iput euros h umber of euros i hidde layer o umber of output euros w umber of weights df degrees of freedom ssq sum of squares of ANN residues sy model error as sy = ssq df (6) The improvemet of model error was tested i e case of M RBF etwor related to RBF oe. The M RBF learig was based o optimum RBF weights.the weights of optimum RBF etwor were coverted to equivalet weights of of M RBF. The optimum parameters of M RBF were searched i e eighborhood of is iitial estimate i costraied space wi % tolerace as local LSQ optimum, of course. 3 Results The tests were performed o ANN time series predictio tas. Data set of aual umber of suspots (suspots.dat) is freely available i e Matlab eviromet. Oly MLP wi characteristics f, RBF wi characteristics g ad M RBF wi ree iput euros ad sigle output euro were tested ad compared. Various umber of hidde euros up to 5 was used for e study of eir properties. Results of MLP, RBF ad M RBF learig were collected i e Tab.. The optimum structures wi e smallest model error (wii type of ANN) were MLP 3 4, RBF 3 3 ad M RBF 3 4. But M RBF etwor had e better performace a MLP or RBF oe. 4 Coclusios Three types of ANN (MLP, RBF, M RBF) were leared to be e best predictors of suspot umber from last ree year history. The results show at M RBF was e best model i e case of model error miimizatio. The M-RBF etwor has more weights e MLP or RBF wi e same umber oliear euros, which reduces e degrees of freedom. However, is effect is icluded i e model error calculatios ad us we recommed e M-RBF etwor as very efficiet tool for data modelig. The learig of M-RBF etwor have to be predeceased wi RBF learig, which is a good geerator of iitial M-RBF weight estimate. Acowledgemet: The paper was created uder e support of grat OHK4-65/ CTU i Prague. Refereces:

5 [] Rumelhart D.E., Hito G.E.,Williams R.J., Learig Represetatios by Bac-Propagatig Error. Nature, 33, 986, pp [] Hayi S., Neural Networs, Macmilla, New Yor, 994. [3] Poggio T, Networs for Approximatio ad Learig, Proceedigs of e IEEE, 78, 99, pp Josef Boští, Jaromír Kual Faculty of Nuclear Scieces ad Physical Egieerig, CTU i Prague Departmet of Software Egieerig Břehová 7, Praha, 59 Czech republic josefbosti@gmail.com, jaromir.ual@fjfi.cvut.cz Tab.: Results of MLP, RBF ad M-RBF learig ANN model i h o w df sy ssq MLP MLP MLP MLP MLP RBF RBF RBF RBF RBF M RBF M RBF M RBF M RBF M RBF