Elliptical Rule Extraction from a Trained Radial Basis Function Neural Network

Transcription

1 Ellptcal Rule Extracton from a Traned Radal Bass Functon Neural Network Andrey Bondarenko, Arkady Borsov DITF, Rga Techncal Unversty, Meza ¼, Rga, LV-1658, Latva Andrejs.bondarenko@gmal.com, arkadjs.borsovs@cs.rtu.lv Abstract: Currently knowledge management and dscovery plays crucal role n dfferent ndustres. Apart from that n many cases decsons made by companes n msson crtcal areas (lke nuclear power ndustry, medcne and fnance, to name a few) should be verfed by doman expert and/or explaned. Ths s needed to fulfll regulatory requrements, to verfy correctness and / or dscover new knowledge. Artfcal neural networks are well known models that can be used to solve classfcaton problems. Unfortunately they are black box models for end users, because classfcaton process s fully hdden from the observer and cannot be easly mapped nto formalzed meanngful form that can be understood by the doman expert. There are multple knowledge representatons, and the one chosen to work wth n ths study s ellptcal rules. A set of such rules can be used to determne the class of an nput data pont by the determnaton of ellpsod that covers provded pont. Current paper addresses the problem of ellptcal rules (ER) extracton from traned artfcal radal bass functon neural network (RBFNN). Ths study uses RBFNN wth tunable nodes - such networks are bult usng orthogonal forward feature selecton algorthm and they usually have much less neurons (n comparson to RBFNNs contanng neurons wth fxed rad) whle havng comparable or even superor accuracy. The artcle poses non-convex optmzaton problem of fndng multple ellpsods of largest volume nscrbed nto RBFNN decson boundary. Non-convexty arses due to complex nature of RBFNN decson boundary whch serves as constrant. Further we descrbe an algorthm for extracton of ER from traned RBFNN. The provded expermental results show that few of the extracted ellptcal rules have comparable and n some cases hgher classfcaton accuracy than orgnal RBFNN. Although curse of dmensonalty s applcable to the provded algorthm, experments have shown that t can be readly used for relatvely low-dmensonal problems. Fnally are descrbed the possble future research drectons. Keywords: radal bass functon networks, knowledge acquston, optmzaton. Introducton There are many problems for whch artfcal neural networks are stll a preferable soluton. They can be traned on data wth outlers and are natural choce for mult classfcaton problems, although the way n whch classfcaton s performed s a black-box algorthm for the end user. Due to specfc requrements of dfferent ndustres for clear and formal decson process descrpton for valdaton and knowledge gatherng purposes, knowledge extracton from traned neural network s a topcal problem. In ths paper we propose an algorthm for the extracton of ellptcal rules from traned artfcal radal bass functon neural network wth tunable nodes (Chen et al., 2009). Radal bass functon neural networks are local n ther nature - meanng each neuron s responsble for classfcaton decson n specfc regon based on defned neuron radus and locaton parameters (and weght). Havng that, t s possble to defne optmzaton problem that would allow us to cover classfcaton regon wth ellpsods - hence gvng us a formalzed vew of how classfcaton s made. The structure of the paper s as follows: Chapter 2 provdes background on neural networks, Chapter 3 poses optmzaton problem and algorthm, Chapter 4 shows expermental results and Chapter 5 concludes. RBF Neural Network Wth Tunable Nodes Radal bass functon artfcal neural network (Moody et al., 1989; Poggo et al., 1990) can be represented as follows: N (x) = a p( x c (1) where ) a s the -th neuron weght, c s the -th neuron center, N s neurons count. The norm s usually taken to be Eucldean dstance and the bass functon p s taken to be Gaussan: p( x c )= exp( β x ) (2) c There are multple strateges for tranng ths knd of network. The network can have neurons wth fxed rad, as ths smplfes learnng procedure. On the other hand, havng neurons wth dfferent rad reduces the amount of neurons requred to get necessary classfcaton accuracy. But havng neurons wth varyng rad requres a

2 specalzed learnng approach. We used the method descrbed n (Chen et al., 2009). The proposed method allows us to buld RBF networks wth a small amount of neurons whle preservng hgh accuracy. Rule Extracton Optmzaton problem We can treat ellptcal rule extracton from traned RBF neural network can as an optmzaton problem of fndng ellpsods of maxmum volume nscrbed nto the space area(-s) defned by RFBNN decson boundary. It s possble to choose maxmzaton crtera other than volume- lke amount of ponts covered by newly shaped ellpsod, but n ths case we wll not be able to use gradent-based solvers. Let's denote ellpsod as ε = Bu +d u (3) a unt ball under affne transformatons. As descrbed n (Boyd and Vandenberghe, 2004) we say that B s n- length vector contanng postve elements, so ellpsod volume s proportonal to detb. We can wrte down the followng optmzaton problem: maxlog ( detb)) s. t. RBNN Ths means we are lookng for ellpsod of maxmum volume fully nscrbed nto RBFNN defned decson boundary. Apart from that we have explct constrant that ellpsod should have non-negatve rad elements of vector B. The descrbed problem allows us to fnd frst ellpsod nscrbed nto the RBFNN decson boundary. We should note that due to RBFNN nature, constrants of our problem are non-convex, thus the found ellpsod can be a local soluton. In most cases t wll be nsuffcent to represent RBFNN wth a sngle ellpsod, thus we need to search for addtonal ellpsods. For that we need to apply the teratve approach. In contrast to recursve volume subdvson appled n (Nunez et al., 2002), we have chosen another strategy. Although recursve subdvson s also a feasble approach as t does not requre objectve functon modfcaton, on the other hand, t generates larger count of ellpsods. Instead of space subdvson for searchable regons on each recursve teraton, we are just lookng for nscrbed ellpsod wth maxmum volume not covered by already found ellpsods, meanng the larger fracton of ellpsod les outsde the regon already covered by exstng ellpsods, the better t fts our needs: ε max(ε vol Εvol ) s. t. RBNN ε here εvol denotes the volume of found ellpsod and Ε vol s the volume of already exstng ellpsods and P s penalty term. Modfcaton n terms of volume calculaton ntroduced large plateau areas wth objectve functon equal to zero. To allow faster convergence of optmzaton procedure, penalty term P calculates mnmal dstance between canddate ellpsod center and border of set formed by ntersecton of all prevously found ellpsods. Thus the 'further' the center of canddate ellpsod contaned wthn other found ellpsods s, the larger penalty term P wll be. Such modfcatons allow us to ensure that, on each optmzaton teraton convergence wll run faster and wll allow us to fnd a new ellpsod, whch wll cover most of not yet covered volume. Of course, there s no guarantee that the found ellpsod wll be a global soluton. Algorthm Here we wll descrbe an algorthm for ellptcal rule extracton from a traned RBFNN. It s necessary to menton that extracted ellpsods wll be orented parallel to coordnate axes, so orgnal unt ball s deformed only by stretchng, but not by rotaton. On the very frst teraton we need to fnd an nscrbed ellpsod of maxmum volume that fts nto the RBFNN decson boundary. In ths case, objectve functon s defned as: 2 P log( prod ( B)) (6) (4) (5) 24

3 Fg. 1. Decson boundary of RBFNN wth 2 neurons and 2 extracted ellpsods. Fg. 2. Decson boundary of RBFNN wth 6 neurons and 4 extracted ellpsods. The algorthm s lsted below and has these nputs: Data - tranng nput vectors, C, R, w - are parameters depctng RBFNN from whch wll be decomposed nto the ellptcal rules. The output of the algorthm s Ellpses set whch contans ellpsods. In the algorthm descrpton crbf s handle for constrant functon RBF whch accepts C, R and w whch are neurons centers, rad and weghts respectvely; ub, lb and x0 are the upper bound, lower bound and ntal startng pont for optmzaton. objvol s objectve functon whch accepts ellpsod vector contanng ellpsod rad and ellpsod center vector. crbfmod and objvolmod are constrants and objectve functon modfed versons. U are ponts covered by RBFNN, but not covered by the suppled set of ellpsods. n s the amount of ponts (from U) that are not covered by newly found ellpsod. If n s 0, then algorthm ads newly found ellpsod and returns the result. Ths verson of the algorthm s dfferent than that provded n (Bondarenko and Borsov, 2012). Although t produces the same results n a two-dmensonal case, t s capable of generatng only a sngle nscrbed ellpsod. In haberman data set we stumbled on cases when a sngle ellpsod was enough to cover data all ponts that are located nsde RBF decson boundary. Thus we slghtly modfed the algorthm allowng t to termnate after acquston of the frst ellpsod. Another pont not depcted n the algorthm below s that n one of the two examned cases the algorthm produced four ellpsods (ths was set by maxmum allowed ellpsods count varable), but ther manual clean-up revealed that only two of them are suffcent for coverng all, but two ponts covered by RBFNN beng decomposed. Such clean-up cannot be easly ncorporated nto the algorthm tself as each found ellpsod drects global search nto new not covered volumes, thus even f ellpsod s not coverng any new not yet covered ponts t should be preserved. In case when genetc algorthm or mult-start global search are used ths should not be a problem. Fg. 3. Decson boundary of RBFNN wth 6 neurons and 5 extracted ellpsods. Fg. 4. Decson boundary of RBFNN wth 7 neurons and 4 extracted ellpsods. 25

4 IN: ( maxellpsodscount, Data, C, R, w ) OUT: ( Ellpsods ) crbf constrantrbf(x, C, R, w); objvol objvolume(x); e1 = solve(crbf, objvol, ub, lb, x0); Ellpsods = e1; U = uncoveredponts(data, Ellpsods, C, R, w); f (count(u) == 0) return e1; end = 2; whle < maxellpsodscount ++; U = uncoveredponts(data, Ellpsods, C, R, w); crbfmod constrantrbfmodfed(x, C, R, w); objvolmod objvolumemodfed(x); e = solve(crbfmod, objvolmod, ub, lb, x0); n = sze(uncoveredponts(u, e, C, R, w), 1); Ellpsods = Ellpsods + e; f (n == 0) break; end end RBFNN Accuracy, Extracted Ellpsods Accuracy and Count Table 1 # of Neurons n RBFNN Experments RBNN Tran RBNN Test Ellpsods Tran Ellpsods Test Ellpsods count max mean mn 2 neurons neurons neurons neurons We have created an algorthm supportng two and three dmensonal nput space. Furthermore, our fnal goal was to show that extracted rules are approxmatng RBFNN as close as possble. We set up experments on synthetc two-dmensonal Rpley data set, whch can be found at (Frank and Asuncon, 2012) as well as haberman survval data set. Both of them have two classes. We run RBFNN constructon algorthm descrbed n (Chen et al., 2009) to construct several neural networks contanng dfferent amounts of neurons. Furthermore we observed only closed RBFNN defned classfcaton boundares whch can be seen n fgures. Lookng at the algorthm one can notce maxellpsodscount varable. We ve ntalzed t wth the number of neurons n the subject RBFNN, the only excepton was a network wth 9 neurons, for whch maxmum ellpsods count to be extracted was set to seven as well as three dmensonal data case where we set that value to three ellpsods. As t was already noted, the algorthm was not executed on open (not bounded areas meanng decson boundary les outsde the lower and upper bounds) decson areas, and RBFNN decson boundares consstng of several separate space volumes (lke two neurons formng two separate postve decson areas). Decson boundares and extracted ellpses can be observed n Fg. 1-8 for two dmensonal case and Fg for three-dmensonal case. Expermental results can be observed n Table 1. One can notce that testng accuraces are hgher than the tranng ones; ths s due to the nature of tranng/testng data beng used. We have not collected accuraces for three-dmensonal cases apart the fact that ellpsods bult for the case depcted n Fg.9 left 3 uncovered ponts, whle 2 ellpsods depcted n Fg. 10 covered all data ponts that led nsde RBFNN decson boundary. Overall computaton tme was partally an ssue for us, fndng the very frst ellpsod turned out to be rather quck operaton whle searchng for subsequent ellpsods was more CPU ntensve task due to the amount of 26

5 computatons needed to calculate modfed objectve functon. Another pont to menton s algorthms used n volume ntersecton and ellpsod contanment wth RBFNN boundary calculatons. In case of 2 dmensons we utlzed mappng / clppng algorthms that allow us to acqure fgure whch corresponds to area belongng to canddate ellpse layng outsde of already found ellpses. In case of three dmensons, we utlzed an approach smlar to the one descrbed n (Persson and Strang, 2004). We defned dstance-based volumes whch were translated nto three-dmensonal meshes. Ths mpled selecton of approprate mesh grd step for constructon of sosurface for whch volume can be calculated wth approprate precson. For checkng whether ellpsod s fully ncluded wthn RBFNN decson boundary we created a set of ponts on ts surface and checked each of ponts to be belongng to the requred bounded volume. Although one can use any of the global optmzaton approaches, we beleve GA s not a good opton here, whle search nvolvng local solvers wth dfferent startng ponts proved to be the best n terms of computaton tme. Overall accuracy of the extracted rules ellpses n 3 out of 4 cases les wthn 1% whch s a good result, takng nto account the amount of rules beng extracted. Fg. 5. Decson boundary of RBFNN wth 7 neurons and 5 extracted ellpsods. Fg. 6. Decson boundary of RBFNN wth 9 neurons and 7 extracted ellpsods. Fg. 7. Decson boundary of RBFNN wth 9 neurons and 5 extracted ellpsods. Fg. 8. Decson boundary of RBFNN wth 9 neurons and 7 extracted ellpsods. 27

6 Fg. 9. Three dmensonal case. Decson boundary of RBFNN wth 2 neurons and 3 extracted ellpsods. Fg. 10. Three-dmensonal case. Decson boundary of RBFNN wth 2 neurons and 2 extracted ellpsods. We beleve t s possble to lower executon tmes by ntroducng heurstcs for ntal pont selecton. Thus bascally we can deal wth the acquston of local solutons whch can be better approach than the utlzaton of multstart optmzaton approach. In our algorthm we have used an ellpse fully resdng nsde RBFNN decson boundary wthout applyng addtonal constrants; though constrants relaxaton can potentally lower the count of extracted rules. Concluson We have nvestgated the possblty of ellptcal rule extracton from radal bass functon neural networks. We developed an algorthm whch was successfully appled to two and three-dmensonal nput data and radal bass functon neural network traned on that data. The results observed ndcate that the proposed algorthm can be successfully appled to low-dmensonal problems. The experments have shown that computaton tme can be a problem especally n case of larger RBF neural networks. Apart from that feasble research drecton s RBF structure explotng to speed up constrants calculaton, along wth that algorthm s not tested on RBFNN decson boundary whch covers open sets and several solated space regons. Here by sayng an open set we mean that classfcaton boundary partally les behnd the upper and lower doman boundares. Ths can be solved by extendng boundares further away or even elmnatng them, although the effect of that needs to be tested and clearly fndng approprate soluton depends on the optmzaton algorthm used. As t was noted n the Chapter Rule Extracton, one can utlze recursve space subdvson for subsequent searches although ths can mply some lmtatons on ftness of found ellpsods. Overall amount of found ellpsods along wth the demonstrated accuracy shows that the proposed approach s feasble, especally for small radal bass functon neural networks wth low-to-moderately szed RBF layer. References Moody, J.and Darken, C.J., Fast learnng n networks of locally tuned processng unts, Neural Computaton, 1, pp Poggo, T.and Gros, F., Networks for approxmaton and learnng, Proc. IEEE 78(9), pp Chen, S.X., Hong, Luk, B.L. and Harrs, C.J., Constructon of tunable radal bass functon networks usng orthogonal forward selecton, IEEE Trans. Systems, Man, and Cybernetcs, Part B, Vol.39, No.2, pp Bondarenko, A. and Jumutc, V., Extracton of Interpretable Rules from Pecewse-Lnear Approxmaton of a Nonlnear Classfer Usng Clusterng-Based Decomposton, Cambrdge, AIKED. Nunez, H., Angulo, C., Catala, A., Rule extracton from support vector machnes, Proceedngs of the 10 th European Symposum on Artfcal Neural Networks, pp , Bruges, Belgum, Aprl Boyd, S., Vandenberghe, L., Convex Optmzaton, Cambrdge Unversty Press. Bondarenko, A. and Borsov, A., Ellptcal Rules Extracton from Traned Radal Bass Functon Neural Network, Scentfc Journal of Rga Techncal Unversty. Frank, A.and Asuncon, A., UCI Machne Learnng Repostory, Avalable at: 29, School of Informaton and Computer Scence, Irvne, Unversty of Calforna. Persson, P.O. and Strang., G.,2004. A Smple Mesh Generator n Matlab, SIAM Revew, June, Volume 46(2), pp