GA-Based Learning Algorithms to Identify Fuzzy Rules for Fuzzy Neural Networks

Transcription

1 Seventh Internatonal Conference on Intellgent Systems Desgn and Applcatons GA-Based Learnng Algorthms to Identfy Fuzzy Rules for Fuzzy Neural Networks K Almejall, K Dahal, Member IEEE, and A Hossan, Member IEEE School of Informatcs, Unversty of Bradford, Bradford BD7 1DP, UK {K.A.Al-mejall; K.P.dahal; M.A.Hossan1}@Bradford.ac.uk Abstract Identfcaton of fuzzy rules s an mportant ssue n desgnng of a fuzzy neural network (FNN). However, there s no systematc desgn procedure at present. In ths paper we present a genetc algorthm (GA) based learnng algorthm to make use of the known membershp functon to dentfy the fuzzy rules form a large set of all possble rules. The proposed learnng algorthm ntally consders all possble rules then uses the tranng data and the ftness functon to perform ruleselecton. The proposed GA based learnng algorthm has been tested wth two dfferent sets of tranng data. The results obtaned from the experments are promsng and demonstrate that the proposed GA based learnng algorthm can provde a relable mechansm for fuzzy rule selecton. 1. Introducton Fuzzy neural networks are hybrd ntellgent systems, whch combne the advantages of both neural networks and fuzzy logc. The neural fuzzy system s a fuzzy system that uses the learnng ablty of the neural networks to determne ts parameters (fuzzy sets, fuzzy membershps and fuzzy rules) by processng data [1]. An mportant topc n desgnng of a fuzzy neural network s the dentfcaton of the fuzzy rules. However, there s no systematc desgn procedure at present [2]. The recent research drecton n the dentfcaton of the fuzzy rules n fuzzy neural networks s to learn and modfy the rules from past experence. Currently, dfferent approaches are used to dentfy the fuzzy rules n the fuzzy neural networks. Quek and Zhou [3] have classfed these rule dentfcaton approaches nto three categores. Frst category of approaches uses lngustc nformaton from experts to dentfy fuzzy rules [4]. Although ths approach converges faster durng tranng and performs better, t s rather subjectve snce lngustc nformaton from experts may vary from person to person, and from tme to tme. The second category of approaches uses unsupervsed learnng algorthms to dentfy fuzzy rules n the fuzzy neural networks pror to the applcaton of neural network technques to adjust the rules [5, 6]. In ths approach the tranng data s the only source of nformaton so t must be representatve, otherwse, the derved fuzzy rules wll be ll defned. The thrd group of approaches s usng supervsed leanng algorthm (partcularly the Backpropagaton technque) to dentfy the fuzzy rules n the fuzzy neural networks[7]. In ths approach the fuzzy neural network appears as black box at the end of the tranng process. Ths paper presents a genetc learnng algorthm to make use of the known membershp functon to dentfy the fuzzy rules. The proposed learnng algorthm belongs to the second approach for rule dentfcaton n fuzzy neural network. The prelmnary work on the GA-based learnng algorthm was presented n our prevous work [8]. Ths paper extends ths dea further by ncorporatng a weght parameter to trade-off between the number of fuzzy rules and overall error. In ths paper we have used GA to make use of the known membershp functon to dentfy the fuzzy rules usng smlar fuzzy neural network structure dscussed n [2]. The proposed genetc algorthm dffers form exstng algorthms such as [5, 6], n terms of beng smple, fast and flexble to control the process of dentfcaton of the fuzzy rules based on the error level. Several authors have proposed a genetc algorthm for fuzzy neural parameters optmzaton to adjust the control ponts of membershp functons or to tune the weghtngs [9-14]. The poneer was Karr[9], who used GAs to adjust membershp functons. Ishbuch et al.[10] proposed a genetc- based method for selectng a small number of sgnfcant fuzzy rules to construct a compact fuzzy classfcaton system wth hgh classfcaton power. Ishbuch and Yamamoto farther developed ths dea by usng mult-objectve genetc local search algorthms n [13]. Wang et al. [11] have pro /07 $ IEEE DOI /ISDA

2 Fgure 1. Structure of the fuzzy neural tool (FNN_Tool) posed a smplfed genetc algorthm to adjust both control ponts of B-splne membershp functons and weghts of fuzzy-neural networks. The proposed algorthm uses sequental-search-based crossover pont method n whch a better crossover pont s determned and only the gene at the specfed crossover pont s crossed as a sngle pont crossover operaton. Ln [14] has proposed a hybrd learnng algorthm for parameters learnng. He has used GA to tune membershp functons at the precondton part of fuzzy rules, whle the least-squares estmate method has been used to tune parameters at the consequent part. Wang et al. [12] have proposed GA-based approach for a feedback drect adaptve fuzzy-neural controller to tune the onlne weghtng factors. Specfcally, they have used a reduced-form genetc algorthm (RGA) to adjust the weghtngs of the fuzzy-neural network. The paper s organzed as follows. The next secton descrbes the structure of the GA-FNN. Secton 3 dscusses the proposed GA-based Learnng Algorthm. The performance of the proposed GA-based learnng algorthm s tested n Secton 4. The conclusons and future works of ths paper are gven n Secton The GA-FNN Structure In ths secton, we descrbe the structure and functon of the proposed GA-FNN model. The structure of the fuzzy neural network model used n ths paper s smlar to the structure proposed n [2]. It s a fvelayer structure, as shown n Fgure 1, where each layer performs an operaton for buldng the fuzzy system. The process of each layer s descrbed below (see Fgure 1): Layer 1: s the nput layer. Nodes at ths layer represent nput lngustc varable and drectly transmt non-fuzzy nput values to the next layer. The nput and the output of ths layer are gven as follows: where (1) o = (1) ( 1 ) s the nput and ( 1 ) (1) o s the output of nput neuron n layer 1. Layer 2: s the fuzzfcaton layer, whch defnes the fuzzy sets and membershp for each of the nput factors. Nodes n ths layer acts as a membershp functon and represents the terms of the respectve lngustc varable. In our model the neurons of ths layer are modeled as a common bell-shaped membershp functon [15], so the nput and the output ( 2 ) ( 2 ), j o of, j fuzzfcaton node at the layer 2 are gven as follows: o = e (2), j ( 2) 2 j,, j ) ( m σ, j (2) where m, and j σ are the centres and the wdths of, j the membershp functon for the nput-label neuron LI, j respectvely. Layer 3: s the fuzzy rule layer, whch defnes all possble fuzzy rules to specfy qualtatvely how the output parameter s determned for varous nstances of the nput parameters. Each node n ths layer represents a fuzzy rule. The nput and the output of a rule node at the layer 3 are gven as follows: (3) (3) y = mn ( x ) (3) where (3 ) ( 3 ) x are the nputs, and k y s the output of fuzzy rule n the layer 3 k 290

3 Layer 4: s the consequence layer (or the output membershp layer). Neurons n the consequence layer represent fuzzy sets used n the consequent part of the fuzzy rules. The nput and the output of a consequence node n the layer 4 are gven as follows: y ( 4 ) ( 4 ) = mn ( 1, Σ x ) (4) (4) where x s the nput (the output of neuron l, l n the (4) fuzzy rule layer), and y s the output of membershp neuron n the layer 4. Layer 5: s the output layer (or the defuzzfcaton layer). Each node at the output layer represents a sngle output varable. The nput and the output of an output neuron n layer 5 are gven as follows: l l, (4) ( a b ) y x = y ( 5 ) = c / c Σ ( y = 1 x ( ( 5 ) 4 ) / b c ) (6) where x s the nput and y s the output neuron n layer 5, a c and b c are the centre and the wdth of the fuzzy set respectvely. Due to the fact that the tranng data set s the only source of nformaton n most cases, the learnng process of the proposed GA-FNN completely reles on tranng data. We proposed the learnng process consstng of three stages. Frst stage s ntalzng the membershp functons of both nput and output varables by determnng ther centres and wdths. To perform ths stage, we have employed a self-organzng algorthm [6] as n other works [2, 5, 16]. A proposed GA based learnng algorthm s performed n the second stage to dentfy the fuzzy rules that are supported by the set of tranng data. In the last stage, the derved structure and parameters are fne tuned by usng the back-propagaton learnng algorthm [15]. The remander of ths paper focuses on only the second stage, whch s usng GA to dentfy the fuzzy rules needed for constructng the fuzzy neural network. 3. The proposed GA-based Learnng Algorthm Most exstng fuzzy neural networks use selforganzng algorthm to dentfy fuzzy rules, lke [6]. However, n most ntegrated fuzzy neural networks, where the membershp functons are determned pror to the dentfcaton of fuzzy rules, self-organzng or compettve learnng becomes redundant, because the boundary of the clusters n the nput and output spaces have already been predefned [2]. To avod ths redundancy of usng self-organzng algorthms to determne the fuzzy parameters, some authors such as Ln and Lee [5] have used the known membershp functons to fnd the fuzzy rules, where compettve learnng s employed to dentfy the fuzzy rules. In Ln and Lee work, all the possble fuzzy rules must be lsted pror to the ntaton of compettve learnng. After compettve learnng, the lnk wth the largest weght s selected and the consequence t connects to s subsequently held as the real consequence of the rule. Although ths method shows satsfactory result wth lmted number of rules, t nvolves teratve tranng before the system comes to a stable state, because at least 50% of all possble fuzzy rules are useless. In ths paper we have proposed a genetc learnng algorthm to make use of the known membershp functon to dentfy only the relevant fuzzy rules. Fgure 2. FNN of two nputs varable and one output varable for all possble fuzzy rules To explan how we have desgned a GA for dentfyng fuzzy rules, consder a smple example of FNN wth two nput lngustc varables x 1 and x 2, and one output lngustc varable y as shown n Fgure 2. After performng the self-organzaton learnng algorthm (frst stage of learnng), each lngustc varable has a number of fuzzy sets, say we have three fuzzy sets. Then the proposed genetc learnng algorthm consders all possble rules. In our smple example there are a total of twenty seven possble rules. In fact these rules are made of nne possble antecedents (precondtons). These antecedents of fuzzy rules are represented by neurons R 1 R 9 of the Fuzzy-Rules Layer n Fgure 2. Each antecedent has lnks wth three possble decson fuzzy sets (neurons n Consequence Layer: Low(L), Medum(M) and Hgh(H)). For example, the three possble fuzzy rules assocated wth neuron R 1 are: If x 1 s L and x 2 s L, then y s L. 291

4 If x 1 s L and x 2 s L, then y s M. If x 1 s L and x 2 s L, then y s H. A number of decsons must be made n order to mplement the GA for dentfyng approprate fuzzy rules. There are problem specfc decsons whch are concerned wth the search space (and thus the representaton) and the form of the ftness functon. The followng steps are employed to dentfy approprate fuzzy rules by the GA. Step 1: Intalzaton: the frst and most mportant step n the mplementaton of the GA technque s encodng of the problem usng an approprate representaton. The encodng used to represent chromosomes (solutons) defnes the sze and the structure of the search space. Here we propose nteger strngs as chromosomes to represent canddate solutons of the problem. The strng s gven by t1,t2,...,t,...,tn, where t s an nteger 0 t M whch ndcates the lnk of neuron R (.e. neurons n Fuzzy-Rules Layer) wth output neurons (.e., neurons n Consequence Layer). N s the number of neurons n the Fuzzy-Rules Layer and M s the number of neurons n the Consequence Layer. For our example, the chromosome has nne ntegers, and 0 t 3. t = 0 ndcates there s no lnk of R wth output neuron; t = 1 ndcates that there s a lnk wth L neuron n consequence Layer and so on. When t = 0, ths means that R has no relaton to that partcular output varables. Hence, that rule and all the correspondng lnks n ths case can be deleted wthout affectng the outputs. Step 2. Ftness functon: n ths step the goodness of every chromosome s evaluated by usng a ftness functon. The ftness functon can be any nonlnear, nondfferentable, or dscontnuous postve functon, because the GA only needs a ftness value assgned to each chromosome. In ths paper, we use a set of tranng data to calculate the ftness of each chromosome based on the followng ftness functon: FIT() = 1 RMS _ ERROR ( ) (7) where RMS_ERROR() represents the root-meansquare error between the fuzzy-neural network outputs and the desred outputs for the th strng. The GA ams to maxmze the ftness functon (7) to mnmze the error value (e). Ths error value s depended on the selected fuzzy rules (by the GA chromosome) and rule weghtngs. The proposed GA chromosome represents a set of fuzzy rules wth t 0 for ncluson and a set of fuzzy rules wth t =0 for gnorng. The weght for all rules assumed to be 1 at ths stage. However, our experment showed that the nclusons of some these rules (t =0) wth low weghtngs can stll mprove the error value. In order to correctly dentfy the mnmum number of the approprate fuzzy rules wthout gnorng any relevant rule that mght mprove the error value, the ftness value of a chromosome s calculated n two stages: Frstly, a chromosome s evaluated as gven by GA (ft_1). I.e. the ftness of the chromosome s calculated wth consderng all rules represented by t 0 (takng weght 1), and rules represented by are gnored. Secondly, for each rule (R) represented by t =0, the ftness of the chromosome s calculated agan wth consderng a low weght LW (e.g. 0.01) for each possble rule assocated wth that rule (R) (n our example there are three possble rules) and then the best one s selected (ft_2). Then these two ftness values (ft_1, ft_2) are compared and the best ftness value s taken. The chromosome s adjusted f the second ftness (ft_2) appeared to be the better one. Step 3. GA operators: Based on our prevous experence wth GA and a number of experments we have selected GA operators and ther parameters to be used for ths applcaton. The GA operators used are steady state replacement approach[17], tournament selecton [18], standard two-pont crossover and a hgher mutaton probablty. The steady state approach drectly nserts a new soluton nto the populaton pool replacng a less ft soluton. The tournament selecton method pcks a subset of solutons at random from the populaton to form a tournament selecton pool, from whch two solutons are selected wth probablty based upon the ftness values of the solutons. The two-pont crossover operator splts the selected solutons at two randomly chosen postons and exchanges the centre sectons wth probablty a crossover probablty. The mutaton operator changes the nteger at each poston n the soluton wthn the allowed range wth a defned mutaton probablty. We use a hgher mutaton prorty n our case because the dversty n the populaton s not drven by recombnaton. The eltst approach, whch ensures that the best soluton n the populaton pool s always retaned, has been appled. The ntal populaton of chromosomes s created randomly. The stoppng crteron for a GA run s to acheve the pre-specfed error level (e). When the GA learnng process has completed (.e. when pre-specfed error level s acheved) after runnng the GA over a large number of runs, we choose the best GA chromosome. Ths best chromosome s decoded to get the structure of the FNN by keepng only the rules that are ndcated by the chromosome. Then the error level (e) can be mproved by usng the back-propagaton learnng algorthm to fne tune the 292

5 rules weghts. By dong so, we only tran the FNN wth the relevant fuzzy rules only. Error Best Average Generatons Fgure 3. Performance graph for 100 generaton of 20 chromosomes. 4. Experments and Comparson In ths secton, the performance of the proposed GA-based learnng algorthm s tested wth two dfferent sets of tranng data. Experment 1: In ths Experment, a set of tranng data conssts of more than thousand data samples whch are generated from 27 rules, some of them lsted below: If x1 s low and x2 s low and x3 s low then y s very low; If x1 s low and x2 s medum and x3 s medum then y s low; If x1 s medum and x2 s hgh and x3 s hgh then y s hgh; If x1 s hgh and x2 s hgh and x3 s hgh then y s very hgh; The weght of all rules set to unty (.e. w = 1). From the rules above, t can be observed that there are three nputs and one output lngustc varables. For each nput lngustc varable, ts term set s defned as {low, medum, hgh}, whle the output lngustc varable term set s defned as { very low, low, medum, hgh, very hgh}. Therefore, there are a total of 135 possble rules, and the resultng FNN conssts of three lngustc nodes, nne nput-label nodes (three for each lngustc node), twenty seven ntal rule nodes, fve output-label nodes, and one output node. After the self-organzed learnng to fnd the membershp functon for each nput-label (output-label) node, the proposed genetc learnng algorthm s employed to dentfy the fuzzy rules. Because of the fact that the weght of all rules used to generate the tranng data was unty, the weght for rules to be gnored used n the calculaton of the ftness n the second stage wth low weght (LW), as dscussed n perfuse secton, does not affect the result. In other words, we dd not have to use thesecond stage of ftness calculaton. After a number of runs, the result shows that the proposed learnng algorthm s able to correctly dentfy all the relevant fuzzy rules wth error level equals zero. Fgure 3 presents an average performance graph of 20 experments created by 100 generatons of 20 chromosomes. Experment 2: In the second experment, we use a smlar set of tranng data whch conssts of more than thousand data samples, but n ths experment the rules have dfferent weghts (.e. w = n, where 0 n 1 ). After the self-organzed learnng to fnd the membershp functon for each nput-label (output-label) node, the proposed genetc learnng algorthm s employed to dentfy the fuzzy rules. In ths case, due to the fact that some rules used n generatng the tranng data may have a low weght (LW) (e.g. 0.1), the weght for rules to be gnored should be set for two stage ftness calculatons. After a number of runs, the result shows that the number of rules dentfed by the algorthm and the error level (e) s affected by the value of LW. Table I shows the relatonshp between LW and the number of generated rules and the error level (e), created by 100 generatons of 20 chromosomes. From Table I, t s clear that, any ncrease of the value of LW results a decrease of the number of relevant rules dentfed by the algorthm and also results an ncrease of the value of the error level (e). The explanaton of these observatons s that when the value of LW s large the possblty of gnorng some relevant rules wth low weght s hgh, whch affects negatvely on the level error. For example, when the value of LW was 0.9 the number of relevant rules was 18 (9 gnored rules) and the error level was , whle the value of LW was 0.1, the number of relevant rules dentfed s 27 (.e. all rules consdered) and the error level decreased to Table 1. The effect of LW on the number of rules and the error level LW Number of Rules e It s usually the case that a low value of LW produces a better error value as ths allows to consder a large number of rules. In ths example, where we have a lmted number of rules (.e. 27), we can consder all rules by settng a low value of LW to get a mnmum error level. However, for a case wth a very large number of possble rules (whch are mpossble to consder), t s mportant to lmt the number of rules for consderaton by settng an approprate value of LW to obtan an acceptable error level. For example, f we have a problem wth a large number of rules (large number of nputs and outputs), and there are two values of LW lw1,lw2 (lw1<lw2). If lw1 results a number of rules N and a level error e, and lw2 results a de- 293

6 crease of N by 30% and an ncrease of e by only 0.1%. In ths case possbly we set LW to lw2. Also, n order to evaluate the effectveness of applcaton of the proposed genetc learnng algorthm n FNN, we have employed the proposed GA-learnng algorthm n GA-FNN to a real world case study of road traffc management. The results obtaned from the case study are promsng and demonstrate that the proposed GA based learnng algorthm can provde a relable mechansm for fuzzy rule selecton. The results of ths case study have been presented and dscussed n [8]. 5. Concluson and Future Work Ths paper has descrbed a GA based learnng algorthm to dentfy the fuzzy rules for FNNs. The proposed algorthm makes use of the known membershp functons to dentfy only the relevant fuzzy rules. Intally, t has consdered all possble rules and then used the tranng data and the ftness functon to select a lmted number of the relevant fuzzy rules by ntroducng a weght parameter to trade-off between number of rules and error value. In order to test the capabltes of the proposed GA-based learnng algorthm for correct dentfcaton of all the relevant fuzzy rules, t has been tested wth two dfferent sets of tranng data. The results of the proposed GA-based learnng algorthm demonstrate ts capabltes and merts n term of dentfcaton of the fuzzy rules fast and correctly. Further experments on ts applcaton to a case study of road traffc management, whch have been reported elsewhere, also show the effectveness of the proposed algorthm n FNN. Currently we have demonstrated the techncal feasblty of the proposed GA-learnng algorthm to dentfy fuzzy rules, n whch all possble rules were used to select the relevant rules. In the next stage we wll nvestgate the possblty of developng the proposed learnng algorthm to start wth a lmted number of relevant rules and then the learnng process wll be ncreased (or decreased) to that number rules nstead of consderng all possble rules. 6. References [1] T. Ln, Neural Fuzzy Control Systems Wth Structure and Parameter Learnng. Sngapore: World Scentfc [2] C. Quek, M. Pasquer, and B. B. S. Lm, "POP- TRAFFIC: a novel fuzzy neural approach to road traffc analyss and predcton," Intellgent Transportaton Systems, IEEE Transactons on, vol. 7, pp. 133, [3] C. Quek and R. W. Zhou, "The POP learnng algorthms: reducng work n dentfyng fuzzy rules," Neural Netw, vol. 14, pp , [4] B. Krause, A. S. von, W, and K. Lmper, "A neurofuzzy adaptve control strategy for refuse ncneraton plants," Fuzzy Sets and Systems, vol. 63, pp , [5] C. T. Ln and C. S. G. Lee, "Neural-Network-Based Fuzzy Logc Control and Decson System," IEEE Transactons on Computers, vol. 40, pp , [6] R. R. Yager, "Modelng and formulatng fuzzy knowledge bases usng neural networks," Neural Networks, vol. 7, pp , [7] H. Ishbuch, H. Tanaka, and H. Okada, "Interpolaton of fuzzy f-then rules by neural networks," Internatonal Journal of Approxmate Reasonng, vol. 10, pp. 3 27, [8] K. Almejall, K. Dahal, and A. Hossan, "Intellgent Traffc Control Decson Support System," n EVOWorkshop2007. Valenca, Span: (to be publshed n Computer Scence Lecture notes by Spnger), [9] C. Karr, "Genetc algorthms for fuzzy controllers," AI Expert, vol. 6, pp , [10] H. Ishbuch, K. Nozak, N. Yamamoto, and H. Tanaka, "Selectng fuzzy f-then rules for classfcaton problems usnggenetc algorthms," Fuzzy Systems, IEEE Transactons on, vol. 3, pp , [11] W. Y. Wang, T. T. Lee, C. C. Hsu, and Y. H. L, "GAbased learnng of BMF fuzzy-neural network," Fuzzy Systems, FUZZ-IEEE'02. Proceedngs of the 2002 IEEE Internatonal Conference on, vol. 2, [12] W. Y. Wang, C. Y. Cheng, and Y. G. Leu, "An onlne GA-based output-feedback drect adaptve fuzzyneural controller for uncertan nonlnear systems," Systems, Man and Cybernetcs, Part B, IEEE Transactons on, vol. 34, pp , [13] H. Ishbuch and T. Yamamoto, "Fuzzy rule selecton by mult-objectve genetc local search algorthms and rule evaluaton measures n data mnng," Fuzzy Sets and Systems, vol. 141, pp , [14] C. J. Ln, "A GA-based neural fuzzy system for temperature control," Fuzzy Sets and Systems, vol. 143, pp , [15] D. E. Rumelhart, G. E. Hnton, and R. J. Wllams, "Learnng nternal representatons by error propagaton, Parallel dstrbuted processng: exploratons n the mcrostructure of cognton, vol. 1: foundatons," MIT Press, Cambrdge, MA, [16] P. J. Werbos, "Neurocontrol and fuzzy logc: connectons and desgns," Internatonal Journal of Approxmate Reasonng, vol. 6, pp , [17] G. Sywerda, "Unform crossover n genetc algorthms," Proceedngs of the thrd nternatonal conference on Genetc algorthms table of contents, pp. 2-9, [18] D. E. Goldberg and K. Deb, "A Comparatve Analyss of Selecton Schemes Used n Genetc Algorthms," Urbana, vol. 51, pp ,