Proceedngs of the 5th Medterranean Conference on Control & Automaton, July 7-9, 007, Athens - Greece T4-003 Tranng ANFIS Structure wth Modfed PSO Algorthm V.Seyd Ghomsheh *, M. Alyar Shoorehdel **, M. Teshnehlab ** * Computer Department of Islamc Azad unversty, Kermanshah, Iran And Computer Department of Scence and research branch Islamc Azad unversty of Tehran, Iran vahdseyd@gmal.com ** K. N. Toos unversty technology Tehran, Iran Abstract- Ths paper ntroduces a new approach for tranng the adaptve network based fuzzy nference system (ANFIS). The prevous works emphaszed on gradent base method or least square (LS) based method. In ths study we apply one of the swarm ntellgent branches, named partcle swarm optmzaton (PSO) wth some modfcaton n t to the tranng of all parameters of ANFIS structure. These modfcatons are nspred by natural evolutons. Fnally the method s appled to the dentfcaton of nonlnear dynamcal system and s compared wth basc PSO and showed qute satsfactory results. Keywords: Partcle Swarm Optmzaton, TSK System, Fuzzy Systems, ANFIS, Swarm Intellgent, Identfcaton, Neuro- Fuzzy. T I. INTRODUCTION he complexty and the dynamcs of some problems, such as predcton of chaotc systems and adaptng to complex plant, requre sophstcated methods and tools for buldng an ntellgent system. Usng fuzzy systems as approxmators, dentfer and predctor, s relable approach for ths purpose [, ]. The combnaton of fuzzy logc wth archtectural desgn of neural network led to creaton of neuro- fuzzy systems whch beneft from feed forward calculaton of output and backpropagaton learnng capablty of neural networks, whle keepng nterpretablty of a fuzzy system [3]. The TSK [4, 5] s a fuzzy system wth crsp functons n consequent, whch perceved proper for complex applcatons [6]. It has been proved that wth convenent number of rules, a TSK system could approxmate every plant [7]. The TSK systems are wdely used n the form of a neuro- fuzzy system called ANFIS [8]. Because of crsp consequent functons, ANFIS uses a smple form of scalng mplctly. Ths adaptve network has good ablty and performance n system dentfcaton, predcton and control and has been appled n many dfferent systems. The ANFIS has the advantage of good applcablty as t can be nterpreted as local lnearzaton modelng and conventonal lnear technques for state estmaton and control are drectly applcable. The ANFIS composes the ablty of neural network and fuzzy system. The tranng and updatng of ANFIS parameters s one of the man problems. The most of the tranng methods are based on gradent and calculaton of gradent n each step s very dffcult and chan rule must be used also may causes local mnmum. Here, we try to propose a method whch can update all parameters easer and faster than the gradent method. In the gradent method convergence of parameters s very slow and depends on ntal value of parameters and fndng the best learnng rate s very dffcult. But, n ths new method so called PSO, we do not need the learnng rate. The rest of the paper s organzed as follows: In Secton II, we revew ANFIS. In Secton III we dscuss PSO method. An overvew of the proposed method and appled ths method to nonlnear dentfcaton s presented n Secton IV. Fnally, Secton V presents our conclusons. II. THE CONCEPT OF ANFIS A. ANFIS Structure Here, type-3 ANFIS topology and learnng method whch use for ths neuro- fuzzy network s presented. Both Neural Network and Fuzzy Logc [9] are model-free estmators and share the common ablty to deal wth the uncertantes and nose. Both of them encode the nformaton n a parallel and dstrbute archtecture n a numercal framework. Hence, t s possble to
Proceedngs of the 5th Medterranean Conference on Control & Automaton, July 7-9, 007, Athens - Greece T4-003 convert fuzzy logc archtecture to a neural network and vce-versa. Ths makes t possble to combne the advantages of neural network and fuzzy logc. A network obtaned ths way could use excellent tranng algorthms that neural networks have at ther dsposal to obtan the parameters that would not have been possble n fuzzy logc archtecture. Moreover, the network obtaned ths way would not reman a black box, snce ths network would have fuzzy logc capabltes to nterpret n terms of lngustc varables [0]. The ANFIS s composed of two approaches neural network and fuzzy. If we compose these two ntellgent approaches, t wll be acheve good reasonng n qualty and quantty. In other words we have fuzzy reasonng and network calculaton. ANFIS s network organzes two parts lke fuzzy systems. The frst part s the antecedent part and the second part s the concluson part, whch are connected to each other by rules n network form. If ANFIS n network structure s shown, that s demonstrated n fve layers. It can be descrbed as a mult-layered neural network as shown n Fg. (). Where, the frst layer executes a fuzzfcaton process, the second layer executes the fuzzy AND of the antecedent part of the fuzzy rules, the thrd layer normalzes the membershp functons (MFs), the fourth layer executes the consequent part of the fuzzy rules, and fnally the last layer computes the output of fuzzy system by summng up the outputs of layer fourth. Here for ANFIS structure (fg. ()) two nputs and two labels for each nput are consdered. The feed forward equatons of ANFIS are as follows: w = μ A ( x) μ B ( y), =,. () = w w, =,. w + w () f = px + q y + r z f = p x + q y + r z w f + w f f = = w f + w f w + w (3) In order to model complex nonlnear systems, the ANFIS model carres out nput space parttonng that splts the nput space nto many local regons from whch smple local models (lnear functons or even adjustable coeffcents) are employed. The ANFIS uses fuzzy MFs for splttng each nput dmenson; the nput space s covered by MFs wth overlappng that means several local regons can be actvated smultaneously by a sngle nput. As smple local models are adopted n ANFIS model, the ANFIS approxmaton ablty wll depend on the resoluton of the nput space parttonng, whch s determned by the number of MFs n ANFIS and the number of layers. Usually MFs are used as bell-shaped wth maxmum equal to and mnmum equal to 0 such as: Where { a b, c } (4) (5), are the parameters of MFs whch are affected n shape of MFs. Fgure (): The equvalent ANFIS (type-3 ANFIS) B. Learnng Algorthms The subsequent to the development of ANFIS approach, a number of methods have been proposed for learnng rules and for obtanng an optmal set of rules. For example, Mascol et al [] have proposed to merge Mn-Max and ANFIS model to obtan neuro-fuzzy network and determne optmal set of fuzzy rules. Jang and Mzutan [] have presented applcaton of Lavenberg-Marquardt method, whch s essentally a nonlnear least-squares technque, for learnng n ANFIS network. In another paper, Jang [3] has presented a scheme for nput selecton and [0] used Kohonen s map to tranng. Jang [8] s ntroduced four methods to update the parameters of ANFIS structure, as lsted below accordng to ther computaton complextes:. Gradent decent only: all parameters are updated by the gradent descent.. Gradent decent only and one pass of LSE: the LSE s appled only once at the very begnnng to get the ntal values of the consequent
Proceedngs of the 5th Medterranean Conference on Control & Automaton, July 7-9, 007, Athens - Greece T4-003 parameters and then the gradent decent takes over to update all parameters. 3. Gradent decent only and LSE: ths s the hybrd learnng. 4. Sequental LSE: usng extended Kalman flter to update all parameters. These methods update antecedent parameters by usng GD or Kalman flterng. These methods have hgh complexty. In ths paper we ntroduced a method whch has less complexty and fast convergence. III. PARTICLE SWARM OPTIMIZATION (PSO) ALGORITHMS A. General PSO Algorthm. The partcle swarm optmzaton (PSO) algorthms are a populaton-based search algorthms based on the smulaton of the socal behavor of brds wthn a flock [4]. They all work n the same way, whch s, updatng the populaton of ndvduals by applyng some knd of operators accordng to the ftness nformaton obtaned from the envronment so that the ndvduals of the populaton can be expected to move toward better soluton areas. In the PSO each ndvdual fles n the search space wth velocty whch s dynamcally adjusted accordng to ts own flyng experence and ts companon flyng experence, each ndvdual s a pont n the D- dmensonal search space[5]. Generally, the PSO has three major algorthms. The frst s the ndvdual best. Ths verson, each ndvdual compares poston to ts own best poston, pbest, only. No nformaton from other partcles s used n these type algorthms. The second verson s the global best. The socal knowledge used to drve the movement of partcles ncludes the poston of the best partcle from the entre swarm. In addton, each partcle uses ts hstory of experence n terms of ts own best soluton thus far. In ths type the algorthms s presented as:. Intalze the swarm, P(t), of partcles such that the poston x (t) of each partcle P P(t) s random wthn the hyperspace, wth t = 0.. Evaluated the performance F of each partcle, usng ts current poston x (t). 3. Compare the performance of each ndvdual to ts best performance thus far: f F ( x then: < pbest pbest = F( x (6) x = x (t) (7) pbest 4. Compare the performance of each ndvdual to global best partcle: f F( x < gbest then: gbest = F( x (8) x = x (t) (9) gbest 5. Change the velocty vector for each: v t) = v ( t ) + ρ( x pbest x + ρ ( x gbest x (0) ( Where ρ and ρ are random varables. The second term above s referred to as the cogntve component, whle the last term s the socal component. 6. Move each partcle to a new poston: x ( t) = x ( t ) + v ( t) () t = t + 7. Go to step, and repeat untl convergence. The random varables ρ and ρ are defned as ρ = rc and ρ = rc, wth, r, r ~ U (0,) and C and are postve acceleraton constant. C Kennedy has studed the effects of the random varables ρ and ρ on the partcle's trajectores. He asserted that C + C 4 guarantees the stablty of PSO [6]. B. Modfed PSO Algorthm In ths approach we remove worst partcle n populaton and replace t by new partcle. Ths s mportant that how to determne worst partcle and how to generate new partcle for current populaton. The partcle s selected wth worst local best value at generaton. Then we randomly choose two partcles from the populaton and use crossover operator to generate two new partcles. Then select the best one from two newly generated partcles and the selected partcle and replace the worse partcle wth t. In other hand we combne GA operator and PSO algorthm to modfy t.ths modfcaton make converges faster than basc algorthm.
Proceedngs of the 5th Medterranean Conference on Control & Automaton, July 7-9, 007, Athens - Greece T4-003 IV. LEARNING BY PSO In ths secton, the way PSO employed for updatng the ANFIS parameters s explaned. The ANFIS has two types of parameters whch need tranng, the antecedent part parameters and the concluson part parameters. The membershp functons are assumed Gaussan as n equaton (5), and ther parameters are{ a, where, b, c } a s the varance of membershp functons and c s the center of MFs. Also b s a tranable parameter. The parameters of concluson part are traned and here are represented wth { p, q, r }. A. How to Apply PSO for Tranng ANFIS parameters There are 3 sets of tranable parameters n antecedent part { a }, each of these, b, c parameters has N genes. Where, N represents the number of MFs. The concluson parts parameters ({ p }) also are traned durng optmzaton, q, r algorthm. Each chromosome n concluson part has ( I + ) R genes that R s equal to Number of rules and I denotes dmenson of data nputs. For example each chromosome n concluson part n fg () has 6 genes. The ftness s defned as mean square error (MSE). Parameters are ntalzed randomly n frst step and then are beng updated usng PSO algorthms. In each teraton, one of the parameters set are beng updated..e. n frst teraton for example a s are updated then n second teraton b are updated and then after updatng all parameters agan the frst parameter update s consdered and so on. B. Nonlnear Functon Modelng Example : Identfcaton of a nonlnear dynamc system. In ths example, the nonlnear plant wth multple tme-delays s descrbed as [8] y p( k + ) = f ( y p( k ), y p( k ), y p ( k ), u( k ), u( k )) () π k sn( ) 0 < k < 50 5.0 50 k < 500 (4) u ( k ) =.0 500 k < 750 π k π k 0.3sn( ) + 0.sn( ) 5 5 π k + 0.6sn( ). 750 k < 000 0 Fg. (a) shows results usng the ANFIS and PSO for dentfcaton (sold lne: actual system output, the dotted lne: the ANFIS and PSO result). Fg. (b) Present the MSEs of modfed algorthm and Fg. (c) Present the MSEs of basc algorthm. The parameters for tranng the ANFIS wth ths PSO algorthm lsted n Table I. Example : Identfcaton of nonlnear systems. We consder here the problem of dentfyng a nonlnear system whch was consdered n [8]. A bref descrpton s as follows. The system model s yk ( + ) = f( yk ( ), yk ( )) + uk ( ) (5) Where xx ( x +.5) f ( x, x ) = + x + x (6) Fg.3 (a) shows results usng the ANFIS and PSO for dentfcaton (sold lne: actual system output, the dotted lne: the ANFIS and PSO result). Fg.3 (b) Present the MSEs of ths algorthm and Fg.3 (c) Present the MSEs of basc algorthm. The parameters for tranng the ANFIS neural network wth ths PSO algorthm lsted n Table II. TABLE I: PARAMETERS FOR EXAMPEL I Where f ( x, x, x 3, x 4, x 5) = x x x 3x 5( x 3 ) + x 4 + x + x 3 (3) Here, the current output of the plant depends on three prevous outputs and two prevous nputs. Testng nput sgnal used s as the followng: NO of nput NO Of Data for tran NO MFs for each nputs NO of partcle for each populaton Epoch for each populaton 5 000 5 00
Proceedngs of the 5th Medterranean Conference on Control & Automaton, July 7-9, 007, Athens - Greece T4-003 (a) (a) (b) (b) MSE=7.000.e-004 MSE=0.0905 (c) (c) MSE=0.97 Fgure : Usng PSO for tranng parameters n ANFIS structure for example. Smulaton results for nonlnear system dentfcaton. (a) (dashed lne), and actual system output (sold lne). (b) MSEs for modfed algorthm.(c) MSEs for basc algorthm Fgure 3: Usng PSO for tranng parameters n ANFIS structure for example.. Smulaton results for nonlnear system dentfcaton. (a) (dashed lne), and actual system output (sold lne). (b) MSEs for modfed algorthm.(c) MSEs for basc algorthm.
Proceedngs of the 5th Medterranean Conference on Control & Automaton, July 7-9, 007, Athens - Greece T4-003 TABLE II: PARAMETERS FOR EXAMPE II NO of nput NO Of Data for tran NO MFs for each nputs NO of partcle for each populaton Epoch for each populaton V. CONCLUSIONS 3 00 0 00 In ths paper, we proposed a novel method for tranng the parameters of ANFIS structure. In our novel method we used PSO for updatng the parameters. The PSO whch we used has some dfferences wth the usual PSO. The smulaton results showed the new approach has better results for complex nonlnear systems than basc PSO. Snce ths algorthm s free of dervaton whch s very dffcult to calculate for tranng of antecedent part parameters complexty of ths new approach s less than other tranng algorthms lke GD and LS. In the other hand the number of computatons requred by each algorthm has shown that PSO requres less to acheve the same error goal as wth the Backpropagaton [9]. Also, the local mnmum problem n GD algorthm for tranng n ths novel approach s solved. The effectveness of the proposed PSO method was shown by applyng t to dentfcaton of nonlnear model. [8] Jyh-Shng Roger Jang, ANFIS: Adaptve-Network- Based Fuzzy Inference System, IEEE Trans. Sys.,Man and Cybernetcs., Vol. 3, No. 3, May/June 993. [9] 3. Yager, R. R. and Zadeh, L. A., Fuzzy Sets Neural Networks, and Soft Computng, Van Nostrand Renhold,994. [0] Mansh Kumar, Devendra P. Garg Intellgent Learnng of Fuzzy Logc Controllers va Neural Network and Genetc Algorthm, Proceedngs of 004 JUSFA 004 Japan USA Symposum on Flexble Automaton Denver, Colorado, July 9-, 004. [] Mascol, F.M., Varaz, G.M. and Martnell, G., Constructve Algorthm for Neuro-Fuzzy Networks, Proceedngs of the Sxth IEEE Internatonal Conference on Fuzzy Systems, 997, Vol., July 997, pp. 459 464. [] Jang, J.-S. R., and Mzutan, E., Levenberg-Marquardt Method for ANFIS Learnng, Bennal Conference of the North Amercan Fuzzy Informaton Processng Socety, June 996, pp. 87 9. [3] Jang, J.-S.R., Input Selecton for ANFIS Learnng, Proceedngs of the Ffth IEEE Internatonal Conference on Fuzzy Systems, Vol., Sep 996, pp. 493 499. [4] J. kennedy, RC. Ebenhart,, partcle swarm optmzaton, Proceedngs of the IEEE Internatonal Conference on neural networks, Vol. 4, 995, pp. 94 948. [5] Andres P.Engelbrecht, Computatonal Intellgence An ntroducton, John Wley & Sons Ltd 00. [6] J. kennedy, the behavor of partcle swarm n VW, N saravan, D Waagen (eds), proceedng of 7 th nternatonal conference on evolutonary programmng, 998, pp58-589. [7] Y Sh, RC Ebenhart, Emprcal study of Partcle Swarm Optmzaton, proceedng of the IEEE Congress on Evolutonary Computaton, Vol3, 999, pp 945-950. [8] K. S. Narendra and K. Parthasarathy, Identfcaton and control of dynamcal system usng neural networks, IEEE Trans. Neural Networks,vol., pp. 4 7, Jan. 990. [9] Venu G. Gudse and Ganesh K. Venayagamoorthy, Comparson of Partcle Swarm Optmzaton and Backpropagaton as Tranng Algorthms for Neural Networks, Swarm Intellgence Symposum, 003. SIS '03. Proceedngs of the 4-6 Aprl 003 IEEE. REFERENCES [] Mannle M, Identfy rule- base TSK fuzzy method, un of Karsruhe, 000. [] Mannle M, Rchard A, and Dorasm T. A, Rule- based fuzzy model for nonlnear system dentfcaton, un of Karsruhe, 996. [3] Jang J. R, Sun C, and Mzutan, Neuro- Fuzzy and soft computng, prentce hall, 997. [4] Sugeno M, and Kang G. T, Structure dentfcaton of fuzzy model, fuzzy sets and systems, pp. 5-33, 998. [5] Takag T, Sugeno M, Fuzzy dentfcaton of systems and ts applcaton to modelng and control, IEEE Transacton on systems, Man & Cybernetcs, pp: 6-3, 985. [6] Alcala R, Casllas J, Cordon O and Herrera F, Learnng TSK rule- based system from approxmate ones by mean of MOGUL methodology Granda un of span, Oct 000. [7] Mannle M, FTSM: Fast Takag- Sugeno fuzzy modelng, un of Karsruhe, 999.