Fuzzy Model Identification Using Support Vector Clustering Method

Transcription

1 Fuzzy Model Identfcaton Usng Support Vector Clusterng Method $\úhj OUçar, Yakup Demr, and Cüneyt * ]HOLú Electrcal and Electroncs Engneerng Department, Engneerng Faculty, ÕUDW Unversty, Elazg, Turkey agulucar@eee.org,ydemr@frat.edu.tr Electrcal and Electroncs Engneerng Department, Dokuz Eylül Unversty, Kaynaklar Campus, ø]plu7xunh\ guzels@eee.deu.edu.tr Abstract. We have observed that the support vector clusterng method proposed by Asa Ben Hur, Davd Horn, Hava T. Segelmann, Vladmr Vapnk, (Journal of Machne Learnng Research, (00, 5 37 can provde cluster boundares of arbtrary shape based on a Gaussan kernel abstanng from explct calculatons n the hgh-dmensonal feature space. Ths allows us to apply the method to the tranng set for buldng a fuzzy model. In ths paper, we suggested a novel method for fuzzy model dentfcaton. The premse parameters of rules of the model are dentfed by the support vector clusterng method whle the consequent ones are tuned by the least squares method. Our model does not employ any addtonal method for parameter optmzaton after the ntal model parameters are generated. It gves also promsng performances n terms of a large number of rules. We compared the effectveness and effcency of our model to the fuzzy neural networks generated by varous nput spacepartton technques and some other networks. Introducton In recent years, fuzzy models have successfully appeared on a lot of applcatons n system dentfcaton, control, predcton and nference. An mportant property of fuzzy models s ther ablty to represent hghly nonlnear systems. In comparson wth other nonlnear black-box modelng technques, fuzzy models have the advantage of gvng nsght nto the relatons between model varables and combnng pror knowledge wth the nformaton dentfed from numercal data. The fuzzy models are accomplshed by structure dentfcaton and parameter adustment procedures. Generally, these models are bult from two learnng phases, the structure learnng phase and the parameter learnng phase. These two phases are usually done sequentally; the structure learnng phase s employed to decde the structure of fuzzy rules frst and then the parameter learnng phase s used to fne tune the coeffcents of each rule obtaned from the frst one. Varous methods have been proposed to solve these problems separately or n a combnatoral way [ 3]. O. Kaynak et al. (Eds.: ICANN/ICONIP 003, LNCS 74, pp. 5 33, 003. Sprnger-Verlag Berln Hedelberg 003

2 6 A. Uçar, Y. Demr, and C. * ]HOLú In ths paper, we deal wth the support vector clusterng method suggested by Asa Ben Hur and co-workers [4] for the structure dentfcaton of fuzzy model. In contrast to some graph theoretc and parametrc clusterng methods [5], a non-parametrc clusterng method based on the support vector approach provdes a set of contours whch enclose the data ponts wthout mposng any assumptons on the cluster number and shape. For the frst tme n ths paper, the method s proposed to partton the sample data nto clusters and smultaneously estmate the parameter, whch defne the best-affne fuzzy models wthout an addtonal parameter dentfcaton cost. The method performs better than other neural and fuzzy neural networks and those generated by partton technques n the lterature. If the tranng set were large, the method would result n a large amount of rules. To elmnate some of these rules, reducng the measure of the tranng set could be consdered as an alternatve soluton. We consdered the technque developed for prunng the Least Squares verson of Support Vector Machnes (LS-SVM n [6] at the problems exhbtng too many rules. Ths technque s based upon smply applcable methods n the neural networks lterature. In these methods, the prunng process s made startng from the huge network and cancelng nterconnecton weghts that are less relevant so as to be obtaned a better generalzaton performance. Smlarly, t s dsmssed data wth the smallest support values havng the least mportant for the constructon of the LS-SVM model. We gradually removed useless samples from the tranng set by the technque, and then appled to ths reduced tranng set the support vector clusterng technque. The paper s organzed nto four sectons. In Secton, the support vector clusterng method s ntroduced. Adaptve Network based Fuzzy Inference System (ANFIS and ts ntalzaton technques are brefly revewed n Secton 3. In Secton 4, smulaton tests n fuzzy modelng are conducted to assess the effectveness of the proposed method and the results are demonstrated n Secton 5. These results are also compared wth the ones obtaned by the Multlayer Perceptron (MLP [7], the Radal Bass Functon (RBF [8], the self-organzed fuzzy rule generaton (SOFRG [9], and LS- SVM [6]. Support Vector Clusterng Gven a data set { } G [ L L = wth [ R, the support vector-clusterng algorthm searches L the smallest enclosng sphere of radus R. The problem can be defned by the followng constrants: ( x a R, ( P PK where Φ( [ M R R nonlnearly maps the nput space to the hgh dmensonal feature space, s the Eucldean norm, and a s the centre of the sphere. To satsfy ( the slack varables s ntroduced such that

3 Fuzzy Model Identfcaton Usng Support Vector Clusterng Method 7 ( x a R + ξ 0 ( and the followng Lagrangan augmented mnmzaton problem s consdered: ( R + ( x a C + L (3 = R a = ( x = C = = are Lagrange multplers, C s a constant, and C where 0 and 0 a penalty term. Applyng the Karush-Kuhn Tucker condtons we obtan 0,. (4 = (5 ( R + ( x a = 0. (6 For > 0 and > 0 the ponts x les outsde the feature space sphere. If 0 > and, the ponts are called as a Bounded Support Vectors 0 = =, C (BSVs. A pont space sphere. For [ wth L = 0 s mapped to the nsde or the surface of the feature 0 < C x les on the surface of the feature boundares <, ( and they are called support vector (SV. As a result, BSVs le outsde the boundares, and all other ponts le nsde them. By constructng the Lagrangan n the Wolfe dual form, the varables R, a, and are elmnated: W ( x ( x. ( x = (7, 0 C, =,..., N. (8 If Mercer s theorem [0] s appled to the kernel matrx of Gaussan wth wdth parameter q, s

4 8 A. Uçar, Y. Demr, and C. * ]HOLú ( x, x exp( q x x ( x. ( x K = =, (9 then the Lagrangan W and the defnton kernel are rewrtten as: R = K ( x, x K( x, x W (0 ( x = ( x, x K( x, x + K( x, x., K. ( The contours that enclose the ponts n data space are defned by the set { 5 ([ 5} [ =., 3 Fuzzy Inference System The fuzzy nference system under consderaton s ANFIS [3]. ANFIS consst of fve layers and the bass functons of each layer are the nput, fuzzfcaton, rule nference, normalzaton, and defuzzfcaton. A detaled descrpton of ANFIS can be found n [3]. An ANFIS archtecture wth two nputs, two rules and one output s shown n Fg.. EINBETTEN Fg. ANFIS archtecture for the two-nput two-rule Sugeno fuzzy model. A typcal fuzzy rule set has the form IF x s A and x s B THEN y =p x +q x +r, IF x s A and x s B THEN y =p x +q x +r, where x and x are the nputs relatng to the node, A and B are Membershp Functons (MFs assocated wth ths node. The parameters of the MFs and the coeffcents of the lnear equatons are called as the premse and the consequent parameters, respectvely. In order to tune the premse parameters, several methods are avalable n the lterature [-3]. The most drect way s to partton the nput space nto grd types wth each grd

5 Fuzzy Model Identfcaton Usng Support Vector Clusterng Method 9 representng a fuzzy f-then rule. The maor problem of such knd of partton s that the number of fuzzy rules ncreases exponentally as the dmenson of the nput space ncreases. Another frequently used method for nput space parttonng s to cluster the nput tranng vectors n the nput space. Such the methods provde a more flexble partton. In ths paper, the grd partton, subtractve clusterng and hyperplane fuzzy clusterng algorthms are consdered n comparson to the proposed method [3]. On the other hand, the consequent parameters are adusted by the least squares method and n order to optmze all the parameters of the ANFIS network, the backpropagaton and the least squares algorthms are used smultaneously [3]. We used the support vector clusterng method to obtan the number, centers, and wdths of the MFs n ths paper. We assgn the obtaned support vectors as the centers of the Gaussan MFs that are radally symmetrc wth kernel wdth /q and also accept ther number as the rule number. Smlar to the ANFIS archtectures ntalzed by the above mentoned methods, we used a least squares technque to fnd the consequent parameters. The only dfference s that our model has a better performance than the other ANFIS archtectures n functon approxmaton wthout requrng any addtonal optmzaton method,.e., wthout the parameter learnng processes. In ths paper, ANFIS-, ANFIS-, and ANFIS-3 ndcate the ANFIS archtectures generated by the grd partton, subtractve and hyperplane fuzzy clusterng methods, respectvely. 4 Smulaton Results To llustrate the valdty of the proposed method, two sets of examples are carred out. Comparng wth the ANFIS archtectures generated by parttonng the nput space s shown n the frst example. In another example, the obtaned results from other neural and fuzzy neural networks are cted from the relevant references so as to be able to compare performance on the same functons. Example In ths example, a D Snc functon whch s a typcal benchmark of functonal approxmaton s examned due to ts hghly varyng characterstc. From evenly dstrbuted grd ponts of the nput space [-0,0] x [-0,0], tranng data pars were obtaned. The performance of the archtecture generated by support vector clusterng method (SVCM was evaluated changng C and q parameters. The accuracy of model was assessed as mean squared error (MSE. The partcular values for C and q parameters were chosen as the results of the smallest MSE search takng nto consderaton the number of both SV and BSV as n [4]. As can be seen from some results gven n Table, the best results were obtaned by small BSV and large SV number. Wthout usng any prunng to reduce the tranng set, good performances were obtaned by approxmately 40 support vectors. When the number of the tranng samples was reduced to 7 by prunng the tranng set, the promsng performances wth respect to the prevous case were exposed by approxmately 6 support vectors. The results obtaned by prunng are lsted n the last sx lne of the Table. On the other hand, all the

6 30 A. Uçar, Y. Demr, and C. * ]HOLú ANFIS archtectures were traned for 300 epochs. Ther performances were evaluated for the number of rules rangng from to 50. As the rule number ncreases, the performances of all ANFISs decrease, however that of the SVCM ncreases. In order to show the generalzaton capablty, other 500 Table. The performance of SVCM n contrast to C and q parameters. C q SV BSV MSE e e e e se e e e e e e e e e e e e-5 checkng data was unformly sampled n the nput space. Fg. llustrates the reconstructed surface of the orgnal functon and the networks. It s clearly seen that the reconstructed surfaces looks smlar to the orgnal functon except at the surfaces around the peaks and at valleys wth hghly varyng nonlnear characterstcs. Table. Comparson of the performances of SVCM archtecture and the ANFIS archtectures. Rule Number MSE SVCM ANFIS ANFIS ANFIS3 6.e-5 4.e-4 5.7e-5 9.4e e-5.7e-3 3.9e-4.8e e-6 5.e-4.e-3.4e- Example SVC procedure s compared wth respect to LS-SVM and the fuzzy neural networks obtaned by usng SOFRG method proposed n [9], where the results of RBF and MLP were also consdered. Therefore three functons consdered n [9] are used.

7 Fuzzy Model Identfcaton Usng Support Vector Clusterng Method 3 Fg.. Fuzzy dentfcaton of a D-Snc functon (a Orgnal functon and reconstructed surfaces wth (b ANFIS- (c ANFIS- (d ANFIS-3, and (e SVCM.

8 3 A. Uçar, Y. Demr, and C. * ]HOLú f f f 3 a ( x,x + ( x, ( x,x ( x,x D = Dbx + e = Dcsn = D d 3. ( 5 x + ( 5 x 0 ( 5 x ( 5 x ( 5 x. + [ [ [0,0] and a small perturbaton ( wth x ntroduced on a specfc regon of the output surface. Here, D s adusted to ensure that the nput and output domans wll be the nterval [0,0]. A tranng set of 400 samples for each functon was generated by a unform spral dstrbuton produced. Model Table 3. Comparson of the dentfcaton performances. MSE Number of Parameters, ( Computatonal Complexty Mean Number Mean f f f 3 f f f 3 of Iteratons tme SVCM C=0.005 q=34.3e-5 4.e s SVCM C=0.039 q= e-4.6e e s SVCM C=0.039 q=7.e-4 9.8e-4 7.7e s SVCM C=0.006 q=3.5e-3 5.e-.6e s SVCM C=0.083 q= 6e-3 5e-6 e s SVCM C=0.083 q=.7e-3 3.7e-3 3.7e s SVCM C=0.083 q=. 3.4e-3.8e-3 3.4e s SOFRG 4.9e- 3.8e-.e s RBF 3.e- 9.8e-.9e- 0-86s MLP.6e-.73e-.e s LS-SVM.e-3 9.3e-5 5e s Fuzzy models determned by the SVCM outperform the other neural and fuzzy networks methods under consderaton n approxmaton accuracy n terms of MSE, as reported n Table 3. Besdes, one can verfy that the computatonal complexty s smlar to that of SOFRG. 5 Conclusons In ths paper, we have proposed a novel automatc desgn method for dentfyng fuzzy models from data. The reason s that the premse parameters are frst dentfed by the

9 Fuzzy Model Identfcaton Usng Support Vector Clusterng Method 33 support vector clusterng method and then the consequent parameters s defned by the least squares method. Perhaps the greatest advantage of the proposed method s that there s no need to any optmzaton technque dfferent than those n lterature after the ntal archtecture s constructed. The generaton of the fuzzy model by applyng support vector clusterng method to the pruned tranng set results n a good performance and less rule. Dfferent prunng technques and alternatvely solutons wth the obectve of reducng the rule number can be expermented. Ths seems as an nterestng pont to be mproved by further study. The valdty of the fuzzy model generated by support vector clusterng method s demonstrated on several tests. The performance of the resultng network s compared wth the best performances obtaned by dfferent neural and fuzzy neural network archtectures and the partton approaches reported n the lterature. The man concluson of ths comparson s that the proposed SCVM provdes hgh modelng accuracy wth a reasonable computatonal complexty. References. Lnkens, D.A., Mn-You, C.: Input Selecton and Partton Valdaton for Fuzzy Modellng Usng Neural Network, Fuzzy Sets and Systems, Vol. 07. ( Mu-Song, C., Shnn-Wen, W.: Fuzzy Clusterng Analyss for Optmzng Fuzzy Membershp Functons, Fuzzy Sets and Systems, Vol. 03. ( Jang, J.S.R., Sun, C.T., Mzutan, E.: Neuro-Fuzzy and Soft Computng: A Computatonal Approach to Learnng and Machne Intellgence, Prentce-Hall Ben-Hur, A., Hor, D., Segelmann, H.T., Vapnk, V.: Support Vector Clusterng, Journal of Machne Learnng Research, Vol.. ( Duda, R.O., Hart, E.P., Stork, D.G.: Pattern Classfcaton, John Wley, New York Suykens, J.A.K., Gestel, T.V., Brabanter, J.D., Moor, D.B., Vandewalle J.: Least Squares Support Vector Machne, World Scentfc Rumelhart, D.E., Hnton, G.E., Wllams, R.J.: Learnng Internal Representatons by Error Propagaton, Parallel Data Processng, Cambrdge, MA: MIT Press, Vol.. ( Bors, A.G., Ptas, I.: Medan Radal Bass Functon Neural Network, IEEE Trans. Neural Networks, Vol. 7. ( Ignaco, R., Hector, P., Julo, O., Alberto, P.: Self-Organzed Fuzzy System Generaton from Tranng Examples, IEEE Trans. Fuzzy Systems, Vol. 8. ( Crstann, N., Shawe-Taylor, J.: An Introducton to Support Vector Machnes, Cambrdge Unversty Press 000