A METHOD OF GENERATING RULES FOR A KERNEL FUZZY CLASSIFIER

Transcription

1 Proceedigs of the Sixth Iteratioal Coferece o Machie Learig ad Cyberetics, Hog Kog, 9- August 7 A METHOD OF GENERATING RULES FOR A KERNEL FUZZY CLASSIFIER AI-MIN YANG,, XIN-GUANG LI, LING-MIN JIANG,YONG-MEI ZHOU, QIAN-QIAN LI 3 School of Iformatics, Guagdog Uiversity of Foreig Studies, Guagzhou 54, Chia School of Computer, Natioal Uiversity of Defese Techology, Chagsha 473, Chia 3 School of Computer ad Commuicatio, Hua Uiversity of Techology, Zhuzhou Hua, 48, Chia amyag8@63.com Abstract: A method of geeratig rules for a kerel fuzzy classifier is itroduced. For this method, firstly, the iitial sample space is mapped ito a high dimesioal feature space by selectig the appropriate kerel fuctio. The i the feature space, the proposed dyamic clusterig algorithm dyamically separates the traiig samples ito differet clusters ad fids out the support vectors of each cluster. For each cluster, a fuzzy rule is defied with ellipsoidal regios. Fially, the rules are adjusted by Geetic Algorithms. This classifier with such fuzzy rules is evaluated by two typical data sets. For this classifier, the learig time is short, the classificatio is better ad the speed of classificatio is quick. Keywords: Fuzzy classificatio rule; Kerel fuctio; Dyamic clusterig; Geetic algorithms. Itroductio Fuzzy classificatio is a importat applicatio of fuzzy set theory. Fuzzy classificatio rules [-6] are widely cosidered a well-suited represetatio of classificatio kowledge, with readability ad iterpretatio. Fuzzy classificatio has bee widely applied i may fields, such as, image processig, words recogitio, voice recogitio, text classificatio, remote sesig etc. But there are some problems i geeratig ad optimizig fuzzy classificatio rules, ad i how to select membership fuctio, that is, it is difficult to gai fuzzy rules with good performace. So, there are may researchers o studyig these problems ad developig variety of systems, which were based o eural etwork [], geetic algorithm [-3,6], clusterig [4-5] ad so o. But fuzzy classificatio systems based o eural etwork ad geetic algorithm have log traiig time ad too may rules, whe the dimesios of iput samples or the umber of class is may. I the system based o clusterig, it is difficult to optimize the rules. Especially, whe the distributio of samples is as Figure (a, the geeral clusterig method has poor classificatio. x x (a z z 3 (b Figure. A distributio of samples I order to trai classifiers fast ad obtai well geeralizatio ability as eural etwork classifier, we propose a method of geeratig ad optimizig fuzzy rules ad costruct a ew fuzzy classifier with these rules, called kerel fuzzy classifier. For this method, firstly, the iitial sample space is mapped ito a high dimesioal feature space by selectig appropriate kerel fuctio, as Figure,(x,x (z,z,z 3, where, z =x, z = / x x, z 3 =x, ad the dimesio of sample space becomes from two to three. The i the feature space, the proposed dyamic clusterig arithmetic dyamically separates the traiig samples ito differet clusters ad fids out the support vectors of each cluster. For each cluster, a fuzzy rule is defied with ellipsoidal regios. Fially, the rules are adjusted by Geetic Algorithms. This paper orgaizes as follows. I sectio, the basic form of kerel fuzzy classificatio rules is itroduced. Sectio 3 presets the method of computig support vectors. Sectio 4 describes the proposed dyamic clusterig algorithm. I sectio 5,the method of adjustig rule is itroduced. Sectio 6 is performace evaluatio for proposed classifier. The last is coclusio ad future work.. Basic form of kerel fuzzy classificatio rules Let R q deote the sample space (q is dimesio of sample space, Φ: R q F, x Φ(x X, F is a higher dimesio space relative to R q, called feature space. Φ is a map fuctio. x is the sample i iitial space, Φ(x is the z X/7/$5. 7 IEEE 695

2 Proceedigs of the Sixth Iteratioal Coferece o Machie Learig ad Cyberetics, Hog Kog, 9- August 7 samples i feature space ad F. Let K(, deote kerel fuctio[]. Its defiitio is as Eq.(. K(x i,x j =Φ(x i.φ(x j ( Where,. is the ier product. I this paper, the kerel fuctio of as Eq.( is used. K(x i,x j =exp(-γ x i -x j (γ ( Fuzzy rule based o ellipsoidal regios ca be described as follows. Let x (x R q ad Φ(x(Φ(x F be a iput sample i the iitial space ad i feature space respectively. The class umber of the classified patters is. C i (i=,, deotes the i-th class. Each class ca be clustered with the icrease of traiig samples, C (j=, deotes the j-th cluster of the i-th class. Whe the samples umber of a cluster icreases to a give value, a ew fuzzy classificatio rule is geerated ad the expressio is as (3. R : IF Φ( x is aroud C THEN x Ci (3 with CF = α Where, R is the label of rule ad geerated from the j-th cluster of i-th classificatio, CF represets the degree of Φ(x belogig to this rule, α [,] is described as Eq(8. Let (cc R q deote the ceter of the j-th cluster of i-th classificatio. mf (Φ(x deotes the membership fuctio for the iput sample. u (Φ(x is the degree of membership fuctio. The followig equatios are defied. mf ( Φ (x = exp( h ( Φ (x (4 d ( Φ(x h ( Φ (x = (5 δ d ( Φ (x = Φ(x = K(x,x K(x,cc + K(cc,cc (6 u ( Φ (x = mf ( Φ (x (7 α = u ( Φ (x (8 Where, the membership fuctio mf (Φ(x is a typical expoetial fuctio like Gauss fuctio. h (Φ(x is called adjustig distace. d (Φ(x is the Euclidea distace betwee Φ(x ad the ceter of C cluster. is determied with our proposed dyamic cluster algorithm i setio 4. δ is a adjustig parameter. Eq(6 is calculated accordig to Eq(4 (i selectio 3 after created rules. 3. The method of fidig support vectors Suppose C is a cluster with samples(,,,. The selected mappig fuctio Φ ad kerel fuctio satisfy Eq.( ad Eq.((i selectio. There are the followig questios to be solved.(how to compute the miimal radius Ru of C. ( how to compute the distace d,l betwee Φ(x L (L s ad the ceter of C. If sample Φ(x s belogs to cluster C, iequatio (9 holds. Φ(x s - Ru (9 Where,. is the stadard Euclidea distace, is the ceter of C. We calculate the miimal value of Ru i Eq(9 by usig Lagragia equatio (as Eq(. s s L= Ru (R Φ(x β ( Where, β s (β s is the lagragia multiplier. To get the miimal value of L, Ru ad have to satisfy the costrait coditio Eq.( ad Eq.(. β = s s s ( Φ (cc = β Φ(x ( The variables of Ru ad i Eq( ca be caceled usig the above equatios. Eq.( is traslated to the Wolfe dual[8] problem as Eq.(3. ( β β β (. ( x s s s s' s s' W = Φ x Φ x Φ, s' = = K(x, x β β β K(x, x s s s s s' s s', s' = (3 Where, W is the fuctio about β s, its max value ca be got usig quadratic programmig as well as β s of each sample Φ(x s, ad these values make Eq( gettig miimal value. The set of samples Φ(x s (,,, is separated to two types accordig to the value of β s. Oe with β s = is i the ier of the sphere with radius Ru (some are outside of the sphere, while the other with β s > is o the border of the sphere, called the support vectors [7-8]. _sv is the umber of support vectors of cluster C, _sv. So, i feature space, the distace betwee ay sample Φ(x L ad the ceter of C ca be calculated usig Eq.(4,ad Ru of C ca be calculated accordig to Eq.(5. 696

3 Proceedigs of the Sixth Iteratioal Coferece o Machie Learig ad Cyberetics, Hog Kog, 9- August 7 d (x = Φ(x,L L L _ sv s,s' _ sv =K(x x β K(x,x + + _ sv L, L s s L ββ K(x,x s s' s s' _ sv = k k, k βs s k R (x K(x x K(x,x s,s' ββ K(x,x s s' s s' (4 (5 Where, x k is oe support vector of cluster C. So,the distace betwee sample ad the ceter of cluster ad the radius of the cluster are calculated accordig to Eq.(4 ad Eq.(5. 4. Dyamic clusterig algorithm We put forward a dyamic clusterig algorithm to fast geerate fuzzy classificatio rules. I feature space F, this algorithm ca separate the traiig samples ito differet clusters by scaig the traiig samples oce.let C k represet C which has bee modified for k times (k=,,,. k is the ceter of the cluster C which has bee modified for k times. Ru k is Ru which has bee modified for k times. d,l is the distace betwee iput sample x L ad the ceter of C. dth i is the radius threshold for each cluster of C i. Dyamic clusterig algorithm: Iput: traiig samples (x L, C i (L=,,,m, i=,,,,the selected kerel fuctio which satisfies Eq.( ad Eq.(, dth i (i=,,,. Output: the radius of each cluster, support vectors. Algorithm: Step : For each iput traiig sample, judge the class, which this sample belogs to (do clusterig for the samples i the same class. Step : If a sample is the first traiig sample of C i (i=,,, C i is created ad this sample becomes the cluster ceter of C i, ad the followig symbols are give,c i, i,ru i, ad Ru i =,else,go to step 3. Step 3: If traiig sample Φ(x L belogs the C i while it is ot the first oe, calculate the distaces d,l = Φ(x L - (calculatig accordig to Eq(4,which is betwee Φ(x L ad created cluster ceters of C i,j=,,c i.here, c i is the umber of created clusters of C i. Step 4: Compare the d,l which is calculated i Step 3 with the correspodig cluster radius Ru.If there is the cluster C ig ( g c i which satisfies Φ(x L - ig =mi( Φ(x L -,ad d,l Ru (j=,,..,c i, Φ(x L is assiged ito C ig,the go to Step,else, go to Step 5. Step 5: Fid the cluster C ia which is earest to Φ(x L amog the created clusters of C i. Firstly,calculate S,L,S,L =d,l +Ru (j=,,c i,the choose C ia accordig to S,L ad Eq.(6. S = d + Ru = mi{ S } ia,l ia,l i a,l (6 ( j =,,, c Step 6: If S ia,l > dth i,it idicates that Φ(x L does t belog to ay created clusters, the a ew cluster should be created, the creatig method ad process are similar to those of Step,the go o Step,else, go to Step 7. Step 7: If S ia,l dth i,the ceter(cc ia ad radius(ru ia of C ia should be updated by the method i sectio 3,ad the support vectors refreshed also. If all the iput traiig samples have bee iput, tur to Step 8, else, tur to Step. Step 8: For every cluster, output the radius ad support vectors. Φ( x C Φ( x C Ru = (a Ru = Φ( cc k C Ru k Sample Sample k Cluster ceter Cluster Cluster radius (c i Φ( x 5 Φ( x Ru = Ru = Φ( x 4 Φ( x C C Φ( x 5 C Φ( x Ru = Φ( x C Φ( x 4 Φ( x 3 Φ( x 6 Ru C Ru (b Φ( x 7 C Ru Φ( x 3 Figure. Illustratio of dyamic clusterig Figure shows the process of the dyamic clusterig algorithm. I Figure (a, the traiig samples Φ(x ad Φ(x belog to two differet classes, ad create clusters C ad C respectively. I Figure (b, sample Φ(x 3 697

4 Proceedigs of the Sixth Iteratioal Coferece o Machie Learig ad Cyberetics, Hog Kog, 9- August 7 satisfies the coditio i Step 7,so the ceter ad radius should be modified,cc ad Ru are geerated, sample Φ(x 4 satisfies the coditio i Step 4, the cluster ceter ad radius eed ot to be modified, sample Φ(x 5 satisfies the coditio i Step 6, ad cluster C is created. I Figure (c, Φ(x 6 ad Φ(x 7 are similar to the sample Φ(x 3. Fuzzy classificatio rules are geerated durig the process of dyamic clusterig, each cluster geerates a rule as expressio(3 whe the umber of traiig samples for this cluster is over set value (default is q.rule R is determied by followig parameters: the support vectors, Lagrage Multiplier β s, ad the adjustig parameter δ. After the creatio of rules, rules should be adjusted because the overlappig of rule regios is ucosidered whe rules were geerated. This is discussed i sectio 5. The samples ca be classified after the rules adjusted. CF ca be calculated correspodig to each iput sample Φ(x.If the value α kj of the rule kj is the most, x ca be classified to the k-th class. 5. Method of adjustig rule by geetic algorithms Because the overlappig of rule regios is ucosidered durig the geeratio of rules, these rules should be adjusted. I this selectio, we propose a method of adjustig rule by Geetic Algorithms. A rule is maily determied by the support vectors, β s, ad δ. If all of these parameters are adjusted, it will cosume much time [6]. Here, we oly adjust the value of δ by Geetic Algorithm to get the maximal recogitio rate [9]. Accordig to Eq.(6,whe the value of δ icreases, the degree of membership fuctio will icrease, vice versa. 5.. Gee codig Geetic iformatio of fuzzy rule R is δ ad the firig iformatio.a chromosome (see Figure 3(a cosists of δ ad firig iformatio. Each traiig sample is ormalized to [ ],ad <δ.δ is expressed by a 8-bit biary strig ad firig iformatio is expressed by a bit. For firig iformatio, is fired for rule ad is ot fired. Figure 3(b shows the compositio of a idividual. Figure 4 shows the compositio of a populatio. I p (p=,,,p deotes a idividual, P deotes the size of this populatio. chromosome Firig iformatio bit δ (a idividual R R (j- (b R j Figure 3. Compoet of chromosome ad its idividual I I I p I P R 5.. Fitess fuctio Populatio R R (j- R R R (j- R R R (j- R R R (j- R j R j R j R j Figure 4. Compoet of a populatio For each idividual, the fitess fuctio is Eq.(9. The recogitio rate of idividual I p for traiig samples ca be calculated accordig to Eq.(7. I Eq.(7,Num all is the total umber of traiig samples ad Num cop is the umber of traiig samples which have bee classified correctly. Eq(8 is the average classificatio of rule R.I (8,Φ(x L is the traiig samples which have bee classified correctly, N corr is the umber of traiig samples which have bee classified corrected by rule R ad N corr is the umber of traiig samples which have bee classified icorrectly by rule R. Eq.(9 is the fitess fuctio of idividual I p. I Eq.(9, is the umber of class ad N R_all is the total umber of created rules. Numcop Accuracy = (7 Num ACF R = Ncorr L= N corr all u( Φ(x + N L corr (8 ACF R i = j= Fit = Accuracy (9 p N R_all 698

5 Proceedigs of the Sixth Iteratioal Coferece o Machie Learig ad Cyberetics, Hog Kog, 9- August Determiatio of firig iformatio I formula (, ACC (R is the correctio rate of classificatio of rule R for traiig samples, Trum corr (R is the umber of traiig samples which are classified correctly by rule R, Trum all (R is the total umber of the traiig samples classified by rule R,ad k is geeral i the rage from.6 to.9. I each geeratio, if formula ( is satisfied, rule R will be fired. Tr um ( R corr ACC( R = > k ( Tr um ( R Each rule maitais a cout for ufired iformatio. fc(r is the cout for R. NFC is a threshold of this cout. Durig the evolvig procedure of the rules, if the rule is ot fired, fc(r will be icreased by. As soo as fc(r =NFC, rule R will be deleted, ad these traiig samples for R are uited with the rule which is the earest to deleted rule R. This strategy ca delete some rules ad make the umber of rules more reasoable Geetic operatio The procedure of geetic operatio is as follows. Step : Iitial populatio is geerated. Geerally, this process is carried out radomly ad all firig bits of rules are. Step : Evaluate the fitess of each idividual I p i the populatio as Eq.(9. Step 3:Create a ew populatio by repeatig followig steps util a populatio size is reached P. Selectio Operatio: Select two paret chromosomes from a populatio accordig to their fitess by usig the roulette wheel method. Crossover Operatio: A sigle poit crossover(firig bit is excluded method is coducted to form a ew offsprig. Mutatio Operatio: With a mutatio rate, mutate ew offer sprig at each locus (positio i chromosome, firig bit excluded. Place ew offsprig i a ew populatio. Step 4 : Geetic algorithm is doe with ewly geerated populatios. If the maximum umber of geeratios has bee reached, Geetic algorithm is stopped. Otherwise, retur to to Step. 6. Performace evaluatio Two typical data sets are used to valuate the proposed method. The results are compared with methods of eural etwork ad referece [6]. all 6.. The recogitio experimet with wie data set Wie data [] iclude 78 datum with 3 iput attributes ad 3 classes.9 datum are used as traiig samples while 88 datum are used as testig samples. Parameters of experimet are as follows. dth =dth =dth 3 =..The miimum umber of traiig samples for creatig a rule is 6. The maximum umber of rules for a class is 8.Iitial value of δ is,p=3,geeratio=3.mutatio probability is..others, γ=.,k=.75,nfc=5.table is the experimetal result of the proposed method. Table is compariso of for proposed method ad referece [6] method. Table. The experimetal result of the proposed method Geeratios Time(sec. Accuracy(% Table. Compariso of for wie data Average Maximum Average Maximum The average umber of rules The umber of evolutio geeratios Traiig samples(9 Test samples(88 PM(% RM(% PM: Proposed method RM:Referece [6] method 6.. Experimet of hadwritte digits data[] This data set icluded 7494 traiig samples ad 3498 test traiig samples with 6 iput attributes ad classes. Table 3 is compariso of ad traiig time for proposed method ad eural etwork method. Parameters of the experimet are as follows. dth = =dth =.5.The miimum umber of traiig samples for creatig a rule is 3.The maximum umber of rules for each class is 699

6 Proceedigs of the Sixth Iteratioal Coferece o Machie Learig ad Cyberetics, Hog Kog, 9- August 7 3.Iitial value of δ is,p=6,geeratio=3, γ=.. Mutatio rate is.,k=.78,nfc=3.the eural etwork has a 3-layer structure ad hidde layer uits. Other parameters are as follows: iitial weight is distributed i [-.,.], Differet iitial values are used to trai for 5 times. epoch is.the learig speed is.8.the mometum is. Table 3. Compariso for Hadwritte Digits data[] Classifier ystem CPU time (s Rule umbers Accuracy of traii -g sample Accuracy of testig sample PM % 9.9% NNM 35.6 (uits 93.9% 9.% PM: Proposed method NNM: Neural Network method These results ad compariso show that the of our proposed classifier are comparable to the maximum of the multilayered eural etwork classifier ad referece [6], the traiig time is much shorter. 7. Coclusio ad future work This paper itroduces a method of geeratig rules with kerel fuzzy classifier. Firstly, the iitial sample space is mapped ito a high dimesioal feature space by selectig appropriate kerel fuctio. The i the feature space, our proposed dyamic clusterig algorithm dyamically separates the traiig samples ito differet clusters ad fids out the support vectors of each cluster. Because dyamic clusterig arithmetic is able to separate traiig data ito clusters by oe-pass, ad Geetic Algorithm oly adjusts δ for each fuzzy classificatio rule, the process is very quick. The of the costructed classifier by our proposed method are comparable to the maximum of the multilayered eural etwork classifier ad referece [6],ad the traiig time is much shorter. I the future, we will discuss the self-adaptive determiatio of cluster radius threshold i the dyamic cluster algorithm, ad select other kerel fuctios ad membership fuctios to do some compariso. Refereces [] F.Uebele,S.Abe,ad M.-S.La, A Neural Network- Based Fuzzy Classifier,IEEE Tras. Systems,Ma, ad Cyberetics, Vol. 5, No., pp , 999. [] K.Shirojima,T.FuKuda,Y.Hasegawa, RBF-Fuzzy System with GA Based Usupervised/Supervised Learig Method,Proc. 4th IEEE It. Cof. Fuzzy Syst., pp53-58,995. [3] Hiroyuki INOUE, Automatic Geeratio of Fuzzy Rules Usig Hyper Elliptic coe Membership Fuctios by Geetic Algorithms,J. Itelliget ad Fuzzy systems, vol.6,o.,pp.65-8,998. [4] M.Ryoke, H.Tarura ad Y.Nakarori, Fuzzy Rule Geeratio by Hyperellipsoidal Clusterig,Methodologies for the Coceptio,Desig ad Applicatio of Itelliget Syst.,pp.86-89,World Scietific,996. [5] S.Abe, Dyamic Cluster Geeratio for a Fuzzy Classifier with Ellipsoidal Regios,IEEE Tras. Systems,Ma, ad Cyberetics-Part B, Vol. 8, No. 6, pp , 998. [6] Hg, L.,Ioue, H., Automatic geeratio of fuzzy classificatio systems usig hyper-coemembership fuctio,computatioal Itelligece i Robotics ad Automatio, 3. Proceedigs. 3 IEEE Iteratioal Symposium o, Volume:, pp ,July, 3. [7] Chu-Fu Li, Sheg-De Wag, Fuzzy support vector machies,neural Networks,IEEE Trasactios o, Volume: 3,Issue:, pp ,March,. [8] Be-Hur,A,Hor,D..Siegelma,H.T., Vapik, V., A support vector clusterig method,patter Recogitio,. Proceedigs. 5th Iteratioal Coferece o, Volume:,3-7 pp vol., Sept. [9] Kaieda,K,Abe,S., A kerel fuzzy classifier with ellipsoidal regios,neural Networks, 3. Proceedigs of the Iteratioal Joit Coferece o, Volume: 3, pp.43 48, July, 3. [] [UCI Repository of Machie Learig Databases Network Documet]. ftp://ftp.ics.uci.edu/pub/machie -learig-databases/. Ackowledgemets This work was supported by the Natioal Natural Sciece Foudatio of Chia (63736 ad the Natural Sciece Foudatio of Hua Provice of Chia (5JJ4. 7