Design of PSO-based Fuzzy Classification Systems

Transcription

1 Tamkang Journal of Science and Engineering, Vol. 9, No 1, pp (006) 63 Design of PSO-based Fuzzy Classification Systems Cia-Cong Cen Department of Electronics Engineering, Wufeng Institute of Tecnology, Ciayi, Taiwan 61, R.O.C. Abstract In tis paper, a metod based on te particle swarm optimization (PSO) is proposed for pattern classification to select a fuzzy classification system wit an appropriate number of fuzzy rules so tat te number of incorrectly classified patterns is minimized. In te PSO-based metod, eac individual in te population is considered to automatically generate a fuzzy classification system for an M-class classification problem. Subsequently, a fitness function is defined to guide te searc procedure to select an appropriate fuzzy classification system suc tat te number of fuzzy rules and te number of incorrectly classified patterns are simultaneously minimized. Finally, two classification problems are utilized to illustrate te effectiveness of te proposed PSO-based approac. Key Words: Particle Swarm Optimization, Fuzzy Classification System, Pattern Classification 1. Introduction *Corresponding autor. cccen@mail.wfc.edu.tw Fuzzy rule-based systems ave been successfully applied to solve many classification problems. In many classification problems, fuzzy classification rules are derived from uman experts as linguistic knowledge. Because it is not usually easy to derive fuzzy rules from uman experts, many approaces ave recently been proposed to generate fuzzy rules automatically from te training patterns of te considered classification problem [35,7,8,13]. In order to generate fuzzy rules from training patterns, fuzzy partition in te input space are generally considered to determine te premise part of a fuzzy classification system. Te grid-type fuzzy partition of te input space [3,4,7,9,11,13] and te scatter-type fuzzy partition of te input data [8,10] ave been often used to model fuzzy systems for training patterns. In [3], a euristic metod for generating fuzzy rules is applied to te grid-type fuzzy partitions, and a rule selection metod, based on genetic algoritms, is ten employed to select te relevant fuzzy rules from te generated fuzzy rules for classifying te training patterns in te considered classification problem. However, tis approac to solving classification problems wit ig-dimensional pattern spaces as a significant sortcoming: te number of fuzzy rules becomes enormous as te number of dimensions increases and te learning time for genetic algoritms is too ig. Furtermore, an adaptive grid partition in te input space is used to design te ANFIS-based fuzzy classifier in [4]. Tis approac takes te uniformly partitioned grid as te initial state. Te grid evolves as te parameters in te premise membersip functions are adjusted. However, te adaptive grid-partition sceme as two problems. First, te number of fuzzy sets for eac input variable is predetermined in advance. Second, te learning complexity suffers from an exponential explosion as te number of inputs increases. In te scatter partition for constructing fuzzy classifiers, fuzzy minmax yperbox classifiers are powerful tools for solving classification problems. Simpson [8] proposed a metod for generating te yperbox regions to construct a fuzzy classifier to te considered classification problem. In tis approac to te classification problem, te learning parameter is very critical, since it directly affects te number and position of te resulting yperboxes. Consequently, it influences te structure of te fuzzy classifier and te classification performance so tat tis approac suffers from a ig sensitivity of te classification accu-

2 64 Cia-Cong Cen racy wit respect to te skill of te user to determine te appropriate learning parameter. Besides, te fixed learning parameter of te above approac to fuzzy min-max yperbox classifier makes te same constraint on coverage resolution in te wole input space so tat some small yperboxes are generated even in te input space far from class boundaries. Tis reduces te generalization capability of te fuzzy min-max yperbox classifier. In tis paper, a metod based on te particle swarm optimization (PSO), called a PSO-based metod, is implemented to select an appropriate fuzzy classification system suc tat a low number of training patterns are misclassified by te selected fuzzy classification system. In te PSO-based approac, eac individual in te population is considered to represent a fuzzy classification system. Ten, a fitness function is implemented to guide te searc procedure to select an appropriate fuzzy classification system suc tat te number of fuzzy rules and te number of incorrectly classified patterns are simultaneously minimized. Te rest of tis paper is organized as follows. Section describes te structure of te fuzzy classification system. Section 3 proposes a PSO-based metod to select an appropriate fuzzy classification system for te considered classification problem. Section 4 considers classification problems of a syntetic data set and te Iris data set to illustrate te learning ability and te generalization ability of te proposed approac, respectively. Finally, Section 5 draws conclusions about te proposed approac to solving te classification problem.. Structure of Fuzzy Classification Systems Wen an M-class classification problem is considered, a rule base of a fuzzy classification system can be expressed as follows [3]: j t rule: If x1 is Aj1 and x is Aj and and xm is Ajm, Ten x ( x1, x,, xm) belongs to Class H j wit CF CFj, j 1,,, R, (1) were R is te number of fuzzy rules in te rule base, A ji, i = 1,,,m, are te premise fuzzy sets of te j-t fuzzy rule, H j {1,,,M}, is te consequent class output of te j-t fuzzy rule, and CF j [0,1] is te grade of certainty of te j-t fuzzy rule. In tis paper, te membersip function of te fuzzy set A ji is described by ( m, m, m ; x ) Aji ( ji,1) ( ji,) ( ji,3) i xi m ( ji,1) exp, if xi m( ji,1), m ( ji,) xi m ( ji,1) exp m ( ji,3), if x m, ( ji,1) () i1 were m (ji,1), m (ji,) and m (ji,3) determine te center position, te left and rigt widt values of te membersip function, respectively. Hence, te sape of te membersip function is determined by a parameter vector m ji = [m (ji,1) m (ji,) m (ji,3) ]. Te j-t fuzzy rule in te rule base is determined by a parameter vector r j =[m j1 m j mjm ]. Consequently, te set of parameters in te premise part of te rule base is defined as r =[r 1 r rr ]. According to (1), te set of parameters in te consequent part of te rule base is defined as a =[H 1 CF 1 H CF HR CF R ]. Wen te input x =(x 1, x,,x m ) is given, te firing strengt of te premise of te j-t rule is calculated by M q j (x)= Aji (x i ). Te class output of te fuzzy classification system wit respect to te input x can be determined by R y arg max q ( x) CF j1 j (3) According to te above description, eac parameter set consisting of te premise and consequent parameters determines a fuzzy classification system. Tus, different parameter sets determine different fuzzy classification systems so tat te generated fuzzy classification systems ave different performances. Te goal of tis paper is to find an appropriate fuzzy classification system to minimize bot of te number of incorrectly classified patterns and te number of fuzzy rules. In te next section, te PSO-based metod is applied to select an appropriate fuzzy classification system wit a low number of fuzzy rules suc tat te number of incorrectly classified patterns is minimized. 3. PSO-Based Fuzzy Classification Systems j Te PSO is an evolutionary computation tecnique i

3 Design of PSO-based Fuzzy Classification Systems 65 proposed by Kennedy and Eberart [,6]. Its development was based on observations of te social beavior of animals suc as bird flocking, fis scooling, and swarm teory. Like te GA, te PSO is initialized wit a population of random solutions. It also requires only te information about te fitness values of te individuals in te population. Tis differs from many optimization metods requiring te derivation information or te complete knowledge of te problem structure and parameter. Compared wit te GA, te PSO as memory so tat te information of good solutions is retained by all individuals. Furtermore, it as constructive cooperation between individuals, individuals in te population sare information between tem. In te PSO-based metod, eac individual is represented to determine a fuzzy classification system. Te individual is used to partition te input space so tat te rule number and te premise part of te generated fuzzy classification system are determined. Subsequently, te consequent parameters of te corresponding fuzzy system are obtained by te premise fuzzy sets of te generated fuzzy classification system. A set of L individuals, P, called population, is expressed in te following: p r 1 1 g 1 p r g P p r g p r L L g L (4) In order to evolutionarily determine te parameters of te fuzzy classification system, te individual p contains two parameter vectors: r and g. Tat is, p =[r g ]. Te parameter vector r =[r 1 r rj rb ] consists of te premise parameters of te candidate fuzzy rules, were B is a user-defined positive integer to decide te maximum number of fuzzy rules in te rule base generated by te individual p. Here, r j =[m j1 m j mji m jm ] is te parameter vector to determine te membersip functions of te j-t fuzzy rule, were m ji =[m (ji,1) m (ji,) m (ji,3)] is te parameter vector to determine te membersip function for te i-t input variable. Te parameter vector g =[g 1 g gj gb ] is used to select te fuzzy rules from te candidate rules r =[r 1 r rj rb ] so tat te fuzzy rule base is generated. g j [0,1] decides weter te j-t candidate rule r j is added to te rule base of te generated fuzzy system or not. If g j 0.5, ten te j-t candidate rule r j is added to te rule base. Consequently, te total number of g j (j = 1,,,B) wose value is greater tan or equal to 0.5 is te number of fuzzy rules in te generated rule base. In order to generate te rule base, te index j of g j (j = 1,,,B) wose value is greater tan or equal to 0.5 is defined as I r {1,,,B}, r = 1,,,r, were r represents te number of te fuzzy rules in te generated rule base. { r, I r 1 I,, r,, I r } generates te premise part of te fuzzy r Ir rule base generated by te individual p =[r g ]. For example, assume tat r and g are denoted as [r 1 r r 3 r 4 r 5 r 6 ] and [ ], respectively. According to g, te generated rule base as tree fuzzy rules and {I 1, I, I 3 } = {,3,6} so tat {r r 3 r 6 } determines te premise part of te generated rule base. Consequently, te rule base of te generated fuzzy classification system is described as follows: r t rule : If x1 is A and x 1 is A and and x, Ir Ir m is AIrm Ten x ( x1, x,, xm) belongs to Class Hr wit CF CFr, r 1,,, r, (5) were A i = 1,, I,,m, are te fuzzy sets of te generated r-t fuzzy rule. Te membersip function associ- r i ated wit te fuzzy set is described as follows: A I r i ( m, m, m ; x ) A ( Iri,1) ( Iri,) ( Iri,3) i Ir i xi m ( Ir i,1) exp if xi m ( Ir i,1) m ( Ir i,),, xi m ( Ir i,1) exp, if xi m, ( Ir i,1) m ( Ir i,3) (6) Consequently, te individual p determines te premise part of te generated fuzzy classification system. Assume tat N training patterns (x n, y n ), n = 1,,,N, are gatered from te observation of te considered M-class classification problem, were x n =(x n1, x n,, x nm ) is te input vector of te n-t training pattern

4 66 Cia-Cong Cen and y n {1,,,M} is te corresponding class output. In order to determine te consequent parameters H r and CF r of te r-t fuzzy rule, a procedure is described as follows [3]: Step 1. Calculate t, t=1,,,m for te r-t fuzzy rule as follows: Step. Determine H r for te r-t fuzzy rule by (7) (8) Step 3. Determine te grade of certainty CF r of te r-t fuzzy rule by were t r p xpclasst H r CF r q ( x ), t 1,,, M. M arg max. M t1 thr H r M t1 t1 t t. M 1 t (9) (10) Tus, te consequent parameters of te generated fuzzy classification system are determined by te above procedure. According to te above description, eac individual corresponds to a fuzzy classification system. In order to construct a fuzzy classification system wic as an appropriate number of fuzzy rules and minimize incorrectly classified patterns simultaneously, te fitness function is defined as follows: f fit( p ) g ( p ) g ( p ) 1 (11) were f is te fitness value of te individual p, g 1 (p ) and g (p ) are defined respectively as follows: NICP( p ) g1( p ) exp e and ( ) exp r g. p r (1) (13) Here, NICP(p ) is te number of incorrectly classified patterns, r is te number of fuzzy rules in te rule base of te generated fuzzy classification system, and e and r are user-defined constants for te fitness function. Consequently, te fitness function is designed to deal wit te tradeoff between te number of incorrectly classified patterns and te number of fuzzy rules. In tis way, as te fitness function value increases as muc as possible based on te guidance of te proposed fitness function, te fuzzy classification system corresponding to te individual will satisfy te desired objective as well as possible. Tat is, te selected fuzzy system as a low number of rules and a low number of incorrectly classified patterns simultaneously. Subsequently, a PSO-based metod is proposed to find an appropriate individual so tat te corresponding fuzzy classification system as te desired performance. Te procedure is described as follows: Step 1. Initialize te PSO-based metod. (a) Set te number of individuals (L), te maximum number of rules (B), te number of generations (K), te constants for te fitness function ( e and r ) and te constants for te PSO algoritm (, d 1,d,c 1, and c ). (b) Generate randomly initial population P. Eac individual of te population is expressed as follows: p =[r g ], (14) were r = [m (11,1) m (11,) m (11,3) m (1m,1) m (1m,) m (1m,3) m (B1,1) m (B1,) m (B1,3) m (Bm,1) m (Bm,) m (Bm,3)] and g =[g 1 g gb ]. m (ji,k), j {1,,,B}, i {1,,,M}, k {1,,3}, is randomly generated as follows: m m ( m m ) rand(), min max min ( ji, k ) ( ji, k ) ( ji, k ) ( ji, k ) (15) were te range of te parameter m (ji,k) is defined min max as [ m( ji, k ), m( ji, k )] and rand() is a uniformly distributed random numbers in [0,1]. g j [0,1], j {1,,,B}, is randomly generated as follows: g j = rand(). (16) (c) Generate randomly initial velocity vectors, = 1,,,L. Eac velocity vector is expressed as follows:

5 Design of PSO-based Fuzzy Classification Systems 67 =[ ], (17) p p v, 1,,, L. (1) were = [ (11,1) (11,) (11,3) (1m,1) (1m,) (1m,3) (B1,1) (B1,) (B1,3) (Bm,1) (Bm,) (Bm,3)] and = [ 1 B ]. (ji,k), j {1,,,B}, i {1,,,M}, k {1,,3}, is randomly generated as follows: m m max min ( ji, k ) ( ji, k ) ( ji, k ) rand(). 0 (18) j,j{1,,,b} is randomly generated as follows: j rand(). 0 (19) Step. Calculate te fitness value of eac individual and set initial p, f for eac individual and initial p best, f best for te initial population. (a) f = fit (p ), = 1,,...,L, Set f = f, p = p, = 1,,...,L. (b) Find te index J of te individual wit te best fitness J arg max L f. 1 Set f best = f J, p beat = p J. (c) Set gen = 1. Step 3. Update te vector g =[g 1 g g j gb ], {1,,,L}, as follows: If rand (), ten g j* =1g j*, were j* =round (B rand () + 0.5). round (B rand () + 0.5) rounds B rand () to te nearest integer. Step 4. Update p, f, and p best, f best. (a)update p, f in te following: Calculate f = fit(p ), if f > f ' ',, ten f = f, p = p, = 1,,,L, (b)update p best, f best in te following: ' If f > f best, ten f best = f ',p best = p, = 1,,,L. Step 5. Update te velocity vector v and te parameter vector p. (a) Update te velocity vectors in te following: 1 best ' v v c rand() ( p p ) c rand() ( p p ), 1,,, L. (b) Update te parameter vectors in te following: (0) Step 6. Decrease and by te constants d 1 [0,1] and d [0,1], respectively. v, 1,,,, vd1 L * d. () (3) Step 7. gen = gen+1, if gen > K ten go to Step 8; oterwise go to Step 3. Step 8. Based on te individual p best =[r best g best ] wit te best fitness f best, te desired fuzzy classification system can be determined. Eac g is randomly updated by Step 3 so tat te fuzzy classification system generated by te individual p as an appropriate number of fuzzy rules for te considered classification problem. Te adjustment by Step 5 is conceptually similar to te mutation operator utilized by te genetic algoritm. Eac individual p keeps track of its own best solution, wic is associated wit te best fitness f ', it as acieved so far in a vector p. Furtermore, te best solution among all te individuals obtained so far in te population is kept track of as te vector p best associated wit te global best fitness f best.according to Step 5, eac parameter vector p is assigned wit a randomized velocity vector according to its own and its companions flying experiences so tat te vector p searces around its best vector p and te global best vector p best. Te adjustment toward p and p best by Step 5 is conceptually similar to te crossover operator utilized by te genetic algoritm. Consequently, te parameter vector p is updated by te PSO so tat te fuzzy classification system generated by te individual p as an appropriate number of rules and a low number of incorrectly classified patterns simultaneously. 4. Simulation Results In tis section, a syntetic data set and te Iris data set are employed to examine te learning ability and te generalization ability of te proposed PSO-based fuzzy classification system, respectively. Example 1. A syntetic data set In tis example, a syntetic data set containing of tree clusters of various sizes, sapes and orientations

6 68 Cia-Cong Cen [1] is utilized to examine te learning ability of te proposed PSO-based fuzzy classification system. Te total number of patterns in tis data set is 579. Figure 1 sows te data set wit a mixture of sperical and ellipsoidal clusters. Following te proposed metod, simulation results of te proposed approac to classifying tis data set is sown in Figure, were te initial conditions for te proposed metod in Example 1 are given in te following: Te number of individuals: L = 100, te maximum number of rules: B = 0, te number of generations: K = 50, te range of m (j1,1), j {1,,,0} : [0,1], te range of m (j1,), j {1,,,0} : [0.05,0.5], te range of m (j1,3), j {1,,,0} : [0.05,0.5], te range of m (j,1), j {1,,0} : [0,1], te range of m (j,), j {1,,,0} : [0.05,0.5], te range of m (j,3), j {1,,,0} : [0.05, 0.5], te constants for te fitness function: { e, r }= {5,10} and te constants for te PSO: {, c 1,c,d 1,d }= {1,1,1,0.75,0.75}. Te rule base of te selected fuzzy classification system as four rules suc tat te number of incorrectly classified patterns is zero. Te parameters of te selected fuzzy classification system are sown in Table 1. From simulation results of tis example, it is clear te proposed PSO-based metod can select significant fuzzy rules to construct a fuzzy classification suc tat te number of incorrectly classified patterns are minimized. Example. Iris data set Te Iris data set [1] contains 150 patterns wit four features, tat belong to tree classes (Iris Setosa, Iris Versicolour and Iris Virginica). Te four features are te sepal lengt in cm, te sepal widt in cm, te petal lengt in cm and te petal widt in cm. Te data set contains tree classes, eac of 50 patterns; eac class refers to a type of iris plant. One class is linearly separable from te oter two; te latter are not linearly separable from eac oter. In tis example, te two-fold cross validation (CV) is employed to examine te generalization ability of te proposed approac to classifying te iris data. In te CV procedure, te Iris data are separated into two subsets of te same size. Tat is, eac subset consists of Table 1. Parameters of te selected fuzzy classification system by te proposed PSO-based metod in Example 1 j m (j1,1) m (j1,) m (j1,3) m (j,1) m (j,) m (j,3) H j CF j Figure 1. A syntetic data set wit a mixture of sperical and ellipsoidal clusters in Example 1. Figure. Simulation results of te proposed PSO-based metod for te syntetic data set in Example 1.

7 Design of PSO-based Fuzzy Classification Systems 69 Table. Simulation results of te proposed PSO-based metod for te Iris data set using CV metod in Example Average classification rate on training patterns Average classification rate on test patterns Average number of fuzzy rules 98.87% 96.8% 4.75 Table 3. Generalization ability for test patterns in te Iris data classification using te same CV metod Metod Average classification rate on test patterns Average number of fuzzy rules Pruning % 8.00 Multi-rule-table % GA-based % Proposed metod % patterns. One subset is used as training patterns to construct a fuzzy classification system by te proposed approac. Te oter subset is used as test patterns to evaluate te generated fuzzy classification system. Te same training-and-testing procedure is also followed after te roles of te subsets are excanged wit eac oter. Te above procedure is iterated 0 times using different partitions of te 150 patterns into two subsets for calculating te average classification rates on tese data. Te initial conditions for te proposed metod in Example are given in te following: Te number of individuals: L = 100, te maximum number of rules: B = 100, te number of generations: K = 50, te range of m (j1,1), j {1,,,0} : [4.4,7.7], te range of m (j1,), j {1,,, 0} : [0.1,1.65], te range of m (j1,3), j {1,,,0} : [0.1,1.65], te range of m (j,1), j {1,,,0} : [,4.4], te range of m (j,), j {1,,,0} : [0.1,1.], te range of m (j,3), j {1,,,0} : [0.1,1.], te range of m (j3,1), j {1,,,0} : [1.,6.7], te range of m (j3,), j {1,,, 0} : [0.1,.75], te range of m (j3,3), j {1,,,0} : [0.1,.75], te range of m (j4,1), j {1,,,0} : [0.1,.5], te range of m (j4,), j {1,,,0} : [0.1,1.], te range of m (j4,3), j {1,,,0} : [0.1,1.], te constants for te fitness function: { e, r } = {5,10} and te constants for te PSO: {,c 1,c,d 1,d } = {1,1,1,0.75,0.75}. Te average number of fuzzy rules of te selected fuzzy classification systems and te average classification rates on training patterns and test patterns are summarized in Table. Table 3 compares our results wit te results in [7] for te classification of te Iris data using te same CV metod. From simulation results for te Iris data set, it is obvious tat te proposed PSO-based fuzzy classification system as ig generalization ability for te classification problem of te Iris data set. 5. Conclusions A PSO-based metod is proposed to design an appropriate fuzzy classification system for pattern classification. In te proposed approac, eac individual p consists of two parameter vectors: r and g Te parameter vector g is updated so tat te generated fuzzy classification system as an appropriate number of fuzzy rules. Te parameter vector r is updated so tat te premise part of te generated fuzzy classification system as appropriate membersip functions for te considered classification problem. Ten, te obtained premise fuzzy sets are used to determine te consequent parameters of te corresponding fuzzy classification system. Consequently, eac individual corresponds to a fuzzy classification system. Subsequently, a fitness function is defined to guide te searcing procedure to select a fuzzy classification system wit te desired performance. Te simulation results sow tat te selected fuzzy classification system not only as an appropriate number of rules for te considered classification problem but also as a low number of incorrectly classified patterns. Acknowledgement Tis researc was supported in part by te National Science Council of te Republic of Cina under contract NSC References [1] Blake, C., Keog, E. and Merz, C. J., UCI Repository of Macine Learning Database, Univ. California, Irvine (1998). ttp:// [] Eberart, R. and Kennedy, J., A New Optimizer Using Particle Swarm Teory, Proc. Int. Sym. Micro Macine and Human Science, Nagoya Japan, Oct., pp (1995).

8 70 Cia-Cong Cen [3] Isibuci, H., Nozaki, K., Yamamoto, N. and Tanaka, H., Selecting Fuzzy If-Ten Rules for Classification Problems Using Genetic Algoritms, IEEE Trans. Fuzzy Systems, Vol. 3, pp (1995). [4] Jang, J. S., ANFIS: Adaptive-Network-Based Fuzzy Inference systems, IEEE Trans. on Systems, Man and Cybernetics, Vol. 3, pp (1993). [5] Jang, J. S., Sun, C. T. and Mizutani, E., Neuro-Fuzzy and Soft Computing, Prentice Hall, New Jersey, U.S.A. (1997). [6] Kennedy, J. and Eberart, R., Particle Swarm Optimization, Proc. IEEE Int. Conf. Neural Networks, Pert, Australia, pp (1995). [7] Nozaki, K., Isibuci, H. and Tanaka, H., Adaptive Fuzzy Rule-Based Classification Systems, IEEE Trans. on Fuzzy Systems, Vol. 4, No. 3, Aug., pp (1996). [8] Simpson, P. K., Fuzzy Min-Max Neural Networks- Part 1: Classification, IEEE Trans. Neural Networks, Vol. 3, Sep., pp (199). [9] Wang, L. X. and Mendel, J. M., Generating Fuzzy Rules by Learning from Examples, IEEE Trans. on Systems, Man and Cybernetics, Vol., pp (199). [10] Wong, C. C. and Cen, C. C., A Hybrid Clustering and Gradient Descent Approac for Fuzzy Modeling, IEEE Trans. on Systems, Man and Cybernetics-Part B: Cybernetics, Vol. 9, pp (1999). [11] Wong, C. C. and Cen, C. C., A GA-Based Metod for Constructing Fuzzy Systems Directly from Numerical Data, IEEE Trans. on Systems, Man and Cybernetics-Part B: Cybernetics, Vol. 30, pp (000). [1] Wong, C. C., Cen, C. C. and Su, M. C., A Novel Algoritm for Data Clustering, Pattern Recognition, Vol. 34, pp (001). [13] Yager, R. R. and Filev, D. P. Essentials of Fuzzy Modeling and Control, Jon Wiley, New York, U.S.A. (1994). Manuscript Received: Apr. 7, 005 Accepted: Jul. 13, 005