Identifying fuzzy systems from numerical data with Xfuzzy

Transcription

1 Identifying fuzzy systems from numerical data with Xfuzzy I. Baturone Instituto de Microelectrónica de Sevilla, (IMSE-CNM), y Dept. Electrónica y Electromagnetismo, Univ. de Sevilla (Spain) lumi@imse.cnm.es F. J. Moreno-Velo Dept. Ing. Electrónica, de Lenguajes Informáticos y Automática, Univ. de Huelva (Spain) velo@diesia.uhu.es A. Gersnoviez Instituto de Microelectrónica de Sevilla, (IMSE-CNM), y Dept. Electrónica y Electromagnetismo, Univ. de Sevilla (Spain) andres@imse.cnm.es Abstract Extracting fuzzy rule bases from numerical data draws a great interest nowadays. Several algorithms based on grid partitions or clustering have been reported in the literature for this purpose, and several CAD tools have been developed to implement one or other type of techniques. This paper presents the CAD tool xfdm that allows applying both types of techniques. In particular, it permits to select among four types of grid-based techniques and five types of clustering algorithms. Advantages of this tool is that it is integrated into the Xfuzzy 3 environment, and, hence, the identified rule base can be described, verified, tuned, simplified, and synthesized with the corresponding tools of Xfuzzy 3. Keywords: Identification, knowledge extraction, clustering, learning techniques. 1 Introduction Techniques to analyze numerical data and obtain significant information are required in many areas such as medicine, engineering, and, in general, the area of knowledge discovery in databases (KDD). Fuzzy logic-based techniques draw a great interest in these areas since they are able to extract linguistic knowledge that could be easily understood by even non expert people. The reason is that a fuzzy rule base contains rules whose antecedent and consequent parts are expressed with linguistic terms similar to those employed by natural language. Two different strategies are usually employed when defining a fuzzy rule base from input/output numerical data: clustering and grid-based fuzzy rule learning. The first one organizes the data into clusters and uses them to create the rules. Each cluster is transformed into a rule by projecting it onto every dimension of the input variables [1-2]. Hence, the whole rule base description (rules and membership functions) is obtained simultaneously. An advantage of these techniques is that the number of rules extracted is usually low. As a drawback, each rule has its own fuzzy sets that do not appear in other rules, which complicates their linguistic meaning. In the other side, the grid-based fuzzy rule learning generates a partition or grid of the input and output spaces prior to creating the rule base [3]. The rules are obtained by selecting the adequate input and output labels according to the numerical data. The grid-based fuzzy rule learning can be understood as a particular case of the clustering technique in which the data are grouped into hyper rectangular clusters formed by the grid partition [4]. Several CAD tools reported in the literature are based on the grid-based fuzzy rule learning technique. Some examples are NEFCLASS [5], NEF- PROX [6], and KBCT [7]. Other tools, such as FCLUSTER [8], are based on the cluster-based techniques. Many of these tools are specialized on knowledge extraction problems so that the use of the extracted fuzzy system in a given application must be done with other tools, such as those in Matlab. The tool presented in this paper, xfdm, can apply grid- as well as cluster-based techniques, since it allows using well-known algorithms of both types. In addition, it is integrated within the Xfuzzy

2 environment [9], so that the fuzzy rule base identified can be used by the rest of the tools of this environment without requiring any translation. The paper is structured as follows: Section II summarizes the design flow that can be followed with the aid of the Xfuzzy environment. It clarifies how this new tool can be used within a fuzzy system design flow. Section III describes the features of this new tool while an illustrative application example is included in Section IV. Finally, some conclusions are given in Section V. 2 The Xfuzzy 3 environment The simplest structure of a fuzzy inference system consists of a unique and simple rule base relating input and output variables with linguistic concepts. The universe of discourse of the variables is covered by fuzzy sets associated with those linguistic concepts. The classic inference mechanism is based on three stages: fuzzification, inference mechanism, and defuzzification. The inference mechanism is performed with three steps: firstly, the activation degree of each rule is calculated by applying conjunctive or disjunctive operators between the antecedents; secondly, an implication function is used to obtain the conclusion of each rule; and, finally, the global conclusion is calculated by an aggregation operator [10]. In order to describe more complex rule bases, complex antecedent parts can be used in the rules, for example, by connecting the several antecedents by any kind of conjunctive and disjunctive connectives, by relating input variables with fuzzy sets by any kind of linguistic hedges, and by even applying linguistic hedges to some connected antecedents [1]. It should be noted that the fuzzy concepts of the rules (such as the membership functions, the conjunctive, disjunctive, implication, and aggregation operators, the linguistic hedges, and the defuzzification methods) can be represented by many mathematical functions and algorithms [11]. In order to describe more complex structures of fuzzy systems, interconnection of several fuzzy modules should be admitted. Each module should apply its own and appropriate fuzzy operators (for instance the defuzzification methods can be different), and may interchange fuzzy or non fuzzy values. The Xfuzzy 3 environment has been developed so as to be able to design complex fuzzy systems with these three features of expressiveness, extensiveness, and modularity. The design methodology with Xfuzzy follows the flow chart in Fig. 1. The aim of the first stage (description) is to describe the whole fuzzy system, whose constituent modules usually respond to the architecture thought by a human expert. The structure of each module can be also determined from the expert linguistic knowledge. A formal specification language, named XFL3 [12], has been defined to facilitate the translation of complex rules expressed linguistically (with linguistic hedges, weights, any kind of connective function, etc.). The use of XFL3 allows sharing the same system definition throughout the stages of the design methodology. Another relevant feature of the language XFL3 is that it distinguishes between the logical and functional structure of the system to define. The logical structure contains the modular architecture of the system, the linguistic variables employed and their associated fuzzy sets, and the rule base of each module with its associated fuzzy operators. On the other hand, the functional structure contains the mathematical functions which define the membership functions of the fuzzy sets and the fuzzy operators of each module rule base. These operators can be defined freely by the user. The tools xfedit and xfpkg can aid the user at this description stage. Once the whole system has been described, its behavior should be tested at the verification stage. Possible errors and deviations have to de detected at this stage in order to correct them. Three tools facilitate this verification process in several ways. expert knowledge Description Verification Synthesis implementation numerical data Tuning / Simplification Xfuzzy 3 Figure 1: Xfuzzy 3 design flow without identification stage. 1258

3 One of them (xfplot) allows showing two- and threedimensional graphics with the input/output behavior of the fuzzy system. Another tool (xfmt) allows monitoring the values of the internal and global variables of the system and the activation degrees of the rules of the different modules. The last tool, named xfsim, simulates the behavior of the fuzzy system working in line with a model of an external system (a plant in the case of a controller). One way of correcting deviations from a desired behavior is to apply supervised learning techniques. The tool xfsl includes a wide set of tuning algorithms [13]. It allows tuning hierarchical fuzzy systems with operators defined freely by the user. After tuning, the resulting module descriptions might be simplified by applying pruning techniques. This is performed by the tool xfsp. Once the whole system description has been verified, tuned, and simplified, the last step (synthesis) is to represent the system by an executable format suitable for the application. Three tools allow software synthesis (to C, C++, and Java) while the tool xfvhdl allows hardware synthesis. This design flow starts with fuzzy rule bases translated from linguistic knowledge, but it did not allow obtaining rule bases from numerical data. The numerical data were employed only to adjust or tune the fine structure of the rule bases (the parameters of the membership functions). The new tool, xfdm, has been developed to also employ these numerical data to obtain the coarse structure of the rule base (number of membership functions, number of rules, etc.). Hence, a new identification stage has been included within this design flow, as shown in Fig. 2. Figure 3: Main window of the xfdm tool. 3 The xfdm tool The main window of the xfdm tool is shown in Fig. 3. It allows selecting: the grid- or clustering-based algorithm employed to extract the fuzzy rule base, the file with the numerical data from which the fuzzy rule base will be extracted, the number of inputs and outputs of the rule base to extract, and the input and system style of that rule base. The input style means the number and type of membership functions to cover the input universes of discourse as well as the name of the inputs. It is configured with the window shown in Fig. 4. Depending on the algorithm, this window will allow or not selecting different number and types of membership functions for each input variable. Free triangles or Gaussian functions as well as families of triangles and B- splines can be selected to represent the input fuzzy sets, as shown in Fig. 4. Another window, shown in Fig. 5, helps the user to configure the system style. The system style means to specify the name of the rule base to extract, the prefix used to name the output variables, the kind of conjunction operator used in the antecedents, and the type of inferencedefuzzification method applied (Fuzzy Mean, Weighted Fuzzy Mean, Takagi-Sugeno, or a classification method, which takes as output the consequent of the most activated rule). Regarding expert knowledge numerical data Description Identification Verification Tuning / Simplification Synthesis Xfuzzy 3 implementation Figure 2: Xfuzzy design flow with identification stage. Figure 4: Window to configure the input style. 1259

4 Figure 5: Window to configure the system style. the system style configuration, the user can also decide or not to generate a fuzzy system containing the extracted rule base. In the first case, the inputs and outputs of the fuzzy system coincide with those of the rule base extracted. Otherwise, the rule base is extracted but the structure of the fuzzy system is not generated. This permits to identify different rule bases that could then be connected adequately by the user with the tool xfedit, thus describing a complex fuzzy system. The buttons Load, Save, Create system, and Close, in the main window of xfdm allows, respectively, to load a previously saved configuration to perform identification, to save a new configuration, to create the description with XFL3 of a fuzzy system containing the extracted rule base(s), and to close the tool. Since the efficiency of a particular grid- or clusterbased identification technique depends very much on the application, several algorithms have been programmed to cover as much as possible the different possibilities. Among the grid-based techniques, three versions of the Wang and Mendel s algorithm can be employed. They have been named Wang&Mendel, Nauck, and Senhadji algorithms. In all of them, the user defines a fixed partition for the input variables (equal or different for each variable), and, for each possible combination of input fuzzy sets (for each possible rule antecedent), it is evaluated which data activates it most (if there is any). That data will give the consequent of that rule. If there is no data, that rule will not appear in the rule base. In the Wang& Mendel algorithm no rule selection is implemented, so that the extracted rule base can be very large due to the curse of the dimensionality affecting gridbased techniques. Nauck and Senhadji algorithms avoid this problem since they allow selecting the maximum number of rules generated according to an efficiency measure [5][14]. Another grid-based algorithm that can be employed with xfdm is named Incremental Grid. This Figure 6: Configuring Incremental Grid algorithm. algorithm, based on the proposal in [15], starts with two fuzzy sets covering each input variable and 2 u rules (with u the number of inputs). Every iteration, the point giving the maximum error between the numerical data and the inference result of the current rule base is determined. New fuzzy sets placed at that point are added as well as the corresponding rule. The user selects if the consequents of the extracted rules are obtained from the numerical data by applying or not learning, as shown in Fig. 6. The algorithm finishes when reaching a maximum number of rules or fuzzy sets per input or a minimum error (Fig. 6). Compared with the other grid-based algorithms (which used a fixed grid), this can find a better covering of the input variables. Regarding the cluster-based techniques, the user can select among four algorithms that use a fixed number of clusters and another algorithm that finds the adequate number of clusters iteratively. The first ones are based on the Hard C-means, Fuzzy C- means [16], Gustafson-Kessel [17], and Gath-Geva [18] algorithms. These algorithms cluster the numerical data into a number of clusters specified by the user. They finish when reaching a maximum number of iterations or a minimum variation in the obtained clusters (Fig. 7). The main difference between them is the shape of the obtained clusters: while the Hard and Fuzzy C-means find hyper spherical clusters (fuzzy or not), Gustafson-Kessel and Gath-Geva find hyper ellipsoidal clusters (of the same or different volumes). Hence, the two latter can better extract fuzzy rule bases with first-order Takagi-Sugeno inference method. The other cluster-based algorithm that can be employed with xfdm is named Incremental Clustering. It is based on the proposal in [19] to find a suitable number of clusters for the data in the file. The algorithm is configured by specifying the radius of influence of the obtained clusters and the maximum number of clusters to obtain (Fig. 8). 1260

5 Figure 7: Configuring Fixed Clustering algorithms. 4 Application example A classification problem is included herein to illustrate the use of the xfdm tool within the design flow of Xfuzzy. A set of numerical data with information about 572 olive oil samples has been employed. Each numerical data contains the values of 8 measurements of each sample and the kind of olive oil (out of 9 types) to which that sample belongs to. The numerical data has been divided into 250 data to identify a fuzzy rule base and 322 data to test its performance. Using a fixed grid-based technique with 3 fuzzy sets per input, the number of possible extracted rules can reach 6561 (3 8 ). If the Wang&Mendel algorithm is selected, only 71 rules are created (they are the rules activated by the numerical data). The classification errors of this extracted rule base are 49 out of 250 (19.6 %) for the training data and 63 out of 322 (19.6%) for the test data. The tuning tool of Xfuzzy, xfsl, has been used to adjust the extracted rule base (the fuzzy sets of the input variables). Selecting the RProp learning algorithm to minimize the classification square error, it is obtained a new rule base that performs better: it fails 11.6% with the training data and 9% with the test data. This adjusted rule base can be simplified with the tool xfsp of Xfuzzy. Applying a simplification technique consisting in pruning those rules whose activation degree never exceeds 0.45 for the numerical data, 30 rules are eliminated, thus resulting a rule base with 41 rules. If this simplified rule base is again tuned, the classification errors are even inferior: 10.8% with the training data and 7.5% with the test data. Using the Incremental Clustering algorithm, 22 clusters are find, which means 22 rules. The classification errors of this rule base are 27 of 250 (10.8%) for the training data and 15 of 320 (4.7%) for the test data. Tuning this rule base with the tool xfsl (using again the RProp algorithm), the performance is improved: the adjusted fuzzy system fails 2.4% with the training data and 3.4% with the test data. This adjusted rule base can be simplified with the tool xfsp of Xfuzzy. Now the simplification technique applied has been to reduce the number of fuzzy sets by using similarity measures. The 22 fuzzy sets per input obtained by projecting the clusters have been reduced, for instance, to 5 or 12 depending on the input. If this simplified rule base is again tuned, the classification errors are 2.4% with the training data and 3.7% with the test data. The system extracted by using the clustering algorithm performs around twice better than the system obtained with the grid-based technique. Besides, it is twice simpler (it contains 22 instead of 41 rules). These results usually appear when comparing clustering and grid-based techniques (independently of the particular algorithm). The disadvantage of clustering techniques is that the linguistic meaning of the extracted rules is inferior because of the covering of the input variables. In the classification example considered herein, Fig. 9a shows some fuzzy sets obtained with the Wang&Mendel algorithm. They can be understood as small, medium and large linguistic concepts. However, the sets obtained with the Incremental Clustering (Fig. 9b) are difficult to express with linguistic terms. 5 Conclusions Figure 8: Window to configure the Incremental Clustering algorithm. The tool xfdm presented in this paper is very useful to complete the fuzzy system design flow that can be carried out with the Xfuzzy 3 environment since it allows that the fuzzy rule bases can be described 1261

6 from numerical data. The rule bases identified can be included and interconnected with other rule bases either also identified from numerical data or translated from linguistic knowledge so as to form a complex fuzzy system. The complete fuzzy system can be verified, tuned, simplified, and synthesized with the rest of the tools of Xfuzzy 3. References (a) (b) Figure 9: Fuzzy sets covering the first and third inputs with (a) the Wang&Mendel, and (b) the Incremental Clustering algorithms. [1] M. Sugeno, T. Yasukawa, A Fuzzy-Logic- Based Aproach to Qualitative Modeling, IEEE Trans. on Fuzzy Systems, Vol. 1, Nº 1, pp.7-31, Feb [2] F. Klawonn, R. Kruse, Constructing a Fuzzy Controller from Data, Fuzzy Sets and Systems, 85, pp , [3] L. Wang, J.M. Mendel, Generation Rules by Learning from Examples. Proc. Int. Symp. on Intelligent Control, pp , [4] D. Nauck, Data Analysis with Neuro-Fuzzy Methods, PhD. Dissertation, Univ. of Magdeburg, Faculty of Computer Science, Germany, [5] D. Nauck, R. Kruse, NEFCLASS - A Neuro- Fuzzy Approach for the Classification of Data, Proc ACM Symp. on Applied Computing, Nashville, Feb , pp ACM Press. [6] D. Nauck, R. Kruse, Function Approximation by NEFPROX, Proc. 2nd European Workshop on Fuzzy Decision Analysis and Neural Networks for Management, Planning, and Optimization (EFDAN 97), pp , Dortmund. [7] J. M. Alonso, L. Magdalena, and S. Guillaume, KBCT: A knowledge Extraction and Representation Tool for Fuzzy Logic Based Systems, Proc. FUZZ-IEEE 2004, Vol. 2, pp , Budapest (Hungary), July [8] FCLUSTER, a tool for fuzzy cluster analysis: [9] Xfuzzy web site: [10] L-X. Wang, A course in fuzzy systems and control, Prentice Hall, Englewood Cliffs, New Jersey, [11] E. H. Ruspini, P. P. Bonissone, W. Pedrycz, Eds., Handbook of Fuzzy Computation, Institute of Physics Pub., [12] F.J. Moreno-Velo, S. Sánchez-Solano, A. Barriga, I. Baturone, D.R. López, An Specification Language for Fuzzy Systems, Mathware & Soft Computing, vol. 8, n. 3, pp , [13] F.J. Moreno-Velo, I. Baturone, S. Sánchez- Solano, A. Barriga, Xfsl: A Tool for Supervised Learning of Fuzzy Systems, Proc. European Symp. on Intelligent Technologies, Hybrid Systems and their implementation on Smart Adaptive Systems (EUNITE 2001), Tenerife, Spain, pp [14] R. Senhadji, S. Sánchez-Solano, A. Barriga, I. Baturone, F. J. Moreno-Velo, NORFREA: An Algorithm for non-redundant fuzzy rule extraction, Proc. IEEE SMC 2002, vol. 1, pp , Tunisia, Oct [15] C.H. Higgins, R.M. Goodman, Fuzzy rulebased networks for control, IEEE Trans. on Fuzzy Systems, vol. 2, n. 1, pp , [16] J. C. Bezdek, Pattern recognition with fuzzy objective function algorithms, Plenum Press, New York, [17] D. E. Gustafson, W. C. Kessel, Fuzzy clustering with a covariance matrix, Proc. IEEE Conf. on Dec. & Control, San Diego, 1979, pp [18] I. Gath, A. B. Geva, Unsupervised optimal fuzzy clustering, IEEE Trans. On Pattern Análisis and Machina Intelligence, vol. 11, pp , [19] S. L. Chiu, A cluster estimation method with extensión to fuzzy model identification, Proc. IEEE Int. Conf. on Decision, pp ,