FuzzConRI - A Fuzzy Conjunctive Rule Inducer

Size: px

Start display at page:

Download "FuzzConRI - A Fuzzy Conjunctive Rule Inducer"

Dorthy Jefferson
6 years ago
Views:

1 FuzzConRI - A Fuzzy Conjunctive Rule Inducer Jacobus van Zyl and Ian Cloete School of Information Technology, International University in Germany, Bruchsal, Germany {jacobus.vanzyl,ian.cloete}@i-u.de Abstract. A variety of methods exist for inductive learning of classification rules using crisp sets. In this paper we illustrate an inductive learner that uses fuzzy sets, where the membership functions of the linguistic terms are given in advance. We also show how the induction of conjunctive rules fit into a fuzzy set covering framework (FuzzyBexa) that we introduced before. 1 Introduction Rule inducers are an important category of machine learning algorithms. Typically they obtain propositional expressions derived from a set of training instances. A multitude rule learning algorithms were proposed, and especially the separate-and-conquer approach to rule learning has been very popular [1]. These algorithms are all based on crisp set covering ideas. Fuzzy sets are a generalization of crisp sets [2] providing increased expressive power and comprehensibility. There have been many attempts to induce fuzzy concept representations from data. These include fuzzy neural networks [3], fuzzy decision trees [4], genetic algorithms [5], grid methods [6], and clustering [7]. These methods, however, are primarily focused on the parameter identification, i.e. finding suitable parameters for a fuzzy inference systems, such as membership functions for the fuzzy sets. Little work has been done thus far on rule induction systems that perform structure identification, i.e. learning incomplete rules that do not contain all the input domain variables in the antecedent. Some efforts have been made by Wang et al who propose an induction method for learning modular fuzzy rules [8] which is based on the crisp rule learner PRISM [9]. Castro et al showed that an assumption-based truth maintenance system (ATMS) can be used to build a fuzzy inductive learner. This algorithm finds the minimal node in an ATMS and uses it to propose fuzzy classification rules [10]. In this paper we present FuzzConRI, a novel induction algorithm capable of inducing incomplete fuzzy classification rules from data. We also show that this algorithm can fit into the fuzzy set covering framework FuzzyBexa [11]. 2 FuzzConRI FuzzConRI is a fuzzy rule induction algorithm based on the CN2 approach [12, 13]. The algorithm is shown in Table 1. It consists of two layers, an upper

2 Table 1. The FuzzConRI algorithm FuzzConRI Input: Set of training instances T, Set of concepts to learn Concepts Output: A rule set describing the concepts Set the rule set to empty FOR EACH concept Concepts P = {i T µ concept(i) α c} N = T P WHILE P is not empty, and more rules can be found DO antecedent=findbestantecedent(p, N) If a suitable antecedent is found, augment the rule set with IF antecedent THEN concept Remove the positive instances covered by the added rule Return the rule set FindBestAntecedent Input: Set of positive instances, Set of negative instances Output: Antecedent that covers the positive instances best Let ST AR contain the antecedent T RUE Let BEST ANT be nil Let T ERMS contain all possible terms While ST AR is not empty NEW ST AR = {x y x ST AR, y T ERMS} NEW ST AR = NEW ST AR ST AR For each antecedent A in NEW ST AR If A is better than BEST ANT according to an evaluation function, then Replace the current best conjunction with the new one Remove all antecedents that cover only positive instances Retain a user defined number of best elements in NEW ST AR ST AR = NEW ST AR Return BEST ANT layer implementing a set cover approach to rule induction, and a lower layer for inducing one rule. The upper layer receives a set of training instances T, and a set of concepts Concepts. It starts by initialising the rule set to the empty set, and then considers each concept concept Concepts, one at a time. The training set T is split into two parts, a set of fuzzy instances P that belong to the concept and a set of fuzzy instances N that do not belong to the concept. The sets P and N are called the set of positive and negative instances, respectively. In contrast to a crisp instance, a fuzzy instance belongs to all linguistic terms (attribute values) to a certain degree. Crisp instances can belong to only one attribute value per attribute at a time. If a concept such as Temperature is hot must be learned, the user specifies a threshold membership α c at which the temperature is considered hot. The procedure FindBestAntecedent is then repeatedly called to induce a conjunction that covers a set of instances in T.

3 The instances in P covered by the conjunction are removed from P and the procedure repeated until P is empty. The rules induced by FuzzConRI are of the form: IF [T emperature is mild hot] [W ind is calm] THEN Sport is tennis where T emperature, W ind and Sport are linguistic variables, and mild, hot, calm and tennis linguistic terms. Here the difference between fuzzy and crisp rule induction becomes clear. The above rule will cover no instances in the crisp case since no instance can have both T emperature = mild and T emperature = hot at the same time. This, however, is perfectly possible in the fuzzy case. The procedure FindAntecedent receives a set of positive and a set of negative instances and returns the antecedent that best covers the positive instances while attempting not to cover any negative instances. It starts by initialising the set ST AR with the mgc (most general conjunction). This conjunction should cover all possible instances, and thereby also the training set. The best conjunction found during the search is stored in the variable BEST ANT. A new ST AR is obtained by forming the conjunction of all possible single linguistic term descriptions with each conjunction in ST AR, and the removing the conjunctions that were not changed, e.g. the conjunction of [T emperature = hot][w ind = calm] with the description [W ind = hot] does not change the conjunction, and is removed. Here we left the symbol out for brevity s sake. After the new conjunctions are formed, they are evaluated according to an evaluation function. This evaluation functions plays a crucial role to guide the search for good rule antecedents. A rule evaluation function that performs very well, especially in the presence of noise, is the fuzzy Accuracy function, A(r) = X P (c) (X N (c) (1) where c is a conjunction, and X S (c) is its extension in the set S, i.e. all the instances that are matched by c. The set cardinality in the fuzzy case, also called the sigma count of the set [14], is computed as, X S (c) = µ c (i) (2) i X S(c) We consider an instance to match a fuzzy conjunction when its membership to the conjunction is non-zero. If necessary, an α-cut can also be applied to the fuzzy sets in the input domain. In this case the value of the α-cut is specified by α a. The fuzzy and operators can be implemented by any t-norm and t- conorm functions, such as minimum and maximum for example. For an overview of possible fuzzy conjunction evaluation functions for iterated concept learning refer to [15, 16]. After the conjunctions are evaluated, they are compared to the best conjunction found thus far, and if a conjunction obtains a higher score than the best conjunction it replaces the best conjunction. Finally, the conjunctions that cover no positive instances are removed from the search, since they cannot be

4 Table 2. FuzzyBexa s top layer routines. CoverConcepts Input: Set of training instances, Set of concepts to learn Output: A rule set describing the concepts Set the current rule set to empty For each concept Find the set of positive and negative instances While there are positive instances Find the best conjunction If a suitable conjunction is found, augment the rule set with IF conjunction THEN concept Remove the positive instances covered by the added rule Return the rule set FindBestConjunction Input: Set of positive instances, Set of negative instances Output: Conjunction that covers the positive instances best Set the current best conjunction to empty Add the most general conjunction to the current conjunctions While more (useful) conjunctions can be found Generate more specializations For each new conjunction If the conjunction is better than the current best conjunction according to an evaluation function, Replace the current best conjunction with the new one Remove conjunctions that cover no negative instances, or that cannot be improved beyond the performance of the best conjunction Retain only the beamwidth best conjunctions If the evaluation function value for the best conjunction is the same or worse than that of the complete training set, return: no conjunction found, otherwise, return the best conjunction found. improved upon. FuzzConRI can implement a beam search by retaining more than one conjunction for further specialization in the new ST AR. The set ST AR will eventually become empty when no more new conjunctions can be formed such that they do not replace older conjunctions. The best conjunction found during the search process is then returned. 3 FuzzConRI and FuzzyBexa The BEXA framework provided a method for comparison of several crisp rule induction algorithms [17]. We recently extended this framework to the fuzzy case [11]. FuzzyBexa consists of three layers. The top layer implements a fuzzy set

5 Table 3. FuzzConRI as specialization model. GenerateSpecializations Input: Set of conjunctions to specialize Output: Set of specializations Let specializtions be the empty set. For each conjunction c For each linguistic term L Add the conjunction cnew = c L to specializations Remove all duplicate conjunctions from specializations Return specializations covering approach, the middle layer implements several heuristics for guiding the search process, and the bottom layer implements conjunction refinement. The framework uses the description language FuzzyVL 1 which allows internal disjunction. The method of conjunction refinement is implemented by the specialization operator. One possible specialization operator is exclusion, where conjunctions are specialized by removing (excluding) a linguistic term from the description. By changing FuzzyBexa s specialization operator from exclusion to append, FuzzConRI behaviour is implemented by the framework. FuzzyBexa s top layer routines are shown in Table 2, and Table 3 shows FuzzConRI as a specialization model within the framework. FuzzyBexa s top layer routines implement many different heuristics that can be shared between different specialization models. As such, the extra work needed to implement FuzzConRI s behaviour is indeed not much. For an in-depth discussion of FuzzyBexa s induction strategies, search heuristics, efficiency measures, rule pre-pruning methods and stopping criteria refer to reference [11]. The rules induced by FuzzConRI are unordered. Recently, FuzzyBexaII was introduced that extended FuzzyBexa to allow the use of a simultaneous concept learning strategy [18]. 4 Experimental Results In this section we provide an empirical evaluation of different parameters for FuzzConRI. The data sets are available from the UCI machine learning repository [19]. Unfortunately there is currently a lack of publicly available originally fuzzy data sets. By originally fuzzy we mean that either the fuzzy membership functions are known or provided by experts, or that instances are characterized by their membership degrees to linguistic terms. We obtain a fuzzy data set by a process of data fuzzification. Crisp nominal attributes are converted to linguistic variables with crisp membership functions. Fuzzy membership functions are obtained for numerical attributes by first clustering the instance values and then placing bell-shaped membership functions of the form published in [20] at the cluster centres. However, here we have to stress that our methods are independent of the method of parameter identification (membership function

6 extraction), and that better methods for membership function extraction will of course improve the overall system performance. Figure 1 shows results obtained by FuzzConRI from 10-fold cross validation on the Iris and Bodyfat data sets. Figure 1(a) shows the influence of changing the value of α a, i.e. applying an α-cut at varying membership degrees. Here we use the same value for α a during rule induction and test set evaluation (inferencing). Results are shown for the classification accuracy, the rule set complexity, and the amount of search performed to induce the rule set. Rule set complexity is measured by the number of linguistic variables used in the rule set. FuzzConRI s classification performance on the Iris data set seems to be very insensitive to α a. Only extreme values of α a results in a degraded classification accuracy. A similar result is obtained for the rule set complexity. However, the number of conjunctions generated during the induction of the rule set is more sensitive to extreme values for α a, with 2.5 times as many needed for α a = 0.95 than for 0 α a Figure 1(b) shows the effect of the beam width on the induction process. Here α a was kept constant and the beam width was set to values 1, 2, 3, 5, 10, 15, 25, and 50. As is clear from the figure, the beam width had no effect on the classification accuracy or the rule set complexity. This implies that FuzzConRI already obtain the best solution without employing a beam search. A further observation from the figure is that beam widths of 10 and more do not result in the generation of more conjunctions. The reason for that is FuzzyBexa s efficiency measures. FuzzyBexa removes conjunctions that cannot result in good concept descriptions and only concentrate on interesting parts of the search space. Thus a very efficient search of the space of conjunctions is performed, and a large beam width is not required to obtain good results. Figure 1(c) shows the effect of α a for the Bodyfat data. Here a good value for α a is clearly important, and the data set shows sensitivity to α a for all measured parameters. The optimum value for classification performance is obtained for α a 0.65, and the least complex rule set is obtained at α a An interesting trend is the correlation between rule set complexity and the size of the search space examined. This correlation is due to the fact the rule set complexity is correlated with the number of rules in the rule set, and for each rule the hypothesis space is searched. Thus, the number of rules per rule set is directly related to the amount of search required, as can clearly be seen in Figure 1(c). Note that in Figure 1(c) the amount of search shown on the graph is multiplied with a scale factor of 0.1. It is not clear how to automatically choose the value for α a in general. The optimum value for α a might depend on the method of membership function extraction. Our fuzzification method placed bell shaped function on the cluster centres. For the Bodyfat data, values of α a greater than 0.6 resulted in little overlap of the membership functions. This had a detrimental effect on performance. When there is very little overlap, the fuzzy membership functions revert back to crisp -like behaviour, with no two fuzzy sets from the same variable being non-zero at the same time. Thus, in Figure 1(c) the beneficial effect of the fuzzy

7 Influence of α a for the Iris Data Number of Conjunctions Generated 120 Influence of the beam width on the Iris data Number of Conjunctions Generated α a (a) Influence of α a for the Iris data Influence of α a on the Bodyfat data Number of Conjunctions Generated (Scale 0.1) Beam Width (b) Influence of the beam width for the Iris data Influence of the beam width on the Bodyfat data Number of Conjunctions Generated (Scale 0.01) α a (c) Influence of α a for the Bodyfat data Beam Width (d) Influence of the beam width for the Bodyfat data Fig. 1. Results the Iris and Bodyfat data induction method is clearly shown in the rule complexity and search requirement behaviour. For values of α a greater then 0.65 the classification performance also decreased steadily. The reason for the behaviour of the classification performance for α a < 0.65 is not clear. Figure 1(d) shows the effect of an increased beam width on the Bodyfat data. Similar to the Iris data, a near optimum classification result is obtained with no beam search. A marginal increase in classification performance is obtained for a beam width greater than 10. Figure 2(a) and (b) shows results for the Colic data. Classification accuracy is relatively insensitive to α a, while the search requirement and rule set complexity shows a complex relationship to α a. The beam width again had relatively little effect, with only slightly less complex rule sets obtained for greater beam widths. A further observation is that the increase in search requirement is rather linear with an increase in beam width

8 Influence of the α a on the Colic data Number of Conjunctions Generated (Scale 0.001) Influence of the beam width on the Colic data Number of Conjunctions Generated (Scale 0.001) α a (a) Influence of α a for the Colic data Influence of α a on the Lymph data Beam Width (b) Influence of the beam width for the Colic data Influence of the beam width on the Lymph data Number of Conjunctions Generated (Scale 0.1) α a (c) Influence of α a for the Lymph data 10 Number of Conjunctions Generated (Scale 0.001) Beam Width (d) Influence of the beam width for the Lymph data Fig. 2. Results the Colic and Lymph data than exponential. In fact, from Figures 1(a) and (c) we may deduce that the trend of the search requirement is initially linear and eventually reach a plateau. Figure 2(c) shows the relative insensitivity to α a for the Lymph data. Figure 2(d) shows that an increased beam width can in some cases have a positive effect on classification performance and rule set complexity. 5 Conclusion In this paper we presented FuzzConRI, a novel algorithm that can induce fuzzy conjunctive classification rules from data. The algorithm is based on the approach by CN2, but differ from crisp inductive algorithms in that fuzzy sets are used in the description languages. Fuzzy instances belong to all linguistic terms in the problem to a given degree, and therefore rules may have antecedents such

9 as T emperature = cold mild. We have also reviewed FuzzyBexa, a set covering framework for rule induction. We have shown that FuzzConRI fits in this framework and how to insert FuzzConRI as a specialization model in Fuzzy- Bexa. Finally we provided empirical results of FuzzConRI s performance on four data sets for different induction parameters. From the experiments we conclude that FuzzConRI can induce very good rule descriptions even when using no beam search. The experiments have further shown that in a few cases Fuzz- ConRI is sensitive to the value of α a, the α-cut applied to the input domain membership degrees. However, performance is never completely degraded, and in most cases FuzzConRI s classification performance is rather insensitive to α a. References 1. Fürnkranz, J.: Separate-and-conquer rule learning. Artificial Intelligence Review 13 (1999) Cox, E.: The Fuzzy Systems Handbook. 2 edn. Academic Press, London (1998) 3. Kasabov, N.K.: On-line learning, reasoning, rule extraction and aggregation in locally optimized evolving fuzzy neural networks. Neurocomputing (2001) Singal, P.K., Mitra, S., Pal, S.K.: Incorporation of fuzziness in id3 and generation of network architecture. Neural Computing and Applications 10 (2001) Cordón, O., Herrera, F., Hoffmann, F., Magdalena, L.: Genetic Fuzzy Systems - Evolutionary Tuning and Learning of Fuzzy Knowledge Bases. World Scientific, Singapore (2001) 6. Wang, L., Mendel, J.M.: Generating fuzzy rules by learning from examples. IEEE Trans. on Systems, Man and Cybernetics 22 (1992) Gedeon, T.D., Kuo, H., Wong, P.M.: Rule extraction using fuzzy clustering for a sedimentary rock data set. International Journal of Fuzzy Systems 4 (2002) Wang, C.H., Liu, J.F., Hong, T.P., Tseng, S.S.: A fuzzy inductive learning strategy for modular rules. Fuzzy Sets and Systems 103 (1999) Cendrowska, J.: PRISM: An algorithm for inducing modular rules. International Journal of Man-Machines Studies 27 (1987) Castro, J.L., Zurita, J.M.: An inductive learning algorithm in fuzzy systems. Fuzzy Sets and Systems 89 (1997) Cloete, I., van Zyl, J.: Fuzzy rule induction in a set covering framework. (2004) (Submitted). 12. Clark, P., Boswell, R.: Rule induction with CN2: Some recent improvements. In: Proceedings of the Sixth European Working Session on Learning. (1991) van Zyl, J., Cloete, I.: An inductive algorithm for learning conjunctive fuzzy rules. In: International Conference on Machine Learning and Cybernetics, Shanghai, China (2004) 14. Klir, G.J., Yuan, B.: Fuzzy Sets and Fuzzy Logic. Prentice Hall, Upper Saddle River, NJ (1995) 15. van Zyl, J., Cloete, I.: Heuristic functions for learning fuzzy conjunctive rules. In: IEEE International Conference on Systems, Man and Cybernetics, The Hague, The Netherlands (2004) 16. Cloete, I., van Zyl, J.: Evaluation function guided search for fuzzy set covering. In: IEEE International Conference on Fuzzy Systems, Budapest, Hungary (2004)

10 17. Theron, H., Cloete, I.: BEXA: A covering algorithm for learning propositional concept descriptions. Machine Learning 24 (1996) van Zyl, J., Cloete, I.: Simultaneous concept learning of fuzzy rules. In: Proceedings of the 15th European Conference on Machine Learning, Pisa, Italy (2004) 19. Blake, C.L., Merz, C.J.: UCI repository of machine learning databases (1998) URL: mlearn/. 20. Surmann, H.: Learning a fuzzy rule based knowledge representation. In: Proc. of 2. ICSC Symp. on Neural Computation, Berlin (2000)

Simultaneous Concept Learning of Fuzzy Rules

Simultaneous Concept Learning of Fuzzy Rules JacobusvanZylandIanCloete School of Information Technology, International University in Germany, 76646 Bruchsal, Germany Abstract. FuzzyBexa was the first algorithm