Fuzzy Partitioning Using Mathematical Morphology in a Learning Scheme

Transcription

1 Fuzzy Partitioning Using Mathematical Morphology in a Learning Scheme Christophe Marsala Bernadette Bouchon-Meunier LAFORIA-IBP, Université Pierre et Marie Curie, Case 69, 4 place Jussieu, Paris Cedex 05, France. marsala, Abstract In this paper, we present an algorithm to infer a fuzzy partition over a set of numerical values. This algorithm is based on mathematical morphology and is expressed in theory of formal language. It is usable when the numerical values are associated with a class. We use this algorithm during the construction of a fuzzy decision tree, when no fuzzy partition is available for a numerical attribute.. Background We consider an inductive learning scheme. From a given training set, we want to find general laws which describe data pertaining to this set. is composed of examples represented by a description (a set of pairs [attribute, value]) and a class. Our goal is to assign a class to any new example for which we only know a description. This generalization is done by means of rules which will be induced from. The problem has been extensively studied in the case of attributes with symbolic values ([], [7], [2]). Problems occur when the set of the values of a given attribute is imprecise or numerical. A solution is to use a fuzzy representation of these values ([3], [5], [3],[5]). However, this solution gives rise to the complex problem of the generation of a fuzzy representation. A natural idea is to obtain it from experts of the studied domain. But it could be difficult to find experts or expertises for particular kinds of data. One possible means to obtain a fuzzy representation of the values of an attribute is to infer a fuzzy partition from the data of the training set. There exist various methods to infer a fuzzy partition from a set of data []. We consider the construction of a decision tree. The fuzzy partition could be inferred in an initial step, before the construction of the decision tree, and there exist various methods to do this. However, we prefer to use an automatic method which infers a fuzzy partition at each step of the tree construction. Thus, the induced fuzzy partition can be related to the training set of the current step of the decision tree construction. We do not want a highly elaborated method which covers the whole space of training examples as the Krishnapuram s method [0], the neural network method, or genetic algorithm method. These methods cluster the space covered by all the attributes of the data, in one step. We prefer to use an algorithm which builds a fuzzy decision tree to cluster this space. Thus, we propose to infer a fuzzy partition in an automatic way, in an intermediate step of the construction of the decision tree. This method is easy to implement and gives good results. In our approach, we have to compare fuzzy partitions on several attributes. There is no problem even if the class depends on several numerical attributes: the final decision tree will take into account all the attributes involved in the recognition of a class and dependencies of these attributes. In this paper, we present a solution based on the use of mathematical morphology operators, and formalized by means of tools from theory of formal language. The implementation of this solution is described in [4]. In Section 2, we recall the basics of mathematical morphology. In Section 3, we present rewriting systems that implement mathematical morphology operators to smooth a training set. In Section 4, we propose an algorithm to infer a fuzzy partition over a smoothed training set. Section 5 gives an example. In Section 6, we present application of this method to the construction of decision trees in two practical cases. And, finally, we conclude with new ideas that we are planning to add to the method.

2 2. Mathematical morphology operators In this section, we present the fundamental operators of the mathematical morphology. The theory of these operators can be found in [4] or [8]. 2.. The basic operators: erosion and dilatation Dilatation Erosion We consider a space of morphological bodies. These two basic operators enable us to modify a morphological body. This modification is related to a. The erosion is a particular substraction of in, and the dilatation is a particular addition of in. Erosion Dilatation Figure. Mathematical morphology operators 2.2. The operators opening and closure Each of these operators is a combination of the two basic operators. Figure 3. Closure operator 2.3. The open-close filter A filter is a combination of openings and closures. It is composed by successive openings followed by successive closures ( IN) applied to all bodies of the space, with the same. Thus it enables the destruction of small bodies present in the space with respect to the chosen. Simultaneously, it enables the filling of small vacuum places occurring in bodies. The value of enables us to control of such modification. 3. Smoothing a set of values In this section, we introduce the representation of the training set as a word describing the distribution of classes in, according to the values of a given attribute which is supposed to take values in an ordered set. We propose the use of rewriting systems ([9]), from formal language theory, to smooth this word in order to obtain fuzzy modalities of the attributes. Erosion Dilatation Figure 2. Opening operator The opening is the combination of an erosion followed by a dilatation applied to a morphological body, with the same. It enables destruction of small bodies in the space, with respect to the size of the chosen. The closure is the combination of a dilatation followed by an erosion applied to a morphological body, with the same. It enables the destruction of small vacuum places occurring in a body, with respect to the size of the chosen. 3.. The training set as a word We first transform a training set into a word, for a given attribute with values in. Let be an alphabet, each letter of representing one of the classes in. We construct the alphabet with. The letter is a particular letter in the system, we will use it to determine uncertain sequences. For any alphabet, denotes the set of all possible words composed by letters from. For example, let the training set be (!, cheap), ("$#, expensive), ("&%, expensive), ('&', expensive), (#(", expensive) with cheap, expensive as set of classes and )*,+-"#.+-"&%/+0''/+#&"2 as set of values of an attribute (e.g. the attribute size ). Let 3 c+ e, 4 representing cheap and 5 representing expensive. The word defined in 6 by the training set is 475*5!5*5. After this transformation of the training set into a word, we define various ways of using this word.

3 3.2. Different operators to smooth a training set We want to obtain sequences of letters from, as homogenous as possible, in order to associate them with fuzzy subsets of (constituting a fuzzy partitioning of ). Each subset will represent a linguistic modality of the attribute (for instance small and big with the previous example). We present several techniques to alter a word. Our goal is to erase non-representative values within a word in order to smooth it. We use operators inspired by mathematical morphology. Each rewriting system is given as a transduction, a particular kind of automaton ([9]). a. Transductions Let us recall that a transduction is a 6-tuple 89+;:<+-=>+0?/+0@A+B&C where is the input alphabet, : is the output alphabet, = is a (finite) set of states,?de= is the set of initial states of the is the set of terminal states of the transduction and BIHJ=LKMNKO:PNK3= is the transition function. A transduction reads a word QN M and rewrites it in a corresponding word RS :. It proceeds sequentially from the first letter of Q to the last one, as a reading head moves letter by letter. The rewriting rules to generate R are based on B. Let TVU7W7+-X,+ZY!+-U[*\ B+ with U7W-+-U[ ]=>+-X^HO and Y6H :P, T_Ù W7+0X/+aY!+-U;[!\ is called a transition of the transduction. If U7W is the current state and we can read X in Q (i.e. X is composed by the successive letters coming just after the reading head), we replace it by Y and the current state becomes U [. A convention is to use b to match the end of the input word, and c (the null word) is introduced when nothing has to be written. For example, let d.+-e*(+: f,+*)+>= S+ S2+ S3(+g? S)+h@ S3 and B )T S+-d.+-f,+ S\`+iT S+0e$+c(+ S2\7+iT S2+-d.+-f,+ S\7+iT S2+-e+*&+ S2\`+ T S+0b/+0b/+ S3\7+,T S2+-b,+-b,+ S3\j b / b / ε S2 a / 0 S $ / $ a / 0 $ / $ S3 Initial state Terminal state a / 0 read a and replace it by 0 Figure 4. Example of transduction more to read in Q, the rewriting is done. b. A transduction for the erosion Let us define a with pq:rs. This 4 x u 2 $ u$ x ε x / y y uy 3 Initial state Terminal state read x and replace it by y Figure 5. Transduction for the erosion related to tu u with v^ u vu kt rewriting system is used for the erosion of a word with a particular letter tn u as. It corresponds to the reduction of sequences of t in the word. From the word Q,+wQx y i, we obtain the z 6. For example, with the word QJt{v&vlv&v, to o TVQ&\ we use the transduction given in Fig. 5, and we o T_t{v&v&vlv&\.vlv&vlv. c. A transduction for the dilatation Now, let us define }~o, another rewriting system with G :3z and tn. This system will dilate a sequence in a word when this sequence is surrounded with letters. It can be proven that for any given word, the computed terminal word is unique [2]. Thus, we are sure o T_Ql\ and }~ o T_Q&\ exist for all word QN 6 and for all letter t, and therefore, for all training sets. Moreover, for any word Q I 6, we o T_Ql\ (resp. }~9o T_Q&\ ), with u zf, the word obtained from Q after consecutive erosions (resp. dilatations). Now, with these two rewriting systems we will define the two usual operations from the mathematical morphology: the opening and the closure. A simple visual representation of this transduction is a graphic form (Fig. 4). Let us rewrite Q d(e7e7d(d(e7d(d)e with this transduction. After the sequence of states (S, S, S2, S2, S, S, S2, S, S, S2, S3), we have RIkf/!f&flf&f and, as the current state is S3, a terminal state, and we have nothing d. The opening operator The opening is the composition }~onƒh@nm7o of the two previous operators (i.e. rewriting systems). The -opening ( IN) of a word Q E i with respect to t E is defined as iˆ o TVQ&\ }~ o T_@nm o T_Q&\;\. The -opening of a

4 W ª W ª Ÿ u ε u x 3 2 x y uy 5 u ε 4 $ u$ u u x / y Initial state Terminal state read x and replace it by y Figure 6. Transduction for the dilatation related to tn u with v^ <vu kt word allows to erase small sequences with length smaller than "l. The advantage of this operation is that we can erase all sequences in Q with length smaller than a fixed value. For example, with the word QŠ vlv&t{t.t{v&t{v&t, we have iˆ o TVQ&\vlv&t{t.t{v&{v& and Žˆ. o T_Ql\ }~V o T_@m o T_Q&\;\ }~ o T_vlv&{.{v&{v&{\izvlv&{.{v&{v&. e. A closure operator The closure is the o ƒ^}~ o of the two operators of erosion and dilatation. This composition allows to join sequences of letters in a word if these sequences are separated by less than two letters. The -closure ( IN) with h respect to t of the word Q 6 is defined as o T_Ql\Pz@m* o T_}~o TVQ&\;\. With this operator, two sequences separated by less than "l letters are unified. For example, with the word Q^k{.t{t{.{.t{t{.t{.{ h, we have o T_Ql\P.{t.t{{.{t.t{t{t.{, h and o TVQ&\wL.{t{t.t{t.t{t{t.t{t.{{. Finally, we introduce the filter operator which transforms a word into a sequence of homogeneous series of letters of. In the framework of the utilization of a training set, a filter allows to smooth the training set to deduce a fuzzy partition. f. A filter operator A filter is a composition of the previously described wordtransforming operators. Let Q^ u i, tn u and u IN. The -filter of the word Q with respect to t is defined as: if N : n~ o T_Ql\w h o T_ Žˆ o TVQ&\;\ if ] : n~ o T_Ql\P h o T_ Žˆ,o TV n~ o, T_Q&\;\\ The particular combination of these operators has some interesting properties. A filter will allow to smooth a word. First, the sequences with a length of "& letters are erased (i. e., replaced by the letter ), then we unify sequences separated by "l letters. In the next section, we will use all these operators to describe our main algorithm. 4. Fuzzy partitioning of a set of values In this Section, we present an algorithm to infer a fuzzy partition on a set of numerical values, after the use of the previous rewriting system. When we apply a filter to the word induced by a training set, we are able to translate small sequences of classes into uncertain sequences. To smooth a training set, we apply a -filter to it. Then, we obtain a word with large sequences (with a length larger than "& ). is a value, empirically fixed, given to the system (we use š*f&œ of the size of the training set in our application ). The sequences of represent uncertain sequences where the classes are highly mixed. The sequences consisting of a single letter tž+0t I. These sequences describe crudely speaking a single class; these sequences do not contain any u character. We call them certain sequences, whatever t may be. We will use these two kinds of sequences to build a fuzzy partition of the training set that is related to an attribute. Certain sequences of letter t correspond to the kernels of the fuzzy sets of the partition. Let m be the number of fuzzy modalities we want for the attribute. We select the m largest certain sequences containing one class (Fig. 7). To each sequence, we assign intervals from, for instance Ÿw W +w i o and Ÿw W +;w i o when m s". In the case where we cannot find m such sequences, we can either reduce the number of applied filters, or select fewer sequences. We summarize this in the following algorithm FPMM, Fuzzy Partitioning using Mathematical Morphology, with mhz" : Algorithm (FPMM) To fuzzify a training set with respect to an attribute defined on in " fuzzy subsets. Tranform into a word Q. 2. For fixed, smooth Q. 3. Find the two larger certain sequences and. (resp.w i o 4. We denote by W ) the value associated with the first (resp. last) letter of W in, ~u l+0". Ÿ W + 6 o (resp. Ÿw W +;w 6 o \ with w i o w W. 5. The fuzzy partition is defined as a family of two fuzzy subsets. The kernel of the first one is Aª +;w i o and its support is «yª +;w W. The kernel of the second one is Ÿw i o +- Ÿ and its support is Ÿw W +0

5 ³ ³ u + + u u u u u u u - - uu u u sequence sequence 2 0 ±.áµ ²±.áµ Figure 7. Induced fuzzy partition Thus, we have an algorithm to induce a fuzzy partition from a training set. This algorithm smoothes a word induced by a training set and, from this smoothed word, we propose a way to define a fuzzy partition. 5. An illustration of the algorithm Let be a training set with a numeric attribute (e. g. the age), and two classes + and -. (5, +), (7, +), (8, +), (3, +), (4, -), (7, +), (20, -), (2, +), (22, +), (25, -), (29, -), (30, -), (35, -), (36, +), (38, -), (40, -). We represent in a graphical form (Fig. 8) Age Age Figure 0. A fuzzy partition of the training set The system is based on an extension of the ID3 Algorithm with a fuzzy measure of entropy, the entropy-star [3],[3]. This measure is usable when a set of numerical values is associated with a fuzzy partition over it. Usually, this fuzzy partition is given by an expert of the considered domain. In a project on learning methods, we were supposed to build a fuzzy decision tree for the Breiman s waveform problem [6]. With these data, no expert knowledge was available. Thus, we had to define the fuzzy partitions from the training set and we used the algorithm FPMM. It appears that the results are more interesting with the fuzzy decision tree than with a traditional ID3-based method: fuzzy trees are shorter and generalize better for new cases. We obtain fuzzy trees with an average size of 2 paths and an average number of node ¹#.º¼». In generalization with these trees, we obtain a 72.5% rate of good classified examples. A more detailed description of the application can be found in [4]. Figure 8. A training set To infer a fuzzy partition on the universe of the values of the age, we apply the algorithm FPMM. First, is transformed into a word. Let 3s +, -, the word associated with is Q^ Petal length 3.0 cms cms u + + u u u u u u u - - uu u u Age Figure 9. The filtered word We filter Q with yš, and we obtain the filtered word on (Fig. 9). Two non uncertain sequences appear in the filtered word, a sequence of + and a sequence of -. We use them as the basis of kernels of two fuzzy subsets (Fig. 0). 6. Application We implement the algorithm to infer fuzzy partition for numerical attributes during the construction of a decision tree. Figure. The root node of the decision tree constructed from the iris training base Other tests were conducted on the Iris data of Fischer (Fig. ) (available at the UCI repository, ftp://ftp.ics.uci.edu). With a cross validation test, we obtain fuzzy trees with an average size of 0 paths and an average number of node ½#.ºž". In generalization with these trees, we obtain a 95.33% rate of good classified examples. 7. Conclusion In this paper, we propose a new algorithm to infer a fuzzy partition for the universe of a set of numerical values, when

6 each of these values is associated with a class. This algorithm is based on several rewriting systems which are represented as transductions. Each of these rewriting systems is based on a mathematical morphology operator. We define an algorithm to reduce an arbitrary sequence of letters and an algorithm to enlarge a sequence of letters. When we compose these two algorithms, then, depending on the order of the composition, we obtain two general operators. These two operators allow us to control how small mixed sequences are erased. Finally, we obtain an algorithm to smooth a word induced by a training set, and, from this word we propose a way to define a fuzzy partition. This method is implemented in our decision tree builder program to help the construction of decision trees when no fuzzy partition from an expert is available. In our future work, we propose to take into account another dimension of the set of values, which is the distance between two consecutive values, and to adapt the fuzzy mathematical morphology to this kind of problems. [] J. R. Quinlan. Induction of decision trees. Machine Learning, ():86 06, 986. [2] J. R. Quinlan. C4.5: Programs for Machine Learning. Morgan Kaufmann, San Mateo, Ca, 993. [3] M. Ramdani. Système d induction formelle à base de connaissances imprécises. Thèse de l Université P. et M. Curie, Rapport TH94/, LAFORIA-IBP, 994. [4] J. Serra. Image Analysis and Mathematical Morphology. Academic Press, New York, 982. [5] Y. Yuan and M. J. Shaw. Induction of fuzzy decision trees. Fuzzy Sets and systems, 69:25 39, 995. References [] N. Aladenise and B. Bouchon-Meunier. Acquisition de connaissances imparfaites : mise en évidence d une fonction d appartenance. Revue Internationale de Systémique, 996. [2] B. Bouchon-Meunier, C. Marsala, and M. Ramdani. Arbres de décision et théorie des sous-ensembles flous. In Actes des 5èmes journées du PRC-GDR d Intelligence Artificielle, pages 50 53, 995. [3] B. Bouchon-Meunier, C. Marsala, and M. Ramdani. Inductive learning and fuzziness. Scientia Iranica, 996. [4] B. Bouchon-Meunier, C. Marsala, and M. Ramdani. Learning from uncertain and imprecise examples. In M. E. Cohen and D. L. Hudson, editors, Proceedings of the th Int. Conf. on Computers and their Applications, pages 75 78, San Francisco, March 996. [5] B. Bouchon-Meunier, C. Marsala, and M. Ramdani. Learning from imperfect data. In H. P. D. Dubois and R. R. Yager, editors, Fuzzy Sets Methods in Information Engineering: a Guided Tour of Applications. John Wileys and Sons, 996, to appear. [6] L. Breiman, J. H. Friedman, R. A. Olshen, and C. J. Stone. Classification And Regression Trees. Chapman and Hall, New York, 984. [7] B. Cestnlk, I. Kononenko, and Y. Bratko. Assistant86: a knowledge elicitation tool for sophisticated users. In Y. Bratko and N. Lavrac, editors, Progress in Machine Learning, Proceedings of EWSL, pages 3 45, 987. [8] M. Coster and J. L. Chermant. Précis d analyse d images. Presses du CNRS, 989. [9] S. Ginsburg. The Mathematical Theory of Context Free Languages. McGraw-Hill, New-York, 966. [0] R. Krishnapuram. Generation of membership functions via possibilistic clustering. In Proceedings of the 3rd IEEE Int. Conf. on Fuzzy Systems, volume 2, pages , Orlando, Florida, June 994.