Bottom-Up Fuzzy Partitioning in Fuzzy Decision Trees

Transcription

1 Bottom-Up Fuzzy arttonng n Fuzzy eson Trees Maej Fajfer ept. of Mathemats and Computer Sene Unversty of Mssour St. Lous St. Lous, Mssour maejf@me.pl Cezary Z. Janow ept. of Mathemats and Computer Sene Unversty of Mssour St. Lous St. Lous, Mssour janow@umsl.edu Abstrat FI s a publly avalable fuzzy deson tree software for lassfyng fuzzy data. T paper desrbes a new doman parttonng tehnque, bottom-up, whh has just been mplement to omplement the prevously avalable topdown tehnque. 1. Introduton eson trees are one of the most popular methods for learnng and reasonng from feature-based examples [7]. However, they often have been rtzed for ther persstent over-relane on near-perfet data, and for the resultng degradaton n the presene of mperfet data. ata mperfeton mght have been the result of nose, mprese measurements, subjetve evaluatons, nadequate desrptve language, or mssng data. Addtonal problems arse from ontnuous or smply large nomnal attrbutes - all suh domans have to be parttoned. Some of these potental problems have been suessfully addressed n the past. For example, Qunlan has proposed some methods for dealng mssng features both n tranng data and n the examples to be lassfed [8]. Contnuous domans have been addressed by CART [1] and subsequently by C4.5, along wth tree prunng tehnques [9]. A more reent method s to ombne fuzzy representaton, and n partular ts ablty to provde omprehensble desrptve language, and ts approxmate reasonng tehnques, wth deson trees. The result s a fuzzy deson tree, suh as that desrbed n [3]. The methodology onssts of three elements. Frst, there are two doman parttonng methods: one s top-down, reatng loal and mnmal parttonng needed for generatng a fuzzy tree, the other s bottom-up, reatng more global parttonng. Here, doman parttonng s performed pror atual tree generaton. Seond, there s a proedure for buldng fuzzy deson tree, whh tree an also be nterpreted as a set of fuzzy rules. The tree an be bult usng a number of potental fuzzy norms. Fnally, there are a number of nferene rules, for assgnng lassfatons to new samples - based on the nformaton extrapolated from the tree. The nferenes fall nto two bas ategores: set-based (followng loal nferenes n fuzzy rules) and exemplarbased (followng exemplar-based learnng). The software an handle a mxture of features: symbol, fuzzy terms, and numer, and t an reason under nomplete/mssng nformaton. T paper desrbes the new bottom-up doman parttonng tehnque. The prevously avalable tehnque was amed at produng the mnmal set of parttons loally neessary for mnmzng tree sze [4]. As suh, t was a topdown tehnque, mplementng doman splttng rules whle buldng a tree. However, the fuzzy deson tree has just been extended to a deson forest (desrbed separately), whh uses redundant nowledge for performng more elaborate lassfaton. T redundany requres more global rather than mnmal loal parttonng. Suh a doman parttonng s presented here. 2. Fuzzy eson Trees eson tree methods use reursve parttonng proedures to buld deson trees. Subsequently, they use mathng nferene proedures for lassfaton of new samples. I3 [7], and ts suessor C4.5 [9], along wth CART [1], are the two most wdely used deson trees. Ther bas deas are the same: partton the sample spae n a datadrven manner, and represent the partton as a tree. An mportant property of these algorthms s that they mpltly attempt to mnmze the sze of the tree whle optmzng some loal qualty measure - suh as entropy or gn ndex [1][7][9]. The tree s onstruted n a data-drven algorthm. Eah node n the tree represents a subspae of the event spae, and thus the whole tree s a parttonng proedure on the event spae. In eah node, the parttonng ontnues by seletng the best deson (an attrbute or a relaton) to further partton the subspae. When a sngle attrbute s seleted for the splt deson, the most ommonly used

2 seleton method s nformaton gan, whh s omputatonally smple and effetve: selet an attrbute for testng (or a new threshold on a ontnuous doman) suh that the nformaton dfferene between that ontaned n a gven node and n ts hldren nodes (resultng from splttng aordng to those tests) s maxmzed. The nformaton ontents s measured aordng to [7]: I N C = ( p log p ), where C s the set of desons, = 1 and p s the probablty that a tranng sample n the node represents lass. 1. The root of the deson tree ontans all tranng examples. It represents the whole desrpton spae sne no restrtons are mposed yet. 2. Wor wth any node N. The node beomes a leaf when ether ts samples ome from a unque lass, when all attrbutes are used on the path leadng to the node, or when possbly nformaton n the node beomes too unrelable (e.g., when too few examples are found). roeed only when deded to further splt the node. 3. Compute the nformaton ontent at the node N. Then, for eah attrbute a not appearng on the path to N and for eah of ts doman values a j, ompute the nformaton ontents n hldren nodes restrted by the addtonal ondton a =a j. Subsequently, ompute a ombned weghted nformaton ontents n all the hldren, and the resultng gan wth respet to N. Note that for the not-pre-parttoned domans, the algorthm tres all applable thresholds, seletng the best one - t results n a bnary test (note that suh attrbutes an appear more than one on a gven path, wth dfferent bnary test). 4. Selet the attrbute maxmzng the gan, and subsequently splt node N usng the applable tests. Afterwards, deson trees use the same bas nferene mehansm for lassfyng new samples. Features of a sample are mathed aganst the tests of the tree, startng from the root and desantng along the mathng path. The sample s lassfed aordng to the lassfaton of the leaf that t reahes. Fuzzy deson trees dffer n two respets: they use splttng rtera based on fuzzy restrtons, and ther nferene proedures are dfferent. Fuzzy sets defnng the fuzzy terms used for buldng a tree are mposed on the algorthm (or generated n the doman parttonng stage). FI s a wdely used publly avalable fuzzy deson tree. It an handle data desrbed by varous nds of attrbute domans. For example, the same data an be desrbed by a mxture of nomnal attrbutes and attrbutes wth ontnuous domans - whh an be preparttoned by the user or not. The same mxture of values an be used as lasses values. Eah pee of data s also augmented wth a onfdene weght (dsregarded here n subsequent presentaton). If some domans la parttonng, the algorthm performs the preproessng, n whh ether the top-down or the bottom-up tehnque s appled to partton suh domans. After the preproessng, the fuzzy deson tree s onstruted smlarly to the standard deson tree, wth a reursve depth-frst proedure. However, there are a number of subtle dfferenes: 1. ata samples may math more than one test of a node. When aggregated over multple levels, t leads to samples fallng nto many nodes, wth a real-valued degree (based on the aggregated math to the fuzzy restrtons on the path). 2. The nformaton ontents formula s modfed to reflet partal membershps (n addton to allowng absent features). For detals, see [3]. 3. Fuzzy math s determned based on preseleted norms, or by seletng best norms from a predefned set. However, the most profound dfferenes are n the proess of lassfyng a new sample. These dfferenes arse from the fat that FI trees have leaves that are more lely to ontan samples of dfferent lasses (wth dfferent degrees of math), the nferene proedure s lely to math the new sample aganst multple leaves, wth varyng degrees of math. Fgure 1 Illustratons of the nowledge of two FI trees traned for the mexan sombrero funton

3 To aount for these potental problems, a number of nferenes routnes have been proposed. Some nferenes follow the deas of approxmate reasonng [3], other follow mahne learnng prnples of exemplar learnng [4]. Some of these nferenes have more global harater, some are more loal and behave le nose flters [3]. Whatever spef nferene s used, the outome s a value from the doman of the lass varable. FI trees have been shown to be apable of produng nowledge whh s both omprehensble yet apable of generatng fner levels of detal - dependng of the atually used nferenes. For more nformaton, see [3]. As an llustraton of the desrptve power of FI trees, onsder the well nown mexan sombrero funton [10]. When the FI tree s traned wth data samples from a 13x13 grd, usng domans wth predefned 13 fuzzy terms, two of ts nterpretatons, followng two dfferent nferenes, are llustrated n Fgure Fuzzy arttonng If at least one attrbute does not have a predefned fuzzy parttonng (does not have the lngust doman), datadrven preproessng s nvoed n order to partton suh domans for relevant attrbutes (and not neessarly all suh attrbutes). All attrbutes wth predefned parttons retan ther gven parttons. The remanng attrbutes are parttoned wth fuzzy sets. There are two dfferent methods avalable. One, reported prevously [4], only parttons attrbutes relevant to buldng a deson tree. Below we desrbe the new method, whh attempts to partton all domans n a more global way. 4. Bottom-up arttonng The bottom-up parttonng s a global data-drven parttonng strategy. It s amed at parttonng all ontnuous domans. Moreover, based on some user parameters, eah doman may be fored to have a number of parttons n some predefned range (between mnnumlngvals and maxnumlngvals). The proess starts wth a maxmal parttonng (at the data level, that s eah attrbute-value s treated as an ndvdual partton of the gven doman) and then uses heursts to generalze that parttonng. The algorthm onssts of three man stages: lusterng, safe mergng and mergng. In the ntal lusterng stage, every data event s assgned to an ndvdual luster. Then teratvely, two nearest lusters, usng some dstane measure, are joned as long as a gven error measure s not exeeded. The lusterng stage ends wth projetng the lusters onto the domans. T reates ntal parttonng for the mergng stage. Safe mergng ombnes neghborng parttons only f they ontan data of exatly the same lassfaton. It s followed wth the proper mergng, n whh entropy measure and error rate are both used to selet parttons to be joned. The proess stops when no more jons an be made under the allowed error rate. T algorthm s an extenson of a smlar algorthm proposed by Grzymala-Busse [2]. 4.1 Clusterng The set of tranng examples s E = { e j e j = ( u 1 j, u n j, y j )}, where y j s the fuzzy lass or ontnuous deson value, j = 1, N, where N s number of examples and n s the number of attrbutes used n dsretzaton (only those subjet to dsretzaton). The set of lusters s C = { }, = 1, m, where m s the number of lusters. Fgure 2 Illustraton of projeton of lusters. U A V 2 B A B B B C L A A 1. We start wth N lusters. Eah luster onssts of one example and t s a pont n the hyperspae defned by the set of attrbutes. 2. Compute the dstane between lusters and j as: d j = n ( u u j ) 2 3. Jon two losest lusters A, B nto AB, and ompute new dstanes to all remanng lusters usng the followng formula d ( C, AB ) d + A d B d AB, = AB For eah luster, ompute the error rate E max, ( ) = L B B C R B V 1

4 5. If the global error rate E = E s greater C C C than a parameter ClusterStop, stop lusterng, undo the last jon and go to the next step, otherwse go ba to step rojet the lusters onto domans and reate parttons. Fgure 2 llustrates parttons for attrbute V 1 : [ L A, L B )[ L B, R B ], and V 2 : [ B C, B B )[ B A, U A ], generated by lusters A, B. To defne parttons we use the mnmal boundary of every luster on a gven attrbute and the maxmal boundary of all maxmal boundares (sppng lusters whh defne subdomans of any other luster). 4.2 Safe Mergng Safe mergng s the proess of jonng neghborhood parttons under the ondton that they ontan examples belongng to exatly the same lass. For example, parttons R 1 and R 2 n Fgure 3 would be merged f they ontaned examples of exatly the same lass. We repeat the proess for all possble unons. Fgure 3 Safe Mergng. number of parttons s stll greater than the parameter mnnumnlngvals. I R, R j 3. Fnd the unon wth mnmum over all attrbutes. 4. Jon parttons and j. Compute error rate for all hyper subspaes defned by parttons on all attrbutes E max, C ( ) = and global error rate E = E HS HS 5. If E s less than parameter MergeStop then aept the unon, update needed entropes and go to step 3. Otherwse, undo the unon, selet next unon wth mnmum I R, R j, and go to step 4. If no more unons an be seleted, go to step If there are attrbutes on whh the number of parttons s greater than parameter maxnumlngvals then ontnue mergng these attrbutes as long as needed, regardless of the error rate. 4.4 Creaton of Fuzzy Terms on arttons V 1 V 2 V 3 V 4 R 1 R 2 R 3 R 4 Let us assume that we have n parttons R 1, R n on a gven attrbute, where R = ( a, b ). artton R generates a fuzzy set, at present trapezodal, n suh as way the fuzzy set ntersets neghborng sets (for R 1 and R + 1 ) at a and, respetvely (exept for the frst and the last sets). b 5. Summary 4.3 Mergng Mergng s the fnal jonng proess, where parttons ontanng dfferent lasses an be joned - the proess s drven by heursts (entropy) and error measures. 1. Compute entropy n every partton : R R I R = log R R 2. For all possble unons ompute the resultng entropy as I R, R j R I R R j I R j =. R + R j ( ) R + R j ( ) However, unon on a gven attrbute s possble f the R We have desrbed a method to partton ontnuous or large-valued domans nto fuzzy sets. The method s data drven and eah doman s parttoned n a data-drven manner. The method s bottom-up as t starts wth maxmal parttons, whh are subsequently refned and merged. The method s espeally applable to just released upgrade to the fuzzy deson forest, whh methodology reles on redundant nowledge and thus requres parttonng of all attrbutes. Therefore, the resultng parttonng are evaluated n the ontext of the deson forest, reported separately. A publ release of FI4.0 s avalable at

5 Referenes [1] L. Breman, J.H. Fredman, R.A. Olsen & C.J. Stone. Classfaton and Regresson Trees. Wadsworth, [2] M.R. Chmelews and J.W. Grzymala-Busse. Global sretzaton of Contnuous Attrrbutes as reproessng for Mahne Learnng. In T.Y. Ln and A.M. Wlderberger, (eds), Soft Computng: Rough Sets, Fuzzy Log, neural Networs, Unertanty Management, Knowledge sovery, 1995, pp [3] C.Z. Janow. Fuzzy eson Trees: Issues and Methods, IEEE Transatons on Systems, Man, and Cybernets, Vol. 28, Issue 1, pp. 1-14, [4] C.Z. Janow and M. Fafer. Fuzzy arttonng wth FI3.1. roeedngs of the 18th Internatonal Conferene of the North Ameran Fuzzy Informaton Soety, IEEE 1999, pp [5] C.J. Merz,.M. Murphy. Repostory of mahne learnng databases. Unv. of CA, ept. of Informaton and Computer Sene, [6] R.S. Mhals. Understandng the Nature of Learnng. In Mahne Learnng: An Artfal Intellgene Approah, R. Mhals, J. Carbonell & T. Mthell (eds.), Vol, II, pp Morgan Kaufmann, [7] J.R. Qunlan. Induton on eson Trees. Mahne Learnng, Vol. 1, 1986, pp [8] J.R. Qunlan. Unnown Attrbute-Values n Induton. In roeedngs of the Sxth Internatonal Worshop on Mahne Learnng, 1989, pp [9] J.R. Qunlan. C4.5: rograms for Mahne Learnng. Morgan Kaufmann, San Mateo, CA [10] I. Suh, Hong & T.W. Km. Fuzzy Membershp Funton Based Neural Networs wth Applatons to the Vsual Servong of Robot Manpulators. IEEE Transatons on Fuzzy Systems, Vol. 2, No. 3, 8/1994, pp