DYNAMIC DISCRETIZATION: A COMBINATION APPROACH

Transcription

1 DYNAMIC DISCRETIZATION: A COMBINATION APPROACH FAN MIN, QIHE LIU, HONGBIN CAI, AND ZHONGJIAN BAI School of Computer Science nd Engineering, University of Electronic Science nd Technology of Chin, Chengdu , Chin Emil: {minfn, qiheliu, cihb, bizj}@uestc.edu.cn Abstrct: Supervised discretiztion refers to the problem of trnsforming continuous ttributes of decision tble into discretized ones. It is importnt for some Artificil Intelligence theories where nominl dt re required or preferred. Insted of depending on the experience of humn experts, supervised discretiztion lgorithms lern from the dt. However, the results of such lgorithms my be sensitive to the chnge of the dt. In this pper, we propose to compute more stble nd informtive discretiztion schemes through subtble smpling nd scheme combintion. Discretiztion schemes computed in this wy re clled dynmic discretiztion schemes. Experimentl results on some well known dtsets show tht they re helpful for obtining decision rules with better ccurcy nd F-mesure. Keywords: Rough Sets, discretiztion scheme, subtble 1. Introduction Some Artificil Intelligence theories such s Decision Trees [1], Rough Sets [2] nd Forml Concept Anlysis [3] use dt tbles, especilly decision tbles s the centrl tool for the development of decision ids. Those theories require, however, ttribute vlues to be numericlly or nominlly discrete. In the cse of continuous ttributes, there is need for discretiztion lgorithm tht trnsforms continuous ttributes into discrete ones [4] using ttributevlue pirs clled cuts. Even in systems tht re ble to hndle continuously vlued dt directly, e.g., C4.5 [5] nd LERS [6], explicit dt discretiztion my still be proven dvntgeous [7]. A number of successful supervised discretiztion lgorithms hve been proposed. They discretize ech ttribute by tking into ccount the interdependence between clss lbels nd the ttribute vlues [4]. Generlly, these lgorithms fll into two ctegories: locl pproches nd globl pproches [8]. Locl pproches discretize ech ttribute independently, nd their difference lies on the discretiztion criterion employed, e.g., CAIR [9], CAIU nd CAIM [4]. Globl pproches discretize ll ttributes simultneously. For exmple, Nguyen [8] converted the discretiztion problem into the reduct problem of Rough Sets nd proposed n efficient lgorithm clled MD-heuristic to obtin suboptiml (sub-miniml) discretiztion schemes. According to empiricl study on mny dtsets, this pproch hs quite good performnce compred with mny other pproches [8]. However, supervised discretiztion lgorithms, especilly globl pproches re not stble. Adding or removing few objects from the decision tble my result in drmtic chnge of the discretiztion scheme. Moreover, the discretiztion process inevitbly result in informtion loss, nd minimizing the discretiztion scheme my mke such loss uncceptble. In this pper, we ddress these issues through introducing the concept of dynmic discretiztion scheme nd proposing two definitions to evlute the stbility of discretiztion scheme.. We do not try to develop new discretiztion lgorithm from scrtch. Insted, we propose to firstly smpling fmily of subtbles from the given decision tble, then compute discretiztion schemes of subtbles using MD-heuristic, nd finlly combine these schemes to obtin new scheme, clled dynmic discretiztion scheme. During the combintion process, it is convenient for us to reserve excessive cuts which re more importnt from sttisticl point of view, thus mking the discretiztion scheme more informtive. It should be noted tht MDheuristic cn be esily replced with ny supervised discretiztion lgorithms. Experimentl results on some UCI dtsets show tht dynmic discretiztion schemes computed by our pproch re helpful for obtining decision rules with better ccurcy nd F-mesure thn tht of MD-heuristic. 2. Preliminries In this section we emulte some bsic concepts. For more detil plese refer to Nguyen [10] nd Komorowski et 1

2 l [11] Decision Tbles Formlly, decision tble is pir S = (U, A {d}) where d A is clled the decision ttribute nd elements of A re clled conditionl ttributes. : U V for ny A {d} where V is the set of ll vlues of clled the domin of. An exemplry decision tble is listed in Tble 1. Tble 1. A decision tble S U d x x x x x Tble 2. The discretized decision tble S P where P = {( 1, 1.4), ( 1, 1.6), ( 2, 0.3)} U p p d p x x x x x Discretiztion Schemes Assume tht V = [l, l b ] R is rel intervl for ny A. Any pir (, c) where A nd c R is clled cut on V. Let P be prtition on V (for A) onto subintervls P = {[ c0, c1 ),[ c1, c2),...,[ ck, c )} where k + 1 l = c0 < c1 <... < ck < c k = r nd + 1 V = [ c0, c1 ) U[ c1, c2) U... U [ ck, c ). Hence ny prtition k + 1 P is uniquely defined nd often identified s the set of cuts: {( c, 1),( c, 2),...,( c, k )} A R. Any set of cuts P = U AP defines from S = (U, A {d}) new decision tble S P = (U, A P {d}) clled P-discretiztion of S, where A P = { P P (x) = i (x) [c i, c i+1 ] for x U nd i {0,, k }}. P is clled discretiztion scheme of S. Tble 2 lists discretized decision tble of Tble Reltive Works In this section we nlyze Nguyen s MD-heuristic nd point out some of its dvntges nd disdvntges. It should be noted tht lthough the ide of subtble smpling is similr with dynmic reduct [12], the essence of our pproch is quite different. Nguyen ddressed the problem of discretiztion s [s]erching for optiml consistent set of cuts, where optimiztion criteri were defined by number of cuts (OD-problem), nd consistency mens preserving the discernibility reltion between objects from different decision clsses [8]. Then he proposed n lgorithm clled MD-heuristics using boolen resoning to obtin semi-miniml set of cuts. This lgorithm is both efficient (the time complexity being O(kn( P + logn)) nd spce complexity being O(kn), where k is the number of conditionl ttributes nd n is the number of objects) nd effective (s will be shown in Section 5 through experiments). It is lso implemented in publicly vilble softwre RSES 2.2 [13]. Although the specifiction of OD-problem, especilly the conversion of the discretiztion problem into the reduct problem seems perfect in theory, it hs number of disdvntges in prctice. First, if discretiztion scheme is irreducible, then removing ny cut from the discretiztion scheme will cuse the discretized decision tble to be inconsistent. As pointed out by Nguyen in his lter work [10], [o]ur heuristic for OD-problem clled MD-lgorithm produces new decision tble with one reduct only. In some pplictions bsed on Rough Sets (e.g. dynmic reduct nd dynmic rule methods), where reducts re n importnt tool, it is not enough to obtin the strong rules. Second, being consistent mens noise in the source decision tble cnnot be eliminted in the discretized decision tble. Reserchers pointed out tht lower this requirement re helpful for voiding overfitting nd improve predicting ccurcy [14]. Third, discretiztion schemes computed re sensitive to the chnge of the dt. In fct, by removing 5% objects from the WDBC dtset [15] we will obtin wholly different discretiztion scheme. Nguyen then proposed the s-optiml discretiztion problem (s-od problem) with the purpose of [o]btining more excessive set of cuts producing new discretized decision tble contining more reducts, nd, t the sme time, reducing the superfluous informtion. But this problem seems rther hrd. 2

3 4. Dynmic Discretiztion Generlly, our pproch is composed of three steps: 1, rndomly smpling fmily of subtbles from given decision tble; 2, compute discretiztion schemes of subtbles; nd 3, compute dynmic discretiztion scheme. We choose RSES 2.2 [13], specificlly the split in two function nd MD-heuristic lgorithm to ccomplish the first two steps. In this section, we suppose tht we hve obtined fmily of discretiztion schemes, one for ech subtble. Specificlly, the discretiztion scheme of subtble B is denoted by P B. Our focus is the lst step. 1 2 k 2, i j i= 1 DIS( CD, CD ) = ( x x ) where CD 1 nd CD 2 cn represent distribution vectors of either one subtble or fmily of subtbles. (5) 4.1. Averge Cut Support It is nturl to expect tht different discretiztion schemes my contin the sme cut. We use the following definition to evlute the importnce of cut: Definition 1 The support of cut ct in subtble fmily F is defined by the number of discretiztion schemes contining it, i.e., CP( F, ct) = { PB B Fnd ct PB}. (1) Obviously, discretiztion schemes contining cuts with lrger support re more preferred. Thus we cn use the following definition to evlute the stbility of discretiztion scheme P: Definition 2 The verge cut support of P is defined by CP( F, ct) ct P SC( F, P) = (2) P 4.2. The Cut Distribution Vector From sttisticl point of view, how mny cuts is pproprite for given ttribute? To nswer this question better, we cn firstly investigte the distribution of cuts. Let CD( PB ) = [ P 1, P2,..., Pk ], (3) or CD B for briefness denotes the cut distribution vector. For exmple, for decision tble listed in Tble 1, suppose tht P S = {( 1, 1.2), ( 2, 0.25), ( 3, 0.35)} then CD A = [1, 0, 2]. Furthermore, we cn denote the cut distribution vector of subtble fmily F by CD B F B CDF =. (4) F Let CD 1 = [x 1, x 2,, x k ] nd CD 2 = [y 1, y 2,, y k ] be two cut distribution vectors, the distnce between these two vectors is given by their Euclid distnce, nmely, Figure 1. Chnge speed of the cut distribution vector of WDBC As we dd more subtbles with the sme size into F, it is expected tht CD F generlly converge, i.e., it does not chnge much ny more. Let F i (i 1) denote the subtble fmily contining the first i subtbles, we use 2 2 DISi = DIS ( CDF, CD ) (6) i F i+ 3/2 to specify the chnge speed of the cut distribution vector. Figure 1 illustrtes the chnge speed of cut distribution vector of the WDBC dtset [15]. If DIS 2 i is stbly below threshold, we cn terminte the process of subtble smpling. This threshold should be, however, ppliction dependent. Let n be the number of subtbles, in order to meet these requirements we require tht: N M 1 DIS (7) 2 j= 3 j mx DIS ε, n j j= 0 N where ε is user specify fctor such s 0.1 or 0.05, nd this fctor cn be constnt, i.e., it is ppliction independent. M nd N re lso user specified constnts, usully it is quite enough to let M = N = 5. Tble 3 lists the miniml number of subtbles required for some dtsets tken from the UCI librry, where Hert-D denotes hert disese, nd ech subtbles hs 80% objects of the given decision tble. 3

4 Tble 3. Miniml number of subtbles required for some dtsets ε WDBC IRIS Hert Hert- Dibetes D Austr lin Obviously, DIS(P, CD F ) cn be lso used to evlute the stbility of discretiztion scheme P Dynmic Discretiztion Scheme Dynmic discretiztion scheme is generl concept, the key ide lies in subtble smpling nd discretiztion scheme selection/combintion. Mny possible definitions of dynmic discretiztion scheme exist. For exmple, it could be the discretiztion scheme of one subtble B with the lrgest verge cut support SC(F, P B ), or with the best distribution vector, i.e., DIS(CD B, CD F ) is miniml. According to our experiments in some UCI dtsets, in most cses it is hrd or even impossible to obtin discretiztion scheme with both the lrgest verge cut support nd the best distribution vector. Moreover, even the discretiztion scheme finlly obtined is stble enough from both viewpoints, it still hs, however, the one reduct problem mentioned in Section 3. To meet ll these requirements, we propose construct new discretiztion scheme by tking into considertion both the verge cut support nd the cut distribution vector s follows: 1. Compute P = P S ; 2. Add cuts with currently the lrgest support of respective ttributes into P until DIS(CD(P), α CD F ) is minimized, where α > 1 is user specified constnt clled the redundnt fctor. We cll this combintion pproch to dynmic discretizion (CADD). In this pproch, the first step ensures tht the discretiztion scheme obtined be consistent with A. The second step blnces the number of cuts for ech ttribute, nd t the sme time ensures the lrgest verge cut support. α > 1 is required to obtin excessive cuts. Formlly, let CD F = [v 1, v 2,, v k ], since P is vector of integers, we hve CD( P) = [ α v ], α v ,..., α v k (8) For exmple, in the IRIS dtset CD F =[1.48, 0.85, 2.76, 1.33]. Let α = 1.5, then α CD = [2.22, 1.275, 4.14, 1.995], nd CD(P) = [2.1, 4.2]. In some extreme cses Eqution (8) cnnot be stisfied. For exmple, it is required tht P hs only 1 cut on 2, but P S hs lredy 2 cuts on 2 ; or it is required tht P hs 4 cuts on 3, but ttribute 3 hs only 3 possible cuts. In the former cse we do not delete cuts on 2, while in the ltter cse we do not dd more cuts. 4.4 Complexity Anlysis Let n denotes the number of objects in ech subtble, the time complexity of the subtble smpling step is O( F n ). In order to clculte the support of ech cut nd determine the best scheme, we should scn ech discretiztion scheme only once, hence the time complexity of the scheme combintion step is O ( F P ), where P denotes the verge number of cuts for ech subtble. The overll time complexity is determined by the subtble discretiztion step where MD-heuristic is employed. Hence it is O ( F ( kn' ( P + log n' ))), (9) which is firly low since in most pplictions F < 100. Moreover, both the subtble smpling step nd the subtble discretiztion step cn esily run in prllel on different mchines, in which cse the time complexity cn be reduced to O(kn( P + log n)), (10) where n is the number of objects in the initil decision tble S, nd P = P S is the discretiztion scheme of S. 5. Experiments This section presents firstly the process of our experiments, then experiment results for some UCI dtsets The Experiment Process Our experiments were conducted using RSES 2.2 nd our progrms. The experiment process is listed s follows: Step 1. Preprocessing. Step 1.1 Lod the decision tble A from the file. Delete irrelevnt ttributes if ny, e.g., ID number of the WDBC dtset; Step 1.2 Specify the decision ttribute of A; Step 1.3 Split A into two subtbles: the trining tble AR nd the testing tble A T ; Step 2. Dynmic discretiztion. Step 2.1 Generte fmily of subtbles, denoted by F, of AR. The number of subtbles is computed s indicted in Subsection 4.2; 4

5 Step 2.2 Obtin discretiztion schemes of subtbles using MD-heuristic (the generte cuts function of RSES 2.2); Step 2.3 Obtin dynmic discretiztion scheme using CADD; Step3.Test the performnce of the dynmic discretiztion scheme; Step 3.1 Discretize AR nd A T using the dynmic discretiztion scheme; Step 3.2 Clculte decision rules using the discretized tble of AR nd certin rule genertion lgorithm. Rules re shortened to obtin better generliztion bility nd prediction ccurcy; Step 3.3 Apply the obtined rule set to discretized tble of AT nd obtin respective coverge nd ccurcy; Step 4. Performnce comprison. Step 4.1 Obtin the discretiztion scheme of AR using MD-heuristic; Step 4.2 Test the performnce of the discretiztion scheme using process similr with tht of Step 3; Step 4.3 Compre results of obtined in Step 3.3 nd Experiments with Dt The overll experiment pproch is the stndrd 5-Cross Vlidtion. A subtble is esy to obtin using the split in two function provided by RSES 2.2. Different from Bzn s pproch [12], subtbles with the sme size (80% of the given decision tble in our experiment) re employed becuse this my be helpful for obtining similr discretiztion schemes in different subtbles, which in turn produce cuts with good support. The rule genertion method is LEM2 with the coverge prmeter set to 0.9. Rule shortening rtio is 0.9 for WDBC nd IRIS, nd 0.8 for others. α = 1.2 for Hert-D, 1.1 for Austrlin, nd 1.5 for others. We do not nlyze the settings nd other detiled issues since they re rther ppliction dependent. The lst point to be noted is tht rule shorten employed in Step 3.2 is very importnt in obtining high coverge. Experiment results in terms of coverge, ccurcy nd F-mesure re listed in Tble Anlysis of the Results For most dtsets CADD produces better results in terms of ccurcy, F-mesure nd stbility (evluted by stndrd devition, std) thn those of MD-heuristic. Specificlly, the decrese of the stndrd devition vlidtes our clim tht dynmic discretiztion schemes re more stble. Compred with MD-heuristic, the verge increse of ccurcy nd F-mesure for these dtsets nd 1.38% nd 1.33%, respectively. One my sy tht the performnce increse is not quite obvious. But we rgue this is vluble since it is compred with n dvnced discretiztion lgorithm. Also note tht the performnce increse of CADD is more obvious for smller dtsets such s IRIS. This is becuse tht MD-heuristic is more sensitive to the chnge of dt in such sitution. Tble 4. Comprison of CADD with MD heuristic using six dtsets Criterion Discretiztion Dtset Method IRIS WDBC Hert Hert-DDibetsAustrlin Coverge men MD-heuristic std men CADD std ccurcy men MD-heuristic std men CADD std men MD-heuristic std F-mesure men CADD std Conclusions In this pper we proposed combintion pproch to dynmic discretiztion, which is imed t obtining more stble nd informtive discretiztion schemes. We ddress the stbility metrics from view points of verge cut support nd the cut distribution vector. In fct, we cn obtin mny similr pproches through replcing some prts of this pproch, for exmple, the discretiztion lgorithm of subtbles nd the stbility definition (especilly Eqution (2)). Some user specified fctors such s α used in Eqution (8) lso deserve in-depth study. Moreover, the vlidity of dynmic discretiztion schemes should be further tested using other kinds of rule genertion lgorithms. Acknowledgement Fn Min ws supported by n informtion distribution project under grnt No. 9140A DZ223 nd the Youth Foundtion of UESTC. Hongbin Ci ws supported by the Youth Foundtion of UESTC. 5

6 The uthors would like to thnk Zichun Zhong, Xuelei Xu, Xioping Xu, Ji Chen, Bei Hui, Qing Zheng, Xuefeng Lio, Song Chen nd Huiyu Men for their help in experiments nd pper proofing. References [1] J. R. Quinln, Induction of decision trees, Mchine Lerning, vol. 1, pp , [2] Z. Pwlk, Rough sets, Interntionl Journl of Computer nd Informtion Sciences, vol. 11, pp , [3] R. Wille, Restructuring lttice theory: n pproch bsed on hierrchies of concepts, Ordered Sets, pp , [4] L. A. Kurgn nd K. J. Cios, CAIM discretiztion lgorithm, IEEE Trnsctions on Knowledge nd Dt Engineering, vol. 16, no. 2, pp , [5] J. R. Quinln, Ed., C4.5 Progrms for Mchine Lerning. Sn Mteo, Cliforni: Morgn kufmnn Publisher, [6] M. R. Chmielewski, J. W. Grzymł-Busse, N. W. Peterson, nd S. Thn, The rule induction system LERS version for personl computers, Foundment Computing Decision Science, vol. 18, no. 3, pp , [7] R. Susmg, Anlyzing discretiztions of continuous ttributes given monotonic discrimintion function, Intelligent Dt Anlysis, vol. 1, pp , [8] H. S. Nguyen, Discretiztion of rel vlue ttributes, boolen resoning pproch, Ph.D. disserttion,wrsw University, Wrsw, Polnd, [9] J. Y. Ching, A. K. C.Wong, nd K. C. C. Chn, Clssdependent discretiztion for inductive lerning from continuous nd mixed-mode dt, IEEE Trnsctions on Pttern Anlysis nd Mchine Intelligence, vol. 17, pp , July [10] H. S. Nguyen, Discretiztion problem for rough sets methods, in RSCTC 98, ser. LNAI 1424, L. Polkowski nd A. Skowron, Eds. Berlin Heidelberg: Springer-Verlg, June 1998, pp [11] J. Komorowski, Z. Pwlk, L. Polkowski, nd A. Skowron, Rough sets: tutoril, in Rough Fuzzy Hybridiztion, S. Pl nd A. Skowron, Eds. Springerverlg, 1999, pp [12] J. G. Bzn, A. Skowron, nd P. Synk, Dynmic reducts s tool for extrcting lws from decision tbles, in Proceeding of the Sympsium on Methodologies for Intelligent Systems, ser. LNAI 869, 1994, pp [13] J. Bzn nd M. Szczuk, The RSES homepge, rses, [14] J. Dougherty, R. Kohvi, nd M. Shmi, Supervised nd unsupervised discretiztions of continuous fetures, in Proceedings of the 12th Interntionl Conference on Mchine Lerning,. Morgn Kufmnn Publishers, 1995, pp [15] C. L. Blke nd C. J. Merz, UCI repository of mchine lerning dtbses, mlern/mlrepository.html,