An Efficient and Effective Case Classification Method Based On Slicing

Transcription

1 An Efficient nd Effective Cse Clssifiction Method Bsed On Slicing An Efficient nd Effective Cse Clssifiction Method Bsed On Slicing Omr A. A. Shib, Md. Nsir Sulimn, Ali Mmt nd Ftimh Ahmd Fculty of Computer Science nd Informtion Technology University Putr Mlysi, UPM Serdng, Selngor E-mil: Abstrct One of the most importnt tsks tht we hve to fce in rel world pplictions is the tsk of clssifying prticulr situtions nd /or events s belonging to certin clss. In order to solve the clssifiction problem, ccurte clssifier systems or models must be built. Severl computtionl intelligence methodologies hve been pplied to construct such clssifier from prticulr cses or dt. This pper introduces new clssifiction method bsed on slicing techniques tht ws proposed for procedurl progrmming lnguges. The pper lso discusses two of common clssifiction lgorithms tht re used either in dt mining or in generl AI. The lgorithms re: Induction of Decision Tree Algorithm (ID3) nd Bse Lerning Algorithm (C4.5). The pper lso studies the comprison between the proposed method nd the two selected clssifiction lgorithms using severl domins. Keywords: - dt mining, clssifiction problem, clssifiction lgorithms, slicing techniques, cse slicing, ID3, C Introduction Clssifiction is the most importnt tsk in mchine lerning. In clssifiction, clssifier is built from set of trining exmples with clss lbels. A key performnce mesure of the clssifier is its predictive ccurcy on the trining nd testing exmples [1]. The clssifiction problem hs been studied extensively by the dtbse nd Artificil Intelligence communities. The problem of clssifiction is defined s follows: The input dt re referred to s the trining set, which contins plurlity of records, ech of which contins multiple ttributes or fetures. Ech exmple in the trining set is tgged with clss lbel. The trining set is used in order to build model of the clssifiction ttribute bsed upon the other ttributes. This model is used in order to predict the vlue of the clss lbel for the test set. Some well-known techniques for clssifiction include the following: ID3 nd C4.5 Algorithm [2, 3]. This pper introduces new clssifiction method bsed on slicing technique. Slicing technique ws proposed for procedurl progrmming lnguges [4]. Slicing is method used by experienced computer progrmmers for restricting, the behviour of progrm to some specified subset of interest. The reminder of the pper is orgnized s follows: Section 2 presents brief description of some relted work; cse clssifiction bsed on slicing technique is described in section 3; the experimentl results nd the conclusion re presented in sections 4 nd 5 respectively. Interntionl Journl of The Computer, the Internet nd Mngement Vol. 14.No.2 (My - August, 2006) pp

2 Omr A. A. Shib, Md. Nsir Sulimn, Ali Mmt nd Ftimh Ahmd 2. Relted Work In this section brief description of two of the common clssifiction lgorithms tht re relted to our work will be presented. 2.1 C4.5 Algorithm C4.5 is n extension to the decision-tree lerning lgorithm ID3 [2, 3]. The lgorithm consists of the following steps: Build the decision tree form the trining set (conventionl ID3). Convert the resulting tree into n equivlent set of rules. The number of rules is equivlent to the number of possible pths from the root to lef node. Prune ech rule by removing ny preconditions tht result in improving its ccurcy, ccording to vlidtion set. Sort the pruned rules in descending order ccording to their ccurcy, nd consider them in this sequence when clssifying subsequent instnces. 2.2 Induction of Decision Tree Algorithm (ID3) ID3 is n lgorithm introduced by Quinln for inducing Clssifiction Model, lso clled Induction of Decision Tree, from dt. We re given set of records. Ech record hs the sme structure, consisting of number of ttribute / vlue pirs. One of these ttributes represents the ctegory of the record. The problem is to determine decision tree tht on the bsis of nswers to questions bout the non-ctegory ttributes predicts correctly the vlue of the ctegory ttribute. Usully the ctegory ttribute tkes only the vlues {true, flse}, or {success, filure}, or something equivlent. In ny cse, one of its vlues will men filure [2, 3]. The bsic ides behind ID3 re tht: In the decision tree ech node corresponds to non-ctegoricl ttribute nd ech rc to possible vlue of tht ttribute. A lef of the tree specifies the expected vlue of the ctegoricl ttribute for the records described by the pth from the root to tht lef. (This defines wht is Decision Tree.) In the decision tree t ech node should be ssocited the non-ctegoricl ttribute which is most informtive mong the ttributes not yet considered in the pth from the root. (This estblishes wht is Good decision tree.) Entropy is used to mesure how informtive is node. (This defines wht we men by Good. By the wy, this notion ws introduced by Clude Shnnon in Informtion Theory) [5]. 3. Cse Clssifiction Bsed On Slicing In this section the proposed clssifiction pproch nd some relted terms re discussed. 3.1 Progrm Slicing Technique Slicing is method used by experienced computer progrmmers for restricting, the behviour of progrm to some specified subset of interest [4, 6]. A slice is constructed by deleting those prts of the progrm tht re irrelevnt to the vlues stored in the chosen set of vribles t the chosen point. The point of interest is usully identified by nnotting the progrm with line numbers, which identify ech primitive sttement nd ech brnch node. Progrm slicing is useful for progrm understnding, mintennce, debugging, testing, differencing, speciliztion, reuse, optimiztion, prlleliztion, nd nomly detection [7]. Progrm slicing hs been widely studied in the context of impertive progrms. Severl slicing lgorithms for impertive lnguges hve been developed 16

3 An Efficient nd Effective Cse Clssifiction Method Bsed On Slicing [8]. Slicing of progrms is performed with respect to some criterion. Weiser proposes s criterion the number i of commnd line nd subset V of progrm vribles [4]. According to this criterion, progrm is nlyzed nd its commnds re checked for their relevnce to commnd line i nd those vribles in V. However, other uthors hve defined different criterion [9]. Progrm slicing cn be summrized s following: Progrm Slice the sttements (nd predictes) tht might ffect the vlue of set of vribles t prticulr sttement. A slice is tken with respect to set of vribles t prticulr sttement, the slicing criterion. A slice nd the ctul progrm behvior re identicl. Not just the finl vlue, but ll intermedite steps Executble (Slice) slice tht cn be compiled nd executed. Closure (Slice) n informtionl presenttion of slice tht might lck semntics. Bsic Types: Sttic vs. Dynmic Type of feedbck: executble, closure Approch: grph rechbility, dtflow equtions using control flow. 3.2 Extending Progrm Slicing To Cse Slicing The cse slicing clssifier described in this pper ddress the problem of clssifiction. The cse slicing clssifier is extended of progrm slicing technique. When we slice cse we re interested in utomticlly obtining tht portion fetures of the cse responsible for specific prts of the solution of the cse t hnd. A cse slicing: is process for utomticlly obtining subprts (fetures) of cse with collective mening. A slicing criterion: denotes the conditions of the slice computtion, with respect to which nd for which cse slice is required. Sliced cse: contins ll fetures tht could hve direct reltions with the fetures of interest t new cse. 3.3 The Cse Slicing Technique Conceptully, the proposed method is vrition of the Nerest Neighbor Algorithms [10] nd is clled Cse Slicing Technique (CST). It compres new cses to the trining cses in the dt file. It computes the similrity between the new cses nd trining cses to clssify the new cses. The proposed method is clssifiction technique bsed on slicing. Slice cse mens we re interested in utomticlly obtining tht portion fetures of the cse responsible for specific prts of the solution of the cse t hnd. By slicing the cse with respect to importnt fetures we cn obtin new cse with smll number of fetures or with only importnt fetures. The proposed pproch consists of dtbse with three clcultion modules s follows: Fetures Weighting Module This module is used to mesure the importnce of ech ttribute in clssifiction. The weight of ech ttribute hs been clculted to clssify the new cse by using simple conditionl probbilities. High weight vlues were ssigning to fetures tht re highly correlted with the given clss using eqution (1). ( ) w(i ) = P C i = instnces instnces contining i contining clss = C (1) Where the weight for feture for clss c is the conditionl probbility tht cse is member of c given the vlue to. i Interntionl Journl of The Computer, the Internet nd Mngement Vol. 14.No.2 (My - August, 2006) pp

4 Omr A. A. Shib, Md. Nsir Sulimn, Ali Mmt nd Ftimh Ahmd Discretiztion Computing Module Discretiztion s used in this pper, nd in the mchine lerning literture in generl, is process of trnsforming continuous ttribute vlues into finite number of intervls nd ssociting with ech intervl discrete, numericl vlue. The usul pproch for lerning tsks tht use mixedmode (continuous nd discrete) dt is to perform discretiztion prior to the lerning process [11, 12, 13, 14]. The discretiztion process first finds the number of discrete intervls, nd then the width, or the boundries for the intervls, given the rnge of vlues of continuous ttribute. Most often the user must specify the number of intervls, or provide some heuristic rule to be used [15]. A vriety of discretiztion methods hve been developed in recent yers. Some models tht hve used the Vlue Difference Metrics (VDM) or vrints of it [16, 17, 18] hve discretized continuous ttributes into somewht rbitrry number of discrete rnges, nd then treted these vlues s nominl (discrete unordered) vlues. When using slicing pproch, continuous vlues re discretized into s equl-width intervls (though the continuous vlues re lso retined for lter use), where s is n integer supplied by the user. Unfortuntely, there is currently little guidnce on wht vlue of s to use. Current reserch is exmining more sophisticted techniques for determining good vlues of s, such s cross-vlidtion, or other sttisticl methods [19]. The width w of discretized intervl for ttribute is given by eqution (2). w mx min = s (2) where mx nd min re the mximum nd minimum vlue, respectively, occurring in the trining set for ttribute. The discretized vlue v of continuous vlue x for ttribute is n integer from 1 to s, nd is given by eqution (3). ( x min ) if ttribute v = disc( x) = w x if ttribute Distnce Computtion Module is is continuou discrete (3) There re mny lerning systems tht store some or ll vilble trining exmples during lerning. During generliztion, new input vector is presented to the system for clssifiction nd distnce function is used to determine how fr ech stored instnce is from the new input vector. The stored instnce or instnces which re closest to the new vector re used to clssify it. A vriety of distnce functions re vilble for such uses, including the Minkowsky [20], Mhlnobis [21], Cmberr, Chebychev, Qudrtic, Correltion, nd Chi-squre distnce metrics [22, 23], the Context- Similrity mesure [24], the Contrst Model [25], hyperrectngle distnce functions [26, 27] nd others. Although there hve been mny distnce functions proposed, by fr the most commonly used is the Eucliden distnce function, which is defined in eqution (4). E m ( x, y ) = ( x ) r, y = 1 (4) Where x nd y re two input vectors (one typiclly being from stored instnce, nd the other n input vector to be clssified) nd m is the number of input vribles 18

5 An Efficient nd Effective Cse Clssifiction Method Bsed On Slicing (ttributes) in the ppliction. The squre root is often not computed in prctice, becuse the closest instnce(s) will still be the closest, regrdless of whether the squre root is tken. An lterntive function, the City-block or Mnhttn distnce function, requires less computtion nd is defined in eqution (5). M m ( x, y) = = 1 x y r (5) The Eucliden nd Mnhttn distnce functions re equivlent to the Minkowskin r-distnce function [20] with r = 2 nd 1, respectively Slicing Technique The objective of slicing technique is to optimize the similrity mtching to chieve best clssifiction results. The proposed pproch is dpting slicing techniques tht hve been used in progrmming lnguges, to slice the cses by removing subset of fetures which re irrelevnt to cse lbel with respect to the selected slicing criterion. Cse clssifiction lgorithm bsed slicing is shown in Figure A Forml Description Of Cse Slicing Technique In this section we will give forml description of Cse Slicing Techniques in bsic version llowing for detiled investigtion of the pproch. Let Ci = {f1, f2, f3 fn} where n is the number of fetures in Ci λ = [{Cs Cs is set of sliced cses}] OR λ = {ll cses tht contins one or more importnt feture(s)} I = {if1, if2,., ifn} where n is the number of importnt fetures in I I Ci S I Cs λ 4. Experimentl Results In this section the results of severl prcticl experiments re presented to exmine the performnce of the proposed pproch nd the performnce of the selected clssifiction lgorithms on rel world problems. 4.1 Selected Dtsets In this pper eight rel-world dtsets hve been used, which re widely used in the mchine-lerning field for evlution of cse slicing technique. The eight dtsets: Clevelnd Hert Disese (CLEV), Brest Cncer (BCO), Germn Credit Crd (GERM), Heptitis Domin (HEPA), Austrlin Credit Crd Approvl (AUS), Iris Plnts Dtbse (IRIS), United Sttes Congressionl Voting Records Dtbse (VOTING) nd Credit Crd Appliction (CRX) were chosen from the UCI: Mchine Lerning Repositories nd Domin Theories [28]. Tble 1 presents the min chrcteristics of these dtsets, where B, C nd D in the tble mens Boolen, continuous nd discrete ttributes respectively. S = {C1, C2, C3 Cn} set of cses in Cse Bse S Ci S φ Interntionl Journl of The Computer, the Internet nd Mngement Vol. 14.No.2 (My - August, 2006) pp

6 Omr A. A. Shib, Md. Nsir Sulimn, Ali Mmt nd Ftimh Ahmd Algorithm: Algorithm Figure for Cse Clssifiction 1. Cse Input: User s Input Problem Specifiction Output: Clssified Cse BEGIN clssifiction lgorithm bsed While True do Step 1 Discretize Continuous Vlues slicing Let x be the input vlue for ttribute of cse i v = disc ( x ) [Which is just x if is discrete] w = bs ( mx min )/ s {The width of discretized intervl for ttribute } {Where mx nd min re the mximum nd minimum vlue, respectively, which re occurring in the trining set for ttribute.}, {the discretized vlue v of continuous vlue x for ttribute is n integer from 1 to s nd s determine by the user.} If is continuous then v = disc ( x ) = Else v = disc ( x ) = x x min / w Endif Step 2 Assign Weights to fetures {using conditionl probbility} Let c be the output clss of cse i. N,v,c = N,v,c + 1 {# of vlue v of ttribute with output clss c} N,v = N,v + 1 {# of vlue v of ttribute } For ech vlue v (of ttribute ) For ech clss c If N,v = 0 P,v,c =0 Else p,v,c = N,v,c/ N,v ENDFOR ENDFOR Step 3 Cse Slicing For ech cse i in T {T- Trining set} Slice cse i w.r.t. selected criterion {removing irrelevnt ttributes depending on ttribute weights} Endfor Step 4 Distnce Clcultions For ech two cses x nd y {One typiclly being from stored cses, nd other the input cse to be clssified} Let m the number of ttributes in the cse For =1 to m do dis tn ce ( x, y) = bs( ) x y Distnce_ Summtion {the cumultive distnce of ll the m ttributes in the cse i} 0Endfor Step 5 Closer Cse Serching While.not. done do Find mtching between cses to get closer cse with less distnce Enddo Figure 1. Cse clssifiction lgorithm bsed slicing 20

7 Tble 1. Chrcteristics of the selected dtsets An Efficient nd Effective Cse Clssifiction Method Bsed On Slicing Dtsets No. Of Type & No. Of No. Dt Attributes Clsses CLEV 303 6C, 9D (15) 2 BECO B, 6C (19) 2 GERM B (16) 2 HEPA B, 6C (19) 2 AUS 690 6C, 9D (15) 2 IRIS 150 4C (4) 3 VOTING B (16) 2 CRX 690 6C, 9D (15) 2 Of 4.2 Empiricl Results We evluted the performnce of the cse slicing technique by compring it ginst the ID3 nd C4.5 clssifiers on vriety of dtsets. The dtsets we hve selected re very good choice to test nd evlute the slicing technique becuse the dtsets re from different domins nd there is good mixture of continues, discrete nd Boolen fetures. In ll the experiments reported here we used the evlution technique 10-fold cross-vlidtion, which consists of rndomly dividing the dt into 10 eqully, sized subgroups nd performing ten different experiments. We seprted one group long with their originl lbels s the vlidtion set; nother group ws considered s the strting trining set; the reminder of the dt were considered the test set. Ech experiment consists of ten runs of the procedure described bove, nd the overll verge re the results reported here. The criterion of choosing the best clssifiction pproch is bsed on the highest percentge of clssifiction. The results, given in Tble 2, list the clssifiction ccurcies chieved by ech pproch for ech of the dtsets, nd Figure 2 shows the difference in clssifiction ccurcy. Tble 2. The clssifiction ccurcy chieved by the different clssifiction lgorithms. Methods C4.5 ID3 CST Dtsets CLEV BCO GERM HEPA AUS IRIS VOTING CRX Interntionl Journl of The Computer, the Internet nd Mngement Vol. 14.No.2 (My - August, 2006) pp

8 Omr A. A. Shib, Md. Nsir Sulimn, Ali Mmt nd Ftimh Ahmd Clssifiction Acurcy (%) CLEV GERMN AUS VOTING Dtsets C4.5 ID3 CST Figure 2. Difference in clssifiction ccurcy of the selected lgorithms 5. Conclusion This pper hs presented nd discussed the Cse Slicing Technique (CST) s new pproch bsed slicing to improve clssifiction tsk in dt mining. CST ws supported with experiments on eight dtsets. The experiments show tht using the CST indeed improves the ccurcy of clssifiction. The pper lso gve brief description of number of common clssifiction lgorithms tht re used either in dt mining or in generl AI. The pper hs presented comprison between proposed method nd other selected clssifiction lgorithms using severl domins. The proposed technique hs possessed competitive result. It gve very high percentge of clssifiction ccurcy. References Ling, C. X. nd Zhng, H. (2002) Towrd Byesin Clssifiers with Accurte Probbilities, Proceedings of the Sixth Pcific-Asi Conference on KDD, Springer. Quinln, J. R. (1986) Induction of Decision Tree, Mchine Lerning, Vol. 1, No. 1, pp Quinln, J. R. (1993) C4.5: Progrms for Mchine Lerning, CA: Morgn Kufmnn Publishers, Inc. Weiser, M. (1984) Progrm Slicing, IEEE Trnsction Softwre Engineering, Vol. 10, No. 4, pp Cover, T. M. nd Thoms, J. A. (1991) Elements of Informtion Theory, Wiley. Horwitz, S., Reps, T. nd Binkley, D.(1990) Interprocedurl Slicing Using Dependence Grphs, ACM Trnsctions on Progrmming Lnguges nd Systems, Vol.12, No.1, pp Gllgher, K. nd Lyle, J. (1991) Using Progrm Slicing in Softwre Mintennce, IEEE Trnsction on Softwre Engineering. Vol. 17, No.8, pp Tip, F. (1995) A Survey of Progrm Slicing Techniques, Journl of Progrmming Lnguges, 3, pp Vsconcelos, W. W. (2000) Slicing Knowledge- Bsed Systems Techniques nd Applictions, Knowledge bsed Systems Journl, Elsevier, Vol. 13, pp

9 An Efficient nd Effective Cse Clssifiction Method Bsed On Slicing Wettschereck, D. nd Thoms, G. (1995) An Experimentl Comprison of Nerest Neighbor nd Nerest-Hyperrectngle Algorithms, Mchine Lerning, Vol. 19, No.1, pp Ctlett, J. (1991) On Chnging Continuous Attributes into ordered discrete Attributes, Proc. Europen Working Session on Lerning, pp Dougherty, J., Kohvi, R. nd Shmi, M. (1995) Supervised nd Unsupervised Discretiztion of Continuous Fetures, Proc. of the 12th Interntionl Conference on Mchine Lerning, pp Fyyd, U. nd Irni, K. (1992) On the Hndling of Continuous-Vlued Attributes in Decision Tree Genertion, Mchine Lerning, Vol. 8, pp Pfhringer, B. (1995) Compression-Bsed Discretiztion of Continuous Attributes, Proc. of the 12th Interntionl Conference on Mchine Lerning, pp Ching, J., Wong, A. nd Chn, K. (1995) Clss-Dependent Discretiztion for Inductive Lerning from Continuous nd Mixed Mode Dt, IEEE Trnsctions on Pttern Anlysis nd Mchine Intelligence, Vol.17, No.7, pp Cost, S. nd Slzberg, S. (1993) A Weighted Nerest Neighbor Algorithm for Lerning with Symbolic Fetures. Mchine Lerning, Vol. 10, pp Rchlin, J., Simon, K., Slzberg, S. nd Dvid, W. (1994) Towrds Better Understnding of Memory-Bsed nd Byesin Clssifiers, In Proceedings of the Eleventh Interntionl Mchine Lerning Conference. New Brunswick, NJ: Morgn Kufmnn, pp Mohri, T. nd Tnk, H. (1994) An Optiml- Weighting Criterion of Cse Indexing for both Numeric nd Symbolic Attributes, In W. Dvid (Ed.), Cse-Bsed Resoning: Ppers from the 1994 Workshop (Report No. WS-94-01). Menlo Prk, CA: AIII Press. pp Tpi, R. nd Thompson, J. (1978) Nonprmetric Probbility Density Estimtion, Bltimore, MD: The Johns Hopkins University Press. Btchelor, B. (199=78) Pttern Recognition: Ides in Prctice, New York: Plenum Press. pp Ndler, M. nd Eric, P. (1993) Pttern Recognition Engineering, New York: Wiley. pp Michlski, R. Robert, E. nd Edwin, D. (1981) A Recent Advnce in Dt Anlysis: Clustering Objects into Clsses Chrcterized by Conjunctive Concepts, Progress in Pttern Recognition, Vol. 1, Lveen N. Knl nd Azriel Rosenfeld (Eds.). (New York: North-Hollnd). pp Edwin, D. (1994) Recent Progress in Distnce nd Similrity Mesures in Pttern Recognition, Second Interntionl Joint Conference on Pttern Recognition. pp Bibermn, Y. (1994) A Context Similrity Mesure, In Proceedings of the Europen Conference on Mchine Lerning (ECML- 94). (Ctlin, Itly: Springer Verlg). pp Tversky, A. (1977) Fetures of Similrity, Psychologicl Review, Vol. 84, No. 4, pp Slzberg, S. (1991) A Nerest Hyperrectngle Lerning Method, Mchine Lerning, 6: pp Domingos, P. (1995) Rule Induction nd Instnce-Bsed Lerning A Unified Approch, To pper in The Interntionl Joint Conference on Artificil Intelligence (IJCAI-95). Murphy, P. (1996) UCI Repositories of Mchine Lerning nd Domin Theories, [online]. University of Cliforni, Irvine Avilble: tory.html, [2001, Nov Dte of brows]. Interntionl Journl of The Computer, the Internet nd Mngement Vol. 14.No.2 (My - August, 2006) pp