A new way to build Oblique Decision Trees using Linear Programming

Transcription

1 A new way to build Oblique Decision Trees using Linear Programming Guy Micel, Jean Luc Lambert Bruno Cremilleux & Micel Henry-Amar GREYC, CNRS UPRESA GRECAN, CNRS UPRES 772 Eslanade de la aix - Université de Caen F Caen Cedex, France Summary : Adding linear combination slits to decision trees allows multivariate relations to be exressed more accurately and succinctly tan univariate slits alone. In order to determine an oblique yerlane wic distinguises two sets, linear rogramming is roosed to be used. Tis formulation yields a straigtforward way to treat missing values. Comutational comarison of tat linear rogramming aroac algoritm wit classical univariate slit algoritms roofs te interest of tis metod. Key words : Oblique decision tree, missing values, linear rogramming. Introduction Classification and decision trees already belong to te way of current researc and many metods still ave to be exlored [5]. Unlike classical decision trees wic roduce univariate trees, linear rogramming is roosed to be used to generate oblique decision trees. For many years, many metods ave been sown to create ODT but finding best multivariate yerlane is a NP comlete roblem. Murty, Kasif and Salzberg [6] ave roosed OC, a solution based on imurity measures and erturbation algoritm. Oter metods roosed by Mangasarian, Setiono and Wolberg [4] or Bennet [] induce ODT using linear rogramming but tese metods find te otimal linear discriminates by otimising secific goodness measures given by te autors. Instead of aving an analytic aroac, geometrical results of linear rogramming is used and no secial measures are needed. Only data set containing two classes can be treated for te moment but an original solution to solve te roblem of missing values is roosed. Tat gives te ossibility to work wit a medical data set and to comare te erformances of tis algoritm to C4.5 [7], a standard univariate decision trees builder. {gmicel,lambert,cremilleux}@info.unicaen.fr amar@baclesse.fr Trees using oblique yerlanes to artition data are called oblique decision trees and noted ODT.

2 Linear Programming : geometrical aroac Two rincial results of linear rogramming are essentially used in tis work : te Simlex Algoritm and te Duality Teorem. Linear rogramming is an algebraic tool and so te data set as to be translated into an algebraic form. Assume, for te moment, tat data set contains two classes, tat all data are numeric and tat tere is no missing value.. Global aroac Te data set can be roected in an Figure : Duality teorem Euclidean sace, noted E, of dimension n ( n is te number of caracteristics describing eac data and one data becomes one oint in E ) and eac class is reresented by a cloud of oints. Te aim of learning, in tis case, is to searate two classes ( or two clouds of oints ) : in te case of te Figure, te yerlane D divides x-class and y-class and allows to decide in wic classes belong new oints. As linear rogramming roducts linear borderline, dividing two clouds of oints or teir convex covers is equivalent. Let be C = { E,...E } and C 2 = { F,...F q } wit E i in R n for all i in [ ] Convex cover of C and C 2 is obtained by introducing convex notions ( Tey are noted Con(C ) and q Con(C 2 ) ). Te intersection of Con(C λiei F = 0 ) and i= = Con(C 2 ) is null ( and a i [.. ] λ i 0 linear borderline can be λi = wit found ) if and only if i= [.. q] 0 q System as no solution. i = In te oter case, an oter = result of te linear System : Intersection of convex covers. rogramming gives a metod for aving good results..2 Tools and metods Te Simlex Algoritm and te Duality Teorem [3] are mainly used in tese twice situations..2. A borderline exists [ ] [ ] n a R, b R : i..,.. q System 2 : System dual t a Ei t a F < b > b Firstly, studying te case in wic a borderline dividing convex covers exists ( e.g. te situation look like at te Figure ).

3 A linear borderline exists ( e.g. a vector a ) if and only if te system 2 is verified. By noticing tat te System 2 is te dual of te System and using te Duality Teorem, following result can be roved : if te System as no solution, te base of te dual is solution of te System 2. Tis base, easily extractable from te System, gives, for eac dimension, te coefficient of te vector a ( and, of course, te equation of te yerlane D )..2.2 No borderline exists But, most of te time, it not exists yerlanes dividing classes and te dual as no solution. See te Figure 2 for aving an examle of tis kind of situation. Figure 2 : Simlex Algoritm Te Simlex Algoritm return a roof of te non existence of yerlanes by giving a small set of oints belonging to te bot classes. Te Figure 2 sows easily tat : Con(C ) Con(C 2 ) and one roof can be given by te following oints : {x 2,x 3,y,y 2 } ( because [x 2,x 3 ] [y,y 2 ] ). In tis case, te idea is to take out one oint from tis set and to consider new convex covers again. Logically, after m iterations, two distinct convex covers ave to be obtained and results of.2. may be alied. Tose twice remarks allow to roose te following algoritm :.3 A trivial algoritm Let C and C 2, two sets of oint in R n. Te rocedure used to induce a slit tat divides two sets of oints can be defined : Algoritm. Te function tat cooses one oint in te intersection s set is very imortant and not easy to define. For examle, in Figure 2, it is more interesting to extract x 2 tan x 3,y or y 2. For te moment, te coice function is trivial and cooses a oint randomly. It is certainly easy to find a better solution and tis oint sould be studied in te future. 2 Missing values treatment Algoritm : Alication Build system Wile Indivisible do Find intersection Coose oint Extract oint from system end wile Extract yerlane from system In tis reresentation of data set, tere is no lace for missing value wereas it is very imortant to consider tis kind of trouble igly resent in real word data extracted.

4 2. Linear Programming interretation of missing values Algoritms usually roose [2][8] to relace missing values by values extracted from te data set ( e.g. median value of te argument ) or to ave a robabilistic aroac [7]. Te linear rogramming aroac allows us to aly an oter treatment. Instead of fixing missing values, tey are relaced by variables. Limits are given to tose variables to be sure tat tey ave logical values. Tat means an exert as to define a maximum value and a minimum value for all dimensions in wic tere exists missing values. Sometimes, tose values could be extracted from te data base by finding real maximum and minimum if tis data base is reresentative enoug. 2.2 Geometric interretation of missing values For understanding te aroac, it is interesting to ave a geometrical Figure 3 : Missing values treatment interretation of tis oeration. In fact, A data aving missing values is relaced by an yercube of dimension ( e.g. an yercube wit 2 vertices ). Studying, in R 2, te following case : A=(?,α) and B=(β,?) and ave a look to te Figure 3. Constraints given by te linear rogramming aroac are stronger tan tose given by te classical aroaces and te coice for finding yerlanes is more limited. It is even ossible to ave situations in wic tere exists yerlanes in te first case wereas te intersection between te two yercubes is not emty ( it can not exist yerlane in tis condition ). Tat roves tose aroaces are not equivalent. Te algoritm given in.3 finds an elegant solution for tis trouble because bad oints are eliminated. 2.3 Generalisation Te algebraic form is generalised for C and C 2, two sets of R n. Let be Ω=(Ω,,Ω n ) and Θ=(Θ,,Θ n ), two sets of n sets. For all i in [..n], let Ω i ( res. Θ i ) te set of indices of variables of C ( res. C 2 ) for wic te i t comonent is unknown. For examle, r is in Ω i if and only if x r is in C and x i is a missing value. Let, for all i in [..n], m i and M i as te limits for te i t comonent of te data base ( read in te data base or given by an exert of te domain ). Te system obtained ( System 3 ) is linear and linear rogramming algoritms may be alied to resolve it.

5 [.. n] i=, i Ω i i, System 3 : Generalised system λ E i k + [.. n] [.. ] λ 0 ; [.. q] [.. n] m λr X r Ω X M λr i 0 m s Ys, k s Θ k Ys, k M s q λ = ; = i= rω i X = k=, k Θ k k, F rθ K Y k = 0 Notice tat if all comonents are not numeric, it is always ossible to transform tem. For examle, [Small,Medium,Big] becomes [,2,3] and [Yellow,Blue,Red] is relaced wit tree binary comonents : [0,] for Yellow, [0,] for Blue and [0,] for Red. Tis treatment is imortant and, at a roug estimate, can reresent a negative oint of tis metod because it could not be done automatically by algoritms and knowledge of exerts is needed. But, wit te view to ave a semantic interretation of attributes, only uman users can be efficient. 3 Comutational results Before giving results of exeriments tat comare te erformances of tose algoritm to C4.5, te data set used as to be described. 3. Te Hodgkin s disease In tis ae reorted results are issued from a data set collected by te Lymoma Cooerative Grou of te Euroean Organisation for Researc and Treatment of Cancer ( EORTC ) and rovided by Dr. M. Henry-Amar. Te data set describes more tan 3000 atients treated wit various rotocols. After treatment 2, te data set currently as 824 entries for te learning data and 70 entries for te test data 3. Te atients, groued as Favourable ( 369 cases ) nor Unfavourable ( 455 cases ), are described troug 6 continuous attributes and tree binaries attributes. Te learning data set contains 330 missing values concentrated on five attributes. 3.2 Comments about results form In tis exerience, extrema were extracted from te data base as described in 2. and were suervised by Dr. M. Henry-Amar. Hyerlanes are given 2 Different rotocols roducts different descritors for atients and coices ave to be done. 3 Tose data sets are fixed like tat for temoral reasons.

6 under te form exlain in Table : for eac attribute of te data base, te value obtained in te dual is te coefficient of te vector a described in.2.. Table : Results form Coef. for age : Coef. for sexe : Coef. for cbdfus : Coef. for cbgfus : Coef. for axdfus : Coef. for axgfus : Coef. for medfus : Coef. for ext : Coef. for sg : Coef. for vs : Coef. for b : Coef. for gb : e-05 Coef. for olfus : 0 Coef. for lymfus : 0 Coef. for monfus : 0 Coef. for laq : 0 Coef. for a : Coef. for ld : e-06 Coef. for istforus : Coef. for lambda_dua : Coef. for mu_dua : In te case of te Table, te vector a is : ( , ,, e - 06, ) a = and let be b = 2 ( ) A oint x belongs to Favourable if ax > b; oterwise, it belongs to Unfavourable. Results are given under a form tat allowed a semantic interretation. For examle, te attribute olfus is unneeded to classify because its coefficient is null and te imortant value of te age s coefficient means tat te youngest are more concerned by te disease tan te oldest. Tose information are well known from medical exerts but oters yoteses are confirmed by tis way. 3.3 Results of te linear rogramming classifier For aving results, ust tree oblique yerlanes are extracted ( Tat means tat te work sace is Learning Class Class 2 Total devised in four regions wit Linear Prog. 88.% 89.2% 88.7% tree linear borderline ) C4.5 ( Rules ) 90.8% 99.3% 94.3% wereas C4.5 use more tan 20 rules. Te results obtained are under tose obtained by C4.5 but tey are very interesting. Wit few yerlanes ( Tree ), results obtained on te test data base rove tat te metod described bellow is valid even on a real world Test Class Class 2 Total Linear Prog. 85.3% 8% 82.8% C4.5 ( Rules ) 87.4% 9.% 89.6% extracted data base ( C4.5 results sow te nontrivial artition of te Hodgkin s disease data base ). Notice tat seed of linear rogramming algoritm, articularly adated for tis kind of situation, allows to obtain results very easily on common comuter.

7 4 Conclusions Tis aer as described a linear rogramming metod to construct linear borderline seedily and easily. A new way to exloit linear rogramming in classification as been resented and an original treatment of missing values as been introduced. Te imlementation roosed was tested on a consistent data set extracted from te medical domain and interesting results were obtained : good yerlanes tat generalise very well were got. But better results may be obtained by trying tose following ideas : build a comlete decision trees wit linear borderline, define a clever coice function (.3 ), introduce te extracted elements in an oter orde generate comlete decision trees, use some runing metods, test te algoritm wit cross validation and oter data sets, generalise to multi classes decision trees. Tese extensions are currently exlored. References []. Bennet ( 992 ). Decision tree construction via linear rogramming, Comuter Sciences Tecnical reort 067. [2]. Celeux ( 988 ). Le traitement des valeurs manquantes dans le logiciel SICLA. [3]. Cvátal ( 993 ). Linear Programming, W.H. Freman and Comagny. [4]. Mangasarian, Setiono & Wolberg ( 990 ). Pattern recognition via linear rogramming : Teory and alication to medical diagnosis, in : S.I.A.M. Workso on otimisation. [5]. Micie, Siegelalter & Taylor ( 994 ). Macine learning, neural and statistical classification. Ellis Herwood Series in Artificial Intelligence. [6]. Murty, Kasif & Salzberg ( 994 ). A system for induction of oblique decision trees in : Journal of Artificial Intelligence Researc 2, -32. [7]. Quinlan ( 993 ). C4.5 : Programs for macine learning. Morgan Kaufmann. [8]. Quinlan ( 989 ). Unknown attribute values in induction in Segre. Proceedings of te sixt International Workso on Macine Learning,