A Non-greedy Decision Tree Algorithm. Kristin P. Bennett. Rensselaer Polytechnic Institute. Troy, NY 12180

Size: px

Start display at page:

Download "A Non-greedy Decision Tree Algorithm. Kristin P. Bennett. Rensselaer Polytechnic Institute. Troy, NY 12180"

Lora Summers
5 years ago
Views:

1 Global Tree Optiization: A Non-greedy Decision Tree Algith Kristin P. Bennett Eail benne@rpi.edu Departent of Matheatical Sciences Rensselaer Polytechnic Institute Troy, NY 28 Abstract A non-greedy approach f constructing globally optial ultivariate decision trees with xed structure is proposed. Previous greedy tree construction algiths are locally optial in that they optiize soe splitting criterion at each decision node, typically one node at a tie. In contrast, global tree optiization explicitly considers all decisions in the tree concurrently. An iterative linear prograing algith is used to iniize the classication err of the entire tree. Global tree optiization can be used both to construct decision trees initially and to update existing decision trees. Encouraging coputational experience is repted. Introduction Global Tree Optiization (GTO) is a new approach f constructing decision trees that classify two e sets of n-diensional points. The essential dierence between this w and pri decision tree algiths (e.g. CART [5] and ID3 []) is that GTO is non-greedy. F greedy algiths, the \best" decision at each node is found by optiizing soe splitting criterion. This process is started at the root and repeated recursively until all alost all of the points are crectly classied. When the sets to be classied are disjoint, alost any greedy decision tree algith can construct a tree consistent with all the points, given a sucient nuber of decision nodes. However, these trees ay not generalize well (i.e., crectly classify future not-previously-seen points) due to over-tting over-paraeterizing the proble. In practice decision nodes are pruned fro the tree. Typically, the pruning process does not allow the reaining decision nodes to be adjusted, thus the tree ay still be over- This aterial is based on research suppted by National Science Foundation Grant This paper appeared in Coputing Science and Statistics, 26, pg. 56-6, 994. paraeterized. The strength of the greedy algith is that by growing the tree and pruning it, the greedy algith deterines the structure of the tree, the class at each of the leaves, and the decision at each non-leaf node. The liitations of greedy approaches are that locally \good" decisions ay result in a bad overall tree and existing trees are dicult to update and odify. GTO overcoes these liitations by treating the decision tree as a function and optiizing the classication err of the entire tree. The function is siilar to the one proposed f MARS [8], however MARS is still a greedy algith. Greedy algiths optiize one node at a tie and then x the resulting decisions. GTO starts fro an existing tree. The structure of the starting tree (i.e. the nuber of decisions, the depth of the tree, and the classication of the leaves) deterines the classication err function. GTO iniizes the classication err by changing all the decisions concurrently while eeping the underlying structure of the tree xed. The advantages of this approach over greedy ethods are that xing the structure helps prevent overtting overparaeterizing the proble, locally bad but globally good decisions can be ade, existing trees can be re-optiized with additional data, and doain nowledge can be e readily applied. Since GTO requires the structure of the tree as input, it copleents (not replaces) existing greedy decision tree ethods. By copleenting greedy algiths, GTO oers the proise of aing decision trees a e powerful, exible, accurate, and widely accepted paradig. Miniizing the global err of a decision tree with xed structure is a non-convex optiization proble. The proble of constructing a decision tree with a xed nuber of decisions to crectly classify two e sets is a special case of the NP-coplete polyhedral separability proble [9]. Consider this seeingly siple but NPcoplete proble [9]: Can a tree with just two decision nodes crectly classify two disjoint point sets? In [4],

2 w x? w x? > 2 3 w 2 x? 2 w 2 x? 2 > 2 w 3 x? 3 w 3 x? 3 > 4 5 w 4 x? 4 3 w 4 x? 4 > 4 Class A Class B (a) Tree found by Greedy LP Algith Figure : A typical two-class decision tree this proble was fulated as a bilinear progra. We now extend this w to general decision trees, resulting in a ultilinear progra that can be solved using the Fran-Wolfe algith proposed f the bilinear case. This paper is ganized as follows. We begin with a brief review of the well-nown case of optiizing a tree consisting of a single decision. The tree is represented as a syste of linear inequalities and the syste is solved using linear prograing. In Section 3 we show how e general decision trees can be expressed as a syste of disjunctive linear inequalities and fulated as a ultilinear prograing proble. Section 4 explains the iterative linear prograing algith f optiizing the resulting proble. Coputational results and conclusions are given in Section 5. GTO applies to binary trees with a ultivariate decision at each node of the following f: If x is a point being classied, then at decision node d, if xw d > d the point follows the right branch, if xw d d then the point follows the left branch. The choice of which branch the point follows at equality is arbitrary. This type of decision has been used in greedy algiths [6, ]. The univariate decisions found by CART [5] f continuous variables can be considered special cases of this type of decision with only one nonzero coponent of w. A point is classied by following the path of the point through the tree until it reaches a leaf node. A point is strictly classied by the tree if it reaches a leaf of the crect class and equality does not hold at any decision along the path to the leaf (i.e. xw d 6= d f any decision d in the path). Although GTO is applicable to probles with any classes, f siplicity we liit discussion to the proble of classifying the two sets A and B. A saple of such a tree is given in Figure. Let A consist of points contained in R n and B consist of points contained in R n. Let A j (b) Tree found by GTO Figure 2: Geoetric depiction of decision trees denote the jth point in A. 2 Optiizing a Single Decision Many ethods exist f iniizing the err of a tree consisting of a single decision node. We briey review one approach which fulates the proble as a set of linear inequalities and then uses linear prograing to iniize the errs in the inequalities [3]. The reader is referred to [3] f full details of the practical and theetical benets of this approach. Let xw = be the plane fed by the decision. F any point x, if xw < then the point is classied in class A, and if xw > then the point is classied in class B. If xw = the class can be chosen arbitrarily. All the points in A and B are strictly classied if there exist w and

3 such that equivalently A j w? < B i w? > j = : : : i = : : :?A j w + j = : : : B i w? i = : : : Note that Equations () and (2) are alternative denitions of linear separability. The choice of the constant is arbitrary. Any positive constant ay be used. If A and B are linearly separable then Equation (2) is feasible, and the linear progra (LP) (3) will have a zero iniu. The resulting (w; ) fs a decision that strictly separates A and B. If Equation (2) is not feasible, then LP (3) iniizes the average isclassication err within each class. in w; X j= y j + i= s:t: y j A j w? + y j j = : : : z i?b i w + + z i i = : : : z i () (2) (3) LP (3) has been used recursively in a greedy decision tree algith called Multisurface Method-Tree (MSMT) []. While it copares favably with other greedy decision tree algiths, it also suers the proble of all greedy approaches. Locally good but globally po decisions near the root of the tree can result in overly large trees with po generalization. Figure 2 shows an exaple of a case where this phenoenon occurs. Figure 2a depicts the planes used by MSMT to copletely classify all the points. The decisions chosen near the root of the tree are largely redundant. As a result the decisions near the leaves of the tree are based on an unnecessarily sall nuber of points. MSMT constructed an excessively large tree that does not reect the underlying structure of the proble. In contrast, GTO was able to copletely classify all the points using only three decisions (Figure 2b). 3 Proble Fulation F general decision trees, the tree can be represented as a set of disjunctive inequalities. A ultilinear progra is used to iniize the err of the disjunctive linear inequalities. We now consider the proble of optiizing a tree with the structure given in Figure, and then briey consider the proble f e general trees. Recall that a point is strictly classied by the tree in Figure if the point reaches a leaf of the crect classication and equality does not hold f any of the decisions along the path to the leaf. A point A j 2 A is strictly classied if it follows the path through the tree to the rst fourth leaf node, i.e. if Aj w? + A j w 2? 2 + *?Aj w (4) A j w 3? 3 +?A j w equivalently (A j w? + ) + (A j w 2? 2 + ) + = (?A j w + + ) + (A j w 3? 3 + ) + (?A j w ) + = where () + := axf; g. Siilarly a point B i 2 B is strictly classied if it follows the path through the tree to the second, third, fth leaf node, i.e. if Bi w? +?B i w equivalently *?Bi w + + B i w 3? 3 + B i w 4? 4 +?Bi w + +?B i w (B i w? + ) + (?B i w ) + = (?B i w + + ) + (B i w 3? 3 + ) + (B i w 4? 4 + ) + = (?B i w + + ) + (?B i w ) + = + (5) (6) (7) A decision tree exists that strictly classies all the points in sets A and B if and only if the following equation has a feasible solution: X j= i= (y j + y 2j ) (z j + y 3j + z 4j ) + (u i + v 2i ) (v i + u 3i + u 4i ) (v i + v 3i ) = where y dj = (A j w d? d + ) + j = : : : z dj = (?A j w d + d + ) + u di = (B i w d? d + ) + i = : : : v di = (?B i w d + d + ) + f d = : : : D and D = nuber of decisions in tree: (8)

4 Furthere, (w d ; d ); d = : : : D; satisfying (8) f the decisions of a tree that strictly classies all the points in the sets A and B. Equivalently, there exists a decision tree with the given structure that crectly classies the points in sets A and B if and only if the following ultilinear progra has a zero iniu: in w;;y;z;u;v X j= i= (y j + y 2j ) (z j + y 3j + z 4j ) + s:t: y dj A j w d? d + j = : : : z dj?a j w d + d + u di B i w d? d + i = : : : v di?b i w d + d + f d = : : : j y; z; u; v (9) The coecients and were chosen so that (9) is identical to the LP (3) f the single decision case, thus guaranteeing that w = is never the unique solution f that case [3]. These coecients also help to ae the ethod e nuerically stable f large training set sizes. This general approach is applicable to any ultivariate binary decision tree used to classify two e sets. There is an err ter f each point in the training set. The err f that point is the product of the errs at each of the leaves. The err at each leaf is the su of the errs in the decisions along the path to that leaf. If a point is crectly classied at one leaf, the err along the path will be zero, and the product of the leaf errs will be zero. Space does not perit discussion of the general fulation in this paper, thus we refer the reader to [2] f e details. 4 Multilinear Prograing The ultilinear progra (3) and its e general fulation can be optiized using the iterative linear prograing Fran-Wolfe type ethod proposed in [4]. We outline the ethod here, and refer the reader to [2] f the atheatical properties of the algith. Consider the proble in f(x) subject to x 2 X where x f : R n! R; X is a polyhedral set in R n containing the constraint x, f has continuous rst partial derivatives, and f is bounded below. The Fran-Wolfe algith f proble is the following: Algith 4. (Fran-Wolfe algith [7, 4]) Start with any x 2 X. Copute x i+ fro x i as follows. (i) v i 2 arg vertex in x2x 5 f(x i )x (ii) Stop if 5 f(x i )v i = 5f(x i )x i (iii) x i+ = (? i )x i + i v i where i 2 arg in f((? )x i + v i ) (u i + v 2i ) (v i + u 3i + u 4i ) (v i + v 3i ) In the above algith \arg vertex in" denotes a vertex solution set of the indicated linear progra. The algith terinates at soe x j that satises the iniu principle necessary optiality condition: 5f(x j )(x? x j ), f all x 2 X, each accuulation point x of the sequence fx i g satises the iniu principle [4]. The gradient calculation f the GTO function is straightfward. F exaple, when Algith 4. is applied to Proble (9), the following linear subproble is solved in step (i) with ( ^w; ^; ^y; ^z; ^u; ^v) = x i : in w;;y;z;u;v X j= X j= i= i= i= (^y j + ^y 2j )(z j + y 3j + z 4j ) + (y j + y 2j ) (^z j + ^y 3j + ^z 4j )+ (^u i + ^v 2i ) (^v i + ^u 3i + ^u 4i ) (v i + v 3i ) + (^u i + ^v 2i ) (v i + u 3i + u 4i ) (^v i + ^v 3i ) + (u i + v 2i ) (^v i + ^u 3i + ^u 4i ) (^v i + ^v 3i ) s:t: y dj A j w j? d + F d = ; : : : ; D z dj?a j w d + d + j = : : : u di B i w d? d + i = : : : v di?b i w d + d + y; z; u; v fixed ^y; ^z; ^u; ^v; 5 Results and Conclusions GTO was ipleented f general decision trees with xed structure. In der to test the eectiveness of the optiization algith, rando probles with nown solutions were generated. F a given diension, a tree with 3 to 7 decision nodes was randoly generated to classify points in the unit cube. Points in the unit cube

5 were randoly generated and classied and grouped into a training set (5 to points) and a testing set (5 points). MSMT, the greedy algith discussed in Section 2, was used to generate a greedy tree that crectly classied the training set. The MSMT tree was then pruned to the nown structure (i.e. the nuber of decision nodes) of the tree. The pruned tree was used as a starting point f GTO. The training and testing set err of the MSMT tree, the pruned tree (denoted MSMT-P), and the GTO tree were easured, as was the training tie. This experient was repeated f trees ranging fro 3 to 7 nodes in 2 to 25 diensions. The results were averaged over trials. We suarize the test results and refer the reader to [2] f e details. Figure 3 presents the average results f randoly generated trees with three decision nodes. These results are typical of those observed in the other experients. MSMT achieved % crectness on the training set but used an excessive nuber of decisions. The training and testing set accuracy of the pruned trees dropped considerably. The trees once optiized by GTO were signicantly better in ters of testing set accuracy than both unpruned and pruned MSMT trees. The coputational results are proising. The Fran- Wolfe algith converges in relatively few iterations to an iproved solution. However GTO did not always nd the global iniu. We expect the proble to have any local inia since it is NP-coplete. We plan to investigate using global optiization techniques to avoid local inia. The overall execution tie of GTO tends to grow as the proble size increases. Parallel coputation can be used to iprove the execution tie of the expensive LP subprobles. The LP subprobles (e.g. Proble (9)) have a bloc-separable structure and can be divided into independent LPs solvable in parallel. We have introduced a non-greedy approach f optiizing decision trees. The GTO algith starts with an existing decision tree, xes the structure of the tree, fulates the err of the tree, and then optiizes that err. An iterative linear prograing algith perfs well on this NP-coplete proble. GTO optiizes all the decisions in the tree, and thus has any potential applications such as: decreasing greediness of constructive algiths, reoptiizing existing trees when additional data is available, pruning greedy decision trees, and incpating doain nowledge into the decision tree. References [] K. P. Bennett. Decision tree construction via linear prograing. In M. Evans, edit, Proceedings of the 4th Midwest Articial Intelligence and Cognitive Percent Errs Percent Errs CPU Seconds MSMT MSMT-P GTO Training Set Accuracy (5 points) diension 9 29 Testing Set Accuracy (5 points) diension Training Tie diension Figure 3: Average results over trials f randoly generated decision trees with 3 decision nodes. Science Society Conference, pages 97{, Utica, Illinois, 992. [2] K. P. Bennett. Optial decision trees through ultilinear prograing. R.P.I. Math Rept No. 24, Rensselaer Polytechnic Institute, Troy, NY, 994. [3] K. P. Bennett and O. L. Mangasarian. Robust linear prograing discriination of two linearly inseparable sets. Optiization Methods and Software, :23{34, 992. [4] K. P. Bennett and O. L. Mangasarian. Bilinear separation of two sets in n-space. Coputational Optiization and Applications, 2:27{227, 993. [5] L. Breian, J. Friedan, R. Olshen, and C. Stone. Classication and Regression Trees. Wadswth International, Califna, 984. [6] C. E. Brodley and P. E. Utgo. Multivariate decision trees. COINS Technical Rept 92-83, University of Massachussets, Aherst, Massachusetts, 992. To appear in Machine Learning. [7] M. Fran and P. Wolfe. An algith f quadratic prograing. Naval Research Logistics Quarterly, 3:95{, 956.

6 [8] J. H. Friedan. Multivariate adaptive regression splines (with discussion). Annals of Statistics, 9:{ 4, 99. [9] N. Megiddo. On the coplexity of polyhedral separability. Discrete and Coputational Geoetry, 3:325{ 337, 988. [] J. R. Quinlan. Induction of decision trees. Machine Learning, :8{6, 984.

Guillotine subdivisions approximate polygonal subdivisions: Part III { Faster polynomial-time approximation schemes for

Guillotine subdivisions approxiate polygonal subdivisions: Part III { Faster polynoial-tie approxiation schees for geoetric network optiization Joseph S. B. Mitchell y April 19, 1997; Last revision: May