Regression trees with R

Transcription

1 Regression trees with R Emanuele Taufer file:///c:/users/emanuele.taufer/google%20drive/2%20corsi/5%20qmma%20-%20mim/0%20labs/l7-regression-trees.html#(1) 1/17

2 Implement decision trees with R To adapt a regression tree (or classification) we can use the tree() function from the tree library. The function automatically determines whether to implement a regression tree or a classification tree based on the dependent variable class: regression tree if Y is numeric classification tree if Y is factor The minimum input is very simple since it is sufficient to indicate the regression equation (or classification) and the data (similar to what has already been learned for the lm() function). file:///c:/users/emanuele.taufer/google%20drive/2%20corsi/5%20qmma%20-%20mim/0%20labs/l7-regression-trees.html#(1) 2/17

3 Example -Hitters First load the tree library and the Hitters dataset from the ISLR library library(tree) library(islr) data("hitters") ## clead data from rows with missing cases Hitters<-Hitters[complete.cases(Hitters),] To fit a tree with Salary f(y ears, Hits), use h.tree<-tree(salary~hits+years,hitters) summary(h.tree) ## ## Regression tree: ## tree(formula = Salary ~ Hits + Years, data = Hitters) ## Number of terminal nodes: 8 ## Residual mean deviance: = / 255 ## Distribution of residuals: ## Min. 1st Qu. Median Mean 3rd Qu. Max. ## file:///c:/users/emanuele.taufer/google%20drive/2%20corsi/5%20qmma%20-%20mim/0%20labs/l7-regression-trees.html#(1) 3/17

4 Plot of the tree plot(h.tree,lwd=3) text(h.tree,pretty=0,cex=1.2,col="blue") In the diagram, the height of the branches indicates the effectiveness of the split in the reduction of RSS. In the example, the first splits are the most effective. file:///c:/users/emanuele.taufer/google%20drive/2%20corsi/5%20qmma%20-%20mim/0%20labs/l7-regression-trees.html#(1) 4/17

5 Analysis of the tree h.tree ## node), split, n, deviance, yval ## * denotes terminal node ## ## 1) root ## 2) Years < ## 4) Hits < * ## 5) Hits > ## 10) Years < * ## 11) Years > * ## 3) Years > ## 6) Hits < ## 12) Years < * ## 13) Years > * ## 7) Hits > ## 14) Hits < ## 28) Years < * ## 29) Years > * ## 15) Hits > * file:///c:/users/emanuele.taufer/google%20drive/2%20corsi/5%20qmma%20-%20mim/0%20labs/l7-regression-trees.html#(1) 5/17

6 Interpret the analytical output To interpret the results in the previous slide, note that: Terminal nodes are indicated with an asterisk (8 in this case). The others are internal nodes The first line of the output gives us the key to reading the results: - node): node number - split: subdivision criterion - n: number of observations - deviance: RSS in that node in the case of regression trees; the value of the Gini index or entropy in the case of classification trees - the value ȳ in that node, that is, the prediction if the node is terminal. file:///c:/users/emanuele.taufer/google%20drive/2%20corsi/5%20qmma%20-%20mim/0%20labs/l7-regression-trees.html#(1) 6/17

7 Example: Node 1): includes all units of the dataset, 263 RSS = the Salary average is $ Node 2): the first split is defined by the Y ears < 4.5 condition, there are 90 players that satisfy the condition, the RSS for this group of players is $ the Salary average for the group is file:///c:/users/emanuele.taufer/google%20drive/2%20corsi/5%20qmma%20-%20mim/0%20labs/l7-regression-trees.html#(1) 7/17

8 Control the parameters of the tree The parameters for constructing the tree can be controlled with the tree.control() function tree.control(nobs, mincut = 5, minsize = 10, mindev = 0.01) Argument: nobs: number of observations in the analyzed data set. mincut: see help minsize: see help mindev: the minimum deviance to proceed to a split, decrease (increment) the default value to get a tree less (more) large. file:///c:/users/emanuele.taufer/google%20drive/2%20corsi/5%20qmma%20-%20mim/0%20labs/l7-regression-trees.html#(1) 8/17

9 Example To get the tree of slide 5 in the decision tree class, use setup<-tree.control(nrow(hitters), mincut = 5, minsize = 10, mindev = 0.15) h.tree<-tree(salary~hits+years,hitters,control=setup) Note that the new parameters have been inserted into the setup object which is then used as the tree argument file:///c:/users/emanuele.taufer/google%20drive/2%20corsi/5%20qmma%20-%20mim/0%20labs/l7-regression-trees.html#(1) 9/17

10 plot(h.tree,lwd=3) text(h.tree,pretty=0,cex=1.2,col="blue") file:///c:/users/emanuele.taufer/google%20drive/2%20corsi/5%20qmma%20-%20mim/0%20labs/l7-regression-trees.html#(1) 10/17

11 To obtain a larger tree, use, for example setup<-tree.control(nrow(hitters), mincut = 5, minsize = 10, mindev = 0.003) h.tree<-tree(salary~.,hitters,control=setup) Note: the Salary~. syntax tells R to use Salary as dependent variable and all other variables in the dataset as independent file:///c:/users/emanuele.taufer/google%20drive/2%20corsi/5%20qmma%20-%20mim/0%20labs/l7-regression-trees.html#(1) 11/17

12 plot(h.tree,lwd=3) text(h.tree,pretty=0,cex=1.2,col="blue") file:///c:/users/emanuele.taufer/google%20drive/2%20corsi/5%20qmma%20-%20mim/0%20labs/l7-regression-trees.html#(1) 12/17

13 Use the option type="uniform" to have a more readable result plot(h.tree,lwd=3,type="uniform") text(h.tree,pretty=0,cex=1.2,col="blue") file:///c:/users/emanuele.taufer/google%20drive/2%20corsi/5%20qmma%20-%20mim/0%20labs/l7-regression-trees.html#(1) 13/17

14 Prune the tree The cv.tree() function cross-validates to achieve the optimal level of tree complexity. To adapt regression trees use the FUN = prune.tree argument (in which case RSS will be the guiding criterion) To adapt classification trees use the FUN = prune.misclass argument (in which case the error rate will be the guiding criterion) The output of cv.tree() will report: the number of terminal nodes of each tree considered (size) the corresponding RSS (or error rate) (dev) other parameters (not discussed in class). file:///c:/users/emanuele.taufer/google%20drive/2%20corsi/5%20qmma%20-%20mim/0%20labs/l7-regression-trees.html#(1) 14/17

15 Output We carry out a cross evaluation of the tree obtained in slide 13 of this lab set.seed (3) cv.hitters =cv.tree(h.tree,fun=prune.tree ) cv.hitters ## $size ## [1] ## ## $dev ## [1] ## [8] ## [15] ## ## $k ## [1] -Inf ## [7] ## [13] ## ## $method ## [1] "deviance" ## ## attr(,"class") ## [1] "prune" "tree.sequence" file:///c:/users/emanuele.taufer/google%20drive/2%20corsi/5%20qmma%20-%20mim/0%20labs/l7-regression-trees.html#(1) 15/17

16 Plot We report the results for size and dev in a graph plot(cv.hitters$size,cv.hitters$dev,type="b", lwd=3,col="blue", xlab="nodi terminali", ylab="rss",main="cost complexity pruning" ) Cross-validation indicates that the optimal size is 3. file:///c:/users/emanuele.taufer/google%20drive/2%20corsi/5%20qmma%20-%20mim/0%20labs/l7-regression-trees.html#(1) 16/17

17 The pruned tree At this point, use the prune.tree() function to prune the initial tree to the number of nodes chosen based on the results obtained by cross-validation. prune.hitters=prune.tree(h.tree, best =3) plot(prune.hitters,lwd=3) text(prune.hitters,pretty =0,cex=1.2,col="blue") file:///c:/users/emanuele.taufer/google%20drive/2%20corsi/5%20qmma%20-%20mim/0%20labs/l7-regression-trees.html#(1) 17/17