22s:152 Applied Linear Regression

Transcription

1 22s:152 Applied Linear Regression Chapter 22: Model Selection In model selection, the idea is to find the smallest set of variables which provides an adequate description of the data. We will consider the available explanatory variables as candidate variables. (Some candidates may be transformations of others). Model selection can be challenging. If we have k candidate variables, there are potentially 2 k models to consider (i.e. each term being in or out of a given model). There are many methods for model selection, and we will only talk about a few in this class. One way to avoid looking at all possible subsets (potentially a very large number of models) is to use a stepwise procedure. For example, consider a backward stepwise method: 1. Start with the absolute largest model. 2. Choose a measure to quantify what makes a good model (R 2 is not a good choice, it will just choose the largest model every time). 3. Remove the term that most greatly increases the good model measure. 4. Continue to remove terms one at a time while the removal still provides a better model. 5. When removal of the next term would give you a worse model, stop the procedure. You ve found the best model. 1 2 The measure we use to make our choice should consider: 1. The number of explanatory variables in the model (we ll penalize models with too many). 2. The goodness of fit that the model provides. These express our conflicting interests... To describe the data reasonably well. (pushes toward more variables) To build a model simple enough to be interpretable. (pushes toward fewer variables) Some model selection measures (or criteria) Adjusted R 2,or R 2 : R 2 =1 RSS n 1 TSS n k 1 We prefer a model with a large R 2. Cross-Validation Criterion: ni=1 (Ŷ( i) CV = Y i) 2 n where Ŷ( i) is from the model fitted without using observation i. If you use a lot of parameters, you tend to over-fit the data, and we will do poorly at predicting a new Y not in the model-fitting (or training) data set. We prefer a model with a small CV. 3 4

2 Akaike information criterion (AIC): (This is assuming we have normal errors) AIC=n log e ˆσ 2 +2(k +1) We prefer a model with a small AIC. Bayesian information criterion (BIC): (This is assuming we have normal errors) BIC=n log e ˆσ 2 +(k +1)log e n We prefer a model with a small BIC. For both AIC and BIC, more parameters will provide smaller ˆσ 2,butthelasttermaddsona penalty related to the number of parameters in the model. Choose a best model using AIC in a Backward Stepwise Algorithm: Example: Crimeratedataset Crime-related and demographic statistics for 47 US states in The data were collected from the FBI s Uniform Crime Report and other government agencies to determine how the variable crime rate depends on the other variables measured in the study. VARIABLES RATE: Crime rate as # of offenses reported to police per million population Age: The number of males of age per 1000 population S: Indicator variable for Southern states (0 = No, 1 = Yes) Ed: Mean # of years of schooling x 10 for persons of age 25 or older Ex0: 1960 per capita expenditure on police by state and local government 5 6 Ex1: 1959 per capita expenditure on police by state and local government LF: Labor force participation rate per 1000 civilian urban males age M: The number of males per 1000 females N: State population size in hundred thousands NW: The number of non-whites per 1000 population U1: Unemployment rate of urban males per 1000 of age U2: Unemployment rate of urban males per 1000 of age W: Median value of transferable goods and assets or family income in tens of $ Pov: The number of families per 1000 earning below 1/2 the median income Use the step procedure in R to choose a good subset of predictors by subtracting terms one at atime. > crime.data=read.delim("crime.txt",sep="\t",header=false) > dimnames(crime.data)[[2]]=c("rate","age","s","ed","ex0", "Ex1","LF","M","N","NW","U1","U2","W","Pov") > attach(crime.data) > head(crime.data) RATE Age S Ed Ex0 Ex1 LF M N NW U1 U2 W X ## Fit the model including all candidate variables: > lm.full.out=lm(rate ~Age + S + Ed + Ex0 + Ex1 + LF + M + N + NW + U1 + U2 + W + Pov) > vifs=vif(lm.full.out) > round(vifs,2) Age S Ed Ex0 Ex1 LF M N NW U1 U2 W Pov

3 ## The starting AIC is ## remove variables one at a time. > model.selection=step(lm.full.out) Start: AIC= RATE ~ Age + S + Ed + Ex0 + Ex1 + LF + M + N + NW + U1 + U2 + W + Pov - NW LF N S Ex M <none> W U Ex U Age Ed Pov ## Remove NW and check if we should remove another. Step: AIC= RATE ~ Age + S + Ed + Ex0 + Ex1 + LF + M + N + U1 + U2 + W + Pov - LF N Ex S M <none> W U Ex U Age Ed Pov ## Remove LF and check if we should remove another. Step: AIC= RATE ~ Age + S + Ed + Ex0 + Ex1 + M + N + U1 + U2 + W + Pov - N S Ex M <none> W U Ex U Age Ed Pov ## Remove N and check if we should remove another. Step: AIC= RATE ~ Age + S + Ed + Ex0 + Ex1 + M + U1 + U2 + W + Pov - S Ex M <none> W U Ex U Age Ed Pov ## Remove S and check if we should remove another. Step: AIC= RATE ~ Age + Ed + Ex0 + Ex1 + M + U1 + U2 + W + Pov - Ex M <none> W U Ex U Age Ed Pov ## Remove Ex1 and check if we should remove another. Step: AIC= RATE ~ Age + Ed + Ex0 + M + U1 + U2 + W + Pov - M <none> W U U Age Ed Pov Ex ## Remove M and check if we should remove another. Step: AIC= RATE ~ Age + Ed + Ex0 + U1 + U2 + W + Pov - U <none> W U Age

4 - Ed Pov Ex ## Remove U1 and check if we should remove another. Step: AIC= RATE ~ Age + Ed + Ex0 + U2 + W + Pov <none> W U Age Ed Pov Ex ######################################### ## Procedure stops because removing ## ## any of the remaining variables ## ## only increases AIC. ## ######################################### ## Get the output from the final chosen model: > summary(model.selection) Call: lm(formula = RATE ~ Age + Ed + Ex0 + U2 + W + Pov) Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) e-06 *** Age ** Ed *** Ex e-07 *** U W Pov e-05 *** --- Signif. codes: 0 *** ** 0.01 * Residual standard error: on 40 degrees of freedom Multiple R-squared: ,Adjusted R-squared: 0.71 F-statistic: on 6 and 40 DF, p-value: 1.441e-10 > lm.out=lm(rate ~Age + Ed + Ex0 + U2 + W + Pov) > vif(lm.out) Age Ed Ex0 U2 W Pov You can use the step function with the BIC instead through an option in the step() statement. Setting k = log(n)inthestatementchangesthe criterion to the BIC rather than the AIC (Eventhough some of the output says AIC). > model.selection.bic=step(lm.full.out,k=log(47)) ##### similar output to previous example... ###### > summary(model.selection.bic) Call: lm(formula = RATE ~ Age + Ed + Ex0 + U2 + Pov) BIC tends to favor smaller models than the AIC. It has a heavier penalty for using more parameters. The only difference between the two chosen models in this example is that W (for wealth) is also removed from the BIC chosen model. You can also use the option direction = forward to build abestmodel. Butstartingwiththefull model is generally more reliable. Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) e-06 *** Age ** Ed *** Ex e-11 *** U * Pov e-05 *** --- Signif. codes: 0 *** ** 0.01 * Residual standard error: 21.3 on 41 degrees of freedom Multiple R-squared: ,Adjusted R-squared: F-statistic: on 5 and 41 DF, p-value: 1.105e

5 We mentioned stepwise procedures as a way to get around looking at every single possible model (there are 2 k possibilities). What about actually considering every possible model? Is this feasible?... Depends on the total number of variables. We can use the regsubsets function in the leaps library to consider the best model of each possible size (1 predictor, 2 predictors, 3 predictors, etc.) > library(leaps) ## The. below means we ll use all the other variables ## besides RATE as predictors in the largest model. > crime.subsets=regsubsets(rate ~., nbest=1,nvmax=13, data=crime.data) ## For each size of model, we ll only record ## the single best model (nbest=1). > summary(crime.subsets) Subset selection object Call: regsubsets.formula(rate ~., nbest = 1, nvmax = 13, data = crime.data) 13 Variables (and intercept) 1 subsets of each size up to 13 Selection Algorithm: exhaustive Age S Ed Ex0 Ex1 LF M N NW U1 U2 W Pov 1 ( 1 ) " " " " " " "*" " " " " " " " " " " " " " " " " " " 2 ( 1 ) " " " " " " "*" " " " " " " " " " " " " " " " " "*" 3 ( 1 ) " " " " "*" "*" " " " " " " " " " " " " " " " " "*" 4 ( 1 ) "*" " " "*" "*" " " " " " " " " " " " " " " " " "*" 5 ( 1 ) "*" " " "*" "*" " " " " " " " " " " " " "*" " " "*" 6 ( 1 ) "*" " " "*" "*" " " " " " " " " " " " " "*" "*" "*" 7 ( 1 ) "*" " " "*" "*" " " " " " " " " " " "*" "*" "*" "*" 8 ( 1 ) "*" " " "*" "*" " " " " "*" " " " " "*" "*" "*" "*" 9 ( 1 ) "*" " " "*" "*" "*" " " "*" " " " " "*" "*" "*" "*" 10 ( 1 ) "*" "*" "*" "*" "*" " " "*" " " " " "*" "*" "*" "*" 11 ( 1 ) "*" "*" "*" "*" "*" " " "*" "*" " " "*" "*" "*" "*" 12 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" " " "*" "*" "*" "*" 13 ( 1 ) "*" "*" "*" "*" "*" "*" "*" "*" "*" "*" "*" "*" "*" As Ex0, the1960policeexpenditures,isfirst chosen and appears in every best model, this may be the most important explanatory variable of crime rate. ## We ll consider models up to a ## 13 variable model (nvmax=13) The 5 variable model matches that of the stepwise BIC best model we saw earlier. And the 6variablemodelmatchesthatofthestepwise AIC best model we saw earlier. If you use regsubsets to record more than just the single best model of each size, you can see how different the BIC values are for the top X best models of each size in a visual plot... Statistic: bic E1-M-P Ed-E1-P E0-M-P Ed-E0-P Ed-E0-U2-P Ed-E0-W-P Ed-E0-M-P A-Ed-E0-P A-Ed-E0-U1-P A-Ed-E0-M-P A-Ed-E0-W-P A-Ed-E0-E1-U2-P A-Ed-E0-L-U2-P A-Ed-E0-U1-U2-P A-Ed-E0-U2-W-P A-Ed-E0-U2-P We ll keep the best 4 models of each size. > crime.subsets.2=regsubsets(rate~., nbest=4, nvmax=8, data=crime.data) ## The next line will give you the graphic ## for models of size 3 to 6. > subsets(crime.subsets.2,min.size=3,max.size=6,legend=f) ## The subsets() function is in the car library Subset Size This may be useful if you re deciding between models with similar BIC values, and some models seem better for you in terms of your research, and which variables are included. The plot also shows the model with the smallest BIC (if you show all subset sizes). 20

6 Here we keep track of the best 4 models of each size up to a model with all 13 variables included. The graphic becomes a bit hard to read when you look at all recorded models, so subsetting the picture (as on the previous page) is useful. > crime.subsets.3=regsubsets(rate~., nbest=4, nvmax=13, data=crime.data) > subsets(crime.subsets.3,legend=f) AIC function To compare model 1 to model 2 using AIC, you can just use the AIC function directly. > model.1=lm(rate ~Ex0 + Ex1 + LF + M + N) > model.2=lm(rate ~Age + S + Ex0 + Ex1 + U1 + Pov) > AIC(model.1) [1] Statistic: bic N W E1 E0 A-E1 E1-P A-E0 E0-P A-Ed-E0-M-NW-U1-U2-W-P A-Ed-E0-E1-M-U1-U2-W-P A-Ed-E0-M-N-U1-U2-W-P A-S-Ed-E0-M-U1-U2-W-P E1-M-P Ed-E1-P A-Ed-E0-E1-U1-U2-W-P A-Ed-E0-N-U1-U2-W-P A-Ed-E0-L-U1-U2-W-P E0-M-PEd-E0-U2-P A-Ed-E0-M-U1-U2-W-P Ed-E0-W-P Ed-E0-M-P A-Ed-E0-U1-P A-Ed-E0-M-P A-Ed-E0-E1-U2-P Ed-E0-P A-Ed-E0-U1-U2-P A-Ed-E0-L-U2-P A-Ed-E0-U1-U2-W-P A-Ed-E0-E1-U2-W-P A-Ed-E0-M-U1-U2-P A-Ed-E0-L-U2-W-P A-Ed-E0-P A-Ed-E0-W-P A-Ed-E0-U2-P A-Ed-E0-U2-W-P A-S-Ed-E0-E1-L-M-N-NW-U1-U2-W-P A-S-Ed-E0-E1-M-N-NW-U1-U2- A-Ed-E0-E1-L-M-N-NW-U1-U2- A-S-Ed-E0-E1-L-M-NW-U1-U2- A-S-Ed-E0-E1-L-M-N-U1-U2-W A-Ed-E0-E1-M-N-NW-U1-U2-W-P A-S-Ed-E0-E1-M-NW-U1-U2-W-P A-S-Ed-E0-E1-M-N-U1-U2-W-P A-S-Ed-E0-E1-L-M-U1-U2-W-P A-Ed-E0-E1-M-NW-U1-U2-W-P A-Ed-E0-E1-M-N-U1-U2-W-P A-S-Ed-E0-E1-M-U1-U2-W-P A-S-Ed-E0-M-N-U1-U2-W-P > AIC(model.2) [1] *smaller AIC is better. To compare the two models using BIC... > AIC(model.1,k=log(47)) [1] > AIC(model.2,k=log(47)) [1] *smaller BIC is better Subset Size 21 22