Applied Statistics : Practical 9

Transcription

1 Applied Statistics : Practical 9 This practical explores nonparametric regression and shows how to fit a simple additive model. The first item introduces the necessary R commands for nonparametric regression using a simulated example. The second item uses the cars dataset in R, while data for the third and fourth items are available on the course website. 1. A simple simulated example We consider first a simple simulated example we will use to compare different methods to fit a regression function. set.seed(1122) # so we all get the same 'random' data x<-seq(0,1,len=100) # equally spaced grid of predictor values true.func<-sin(2*pi*x)+2*x # defines the true (unkown) function f y<-true.func+rnorm(length(x),0,0.3) # add random i.i.d errors points(x,true.func,type='l',lwd=2) # plot the true function y x #superimposed to the observed data You can later try to change the amount of noise in the data (the standard deviation in the rnorm function) or the period in the sinusoidal function to see how this impact the estimation. Let us now try to estimate the regression function using a nearest neighbors estimator. We need to install and load the R package FNN which contains the function to find the nearest neighbors in a dataset. 1

2 install.packages("fnn") and now we can define a function for the estimator: library(fnn) fknn<-function(x,y,k=5){ # this is a function definition + fx<-rep(na,length(x)) + index<- get.knn(data=x,k=k)$nn.index # finds the K nearest neighbors for every x + for (i in 1:length(x)){ + fx[i]<-mean(y[index[i,]]) # avereges the K nearest neighbors + } + fx # returns the estimated f + } We can use the function above to get the nearest neighbors estimators: NN_fhat<-fkNN(x,y,K=5) points(x,nn_fhat,type='l',lwd=2) Try with different values of K and choose the best one by visual inspection. The visual inspection suggests a number of neighbors between 10 and 15. Consider now the kernel smoothing estimator, which is available in R with the command ksmooth: KN_fhat<-ksmooth(x,y,kernel="normal",bandwidth=0.5) points(kn_fhat$x,kn_fhat$y,type='l',lwd=2) Try using different values for the bandwidth h, which one provides the best fit? Remember that smaller values of the bandwidth leads to a high variation in the fitted curve, while larger values of h give a smoother curve. The visual inspection suggests a bandwidth around You can look at the help of the ksmooth function to learn which other kernel functions (in addition to the Gaussian one) are available. Does the change of the kernel impact on the curve estimate? The other only available kernel is the box function 1 [x h/2,x+h/2], which provides a less smooth estimate, whose irregularities can be detected by the human eye. To apply a local polynomial smoother to the same data: LP_fhat<-loess(y~x,span=2,degree=2) points(x,lp_fhat$fitted,type='l',lwd=2) Here you have two parameters that control the fit: the degree of the piecewise polynomial (set by the option degree) and the bandwidth (option span). Try different values for these two parameters (check the help for admissible values), which ones provide the best fit? 2

3 Visual inspection suggests a bandwidth of 0.5 for a second degree polynomial or a bandwidth between 0.25 and 0.3 for a first degree polynomial. The smoothing splines estimator is implemented in R in the function smooth.spline. To control the smoothing, you can alternatively specify the smoothing parameter λ, via the option spar in the function (where λ is a monotone function of spar, see the help for more details) SS_fhat<-smooth.spline(x,y,spar=0.5) points(x,ss_fhat$y,type='l',lwd=2) or you can specify the effective degrees of freedom of the fitted curve. SS_fhat<-smooth.spline(x,y,df=3) points(x,ss_fhat$y,type='l',lwd=2) You can get the equivalent the parameters spar, λ and the effective degrees of freedom associated with a fitted model with SS_fhat$lambda SS_fhat$df SS_fhat$spar Choose visually an appropriate fit and report your choice of λ and the correspondent effective degrees of freedom. A reasonable fit is obtained with spar= 0.7, corresponding to λ = and 8.4 effective degrees of freedom. An automatic choice of the smoothing parameters can be obtained using the option cv=true. SS_fhat<-smooth.spline(x,y,cv=TRUE) points(x,ss_fhat$y,type='l',lwd=2,col=3) What are the value of λ and the effective degrees of freedom selected by cross-validation? The cross-validation method selects λ = and 6.57 effective degrees of freedom, not too far from what could be expected from visual inspection. If we set cv=false (without selecting the parameter ourselves), the function chooses the smoothing parameter via generalized cross validation, a modified version of the cross-validation error. Now superimpose in the same plot the fitted curves obtained from the various non parametric estimators, with your choice for the best smoothing parameters. Which one perform best in this case? NN_fhat<-fkNN(x,y,K=13) points(x,nn_fhat,type='l',lwd=2) KN_fhat<-ksmooth(x,y,kernel="normal",bandwidth=0.15) points(kn_fhat$x,kn_fhat$y,type='l',lwd=2, col=2) 3

4 LP_fhat<-loess(y~x,span=0.5,degree=2) points(x,lp_fhat$fitted,type='l',lwd=2,col=3) SS_fhat<-smooth.spline(x,y,cv=TRUE) points(x,ss_fhat$y,type='l',lwd=2,col=4) y x Local polynomial and smoothing splines perform best. Regression (cubic) splines can be implemented using the package splines and specifying either the knots or the effective degrees of freedom (in this case equispaced knots are assumed). The function ns build the matrix G we have seen in the lecture notes, than we can simply use lm to fit the model. library(splines) reg_mod<-lm(y~ns(x,df=8)) points(x,reg_mod$fitted.values,type='l',lwd=2) Imagine now you are confronted with a more complicated function which contains some local feature you want to describe in your model: set.seed(1122) # so we all get the same 'random' data x<-seq(0,1,len=100) # equally spaced grid of predictor values true.func<-sin(2*pi*x)+2*x # defines the true (unkown) function f 4

5 true.func[71:80]<-3*sin(2*seq(0,pi,len=10)) y<-true.func+rnorm(length(x),0,0.3) # add random i.i.d errors points(x,true.func,type='l',lwd=2) # plot the true function superimposed to the observed reg_mod<-lm(y~ns(x,df=8)) points(x,reg_mod$fitted.values,type='l',lwd=2) As you can see, we are completely missing the local feature in the fit. We would like therefore to place additional knots between x = 0.7 and x = 0.8. knots<-c(0.2,0.4,0.6,0.7,0.72,0.74,0.75,0.76,0.77,0.8) reg_mod<-lm(y~ns(x,knots=knots)) points(x,reg_mod$fitted.values,type='l',lwd=2) Try to fit this curve using a smoothing spline. What happens? To get an accurate fit for the local feature, we are forced to choose lambda small enough that in the rest of the domain we get a very irregular curve. 2. Cars data We have seen in the first part of the course the dataset cars that which is provided by R and contains speeds and stopping distances for a set of cars. The aim was to predict the stopping time (dist) from the speed (speed). However, neither linear models nor generalized linear models provided a completely satisfactory fit for the data. We try now fitting a nonparametric regression. data(cars) attach(cars) plot(speed,dist) Fit a regression function using smoothing spline and choosing the smoothing parameter using generalized cross-validation. What is an appropriate choice of λ in this case? What are the effective degrees of freedom of the selected model? lambda = and the effective degrees of freedom are cars_fit<-smooth.spline(speed,dist,cv=false) cars_fit$df ## [1] cars_fit$lambda ## [1] plot(speed,dist) points(cars_fit$x,cars_fit$y,type='l') 5

6 dist speed Compare the fitted curve with what you would obtain from a linear or quadratic parametric model. cars_mod<-lm(dist~speed) cars_mod2<-lm(dist~speed+i(speed^2)) plot(speed,dist) points(speed,cars_mod$fitted.values,type='l',lwd=2,col=1) points(speed,cars_mod2$fitted.values,type='l',lwd=2,col=2) points(cars_fit$x,cars_fit$y,type='l',lwd=2,col=4) 6

7 dist speed As suggested by the number of effective degrees of freedom, the nonparametric model selects an intermediate choice between a linear and a quadratic model. 3. Signature acceleration data In a neurophysiological study, researchers put an accelerometer on the index finger of the participants when they are asked to write their signature. The researchers need first to estimate the acceleration as function of time for each participant. The file signature.txt on the course website contains the data for the first participant of the experiment. data<-read.table("signature.txt",header=true) attach(data) plot(time, acceleration) Looking at the plot, which method should be preferred among the ones we have considered in this practical? Why? The prominent presence of a localized feature around 0.75 seconds suggests the use of regression splines. Fit the nonparametric regression curve and find the value of the estimated value and position of the acceleration peak. We fit the curve using regression splines and choosing the knots so that they are dense around the localized feature at time We realize that an additional knot is also needed at the beginning to catch the rapid increase of the acceleration. If we use the predict command to evaluate the function on a finer grid, we find that the peak is at seconds and its estimated value is

8 knots<-c(0.05,0.1,0.4,0.6,0.7,0.72,0.74,0.75,0.76,0.77,0.8) sig_mod<-lm(acceleration~ns(time,knots=knots)) plot(time,acceleration) points(time,sig_mod$fitted.values,type='l',lwd=2) acceleration time 4. A simple additive model We see now how to fit a simple additive model. The file ozone_data contains 330 observations of the concentration of ozone, a measure of the pressure gradient and the day of the year in which the measurement have been taken. The aim is to fit an additive model for the concentration of ozone. You may need to install the package gam first. library(gam) data<-read.table("ozone_data.txt",header=true) attach(data) ozone_model<-gam(ozone~0+lo(pressure_grad,span=0.5,degree=2) + +lo(day,span=0.5,degree=2)) The lo function specifies that the predictor has to be smoothed with a local regression with the chosen degree and bandwidth. Then the function gam fits the additive model using the backfitting algorithm. What is the algebraic form of the model? Let Y i be the concentration of ozone for the i th observation, P i the correspondent pressure gradient and D i the day. The algebraic form of the model is Y i = f 1 (P i ) + f 2 (D i ) + ε i, 8

9 where ε i are i.i.d errors with zero mean and variance σ 2 and f 1 and f 2 are unknown regression functions. If we had use the formula ozone~ lo(pressure_grad,span=0.5,degree=2)+..., R would have included an intercept in the model. What would have been its algebraic form in this case? Y i = β 0 + f 1 (P i ) + f 2 (D i ) + ε i. We can now evaluate the fit by plotting the estimated regression functions and the marginal residuals (the difference between the observations and the other regression function): plot(ozone_model,residuals=true) Is the smoothing appropriate? Try changing the bandwidth in the lo function. The smoothing appears reasonable, it may be possible to slightly shorten the bandwidth for the pressure gradient. Alternatively, it is possible to use smoothing splines: ozone_spl<-gam(ozone~s(pressure_grad,df=3)+s(day,df=3)) #or ozone_sp2<-gam(ozone~s(pressure_grad,spar = 0.5)+s(day,spar=0.5)) 9