Curve fitting using linear models

Transcription

1 Curve fitting using linear models Rasmus Waagepetersen Department of Mathematics Aalborg University Denmark September 28, / 12

2 Outline for today linear models and basis functions polynomial regression and overfitting piecewise linear functions 2 / 12

3 Basis functions representations of unknown function Suppose we are given measurements (x i, y i ) where y i are observations of Y i with EY i = f (x i ) for some unknown function f. Idea: represent f ( ) as a linear combination of specified basis functions p 1 f (x) = β i B i (x) i=0 Example (linear regression): p = 2, B 0 (x) = 1, B 1 (x) = x. Polynomial regression: B i (x) = x i, i = 0,..., p 1 Trigonometric polynomials/discrete Fourier transform: B l1 (x) = cos(2πlx), B l2 (x) = sin(2πlx) x [0, 1] (or B l (x) = exp(i2πlx) with complex coefficients). 3 / 12

4 Overfitting Suppose we are given observations (x i, y i ) i = 1,..., n. Then we can always find a nth order polynomial ˆf (x) that fits exactly these observations - i.e. y i ˆf (x i ) = 0 for all i (Note: if design matrix n n and full rank then L = R n and P = I ). However, typically such a high order polynomial fits actual data too well - it fits not only f but also the noise. This means fitted ˆf bad for prediction of new observations. Another problem: polynomials global - if just one (x i, y i ) is changed this affects the whole fitted polynomial. 4 / 12

5 Piecewise linear function A first approximation of f might be a linear regression f (x) = a + bx but this is often too crude. A next step might be a piecewise linear function f f (x) = a l + b l x, x [c l, c l+1 [ for some cut -points or knots c l, l = 1,..., p. However, we typically want f to be continuous! This is ensured if we require a l + b l c l+1 = a l+1 + b l+1 c l+1. 5 / 12

6 A continous piece-wise linear curve from c 1 to c p is obtained with the following parametrization: β 0 + β 1 x x [c 1, c 2 ] f (c 2 ) + β 2 (x c 2 ) x ]c 2, c 3 ] f (x) = f (c 3 ) + β 3 (x c 3 ) x ]c 3, c 4 ] etc. This still defines a linear model! Basis functions: B 0 (x) = 1, B 1 (x), B 2 (x) = 1(x ]c 2, c 3 ])(x c 2 ) + 1[x > c 3 ](c 3 c 2 ),... 6 / 12

7 Alternative basis functions for piecewise linear functions Cut-points c 0, c 1,..., c p 1, c p. For i = 1,..., p: x c i 1 c i c i 1 x [c i 1, c i [ B i (x) = 1 x c i c i+1 c i x [c i, c i+1 [ 0 otherwise Note: f (x) = p 1 i=0 β ib i (x) piecewise linear and continuous. Hence we obtain exactly same set of functions as with basis on previous slide! Note: new set of basis functions local - only non-zero on intervals [c i 1, c i+1 [. Thereby more sparse X T X matrix. Disadvantage: f above is not smooth at cutpoints. 7 / 12

8 B-spline basis functions Consider doubly infinite sequence of equi-distant cut points..., c 1, c 0, c 1, c 2,... with (wlog) c i+1 c i = 1. Define B i (x) = B(x c i ) where 1 6 (x + 2)3 x [ 2, 1[ 1 6 (1 + 3(x + 1) + 3(x + 1)2 3(x + 1) 3 ) x [ 1, 0[ B(x) = 1 6 (4 6x 2 + 3x 3 ) x [0, 1[ 1 6 (1 3(x 1) + 3(x 1)2 (u 1) 3 )) x [1, 2[ 0 otherwise B(x) is a cubic spline: composed of the constant function g(x) = 0 and 4 third-order polynomials such that it is everywhere continuous and twice-differentiable. 8 / 12

9 The B-spline basis function: B(x, 0, 1) x 9 / 12

10 Cubic spline f (x) = f i (x) = a i0 +a i1 (x c i )+a i2 (x c i ) 2 +a i3 (x c i ) 3 x [c i, c i+1 [ Require continuity and twice differentiability: f i (c i+1 ) = f i+1 (c i+1 ) f i (c i+1 ) = f i+1(c i+1 ) f i (c i+1 ) = f i+1(c i+1 ) Again possible to compute basis functions and fit model in R. R Function bs() can be used to generate required basis functions for linear model. Suppose we use cut-points/knots c 1,..., c q and the q 1 associated cubic polynomials. Then we have p = (q 1) 4 3 (q 2) = q + 2 free parameters. 10 / 12

11 Equivalence of bases for cubic splines Fitting a cubic spline with knots 0, 1,..., q 1 (starting at c 1 and ending at c q ) is equivalent to fitting the linear model based on the B-spline basis functions B(x i), i = 1,..., q. Note: same number of free parameters. Intuitively makes sense, since both models generate continous piecewise cubic splines with continuous first and second derivatives. 11 / 12

12 Exercises 1. Write down the design matrix for a piece-wise linear regression model with cut-points c 1 and c 2 (i.e. the curve is composed of three segments). 2. Implement in R the above piece-wise model for your wind/power data. Try both types of basis functions. 3. Write down the equations for a cubic spline with knots 0, 1, 2 starting at 0 and ending at 2. Write down the associated design matrix. Do the same using the B-spline basis functions B(x i), i = 1, 0, 1, 2, 3. Compare the two design matrices. 4. Fit a cubic spline to your wind/power data (use R-function bs()). 12 / 12