Boston-Keio Workshop 2016. Doubly Cyclic Smoothing Splines and Analysis of Seasonal Daily Pattern of CO2 Concentration in Antarctica... Mihoko Minami Keio University, Japan August 15, 2016 Joint work with Ryo Kiguchi CO2 Data were provided by National Institute of Polar Research 1 / 37
Hourly observations of CO2 concentration at Syowa station in Antarctica (1984/ 2/ 3-2009/12/31) 25+ years (ppm) 1year 5days 2 / 37
Hourly observations of CO2 concentration at Syowa station in Antarctica The data clearly show strong temporal trend strong seasonal variation We are also interested to know if there is a daily pattern? If so, does it vary seasonally? an effect due to wind speed? If so, does it vary seasonally? an effect due to wind direction? In this talk, we introduce the method to analyze a seasonal daily pattern, doubly cyclic smoothing splines, and show the results of analysis that give answers to the above questions. 3 / 37
Outline...1 Cyclic cubic smoothing splines A cyclic cubic spline function Cyclic cubic smoothing splines Smoothing mechanism of cyclic cubic smoothing splines...2 A tensor product method: an extenstion to a multivariate smoothing method Roughness penalty for the tensor product method...3 Doubly cyclic cubic smoothing splines What do doubly cyclic cubic smoothing splines do? Wiggly components are shrunk more...4 Analysis of CO2 concentration at Syowa station in Antarctica Seasonal variation of daily Pattern Confidence interval curves for daily pattern Model selection Seasonal change of wind speed effect...5 Conclusions 4 / 37
1. Cyclic cubic smoothing splines 5 / 37
Cyclic cubic smoothing splines The cyclic cubic smoothing spline is a smoothing method to estimate periodic variation such as daily or annual pattern of time series observations. Day 1 Day2 Day 3 Day 4 It fits a cyclic cubic spline function which is a periodic piece-wise cubic function with continuity up to the second derivative. 6 / 37
A cyclic cubic spline function A cyclic cubic spline function g(t) is periodic When the period is T, g(t + kt ) = g(t) for k = 0, ±1, ±2, piece-wise cubic polynomial Given knots t (0) < t (1) < < t (K 1) < t (K) with t (K) t (0) = T, g(t) = f j (t) for t [t (j 1), t (j) ), j = 1, 2,, K where f j (t)s are cubic polynomial functions. That is, for t [t (0), t (K) ], it can be expressed as g(t) = K I [t (j 1),t (j) ) f j(t) j=1 7 / 37
A cyclic cubic spline function A cyclic cubic spline function K g(t) = I [t (j 1),t (j) ) f j(t) j=1 is also continuous up to the second derivative For j = 1, 2,, K 1, f j (t (j) ) = f j+1 (t (j) ), f j(t (j) ) = f j+1(t (j) ), f j (t (j) ) = f j+1(t (j) ) The values at the both endpoints t (0) and t (K) are equal up to the second derivative. f 1 (t (0) ) = f K (t (K) ), f 1 (t(0) ) = f K (t(k) ), f 1 (t(0) ) = f K (t(k) ) 8 / 37
Cyclic cubic smoothing splines A cyclic cubic spline function is flexible. To avoid overfitting, we impose a roughness penalty. In a most simplified case, the model and object function are defined as: Model y i = g(t i ) + ϵ i, ϵ i N(0, σ 2 ), i = 1,, n, i.i.d. where g(t) is a cyclic cubic spline function Penalized squared errors n t (K) Ω λ (g) = {y i g(t i )} 2 + λ g (t) 2 dt, λ > 0 (1) i=1 t (0) λ is called a smoothing parameter. 9 / 37
Example: Daily pattern of PM2.5 1/3 Hourly observations of PM2.5 1/3 in air for 28 days at Fukuoka In the left plot, 2.70 2.75 2.80 2.85 2.90 2.95 3.00 lambda/m = 8 lambda/m=1.2 5 10 15 20 black dots with solid lines depict hourly averages of PM2.5 1/3 the red dashed curve is the fitted cyclic cubic spline function with λ = 8 28 the green dotted curve is the fitted cyclic cubic spline function with λ = 1.2 28 10 / 37
Smoothing mechanism of cyclic cubic smoothing splines In the following, we assume knots are evenly spaced with Function value parameterization t (j) t (j 1) = h for j = 1, 2,, K We employ the function value parameterization to express cyclic cubic functions. For j = 1, 2,, K, let β j be the function value of g(t) at t (j), that is, β j = g(t (j) ), b j (t) be the corresponding cyclic cubic spline basis function with b j (t (i) ) = δ ij for i = 1, 2,, K, so that a cyclic cubic spline function g(t) can be expressed as g(t) = and K β j b j (t) (2) j=1 11 / 37
Penalty term with function value parameterization The penalty term can be expressed as t (K) g (t) 2 dx = β T D T B 1 Dβ (3) t (0) where β = (β 1, β 2,, β K ) T B and D are cyclic band matrices, B = h 6 G(4, 1) and D = 1 G( 2, 1) h where G(a, b) denotes a cyclic band matrix a b b b a b G(a, b) =......... b a b b b a 12 / 37
Least penalized squared error estimate ˆβ Suppose now that all observations were made at knots at each knot, m observations were obtained. Let y denote the sample average vector at the knots. Then, we have n {y i g(t i )} 2 = y 1 m y 2 + m y β 2 i=1 so that the minimization of penalized squared errors is equivalent to the minimization of S(β) = m y β 2 + λβ T D T B 1 Dβ (4) Least penalized squared error estimate ˆβ = Hy where H = ( I K + λ ) 1 m DT B 1 D 13 / 37
Eigenvalues and eigenvectors of G(a, b) For the even number of knots (K = 2q), eigenvalues of a cyclic band matrix G(a, b) with b > 0 are in descending order l 1 = a + 2b, l 2j = l 2j+1 = a + 2b cos 2πj (j = 1,, q 1) and the corresponding eigenvectors are u 2j = cos ( ) 2πj 1 k 2. k cos ( ) 2πj i k. cos (2πj) k, u 1 = 1 k (1, 1,, 1, 1) T,, u 2j+1 = 2 k sin ( ) 2πj 1 k. sin ( ) 2πj i k. sin (2πj) u 2q = 1 k (1, 1,, 1, 1) T. l 2q = a 2b, j = 1, 2,, q 1 14 / 37
Eigenvalues and eigenvectors of influence matrix H Recall that the estimate ˆβ of the function values at knots is given by ˆβ = Hy where ( I K + λ ) 1 m DT B 1 D Matrices D, B and I K share the same eigenvectors, so does H. The eigenvalues of the influence matrix H are given in descending order by ( γ 1 = 1, γ 2j = γ 2j+1 = 1 + λ m 12 1 cos 2πj ) 2 h 3 k ( 2 + cos 2πj ) k and γ 2q = ( 1 + λ m 48 ) 1 h 3 1 j = 1,, q 1, 15 / 37
Eigenvalues and eigenvectors of influence matrix H Eigen Values 0.15 0.25 0.3 0.1 0.1 0.3 0.3 0.1 0.1 0.3 0.0 0.2 0.4 0.6 0.8 1.0 Eigen Values of Influence Matrix 5 10 15 20 i= 1 0 5 10 15 20 i= 4 0 5 10 15 20 i= 8 0 5 10 15 20 j lambda/m = 8 lambda/m = 1.2 0.3 0.1 0.1 0.3 0.3 0.1 0.1 0.3 0.3 0.1 0.1 0.3 γ j i= 2 0 5 10 15 20 i= 6 0 5 10 15 20 i= 10 0 5 10 15 20 ˆβ = = ( I k + λ m DT B 1 D) 1 y k γ j (u j, y) u j j=1 where γ j s are eigenvalues and u j are eigenvectors of H. u j The black solid curves are cyclic cubic spline basis functions corresponding to u j. The red dashed curves are cyclic cubic spline basis functions multiplied by γ j (=γ j u j ). 16 / 37
Smoothing mechanism of cyclic cubic smoothing splines 2.70 2.75 2.80 2.85 2.90 2.95 3.00 lambda/m = 8 lambda/m=1.2 5 10 15 20 ˆβ = = y = k γ j u j u T j y j=1 k γ j (u j, y) u j j=1 k (u j, y) u j j=1 The smoothing mechanism can be understood as follows It decomposes the average observation vector y into the constant component (overall mean), and sin and cos components with frequencies 1 to q(= m/2). sin and cos components are shrunk. The higher the frequency is, the more the component is shrunk. The overall mean and shrunk components are summed up to produce ˆβ 17 / 37
2. A tensor product method: an extenstion to a multivariate smoothing method 18 / 37
A tensor product method : an extenstion to a bivariate smoothing method Suppose we have basis functions for a function space Ω 1 a 1 (s), a 2 (s),, a K1 (s) basis functions for a function space Ω 2 b 1 (t), b 2 (t),, b K2 (t). A tensor product method uses products of basis functions on Ω 1 Ω 2 a i (s)b j (t), i = 1, 2,, K 1, j = 1, 2,, K 2 as its basis functions. Thus, a bivariate function for the tensor product method can be expressed as K 1 K 2 f st (s, t) = β ij a i (s)b j (t) (5) i=1 j=1 19 / 37
Roughness penalty for the tensor product method Roughness penality for a tensor product smoothing function is defined as J(f st ) = Ω s Ω t λ s ( 2 f st s 2 ) 2 ( 2 ) 2 f st + λ t t 2 ds dt. When knots are evenly spaced, the penalty term can be approximated as Penalty term for a tensor product smoothing function J(f st ) λ s β T (S s I Kt )β + λ t β T (I Ks S t )β (6) where β is a vector of appropriately rearranged function values at grids. (Wood, 2006) 20 / 37
3. Doubly cyclic cubic smoothing splines 21 / 37
Doubly cyclic cubic smoothing splines Doubly cyclic cubic smoothing splines are generated using a tensor product method with: - basis functions for a function space with a yearly period f a 1, f a 2,, f a K a - basis functions for a function space with a daily period f d 1, f d 2,, f d K d. We start with a univariate function of time t, defined on a coil, that winds around a torus: K d K a f(t) = β ij fi a (t)fj d (t) i=1 j=1 Then, to have a function that is smooth in two directions, we re-express this as: K d K a f(s, t) = β ij fi a (s)fj d (t) i=1 j=1 and consider penalty to this function. 22 / 37
What does doubly cyclic cubic smoothing spline do? When knots are evenly spaced, and the numbers of observations are equal for all knots, then, original basis functions can be linearly transformed into Oorthogonal basis functions fij cos : cyclic cubic spline function whose values at knots are equal to cos (2πt(ih a + jh d )) for i = 0, qa, j = 0, ±1, ± qd fij sin : cyclic cubic spline function whose values at knots are equal to sin (2πt(ih a + jh d )) for i = 0, qa, j = 0, ±1, ± qd where q d K d/2 1, q a K a /2 1 and for h a = 1/K a, h d = 1/K d, These are the eigenvectors of the influence matrix for doubly cyclic cubic smoothing splines. 23 / 37
What do doubly cyclic cubic smoothing splines do? Estimated function ˆf(t) = q a i=0 j= qd qd (1 cos 2πih a ) (1 2 (1 cos 2πjh d ) 2 + λ a + λ d 2 + cos 2πih a 2 + cos 2πjh d ( < u cos ij, y > ) u cos f cos ij 2 ij + < usin ij, y > u sin f sin ij 2 ij where qd K d/2 1, qa K a /2 1 and for h a = 1/K a, h d = 1/K d, ) 1 u cos ij : vectors of values of cos (2πt(ih a + jh d )) at knots u sin ij : vectors of values of sin (2πt(ih a + jh d )) at knots fij cos : cyclic cubic spline function with cos (2πt(ih a + jh d )) as values at knots fij sin : cyclic cubic spline function with sin (2πt(ih a + jh d )) as values at knots y : vector of the averages at knots 24 / 37
Wiggly components are shrunk more The doubly cyclic cubic smoothing spline shrinks the components of basis function with values at knots cos (2πt(ih a ± jh d )), and basis function with values at knots sin (2πt(ih a ± jh d )) by multiplying them by the shrinkage rate (1 cos 2πih a ) (1 2 (1 cos 2πjh d ) 2 ) 1 + λ a + λ d 2 + cos 2πih a 2 + cos 2πjh d Shrinkage rate and sum them up. 0.0 0.2 0.4 0.6 0.8 1.0 5 10 15 0 5 10 15 20 i(annual) 0 5 10 15 j(daily) The larger i or j is, the more wiggly the basis function is in either direction. The more wiggly the basis function is in either direction, the more its coefficient is shrunk. 25 / 37
4. Analysis of CO2 concentration at Syowa station in Antarctica 26 / 37
A model with temporal trend and seasonal daily pattern We start with a linear additive model for CO2 concentration with temporal trend and seasonal daily pattern as explanatory terms. where Model 1: Y = f tr (t) + f day,year (t) + ϵ Y f tr (t) f day,year (t) : CO2 concentration : a cubic spline function of time t for temporal trend : a doubly cyclic cubic spline function of time t with daily and annual cycles ϵ : random error with variance σ 2 We used R package mgcv by Simon Wood for analysis. 27 / 37
Temporal trend and annual variation Temporal trend is almost linear CO2 concentration has increased 40ppm in 25 years The range of annual variation is 1.1ppm CO2 concentration is low in summer and high in winter 28 / 37
Seasonal variation of daily Pattern in CO2 concentration Daily pattern of CO2 concentration has a seasonal variation It has the largest daily variation (0.017ppm) in summer (January 4th) 29 / 37
Confidence interval curves for daily pattern 95% confidence interval curves for January 4th and July 4th. CO2 concentration 0.015 0.010 0.005 0.000 0.005 0.010 0.015 January 4 July 4 0 5 10 15 20 25 30 Hourly variation is significant in summer (January 4th), but not significant in winter (July 4th). Time 30 / 37
Effects of wind speed and direction Wind speed might have an effect on CO2 concentration. The effect of wind speed might depend on the wind direction. where Model 2: y = f tr (t) + f day,year (t) + f ws,wd (s, d) + ϵ Y f tr (t) f day,year (t) : Co2 concentration : a cubic spline function of time t for temporal trend : a doubly cyclic cubic spline function of time t with daily and annual cycles f ws,wd (s, d) : tensor product of a cubic spline function of wind speed s and a cyclic spline function of wind direction d ϵ : random error with variance σ 2 31 / 37
Seasonal effect of wind speed The effect of wind speed might differ by season. where Model 3: y = f tr (t) + f day,year (t) + f ws,year (s, t) + ϵ Y f tr (t) f day,year (t) : Co2 concentration : a cubic spline function of time t for temporal trend : a doubly cyclic cubic spline function of time t with daily and annual cycles f ws,year (s, t) : tensor product of a cubic spline function of wind speed s and a cyclic cubic spline function with annual cycle of t ϵ : random error with variance σ 2 32 / 37
Model selection for CO2 concentration We fitted the following three models and compared AIC where model formula AIC model 1 y = f tr (t) + f day,year (t) + ϵ 19942.5 model 2 y = f tr (t) + f day,year (t) + f ws,wd (s, d) + ϵ 18725.6 model 3 y = f tr (t) + f day,year (t) + f ws,year (s, t) + ϵ 16189.5 f tr (t) f day,year (t) : a cubic spline function of time t : a doubly cyclic cubic spline function of time t with daily and annual cycles f ws,wd (s, d) : tensor product of a cubic spline function of wind speed s and a cyclic spline function of wind direction d f ws,year (s, t) : tensor product of a cubic spline function of wind speed s and a cyclic cubic spline function with annual cycle of t 33 / 37
Seasonal change of wind speed effect January - March CO2 concentration January 4 0 100 200 300 400 500 Wind Speed CO2 concentration February 4 0 100 200 300 400 500 Wind Speed CO2 concentration March 4 0 100 200 300 400 500 Wind Speed April 4 May 4 June 4 April - June CO2 concentration 0 100 200 300 400 500 Wind Speed CO2 concentration 0 100 200 300 400 500 Wind Speed CO2 concentration 0 100 200 300 400 500 Wind Speed July 4 August 4 September 4 July - September CO2 concentration 0 100 200 300 400 500 CO2 concentration 0 100 200 300 400 500 CO2 concentration 0 100 200 300 400 500 Wind Speed Wind Speed Wind Speed October 4 November 4 December 4 October - December CO2 concentration 0 100 200 300 400 500 CO2 concentration 0 100 200 300 400 500 CO2 concentration 0 100 200 300 400 500 Wind Speed Wind Speed Wind Speed 34 / 37
Conclusions We proposed the doubly cyclic cubic smoothing spline method. For a simple model, the eigenvalues and eigenvectors of the influence matrix can be explicitly expressed with the values of trigonometric functions with different frequencies. This expression shows that the more wiggly the basis function is, the more its coefficient is shrunk. We analyzed CO2 concentration at Syowa station in Antarctica using this method. CO2 concentration has a strong temporal trend and annual variation. Daily pattern of CO2 concentration has a seasonal variation. Hourly variation is significant in summer (January), but not significant in winter (July). The effect of wind speed also has annual variation. Flexible regression models using nonparametric smoothing methods enable us to analyze the data of interest more precisely. 35 / 37
Reference..1 Green, P.J. and Silverman, B.W. (1993) Nonparametric Regression and Generalized Linear Models: A roughness penalty approach, Chapman&Hall/CRC...2 Wood, S.N. (2003) Thin plate regression splines. J. R. Statist. Soc. B 65, 95-114..3 Wood, S.N. (2006a) Low-Rank Scale-Invariant Tensor Product Smooths for Generalized Additive Mixed Models. Biometrics 62(4): 1025-1036..4 Wood, S.N. (2006b) Generalized Additive Models: An Introduction with R, Chapman Hall/CRC...5 Kiguchi, R. and Minami, M. (2012) Cyclic Cubic Regression Spline Smoothing and Analysis of CO2 Data at Showa Station in Antarctica, Proceedings of International Biometric Conference 2012...6 Kiguchi, R. (2014) Doubly Cyclic Smoothing Splines and Analysis of CO2 Data at Syowa Station in Antarctica, Master s thesis, Keio University. 36 / 37
Thank you for your attention! 37 / 37