1. INTRODUCTION. condi tioned, 2) this degrading

Transcription

1 EVALUATING LINEAR MODEL REPRESENTATIONS OF CUBIC SPLINES USING PROC REG Hina Mehta, Thomas Capizzi Merck Sharp & Dohme Research Laboratories l, i, ) 1. INTRODUCTION Spline functions, are defined as piecewise continuous polynomials of degree n with n-l continuous derivatives (1-13). The abscissa values corresponding to the join points of the spline are ~alled. knots. Quadratic and CUbLC sp!lnes seem to be the most popular among statistical practitioners because of their computational simplicity. At least five papers presented at previous SUGI conferences ~ave discussed applications of spllne functions (1-5). Once the number and location of the knots have been specified, the spline can be fit to data using ordinary least squares. There are several linear model representations of a sp~i~e func~ tion. They are: (a) + functlons (6;7;8) (b) ANW-sp1ines (9;10;11) and (c) B-splines (12;13.) The '+' function representation is the simpler approach both for under-. standing and statistical hypothesls testing. If the number of knots (i.e., polynomial pieces) is large or if the knots are placed close together, then this representation is thought to be an ill-condi~ioned linear system because of multlcollinearity (cf. 7;8;11). However, no formal work seems to have been done in determining specific circumstances which will produce ill-conditioning. The other two representations are considered more stable and computationally efficient basis for fitting splines using linear models. Their motivation is analagous to that of orthogonal polynomials and transformations of the predictor variables in order to make the columns of the design matrix orthogonal in ordinary polynomial or multiple regression. The purpose of this paper is to show by applying diagnostic procedures for col linearity obtained with PROC REG to a published data set (14) that 1) the '+' function linear model approach suffers from degrading collinearity with just two interior knots whereas the 'ANW' and IB' splines bases are well- condi tioned, 2) this degrading collinearity has potentially harmful effects on tests for structural change (9; 10) and to discuss the noneffectiveness of a remedy for the collinearity problem. Collineari ty and its diagnostic techniques are reviewed in section 2. In section 3, the linear model formulations of spline functions, a test for structural change, and end conditions are discussed. The results of our col linearity analysis of the published data are given in Section 4. Section 5 contains conclusions, recommendations and areas for further research. 2. COLLINEARITY IN REGRESSION 2.1 Definition Consider the usual linear model Y = X~ + E (2.1) where-y is-the-n x 1 vector of observations, X is tren x p matrix of known predictor (or explanatory) variables, B is the p x 1 vector of regression coefficients to be estimated, and E is the N x 1 vector of errors with-the usual assumptions. COllinearity denotes the presence of near linear relationships among the predictor variables (i.e., the colu~ns of X). This causes the matrix X X to be ill-conditioned (i.e., almost singular). The potential harmful effects that result from collinear data are twofold: (a) numerical instability of the least squares solution, b and its variance matrix, and (b) decreased precision (inflated variances) for estimated parameters. Inflated variances are quite harmful to the hypothesis testing, estimation and forecasting aspects of linear regression. 2.2 Col linearity Diagnostics in SAS Options in PROC REG allow the computation of two diagnostics for collinearity. The two diagnostics are (a) variance inflation factors (VIF) (IS) and (b) the diagnostic procedure of Belsley, Kuh, and Welsch (BKW) (16). Variance inflation factors are the diagonal elements of the inverse correlation matrix of X. Their colline- 562

2 r, arity diagnostic potential sterns from the relation 2 VIF. = l!(l-r. ), (2.4) where R. is the multiple correlation co~fficient of the ith variable regressed on the remaining predictor variables. A large VIF (greater than 10) indicates R~ near- unity and hence points to colliffearity. BKW (16) used the singular value decomposi-tion of the matrix X (scaled to unit length) to provide a diagnostic procedure. This procedure involves (a) the calculation of the condition indices of the matrix which are defined as the ratios of the largest singular value of X to each individual singular value (or equivalently the square roots of the ratio of the largest eigenvalue of X X to each individual eigenvalue and (b) the proportion of the variance of each regression estimate, bi' j=l.., P, acc~unted for by each eigenvector of X X. BKW claimed that degrading col linearity exists if the following two conditions hold: I) A singular value with a high condition index (greater than 30 or 100) which is associated with 2) high variance decomposition proportions (greater than SO%) for two or more estimated regression coefficient variances. The number of large condition indices gives the number of dependencies among the columns of X. The variancedecomposition proportions identifies the variables involved in the near dependencies. 2.3 Remedies for Collinearity There are several possible corrective measures for collinearity. 'l'hey include: (a) Collecting new data with a well designed experiment (which often is not practical), (bj Bayesian or mixed estimation, (16), (c) A biased estimation procedure such as ridge regression, (15) or (d) restricted least squares (RLS), (17). 3. Spline Regression We limit our discussion to cubic splines. 3.1 Linear Model Representations Suppose data are g1ven by Y i = S(t i ) + e i, i=l,,n ( 3.1) where S(t) is a cubic spline with knots at a l =k l <k 2 <,, <k,_l<k,= a 2 and e., independent identically distribueed random variabtes with mean 0 and variance, cr. It can be easily shown that the expression t-l S(t)=SO+Slt+S2t2+S3t3+" S (t_k.)3 J=2 j+2 J + where (3.2) (t-k j )+ = t-k j, if t>k j, j=2, o, otherwise is a cubic spline over the interval (ai' a 2 ) The '+' function formulation of a cubic spline can be set up as a linear regression model with the elements of the design matrix given by j-l._ ti ' J-l,2,3,4 x. 1.J. ( t -k _ ), 3 j=s,6, i j 3 + i=1,2,...,n (3.3) It is believed (7,8,11) that the '+'function representation has severe collineari ty when there are large numbels of interior knots k 2, k it-i. In this case, two alternative bases for splines (ANW-splines (9,11) and B-splines (12,13)) are recommended because they are more stable linear systems. These representations are just linear reparameterizations of the '+' function approach i.e., (3.4) where T is a pxp non-singular matrix and Z is the reparametrized matrix of predictors. For ANW-splines, the parameter vector, a, represents the ordinate values of the spline at the knots and the two pseudo-knots which estimate the boundary conditions of the spline (10,11). For B-splines it is difficult to interpret what the parameter vector ~ represents (8). Th~ choice. of a right linear nodel formulat1.oll of spll.lle depends on the spec i fic problem at hand. If it is desired to test hypotheses on the parameters B in "(3.3), then the '+' function ' bases should be used and assessed for any collinearity problem. Likewise if the objective of the analysis involves testing hypotheses on linear combination ~ = T~, then the appropriately reparametrized model (3.4) should be employed and also examined for col linearity_ Using a well-conditioned basis to fit a spline, when one is actually interested in estimating the parameters of an equivalent ill-conditioned parameterization, does little good and provides 563

3 a false sense of security. 3.2 Testing for Structural Change Spline functions have been used to test for structural change (10;11), that is to test for changes in the regression function at the interior knots (k Z ',.,.,k,). To check for structural-change one tests the hypothesis,j=2,.,t-l where~. is given in (3.2). )+2 (3.5) In this case, the '+' function linear model should be the procedure of choice. However, the usual approach has been to use ANW-splines and then backtransform the estimated parameters to obtain the test statistic for structural change. As we show in section 4, this strategy hides potentially harmful collinearity in tests for structural change and thus causes problems in the interpretation of results. 3.3 Specifying End Conditions Cubic splines are completely determined if the values of the spline at the knots and either (a) the values of the first derivative or (b) the second derivative at the end points are known. For the linear model formulations discussed in section (3.1), the end point conditions are implicitly estimated by ordinary least squares. However, another method is to specify the end conditions in advance i.e., incorporate them in the model formulation or as a set of linear restrictions on the regression parameters (RLS). There are several ways to express end conditions (9,10,11,18), e.g., s " " S (k,) = ~,S (k'_l) (3.8) where 5" (k) is the second derivative and TIl and TIt are proportionality constants to be specified. Recalling that RL5 (or equivalently model reformulation) is a possible remedy for collinearity, specifying end conditions in advance could have the beneficial effect of more precise estimates in terms of mean squared error. Unfortunately there are few guidelines concerning the choice of end conditions, and incorrect specication could result in biased estimates and, hence, affect the interpretation of results. 4. Col linearity Analysis of the Indy 500 Data This example originates from an article by Barzel (14), in which the growth of technical progress in the Indianapolis 500 race from its inception in 1911 and the effect that the non-racing war years had on this progress were examined. Poirer (10) examined whether there were structural changes in the Indy 500 winning speed vs. time (Year-1910) relationship due to two wars by employing a ANW-Spline regression model with knots {I, 7.5, 33.5, 61} (the interior knots were placed midway through the non-racing years) and end conditions constants 'IT1 = TI4 = 2 (given without justification). Like Barzel, Poirier concluded that the non-racing war years hampered technical progress. Sampson (18) showed that the results of the structural change tests were sensitive to the specification of end conditions. He used ANW-splines with a) no end conditions and b) Tl = 2 and TI4 = 4, and found that the results and conclusions differed with those obtained by Poirier. Further analysis, led Sampson to conclude that a quadratic spline with one interior knot at 33.5 provided the most parsimonious description of the data. The purpose of our reanalysis was twofold. First we wanted to provide an example in which the '+' function approach can suffer from degrading collinearity with just a few interior knots whereas ANW-splines and -B-splines do not. Secondly, that the tests for structural change are degraded by '+' function col linearity and this cannot be ascertained by employing ANW-splines. To this end we present the '+' function formulation of the three cubic spline models described above in Table 2. The coefficients for th) variables containing PI = (X-7.5)+ dnd P, = (X-33.S)!, are estimates of struceu:al change due to WWI and WWII, respectlvely. The collinearity diagnostics for the model (a) in Table 2 and the equivalent ANW and B-spline formulations are given in Table 1. For the '+' function representation (Table la), there are two condition indices greater than 100 indicating two near dependencies. The variance proportions as well as the large variance inflation factors show that a 1.1 the regression estimates suffer from degrading collinearity. Also, the WWI structural change estimate (b ) appears to be degraded more than the 4 WWII estimate 564

4 (b S )' The diagnostics applied to the corresponding ANW-spline and B-splines (Tables 2b and 2c, respectively) show that, as expected, these respresentations are well conditioned bases for this cubic spline function. Table 3 contains the collinearity diagnostics for the restricted models (bl and (el in 'I'able l. Also as advertised, the restricted models, while exhibiting degrading col linearity according to the usual rules of thumb, provide a large reduction in the degree of collinearity. 'l'his reanalysis of the Indy 500 data has shown that, in addition to the problems associated with the specification of end conditions (18), the potentially harmful effects of col linearity in assessing structural change must be considered when interpreting results. 5. Concluding Remarks We have shown that for a specific cubic spline applied to a published data set, the '+' function approach with just two interior knots is an ill-conditioned linear system. This should not have been a surprising result since it is well known that simple quadratic and cubic polynomial regression is collinear (191 (See Appendix AI. Unpublished work has shown that degrading col linearity for '+' functions, exists in many applications including the use of linear splines. Thus, the use of '+' functions should always be accompanied by a col linearity analysis. As advertised the 'ANW' and 'B' spline bases are well conditioned. Obviously the choice of bases depends on the purposes of the application. If the purpose is to test for structural change, the '+' function basis should be employed. To apply the col linearity diagnostics to a wellconditioned bases, and then backtransform these parameter estimates to obtain the test statistics for structural change would yield misleading information in that the potentially harmful effects of collinearity would be hidden (S1nce var (T a) = var (T Tb) var (bl = (X 1 XI- l o 2 11 Further, based on the results discussed in section 4, it would appear that the use of spline functions to test for structural change has limited practical value. If the test statistic is not statistically significant, it would be difficult to ascertain whether this was because of collinearity or because there actually is no structural change (although a procedure developed by Belsley (20) might help in deciding whether the degrading collinearity is harmful). Even in the case where a statistically significant result is obtained, the magnitude of the structural change could be affected by collinearity. Employing a restricted model through the specification of end conditions appeared to give a large reduction in the degree of degrading collinearity (equivalent to that of a cubic polynomial). However, the lack of guidelines in specifying these restrictions also makes this approach unattractive. Finding a data-based strategy for selection of end-conditions (such as cross-validation) might make the spline approach more appealing. However, in the absence of such a strategy, other statistical techniques such as intervention analysis (21) would appear to be more appropriate methods for testing structural change. Splines also have been extensively used for numerical differentiation and integration. Since derivative and integral estimates are linear combinations of the regression estimates, investigating the effects of col linearity on these estimates would also seem warranted. 6. References 1. Brunelle, R. L. and Johnson, D. W., 19RO. Proceedings of the 5th Annual SUGT Conference pp Carmer, S. G., proceedings of the 5th Annual SUGI Conference pp Hwang, I., Proceedings of the 3rd Annual SUGI Conference pp 4. Kral, K., Capizzi, T. P., and Small, R. D., Proceedings of the 4th Annual SUGI Conference pp Mehta, H., Capizzi, T. P., Oppenheimer, L., Proceedings of the 6th Annual SUGI Conference pp 2ll-216. S. Fuller, W., Introduction to Statistical Time Series, John Wiley, New York. 7. Lenth, R. V., Commun. Statist (A.) 6, Smith, P. L., Amer. Statist, 33,

5 9. Ahlberg, J. H., Nelso~, E. N., and Walsh, J.', The Theory of Splines and their Applications, Academic press, New York. 10. Poirier, D. J., J. Amer. Statist. Assoc., 68, Poirier, D. J., The Econometrics of Structural Change, Amer. Elsevier Publishing Co. r New York. 12. de Boor, C A Practical Guide to Splines, Springer-Verlag, New York. 13. Wold,S., J. Econ. Theory, 4, Barze1, Y., J. Econ. Theory, 4, Marquardt, D. W., Technometrics, 12, Be1s1ey, D.A., Kuh, E. and Welsch, R. E., Regression Diagnostics: Identifying Influential Data and Sources of Collinearity. John Wiley & Sons, New York. 17. Searle, G., Linear Models. John Wi. ley, New York. 18. Sampson, P. D., J. Amer. Statist. Assoc., 74, Mosteller, F. and Tukey, J. w., Data Analysis and Regression. Addison-Wesley, Reading, Mass. 20. Belsley, D. A., MIT Technical Report ttr-i BOX, G. E. P. and Tiao, G., J. Amer. Statist. Assoc. 70, TABLE 1 COLLINEARITY DIAGNOSTICS OF CUBIC SPLINE REGRESSION MODEL WITH NO END CONDITIONS a) '+' function VIF b b l b 2 b 3 b 4 b S ,188 1,139,053 8,364,337 3,800, b) ANW-Splines Wo WI 1<2 1<3 1<4 W VIF t c ~, r f r, I ~ t ~ ; c) B-Splines VIF S S

6 ~-, TABLE 2 FITTED '+' FUNCTION SPLINES FOR INDY 500 WINNING SPEED VS. TIME DATA BY END CONDITIONS MODEL a) No End Conditions 1T4 = SPLINE S1t t t Pl P 2 (0.0163) (0.0007) Sit) t - O.OOOR (-0.19C+P ) (-3.44C+P ) 2 Sit) l.b8t (0.0003) (-1.63C+P 1 ) (0.0025) (0.0001) (0.89C+P2) (0.0006) where, PI ~ (t-7.s)3, P 2 ~ (t-33.5)3, C ~ (-42 t 2 + t 3 ), lsts a: Number in parenthesis is standard error for tests of structural change a) 1T 1 = 1T 4 = VIP TABLE 3 COLLINEARITY DIAGNOSTICS FOR RESTRICTED CUBIC SPLINE MODELS (Eqs. (4.2) and (4.3)) b b 1 b b s G<ll ) b) '1 ~ 2 '4 ~ VIP b o b 1 b b ?'? APPENDIX A COLLINEARITY ANALYSIS POR A CUBIC POLYNOMIAL FIT TO INDY 500 DATA (P(t) ~ ~O+~lt+~2t2+~3t3) VIP Est.:l..mates b b 1 b b