A Survey of Regression Methods for Proxy Functions

Transcription

1 MARCH 2014 ENTERPRISE RISK SOLUTIONS B&H RESEARCH INSURANCE ERS Douglas McLean Ph.D David Redfern Ph.D Kate Pyper Moody's Analytics Research Contact Us Global A Survey of Regression Methods for Proxy Functions Overview Within the context of asset and liability proxy generation, this document discusses:» The features of a good proxy function.» Some relevant applications of proxy modeling.» Available linear and nonlinear regression techniques with their advantages and disadvantages. In addition, we comment on the availability of statistical power, bias and model misspecifation.

2 Table of Contents 1. Introduction 3 2. General Statistical Considerations 3 Statistical Power 3 Model Specification 3 The Bias-Variance Trade-off 3 3. Features of a Good Proxy Function Regression Model 4 4. Applications 5 5. The Current Approach - Multiple Polynomial Regression 5 6. Alternative Regression Methods 5 Mixed Effects in Multiple Polynomial Regression 6 Generalized Additive Models 6 Artificial Neural Networks 7 Regression Trees 7 Kernel Smoothing 7 Locally Weighted Scatterplot Smoothing 8 Finite Element Methods 8 7. Conclusion 8 2 MARCH 2014

3 1. Introduction The current implementation of the B&H Proxy Generator uses the method of multiple polynomial regression of the liability on a number of risk drivers that represent market and/or non-market risks. Recently, a number of limitations of the multiple polynomial regression method have been identified. This document aims to present a survey of proxy function regression methodologies that could be used in place of multiple polynomial regression. In this document, the features of a good proxy function regression model are clarified, and some of the key applications are identified. The advantages and disadvantages of the existing multiple polynomial regression models are identified. Then we examine a number of alternative regression methods and highlight their advantages and disadvantages. Proxy functions can be used to model liability values, asset values or any other financial or economic quantity. We will refer loosely to the liability as being the object we wish to model whether it is a liability, an asset or another type of quantity. 2. General Statistical Considerations Before discussing the features of a good proxy function, we define some of the general concepts that should be understood as part of a general statistical background. Statistical Power Statistical power refers to the overall amount of information contained in a set of data. It is important to consider this issue and what can and cannot be estimated from a set of fitting scenarios. Although it may be possible to estimate a proxy model by producing parameter estimates, there may be insufficient numbers of fitting scenarios to enable good estimates. Statistical power determines the amount of signal we can reasonably expect to resolve from a data set. Furthermore, the candidate terms with the most power tend to be the linear terms with the quadratics and cubics following on with successively less power. For example, in a univariate data set, a minimum of two points are needed to describe a straight line, three for a quadratic, four for a cubic etc., so the lower powered terms are most easily estimated. The quality of a candidate term s parameter estimate is seen in its standard error: a relatively large ratio of a standard error to a parameter s estimate most likely implies that this term does not contribute meaningfully in explaining the data. Were this term to be a truly meaningful predictor, it is likely that including more fitting scenarios would better resolve it. The effect of a lack of statistical power is most keenly felt when many candidate terms are contributing to the true data generation process in a nonlinear way. An insufficient number of scenarios leads to a failure to properly resolve the higher order terms, meaning that there is insufficient statistical power. For processes involving larger numbers of risk drivers, it will quickly become costly to add more fitting scenarios as these need to fill out a progressively larger multi-dimensional space. The desire to move to larger and more intricate models quickly becomes futile: our lens is insufficiently powerful to resolve all the detail we demand of it. Model Specification As described above, a solution to the issue of a lack of statistical power is to increase the number of fitting scenarios. Whilst indefinitely increasing the number of fitting scenarios increases statistical power, it does not take model misspecification into account. This occurs when the model predictions cannot arbitrarily match the process by which the data were generated in the first place, however many fitting scenarios are supplied. Insufficient power from the sample will only confound this issue. When considering reasons why a performance metric (such as R-squared) differs from its ideal value, it is necessary to consider model misspecification as well as sampling error. The Bias-Variance Trade-off Models of increasing complexity for a fixed scenario budget suffer from the bias-variance trade-off. If the data generation process is complex then one naturally seeks increasingly complex proxy forms. However, the issue of a lack of statistical power for more complicated processes plus a desire to fit more complicated models leads to poor predictive performance on out-of-sample data whilst the performance in-sample improves: we begin to fit to the noise in the sample. The estimator necessarily has a high variance for more complex models and can only be seen out-of-sample. Conversely, when a much smaller proxy form is used, the bias in the estimator is large due to model underfit. 3 MARCH 2014

4 3. Features of a Good Proxy Function Regression Model A good proxy function should exhibit the following features:» Parsimony It should use a minimally sufficient set of risk drivers (including powers and crossed terms) to achieve a fit that is measurably fit-for-purpose using any of the standard performance measures such as the adjusted R-squared measure, Mallows Cp or either of the Akaike or Bayesian Information Criteria.» Compatibility with downstream software Ease of communication with downstream software such as the B&H Economic Capital Calculator is also a big factor, which effectively means that it should use a relatively small number of function parameters in a succinct representation.» Good validation on accurate validation scenarios During the proxy fitting process, a small number of scenarios will have been evaluated accurately by a client s ALM system. These so-called validation scenarios should be sufficiently close, and lie within a given confidence interval around a proxy function s predictions.» High goodness-of-fit measure without overfitting Although good validation on accurate validation scenarios is necessary for a good proxy, it is not sufficient. This is because the number of validation scenarios is significantly smaller than the number of fitting scenarios. To measure goodness-of-fit:» the in-sample R-squared should be as high as possible; and» the out-of-sample R-squared (computed via resampling of the fitting scenarios) should be as close as possible to the in sample R-squared. The former condition establishes an upper bound on the goodness-of-fit, whilst the difference between the former and the latter determines the degree of overfitting» Unbiased predictions of minimum variance Any evidence of systematic under- or over-estimation in the model predictions given by the chosen estimator is evidence of bias. In addition, over the space of possible estimators, the estimator with the smallest variance in predictions should be used. This will often involve trading bias against variance in determining an optimal estimator. Resampling techniques applied to the fitting scenarios can achieve this. Since our ultimate objective is to use the proxy function for predictions, we need not concern ourselves with bias or variance in parameter estimates. In terms of predictive ability, more complex models will have a higher variance but a low bias and vice versa: this is the bias-variance-trade-off. Hitting the optimum middle ground is our primary goal.» Scalability to high dimensions For large numbers of risk drivers and fitting scenarios the memory requirements can become considerable and the time taken to summarise the information contained in so many scenarios can be prohibitive. When the data generation process is detailed, many more parameters are needed to accurately resolve the signal. When so many parameters are being estimated, however, their standard errors are large and our ability to recover a meaningful model is reduced. Adding more noisy inner scenarios may be costly.» Short model fitting time Users will potentially want to fit several, and perhaps hundreds of proxy models to their books of business. The time required to fit and then perform goodness-of-fit testing on all of the required proxy functions could become prohibitively large. Any speed advantage is clearly beneficial.» Good model specification Proxy models which are well specified will be able to approximate arbitrarily closely the underlying data generation process given enough fitting scenarios. Model misspecification, on the other hand, means that there would still be significant differences between the model fit and the generation process despite arbitrarily large numbers of fitting scenarios.» Efficient use of available data That is, statistical power of the data should be put to best use. Proxy models which are well specified will be able to approximate arbitrarily closely the underlying data generation process given enough fitting scenarios. Model misspecification, on the other hand, means that there would still be significant bias. 4 MARCH 2014

5 4. Applications To date, the most widely used application of the B&H Proxy Generator is to generate proxy functions for single time-step one year VaR calculations. This involves projecting the required risk drivers within the real- world ESG through one year and using the fitted proxy model to price those parts of the business (e.g. a swaptions interest rate hedge) that do not have a value at one year. Having used B&H Proxy Generator to price a book of business, the client may then determine their 99.5% VaR from their loss distribution at one year. Either time zero or time one stresses may also be applied when a client wishes to consider certain what if scenarios (e.g. 200 bps down on the yield curve in the swaptions hedge). The difficulty of the application largely depends on the type and features of the liability that is to be represented. Other applications can be understood as following variations on this theme. They are:» Multiple time-step models.» Calculation of liability sensitivities.» Calculation of liability sensitivities at multiple time-steps. 5. The Current Approach - Multiple Polynomial Regression This is the current regression method in the B&H Proxy Generator. The proxy function takes the form of a multivariate polynomial, and the fitting method is required to select which multivariate polynomial terms should be included in the model along with the magnitude/coefficient of each term.» In theory, can reproduce any functional form to arbitrary accuracy.» Regression coefficients have readily available formulae.» The Gauss-Markov theorem guarantees that if the noise in the fitting scenarios has a zero mean, is uncorrelated and is homoscedastic, then the regression model is the best linear unbiased estimator.» Produces model formulae with an intuitively simple appeal for less complicated data generation processes.» Readily produces standard errors for parameters and predictions when the residual noise is normal.» Communicates with and is easily implemented within the B&H Economic Capital Calculator.» In practice, statistical power limits the maximum degree of the polynomial terms that can be estimated and so some highly nonlinear functional forms, particularly those that are non-smooth, cannot be reproduced very accurately without using an unrealistically high number of fitting scenarios.» Model fitting time can be long.» Out-of-sample validation time can be long.» Subject to large model misspecification: model validation issues and prediction bias.» Intuition lost when the number of terms in the model formula is large.» Proxy function behaviour poor for nonlinear liabilities.» Cannot perform multiple time-step regressions unless time is taken as a risk driver.» Poor scaling of calculation time and complexity with increasing polynomial degree and/or number of risk drivers.» Error analysis relies on assumption of normality.» The functional form of sensitivities are limited to less than the degree of the polynomials used in the proxy function» Very easy to achieve overfitting when models of arbitrarily high complexity are force-fit. 6. Alternative Regression Methods In this section we survey a selection of the more popular regression models and point out their advantages and disadvantages with respect to the features discussed in Sections 3 and 4. The best method for a particular application will depend on how the advantages and disadvantages of the methods align with the specific requirements for that application. Often a trade-off will need to be made since some methods might perform very well in some areas of consideration but very poorly in other areas. Other methods may be more well-rounded, with no disadvantages that are significant for the particular application. 5 MARCH 2014

6 Mixed Effects in Multiple Polynomial Regression The current B&H Proxy Generator tool models risks as fixed effects. Simply put, the effect each specific risk has on a liability is measured as an absolute level by its regression coefficient. Uncertainty about this level is further quantified by an assessment of the standard errors. A random effect allows for the situation where groups of similar units behave in a correlated way. Rather than model each group separately, with a large increase in the number of regression coefficients, they are modeled as though they have come from a larger distribution. For example, in multi-time step modeling, the effect that autocorrelation has on individual Monte Carlo trials cannot reasonably be modeled by a set of fixed effects, but it can be modeled as a random effect. The additional random variable induces a temporal autocorrelation with the nuisance noise term already present and improves the fit.» In theory, can reproduce any functional form to arbitrary accuracy.» The Gauss-Markov theorem guarantees that if the noise in the fitting scenarios has a zero mean, is uncorrelated and is homoscedastic, then the regression model is the best linear unbiased estimator.» Produces model formulae with an intuitively simple appeal for less complicated data generation processes.» Readily produces standard errors for parameters and predictions when the residual noise is normal.» Communicates with and is easily implemented within the B&H Economic Capital Calculator.» Reduces total parameter count over multiple polynomial regression.» Handles multi-timestep modeling by inducing serial autocorrelation between steps.» Error analysis available for parameters and fits.» Requires the Expectation-Maximisation Algorithm or the Restricted Maximum Likelihood method to fit.» In practice, statistical power limits the maximum degree of the polynomial terms that can be estimated and so some highly nonlinear functional forms cannot be reproduced without using an unrealistic number of fitting scenarios.» Model fitting time can be long.» Out-of-sample validation time can be long.» Subject to large model misspecification: model validation issues and prediction bias.» Intuition lost for large number of terms in the model formula.» Proxy function behaviour poor for nonlinear liabilities.» Poor scaling of calculation time and complexity with increasing polynomial degree and/or number of risk drivers.» Error analysis relies on assumption of normality. Generalized Additive Models Additive models provide a way to extend the ideas of multiple polynomial regression to the use of other parametric (or even nonparametric) functions. For example, instead of using weighted sums of standard monomial functions of the risk drivers, sums of sigmoidal functions could be used. The exact choice of functions would depend on the nature of the data being modelled. For example, sigmoidal functions would be useful if the asymptotic behaviour of the liability were known to be bounded. Simple monomial functions become unbounded for large values of their risk drivers. For additive models, the liability is modeled as a linear sum of functions of the risk drivers. Generalized additive models subsume this category and allow the liability to be modeled as a nonlinear function of the linear sum of functions with the potential to improve proxy fit.» Allows a very large choice of basis functions including non-parametric univariate regression splines.» Produces a parsimonious, semi-parametric model representation.» Can be fitted using the existing techniques for multiple polynomial regression.» Choice of basis function can be difficult without prior assumptions of the functional form.» Regression spline basis functions do not apply to designed experiments like LSMC. 6 MARCH 2014

7 Artificial Neural Networks An artifical neural network (or simply a neural network ) is a computational structure that mimics the arrangement of neurons in the brain. For proxy generation, however, a neural network is simply used as a regression tool. The simplest neural network, the single hidden layer perceptron, is composed of three layers: an input layer, a hidden layer and an output layer. Each layer is composed of a (small) number of nodes which are joined (as in the brain) by directed line segments (synapses). In the neural network literature, its regression coefficients are called weights. Each node in the hidden layer combines its incident signal from the input layer by taking a weighted sum. Each node then processes its signal through a (sigmoidal) activation function and sends it to the output layer. Typically there is only one output node and this takes a weighted sum of its incident signals as the final output from the network. The fitting process estimates the weights by minimising the sum of squared differences between the neural network s output signal and the liability value. :» Model is parsimonious, with typically of the order of less than several tens of nodes defining the entire structure. It could easily be implemented into the ECC.» Since neural networks often only contain a small number of nodes, they are not memory intensive.» The nonlinear construct allows a good fit to a wide variety of function forms.» Sensitivities are robust and may be obtained by differentiating the functional formulae.» Well defined generalisation to multiple time-steps.» Includes multiple linear regression as a special case (via skip-layer modeling)» Is capable of representing more than one proxy function within the same neural network and with a single fitting process (by having more than one outer node).» Requires an iterative scheme to find coefficients.» Multiple redundancy in model parameters (although predictions are robust).» Small overhead in time to choose optimal number of nodes in the hidden layer.» Additional decay parameter may need tuning by cross validation.» Limited intuition for model parameters. Regression Trees As the name suggests, prediction from regression trees involves following a value from a root node to nodes on downstream layers. At each node, a decision is made based on the value of one of more of the regression variables as to which downstream node to progress to. The leaf nodes (i.e. nodes at the end of the tree with no downstream nodes) are associated with particular numerical values.» Non-parametric method allowing a good fit to a wide variety of liability surfaces.» Intuitive non-recombinant binomial tree structure.» Functional form may not be differentiated to obtain sensitivities.» Model fit time is very memory intensive and time consuming. Kernel Smoothing This is a non-parametric method that takes a weighted average of the values at each fitting point within a localized neighbourhood of the point in question. The weights are defined by a kernel function.» Allows a good fit to a wide variety of liability surfaces.» Functional form may be differentiated to obtain sensitivities.» Requires arbitrary choice of kernel functional form.» Requires arbitrary choice of what is meant by the localized neighbourhood.» Model is not parsimonious, in that all of the fitting scenarios are required for the proxy function to be fully specified.» Does not easily generalize to high dimensional problems. 7 MARCH 2014

8 Locally Weighted Scatterplot Smoothing Sometimes called LOESS or LOWESS, this is a semi-parametric (or non-parametric?) method in which a functional form is fitted to the values at points within a localized neighbourhood of the point in question. Weighting of the points can be defined by a kernel function.» Semi-parametric method allowing good fit to a wide variety of liability surfaces.» Functional form may be differentiated to obtain sensitivities.» Requires arbitrary choice of kernel functional form.» Requires arbitrary choice of what is meant by the localized neighbourhood.» Model is not parsimonious, in that all of the fitting scenarios are required for the proxy function to be fully specified.» Does not easily generalise to high dimensional problems.» Requires a look-up table and evaluation of a local proxy for evaluation.» Model fitting is highly computationally onerous in multiple-dimensions.» Does not generalise to multi-timesteps except through using time as a risk driver. Finite Element Methods The sample space is broken down into a mesh/grid (often irregular) and a particular simple functional form (i.e. linear or quadratic) is fitted to the points within each mesh element, maintaining consistency across element boundaries. In essence this is regression via piecewise functional forms.» Semi-parametric method allowing good fit to a wide variety of liability surfaces.» Functional form may be differentiated to obtain sensitivities.» Requires choice of element form.» Model is not parsimonious.» Model fitting is highly computationally onerous in multiple dimensions.» Does not easily generalize to high dimensional problems.» Requires a look-up table and evaluation of a local proxy for evaluation.» Does not generalise to multi-timesteps except through using time as a risk driver. 7. Conclusion The multiple polynomial representation that is being used in today's proxy function fitting methods have clear limitations, most notably around fitting time, high dimensional applications and their ability to fit to highly nonlinear liabilities. In this document, a number of methods for representing proxy functions have been presented and the advantages and disadvantages of each have been summarized. The best method for a particular application will depend on how the advantages and disadvantages of the methods align with the specific requirements for that application. 8 MARCH 2014

9 Copyright 2014, Moody s Analytics, Inc., and/or its licensors and affiliates (together, "MOODY'S). All rights reserved. ALL INFORMATION CONTAINED HEREIN IS PROTECTED BY COPYRIGHT LAW AND NONE OF SUCH INFORMATION MAY BE COPIED OR OTHERWISE REPRODUCED, REPACKAGED, FURTHER TRANSMITTED, TRANSFERRED, DISSEMINATED, REDISTRIBUTED OR RESOLD, OR STORED FOR SUBSEQUENT USE FOR ANY SUCH PURPOSE, IN WHOLE OR IN PART, IN ANY FORM OR MANNER OR BY ANY MEANS WHATSOEVER, BY ANY PERSON WITHOUT MOODY S PRIOR WRITTEN CONSENT. All information contained herein is obtained by MOODY S from sources believed by it to be accurate and reliable. Because of the possibility of human or mechanical error as well as other factors, however, such information is provided as is without warranty of any kind and MOODY S, in particular, makes no representation or warranty, express or implied, as to the accuracy, timeliness, completeness, merchantability or fitness for any particular purpose of any such information. Under no circumstances shall MOODY S have any liability to any person or entity for (a) any loss or damage in whole or in part caused by, resulting from, or relating to, any error (negligent or otherwise) or other circumstance or contingency within or outside the control of MOODY S or any of its directors, officers, employees or agents in connection with the procurement, collection, compilation, analysis, interpretation, communication, publication or delivery of any such information, or (b) any direct, indirect, special, consequential, compensatory or incidental damages whatsoever (including without limitation, lost profits), even if MOODY S is advised in advance of the possibility of such damages, resulting from the use of or inability to use, any such information. The credit ratings and financial reporting analysis observations, if any, constituting part of the information contained herein are, and must be construed solely as, statements of opinion and not statements of fact or recommendations to purchase, sell or hold any securities. NO WARRANTY, EXPRESS OR IMPLIED, AS TO THE ACCURACY, TIMELINESS, COMPLETENESS, MERCHANTABILITY OR FITNESS FOR ANY PARTICULAR PURPOSE OF ANY SUCH RATING OR OTHER OPINION OR INFORMATION IS GIVEN OR MADE BY MOODY S IN ANY FORM OR MANNER WHATSOEVER. Each rating or other opinion must be weighed solely as one factor in any investment decision made by or on behalf of any user of the information contained herein, and each such user must accordingly make its own study and evaluation of each security and of each issuer and guarantor of, and each provider of credit support for, each security that it may consider purchasing, holding or selling. MOODY S hereby discloses that most issuers of debt securities (including corporate and municipal bonds, debentures, notes and commercial paper) and preferred stock rated by MOODY S have, prior to assignment of any rating, agreed to pay to MOODY S for appraisal and rating services rendered by it fees ranging from $1,500 to approximately $2,400,000. Moody s Corporation (MCO) and its wholly-owned credit rating agency subsidiary, Moody s Investors Service (MIS), also maintain policies and procedures to address the independence of MIS s ratings and rating processes. Information regarding certain affiliations that may exist between directors of MCO and rated entities, and between entities who hold ratings from MIS and have also publicly reported to the SEC an ownership interest in MCO of more than 5%, is posted annually on Moody s website at under the heading Shareholder Relations Corporate Governance Director and Shareholder Affiliation Policy. Credit Monitor, CreditEdge, CreditEdge Plus, CreditMark, DealAnalyzer, EDFCalc, Private Firm Model, Portfolio Preprocessor, GCorr, the Moody s logo, the Moody s KMV logo, Moody s Financial Analyst, Moody s KMV LossCalc, Moody s KMV Portfolio Manager, Moody s Risk Advisor, Moody s KMV RiskCalc, RiskAnalyst, RiskFrontier, Expected Default Frequency, and EDF are trademarks or registered trademarks owned by MIS Quality Management Corp. and used under license by Moody s Analytics, Inc. 9 MARCH 2014