GLM II. Basic Modeling Strategy CAS Ratemaking and Product Management Seminar by Paul Bailey. March 10, 2015

Transcription

1 GLM II Basic Modeling Strategy 2015 CAS Ratemaking and Product Management Seminar by Paul Bailey March 10, 2015

2 Building predictive models is a multi-step process Set project goals and review background Gather and prepare data Explore Data Build Component Predictive Validate Component Combine Component Incorporate Constraints Ernesto walked us through the first 3 components We will now go through an example of the remaining steps: Building component predictive models We will illustrate how to build a frequency model Validating component models We will illustrate how to validate your component model We will also briefly discuss combining models and incorporating implementation constraints Goal should be to build best predictive models now and incorporate constraints later 1

3 Building component predictive models can be separated into two steps Set project goals and review background Gather and prepare data Explore Data Build Component Predictive Validate Component Combine Component Incorporate Constraints Initial Modeling Selecting error structure and link function Build simple initial model Testing basic modeling assumptions and methodology Iterative modeling Refining your initial models through a series of iterative steps complicating the model, then simplifying the model, then repeating 2

4 Initial modeling Initial modeling is done to test basic modeling methodology Is my link function appropriate? Is my error structure appropriate? Is my overall modeling methodology appropriate (e.g. do I need to cap losses? Exclude expense only claims? Model by peril?) 3

5 Examples of error structures Error functions reflect the variability of the underlying process and can be any distribution within the exponential family, for example: Gamma consistent with severity modeling; may want to try Inverse Gaussian Tweedie consistent with pure premium modeling Poisson consistent with frequency modeling Normal useful for a variety of applications 4

6 Generally accepted error structure and link functions Use generally accepted standards as starting point for link functions and error structures Observed Response Most Appropriate Link Function Most Appropriate Error Structure Variance Function Normal µ 0 Claim Frequency Log Poisson µ 1 Claim Severity Log Gamma µ 2 Claim Severity Log Inverse Gaussian µ 3 Pure Premium Log Gamma or Tweedie µ T Retention Rate Logit Binomial µ(1-µ) Conversion Rate Logit Binomial µ(1-µ) 5

7 Build an initial model Reasonable starting points for model structure Prior model Stepwise regression General insurance knowledge CART (Classification and Regression Trees) or similar algorithms 6

8 >.2 >.5 >.7 > 1.4 > 2.2 > 2.9 > 3.6 > 4.3 > 5.0 > 5.7 > 6.5 > 7.2 > 7.9 > 8.6 > 9.3 > 10.1 > 10.8 > 11.5 > 12.2 > 12.9 > 13.6 > 14.4 > 15.1 > 15.8 > 16.5 > 17.2 > 17.9 > 18.7 > 19.4 > 20.1 > 20.8 > 21.5 > 22.3 > 23.0 > 23.7 > 24.4 > 25.1 > 25.8 > 26.6 > 27.3 Test model assumptions Plot of all residuals tests selected error structure/link function Studentized Standardized Deviance Residuals , ,000 2,000 3,000 4,000 5,000 6,000 7,000 8,000 9,000 Fitte d Value Two concentrations suggests two perils: split or use joint modeling Normal Error Structure/Log Link (Studentized Standardized Deviance Residuals) Transformed Fitted Value Asymmetrical appearance suggests power of variance function is too low > 1 > 3 > 6 > 11 > 16 > 21 > 26 > 31 > 37 > 42 > 47 > 52 > 57 > 62 > 68 > 73 > 78 > 83 > Crunched Residuals (Group Size: 72) Elliptical pattern is ideal Fitted Value Use crunched residuals for frequency 7

9 Example: initial frequency model Link function: Log Gender Relativity Error structure: Poisson Initial variable selected based on industry knowledge: Gender Driver age Vehicle value Area (territory) Variable NOT in initial model: Vehicle body Vehicle age 8

10 Example: initial frequency model Link function: Log Driver Age Relativity Error structure: Poisson Initial variable selected based on industry knowledge: Gender Driver age Vehicle value Area (territory) Variable NOT in initial model: Vehicle body Vehicle age 9

11 Example: initial frequency model Link function: Log Vehicle Value Relativity Error structure: Poisson Initial variable selected based on industry knowledge: Gender Driver age Vehicle value Area (territory) Variable NOT in initial model: Vehicle body Vehicle age 10

12 Example: initial frequency model Link function: Log Area Relativity Error structure: Poisson Initial variable selected based on industry knowledge: Gender Driver age Vehicle value Area (territory) Variable NOT in initial model: Vehicle body Vehicle age 11

13 Example: initial frequency model - residuals Frequency residuals are hard to interpret without Crunching Two clusters: Data points with claims Data points without claims 12

14 Example: initial frequency model - residuals Order observations from smallest to largest predicted value Group residuals into 500 buckets The graph plots the average residual in the bucket Crunched residuals look good! 13

15 Building component predictive models can be separated into two steps Set project goals and review background Gather and prepare data Explore Data Build Component Predictive Validate Component Combine Component Incorporate Constraints Initial Modeling Selecting error structure and link function Build simple initial model Testing basic modeling assumptions and methodology Iterative modeling Refining your initial models through a series of iterative steps complicating the model, then simplifying the model, then repeating towerswatson.com 14

16 Iterative Modeling Initial models are refined using an iterative modeling approach Iterative modeling involves many decisions to complicate and simplify the models Review Model Your modeling toolbox can help you make these decisions We will discuss your tools shortly Simplify Exclude Group Curves Complicate Include Interactions 15

17 Ideal Model Structure To produce a sensible model that explains recent historical experience and is likely to be predictive of future experience Overall mean Best One parameter per observation Underfit: Predictive Poor explanatory power Model Complexity (number of parameters) Overfit: Poor predictive power Explains history 16

18 Your modeling tool box Model decisions include: Simplification: excluding variables, grouping levels, fitting curves Complication: including variables, adding interactions Your modeling toolbox will help you make these decisions Your tools include: Judgment (e.g., do the trends make sense?) Balance tests (i.e. actual vs. expected test) Parameters/standard errors Consistency of patterns over time or random data sets Type III statistical tests (e.g., chi-square tests, F-tests) 17

19 Modeling toolbox: judgment Modeled Frequency Relativity Vehicle Value The modeler should also ask, does this pattern make sense? Patterns may often be counterintuitive, but become reasonable after investigation Uses: Inclusion/exclusion Grouping Fitting curves Assessing interactions 18

20 Modeling toolbox: balance test Actual vs. Expected Frequency - Vehicle Age Balance test is essentially an actual vs. expected Can identify variables that are not in the model where the model is not in balance Indicates variable may be explaining something not in the model Uses: Inclusion 19

21 Modeling toolbox: parameters/standard errors Modeled Frequency Relativities With Standard Errors - Vehicle Body Parameters and standard errors provide confidence in the pattern exhibited by the data Uses: Horizontal line test for exclusion Plateaus for grouping A measure of credibility 20

22 Modeling toolbox: consistency of patterns Checking for consistency of patterns over time or across random parts of a data set is a good practical test Uses: Validating modeling decisions Including/excluding factors Grouping levels Fitting curves Adding Interactions Modeled Frequency Relativity Age Category 21

23 Modeling toolbox: type III tests Chi test and/or F-Test is a good statistical test to compare nested models H o : Two models are essentially the same H 1 : Two models are not the same Principle of parsimony: If two models are the same, choose the simpler model Uses: Inclusion/exclusion Chi-Square Percentage Meaning Action* <5% Reject H o Use More Complex Model 5%-15% Grey Area??? 15%-30% Grey Area??? >30% Accept H o Use Simpler Model 22

24 Example: frequency model iteration 1 simplification Modeling decision: Grouping Age Category and Area Tools Used: judgment, parameter estimates/std deviations, type III test Age Category Relativity Area Relativity Chi Sq P Val = 97.4% Chi Sq P Val = 99.9% 23

25 Example: frequency model iteration 1 simplification Modeling decision: fitting a curve to vehicle value Tools used: judgment, type III test, consistency test Vehicle Value Relativity Initial Model Vehicle Value Relativity Curve Fit Chi Sq P Val = 100.0% 24

26 Example: frequency model iteration 2 complication Modeling decision: adding vehicle body type Tools used: balance test, parameter estimates/std deviations, type III test Balance Test: Actual vs. Expected Across Vehicle Body Type Vehicle Body Type Not In Model Vehicle Body Type Relativities Vehicle Body Type Included in Model Chi Sq P Val = 1.3% 25

27 Example: iterative modeling continued. Iteration 3 - simplification Group vehicle body type Iteration 4 complication Add vehicle age Iteration 5 simplification group vehicle age levels 26

28 Example: frequency model iteration 6 complication Action: adding age x gender interaction Tools used: balance test, type III test, consistency test, judgment Balance Test: Two Way Actual vs. Expected Across Age x Gender Age x Gender Interaction NOT in model Vehicle Body Type Relativities Vehicle Body Type Included in Model M F Chi Sq P Val = 47.5% 27

29 Predictive models must be validated to have confidence in the predictive power of the models Set project goals and review background Gather and prepare data Explore Data Build Component Predictive Validate Component Combine Component Incorporate Constraints Model validation techniques include: Examining residuals Examining gains curves Examining hold out samples Changes in parameter estimates Actual vs. expected on hold out sample Component models and combined risk premium model should be validated 28

30 Model validation: residual analysis Recheck residuals to ensure appropriate shape 10 Studentized Standardized Deviance Residuals by Policyholder Age lt Crunched residuals are symmetric For Severity - Does the Box- Whisker show symmetry across levels? 29

31 >.2 >.5 >.7 > 1.4 > 2.2 > 2.9 > 3.6 > 4.3 > 5.0 > 5.7 > 6.5 > 7.2 > 7.9 > 8.6 > 9.3 > 10.1 > 10.8 > 11.5 > 12.2 > 12.9 > 13.6 > 14.4 > 15.1 > 15.8 > 16.5 > 17.2 > 17.9 > 18.7 > 19.4 > 20.1 > 20.8 > 21.5 > 22.3 > 23.0 > 23.7 > 24.4 > 25.1 > 25.8 > 26.6 > 27.3 Model validation: residual analysis (cont d) Common issues with residual plots Studentized Standardized Deviance Residuals , ,000 2,000 3,000 4,000 5,000 6,000 7,000 8,000 9,000 Fitte d Value Two concentrations suggests two perils: split or use joint modeling Normal Error Structure/Log Link (Studentized Standardized Deviance Residuals) Transformed Fitted Value Asymmetrical appearance suggests power of variance function is too low > 1 > 3 > 6 > 11 > 16 > 21 > 26 > 31 > 37 > 42 > 47 > 52 > 57 > 62 > 68 > 73 > 78 > 83 > Crunched Residuals (Group Size: 72) Elliptical pattern is ideal Fitted Value Use crunched residuals for frequency 30

32 Model validation: gains curves Gains curve are good for comparing predictiveness of models Order observations from largest to smallest predicted value on X axis Cumulative actual claim counts (or losses) on Y axis As you move from left to right, the better model should accumulate actual losses faster 31

33 Model validation: hold out samples Holdout samples are effective at validating models Determine estimates based on part of data set Uses estimates to predict other part of data set Full Test/Training for Large Data Sets Partial Test/Training for Smaller Data Sets Train Data Build All Data Build Data Split Data Data Train Data Refit Parameters Test Data Compare Predictions to Actual Split Data Test Data Compare Predictions to Actual Predictions should be close to actuals for heavily populated cells 32

34 Model validation: lift charts on hold out data Actual vs. expected on holdout data is an intuitive validation technique Good for communicating model performance to non-technical audiences Can also create actual vs. expected across predictor dimensions 33

35 Component frequency and severity models can be combined to create pure premium models Set project goals and review background Gather and prepare data Explore Data Build Component Predictive Validate Component Combine Component Incorporate Constraints Component models can be constructed in many different ways The standard model: COMPONENT MODELS Frequency Severity COMBINE Frequency x Severity Poisson/ Negative Binomial Gamma 34

36 Building a model on modeled pure premium When using modeled pure premiums, select the gamma/log link (not the Tweedie) 1,400 1,200 1, Density: Severity Severity Modeled pure premiums will not have a point mass at zero Density ,000 4,000 6,000 8,000 10,000 12,000 14,000 Range Density 2,400 2,200 2,000 1,800 1,600 1,400 1,200 1, Density: Pure Premium Pure Premium Raw pure premiums are bimodal (i.e., have a point mass at zero) and require a distribution such as the Tweedie ,000 4,000 6,000 8,000 10,000 12,000 14,000 Range 35

37 Various constraints often need to be applied to the modeled pure premiums Set project goals and review background Gather and prepare data Explore Data Build Component Predictive Validate Component Combine Component Incorporate Constraints Goal: Convert modeled pure premiums into indications after consideration of internal and external constraints Not always possible or desirable to charge the fully indicated rates in the short run Marketing decisions Regulatory constraints Systems constraints Need to adjust the indications for known constraints 36

38 Constraints to give desired subsidies Offsetting one predictor changes parameters of other correlated predictors to make up for the restrictions The stronger the exposure correlation, the more that can be made up through the other variable Consequently, the modeler should not refit models when a desired subsidy is incorporated into the rating plan Example Result of refitting with constraint Potential action Insurer-Desired Subsidy Sr. mgmt wants subsidy to attract drivers 65+ Regulatory Subsidy Regulatory constraint requires subsidy of drivers 65+ Correlated factors will adjust to partially make up for the difference. For example, territories with retirement communities will increase. Do not refit models with constraint Consider implication of refitting and make a business decision 37