Predictive Analytics: Demystifying Current and Emerging Methodologies. Tom Kolde, FCAS, MAAA Linda Brobeck, FCAS, MAAA

Transcription

1 Predictive Analytics: Demystifying Current and Emerging Methodologies Tom Kolde, FCAS, MAAA Linda Brobeck, FCAS, MAAA May 18, 2017

2 About the Presenters Tom Kolde, FCAS, MAAA Consulting Actuary Chicago, Illinois Linda Brobeck, FCAS, MAAA Senior Consulting Actuary San Francisco, CA 1

3 Agenda Machine Learning Overview Deeper Dive GLM and GAM Decision Trees (Recursive Partitioning) Ensembles Networks Summary 2

4 Machine Learning Overview Machine Learning Supervised Predictive Target Variable Task Driven Regression, Classification Unsupervised Descriptive No Target Variable Data Driven Clustering, Pattern Discovery, Dimension Reduction Reinforcement Algorithm Learns to React 3

5 How do you define Machine Learning? A B C D Analytics with reinforcement Predictive modeling any technique - no reinforcement Predictive modeling beyond GLM without reinforcement Other Polling Question #1 4

6 Machine Learning Overview Machine Learning Regression Instance based Regularization Decision Trees Bayesian Dimensionality Reduction Association Rule Learning Clustering Artificial Neural Network Deep Learning 5

7 Machine Learning Overview Regression Ordinary Least Squares Linear, Logistic, Stepwise Multivariate Adaptive Locally Estimated Scatterplot Smoothed k-nearest Neighbor Learning Vector Quantization Self-Organizing Map Locally Weighted Learning Instance based Regularization Ridge Regression Least Absolute Shrinkage and Selection Operator (LASSO) Elastic Net Least-Angle Regression Iterative Dichotomizer 3 (ID3) Chi-squared Automatic Interaction Detection (CHAID) Decision Stump Conditional Decision Trees Decision Trees Naïve, Gaussian Naïve, Multinomial Naïve Bayesian Belief Network Bayesian Network Bayesian Principal Component Analysis PC Regression Partial Least Squares Sammon Mapping Dimensionality Reduction Multidimensional Scaling Projection Pursuit Linear, Mixture, Quadratic, Flexible Discriminant Analysis A priori algorithm Eclat algorithm Association Rule Learning Clustering k-means, k-medians Expectation Maximization Hierarchical Clustering Perceptron Back-Propagation Hopfield Network Radial Basis Function Network Artificial Neural Network Deep Learning Deep Boltzmann Machine Deep Belief Networks Convolutional Neural Network Stacked Auto-Encoders 6

8 Generalized Linear Modeling (GLM) Generalized Linear Model η = Xβ = β 0 + β 1 x 1 + β 2 x β p x p μ = g -1 (η) Goodness-of-fit Conceptually equivalent to sum of squares in ordinary linear regression Deviance: i 2wi ø y i y i θ µ i V(θ) dθ Key Assumptions Link function g() describes the functional relationship between the parameters and the fitted value Variance function V(µ i ) describes the relationship between the mean and variance for an observation Scale Parameter ø - accounts for misspecification of the variance function Weights w i - the amount of weight given to an individual record 7

9 Distributions 8

10 GLMs have become the primary tool Advantages Transparent results Allows modeler flexibility in setting assumptions beyond ordinary least squares Results are in a form that is readily implementable Allows for explicit adjustments for business considerations Methodology is applicable to pricing, underwriting, and other insurance functions Widely accepted Limitations Requires a greater number of assumptions than nonparametric methods Requires significant domain expertise and modeler judgment 9

11 Parametric vs Non-parametric Parametric The shape of the predictor function is defined by a few parameters o Algorithms simplify the function to a known form o Machine learning finds coefficients Nonparametric The shapes of predictor functions are fully determined by the data 10

12 Generalized Additive Models (GAM) Replaces estimation of linear form parameters with smooth linear or non-linear functions Generalized Additive Model η = β 0 + f 1 ( x 1 ) + f 2 (x 2 ) + + f p ( x p ) μ = g -1 (η) Goodness-of-fit Similar to GLMs Functions f i can be: Parametric with a specified form (i.e., a polynomial) Non-Parametric Each f i can be a different function 11

13 Generalized Additive Models (GAM) Advantages Greater flexibility, particularly when the relationship is not linear Estimate these smooth relationships simultaneously and then predict by simply adding them up GAM scales well with the increasing dimensionality and it yields interpretable models Exploratory modeling can uncover hidden patterns in the data and inspire parsimonious parametric models Limitations Subject to overfitting Requires specification of each f i Requires significant domain expertise and modeler judgment 12

14 Which modeling technique(s) does your company utilize? A B C D E F G Generalized Linear Models (GLM) Generalized Additive Models (GAM) Decision Trees Random Forests Other Ensemble Models Unsupervised Learning (Dimensionality Reduction and/or Clustering) Neural Networks Polling Question #2 13

15 Decision Trees - Methodology Split data according to measures of similarity If the Target Variable is: Categorical Classification Tree Continuous Regression Tree Two Competing Objectives: Purity Measure of Variation Parsimony Desire for Simple 14

16 Decision Trees The Process Splitting Procedure The domain space of explanatory variables X 1, X n is split into two subsets where observed values in X j belong to one of the subsets i.e. < s or >= s OR s 1 =male s 2 =female Improvement Value The dimensions j and s above are chosen to minimize the error in the prediction among all such binary (two-leveled) trees. Stopping Criteria No stopping criterion Minimum leaf (node) size; Maximum levels or splits Let data determine the stopping criterion 15

17 Decision Trees Measures for Splitting Criteria Significance Measures Independence Numeric purity p-values of Chi-square variance reduction Entropy Measures Disorder Categorical Measures pureness of the level Gain Ratio Measures Gain in Intrinsic Information Information Gain = Entropy (parent) Weighted sum of Entropy (children) Penalizes large values/splits Gini Measures Misclassification Max = 1 (1 / # of classes) Minimum = 0 (all records belong to one class) 16

18 Decision Trees Advantages/Limitations Advantages Non-parametric Simple to understand / Easy to interpret Automatic variable and interaction selection Handles missing values and outliers Limitations Over-fit concerns (unstable) Some relationships difficult to find 17

19 Decision Trees Finding Interactions Decision Tree automatically captures interactions Two explanatory variables interact if they combine nonadditively to affect the target Traditional regression requires an explicit interaction term identified upfront A Decision Tree of depth D can capture interactions of order up to D All Drivers Age <=30 Age > 30 Male Female Annual Mileage <10,000 Annual Mileage >10,000 Single Married 18

20 Applications of Decision Trees Enhancing GLMs Screening predictor variables Analyzing residuals Identifying transformations and/or interactions Portfolio diagnostics Checking/Quality Control 19

21 Ensembles Combine many weak classifiers in order to strengthen the overall result Bagging (Bootstrap Aggregating) Many models each based on sample Each model in the ensemble gets a vote Boosting Iterative models dependent on previous model Stacked Generalization (Blending) Use of diverse models in combination 20

22 Random Forests Each tree is built using a random sample With replacement of the observations Without replacement of the explanatory variables The predicted target value is the mean predicted target value over the ensemble Perturbation (interjecting randomness) is implemented, at each node, by only searching for optimal splits among a randomly chosen subset of the explanatory variables Random Sample 1 Random Sample 2 Random Sample 3 Random Sample 4 21

23 Boosting Gradient Boosting Trees built sequentially, tree based on previously built because the new tree is built on the residual of prior tree(s) Multiplicative Boosted Trees Multiplicative residuals Multiplicative combining of trees AdaBoost Adaptive boosting Iteratively changes weights of training observations based on errors of previous prediction 22

24 Stacked Generalization Blending of many model types Use diverse models for the blending/stacking 2 stages 1. Fit base learners to data 2. Fit a combiner algorithm to the predictions of the base learners Combiner Algorithm Logistic Regression Learner 1 Linear Regression Learner 2 Decision Tree Learner 3 Neural Network 23

25 Neural Networks Requires limited assumptions regarding the relationship of explanatory variables. Target layer regression model on a series of derived input, called hidden units Hidden units are regressions on the original inputs Regression parameters are adjusted iteratively to minimize the squared residuals Input layer Hidden layer Target layer X 1i 1 H 1i 1 X 2i 2 H 2i 2 Y i X 3i 3 H 3i 3 24

26 Neural Networks Advantages Assists discovery of local interactions Software packages reduce learning curve Limitations Subject to overfitting Identifying underlying drivers of model is difficult 25

27 What do you consider the significant obstacles to utilizing methodologies other than GLM? A B C D E Transparency of Results Unfamiliarity of methods Time constraints associated with running multiple methodologies Software limitations Other Polling Question #3 26

28 Summary Selecting/Combining Techniques Depends on the application/objective No silver bullet Machine Learning Reinforcement vs. Domain Expertise 27

29 Questions 28

30 Join Us for the Next APEX Webinar 29

31 Final notes We d like your feedback and suggestions Please complete our survey For copies of this APEX presentation Visit the Resource Knowledge Center at Pinnacleactuaries.com 30

32 Thank You for Your Time and Attention Tom Kolde Linda Brobeck Commitment Beyond Numbers 31