Ensemble Learning and the Heritage Health Prize

Transcription

1 and the Heritage Health Prize Jonathan Stroud, Igii Enverga, Tiffany Silverstein, Brian Song, and Taylor Rogers icamp 2012 University of California, Irvine Advisors: Max Welling, Alexander Ihler, Sungjin Ahn, and Qiang Liu August 14, 2012

2 The Heritage Health Prize Goal: Identify patients who will be admitted to a hospital within the next year, using historical claims data.[1] 1,250 teams

3 Purpose Reduce cost of unnecessary hospital admissions per year Identify at-risk patients earlier

4 Kaggle Public online competitions Gives feedback on prediction models

5 Data Provided through Kaggle Three years of patient data Two years include days spent in hospital (training set)

6 Evaluation Root Mean Squared Logarithmic Error (RMSLE) ε = 1 n n [log(p i + 1) log(a i + 1)] 2 i Threshold: ε.4

7 The Netflix Prize $1 Million prize Leading teams combined predictors to pass threshold

8 Blending Blend several predictors to create a more accurate predictor

9 Prediction Models Optimized Constant Value K-Nearest Neighbors Logistic Regression Support Vector Regression Random Forests Gradient Boosting Machines Neural Networks

10 Feature Selection Used Market Makers method [2] Reduced each patient to vector of 139 features

11 Optimized Constant Value Predicts same number of days for each patient Best constant prediction is p = RMSLE: (800th place)

12 K-Nearest Neighbors Weighted average of closest neighbors Very slow

13 Eigenvalue Decomposition Reduces number of features for each patient X k = λ 1/2 k Uk T X c

14 K-Nearest Neighbors Results Neighbors: k = 1000 RMSLE: (600th place)

15 Logistic Regression RMSLE: (375th place)

16 Support Vector Regression ε =.02 RMSLE: (400th place)

17 Decision Trees

18 Random Forests RMSLE: (315th place)

19 Gradient Boosting Machines Trees = 8000 Shrinkage = Depth = 7 Minimum Observations = 100 RMSLE: (200th place)

20 Artificial Neural Networks

21 Back Propagation in Neural Networking

22 Neural Networking Results Number of hidden neurons = 7 Number of cycles = 3000 RMSLE: (340th place)

23 Individual Predictors (Summary) Optimized Constant Value K-Nearest Neighbors Logistic Regression Support Vector Regression Random Forests Gradient Boosting Machines Neural Networks (800th place) (600th place) (375th place) (400th place) (315th place) (200th place) (340th place)

24 Individual Predictors (Summary)

25 Deriving the Blending Algorithm Error (RMSE) ε = 1 n (X i Y i ) n 2 nε 2 c = i=1 n (X i Y i ) 2 i=1 nε 2 0 = n i=1 Y 2 i

26 Deriving the Blending Algorithm (Continued) X as a combination of predictors X = Xw or X i = c w c X ic

27 Deriving the Blending Algorithm (Continued) Minimizing the cost function C = 1 n N (Y i X i ) 2 i=1 C w = i (Y i c w c X ic )( X ic ) = 0

28 Deriving the Blending Algorithm (Continued) Minimizing the cost function (continued) Y i X ic = i i w c X ic X ic Y T X = w T c X T c X c

29 Deriving the Blending Algorithm (Continued) Optimizing predictors weights w c = (Y T X )(X T X ) 1 Y i X ic = Xic 2 + i i i Y i X ic = i i Y 2 ic i X 2 ic + nε 2 0 nε 2 c (Y i X ic ) 2

30 Deriving the Blending Algorithm (Continued) Error (RMSE) ε = 1 n (X i Y i ) n 2 nε 2 c = i=1 n (X i Y i ) 2 i=1 nε 2 0 = n i=1 Y 2 i

31 Deriving the Blending Algorithm (Continued) Optimizing predictors weights w c = (Y T X )(X T X ) 1 Y i X ic = Xic 2 + i i i Y i X ic = i i Y 2 ic i X 2 ic + nε 2 0 nε 2 c (Y i X ic ) 2

32 Deriving the Blending Algorithm (Continued) X as a combination of predictors X = Xw or X i = c w c X ic

33 Blending Algorithm (Summary) 1. Submit and record all predictions X and errors ε 2. Calculate M = (X T X ) 1 and v c = (X T Y ) c = 1 i 2 (X ic 2 + nε2 0 nε2 c) 3. Because w c = (Y T X )(X T X ) 1, calculate weights w = Mv 4. Final blended prediction is X i = Xw

34 Blending Results RMSLE: (98th place)

35 Future Work Optimizing Blending Equation with Regularization Constant w c = (Y T X )(X T X + λi ) 1 Improved feature selection More predictors

36 Questions

37 References Heritage provider network health prize, David Vogel Phil Brierley and Randy Axelrod. Market makers - milestone 1 description. September 2011.