The Consequences of Poor Data Quality on Model Accuracy

Transcription

1 The Consequences of Poor Data Quality on Model Accuracy Dr. Gerhard Svolba SAS Austria Cologne, June 14th, 2012

2 From this talk you can expect The analytical viewpoint on data quality Answers to the questions How much data do I need? How do missing values affect predictive power? Business case considerations for data quality

3 These are your options, if you learn that data quality is poor

4 Typical criteria for data quality for analytics Data Availability Actual data, historic data, historic snapshot of data Ensure periodic availability of data Level of granularity: aggregations or detail data Data Quantity Number of analysis subjects and events, length of observations period Data Completeness Random or systematic missign values, patterns Effort to get complete data Data Correctness Global and individual reference values Univariate and multivariate plausibility checks Statistical Features Correlation, variability, distributions

5 SAS helps to PROFILE data quality DataFlux / SAS Data Management Platform, Base SAS SAS Enterprise Miner, SAS STAT, SAS ETS SAS Forecast Server, Complex patterns of missing values Outliers detection based on multivariate methods Early detection of predictive power and variable importance JMP for interactive visual data quality control

6 SAS helps to IMPROVE data quality Imputation of missing values Calculation of individual replacement values Treat exceptional subgroups and time periods differently in the model Similarity measures for standardisation and record matching Methods for rare events Sample size planning

7 Consequences of poor data quality Cost Time Delays No Results Trust Risk of wrong decisions Insignificance

8 The consequences of the following effects have been studied How much data do I need? Varying the available number of observations and events (SIM_1) Gradually increasing the available length of data history (SIM_3) How do missing values affect predictive power? Random and systematic missing values (SIM_2) Other questions / simulations Withholding the set of the most important variables Introducing random and systematic bias in the input and target variables in predictive modeling Effect of random and systematic missing values and bias in time series forecasting

9 Real life data is used for the simulation studies Four real life datasets from different industries with a binary target variable were used How to end up with perfect data: Variable with > 5 % of missing values drop variable Variable with <= 5 % of missing values filter observations Large number of categories drop variable or group sparse categories to OTHERS Run multiple model cycles for each data source to retrieve a stable model with good predictive power Freeze the list of variables for the simulations

10 Simulation studies help to quantify the consequences of poor data quality Delete Observations Insert Missings Replace Missings Use frozen list of variables on scenario data Iterate

11 Studying the effect of data quantity in event prediction (SIM_1) Event prediction has many applications: credit scoring, churn, claim, response failure, fraud, Additional observations and events provide increase in % Correct Response rate But: Linear or non-linear effect How can this effect be quantified? Do also non-events contribute to an increase? Is it worth waiting for more events?

12 Findings of the data quantity scenarios Marginal benefits flattens out in the area of 500 to 1000 events Also non-events provide additional information especially in the area of up to 500 events Varying the number of events and non-events

13 Process for the missing value scenarios (SIM_2) For a specified proportion of observations (10%, ) Set interval and nominal input variables to missing» Random selection» Systematic selection based on segments Impute missing values with the IMPUTE node of SAS Enterprise Miner Train the model with frozen set of variables Perform treatment also for scoring data Assess model quality

14 Findings of the missing value scenarios Random missing values in training data only have limited effect. Missing values in the scoring data as well affect much more. Systematic missing values have a much larger effect. Things that matter: Not only the proportion of missing values but especially the type Missings in scoring data

15 Quantifying the results of the missing value scenarios Running a general linear model: Response = f(%missing, Systematic_YN, ScoringData_YN) Parameter Value Interpretation Intercept Response with no missing values %missing % missing ~ 1% less response Systematic_YN Systematic error causes 3.6 % less response Scoring_YN Missings in scoring data cause 4.23 % less response

16 Gradually increasing the available length of data history in time series forecasting Business Questions Is it possible to start time series analysis if only 18 months of history are available? Do we have to wait for an additional history month? What is the benefit of additional data management effort? What is the best length of data history for time series forecasting? Methods Simulation environment built with SAS High Performance Forecasting Restricted to forecasting method exponential smoothing Minimum history for each time series: 48 months Validation based on MAPE calculated on 12 lead months 788 time series on monthly data from different industries

17 How far should we remember back? The expected decrease in MAPE with increasing history length can be seen. There is an exponential decrease in the additional value of additional months Larger steps after 12, 24 and 36 months can be seen.

18 What is the best length of data history for time series forecasting? Method: for each time series query how many history months give the smallest error for the future 12 months Results: Not in all cases it is beneficial to have a long data history.

19 Final takeaways Data Quality for Analytics is more More requirements More possibilities Get into details! Random or systematic bias? Permanent or historic/temporary problem? Quantity matters! But balance effort and benefit! SAS helps to Profile, Improve, Assess, Simulate

20 Data Quality for Analytics Using SAS SAS Press, April 2012 Dr. Gerhard Svolba LinkedIn Analytics has additional requirements on data quality Analytics contributes methods for better data quality Simulation studies show the consequences of poor data quality on model quality