Missing Data Techniques

Transcription

1 Missing Data Techniques Paul Philippe Pare Department of Sociology, UWO Centre for Population, Aging, and Health, UWO London Criminometrics ( 1

2 Introduction Missing data is a common problem in quantitative social research Multivariate analysis of large sample surveys: even a small proportion of missing data on many variables quickly adds up to a large number if cases aredeleted Analysis of small sample datasets (e.g. Clinical data, cross national data, quantification of qualitative data): every case counts! Analysis of variables involving sensitive topics 2

3 Introduction Main problems pobe with missing data If missing cases are deleted Reduced sample size and lower statistical power Biased estimates: sample selection If missing cases are treated (imputation) Biasedestimates: estimates: inadequate imputation Biased standard errors and sig. Tests: over fitted imputation Publication of results Journal editors and reviewers are increasingly strict about missing data problems and solutions Gone are the days when one could just add a footnote: missing data were deleted or replaced with the variables average 3

4 I am not: My background An inventor of new techniques to deal with missing data Publishing studies on the substantive topic of missing data I am: A reader (or consumer) of studies on missing data A user of missing data techniques in my own studies 4

5 Missing data theory: Paradise, Purgatory, and Hell Hll Paradise: Missing completely at Random (MCAR) The missing cases are unrelated to any variable in the analysis (including the variable with missing data itself) The pattern of missing cases is random in the common sense definition of random: roll of dices Example: A survey asked respondents about their age. 3% did not answer the question. No pattern is observed with ihother variables. ibl Why MCAR is paradise? Most missing data techniques will work well and provide unbiased and adequate estimates 5

6 Missing data theory: Paradise, Purgatory, and Hell Hll Purgatory: Missing at Random (MAR) The missing cases are associated with other variables in the analysis, but not with the variable with missing data itself. The pattern of missing cases is initially not random but this is taken into account by more sophisticated missing data techniques. The missing data are random once relevant correlates are included. Example: A survey asked respondents about their age. 2% of men and 6% of women did not answer the question. Gender, however, is used as a variable for the missing data technique and the analysis. Why MAR is purgatory? There is an original bias involving the missing data, but the problem can be solved with more sophisticated missing data techniques 6

7 Missing data theory: Paradise, Purgatory, and Hell Hll Hell: Non Random (NR) The missing cases are associated with the variable with missing data itself. The pattern of missing cases is not random and there is no real solution. More sophisticated missing data techniques provide some improvement but the bias is still there. The problem should be reported as a limitation of the study. Example: A survey asked respondents about their income. 10% of respondents did not answer the question. Respondents with particularly high or low income were less likely to answer the income question. Why NR is hell? Non random missing data create biases that cannot totally be solved. At best, more sophisticated missing data techniques will reduce the bias. 7

8 Missing Data as Substantive Dependent Variable 8

9 Missing Data as DV A first step in dealing with missing data is to understand the nature of the missing data pattern This can be done empirically: Cases with missing i values are coded dd1, cases without t missing values are coded 0 This variable is treated as a DV in a logistic regression or similar model, and other variables are used as predictors This analysis will reveal the characteristics of cases that are more likely to have missing values Similar to attrition analysis in longitudinal studies, but can also be done with non longitudinal data 9

10 Missing Data as DV Why is analyzing missing data as a DV useful? Provide the researcher with a substantive bt ti understanding of the missing data pattern Canhelp with selecting the best technique to address the missing data problem Canhelp with using the technique: creating weights, creating imputation data Depending on space, can be a part of your story: Missing data analysis as a section of a masters thesis, doctoral dissertation, or book. Appendix or major footnote in a journal article. 10

11 Missing Data Techniques: Traditional and Modern Approaches 11

12 Traditional Techniques: Listwise dl deletion Deleting any case with missing data Advantages: Simple (default option in many stat. ttp Programs) Acceptable with small number of MD (i.e. less than 5% for the full sample). Disadvantages: Quickly reduces sample size and statistical power when many variables have missing data Undetected dselection biases 12

13 Traditional Techniques: Pairwise dl deletion Using all available data between pair of variables to calculate covariance; only delete pairs with specific missing data (not the whole case is deleted) Advantages: Simple All available data is used Disadvantages: Mathematical problems: covariances have different sample sizes; different parts of the model have different degrees of freedom False advantage: given the mathematical problems, the point that All available data is used is more a source of bias than a source of validity 13

14 Traditional Techniques: Mean imputation Substituting thevariable s mean instead of missing data Advantages: Simple Full sample size is preserved Disadvantages: Theoretically illogical Sometimes overestimate coefficients, sometimes underestimate coefficients 14

15 Traditional Techniques: Hot deck imputation Substituting tut the answer from a randomly selected ected similar unit instead of missing data Advantages: Full sample size is preserved Logical Disadvantages: Over fitting: artificially increases statistical power by assuming that similar units are actually all identical. Lower SE and generous significance test. Justify the method for selecting similar units (how many variables?; how similar?, etc.) 15

16 Traditional Techniques: Single imputation Substituting model predicted values instead of missing data. Typically based on a multivariate model and the same variables used in the main analysis. Advantages: Logical Fullsamplesize size ispreserved Powerful yet simple method Disadvantage: Over fitting: artificially increases statistical power by assuming that predicted values are the same as observed values. Lower SE and generous significance test. 16

17 Traditional Techniques: Extra dummy variable Adding an extra dummy variable coded 1 for all the missing values and 0 otherwise to a series of dummy variables. Ex: Education: University graduate (ref); High School graduate (dummy); Less than high school graduate (dummy); Unknown education (dummy) Advantages: Full sample size is preserved The effect of the missing data dummy variable is empirically measured Disadvantages: Heterogeneity: the missing data dummy variable possibly combines very different cases together Requires many extra dummy variables if missing data are on multiple variables Requires original variables to be dummy coded 17

18 Traditional Techniques: Weighting Creating a statistical weight to compensate for the pattern of missing data (i.e. More weight to cases that are more likely to be missing) Advantages: Logical Not based on artificial a (imputed) ed)dataa Disadvantages: Decision must be made about how to create the weight iht Conflict with other recommended weights (e.g. Survey based sampling weight) Some scholars are skeptical of weighted analysis 18

19 Modern Techniques: Multiple l imputation ti Substituting model based predicted values, including random error or noise, instead ofmissing data. Repeat 5 10 times. Analyze datasets (Regression, SEM, etc.). Calculate average pooled coefficients and standard errors from these 5 10 datasets. Advantages: Same as single imputation: logical, full sample is preserved By including random error in the repeated imputations, imputed missing data are more noisy than observed data, thus overfitting is prevented Disadvantages: Requires more statistical work and knowledge; might require the use of different statistical programs Not necessarily an available optionfor all kindsof models (e.g. Propensity score matching within a probit Heckman selection model). 19

20 Modern Techniques: Maximum Likelihood Eti Estimation Using advanced mathematics and computer estimations (Maximum likelihood estimation, Expectation Maximization algorithms) to estimate model s coefficients and standard errors with missing data. Missing data are not imputed. Best fitting parameter estimates are selected via iterations that maximize the probability of observing the data that were collected. With FIML, parameters are directly estimated with the missing data With other ML methods, missing data are mathematically removed from the estimation, under MAR assumptions Advantages: Powerful and mathematically beautiful Not based on artificial (imputed) data Disadvantages: Requires more statistical work and knowledge; might require the use of different statistical programs Not necessarily an available option for all kinds of models. More common in recent SEM programs. 20

21 Other issues: General approaches vs. model specific dl Today s presentation focused on the general approaches to missing data Basic multivariate models: OLS regression; Logistic regression; ANOVA, etc. Literature on model specific techniques to deal with missing data Multilevel models Structural equation models Scales/factor analysis/principal component analysis Panel models/discrete dl/di time series/event history If using these models, it is recommended to also read the specific literature in addition to the general missing data literature. Some results/conclusions are different. 21

22 Other issues: Softwares General Statistical Programs: SPSS: Limited. Expectation Maximization imputation best function. Controversy: More likea sophisticated single imputation than a Maximum likelihood estimation. AMOS: can estimate models with missing data by Full Information Maximum Likelihood (extra cost) MI: Multiple imputation: new SPSS upgrade (extra cost) Stata: Multiple imputation: mi, ice, micombine SAS: Multipleimputation: imputation: MI, MIANALYZE R: AMELIA 2 Multiple Imputation Mplus: many Maximum likelihood estimation options are available HLM: some multiple imputation and Maximum Likelihood options Missing i data dt program: NORM: Multiple imputation 22

23 Conclusion Many techniques are available to deal with missing data Multiple imputation and maximum likelihood estimation are the most powerful techniques, but are also the most complicated Traditional techniques are sometimes adequate, but not always (MI and ML are popular for a reason) Some techniques should be avoided: mean imputation and pairwise deletion It is typically a good idea to analyze your missing data like a DV to better understand dtheir hipatterns If you plan on using more sophisticated missing data techniques, take time to understand the options and limitations of your statistical software. 23

24 General literature: References Allison, P. (2002) Missing data. Sage (one of Sage s green books on methodology) Johnson, D. & R. Young (2011) Toward best practices in analyzing datasets with missing data: Comparisons and recommendations. Journal of Marriage and Family 73: Acock, A. (2005) Working with missing values. Journal of Marriage and Family 67: Raghunathan, T. (2004) What to do with missing data? Some options for analysis of incomplete data. Annual lreview of Public Health h25: Graham, J. (2009) Missing data analysis: Making it work in the real world. Annual Review of Psychology 60: Model specific literature: Roth et al. (1999) Missing data in multiple item scales: Monte Carlo analysis of missing data techniques. Organizational Research Methods 2: Honaker, J. & G. King (2010) What to Do about Missing Values in Time Series Cross Section Data. American Journal of Political Science 54: