Types of missingness and common strategies

Transcription

1 9 th UK Stata Users Meeting 20 May 2003 Multiple imputation for missing data in life course studies Bianca De Stavola and Valerie McCormack (London School of Hygiene and Tropical Medicine) Motivating example Types of missingness and common strategies Multiple imputation A A suite of MI programs

2 Motivating example Breast cancer aetiology: Several established risk factors (adult HT, menarche) New focus on early life and childhood growth Breast cancer. Birth size Growth Growth Adult HT Menarche 2

3 MRC 1946 birth cohort (N=2187): repeated anthropometric measures in childhood Childhood height measured at: N % missing 2 yrs yrs yrs yrs yrs ALL

4 Pattern and type of missingness Data: Y=(y,X 1,, X p )= (Y obs, Y mis ) n y X 1 X 2 X p MCAR: Pr (missing) = not f(y obs, Y mis ) MAR: Pr (missing) = f (Y obs ) NMAR: Pr (missing) = f (Y obs, Y mis ) 4

5 Strategies 1. Analyse only those with complete data 2. Available case analysis 3. Inclusion of a missing value category 4. Use methods not requiring complete data 5. Replacing missing value with imputed 5

6 Strategies X 1. Analyse only those with complete data X 2. Available case analysis 3. Inclusion of a missing value category Biased even when data are MCAR (Greenland and Finkle 1995) confounder Level 1 Level 2 NK overall RR E

7 Strategies 1. Analyse only those with complete data X 2. Available case analysis X 3. Inclusion of a missing value category X 4. Use methods not requiring complete data 5. Replacing missing value with imputed 7

8 5 - Imputations If MAR: Idea: replace missing values with a guess Analysis: same as with complete data Two types, many variants: I. SINGLE IMPUTATION II. MULTIPLE IMPUTATION 8

9 I - SINGLE IMPUTATION a) from a regression model: eplace missing values ith predicted not good: true data variation Ht at age 15(cm)/Fitted values X Birth weight (gr) Ht at age 15(cm) X Fitted values impute.ado 9

10 SINGLE IMPUTATION b) predicted value + random term Size of random term depends on residual variance of the model NSATISFACTORY: Ht at age 15(cm)/Fitted values X Birth weight (gr) Ht at age 15(cm) X Fitted values till pretending the data are observed! 10

11 SINGLE IMPUTATION c) Hot-deck Replaces record with any missing values with another, but complete, selected at random Not recommended if several records are incomplete 11

12 II - MULTIPLE IMPUTATION Not one but several data sets are created Each has a different set of random draws to replace missing value Separate analyses on each data set Results summarised PROBLEM: generating a `proper predictive distribution 12

13 More technically.. MI replacements are simulated draws from a predictive distribution of the missing data: Y* mis ~P(Y mis Y obs, θ*) where θ * ~ P(θ Y obs ) Require a model for the complete data P(Y θ ) Proper, i.e. reflect uncertainty about missing data and the parameters (Shafer, Multiple imputation: a primer. Stat Methods in 13

14 MI: three steps A. Imputation of plausible values: Missing values replaced by imputed m times B. Analysis of the imputed datasets C. Combination of the results 14

15 B. Analysis Each dataset is analysed in the same way: e.g. : logistic regression Save : Point estimates of the statistics of interest: log(or) =Q (l) Their variance matrix: U (l) All stored for l=1,2,..,m 15

16 C. Combination Take the m sample estimates Q j and variance U j For one parameter: Overall estimator: Mean (Q j ) Its variance: Mean(U j ) + (1+1/m) Var(Q j ) For k parameters: Overall estimators: Mean (Q j ) Variance matrix : ( 1+ r 1 ) Mean (U j ) r 1 = (1+1/m) trace[ var(q j ) ( mean(u j ) -1 ] / k 16

17 ost difficult part A. Imputation ay x 1 has missing values. Approaches: i. Use draws from available observations of x 1 (unconditional draws) ii.use draws from regression models of x 1 (conditional draws) iii. [Hot-deck imputation] iv.[markov Chain Monte Carlo techniques] 17

18 i) unconditional draws x 1i ~N(µ, σ 2 ), i=1,,n only x 11 x 12. x 1a observed, for a<n : ( x 1,obs, S 2 obs ) For imputation run l =1,,m: 1. Draw σ 2 (l) from (a-1) S 2 obs / χ2 (a-1) 2. Draw µ (l) from N( x 1,obs, σ 2 (l) /a) Not good for MAR 3. Draw missing values from N(µ (l), σ 2 (l)) 4. New dataset l: observed x 11 x 12. x 1a plus imputed in step 3 should condition on: Factors affecting x 1 actors influencing missingness 18

19 ii) Simple conditional draws ssume x 1i ~ N( β 0 + β 1 x 2i, σ 2 ), 2i always observed, x 2 missing mechanism, X=[1 x2], For imputation run l =1,, m: 1. Draw σ 2 (l) from (a-2) S 2 obs / χ2 (a-2) ^ ^ 2. Draw (β 0 (l), β 1 (l)) from N(( β 0, β 1 ), σ 2 (l) (X X) -1 ) 3. Draw missing values from N(β 0 (l) + β 1 (l) x i, σ 2 (l))) 4. New dataset: observed plus imputed in step 3 19

20 Example RC 1946 birth cohort: 2187 women 1 : HT at 15 and breast cancer by age 53: 1689 I procedure: HT15= f(birth weight) Prob(missing)=f(birth weight, breast cancer) Height at age 15 (cm) Birth weight (gr) Ht at age 15(cm) Fitted values 20

21 MI programs: A. mi_create_reg.ado draws missing HT15 using results from regression of observed HT15 on (BW, BRCA) m times to create m imputed datasets B. mi_logit.ado runs logistic regression on each imputed dataset and saves the results C. mi_summary.ado summarises the results as in Shafer (1997) 21

22 Draws for ht at age 15 cond. on BW and BRCA: riginal estimates: ^ ^ =6.19, β 0 =149.79, ^ ^ 1 =2.60, β 2 =1.48 N=1683) (l) σ (l) β 0 (l) β 1 (l) β 2 (l) OBSERVED MI OR (95% CI) OR 95% CI (1.00,1.10) (1.00,1.09)

23 iii) conditional draws from a random effects model Using all childhood growth data y i (px1 vector): p Observed values: y i = Z η i + + ε i q Latent factors: η i = β X i + u i ε~ N(0, Σ ), u ~ N(0, Ψ), independence assumptions Explanatory variables: X i Loading factors (fcn of observation times): Z i.e. y i ~ N(β X i, Z i Ψ Z i +Σ) Imputation procedure in similar steps 23

24 For imputation run l =1,,m: 1. Draw Σ (l) from inverse Wishart based on Σ 2. Draw Ψ (l) from inverse Wishart based on Ψ 3. Draw η (l) from N(η pred, Z Ψ ( l) Z ) 4. Draw missing values from N(η (l), Σ (l)) ) 5. New dataset: observed plus imputed in step 3 ^ ^ MI program: A. mi_create_growth.ado 24

25 Logistic regression with imputed growth variables Use conditional draws, with several explanatory factors (including breast cancer) Observed data (N=904, D=33) Observed and imputed data (N=2187, D=59) HEIGHT Units OR 95%CI OR 95%CI Intercept at 2yrs cm , ,1.60 Velocity 2-4 yrs cm/yr , ,1.51 Velocity 4-7 yrs cm/yr , ,1.85 Velocity 7-11yrs cm/yr , ,1.62 Velocity 11-15ys cm/yr , ,1.70 Velocity 15- adulthood cm/yr , ,

26 Summary MI requires great care in creating imputed values A. mi_create_reg.ado & mi_create_growth.ado B. mi_logit.ado & mi_ologit.ado C. mi_summary.ado Other Stata programs: impute.ado regmsng.ado hotdeck.ado implogit.ado Gary Kings programs: clarify Ken Scheve s programs 26