Small Sample Robust Fit Criteria in Latent Growth Models with Incomplete Data. Dan McNeish & Jeff Harring University of Maryland

Transcription

1 Small Sample Robust Fit Criteria in Latent Growth Models with Incomplete Data Dan McNeish & Jeff Harring University of Maryland

2 Growth Models With Small Samples An expanding literature has addressed the issue of small samples with mixed effects models (MEMs) Main issue is biased estimates and underestimated standard errors REML estimation and the Kenward-Roger correction have been shown to provide unbiased estimates for sample sizes as low as 6 with MEMs 2

3 Growth Models With Small Samples II Despite the popularity of LGMs in behavioral sciences along with the high prevalence of small samples, very little research on small sample LGMs exists No REML estimation No Kenward-Roger correction Often uses FIML which has well-known finite sample bias Little research on fit indices Little research on missing data 3

4 4

5 5

6 Model Fit with Small Samples The ability to obtain global model fit information is one advantage of modeling in the LGM framework compared to the MEM framework With smaller samples though, the minimum-fit function chi-square test is known to be artificially inflated Well fitting models will be rejected much too often The various indices that are based upon this χ 2 test will also be adversely affected 3 approaches to accommodate small samples have been advanced 6

7 Bartlett Correction T B 1 (2v4 f 5) 6( n 1) T ML V = number of observed variables f = number of factors n = total sample size T ML = minimum fit function chi-square statistic 7

8 Yuan Correction T Y 1 (2v2 f 7) 6( n 1) Very similar to Bartlett correction but applies a less harsh penalty V = number of observed variables f = number of factors n = total sample size T ML = minimum fit function chi-square statistic T ML 8

9 Swain Correction T S v(2v 3v 1) q(2q 3q 1) (12 n) df T ML 1 4 v( v 1) 8df 1) q 2 V = number of observed variables n = total sample size T ML = minimum fit function chi-square statistic 9

10 T B 1 (2v4 f 5) 6( n 1) T ML T Y 1 (2v2 f 7) 6( n 1) T ML T S v(2v 3v 1) q(2q 3q 1) (12 n) df T ML 10

11 Issue with Small Sample Corrections With Missing Data Each of the 3 small sample corrections include sample size somewhere in the correction formula With FIML, what sample size should be in not always straightforward FIML makes full use of all observed values, but does not impute values 100 people may not give you 100 people s worth of information 11

12 Issue with Small Sample Corrections With Missing Data II The information contained in a complete observation with FIML is not equivalent to the information in an observation with missing data Weighing all observations equally assumes that there is more information than is actually present This will lead to under-corrections of the small sample formula 12

13 Analogous Situation with Multiple Imputation Inferential tests from analysis with multiple imputation encounter similar difficulties with appropriate degrees of freedom Degrees of freedom based on total sample size is not appropriate because many values are imputed Assumes that more information is present than there is in actuality Rubin & Schenker (1986) address this by calculating df as dfrs 1 FMI ( m 1) 2 13

14 Proposed Missing Data Scaling Factor FIML and MI don t not overlap exactly, but Rubin-Schenker degrees of freedom will serve as a basis for our proposed method Goal is to come up with a multiplicative factor M that can be applied to the total sample in the Bartlett, Swain, and Yuan corrections such that n M ( n Total ) As a result, the small sample corrections will no longer undercorrect dfrs 1 FMI ( m 1) 2 14

15 Specifically, Proposed Missing Data Scaling Factor II (1) as missingess 0, M 1 (2) and 0 < M 1 Leaving FMI in the denominator violates (1), so we will move it into the numerator Also, FMI is often considered a most relevant in the context of MI, so we will use the closely related (but not identical) and more easily calculable percentage of complete values for M dfrs 1 FMI ( m 1) 2 15

16 Proposed Missing Data Scaling Factor III As a result, we will multiply the total sample size by the square of the percentage complete observations n 2 (% Complete Observations) ntotal For example, if a data matrix has 4 measurements on 25 people and 20 of the measurements were missing (regardless of how they were distributed), 2 n (80 / 100) Preserves the asymptotic properties because as n becomes very large and/or as missing data goes to 0, the missing data factor has less and less influence dfrs 1 FMI ( m 1) 2 16

17 Bartlett Correction with Missing Data Scaling factor T BM (2v4 f 5) 1 6[ M ( n ) 1] Total T ML Where M (% CompleteObservations) 2 T B 1 (2v4 f 5) T 6( n 1) ML 17

18 Bartlett Correction with Missing Data Scaling factor T Where BM M 1 (2v4 f 5) 6[ M( n ) 1] Total T ML Seems painfully simple, but, as shown shortly, it makes a very big difference (% CompleteObservations ) 2 T B 1 (2v4 f 5) 6( n 1) T ML 18

19 4 sample size conditions (20,30,50,100) Simulation Conditions 2 repeated measures conditions (4,8) 3 % missing data conditions (0%, 10%, 20%) 2 missing data pattern conditions (arbitrary, monotone) 2 model conditions (Linear, latent basis) 19

20 Results- No Missing Data T ML can clearly be seen to have inflated rejection rates All three corrections return the rejection rates very close to the nominal rate when no data are missing Values for n is not problematic here 20

21 Results- 10% Missing Data With even a small amount of missing data, the n-based corrections no longer perform very well Using the missing data scaling factor in the corrections is able to yield reject rates much closer to the nominal rate, particularly with Bartlett 21

22 Results- 20% Missing Data As percentage of missing data grows, using n in the sample size correction formula doesn t correct enough Assumes more information is present 22

23 RMSEA Results- 20% Missing Data Inflated T ML values also affect fit indices Missing data corrections can similarly be applied to account for missing data with small samples Most salient at smallest sample sizes 23

24 Discussion The standard ML chi-square and standard fit indices are inflated with small samples, leading to over-rejection of models that truly fit well Bartlett, Swain, and Yuan corrections perform quite well when the no data are missing Has been previously found (e.g., Herzog & Boomsma 2009; Nevitt & Hancock, 2004) However, as the percentage of missing data increases, using FIML, the appropriate value for n in these corrections becomes unclear 24

25 Discussion II The focus of this study was solely on obtaining appropriate rejection rates, additional work would be needed to address the power of the proposed method Normality can also be problematic with small samples. The corrections mentioned here have been applied to the Satorra- Bentler chi square and a missing data scaling factor may be useful in this context as well. 25

26 Thank You! For questions, comments, or a copy of the full paper, dmcneish@umd.edu harring@umd.edu 26