C2 Tranng: June 8 9, 2010 Introducton to meta-analyss The Campbell Collaboraton www.campbellcollaboraton.org Combnng effect szes across studes Compute effect szes wthn each study Create a set of ndependent effect szes Compute weghted mean and varance of effect szes Compute 95% confdence nterval for weghted mean effect sze Test for the homogenety of effect szes C2 Tranng Materals Oslo June 2010 www.campbellcollaboraton.org Create a set of ndependent effect szes Lely to have multple effect szes per study Effect szes wthn a study can be: Multple measures of study partcpants Measures of ndependent groups of partcpants C2 Tranng Materals Oslo June 2010 www.campbellcollaboraton.org 1
What to do wth multple effect szes per study? Use only ndependent groups n each analyss Use only one measure of the study partcpants n each analyss Use results wthn studes that are derved from ndependent groups of study partcpants Compute weghted mean and varance of effect szes Compute ndvdual study effect szes Correct effect szes for any bases, e.g., Hedges correcton Compute study effect sze varance 2 2 (75 1)5.74 + (65 1)4.16 sp = = 25.69 = 5.06 (75 + 65 2) 32.55 32.40 sm = = 0.03 5.06 2
p. 62 of MST pdf 2 75+ 65 (0.03) 1 sm 2 SE = + = 0.17, w = = 34.60 75*65 2(75+ 65) (0.17) 95% CI :(0.03 ± 1.96*0.17) = [ 0.30, 0.36] Use weghted mean for effect szes Use weghted means because each effect sze has a dfferent varance that depends on the study s sample sze and effect sze value Weght all analyses by the nverse of the effect sze varance or 1 w = (SE ) 2 Weghted Mean Effect Sze s the number of effect szes = w w 3
sm = 0.14 Standard error of weghted mean effect sze SE = 1 w Confdence Interval for Mean Effect Sze α = Sgnfcance level, z = Crtcal value from standard normal dstrbuton = - z (SE ) L (1-α) = + z (SE ) U (1-α) 4
SE L U = 0.10 = 0.14 1.96*0.10 = 0.06 = 0.14 + 1.96*0.10 = 0.34 Test of Homogenety Statstcal test that addresses whether the effect szes that are averaged nto a mean value all estmate the same populaton effect sze In a homogeneous dstrbuton, effect szes dffer from populaton mean only by samplng error Form of the homogenety test Q= w(-) 2 When we reject the null hypothess of homogenety, the varablty of the effect szes s more than would be expected from samplng error 5
Computatonal form Q = w - 2 w 2 w To Test Homogenety Compare Q to the (1 α) crtcal value of the ch-square dstrbuton wth -1 degrees of freedom Sgnfcant Q = heterogenety Non-sgnfcant Q = homogenety χ 2 (3) = 3.52, p= 0.32, ns.. 6
Prelmnary analyss Graph effect szes and 95% CI Compute overall mean effect sze and 95% CI Compute homogenety test Interpret fndngs p. 59 of MST pdf Moderator models: ANOVA When we fnd that a group of studes are heterogeneous, we can explore whether moderator varables explan ths varaton When we have contnuous moderators, we use regresson models When we have categorcal moderators, we use ANOVA 7
Example from Srn paper Srn, S. R. (2005). Socoeconomc status and academc achevement: A meta-analytc revew of research. Revew of Educatonal Research, 75, 417-453. 8
Group exercse Usng one of the studes provded from the MST revew, code the elements n Level 4. Choose at least 2 outcomes from the study. 9