Constrained Optimal Sample Allocation in Multilevel Randomized Experiments Using PowerUpR

Transcription

1 Constrained Optimal Sample Allocation in Multilevel Randomized Experiments Using PowerUpR Metin Bulus & Nianbo Dong University of Missouri March 3, 2017 Contents Introduction PowerUpR Package COSA Functions in PowerUpR Installation and Usage Constraints in COSA Example use of COSA Functions in PowerUpR Statistical Model Parameter Values Scenario 1: Constrain Power Scenario 2: Constrain MDES and J Scenario 3: Constrain Cost and J Scenario 4: Constrain Cost, n and J Plots [i] Plots [ii] Plots [iii] Plots [iv] Online GUI for PowerUpR 95% CI for MDES and Percentiles Summary Future Directions Thank you! References Note. To access an HTML version of this document please use the following link 1

2 Introduction There are several commercial and publicly available tools to assist designing multilevel randomized experiments such as CRT-Power (Borenstein, Hedges, & Rothstein, 2012), PowerUp! (Dong & Maynard, 2013), and Optimal Design (Raudenbush et al., 2011). Hedges and Borenstein (2014) has introduced the idea of constrained optimal sample allocations (COSA) considering limitations with the sample sizes in CRT-power. However, there isn't any statistical tool to assist researchers in this endeavor in R environment (R Core Team, 2017), which is free and increasingly used. 2

3 PowerUpR Package So we translated PowerUp! into R and added COSA functions. CRAN page includes all the information about the package, and is available at 3

4 COSA Functions in PowerUpR COSA functions begin with optimal, followed by a period, and a design name. There are 13 types of designs where COSA is applicable: bira2r1, bira2f1, bira2c1, cra2r2, bira3r1, bcra3r2, bcra3f2, cra3r3, bira4r1, bcra4r2, bcra4r3, bcra4f3, and cra4r4. The first three or four letters of the design stands for the type of assignment, for blocked/multisite individual random assignment bira, for cluster random assignment cra, and for blocked/multisite cluster random assignment bcra. It is followed by a number indicating number of levels. A single letter followed by a number indicates whether a block is considered to be r, random; f, fixed; orc, constant and the level of randomization. e.g., optimal.cra3r3 4

5 Installation and Usage 1. Download and install R from 2. Download and install RStudio from (optional) Open R or RStudio, then type the following code to install the package install.packages("powerupr") Load the package into the current session library(powerupr) Get help for a specific function?optimal.cra3r3 Copy and paste the code from Examples section in help file, and modify it to your own case (Important: Note default values in Usage section!) optimal.cra3r3(cn=5, cj=50, ck=200, constrain="cost", cost=100000, rho3=.06, rho2=.17) 5

6 Constraints in COSA PowerUpR package provides functions to deal with COSA problems under the following conditions under budgetary constraints given marginal costs per unit while minimizing sampling variance of the treatment effect, under power constraints given marginal costs per unit while minimizing total cost, under MDES constraints given marginal costs per unit while minimizing total cost, and under sample size constraints for one or more levels along with any of the options above. 6

7 Example use of COSA Functions in PowerUpR Institue of Education Science (IES) reported that evidence on effectiveness of Saxon Math curriclum on secondary school student's algebra subject is lacking and further research is needed (U.S. Department of Education, 2016). To evaluate the effectiveness Saxon Math curriculum, assume a three-level cluster randomized trial is considered where schools are randomly assigned to treatment (Saxon Math curriclum) and a control condition. In the grant proposal number of schools, classrooms, and students, and the cost associated with data collection from this sample needs to be justified. We will use PowerUpR package (Bulus & Dong, 2017) an implementation of PowerUp! (Dong & Maynard, 2013) software in R environment (R Core Team, 2017). 7

8 Statistical Model Due to nesting of students within classrooms and nesting of classrooms within schools, the treatment effect (δ) can be estimated via the following 3-level hierarchical linear model (HLM, Raudenbush & Bryk, 2002) where, Y: Student algebra posttest score. X: Student algebra pretest score. W: Classroom mean algebra pretest score. V: School mean algebra pretest score. T: Treatment status. 8

9 Parameter Values Consider the following hypothetical values for design parameters Proportion of variance in algebra posttest at the school level (τ3 2 ):.15 Proportion of variance in algebra posttest at the classroom level (τ2 2 ):.05 Proportion of variance in algebra posttest at the student level (σ 2 ):.65 Then, intraclass correlation coefficients are ρ2 =.05/( )=0.059 ρ3 =.15/( )=0.176 Proportion of variance in algebra posttest explained by school mean algebra pretest: (R3 2 ):.45 Proportion of variance in algebra posttest explained by classroom mean algebra pretest: (R2 2 ):.50 Proportion of variance in algebra posttest explained by student algebra pretest: (R1 2 ):.55 9

10 Scenario 1: Constrain Power Further assumptions Alpha level (α): 0.05 Statistical power (1 β):.80 Minimum detectable effect size (MDES):.232 Marginal cost per student (cn): 10 Marginal cost per classroom (cj): 200 Marginal cost per school (ck): 500 Given design parameters and further assumptions, how many schools, classrooms and students should be recruited to achieve 80% power while minimizing the total cost? library(powerupr) design1 <- optimal.cra3r3(cn=10, cj=200, ck=500, constrain="power", power=.80, mdes=.232, rho2=.06, rho3=.18, g3=1, R12=.55, R22=.50, R32=.45, P=.40) print(design1$round.optim) ## n J K cost mdes power ## [1,] print(design1$integer.optim) ## n J K cost mdes power ## [1,] ## [2,] ## [3,] ## [4,] ## [5,] ## [6,] ## [7,] ## [8,] ## [9,] ## [10,]

11 Scenario 2: Constrain MDES and J Further assumptions Alpha level (α): 0.05 Statistical power (1 β):.80 Minimum detectable effect size (MDES):.232 Average number of classrooms per school (J): 2 Marginal cost per student (cn): 10 Marginal cost per classroom (cj): 200 Marginal cost per school (ck): 500 Given design parameters and further assumptions, how many schools, and students should be recruited to achieve an MDES of.232 while minimizing the total cost? design2 <- optimal.cra3r3(cn=10, cj=200, ck=500, constrain="mdes", mdes=.232, J=2, power=.80, rho2=.06, rho3=.18, g3=1, R12=.55, R22=.50, R32=.45, P=.40) print(design2$round.optim) ## n J K cost mdes power ## [1,] print(design2$integer.optim) ## n J K cost mdes power ## [1,] ## [2,] ## [3,] ## [4,] ## [5,] ## [6,] ## [7,] ## [8,] ## [9,] ## [10,]

12 Scenario 3: Constrain Cost and J Further assumptions Alpha level (α): 0.05 Statistical power (1 β):.80 Minimum detectable effect size (MDES) Average number of classrooms per school (J): 2 Marginal cost per student (cn): 10 Marginal cost per classroom (cj): 200 Marginal cost per school (ck): 500 Budget: 130, 000 Given design parameters and further assumptions, how many schools, and students should be recruited considering a budget of $130,000 while minimizing the sampling variance of the treatment effect? design3 <- optimal.cra3r3(cn=10, cj=200, ck=500, constrain="cost", cost=130000, J=2, power=.80, mdes=.20, rho2=.06, rho3=.18, g3=1, R12=.55, R22=.50, R32=.45, P=.40) print(design3$round.optim) ## n J K cost mdes power ## [1,] print(design3$integer.optim) ## n J K cost mdes power ## [1,] ## [2,] ## [3,] ## [4,] ## [5,] ## [6,] ## [7,] ## [8,] ## [9,] ## [10,]

13 Scenario 4: Constrain Cost, n and J Further assumptions Alpha level (α): 0.05 Statistical power (1 β):.80 Minimum detectable effect size (MDES):.20 Average number of students per classroom (n): 20 Average number of classrooms per school (J): 2 Marginal cost per student (cn): 10 Marginal cost per classroom (cj): 200 Marginal cost per school (ck): 500 Budget: 130, 000 Given design parameters and further assumptions, how many schools should be recruited considering a budget of $130,000 while minimizing the sampling variance of the treatment effect? design4 <- optimal.cra3r3(cn=10, cj=200, ck=500, constrain="cost", cost=130000, n=20, J=2, power=.80, mdes=.20, rho2=.06, rho3=.18, g3=1, R12=.55, R22=.50, R32=.45, P=.40) print(design4$round.optim) ## n J K cost mdes power ## [1,] print(design4$integer.optim) ## n J K cost mdes power ## [1,] ## [2,] ## [3,] ## [4,] ## [5,] ## [6,] ## [7,] ## [8,] ## [9,] ## [10,]

14 Plots [i] ## MDES - K ## Red point indicates the design location plot(design4, pars=c("mdes","k")) 14

15 Plots [ii] ## Power - K ## Red point indicates the design location plot(design4, pars=c("power","k")) 15

16 Plots [iii] ## Power - MDES - K ## Red point indicates the design location plot(design4, pars=c("power","mdes", "K")) 16

17 Plots [iv] ## Power - MDES - K ## Red point indicates the design location plot(design4, pars=c("power","mdes", "K"), type="c") 17

18 Online GUI for PowerUpR Online version is accesible through: Replicate scenarios 1-4 in PowerUpR online version 18

19 95% CI for MDES and Percentiles ## confidence intervals for MDES design5 <- optimal.to.mdes(design4) print(design5$mdes) ## mdes 95% LCL 95% UCL ## [1,] ## percentile equivalance mdes.to.pctl(design5) ## pctl 95% LCL 95% UCL ## [1,] "58%" "52%" "63%" 19

20 Summary A three-level hierarchical linear model is proposed to estimate treatment effect, where schools are randomly assigned to treatment and control conditions with 40% probability. We assume that proportion of variance in algebra posttest explained by school mean algebra pretest is 45%, proportion of variance in algebra posttest explained by classroom mean algebra pretest is 50%, proportion of variance in algebra posttest explained by student algebra pretest is 55%. Furthermore we assume intraclass correlation coefficient at the school level is 17.6%, and intraclass correlation coefficient at the classroom level is 5.9%. Having $10 marginal cost per student, $200 marginal cost per classroom, $500 marginal cost per school, based on PowerUpR package (Bulus & Dong, 2017), with a budget of $ we can afford recruting 100 schools in total, each with two classrooms and 40 students on average. For a two-tailed hypothesis test for treatment effect, and Type I error rate of 5%, such sample is estimated to detect an effect size as small as % CI[.060,.344] with statistical power of 80%, which is equaivalent to increasing an average student's algebra posttest score from 50th percentile to 58th percentile. 20

21 Future Directions Moderator and mediator effects (in ~ v0.2.3) Binary outcomes for main, moderator and mediator effects (in ~ v1.2.3) 21

22 Thank you! Thank you for your participation and comments! 22

23 References Bulus, M., & Dong, N. (2017). PowerUpR: Power Analysis Tools for Multilevel Randomized Experiments. R package version Borenstein, M., Hedges, L. V., & Rothstein, H. (2012). CRT Power. Teaneck, NJ: Biostat.[Software] Dong, N., & Maynard, R. A. (2013). PowerUp!: A Tool for Calculating Minimum Detectable Effect Sizes and Minimum Required Sample Sizes for Experimental and Quasi-Experimental Design Studies, Journal of Research on Educational Effectiveness, 6(1), Hedges, L. V., & Borenstein, M. (2014). Conditional Optimal Design in Three- and Four-Level Experiments. Journal of Educational and Behavioral Statistics, 39(4), R Core Team (2017). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. URL Raudenbush, S. W., Spybrook, J., Congdon, R., Liu, X. F., Martinez, A., & Bloom, H. (2011). Optimal design software for multi-level and longitudinal research (Version 3.01) [Software]. U.S. Department of Education, Institute of Education Sciences, What Works Clearinghouse. (2016, May). Secondary Mathematics intervention report: Saxon Math. Retrieved from 23