Evaluating an Alternative CS1 for Students with Prior Programming Experience Michael S. Kirkpatrick Chris Mayfield SIGCSE Technical Symposium March 2017
JMU Introductory Sequence CS1 Java CS2 Not Java CS 139 (4 cr.) CS 239 (4 cr.) CS 240 (3 cr.) Java Development process Control structures Variables and expressions Functions Arrays Classes/objects Console I/O Classes/objects OOP concepts Packages References Recursion Exceptions Basic file I/O MA 205 (3 cr.) Applied Calculus Design and testing Collections New language Recursion Asymptotics Searching Sorting Hashing Trees
JMU Introductory Sequence CS 139 (4 cr.) CS 239 (4 cr.) CS 240 (3 cr.) CS1 Goals and benefits No prior exposure CS 139 required CS 239 Emphasis on (4 cr.) algorithmic thinking (4 cr.) Maintain positive climate Small class sizes Exposure to two languages CS2 CS 240 (3 cr.) MA 205 (3 cr.)
JMU Introductory Sequence Fall 2013: 2014: Accelerated Prereq + Calculus Java C students just left later... CS1 Prereq: CB- CS2 CS 139 (4 cr.) CS 149 Students with (3 cr.) prior exposure CS 239 159 (4 (3 cr.) MA 205 235 (3 cr.) CS 240 (3 cr.) Not enough for Big-O analysis
Key (Research) Questions Effects on retention Did either change affect retention into CS 159? Did either change affect retention beyond CS 159? Effects on successful progression How did they do in CS 240? Effects on underrepresented groups Do overall effects extend to women and underrepresented minority students?
Research Hypotheses CS 159 and CS 240 retention The split sections improve retention. The Calculus/B- change led to drop in retention. CS 159 and CS 240 grades CS 139/149 has no effect on CS 159 or CS 240 grades. Skip CS 139/149 with AP 4 yields difference in CS 159. AP 4 + CS 139/149 no difference in CS 159 or CS 240 relative to AP 5. Effect on women and URM students Split sections improve women/urm retention. Calculus/B- had disproportionate effect on women/urm.
CS 139/149 Demographics 100% 80% 60% 40% 20% 0% Baseline 2014-15 White Asian URM Male Female
CS 139/149 CS 159/239 Retention No overall effect after splitting sections Difference between CS 139 and previous combined CS1/1a combined CS1a CS1 2014-15 X-sq = 0.02, p = 0.90 2014-15 X-sq < 0.01, p = 0.95 2014-15 X-sq = 0.04, p = 0.85 Baseline (pre-2013) CS1/1a combined X-sq = 1.81, p = 0.18 CS1 only X-sq = 4.83, p = 0.03* CS1a only X-sq = 1.66, p = 0.20 0% 20% 40% 60% 80% 100% Retention Attrition 0% 20% 40% 60% 80% 100% No effect after adding B- prerequisite
CS 159 240 Retention No effect after accelerated split Baseline (pre-2013) CS1/1a combined X-sq = 1.97, p = 0.16 CS1 only X-sq = 3.33, p = 0.07 CS1a only X-sq = 0.02, p = 0.88 0% 20% 40% 60% 80% 100% CS1/1a combined CS1a CS1 2014-15 X-sq = 5.44, p = 0.02* 2014-15 X-sq = 6.89, p < 0.01** 2014-15 X-sq = 0.18, p = 0.67 0% 20% 40% 60% 80% 100% Significant difference after Calculus/B-, but...
Research Hypotheses CS 159 and CS 240 retention The split sections improve retention. The Calculus/B- change led to drop in retention. / CS 159 and CS 240 grades CS 139/149 has no effect on CS 159 or CS 240 grades. Skip CS 139/149 with AP 4 yields difference in CS 159. AP 4 + CS 139/149 no difference in CS 159 or CS 240 relative to AP 5. Effect on women and URM students Split sections improve women/urm retention. Calculus/B- had disproportionate effect on women/urm.
How were their grades? Linear regression on CS 159 grades Choice of CS 139 vs CS 149? Significant (p < 0.05) Grade in CS 139/149? Very significant (p < 0.001) # of attempts in CS 139/149? Very significant (p < 0.01) CS1.5 Grade Factor Est. SE t value Pr(> t ) (Intercept) CS1 0.745 0.436 = 0.158,p=0.009 1.708 0.089. ** CS1/1a Grade-0.319 0.126 = 0.492,p<3e-16-2.525 0.012 **** CS1X CS1/1a GradeAttempts 0.834 0.096 = 0.182,p= 8.685 < 0.002 4e-16 CS1X Attempts -0.600 0.224-2.675 0.008 ** Residual standard error: 0.917 on 270 degrees of freedom Multiple R-squared: 0.2633, Adjusted R-squared: 0.2552 F-statistic: 32.17 on 3 and 270 DF, p-value: < 2.2e-16 CS1.5 grade factor Spearman s rank coe cient
How were their grades? Linear regression on CS 240 grades Choice of CS 139 vs CS 149? NOT significant Grade in CS 159? Very significant (p < 0.001) # of attempts in CS 139/149? NOT significant CS2 Grade Factor CS1 Est. SE =0.052,p=0.489 t value Pr(> t ) (Intercept) 1.187 0.234 5.079 < 1e-06 *** CS1.5a Grade = 0.467,p<5e-11 *** CS1.5 Grade 0.512 0.075 6.788 < 2e-10 *** Residual standard error: 0.723 on 177 degrees of freedom Multiple R-squared: 0.2066, Adjusted R-squared: 0.2021 F-statistic: 46.08 on 1 and 177 DF, p-value: 1.651e-10 CS2 grade factor
What about AP students? Average grades in CS 159/240 AP 4 who skipped CS 139/149 did worse in CS 159 Course Mean Samples Result CS1.5/1.5a AP 5, µ =3.4 t =2.4,df =56.2 AP 4 (skip), µ =2.8 p =0.02 * CS1.5/1.5a AP 5, µ =3.4 t =1.9,df =53.9 AP 4 (no skip), µ =3.0 p =0.06 CS1.5/1.5a AP 4 (skip), µ =2.8 t = 0.7,df =55.2 AP 4 (no skip), µ =3.0 p =0.47 CS2 AP 5, µ =2.7 t =1.6,df =44.6 AP 4 (skip), µ =2.2 p =0.12 CS2 AP 5, µ =2.7 t =0.2,df =39.0 AP 4 (no skip), µ =2.7 p =0.82 CS2 AP 4 (skip), µ =2.2 t = 1.5,df =42.2 AP 4 (no skip), µ =2.7 p =0.15
Research Hypotheses CS 159 and CS 240 retention The split sections improve retention. The Calculus/B- change led to drop in retention. / CS 159 and CS 240 grades CS 139/149 has no effect on CS 159 or CS 240 grades. Skip CS 139/149 with AP 4 yields difference in CS 159. AP 4 + CS 139/149 no difference in CS 159 or CS 240 relative to AP 5. / Effect on women and URM students Split sections improve women/urm retention. Calculus/B- had disproportionate effect on women/urm.
CS 139/149 CS 159/239 Retention URM CS1/1a No effect after accelerated split 2014-15 X-sq = 0.11, p = 0.74 Women URM Baseline (pre-2013) CS1/1a X-sq < 0.01, p = 0.98 CS1 X-sq < 6e-31, p 1 Baseline (pre-2013) CS1/1a X-sq < 2e-30, p 1 CS1 X-sq = 0.01, p = 0.91 0% 20% 40% 60% 80% 100% URM CS1 only Women CS1/1a 2014-15 X-sq < 0.01, p = 0.96 2014-15 X-sq = 0, p 1 No effect after increasing to B- Women CS1 only 2014-15 X-sq = 0, p 1 0% 20% 40% 60% 80% 100%
CS 159 CS 240 Retention Men Women URM Significant drop in women retention after increasing Calculus requirement... Baseline CS1 (pre-2013) 2013-15 CS1/1a combined X-sq < 8e-31, p 1 Baseline CS1 (pre-2013) CS1/1a combined X-sq = 0, p 1 2014-15 CS1/1a (vs. ) X-sq = 5.11, p = 0.02* CS1/1a combined 2014-15 CS1/1a combined X-sq = 1.15, p = 0.29 0% 20% 40% 60% 80% 100%
Research Hypotheses CS 159 and CS 240 retention The split sections improve retention. The Calculus/B- change led to drop in retention. / CS 159 and CS 240 grades CS 139/149 has no effect on CS 159 or CS 240 grades. Skip CS 139/149 with AP 4 yields difference in CS 159. AP 4 + CS 139/149 no difference in CS 159 or CS 240 relative to AP 5. / Effect on women and URM students Split sections improve women/urm retention. Calculus/B- had disproportionate effect on women/urm.? /
Key Takeaways Extended intro helps level playing field No difference by the end of CS2 Extended intro helps students with AP 4 No difference by the end of CS2 Extended intro helps for fair enrollment mgmt No evidence of prior exposure bias
Thank you kirkpams@jmu.edu mayfiecs@jmu.edu