1 Automatically Generating Tutorials to Enable Middle School Children to Learn Programming Independently Kyle Harms, Dennis Cosgrove, Shannon Gray, Caitlin Kelleher
Shortage of Programmers An estimated 1.4 million computing jobs will be added to the United States economy between 2008-2018. 1 61% of these jobs can be filled based on current college graduation rates. 1 Shortage of information communications technology workers across the European Union. 2 1 Computing Education and Future Jobs: A Look at National, State, and Congressional District Data (2011) 2 IEEE Job Site: http://careers.ieee.org/article/european_job_outlook_0312.php 2
1 Shedding Some New Light on Old Truths: Student Attitudes to School in Terms of Year Level and Gender (1994) 2 Pryor, J.H. et al. 2010. The American Freshman: National Norms for Fall 2009. 3 Middle School Children & Computer Programming Middle school is the time many children decide to opt-out of advanced math or science courses. 1 By college these students are too far behind to realistically succeed in these majors. 2 Maintain interest and develop programming skills through independent learning.
4 Contributions Demonstrate a process for automatically generating programming tutorials from unfamiliar code. The tutorials improved independent learning of programming constructs in near transfer tasks by 64%.
Looking Glass 5
Independent Learning In Looking Glass Talent Show Program 6
Learning From Unfamiliar Code Find Program Select Snippet Remix Snippet Code Snippet Remixed Code Snippet 7
Snippet Copied Into Program 8
9 Exposure to New Programming Concepts
Independent Learning In Looking Glass Find Program 1 Select Snippet 2 Remix Snippet 2 Programming Tutorial Existing Support for Independent Learning This Talk 1 Harms, K.J. et al. 2012. Designing a community to support long-term interest in programming for middle school children. Proc. IDC. 2 Gross, P.A. et al. 2010. A code reuse interface for non-programmer middle school students. Proc. IUI. 10
Automatic Tutorial Generation Bergman, L. et al. 2005. DocWizards: a system for authoring follow-me documentation wizards. Proc. UIST. Grabler, F. et al. 2009. Generating photo manipulation tutorials by demonstration. ACM SIGGRAPH. Fernquist, J. et al. 2011. Sketch-sketch revolution: an engaging tutorial system for guided sketching and application learning. Proc. UIST. 11
12 Current Generated Tutorial Systems Users must adapt tutorial content to their contexts. Require explicit authoring phase. Users may skip steps or make mistakes.
13 Walk-through Tutorial to Reconstruct the Snippet
Interactive Stencils Tutorial Interface Harms, K.J. et al. 2011. Improving learning transfer from stencils-based tutorials. Proc. IDC. 14
Programming Tutorial Find Program Select Snippet Remix Snippet Programming Tutorial Code Snippet Remixed Code Snippet 15
16
17 Snippet Reconstructed Through Walkthrough Tutorial
18 Generating Walkthrough Tutorials from Code Snippets
19 Reconstructing a Code Snippet 1. Insert a Do Together statement. 2. Insert move statement into the Do Together. 3. Insert roll statement into the Do Together.
20 Walkthrough Tutorial Steps Code Snippet Tutorial Steps
21 Model-Driven Architecture On-screen Widget Model Data Code Statement
22 Translate Code Statements into Tutorial Steps Tutorial Step Model Code Statement
23 Draft Tutorial Code Snippet Tutorial Steps
What if the interface is in the wrong state to complete the current step? 24
Insert Do Together 25
Insert Do Together 26
Insert Do Together 27
28 Tutorial Step Dependencies On-screen Widgets Dependent Tutorial Steps
29 How can we present a valid tutorial to the user?
30 Presenting the Draft Tutorial Draft Tutorial Check if a step s dependencies are satisfied. Correct unsatisfied dependencies. Initialize the tutorial interface for the step. Ensure user correctly completes the step.
31 Algorithm for Presenting Steps For each draft tutorial step do: If step s dependencies are satisfied Then: Present the step to the user. Validate the user s progress. Advance to the next step. Else: Create and insert prerequisite step.
32 Presenting the Tutorial For each draft tutorial step do: If the step s dependencies are satisfied Then: Present the step to the user. Validate the user s progress. Advance to the next step. Else: Create and insert a prerequisite step.
33 Check Dependencies For each draft tutorial step do: If the step s dependencies are satisfied Then: Present the step to the user. Validate the user s progress. Advance to the next step. Else: Create and insert a prerequisite step.
Is the interface in a state where we can present this step? 34
35 Model-Driven Architecture + Dependencies Dependency On-screen Widgets Models
36 Check Dependencies Dependency Tutorial Step Models Current Interface State
37 Check Dependencies Dependency Tutorial Step Models Current Interface State
38 Check Dependencies For each draft tutorial step do: If the step s dependencies are satisfied Then: Present the step to the user. Validate the user s progress. Advance to the next step. Else: Create and insert a prerequisite step.
39 Insert Prerequisite Step For each draft tutorial step do: If the step s dependencies are satisfied Then: Present the step to the user. Validate the user s progress. Advance to the next step. Else: Create and insert a prerequisite step.
How do we adapt the tutorial to put the interface in the correct state? 40
41 Model-Driven Architecture + Insert Prerequisite Step Generate & Insert Dependency Models Tutorial Steps
42 Present Prerequisite Step For each draft tutorial step do: If the step s dependencies are satisfied Then: Present the step to the user. Validate the user s progress. Advance to the next step. Else: Create and insert a prerequisite step.
43 Check Dependencies For each draft tutorial step do: If the step s dependencies are satisfied Then: Present the step to the user. Validate the user s progress. Advance to the next step. Else: Create and insert a prerequisite step.
44 Present Step For each draft tutorial step do: If the step s dependencies are satisfied Then: Present the step to the user. Validate the user s progress. Advance to the next step. Else: Create and insert a prerequisite step.
45 How do we present the step to the user?
46 Model-Driven Architecture + Present Tutorial Step Tutorial Step Model Visible Widget
47 Present Step with Stencils Widget Tutorial Step Stencils-Based Interface
48 Validate User s Progress For each draft tutorial step do: If the step s dependencies are satisfied Then: Present the step to the user. Validate the user s progress. Advance to the next step. Else: Create and insert a prerequisite step.
49 How do we prevent mistakes from derailing the tutorial?
50 Model-Driven Architecture + Record User s Actions On-screen Widgets Model Transaction History
51 Validating the User s Progress Equivalent Current Tutorial Step Latest Recorded Transaction
52 Advance to Next Step For each draft tutorial step do: If the step s dependencies are satisfied Then: Present the step to the user. Validate the user s progress. Advance to the next step. Else: Create and insert a prerequisite step.
53 Advance to Next Step For each draft tutorial step do: If the step s dependencies are satisfied Then: Present the step to the user. Validate the user s progress. Advance to the next step. Else: Create and insert a prerequisite step.
54 Check Dependencies For each draft tutorial step do: If the step s dependencies are satisfied Then: Present the step to the user. Validate the user s progress. Advance to the next step. Else: Create and insert a prerequisite step.
55 Check Dependencies For each draft tutorial step do: If the step s dependencies are satisfied Then: Present the step to the user. Validate the user s progress. Advance to the next step. Else: Create and insert a prerequisite step.
56 Present Step For each draft tutorial step do: If the step s dependencies are satisfied Then: Present the step to the user. Validate the user s progress. Advance to the next step. Else: Create and insert a prerequisite step.
Automatically Generated Walkthrough Programming Tutorial 57
58 Evaluation 40 Middle school aged (10 16 years) participants 1.5 hour sessions each with no more than 5 participants Control Experimental
Easy Programming Constructs Do Together Execute in Parallel Count Loop Loop n times Hard For Each in Array Loop Iterate over array 59
Training Phase Control Snippet Copied into Program Remix Animation Experimental Reconstruct Snippet in Tutorial 60
Transfer Phase Initial Transfer Task Program Completed Transfer Task Program 61
62 Grading Transfer Task Programs Grading Criteria for the For Each Transfer Program: (5 points) 1. Program contains a For Each construct. If not, stop grading. (+1) 2. For Each contains at least one statement. If not, stop grading. (+1) 3. Array is defined correctly for the animation. (+1) 4. Programming statements use the loop iterator. (+1) 5. Animation is correct. (+1)
Results Average Transfer Program Scores for Each Programming Construct Task Control Experimental 86.67% 61.90% 70.77% 40.00% 8.33% 18.75% Easy Do Together Count For Each Hard Experimental condition performed 64% better. ANCOVA (F[2,37], p < 0.05). 63
64 Implications Any code can be used as a learning resource. Users can learn while they follow their own interests. Personalize tutorials to the learner s abilities.
65 Thanks Kyle J. Harms Washington University in St. Louis harmsk@seas.wustl.edu
66 Why ANCOVA? New Looking Glass users often have difficulty locating the Control Flow Tab. We provided a Control Flow Tab Hint Offered during the transfer program after 5 minutes Pointed to tab: To complete this task, look here. We used ANCOVA with the presence or absence of this hint as a covariate. The hint was not significant (p = 0.48)