WEB-CAT. Exploring Trends and Student Behaviors from Data Collected on an Automated Grading and Testing System

Transcription

1 Exploring Trends and Student Behaviors from Data Collected on an Automated Grading and Testing System WEB-CAT Tony Allevato, Stephen Edwards Virginia Tech Department of Computer Science

2 Overview Introduction to Web-CAT Primary strengths Approach to scoring Assignments and grading plug-ins Data mining with Web-CAT Motivation Brief discussion of implementation Case studies Community building efforts

3 Introduction to Web-CAT Web-based Center for Automated Testing Designed as a general purpose automated grading system for programming assignments Strong emphasis on test-driven development Encourage students to test early and often with timely, meaningful feedback

4 Primary Strengths Were virtually no mainstream automated grading systems in widespread use Web-CAT is now in use at 30 institutions Designed to fill this role based on four major strengths Security Portability Extensibility Support for manual grading

5 Security Authentication Local user database, POP server, LDAP, or institution/directory-specific authentication adapter Both user-specific and role-based access controls Designate users to create sections of a course Only users listed as instructor or TA for a course can grade student code Detection of erroneous or malicious code

6 Portability Written in Java, Apple WebObjects Distributed as a single WAR (Web Archive) file Runs in a servlet container, such as Apache Tomcat All student, TA, instructor, and administrator functionality accessed through web browser

7 Extensibility Web-CAT as a large plug-in manager Student submissions are processed entirely by plug-ins Plug-ins can be configured on perassignment basis Language neutral has been used with Java, C++, Scheme, Prolog, Standard ML, and Pascal

8 Support for Manual Grading WYSIWYG source code view lets course staff directly mark up student code with comments, suggestions, deductions Deductions can be applied per-item, or the grader can adjust the overall score directly Students automatically notified when manual grading is completed

9 Setting up an Assignment Processing for an assignment consists of a tool chain of one or more grading plugins Instructor has complete control over which plug-ins to use, order of execution, and configuration parameters for each Simple and flexible way for plug-ins to communicate with Web-CAT, each other

10 Plug-in Management We already have plug-ins for many commonly used languages Plug-ins can be automatically updated, as with the Web-CAT system itself Instructors can write and upload their own plug-ins easily Can be written in any language executable on the server (we use Perl)

11 Approach to Scoring Code correctness Based on pass rate of student s own tests Test completeness How thoroughly student s tests cover the problem Measured by code coverage statistics Test validity Are student s tests accurate? Checked using hidden instructor-provided reference tests

12 Grading Java Assignments ANT-based automated build of any Java project with arbitrary package structures Automatic identification, execution of JUnit tests Code coverage measurements with Clover Static analysis with Checkstyle and PMD Optional instructor-provided reference tests

13 Grading C++ Assignments ANT-based automated build of any C++ project with arbitrary directory structures Automatic identification/execution of unit tests written with the CxxTest library Code coverage measurements with Bullseye Optional instructor-provided reference tests Heap tracking to identify memory leaks and memory corruption Trapping of assert failures, segmentation faults mapped to test case failures

14

15

16 Web-CAT Data Mining: Motivation Web-CAT generates a large amount of useful intermediate data while grading For Java and C++ assignments: Code coverage metrics Success/failure rates of student s unit tests and instructor s reference tests Code size statistics (# lines of code, # lines of commenting, etc.) Code style violations (Java only)

17 Web-CAT Data Mining: Motivation Grading plug-ins can process submissions and collect data in arbitrary ways In the end, plug-ins must write out a score data for the submission results report Intermediate data used to compute this score are trapped in the database and file system and not publicly exposed

18 BIRT Report Engine Eclipse-based open-source report generation system for Java Uses Eclipse IDE to provide a WYSIWYG report designer for end-users Reports can include a wide variety of tables, charts, and crosstabs Extensive JavaScript support allows for highly dynamic content

19

20

21 A Library of Report Templates We plan to develop a library of report templates so that instructors can begin analyzing data with little effort Instructors can design their own report templates and share them, as with grading plug-ins

22 Access to Data in Reports Reports are given full access to Web-CAT data model User- and role-based security policies to limit access (to be implemented) Data collected and output by grading plug-ins also accessible So, collecting new data for submissions is as easy as writing a new grading plug-in; new data automatically accessible

23 Report Deliverables Reports available in a variety of formats HTML for online viewing PDF for downloading and distributing Export data in a report to an Excel or CSV file for further, more sophisticated analysis

24 Number of Submissions vs. Final Score How do students scores correspond to the number of attempts they made on the assignment? Students with few attempts may fall into two categories advanced students and very poor students; more likely the latter Hypothesis: Scores improve as the number of attempts increases

25 Data does support the original hypothesis.

26 Code Complexity vs. Final Score How does the complexity of a student s submission correspond to his or her final grade? Complexity can be defined various ways We use a simple, easy to compute metric Hypothesis: Submissions with very low or very high complexity score poorly; there is a sweet spot in-between

27 Data does not support the original hypothesis.

28 Examining Testing Habits We require students to thoroughly test the code that they write Would like to know if they are testing incrementally, or only toward the end Generate time-based charts to show how much of student submissions consist of test code vs. non-test code

29

30 Final Assignment of the Semester First Assignment of the Semester

31 Final Assignment of the Semester First Assignment of the Semester

32 Earliness in Testing vs. Final Score Do students who test early perform better than those who do not? Earliness of testing can be defined various ways Hypothesis: Students who test early catch more bugs and have better programming habits than those who do not, and thus score higher

33 Data does not support the original hypothesis.

34 Future Work Scheduling reports for future generation One-time or recurring schedule notification when report is ready Data-directed scheduling parameters Start when the first submission is made Run every 12 hours, but only if new submissions have been made during that time End after the assignment is closed, or no new submissions have been made Predictive analysis, student interventions

35 Conclusions Students surveyed felt more confident about their assignments thanks to easy and thorough feedback from Web-CAT Initial evaluation showed improvements: 28% reduction in number of bugs per thousand lines of code, on average

36 Conclusions Instructors now have available a wide range of data to analyze the success of their course material, habits of students Intent is not to draw universal conclusions on how to improve learning; just make data accessible, instructors use it to draw own conclusions

37 Community Building Efforts Eclipse, Visual Studio plug-ins CATspace (in development) Facebook application Collaborate with other instructors and Web-CAT users Share assignment specs, sample solutions, and other resources