Fuzz Testing Projects in Massive Courses

Size: px

Start display at page:

Download "Fuzz Testing Projects in Massive Courses"

Margaret Lang
6 years ago
Views:

Learning@Scale 2016 Fuzz Testing Projects in Massive Courses Sumukh Sridhara, Brian

1 Fuzz Testing Projects in Massive Courses Sumukh Sridhara, Brian Hou, Jeffrey Lu and John DeNero UC Berkeley April 26th 2016 Edinburgh, UK

2 Programming Projects in MOOCs IMPLEMENT Programming Projects 1 Primarily Instructional FEEDBACK 2 Instructor Solution Exists REVISE 3 Automated Feedback 2

3 Feedback Goals 1 Help students arrive at a correct answer 2 Students can help themselves 3 Every missed bug is a missed learning opportunity 3

4 Targeted Comprehensive 1 Isolates One Issue 100% Tests every case Guide Student Attention Hard To Engineer Many Targeted Tests Hard To Compute

5 Fuzz Testing

6 Fuzz Testing Testing the behavior of the program on many random inputs Complementary to Manual Testing Historically used for security

7 Fuzz Testing (Security) Generate Random Inputs User Program Verify Security Constraints 7

8 Fuzz Testing (Programming Projects) Generate Random Inputs Student Program Verify Correctness 8

9 Fuzz Testing (Programming Projects) Generate Random Inputs Student Program Instructor Program Student Output Expected Output 9

10 How to compare output? How many inputs are required? How to improve upon Fuzz Tests?

11 Creating & Distributing Tests Generate Inputs Input domain is known Generating Random Inputs is easy Student Program Instructor Program Verify Output 11

12 Implementation Generate Inputs q5(93, q5(93, 97, 97, 3) q5(23, 17, 3) Test #1007) Random input generation All students were provided with an identical set of tests Large number of inputs needed 12

13 Creating & Distributing Tests Generate Inputs Input domain is known Generating Random Inputs is easy Student Program Instructor Program Verify Output 13

14 Creating & Distributing Tests Generate Inputs Raw Output Check Hashing Student Program Instructor Program Program Tracing Verify Output 14

15 RQ1: How to Compare Outputs? Verify Output Raw comparison of output Compare against precomputed result q5(93, q5(93, 97, 97, 3) q5(23, 17, 3) Test #1007) q5(93, q5(93, 97, 97, 3) q5(23, 17, 3) Student output 7) Output Vector 15

16 RQ1: How to Compare Outputs? Hashing Output Hash combination of outputs Compare against precomputed result q5(93, q5(93, 97, 97, 3) q5(23, 17, 3) Test #1007) q5(93, q5(93, 97, 97, 3) q5(23, 17, 3) Student output 7) Output Vector Combine & Hash

17 Evaluation UC Berkeley cs61a.org In Person CS1 Course with 1400 Students Enrolled 17

18 Student Autograder OK Server Analytics Feedback okpy.org 18

19 Collected Dataset Students Completing Project 1,331 Code Snapshots 486,482 Average Snapshots per Student 349 Incorrect Attempts at Target Question 48,079 19

20 Output Vectors Generated Inputs Output Vector Correct Student Attempt Student Attempt Student Attempt Student Attempt 20

21 Output Vectors Output Vector 21

22 Frequency of Incorrect Outputs

23 RQ2: How many inputs are needed? 23

24 R2: How many inputs are needed? 1 Input 24

25 RQ2: How many inputs are needed? 2 Inputs 25

26 RQ2: How many inputs are needed? 3 Inputs 26

27 RQ2: How many inputs are needed? 4 Inputs 27

28 RQ2: How many inputs are needed? 7000 Number of False Positives Number of Inputs (Log Scale) 28

29 Fuzz Testing Effectiveness 52% 48% 656 Students (48.4%) passed all of the targeted tests but still had an error caught by the Fuzz Tests Targeted + Fuzz Tests Targeted Tests 29

30 RQ3: Improving Fuzz Tests Cumulative Percent Fixed Targeted Test Fuzz Test Attempts Required To Fix Error 30

31 How to Improve On Fuzz Tests? As a result of the Fuzz Test: 19% 7% 46% 46% of students reported spending 1 to 4 hours debugging 19% of students reported spending more than 4 hours debugging 28% Obfuscating output made it harder for instructors to help students No Time < 1 Hour >4 Hour >1 Hour 31

32 Improvements Program Inspection Incorrect result after playing 1 game(s): score0 score1 Turn Summary Turn 0: 0 0 Player 0 rolls 0 dice: Turn 1: 1 0 Player 1 rolls 7 six- sided dice: +37 3, 4, 6, 3, 3, 4, Dice sum: Incorrect implementation of game at turn 1. Please read over the trace to find your error. (error_id: ) 32

33 Thank okpy.org cs61a.org Fuzz Testing Projects in Massive Courses 2016

Create a Test Manually

Purpose: This guide includes important terminology and the steps needed to successfully create a Manual Test in Schoolnet. Term/Concept Description Page(s) Create a Test Manually Creating a test manually