From Runtime Failures to Patches: Study of Patch Generation in Production

Transcription

1 From Runtime Failures to Patches: Study of Patch Generation in Production Advisors: Martin Monperrus & Lionel Seinturier Thomas Durieux July 12, 2018 INRIA & University of Lille

2 Windows sends more than 100 million error reports per day! 0

3 Outline Automatic Program Repair BikiniProxy: Patch Generation for Client-side applications BikiniProxy Demo BikiniProxy Architecture BikiniProxy Evaluation Itzal: Patch Generation for Server-side applications Itzal Concept Itzal Architecture Itzal Evaluation Conclusion 1

4 Automatic Program Repair Buggy Application Repair Strategy Oracle (e.g: Crash) 2

5 Automatic Patch Generation Buggy Program GenProg, Nopol,... Failure Oracle: Failing Tests Regression Oracle: Passing Tests 3

6 Test-based automatic program repair Use the test suite as the specification of the program. Status Tests Test Functionality 1 Test Functionality 2 Test Functionality 3 4

7 Test-based automatic program repair Use the test suite as the specification of the program. Common usage: reproduce a bug with a test suite Status Tests Test Functionality 1 Test Functionality 2 Test Functionality 3 Reproduce Bug-X 4

8 Test-based automatic program repair Use the test suite as the specification of the program. Goal: make the failing test pass Status Tests Test Functionality 1 Test Functionality 2 Test Functionality 3 Reproduce Bug-X 4

9 Automatic Program Repair Problem Automatic Program Repair relies of a failing test-case. Our solution Use the production system for patch generation. 5

10 Outline Automatic Program Repair BikiniProxy: Patch Generation for Client-side applications BikiniProxy Demo BikiniProxy Architecture BikiniProxy Evaluation Itzal: Patch Generation for Server-side applications Itzal Concept Itzal Architecture Itzal Evaluation Conclusion 6

11 BikiniProxy Demo 6

12 BikiniProxy Architecture User Firefox Web Server 7

13 BikiniProxy Architecture image.png style.css script.js page.html User Firefox Web Server 7

14 BikiniProxy Architecture image.png JS Error style.css script.js page.html User Firefox Web Server 7

15 BikiniProxy Architecture User Firefox Web Server 7

16 BikiniProxy Architecture Proxy User Firefox Web Server 7

17 BikiniProxy Architecture image.png style.css script.js page.html Proxy User Firefox Web Server 7

18 BikiniProxy Architecture image.png style.css script.js biki-page.html Proxy User Firefox Web Server 7

19 BikiniProxy Architecture image.png JS Error style.css script.js biki-page.html Proxy User Firefox Web Server 7

20 BikiniProxy Architecture BikiniProxy Backend image.png JS Error style.css script.js biki-page.html Proxy User Firefox Web Server 7

21 BikiniProxy Architecture BikiniProxy Backend image.png style.css biki-script.js biki-page.html Proxy User Firefox Web Server 7

22 Repair Strategies 1. HTTP/HTTPS Redirector changes HTTP to HTTPS 2. HTML Element Creator creates HTML elements 3. Library Injector injects missing libraries 4. Line Skipper adds a precondition to a statement 5. Object Creator initializes a null variable 8

23 Evaluation Protocal 1. Access a buggy page 2. Collect the errors on the page 3. Activate BikiniProxy 4. Access the buggy page with BikiniProxy activated 5. Collect the errors on the page with BikiniProxy activated 9

24 Evaluation Benchmark: DeadClick Crawling stats Value # Visited Pages # Pages with Error 4282 (4.5%) Benchmarks stats Value # Pages with Reproduced Errors 555 # Domains 466 # Average # resources per page # Average scripts per page # Min errors per page 1 # Average errors per page 1.49 # Max errors per page 10 # Average pages size 1.98mb 10

25 Evaluation Results Error Results XXX is not defined 184/307 (59.93%) Cannot read property XXX of null 42/176 (23.86%) XXX is not a function 11/111 (9.9%) Unexpected token X 2/61 (3.27%) Cannot set property XXX of null 11/24 (45.83%) Invalid or unexpected token 0/21 (0%) Unexpected identifier 0/15 (0%) Script error for: XXX 2/10 (20%) The manifest specifies... 0/7 (0%) adsbygoogle.push() error: No slot 0/7 (0%) 53 different errors 248/826 (30.02%) 11

26 Evaluation Results Metric Name # Pages Percent All Errors Disappeared 176/ % Some Errors Disappeared 42/ % Different/Additional Errors 140/ % No Strategy Applied 196/ % 12

27 BikiniProxy Highlights First fully automatic patch generation Patch generation without changing the production applicatio First benchmark of JavaScript failures 13

28 Outline Automatic Program Repair BikiniProxy: Patch Generation for Client-side applications BikiniProxy Demo BikiniProxy Architecture BikiniProxy Evaluation Itzal: Patch Generation for Server-side applications Itzal Concept Itzal Architecture Itzal Evaluation Conclusion 14

29 Itzal Concept Firefox Web Server 15

30 Itzal Concept Buggy output Firefox Web Server 15

31 Itzal Concept Expected output Firefox Patched Web Server 15

32 Itzal Concept Inconsistent output Firefox Patched Web Server 15

33 Itzal Concept Buggy output Firefox Web Server 15

34 Itzal Concept Buggy output Firefox Web Server Patch Service Patched Web Server 15

35 Itzal Concept Buggy output Firefox Web Server Patch Service Patched Web Server Expected output 15

36 Itzal Concept Expected output? Firefox Web Server Patch Service Patched Web Server Expected output 15

37 Itzal Concept Highlights Patch generation in production. Patch regression with production inputs. Sandboxed repair environment repair the program without affecting the production. 16

38 Itzal Architecture Client Application Client: e.g. a browser Application: e.g. a web server 17

39 Itzal Architecture Application Client Shadower Shadower: intercepts and duplicates the requests 17

40 Itzal Architecture Application Failure Oracle Client Shadower Patch Service Patch Service: generates patches that fix the requests Failure Oracle: detects if a request is passing or failing 17

41 Itzal Architecture Application Failure Oracle Client Shadower Patch Service Regression Service Regression Oracle Regression: executes passing request on patched applications Regression Oracle: compares the output of the application and the patched application 17

42 Itzal Architecture Application Failure Oracle Client Shadower Patch Service Developer Regression Service Reporting Regression Oracle Reporting: communicates the patches to the developers (Dashboard, Pull Request,...) 17

43 Itzal Oracle Failure Oracle: decides if a request is valid or not (e.g. HTTP status 5xx) Regression Oracle: decides if the patch does not modify the behavior for all non-failing requests. 18

44 Itzal Prototype e-commerce HTTP status 5xx Workload Shadower NPEFix 19

45 NPEFix Ok Execution Start Execution End Failure 20

46 NPEFix NPE1 Line 3 Strategy: return new Date(); Ok Execution Start Execution End Failure 20

47 NPEFix NPE1 Line 3 Strategy: return new Date(); Ok Execution Start Strategy: return null; Strategy: new Date().toString(); Execution End NPE2 Line 6 Failure 20

48 NPEFix NPE1 Line 3 Strategy: return new Date(); Ok Execution Start Strategy: return null; Strategy: new Date().toString(); Execution End NPE2 Line 6 Strategy: return; Failure 20

49 Itzal Evaluation 3 evaluations 1. Evaluate the patch generation from a failing execution 2. Evaluate the regression oracle to detect behavior changes 3. Cases studies 21

50 Itzal Patch Generation Evaluation Patch Models Bug NPEFix Exception-Stopper # Valid # Invalid # Valid # Invalid 34 bugs from 14 applications

51 Itzal Regression Oracle Evaluation HTTP status Oracles HTTP content # Method # Block Is Valid Patch? Patches Patch 1 0% 0% 0% 0% Yes Patch 2 0% 21% 6% 6% No Patch 3 0% 15% 7% 7% No 23

52 Itzal Case Study Evaluation 24

53 Itzal Case Study Evaluation 25

54 Generated Patch for Mayocat FlatStrategyPriceCalculator.java 37,2 +37,5 +if (carrier.getperitem() == null) { +return null; +} price = price.add(carrier.getperitem().multiply( BigDecimal.valueOf(numberOfItems))); 26

55 Outline Automatic Program Repair BikiniProxy: Patch Generation for Client-side applications BikiniProxy Demo BikiniProxy Architecture BikiniProxy Evaluation Itzal: Patch Generation for Server-side applications Itzal Concept Itzal Architecture Itzal Evaluation Conclusion 27

56 Conclusion Take Away Patch generation in production. Patch regression with production inputs. Evaluation production system is hard. Future Work New regression oracles Automatic patch generation with developers 28