LEVERAGING LIGHTWEIGHT ANALYSES TO AID SOFTWARE MAINTENANCE ZACHARY P. FRY PHD PROPOSAL

Size: px

Start display at page:

Download "LEVERAGING LIGHTWEIGHT ANALYSES TO AID SOFTWARE MAINTENANCE ZACHARY P. FRY PHD PROPOSAL"

Clyde Wheeler
5 years ago
Views:

1 LEVERAGING LIGHTWEIGHT ANALYSES TO AID SOFTWARE MAINTENANCE ZACHARY P. FRY PHD PROPOSAL

MAINTENANCE COSTS For persistent systems,

Verification Maintenance R.C. Seacord, D.

2 MAINTENANCE COSTS For persistent systems, software maintenance can account for up to 90% of the software lifecycle costs. 90% Requirements Design Implementation Verification Maintenance R.C. Seacord, D. Plakosh, and G. A. Lewis. Modernizing Legacy Systems: Software Technologies, Engineering Process and Business Practices. Addison-Wesley Longman Publishing Co. Inc., Boston, MA, USA,

3 KEY PARTS OF THE MAINTENANCE PROCESS File: Lines: Bug Reporting /*loop through all keys, removing! corrupted values from map */! Vector keys =! new Vector(map.keySet());! for(string s : keys){! if(map.get(s).iscorrupted()){! map.remove(s);! keys.remove(s);! Bug Fixing /*loop through all keys, removing! corrupted values from map */! Vector keys =! new Vector(map.keySet());! for(string s : keys){! if(map.get(s).iscorrupted()){! map.remove(s);! Update Documentation 3

4 MAINTENANCE PROCESSES IN PRACTICE Manual bug reporting is costly Reputation Human effort Automatic bug finders yield thousands of bugs, requiring verification and triage. 4

MAINTENANCE PROCESSES IN PRACTICE 10000 Number of Automatically

5 MAINTENANCE PROCESSES IN PRACTICE Number of Automatically Reported Defects by Program Defect Reports Benchmark Programs 5

6 MAINTENANCE PROCESSES IN PRACTICE OpenOffice bugs: Confirmed New Bugs Confirmed Resolved Bugs Bug reports come in at an alarming rate, humans simply cannot triage and fix them all. Automatic program repair 6

MAINTENANCE PROCESSES IN PRACTICE Fixing bugs

of page faults since startup should have units

7 MAINTENANCE PROCESSES IN PRACTICE Fixing bugs means lots of code changes. Comments are often overlooked Out-of-date documentation /* A reporter reporting the number of page faults since startup should have units UNITS_COUNT. */ /* The number of tabs currently open would have UNITS_COUNT. */ 7

8 MAINTENANCE PROCESSES IN PRACTICE Automated techniques have helped to facilitate the maintenance process. However, the process remains costly. Research question: Can we reduce the effort necessary for specific parts of the maintenance process, thereby reducing the overall cost? 8

9 PROPOSAL THESIS By using lightweight analyses to extract and use latent information encoded by humans in software development artifacts we can reduce the costs of software maintenance by relieving bottlenecks in various stages throughout the process. 9

10 RESEARCH CONSIDERATIONS Overall Goal: Reduce maintenance costs Design Constraint: Minimize additional human effort Ease of incremental adoption Overall Intuition: Leverage information often overlooked by existing techniques 10

11 THE REST OF THIS PRESENTATION An overview of the proposed thrusts Clustering Duplicate Automatically- Generated Defect Reports Improved Fitness Functions for Automatic Program Repair Ensuring Documentation Consistency Proposed research timeline Conclusion and Questions 11

12 PROJECT OUTLINE File: Lines: Clustering Duplicate Automatically-Generated Defect Reports /*loop through all keys, removing! corrupted values from map */! Vector keys =! new Vector(map.keySet());! for(string s : keys){! if(map.get(s).iscorrupted()){! map.remove(s);! keys.remove(s);! Improved Fitness Functions for Automatic Program Repair /*loop through all keys, removing! corrupted values from map */! Vector keys =! new Vector(map.keySet());! for(string s : keys){! if(map.get(s).iscorrupted()){! map.remove(s);! Ensuring Documentation Quality 12

13 CLUSTERING DUPLICATE DEFECT REPORTS Automatic bug finders successfully report many bugs with little developer effort Bug Finder Defect Reports Verification and Triage False Positives Actual Defects 13

developer effort However Bug Finder Defect Reports x

14 CLUSTERING DUPLICATE DEFECT REPORTS Automatic bug finders successfully report many bugs with little developer effort However Bug Finder Defect Reports x 1000s Verification and Triage False Positives Actual Defects 14

15 CLUSTERING DUPLICATE DEFECT REPORTS Intuitions: Duplicates are detrimental in related fields. Source Code Code Clone Detectors Manual Reports Duplicate Report Detector 15

16 CLUSTERING DUPLICATE DEFECT REPORTS Intuitions: Duplicates are detrimental in related fields. Source Code Code Clone Detectors Manual Reports Duplicate Report Detector 16

17 CLUSTERING DUPLICATE DEFECT REPORTS Hypothesis: By exploiting the special structure of automatic defect detection tools output we can accurately cluster defect reports to save effort by handling similar defect reports aggregately. Success depends on: Internal accuracy of the produced clusters Amount of effort saved from clustering defect reports 17

18 CLUSTERING DUPLICATE DEFECT REPORTS Defect Report 1 File: NSReader.java Suspected Line: plot = lst.get(i);! Defect Report 2 File: NSReader.java Suspected Line: p = lst.get(i);! Defect Report 3 File: NSReader.java Suspected Line: plot = lst.get(n);! Defect Report 4 File: UI_Impl.java Suspected Line: plot = lst.get(i);! 18

19 CLUSTERING DUPLICATE DEFECT REPORTS Defect Report 1 File: NSReader.java Suspected Line: plot = lst.get(i);! Defect Report 2 File: NSReader.java Suspected Line: p = lst.get(i);! Defect Report 3 File: NSReader.java Suspected Line: plot = lst.get(n);! Defect Report 4 File: UI_Impl.java Suspected Line: plot = lst.get(i);! 19

20 CLUSTERING DUPLICATE DEFECT REPORTS Defect Report 1 File: NSReader.java Suspected Line: plot = lst.get(i);! Defect Report 2 File: NSReader.java Suspected Line: p = lst.get(i);! Defect Report 3 File: NSReader.java Suspected Line: plot = lst.get(n);! Defect Report 4 File: UI_Impl.java Suspected Line: plot = lst.get(i);! 20

java Suspected Line: p = lst.get(i);! Defect Report 3 File: NSReader.

21 CLUSTERING DUPLICATE DEFECT REPORTS Defect Report 1 File: NSReader.java Suspected Line: plot = lst.get(i);! Defect Report 2 File: NSReader.java Suspected Line: p = lst.get(i);! Defect Report 3 File: NSReader.java Suspected Line: plot = lst.get(n);! Defect Report 4 File: UI_Impl.java Suspected Line: plot = lst.get(i);! 21

22 CLUSTERING DUPLICATE DEFECT REPORTS Clustering technique: R1 R4 R3 R6 R7 R8 R10 R11 R2 R5 R9 R12 22

23 CLUSTERING DUPLICATE DEFECT REPORTS Clustering technique: R1 R4 R3 R6 R7 R8 R10 R11 R2 R5 R9 R12 23

24 CLUSTERING DUPLICATE DEFECT REPORTS Preliminary Cluster Accuracy vs. Effort Savings Effort Saved (% of defects collapsed) Pareto Frontier - All Java Benchmark Programs Our Technique ConQAT PMD Checkstyle Accuracy (fraction of correctly clustered reports) 24

$6 0.8 1 Accuracy (fraction of correctly clustered$

25 CLUSTERING DUPLICATE DEFECT REPORTS Preliminary Cluster Accuracy vs. Effort Savings Pareto Frontier - All Java Benchmark Programs Effort Saved (% of defects collapsed) Our Technique ConQAT PMD Checkstyle Accuracy (fraction of correctly clustered reports) Saving more effort at all levels of accuracy 25

60 40 20 0 Our Technique ConQAT PMD Checkstyle

26 CLUSTERING DUPLICATE DEFECT REPORTS Preliminary Cluster Accuracy vs. Effort Savings Pareto Frontier - All Java Benchmark Programs Effort Saved (% of defects collapsed) Our Technique ConQAT PMD Checkstyle Capable of perfect accuracy Accuracy (fraction of correctly clustered reports) 26

27 PROJECT OUTLINE File: Lines: Clustering Duplicate Automatically-Generated Defect Reports /*loop through all keys, removing! corrupted values from map */! Vector keys =! new Vector(map.keySet());! for(string s : keys){! if(map.get(s).iscorrupted()){! map.remove(s);! keys.remove(s);! Improved Fitness Functions for Automatic Program Repair /*loop through all keys, removing! corrupted values from map */! Vector keys =! new Vector(map.keySet());! for(string s : keys){! if(map.get(s).iscorrupted()){! map.remove(s);! Ensuring Documentation Quality 27

28 IMPROVED FITNESS FUNCTIONS Automatic program repair GenProg Bugs? 28

29 IMPROVED FITNESS FUNCTIONS Automatic program repair can fix bugs. GenProg Bugs Fixes 29

30 IMPROVED FITNESS FUNCTIONS Automatic program repair can fix bugs. GenProg Bugs Fixes 30

31 IMPROVED FITNESS FUNCTIONS Measuring proximity to a fix Insert, delete, and swapping lines in the program FIX d(135) i(251,205) i(774,111) s(598,324) NO FIX 31

32 IMPROVED FITNESS FUNCTIONS Measuring proximity to a fix Insert, delete, and swapping lines in the program FIX d(135) i(251,205) i(774,111) s(598,324) i(251,205) i(774,111) s(598,324) d(63) 75% NO FIX 32

33 IMPROVED FITNESS FUNCTIONS Measuring proximity to a fix Insert, delete, and swapping lines in the program FIX d(135) i(251,205) i(774,111) s(598,324) i(251,205) i(774,111) s(598,324) d(63) 75% d(84) s(844,265) i(774,111) i(735,431) 25% NO FIX 33

34 IMPROVED FITNESS FUNCTIONS The current model of fitness does not correlate well with proximity to a fix. Intuitions: Not all test cases are created equal. Not all bugs are created equal. Not all fixes are created equal. We propose to address the naivety of the current fitness representation. 34

35 IMPROVED FITNESS FUNCTIONS Hypothesis: By taking into account previously unused information about test cases, bugs, and fixes we can better inform the evolutionary bug fixing process to fix bugs faster and more often. Success depends on: Increase the number of bugs fixed For bugs that can currently be fixed, shorten the time it takes to fix them 35

36 IMPROVED FITNESS FUNCTIONS Approach: weight test cases based on known fixes FIX NO FIX Test Case 1 Test Case 2 36

37 IMPROVED FITNESS FUNCTIONS Approach: weight test cases based on known fixes FIX NO FIX Test Case 1 Test Case 2 37

38 IMPROVED FITNESS FUNCTIONS Approach: weight test cases based on known fixes FIX NO FIX Test Case 1 Test Case

39 IMPROVED FITNESS FUNCTIONS Evaluation: How many more bugs can we fix? 55 out of 105 bugs fixed in the most recently published work 1 How much can we speed up fixes? Computational time and monetary cost 1. Claire Le Goues, Westley Weimer, Stephanie Forrest: Representations and Operators for Improving Evolutionary Software Repair. Genetic and Evolutionary Computing Conference (GECCO)

40 PROJECT OUTLINE File: Lines: Clustering Duplicate Automatically-Generated Defect Reports /*loop through all keys, removing! corrupted values from map */! Vector keys =! new Vector(map.keySet());! for(string s : keys){! if(map.get(s).iscorrupted()){! map.remove(s);! keys.remove(s);! Improved Fitness Functions for Automatic Program Repair /*loop through all keys, removing! corrupted values from map */! Vector keys =! new Vector(map.keySet());! for(string s : keys){! if(map.get(s).iscorrupted()){! map.remove(s);! Ensuring Documentation Quality 40

41 ENSURING DOCUMENTATION QUALITY The documentation becomes increasingly inaccurate thereby making future changes even more difficult. (Parnas) Real developers: 76% agree documentation is crucial to understanding But poorly executed in practice (27% complete, 33% consistent) 41

42 ENSURING DOCUMENTATION QUALITY /*loop through all keys, removing! corrupted values from map */! Vector keys =! new Vector(map.keySet());! for(string s : keys){! if(map.get(s).iscorrupted()){! map.remove(s);! 42

43 ENSURING DOCUMENTATION QUALITY /*loop through all keys, removing! corrupted values from map */! Vector keys =! new Vector(map.keySet());! for(string s : keys){! if(map.get(s).iscorrupted()){! map.remove(s);! /*loop through all keys, removing! corrupted values from map */! HashMap validmap = new HashMap();! Vector keys =! new Vector(map.keySet());! for(string s : keys){! if(map.get(s).isvalid()){! validmap.put(s,map.get(s));! map.remove(s);! 43

ENSURING DOCUMENTATION QUALITY /*loop through all keys, removing! corrupted values from map */! Vector keys =! new Vector(map.keySet());! for(string s : keys){! if(map.get(s).iscorrupted()){! map.remove(s);!

44 ENSURING DOCUMENTATION QUALITY /*loop through all keys, removing! corrupted values from map */! Vector keys =! new Vector(map.keySet());! for(string s : keys){! if(map.get(s).iscorrupted()){! map.remove(s);! INCONSISTENT! Comment incorrectly describes the functionality /*loop through all keys, removing! corrupted values from map */! HashMap validmap = new HashMap();! Vector keys =! new Vector(map.keySet());! for(string s : keys){! if(map.get(s).isvalid()){! validmap.put(s,map.get(s));! map.remove(s);! 44

45 ENSURING DOCUMENTATION QUALITY /*loop through all keys, removing! corrupted values from map */! Vector keys =! new Vector(map.keySet());! for(string s : keys){! if(map.get(s).iscorrupted()){! if(s.equals( Primary ))! printf( debug: %s\n,! map.get(s).tostring());! map.remove(s);! /*loop through all keys, removing! corrupted values from map */! Vector keys =! new Vector(map.keySet());! for(string s : keys){! if(map.get(s).iscorrupted()){! map.remove(s);! 45

46 ENSURING DOCUMENTATION QUALITY INCOMPLETE! Comment fails to describe all relevant functionality /*loop through all keys, removing! corrupted values from map */! Vector keys =! new Vector(map.keySet());! for(string s : keys){! if(map.get(s).iscorrupted()){! if(s.equals( Primary ))! printf( debug: %s\n,! map.get(s).tostring());! map.remove(s);! /*loop through all keys, removing! corrupted values from map */! Vector keys =! new Vector(map.keySet());! for(string s : keys){! if(map.get(s).iscorrupted()){! map.remove(s);! 46

47 ENSURING DOCUMENTATION QUALITY Reduce understandability over time Intuitions: Existing tools can accurately extract concepts from code and generate comments about those concepts. There should be natural language overlap in a high quality comments and the associated code. 47

48 ENSURING DOCUMENTATION QUALITY Hypothesis: By comparing concepts extracted from the code with the existing comments, we can accurately identify inconsistent and incomplete documentation. Success depends on: The accuracy of our incomplete and inconsistent comment identification technique The ease with which humans update and understand comments when using our tool 48

49 ENSURING DOCUMENTATION QUALITY Approach: /*loop through all keys, removing! corrupted values from map */! Vector keys =! new Vector(map.keySet());! for(string s : keys){! if(map.get(s).iscorrupted()){! if(s.equals( Primary ))! printf( debug: %s\n,! map.get(s).tostring());! map.remove(s);! 49

$equals( Primary ))! printf( debug: %s\n,! map.get(s).tostring());! map.remove(s);!$

50 ENSURING DOCUMENTATION QUALITY Approach: /*loop through all keys, removing! corrupted values from map */! Vector keys =! new Vector(map.keySet());! for(string s : keys){! if(map.get(s).iscorrupted()){! if(s.equals( Primary ))! printf( debug: %s\n,! map.get(s).tostring());! map.remove(s);! 50

51 ENSURING DOCUMENTATION QUALITY Approach: /*loop through all keys, removing! corrupted values from map */! Vector keys =! new Vector(map.keySet());! for(string s : keys){! if(map.get(s).iscorrupted()){! if(s.equals( Primary ))! printf( debug: %s\n,! map.get(s).tostring());! map.remove(s);! 51

52 ENSURING DOCUMENTATION QUALITY Approach: Generated Documentation (DeltaDoc): Now call printf if s is Primary! /*loop through all keys, removing! corrupted values from map */! Vector keys =! new Vector(map.keySet());! for(string s : keys){! if(map.get(s).iscorrupted()){! if(s.equals( Primary ))! printf( debug: %s\n,! map.get(s).tostring());! map.remove(s);! 52

53 ENSURING DOCUMENTATION QUALITY Approach: Generated Documentation (DeltaDoc): Now call printf if s is Primary! Existing comment lacks this info, thus is incomplete. /*loop through all keys, removing! corrupted values from map */! Vector keys =! new Vector(map.keySet());! for(string s : keys){! if(map.get(s).iscorrupted()){! if(s.equals( Primary ))! printf( debug: %s\n,! map.get(s).tostring());! map.remove(s);! 53

54 ENSURING DOCUMENTATION QUALITY Evaluation: Human studies Study 1 (FIX) Humans identify and fix low-quality comments with and without our tool Study 2 (RATE) Using the resulting data set, have different humans identify and rate the modified comments 54

55 ENSURING DOCUMENTATION QUALITY Evaluation: Compare our tool s accuracy when identifying low-quality comments with humans abilities to do the same task Use identification data from both FIX and RATE Inter-annotator agreement vs. tool-human agreement 55

56 ENSURING DOCUMENTATION QUALITY Evaluation: Measure our tools effectiveness in helping humans identify and fix low quality comments Effort (time) in FIX Use ratings from RATE to compare data from groups in FIX 56

57 SUMMARY We propose work that will specifically target three parts of the maintenance process to reduce the overall cost: 1. Cluster automatically-generated defect reports to facilitate triage and bug fixing 2. Improve fitness functions to aid in automatic program repair to fix more bugs, faster 3. Identify incomplete and inconsistent comments to promote continued documentation quality and foster program understanding 57

58 COMPREHENSIVE GOALS - REVISITED We desire techniques that add minimal human effort Techniques work off the shelf Encourages incremental adoption Use latent, often-overlooked information Syntactic, semantic defect report fields Test case quality, types of bugs/fixes Natural language in code and comments 58

59 RESEARCH TIMELINE Patch&Quality&[ISSTA&'12]& Duplicate&Defect&Detec9on& 2011& 2012& 2013& 2014& today& expected& gradua9on& Enhanced&Fitness&Func9ons& Documenta9on&Quality& Op9onal&Documenta9on&Quality& Journal&Submission& Research&Period& Publica9on&Lag& Publications to date: E. Schulte, Z. Fry, E. Fast, W. Weimer, S. Forrest. Software Mutational Robustness. Genetic Programming and Evolvable Machines (under submission) Z. Fry, W. Weimer. Clustering Static Analysis Defect Reports to Reduce Maintenance Costs. International Conference on Tools and Algorithms for the Construction and Analysis of Systems 2013 (TACAS). (under submission) Z. Fry, B. Landau, W. Weimer. A Human Study of Patch Maintainability. International Symposium on Software Testing and Analysis 2012 (ISSTA). (Acc Rate: 29%) Z. Fry, W. Weimer. A Human Study of Fault Localization Accuracy. International Conference on Software Maintenance 2010 (ICSM). (Acc. Rate: 26%) 59

60 QUESTIONS? 60

Program Synthesis. SWE 795, Spring 2017 Software Engineering Environments

Program Synthesis. SWE 795, Spring 2017 Software Engineering Environments Program Synthesis SWE 795, Spring 2017 Software Engineering Environments Today HW3 is due next week in class! Part 1 (Lecture)(~50 mins) Break! Part 2 (Discussion)(~60 mins) Discussion of readings Part