Predicting Bugs. by Analyzing History. Sunghun Kim Research On Program Analysis System Seoul National University

Transcription

1 Predicting Bugs by Analyzing History Sunghun Kim Research On Program Analysis System Seoul National University

2 Around the World in 80 days

3 Around the World in 8 years

4 Predicting Bugs Severe consequences July 28, 1962: Mariner I space probe 1982 : Soviet gas pipeline 1985 ~ 1987: Therac-25 medical accelerator June 4, 1996: Ariane 5 Flight 501

5 Predicting Bugs Severe consequences July 28, Mariner I space probe Soviet gas pipeline Therac-25 medical accelerato June 4, Ariane 5 Flight 501 The Ariane5 exploded seconds after launching. Boring and labor intensive work

6 Analyzing History "Those who cannot learn from history are doomed to repeat it." - George Santayana

7 Analyzing History "Developers who cannot learn from history are doomed to repeat it." - George Santayana

8 Analyzing History "Developers who cannot learn from Software history are doomed to repeat it." - George Santayana

9 Available Information Bug Prediction Feedback Change Impact Analysis Resource Allocation Produce Software Understanding Models s Tests Raw data Bug reports Mine History Information

10 Dream All modules are bug-free! Bug-free module

11 Some are Buggy Some modules are buggy! Bug-free module Buggy module

12 Changes introduce Bugs Often changes introduce bugs! change Bug-free module Buggy module

13 Two Bug Prediction Algorithms Change Classification: predicting if a change introduces a bug change Bug cache: predicting buggy modules Bug-free module Buggy module

14 Change Classification [TSE08, the featured article of March/April issue]

15 Change Classification Development history of JEditTextArea.java Rev 1 Rev 2 Rev 3 change change

16 Change Classification Development history of JEditTextArea.java Rev 1 Rev 2 Rev 3 Rev 4 change change change

17 Change Classification Development history of JEditTextArea.java Rev 1 Rev 2 Rev 3 Rev 4 change change change Did I just introduce a bug?

18 What to do when a change is likely to introduce a bug? Review the submitted code carefully The submitted code change is fresh Focus additional software quality assurance (QA) efforts on those changes Software inspections Additional test cases

19 Change Classification Rev n Rev n+1 Rev n Rev n+1 Rev n Rev n+1 Rev n Rev n+1 buggy clean buggy clean buggy Rev n Rev n+1? Machine learner clean It classifies all changes (as buggy or clean) with 70% recall and 94% precision.

20 Label Historical Changes [MSR06, ASE06] Rev 1 Rev 100 change Rev 101 change Rev 102 Development history of JEditTextArea.java

21 Label Historical Changes Rev 101 (with BUG) settext( \t ) fixed Rev 102 (no BUG) inserttab() Rev 1 Rev 100 Rev 101 Rev 102 change change Development history of JEditTextArea.java

22 Label Historical Changes Change message: fix for bug Rev 101 (with BUG) settext( \t ) fixed Rev 102 (no BUG) inserttab() Rev 1 Rev 100 Rev 101 Rev 102 change change Development history of JEditTextArea.java

23 Label Historical Changes Rev 101 (with BUG) settext( \t ) fixed Rev 102 (no BUG) inserttab() Rev 1 Rev 100 Rev 101 Rev 102 change change Development history of JEditTextArea.java

24 Tracking Line Changes [MSR05] 1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: Rev12 1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: Rev23 1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: Rev42 1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: Rev101 3: 4: 5: 10: 11: 12: 7: 8: 10: CHG CHG CHG DEL ADD 1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: Rev12 Rev12 Rev23 Rev23 Rev23 Rev12 Rev12 Rev12, 23 Rev12, 23 Rev12, 23, 42 Rev12 Rev12 Rev12 Rev12 Line origins

25 Label Historical Changes Line origins Rev 101 (with BUG) Rev 102 (no BUG) Buggy change Rev 23 Rev 12 settext( \t ) fixed inserttab() Rev 1 Rev 100 Rev 101 Rev 102 change change Development history of JEditTextArea.java

26 Label Historical Changes Rev 22 Rev 23 Rev 24 Rev 25 Rev 26 buggy clean clean buggy

27 Extracting Features JEditTextArea.java Revision 10 change Revision 11 Author: hunkim Check-in time: March 23, :30 AM Log: Never convert propresult from utf-16

28 Training Classifiers Historical changes Machine learning techniques Bayesian Network, SVM

29 Classifying New Changes Historical changes New change prediction

30 Evaluation Training and testing sets 10-fold cross-validation Performance measurement Precision Recall

31 Training and Testing Sets Training Testing real prediction

32 Performance Measurement 4 possible outcomes from using a classifier Real Prediction Buggy Clean Buggy n b b n b c Clean n c b n c c Precision: n b b, Recall: n b b + n c b n b b n b b + n b c

33 Subject Systems Bugzilla jedit PostgreSQL Mozilla and more

34 Bug Prediction Accuracy (Bayesian Network after feature selection) A1 BUG COL GAI GFO JED MOZ ECL PLO POS SCA SVN Bug Recall Bug Precision

35 Related Work Mizuno et al. [FSE07] Aversano et al. [IWPSE07] ChangeClassfication Recall Precision

36 Related Work Mizuno et al. [FSE07] Aversano et al. [IWPSE07] ChangeClassfication Recall Precision

37 Change Classification Summary Use machine learning techniques to analyze changes After training, can predict whether a new change has a bug, or doesn t have a bug Average recall is 70% Average precision is 94% Applicable in real development process Yahoo and Apple are interested

38 Bug Cache [ ICSE07, Distinguished paper ]

39 Bug Cache Bug cache: locating buggy modules Bug-free module Buggy module

40 Motivation Which files should we focus on?

41 Where are the Bugs? In new files! [Graves et al.] In modified files! [Khoshgoftaar et al.] Spatial locality: nearby other bugs! [Zimmermann et al.] Temporal locality: Defected files are likely to have more soon. [Ostrand, Hassan]

42 Our Solution List of most bug-prone files Dynamically adaptive and intuitive Combine bug prediction models 10% BugCache predicts 73~95% of bugs

43 Bug Cache Model

44 Cache Model C Cache size: 2 A B C Miss

45 Cache Update Load missed files Pre-fetch nearby files (spatial locality) File A B D Number of common changes with C

46 Cache Model C B A Cache size: 2 Block size: 2 A B C C A B B D A Miss Hit Hit Miss Which one should be replaced?

47 Replacement Policies Least recently used (LRU) Unload the files that have the least recently found defect. Least frequently changed (CHANGE) Unload the files that have the fewest changes. Least frequent defects (BUG) Unload the files that have the fewest defects.

48 File C B BUG 2 1 (replace) 1 C B A Cache size: 2 Block size: 2 Replacement: BUG A B C C A B B D A Miss Hit Hit Miss

49 Cache Evaluation C A Cache size: 2 Block size: 2 Replacement: BUG A B C C A B B D A Miss Hit Hit Miss Hit rate = #Hits / #Bugs = 50%

50 Hit Rates Apache 1.3 Columba Eclipse JEdit Mozilla PostgreSQL Subversion Cache size = 10% of all files File

51 Hit Rates Apache 1.3 Columba Eclipse 95 JEdit Mozilla PostgreSQL 79 Subversion Cache size = 10% of all files File

52 Related Work Khoshgoftaar et al. Top 10% 64% Khoshgoftaar et al. Top 20% Ostrand et al. Top 20% Hassan et al. Top 10% 82% 71~93% 44~78% BugCache. Top 10%

53 Related Work Khoshgoftaar et al. Top 10% 64% Khoshgoftaar et al. Top 20% Ostrand et al. Top 20% Hassan et al. Top 10% BugCache. Top 10% 82% 71~93% 44~78% 73~95%

54 Related Work Khoshgoftaar et al. Top 10% Khoshgoftaar et al. Top 20% Ostrand et al. Top 20% Hassan et al. Top 10% Previous State of the Art, 10% predicts 44%~78% 20% predicts 71~93% 10% BugCache predicts 73~95% 64% 82% 71~93% 44~78% BugCache. Top 10% 73~95%

55 Conclusion Analyzing history is an effective way to predict bug locations Change classification can classify changes as buggy or clean with very good accuracy BugCache can identify the most bugprone files

56 Research Overview and Future Work

57 Research Goal Developer productivity Reliable software Predicting bugs

58 Research Overview History Mining Kenyon [FSE05] Bug Introducing Changes [ASE06] Prioritization of Warnings [FSE07] Static Software Understanding Signature Change Patterns [ICSM06] Micro Pattern Evolution [MSR06] Matching Name Changes [WCRE06] Bug Prediction Memories of Bug Fixes [FSE06] Change Classification [TSE08] Bug Cache [ICSE07] Dynamic Dynamic Monitoring ReCrash [ECOOP08] Zero-day patch

59 Future Work Change classification aware repository Personal coding assistance Mining bug and crash reports Mining APIs Combining complementary techniques Micro commits and explicit fix-change marks Change feedback to developers Adaptive change classification Mining common error patterns in my code Showing code survival rates Identifying/predicting crashed methods Predicting locations based on bug reports Increasing bug report quality Example oriented API documents Identifying more/less error prone APIs Automatic API version upgrading To find bugs effectively and efficiently Static and Dynamic (ReCrash + BugCache)

60 Static and Dynamic Analysis Overhead False positives Static Low High Dynamic High Low Combine static and dynamic analysis BugCache + ReCrash

61 Reproducing Crashes Reproducing crashes (faults) is hard! Require the exact configuration of crash (in field) Crashes usually involve nondeterministic facts Must be able to reproduce crashes to fix bugs and validate fixes

62 ReCrash [ECOOP08] Subject program ReCrash Test case 13-64% performance overhead

63 ReCrash + BugCache ReCrash Monitoring all modules

64 ReCrash + BugCache ReCrash Monitoring all modules BugCache Identify crashable modules

65 ReCrash + BugCache ReCrash Monitoring all modules BugCache Identify crashable modules ReCrash + BugCache Monitoring only identified modules

66 ReCrash + BugCache ReCrash Monitoring all modules BugCache Identify crashable modules ReCrash + BugCache Monitoring only identified modules 10% of modules account for 70% of crashes ReCrash + BugCache can possibly run with 1~6% overhead reproduce 70% of crashes

67 Acknowledgement Advisors Jim Whitehead (UCSC), Michael Ernst (MIT) Collaborators (co-authors) MIT Shay Artzi Adam Kiezun Danny Dig UCSC Guozhang Ge Yi Zhang Kai Pan Ramak Akella Jen Bevan Elias Sinderson Saarland U Andreas Zeller Nicolas Bettenburg Rahul Premraj Iowa Tien Nguyen Hojun Jaygarl Sean Chen (Taiwan) Canada Europe Industry / UW Tom Zimmermann (U. Calgary) Michael Godfrey (Waterloo) Ahmed Hassan (Queen's U) Tudor Girba (SW-ENG) Martin Pinzger (U. Zurich) Audris Mockus (Avaya) Shiv Shivaji (Yahoo) Miryung Kim (UW)

68 Summary

69

70 Predicting Bugs by Analyzing History Sunghun Kim Research On Program Analysis System Seoul National University