Steven Davies Marc Roper Department of Computer and Information Sciences University of Strathclyde. International Workshop on Program Debugging, 2013

Transcription

1 1/22 Bug localisation through diverse sources of information Steven Davies Marc Roper Department of Computer and Information Sciences University of Strathclyde International Workshop on Program Debugging, 2013

2 2/22 Introduction Developers can spend up to 50% of their time on bug-fixing and maintenance, and often a large part is locating where the bug is Given a bug report, bug localisation tools present developers with recommendations for where the bug is likely to be located The aim is to reduce the time, and therefore the cost, spent on fixing bugs

3 Bug localisation 3/22

4 3/22 Bug localisation I just updated to pre 10 from pre 9 and my perl Syntax coloring is now messed up. It seems that escaped double quotes within double quoted strings are messing up my matches (and they weren t before). This is causing all kinds of colorization issues. How can I correct this (which file do I look at). and is there a simple way to put the perl mode back to pre9 and keep pre 10 installed? I have looked for the file and not had any luck. Thanks

5 3/22 Bug localisation ParserRule.ParserRule() ParserRule.createEOLSpanRule() ParserRule.createEscapeRule() ParserRule.createMarkFollowingRule() ParserRule.createMarkPreviousRule() ParserRule.createRegexpEOLSpanRule() ParserRule.createRegexpSequenceRule() ParserRule.createRegexpSpanRule() ParserRule.createSequenceRule() ParserRule.createSpanRule() ParserRule.toString() ParserRuleSet.setEscapeRule() TokenMarker.LineContext.LineContext()...

6 4/22 Bug localisation Existing bug localisation techniques are somewhat effective but often first relevant method shown is vastly outwith those a developer would realistically examine Still not widely used, and no existing implementations We aim to improve the performance of existing bug localisation techniques by adding multiple additional novel sources of information Previous bug fixes Buggy methods Stack traces Others in future

7 5/22 Existing source code approach First, each method in the system is extracted, including comments public LineContext marktokens(linecontext,tokenhandler,segment) //\{\{\{ Set up some instance variables // this is to avoid having to pass around lots and lots of // parameters. this. tokenhandler = tokenhandler ; this. line = line ; lastoffset = line. offset ; linelength = line. count + line. offset ; context = new LineContext () ;...

8 5/22 Existing source code approach Terms are split up on changes in case or punctuation, and common stop-words are removed TokenMarker.markTokens(LineContext,TokenHandler,Segment) around avoid context count handler having instance last length line lots new offset parameters pass set token up variables...

9

10

11 5/22 Existing source code approach The description is pre-processed in the same way and submitted as a query Bug accepted brackets curly documented handle html http like only operators org patch perl perldoc perlop quote round section slash without xml...

12 5/22 Existing source code approach The similarity of each method is calculated (Currently we use cosine similarity and TF-IDF) Bug accepted brackets curly documented handle html http like only operators org patch perl perldoc perlop quote round section slash without xml

13 5/22 Existing source code approach The methods are presented to the developer in order Bug XmlParser.setHandler(XmlHandler) 0.7 jedit.getinputhandler() 0.6 TokenMarker.markTokens(LineContext,TokenHandler,Segment

14 6/22 Additional sources of information In previous work, we used a classifier to partition methods that had been involved in previous similar bug fixes, before sorting them according to the source code approach We now aim to also add in information from: Buggy methods Stack traces None of these are widely applicable, and are probably less accurate than source code approach Combining multiple sources should lead to more accurate performance than any single source of information Each individually outputs a probability for each method of it being related to a particular bug

15 7/22 Previous bug fixes For each method in the system we train a classifier to answer a simple question: Is this new bug related to this method? Training is based on the text of the bugs related to the method, not the text of the method itself Classifiers are updated whenever new bugs are fixed, identified through commit comments

16 8/22 Previous bug fixes We create an empty classifier for each method in the system TokenMarker.markTokens(LineContext,TokenHandler,Segment) N

17 8/22 Previous bug fixes When bugs fixed, relevant methods updated with bug description Bug abnormal adjustments appears attachment basically behaviour calculated component correctly determined dialog display displayed... TokenMarker.markTokens(LineContext,TokenHandler,Segment) Y N

18 8/22 Previous bug fixes The classifiers grow as more bugs are fixed Bug after append appendrows autosizecolumn before bug causes code command ctrl data else example filename found getnumberrows... TokenMarker.markTokens(LineContext,TokenHandler,Segment) Y Y N

19 8/22 Previous bug fixes Classifiers are also updated with which bugs were not related Bug appears can changed class click clicking default list model delete double edit entered entries history text insert limited list model... TokenMarker.markTokens(LineContext,TokenHandler,Segment) Y N Y N

20 8/22 Previous bug fixes When bugs are reported description is submitted as a query Bug accepted brackets curly documented handle html http like only operators org patch perl perldoc perlop quote round section slash...

21 8/22 Previous bug fixes Each classifier predicts probability of bug being related to method Bug accepted brackets curly documented handle html http like only operators org patch perl perldoc perlop quote round section slash

22 9/22 Buggy methods Previous work has shown the methods with bugs in them are more likely to have bugs in future We exploit this by simply ranking methods which have had more bugs as being more likely to be related to the current bug Probability output is the number of bugs related to the method, divided by the total number of bugs to-date Note that this is not specific to a bug report: at any given time, the same values will be returned regardless of the contents of the bug report

23 10/22 Stack traces Work by other researchers has shown that when stack traces are given in bug reports, usually at least one method in the stack trace has to be changed to fix the bug Also, the higher up the stack trace, the more likely it is the method is involved Where bug reports have stack traces, we output a value based on the position of the method in the stack trace

24 11/22 Combining results Results are combined in a weighted linear manner Source Bug Count Stack Result Method 1 Method 2 Method 3 Method 4 Weight

25 11/22 Combining results When a bug is reported, values are obtained for each method Source Bug Count Stack Result Method Method Method Method Weight

26 11/22 Combining results These are multiplied by the weights and summed Source Bug Count Stack Result Method Method Method Method Weight

27 11/22 Combining results The process is repeated for other bugs Source Bug Count Stack Result Method Method Method Method Weight

28 11/22 Combining results The process is repeated for other bugs Source Bug Count Stack Result Method Method Method Method Weight

29 11/22 Combining results When a bug is fixed, the list of methods involved is obtained Source Bug Count Stack Result Method Method Method Method Weight

30 11/22 Combining results These are used to update the weights for future bugs Source Bug Count Stack Result Method Method Method Method Weight

31 12/22 Combining results In this manner, we should learn which sources of information for a project are most accurate Matches the way a tool would be used in practice Currently uses a linear regression, but this is unlikely to be the best technique

32 13/22 Evaluation Evaluated technique on three open-source projects: Ant: 925 bugs and methods JMeter: 160 bugs and 9444 methods JodaTime: 58 bugs and methods Measured the position of the first method presented in the list which was actually changed to fix the bug Compared the results to the original source code approach, as this performs better than any other individual approach, and is the most widely used in other research

33 14/22 Ant bug Summary: NPE in JDependTask if forked First relevant method returned by source code approach is org. apache.tools.ant.taskdefs.optional.jdepend. JDependTask.executeAsForked(CommandlineJava, ExecuteWatchdog) in position 52 Individually, the other sources of information put this method at positions 5658, 6779 and 2 When the information is combined however, the position of the method improves to 1

34 15/22 JMeter bug Summary: StatCalculator.addAll(StatCalculator<T> calc) joins incorrect if there are more samples with the same response time in one of the TreeMap First relevant method returned by source code approach is org. apache.jorphan.math.statcalculator.addvalue( Number,int) in position 1 However, as the other evaluators all rank this method much lower, the position of the method falls to 9703

35 16/22 Results In total for the combined approach 399 bugs improved their position over the text approach, by an average of 33 places Ant JMeter JodaTime 1000 Comb Text 10

36 16/22 Results 396 bugs got worse, but the amount they did so by was smaller, an average of 10 places Ant JMeter JodaTime 1000 Comb Text 10

37 17/22 Results Bugs where the first relevant method was in position 1 increased from 98 to Comb 1 Comb <=10 Comb <=100 Comb <=1000 Comb >1000 Text 1 Text <=10 Text <=100 Text <=1000 Text >1000

38 17/22 Results And in the top-10 from 271 to Comb 1 Comb <=10 Comb <=100 Comb <=1000 Comb >1000 Text 1 Text <=10 Text <=100 Text <=1000 Text >1000

39 17/22 Results Most of the improvement comes from Ant Ant JMeter JodaTime Comb 1 Comb <=10 Comb <=100 Comb <=1000 Comb >1000 Text 1 Text <=10 Text <=100 Text <=1000 Text >1000

40 17/22 Results Changes are less pronounced for JMeter, and no real improvement for JodaTime 0 JMeter JodaTime 150 Comb 1 Comb <=10 Comb <= Comb <=1000 Comb >1000 Text 1 50 Text <=10 Text <=100 Text <=1000 Text >1000

41 18/22 Results Also measured change in mean reciprocal rank (the inverse of the position of the first relevant method) Mean reciprocal rank rose for all 3 projects Text Comb Ant JMeter JodaTime

42 19/22 Future work: Other sources of information This work has only examined taking in four sources of information Design is extensible however, and there are multiple additional sources that could be used Test coverage Are methods with low test coverage more likely to contain bugs? Error logs Similar to stack traces Metrics & static analysis warnings

43 20/22 Future work: Combining sources This work combined sources of information using linear regression This is simple, and easy to interpret, aiding experimentation, but unlikely to be best method Data is not likely to be independent Exploring other methods of combining input, such as machine learning techniques

44 21/22 Future work: Implementation Overall aim is to implement a bug localisation tool Crucial to test whether this research is actually useful, and if assumptions are valid Proposed integration with bug tracking systems, to show developers a list of locations as they examine a bug Integrate with continuous integration systems, to update data whenever new bugs are fixed

45 22/22 Summary Used multiple sources of information to improve on textual bug localisation techniques Evaluated 1143 bugs from 3 projects Improved mean reciprocal rank for all projects Increased the number of bugs where the first relevant method is at position 1 from 98 to 132 Thank You Steven.Davies@cis.strath.ac.uk