Automatic Generation of Oracles for Exceptional Behaviors

Transcription

1 Automatic Generation of Oracles for Exceptional Behaviors ISSTA * * Artifact Consistent * Complete * Well Documented * Easy to Reuse * Evaluated * AEC * Alberto Goffi alberto.goffi@usi.ch USI Università della Svizzera italiana (Lugano, Switzerland) Alessandra Gorla IMDEA Software Institute (Madrid, Spain) Michael D. Ernst University of Washington (Seattle, Washington, USA) Mauro Pezzè USI Università della Svizzera italiana (Lugano, Switzerland)

2 The Oracle Problem Automatically generated test cases need oracles Formal specifications are often missing Developers write Javadoc comments Automatically generate oracles from Javadoc comments

3 The Oracle Problem Automatically generated test cases need oracles Developers do not write formal specifications Developers write Javadoc comments Automatically generate oracles from Javadoc comments

4 The Oracle Problem Automatically generated test cases need oracles Developers do not write formal specifications Developers do write Javadoc comments Automatically generate oracles from Javadoc comments

5 The Oracle Problem Automatically generated test cases need oracles Developers do not write formal specifications Developers do write Javadoc comments Automatically generate oracles from Javadoc comments

6 Testers under-test exceptions 100 % covered statements % covered throw statements cli collections compress dbcp io lang math net pool primitives guava average

7 Testers under-test exceptions 100 % covered statements % covered throw statements cli collections compress dbcp io lang math net pool primitives guava average

8 Testers under-test exceptions 100 % covered statements % covered throw statements cli collections compress dbcp io lang math net pool primitives guava average

9 Testers under-test exceptions % covered statements % covered throw statements -25% cli collections compress dbcp io lang math net pool primitives guava average

10 and automatic testing does not help public class FilterIterator implements Iterator { public FilterIterator(Iterator i, Predicate p) { } public Object next() { } }

11 and automatic testing does not help public class FilterIterator implements Iterator { public FilterIterator(Iterator i, Predicate p) { } public Object next() { } } public void test { FilterIterator i = new FilterIterator(null, null); i.next(); }

12 What does an exception mean? public class FilterIterator implements Iterator { public FilterIterator(Iterator i, Predicate p) { } public Object next() { } } public void test { FilterIterator i = new FilterIterator(null, null); i.next(); } NullPointerException

13 What does an exception mean? public class FilterIterator implements Iterator { public FilterIterator(Iterator i, Predicate p) { } public Object next() { } } public void test { FilterIterator i = new FilterIterator(null, null); i.next(); } NullPointerException 1. Expected behavior? 2. Illegal input 3. Implementation bug

14 What does an exception mean? public class FilterIterator implements Iterator { public FilterIterator(Iterator i, Predicate p) { } } /** NullPointerException if either * the iterator or predicate are null */ public Object next() { } public void test { FilterIterator i = new FilterIterator(null, null); i.next(); } NullPointerException 1. Expected behavior? 2. Illegal input 3. Implementation bug

15 What does an exception mean? public class FilterIterator implements Iterator { public FilterIterator(Iterator i, Predicate p) { } } /** NullPointerException if either * the iterator or predicate are null */ public Object next() { } public void test { FilterIterator i = new FilterIterator(null, null); i.next(); } NullPointerException 1. Expected behavior 2. Illegal input 3. Implementation bug

16 What does an exception mean? public class FilterIterator implements Iterator { public FilterIterator(Iterator i, Predicate p) { } } /** NullPointerException if either * the iterator or predicate are null */ public Object next() { } False Alarm public void test { FilterIterator i = new FilterIterator(null, null); i.next(); } NullPointerException 1. Expected behavior 2. Illegal input 3. Implementation bug

17 Consider another implementation public class FilterIterator implements Iterator { public FilterIterator(Iterator i, Predicate p) { } } /** NullPointerException if either * the iterator or predicate are null */ public Object next() { } public void test { FilterIterator i = new FilterIterator(null, null); i.next(); } No Exceptions 1. Expected behavior bug? 2. Illegal input 3. Implementation

18 Consider another implementation public class FilterIterator implements Iterator { public FilterIterator(Iterator i, Predicate p) { } } /** NullPointerException if either * the iterator or predicate are null */ public Object next() { } public void test { FilterIterator i = new FilterIterator(null, null); i.next(); } No Exceptions 1. Expected behavior 2. Illegal input 3. Implementation bug

19 Consider another implementation public class FilterIterator implements Iterator { public FilterIterator(Iterator i, Predicate p) { } } /** NullPointerException if either * the iterator or predicate are null */ public Object next() { } Missed Alarm public void test { FilterIterator i = new FilterIterator(null, null); i.next(); } No Exceptions 1. Expected behavior 2. Illegal input 3. Implementation bug

20 Test public void test { FilterIterator i = new FilterIterator(null, null); i.next(); }

21 Test Documentation public void test { FilterIterator i = new FilterIterator(null, null); i.next(); } /** NullPointerException if either * the iterator or predicate are null */ public Object next() { }

22 Test + Documentation public void test { FilterIterator i = new FilterIterator(null, null); i.next(); } /** NullPointerException if either * the iterator or predicate are null */ public Object next() { }

23 Test + Documentation public void test { FilterIterator i = new FilterIterator(null, null); i.next(); } /** NullPointerException if either * the iterator or predicate are null */ public Object next() { } Test + Oracle public void test { FilterIterator i = new FilterIterator(null, null); if (i.getiterator() == null i.getpredicate() == null) { try { i.next(); fail( Expected NPE not thrown ); } catch(nullpointerexception e) { // Expected exception } } else { i.next(); } }

24 Test + Documentation public void test { FilterIterator i = new FilterIterator(null, null); i.next(); } /** NullPointerException if either * the iterator or predicate are null */ public Object next() { } Test + Oracle public void test { FilterIterator i = new FilterIterator(null, null); if (i.getiterator() == null i.getpredicate() == null) { try { i.next(); fail( Expected NPE not thrown ); } catch(nullpointerexception e) { // Expected exception } } else { i.next(); } }

25 Test + Documentation public void test { FilterIterator i = new FilterIterator(null, null); i.next(); } /** NullPointerException if either * the iterator or predicate are null */ public Object next() { } Test + Oracle public void test { FilterIterator i = new FilterIterator(null, null); if (i.getiterator() == null i.getpredicate() == null) { try { i.next(); fail( Expected NPE not thrown ); } catch(nullpointerexception e) { // Expected exception } } else { i.next(); } }

26 Toradocu Source Javadoc Extractor Condition Translator Oracle Generator Enhanced Tests Tests

27 Source Javadoc Extractor Condition Translator Oracle Generator Enhanced Tests

28 Source Javadoc Extractor Condition Translator Oracle Generator Enhanced Tests Source 1. Method Signature 2. Expected Exceptions 3. Conditions (Text)

29 Source Javadoc Extractor Condition Translator Oracle Generator Enhanced Tests Source 1. Method Signature 2. Expected Exceptions 3. Conditions (Text) /** NullPointerException * if either the iterator * or predicate are null */ public Object next() { } 1. FilterIterator.next() 2. NullPointerException 3. if either the iterator or predicate are null

30 Source Javadoc Extractor Condition Translator Oracle Generator Enhanced Tests 1. Method Signature 2. Expected Exceptions 3. Conditions (Text) 1. Method Signature 2. Expected Exceptions 3. Conditions (Code)

31 Source Javadoc Extractor Condition Translator Oracle Generator Enhanced Tests 1. Method Signature 2. Expected Exceptions 3. Conditions (Text) 1. Method Signature 2. Expected Exceptions 3. Conditions (Code) 1. FilterIterator.next() 2. NullPointerException 3. if either the iterator or predicate are null 1. FilterIterator.next() 2. NullPointerException 3. getiterator() == null getpredicate() == null

32 Source Javadoc Extractor Condition Translator Oracle Generator Enhanced Tests Tests 1. FilterIterator.next() 2. NullPointerException 3. getiterator() == null getpredicate() == null

33 Source Javadoc Extractor Condition Translator Oracle Generator Enhanced Tests Tests Test + Oracle = Test + Oracle public void test { FilterIterator i = new FilterIterator(null, null); i.next(); } 1. FilterIterator.next() 2. NullPointerException 3. getiterator() == null getpredicate() == null public void test { FilterIterator i = new FilterIterator(null, null); if (i.getiterator()==null i.getpredicate()==null) { try { i.next(); fail( Expected NPE not thrown ); } catch(nullpointerexception e) { // Expected exception } } else { i.next(); } }

34 Natural Language Condition Source Javadoc Extractor Condition Translator Oracle Generator Enhanced Tests Java boolean condition

35 if the array is empty Source Javadoc Extractor Condition Translator Oracle Generator Enhanced Tests array.length == 0

36 if foo is positive Source Javadoc Extractor Condition Translator Oracle Generator Enhanced Tests foo > 0

37 Natural Language Condition Condition Translator NL Parser Subject Matcher Predicate Matcher Java Condition

38 NL Parser Subject Matcher Predicate Matcher if either the iterator or predicate are null

39 NL Parser Subject Matcher Predicate Matcher if either the iterator or predicate are null Proposition Subject Predicate 1 iterator are null 2 predicate are null

40 NL Parser Subject Matcher Predicate Matcher if either the iterator or predicate are null Proposition Subject Predicate 1 iterator are null 2 predicate are null

41 NL Parser Subject Matcher Predicate Matcher if either the iterator or predicate are null

42 NL Parser Subject Matcher Predicate Matcher if either the iterator or predicate are null or Proposition Subject Predicate 1 iterator are null 2 predicate are null iterator, are null predicate, are null

43 NL Parser Subject Matcher Predicate Matcher Proposition Subject Predicate 1 iterator are null 2 predicate are null

44 NL Parser Subject Matcher Subject: iterator Predicate Matcher

45 NL Parser Subject Matcher Subject: iterator Predicate Matcher Candidates Formal Parameters Class Name Non-void Nullary Methods Fields

46 NL Parser Subject Matcher Predicate Matcher Subject: iterator Candidates Formal Parameters Class Name Non-void Nullary Methods Fields Candidate FilterIterator getiterator getpredicate next Distance public class FilterIterator implements Iterator { public FilterIterator(Iterator i, Predicate p) { } public Iterator getiterator() { } public Predicate getpredicate() { } public Object next() { } }

47 NL Parser Subject Matcher Predicate Matcher Subject: iterator Candidates Formal Parameters Class Name Non-void Nullary Methods Fields Candidate Distance FilterIterator 6 getiterator 3 getpredicate 9 next 6

48 NL Parser Subject Matcher Predicate Matcher Subject: iterator => getiterator() Candidates Formal Parameters Class Name Non-void Nullary Methods Fields Candidate Distance FilterIterator 6 getiterator 3 getpredicate 9 next 6

49 NL Parser Subject Matcher Predicate Matcher Proposition Subject Predicate 1 iterator are null 2 predicate are null Proposition Subject Predicate 1 FilterIterator.getIterator() are null 2 FilterIterator.getPredicate() are null

50 NL Parser Subject Matcher Predicate Matcher Proposition Subject Predicate 1 FilterIterator.getIterator() are null 2 FilterIterator.getPredicate() are null

51 NL Parser Subject Matcher Predicate Matcher Proposition Subject Predicate 1 FilterIterator.getIterator() are null 2 FilterIterator.getPredicate() are null Proposition Subject Predicate 1 FilterIterator.getIterator() ==null 2 FilterIterator.getPredicate() ==null

52 NL Parser Subject Matcher Predicate Matcher /** IllegalArgumentException if employees is empty */ public void sortbyage(list employees) { } Proposition Subject Predicate 1 employees is empty

53 NL Parser Subject Matcher Predicate Matcher /** IllegalArgumentException if employees is empty */ public void sortbyage(list employees) { } Proposition Subject Predicate 1 employees is empty Candidate Distance clear 7 isempty 1 iterator 6 size 7

54 NL Parser Subject Matcher Predicate Matcher /** IllegalArgumentException if employees is empty */ public void sortbyage(list employees) { } Proposition Subject Predicate 1 employees is empty Condition Translation 1 employees.isempty()

55 NL Parser Subject Matcher Predicate Matcher if either the iterator or predicate are null Condition Translation 1 FilterIterator.getIterator()==null 2 FilterIterator.getPredicate()==null

56 Positive conditions if either the iterator or predicate are null Condition Translation 1 (FilterIterator.getIterator()==null)==true 2 (FilterIterator.getPredicate()==null)==true

57 Positive conditions if either the iterator or predicate are null Condition Translation 1 (FilterIterator.getIterator()==null)==true 2 (FilterIterator.getPredicate()==null)==true Negative conditions if either the iterator or predicate aren t null Condition Translation 1 (FilterIterator.getIterator()==null)==false 2 (FilterIterator.getPredicate()==null)==false

58 Evaluation

59 From NL to Java Conditions Correct Partially Correct Wrong Missing 100% 75% 50% 25% 0% ArrayListMultimap AtomicDoubleArray ConcurrentHashMultiset Doubles Floats MoreObjects Shorts Strings Verify FilterIterator FixedOrderComparator ListPopulation PredicatedMap

60 From NL to Java Conditions Correct Partially Correct Wrong Missing 100% 75% 50% 25% 0% ArrayListMultimap AtomicDoubleArray ConcurrentHashMultiset Doubles Floats MoreObjects Shorts Strings Verify FilterIterator FixedOrderComparator ListPopulation PredicatedMap

61 From NL to Java Conditions Correct Partially Correct Wrong Missing 100% 75% 50% 25% Precision: 92% 0% ArrayListMultimap AtomicDoubleArray ConcurrentHashMultiset Recall: 73% Doubles Floats MoreObjects Shorts Strings Verify FilterIterator FixedOrderComparator ListPopulation PredicatedMap

62 False Alarms Google Guava Implicit oracle Toradocu ArrayListMultimap AtomicDoubleArray ConcurrentHashMultiset Doubles Floats MoreObjects Shorts Strings Verify Average (EvoSuite Test Cases)

63 False Alarms Google Guava Implicit oracle Toradocu % ArrayListMultimap AtomicDoubleArray ConcurrentHashMultiset Doubles Floats MoreObjects Shorts Strings Verify Average (EvoSuite Test Cases)

64 True Alarms Apache Commons Pass False alarms True alarms Implicit Toradocu 0% 20% 40% 60% 80% 100%

65 True Alarms Apache Commons Pass False alarms True alarms Implicit Toradocu 0% 20% 40% 60% 80% 100%

66 True Alarms Apache Commons Pass False alarms True alarms Implicit Toradocu 0% 20% 40% 60% 80% 100% Regression Toradocu 0% 20% 40% 60% 80% 100%

67 True Alarms Apache Commons Pass False alarms True alarms Implicit Toradocu 0% 20% 40% 60% 80% 100% Regression Toradocu 0% 20% 40% 60% 80% 100%

68 True Alarms Apache Commons Pass False alarms True alarms Implicit Toradocu 0% 20% 40% 60% 80% 100% Regression Toradocu 0% 20% 40% 60% 80% 100% Heuristics Toradocu 0% 20% 40% 60% 80% 100%

69 True Alarms Apache Commons Pass False alarms True alarms Implicit Toradocu 0% 20% 40% 60% 80% 100% Regression Toradocu 0% 20% 40% 60% 80% 100% Heuristics Toradocu 0% 20% 40% 60% 80% 100%

70 What s next? Oracles for exceptions on arrays/containers IllegalArgumentException if the array contains null input foo must not be null void sort(@notnull int[] a) return the size of the array more

71 What s next? Oracles for exceptions on arrays/containers IllegalArgumentException if the array contains null input foo must not be null void sort(@notnull int[] a) return the size of the array more

72 Testers under-test exceptions % covered statements % covered throw statements -25% cli collections compress dbcp io lang math net pool primitives guava average

73 Testers under-test exceptions % covered statements % covered throw statements % Source Javadoc Extractor Toradocu Condition Translator Oracle Generator Enhanced Tests 20 Tests 0 cli collections compress dbcp io lang math net pool primitives guava average

74 Testers under-test exceptions % covered statements % covered throw statements % Toradocu Source Javadoc Extractor Condition Translator Oracle Generator Enhanced Tests 20 Tests 0 cli collections compress dbcp io lang math net pool primitives guava average From NL to Java Conditions Correct Partially Correct Wrong Missing 100% 75% 50% 25% 0% ArrayListMultimap AtomicDoubleArray Precision: 92% ConcurrentHashMultiset Recall: 73% Doubles Floats MoreObjects Shorts Strings Verify FilterIterator FixedOrderComparator ListPopulation PredicatedMap

75 Testers under-test exceptions % covered statements % covered throw statements % Source Javadoc Extractor Toradocu Condition Translator Oracle Generator Enhanced Tests 20 Tests 0 cli collections compress dbcp io lang math net pool primitives guava average From NL to Java Conditions 100% 75% 50% 25% 0% Correct Partially Correct Wrong Missing ArrayListMultimap AtomicDoubleArray Precision: 92% ConcurrentHashMultiset Recall: 73% 0 Doubles Floats MoreObjects Shorts Strings Verify FilterIterator FixedOrderComparator ListPopulation PredicatedMap False Alarms Google Guava ArrayListMultimap AtomicDoubleArray Implicit oracle ConcurrentHashMultiset Doubles Floats MoreObjects Shorts Toradocu -36% Strings Verify Average (EvoSuite Test Cases)

76 Testers under-test exceptions % covered statements % covered throw statements % Source Javadoc Extractor Toradocu Condition Translator Oracle Generator Enhanced Tests 20 Tests 0 cli collections compress dbcp io lang math net pool primitives guava average github.com/albertogoffi/toradocu From NL to Java Conditions 100% 75% 50% 25% 0% Correct Partially Correct Wrong Missing ArrayListMultimap AtomicDoubleArray Precision: 92% ConcurrentHashMultiset Recall: 73% 0 Doubles Floats MoreObjects alberto.goffi@usi.ch Shorts Strings Verify FilterIterator FixedOrderComparator ListPopulation PredicatedMap False Alarms Google Guava ArrayListMultimap AtomicDoubleArray Implicit oracle ConcurrentHashMultiset Doubles Floats MoreObjects Shorts Toradocu -36% Strings Verify Average (EvoSuite Test Cases)

77

78 Related Work Heuristics JCrasher, Crash n Check (Csallner, and Smaragdakis. ICSE 05) Randoop (Pacheco, Lahiri, Ernst, and Ball. ICSE 07) Specifications ASTOOT (Doong, and Frankl. TOSEM 94) Models, contracts, Properties Cross-checking oracles (Carzaniga, Goffi, Gorla, Mattavelli, and Pezzè. ICSE 14) Metamorphic testing (Chen, Kuo, Tse, and Zhou. STEP 13) Symmetric testing (Gotlieb. ISSRE 03) Natural Language (Tan, Marinov, Tan, and Leavens. ICST 12) acomment (Tan, Zhou, and Padioleau. ICSE 11) icomment (Tan, Yuan, Krishna, and Zhou. SOSP 07)

79 When does Toradocu fail? Conditions on the elements in a container - if collection or any of its elements is null - if any element in the arrays is null - if notes has fewer than 2 elements Specific checks - if the runtime type of the specified array is not a supertype of the runtime type of every element in this collection - if x > 0 Missing predicate translation (No output) - if the charset is not valid - if closed

80 NullPointerException if the predicate or either closure is null ifclosure(predicate predicate, Closure trueclosure, Closure falseclosure) { } ==> predicate == null trueclosure == null falseclosure == UnsupportedOperationException if the comparator is locked checklocked() { } ==> target.islocked()