Translating Code Comments to Procedure Specifications

Transcription

1 Translating Code Comments to Procedure Specifications Arianna Blasi Alberto Goffi Konstantin Kuznetsov Alessandra Gorla Michael D. Ernst Mauro Pezzè Sergio Delgado USI Università della Svizzera italiana, Switzerland IMDEA Software Institute, Spain Saarland University/CISPA, Germany University of Washington Seattle, WA, USA

2 public static boolean removeall(list mylist){ }

3 public static boolean removeall(list mylist){ }

4 public static boolean removeall(list mylist){ public void testremoveall(){ }

5 public static boolean removeall(list mylist){ public void testremoveall(){ List<Integer> list = Arrays.asList(1, 2, 3); }

6 public static boolean removeall(list mylist){ public void testremoveall(){ List<Integer> list = Arrays.asList(1, 2, 3); boolean result = removeall(list); }

7 public static boolean removeall(list mylist){ public void testremoveall(){ List<Integer> list = Arrays.asList(1, 2, 3); boolean result = removeall(list); assert? }

8 public static boolean removeall(list mylist){ public void testremoveall(){ List<Integer> list = Arrays.asList(1, 2, 3); boolean result = removeall(list); assert? } We have no idea

9

10

11 Where is the oracle?

12 Where is the oracle? Where is the specification?

13

14 Why not generating executable specifications automatically?

15 Why not generating executable specifications automatically? Let s rely on Javadoc comments!

16 Our solution /** true if the list is empty */ public static boolean removeall(list mylist) { }

17 Our solution /** true if the list is empty */ public static boolean removeall(list mylist) { } mylist.isempty()? result==true

18 Our solution /** true if the list is empty */ public static boolean removeall(list mylist) { } mylist.isempty()? result==true

19 Our solution /** true if the list is empty */ public static boolean removeall(list mylist) { } mylist.isempty()? public void testremoveall(){ List<Integer> list = Arrays.asList(1, 2, 3); boolean result = removeall(list); }

20 Our solution /** true if the list is empty */ public static boolean removeall(list mylist) { } mylist.isempty()? public void testremoveall(){ List<Integer> list = Arrays.asList(1, 2, 3); boolean result = removeall(list); if(list.isempty()) assert result==true; }

21 Our solution /** true if the list is empty */ public static boolean removeall(list mylist) { } mylist.isempty()? public void testremoveall(){ List<Integer> list = Arrays.asList(1, 2, 3); boolean result = removeall(list); if(list.isempty()) assert result==true; }

22 The idea is not new...

23 The idea is not Tan et al. ICST 2012

24 The idea is not Tan et al. ICST 2012 Toradocu Goffi et al. ISSTA 2016

25 The idea is not Tan et al. ICST 2012 Toradocu Goffi et al. ISSTA 2016 ALICS Pandita et al. ICSE 2012

26 The idea is not Tan et al. ICST 2012 Preconditions Normal Postconditions Exceptional Postconditions Toradocu Goffi et al. ISSTA 2016 ALICS Pandita et al. ICSE 2012

27 The idea is not Tan et al. ICST 2012 Preconditions Normal Postconditions Exceptional Postconditions Toradocu Goffi et al. ISSTA 2016 ALICS Pandita et al. ICSE 2012

28 The idea is not Tan et al. ICST 2012 Preconditions Normal Postconditions Exceptional Postconditions Toradocu Goffi et al. ISSTA 2016 ALICS Pandita et al. ICSE 2012 Jdoctor

29 Jdoctor Source Javadoc Extractor Comment Translator Specs

30 Source Javadoc Extractor Comment Translator Specs

31 Source Javadoc Extractor Comment Translator Specs Source 1. Expected result 2. Condition (Text)

32 Source Javadoc Extractor Comment Translator Specs Source 1. Expected result 2. Condition (Text) /** true * if the list is empty */ public boolean removeall(list list){ } 1. true 2. if the list is empty

33 Source Javadoc Extractor Comment Translator Specs 1. Expected result 2. Condition (Text) /** true * if the list is empty */ public boolean removeall(list list){ } 1. true 2. if the list is empty

34 Source Javadoc Extractor Comment Translator Specs 1. Expected result 2. Condition (Text)

35 Source Javadoc Extractor Comment Translator Specs 1. Expected result 2. Condition (Text) 1. Expected result 2. Condition (Code)

36 Source Javadoc Extractor Comment Translator Specs 1. Expected result 2. Condition (Text) 1. Expected result 2. Condition (Code) 1. true 2. if the list is empty 1. true 2. mylist.isempty()

37 Source Javadoc Extractor Comment Translator Specs 1. Expected result 2. Condition (Text) 1. Expected result 2. Condition (Code) 1. true 2. if the list is empty 1. true 2. mylist.isempty() mylist.isempty()? result==true

38 Natural Language Specification Comment Translator Java Specification

39 Natural Language Specification Comment Translator NL Parser Subject Matcher Predicate Matcher Java Specification

40 NL Parser Subject Matcher Predicate Matcher if the list is empty

41 NL Parser Subject Matcher Predicate Matcher if the list is empty Subject list Predicate is empty

42 NL Parser Subject Matcher Subject: list Predicate Matcher

43 NL Parser Subject Matcher Subject: list Predicate Matcher Candidates Formal Parameters Class Name Methods Fields

44 NL Parser Subject Matcher Predicate Matcher Subject: list Candidates Formal Parameters Class Name Methods Fields Candidate Distance mylist 2 isempty 7 iterator 8 size 4

45 NL Parser Subject Matcher Predicate Matcher Subject: list Candidates Formal Parameters Class Name Methods Fields Candidate Distance mylist 2 isempty 7 iterator 8 size 4

46 NL Parser Subject Matcher Predicate Matcher Subject: list => mylist Candidates Formal Parameters Class Name Methods Fields Candidate Distance mylist 2 isempty 7 iterator 8 size 4

47 NL Parser Subject Matcher Predicate Matcher Subject Predicate list is empty

48 NL Parser Subject Matcher Predicate Matcher Subject Predicate list is empty

49 NL Parser Subject Matcher Predicate Matcher Subject Predicate list is empty 1: Exact pattern match?

50 NL Parser Subject Matcher Predicate Matcher Subject Predicate list is empty is null == null 1: Exact pattern match?

51 NL Parser Subject Matcher Predicate Matcher Subject Predicate list is empty 1: Exact pattern match? is null == null is negative < 0

52 NL Parser Subject Matcher Predicate Matcher Subject Predicate list is empty 1: Exact pattern match? is null == null is negative < 0 is positive > 0

53 NL Parser Subject Matcher Predicate Matcher Subject Predicate list is empty 1: Exact pattern match? is null == null is negative < 0 is positive > 0...

54 NL Parser Subject Matcher Predicate Matcher Subject Predicate list is empty 1: Exact pattern Toradocu ALICS is null == null is negative < 0 is positive > 0...

55 NL Parser Subject Matcher Predicate Matcher Subject Predicate list is empty "is empty" not Toradocu ALICS is null == null is negative < 0 is positive > 0...

56 NL Parser Subject Matcher Predicate Matcher Subject Predicate list is empty

57 NL Parser Subject Matcher Predicate Matcher Subject Predicate list is empty 2: Syntax match? Toradocu

58 NL Parser Subject Matcher Predicate Matcher Subject Predicate list is empty Candidate Distance mylist 9 2: Syntax match? isempty 1 iterator 6 Toradocu size 7

59 NL Parser Subject Matcher Predicate Matcher Subject Predicate list is empty Candidate Distance 2: Syntax match? mylist 9 isempty 1 iterator 6 Toradocu size 7

60 Pattern and syntax match are not enough

61 Pattern and syntax match are not enough Return true if the list is empty

62 Pattern and syntax match are not enough Return true if the list is empty Candidate Distance clear 7 isempty 1 iterator 6 size 7

63 Pattern and syntax match are not enough Return true if the list is empty contains no element

64 Pattern and syntax match are not enough Return true if the list is empty contains no element has no content

65 Pattern and syntax match are not enough Return true if the list is empty contains no element has no content Candidate mylist isempty iterator size Distance

66 Pattern and syntax match are not enough Return true if the list is empty contains no element has no content Candidate mylist isempty iterator size Distance

67 Pattern and syntax match are not enough Return true if the list is empty contains no element has no content Candidate mylist isempty iterator size Toradocu ALICS

68 Analyzing complex comments /** * Checks if there is a credible threatening unit to * this unit within a range of moves. * moves list of moves True if a threat was found. */ public boolean isindanger(list<object> moves)

69 Analyzing complex comments /** * Checks if there is a credible threatening unit to * this unit within a range of moves. * moves list of moves True if a threat was found. */ public boolean isindanger(list<object> moves)

70 Analyzing complex comments /** * Checks if there is a credible threatening unit to * this unit within a range of moves. * moves list of moves True if a threat was found. */ public boolean isindanger(list<object> moves) searchfordanger(list<object>)

71 threat was found search for danger

72 threat was found search for danger

73

74 search is not displayed as a result

75 We search need more is not than displayed semantic as equivalence a result

76 Word embedding Word embedded in vector space: GloVe, Word2Vec man king woman queen

77 Word embedding Word embedded in vector space: GloVe, Word2Vec danger threat found search

78 Word embedding Word mover's distance (Matt J. Kusner et al., 2015) danger threat found search

79 Word embedding Word mover's distance (Matt J. Kusner et al., 2015) danger threat found search

80 Word Mover's distance Obama speaks in Illinois. In Chicago the president greets the press

81 Word Mover's distance Obama speaks in Illinois. In Chicago the president greets the press

82 Word Mover's distance Obama speaks in Illinois. In Chicago the president greets the press

83 Word Mover's distance Obama speaks in Illinois. In Chicago the president greets the press

84 Word Mover's distance Threat was found Search For Danger

85 Word Mover's distance Threat was found Search For Danger

86 Word Mover's distance Threat was found Search For Danger

87

88 How accurate is Jdoctor?

89 Experimental setup 6 Popular open-source Java systems

90 Experimental setup 563 Analyzed Java methods

91 Experimental setup 829 Manually-written Java conditions

92

93 92 % Precision

94 92 % 83 % Precision Recall

95 Exceptional Postconditions 100% Precision Recall 90% 80% 70% 60% 50% 40% 30% 20% 10% Toradocu Jdoctor

96 Preconditions 100% Precision Recall 90% 80% 70% 60% 50% 40% 30% 20% 10% Toradocu Jdoctor

97 Normal Postconditions 100% Precision Recall 90% 80% 70% 60% 50% 40% 30% 20% 10% Toradocu Jdoctor

98 Overall accuracy 100% Precision Recall 80% 60% 40% 20% Toradocu Jdoctor

99

100 How can you use Jdoctor?

101 Automatic Test Case Generator + Jdoctor

102 + Jdoctor

103 Randoop+Jdoctor reclassification passing passing failing invalid

104 Randoop+Jdoctor reclassification passing passing failing invalid

105 Randoop+Jdoctor reclassification +Jdoctor passing passing failing invalid

106 Randoop+Jdoctor reclassification +Jdoctor passing invalid passing failing invalid

107 Randoop+Jdoctor reclassification +Jdoctor passing invalid = invalid passing failing invalid

108 Randoop+Jdoctor reclassification +Jdoctor passing invalid = invalid passing failing failing invalid

109 Randoop+Jdoctor reclassification +Jdoctor passing invalid = invalid passing failing = missed alarm failing invalid

110 Randoop+Jdoctor reclassification +Jdoctor passing invalid = invalid passing failing = missed alarm failing passing invalid

111 Randoop+Jdoctor reclassification +Jdoctor passing invalid = invalid passing failing = missed alarm failing passing = false alarm invalid

112 Randoop+Jdoctor reclassification +Jdoctor passing invalid = invalid passing failing = missed alarm failing passing = false alarm invalid passing

113 Randoop+Jdoctor reclassification +Jdoctor passing invalid = invalid passing failing = missed alarm failing passing = false alarm invalid passing = new test

114 Missed alarm public static boolean removeall(list mylist){ }

115 Missed alarm public static boolean removeall(list mylist){ public void testremoveall(){ List<Integer> list = Arrays.asList(1, 2, 3); boolean result = removeall(list); }

116 Missed alarm /** true if the list is empty public */ static boolean removeall(list mylist){ } public static boolean removeall(list mylist){ } + public void testremoveall(){ List<Integer> list = Arrays.asList(1, 2, 3); boolean result = removeall(list); }

117 Missed alarm /** true if the list is empty public */ static boolean removeall(list mylist){ } public static boolean removeall(list mylist){ } + public void testremoveall(){ List<Integer> list = Arrays.asList(1, 2, 3); boolean result = removeall(list); if(list.isempty()) assert result==true; }

118 Missed alarm /** true if the list is empty public */ static boolean removeall(list mylist){ } public static boolean removeall(list mylist){ } + public void testremoveall(){ List<Integer> list = Arrays.asList(1, 2, 3); boolean result = removeall(list); if(list.isempty()) assert result==true; }

119 Invalid test public static float max(float[] floatarray){ }

120 Invalid test public static float max(float[] floatarray){ public void testremoveall(){ float[] floatarray0 = null; float float1 = max(floatarray0); }

121 Invalid test /** floatarray array of floats, not null */ public static float max(float[] floatarray){ } + public void testremoveall(){ float[] floatarray0 = null; float float1 = max(floatarray0); }

122 Invalid test /** floatarray array of floats, not null */ public static float max(float[] floatarray){ } + public void testremoveall(){ float[] floatarray0 = null; float float1 = max(floatarray0); } INVALID

123 Reclassification False alarm Missed alarm New test Invalid test

124 Jdoctor Source Javadoc Extractor Comment Translator Specs

125 Jdoctor Source Javadoc Extractor Comment Translator Specs Preconditions

126 Jdoctor Source Javadoc Extractor Comment Translator Specs Preconditions Normal Postconditions

127 Jdoctor Source Javadoc Extractor Comment Translator Specs Preconditions Normal Postconditions Exceptional Postconditions

128 Jdoctor Source Javadoc Extractor Comment Translator Specs Preconditions Pattern Match Normal Postconditions Exceptional Postconditions

129 Jdoctor Source Javadoc Extractor Comment Translator Specs Preconditions Pattern Match Normal Postconditions Syntax Match Exceptional Postconditions

130 Jdoctor Source Javadoc Extractor Comment Translator Specs Preconditions Pattern Match Normal Postconditions Exceptional Postconditions Syntax Match Semantic Match

131 Jdoctor Source Javadoc Extractor Comment Translator Specs Preconditions Pattern Match Normal Postconditions Exceptional Postconditions Syntax Match Semantic Match