Shin Hwei Tan Darko Marinov Lin Tan Gary T. Leavens University of Illinois. University of Illinois

Transcription

1 Shin Hwei Tan Darko Marinov Lin Tan Gary T. Leavens University of Illinois University of Illinois University of Waterloo University of Central Florida 1

2 /* * Returns a synchronized map backed by the given map. map the map to synchronize, must not be null a synchronized map backed by the given map IllegalArgument if the map is null */ static <K,V> Map<K,V> synchronizedmap(map<k,v> map) Overall Description Block - Parameter name, - - name, Condition under which the exception is thrown 2

3 Writing Javadoc comments is a common practice Lots of Javadoc comments exist in Java libraries. Number of Number of Javadoc Ratio Project methods comments for methods (%) Collections 3,874 2, GlazedLists 2,753 1, JFreeChart 6,205 6, JodaTime 3,887 2, Log4j 2, Lucene 5,222 2, Xalan 5,404 3,

4 /* map the map to synchronize, must not be null a synchronized map backed by the given map IllegalArgument if the map is null */ static <K,V> Map<K,V> synchronizedmap(map<k,v> map) Javadoc comments can be inconsistent with code Expected behavior for synchronizedmap (null): Throws IllegalArgument Actual behavior: Throws NullPointer 4

5 Important to find bugs in comments Comments are not executed Comments are read by developers to understand code Incorrect comments could cause developers to write wrong code [SOSP 07] Challenging to automatically analyze comments Ambiguities in understanding general text NLP made progress but struggles with general text 5

6 Build domain-specific analyses icomment (locking protocols & function calls) [SOSP 07] acomment (interrupts) [ICSE 11] System code (C/C++) Extract machine-checkable rules from comments Use static analysis to check consistency between code and comments/rules 6

7 Javadoc Comments Java Tests with Inconsistency New Domain Method properties for null values and related exceptions Dynamic Analysis (Random Testing) Fewer false alarms compared to static analysis Improved Testing Reduce false alarms in test generation tool Evaluation on 7 Libraries Found 28 inconsistencies, 12 were fixed 7

8 /* map the map to synchronize, must not be null a synchronized map backed by the given map IllegalArgument if the map is null */ static <K,V> Map<K,V> synchronizedmap(map<k,v> map) 8

9 /* map the map to synchronize, must not be null a synchronized map backed by the given map IllegalArgument if the map is null */ static <K,V> Map<K,V> synchronizedmap(map<k,v> map) 9

10 /* map the map to synchronize, must not be null a synchronized map backed by the given map IllegalArgument if the map is null */ static <K,V> Map<K,V> synchronizedmap(map<k,v> map) public void test1() throws Throwable { java.util.map var0 = null; try { java.util.map var1=...synchronizedmap(var0); } catch (IllegalArgument expected) {return;} fail("expected exception of type IllegalArgument but got NullPointer ); } 10

11 /* map the map to synchronize, must not be null a synchronized map backed by the given map IllegalArgument if the map is null */ static <K,V> Map<K,V> synchronizedmap(map<k,v> map) Confirmed & fixed by Collections developers public void test1() throws Throwable { java.util.map var0 = null; try { java.util.map var1=...synchronizedmap(var0); } catch (IllegalArgument expected) {return;} fail("expected exception of type IllegalArgument but got NullPointer ); } 11

12 /* anchor the anchor (<code>null</code> not permitted). */ void setrotationanchor(textanchor anchor) 12

13 /* anchor the anchor (<code>null</code> not permitted). */ void setrotationanchor(textanchor anchor) 13

14 /* anchor the anchor (<code>null</code> not permitted). */ void setrotationanchor(textanchor anchor) public void test2() throws Throwable {...CategoryPointerAnnotation var0 = new CategoryPointerAnnotation( $0.00, (java.lang.comparable)'#', 10.0d, 10.0d); org.jfree.ui.textanchor var1 = null; try { var0.setrotationanchor(var1); fail("expected exception but got Normal Execution"); } catch ( expected) {} } 14

15 /* anchor the anchor (<code>null</code> not permitted). */ void setrotationanchor(textanchor anchor) Confirmed & fixed by JFreeChart developers public void test2() throws Throwable {...CategoryPointerAnnotation var0 = new CategoryPointerAnnotation( $0.00, (java.lang.comparable)'#', 10.0d, 10.0d); org.jfree.ui.textanchor var1 = null; try { var0.setrotationanchor(var1); fail("expected exception but got Normal Execution"); } catch ( expected) {} } 15

16 @tcomment Javadoc Comments Infer properties Properties Java Methods Modified Randoop Tests with Inconsistency 16

17 @tcomment Javadoc Comments 2 Infer properties 1 Properties Java Methods Modified Randoop 3 4 Tests with Inconsistency 17

18 Null Normal, the method should execute normally (no exception) predicate the predicate to use, may be null Null Any, the method should throw some exception collection the collection to add to, must not be null Null Specific (id==null => IllegalArgumentExc), the method should throw a specific type of exception IllegalArgument if the id is null Null Unknown, the method behavior is unknown array the array over which to iterate 1 18

19 Null Normal, the method should execute normally (no exception) predicate the predicate to use, may be null Null Any, the method should throw some exception collection the collection to add to, must not be null Null Specific (id==null => IllegalArgumentExc), the method should throw a specific type of exception IllegalArgument if the id is null Null Unknown, the method behavior is unknown array the array over which to iterate 1 19

20 Null Normal, the method should execute normally (no exception) predicate the predicate to use, may be null Null Any, the method should throw some exception collection the collection to add to, must not be null Null Specific (id==null => IllegalArgumentExc), the method should throw a specific type of exception IllegalArgument if the id is null Null Unknown, the method behavior is unknown array the array over which to iterate 1 20

21 Null Normal, the method should execute normally (no exception) predicate the predicate to use, may be null Null Any, the method should throw some exception collection the collection to add to, must not be null Null Specific (id==null => IllegalArgumentExc), the method should throw a specific type of exception IllegalArgument if the id is null Null Unknown, the method behavior is unknown array the array over which to iterate 1 21

22 2 Parse Javadoc (Standard Doclet) Analyze text 22

23 2 Parse Javadoc (Standard Doclet) Analyze text Negation words (±3 distance within null) Null Any No negation words Null Normal null tag, then searches the list of parameter names. If found Null Specific param == null =>Specific Generates multiple properties for or and either in tag 23

24 while timelimit not reached: a. Create a new sequence 1. Randomly pick a method call m(t 1...T k ) 2. For each parameter, randomly pick a sequence S i that can construct the object for that parameter Select null with some probability (nullratio) 3. Create new sequence S new S 1;...S k ; m(var 1...var k ) b. Classify the new sequence S new Execute & check contract violations? - Output as failure-revealing test case No - Add to sequences if not redundant, else discard 24

25 3 while timelimit not reached: a. Create a new sequence 1. Randomly pick a method call m(t 1...T k ) 2. For each parameter, randomly pick a sequence S i that can construct the object for that parameter Select null with some probability (nullratio) 3. Create new sequence S new S 1;...S k ; m(var 1...var k ) b. Classify the new sequence S new Execute & check contract violations properties? Violate contract - Output as failure-revealing tests Violate properties - Output as comment-code inconsistency No - Add to sequences if not redundant, else discard 25

26 4 /** logger logger, may not be null. sourceclass source class, may be null. sourcemethod method, may be null. */ void exiting(logger logger, String sourceclass, String sourcemethod) 26

27 4 /** Null Any logger logger, may not be null. sourceclass source class, may be null. sourcemethod method, may be null. */ void exiting(logger logger, String sourceclass, String sourcemethod) 27

28 4 /** Null Any logger logger, may not be null. sourceclass source class, may be null. sourcemethod method, may be null. */ void exiting(logger logger, String sourceclass, String sourcemethod) 28

29 4 /** Null Any logger logger, may not be null. sourceclass source class, may be null. sourcemethod method, may be null. */ void exiting(logger logger, String sourceclass, String sourcemethod) Execute 29

30 4 /** Null Any logger logger, may not be null. sourceclass source class, may be null. sourcemethod method, may be null. */ void exiting(logger logger, String sourceclass, String sourcemethod) Execute Actual Thrown 30

31 4 /** Null Any logger logger, may not be null. sourceclass source class, may be null. sourcemethod method, may be null. */ void exiting(logger logger, String sourceclass, String sourcemethod) Execute Actual Thrown Matches 31

32 4 /** Null Any logger logger, may not be null. sourceclass source class, may be null. sourcemethod method, may be null. */ void exiting(logger logger, String sourceclass, String sourcemethod) Execute Actual Thrown Null Any Matches 32

33 4 /** Null Any logger logger, may not be null. sourceclass source class, may be null. sourcemethod method, may be null. */ void exiting(logger logger, String sourceclass, String sourcemethod) Execute Actual Thrown Null Any Matches iscommentcodeinconsistency? 33

34 4 Throws? Execute 34

35 4 Throws? Execute No Null Any / Null Specific? 35

36 4 Execute Throws? No Null Any / Null Specific? Missing 36

37 4 Execute Throws? No Null Any / Null Specific? Missing 1 Null Specific? 37

38 4 Execute Throws? No Null Any / Null Specific? Missing 1 Null Specific? No 1 Null Normal? 38

39 4 Execute Throws? No Null Any / Null Specific? Missing 1 Null Specific? No 1 Null Normal? Unexpected 39

40 4 Execute Throws? No Null Any / Null Specific? Missing 1 Null Specific? No 1 Null Normal? Unexpected Unknown Status 40

41 4 Execute Throws? No Null Any / Null Specific? Missing 1 Null Specific? No 1 Null Normal? Unexpected in the set of expected? Unknown Status 41

42 4 Execute Throws? No Null Any / Null Specific? Missing 1 Null Specific? No 1 Null Normal? Unexpected in the set of expected? No Different Unknown Status 42

43 4 Execute Throws? No Null Any / Null Specific? Missing 1 Null Specific? No 1 Null Normal? Unexpected in the set of expected? No Different Unknown Status Expected 43

44 4 Execute Throws? No Null Any / Null Specific? Missing 1 Null Specific? No 1 Null Normal? Unexpected in the set of expected? No Different Unknown Status Expected Missing Different Unexpected Unknown Status Expected Is Comment-code Inconsistency? 44

45 7 open source projects to infer properties Ran Modified Randoop to check properties Varying 2 options on Modified Randoop nullratio ( How often null is chosen as input for a method?) timelimit ( How long should Modified Randoop take for generating tests before stopping?) Best configuration for all projects nullratio = 0.6 timelimit = 3600s (reached plateau effect after that) 45

46 Project Description # LOC # Classes # Methods Collections Collection library and utilities 19, ,874 GlazedLists List transformations in Java 19, ,753 JFreeChart Chart creator 51, ,205 JodaTime Date and time library 18, ,887 Log4j Logging service 14, ,115 Lucene Text search engine 38, ,222 Xalan XML transformations 53, ,404 46

47 Project Missing Different Unexpected Unknown Status Expected Tested Properties Collections GlazedLists JFreeChart JodaTime Log4j Lucene Xalan Total

48 Project Missing Different Unexpected Unknown Status Expected Tested Properties Collections GlazedLists JFreeChart JodaTime Log4j Lucene Xalan Total Tested only a fraction of properties inferred 48

49 Project Missing = TI + FA Different = TI + FA Unexpected = TI + FA Collections GlazedLists JFreeChart JodaTime Log4j Lucene Xalan Total True Inconsistencies (TI) Total False Alarms (FA)

50 45% 27% 28% Incorrectly inferred properties Missing properties Incorrect/missing properties for another method Incorrect inference Wrong type of inferred properties Missing properties Missing property (Null Unknown) causes unexpected exception Incorrect/missing properties for another method Method under test depend on other method, other method causes unexpected exception 50

51 Project Precision [%] Recall [%] Accuracy Norm Any Spec Norm Any Spec [%] Collections GlazedLists JFreeChart JodaTime Log4j Lucene Xalan Total/Overall properties are inferred, 1 incorrect High accuracy (97-100%) 51

52 Comment-Code Inconsistency Detection Detected 28 comment-code inconsistencies 40 false alarms Comment Analysis Result High accuracy of % without using NLP techniques Javadoc comments are well-structured Not much variance in paraphrases 52

53 An inconsistency between comment and code is highly indicative of program checks consistency of Java method bodies and Javadoc comments properties related to null values and exceptions Evaluated on 7 open-source projects Discovered 28 inconsistencies and 12 were already fixed Acknowledgement: Travel expenses sponsored by NSF CCF grants and ACM-W Scholarships for Attendance at Research Conferences. 53