Evaluating Bug Finders

Transcription

1 Evaluating Bug Finders Test and Measurement of Static Code Analyzers Aurelien DELAITRE Bertrand STIVALET ICSE - COUFLESS 2015 May 23, 2015

2 Authors Aurelien DELAITRE West Virginia University Bertrand STIVALET National Institute of Standards and Technology 2

3 Authors Elizabeth FONG NIST Vadim OKUN NIST 3

4 "If debugging is the process of removing software bugs, then programming must be the process of putting them in" E. Dijkstra 4

5 1. SAMATE Project Software Assurance Metrics And Tool Evaluation 5

6 Software Assurance Reference Dataset (SARD) SARD contains Small test cases w/ specific vulnerabilities Large test suites Software w/ CVEs SARD in numbers 34 Test suites 243 CWEs 148,903 Test cases 665,481 Files 6

7 Static Analysis Tool Expositions (SATE) 5 editions of SATE 3 programming languages 5M+ lines of code for SATE V 7

8 2. Software as Big Data Introduction to Static Analysis 8

9 Static Analysis Automated analysis of large software Defect detection and remediation Use different approaches: Syntax checking Heuristics Formal methods 9

10 Static Analysis Automated analysis of large software Defect detection and remediation Use different approaches Buggy Source Code Compilation Buggy Software 10

11 Static Analysis Automated analysis of large software Defect detection and remediation Use different approaches Buggy Source Code Bug Report Static Analysis Remediation 11

12 Static Analysis Automated analysis of large software Defect detection and remediation Use different approaches Fixed Source Code Compilation Secure Software 12

13 Pros and Cons Improves software assurance Saves time and money Takes customized rule sets False positive (noise) False negative (missed defects) Limited scope 13

14 3. Metrics Measuring the Effectiveness of Tools 14

15 Evaluation Metrics Flawed code Safe code Tool Warnings True False Positives Positives Source Code False Negatives True Negatives 15

16 Evaluation Metrics How much can I trust a tool? Flawed code Safe code Tool Warnings True False Positives Positives Source Code False Negatives True Negatives 16

17 Evaluation Metrics Precision How much can I trust a tool? Flawed code Safe code Tool Warnings True False Positives Positives Prec. False Negatives Source Code True Negatives 17

18 Evaluation Metrics Precision How much can I trust a tool? Flawed code Safe code What proportion of flaws can a tool find? Tool Warnings True False Positives Positives Prec. False Negatives Source Code True Negatives 18

19 Evaluation Metrics Precision How much can I trust a tool? Recall What proportion of flaws can a tool find? Flawed code Safe code Re ca ll Tool Warnings True False Positives Positives Prec. False Negatives Source Code True Negatives 19

20 Evaluation Metrics Precision How much can I trust a tool? What kind of flaws can a tool find? Recall What proportion of flaws can a tool find? Buggy Code Static Analysis 20

21 Evaluation Metrics Precision Coverage How much can I trust a tool? What kind of flaws can a tool find? Recall What proportion of flaws can a tool find? Buggy Code Bug Report Static Analysis 21

22 Evaluation Metrics Precision Coverage How much can I trust a tool? What kind of flaws can a tool find? Recall What proportion of flaws can a tool find? How smart is a tool? Safe Code Static Analysis Buggy Code 22

23 Evaluation Metrics Precision Coverage How much can I trust a tool? What kind of flaws can a tool find? Recall Discrimination What proportion of flaws can a tool find? How smart is a tool? Safe Code Safe Code Static Analysis Buggy Code Buggy Code 23

24 Evaluation Metrics Precision Coverage How much can I trust a tool? What kind of flaws can a tool find? Recall Discrimination What proportion of flaws can a tool find? How smart is a tool? How similar are unrelated tools? report report 24

25 Evaluation Metrics Precision Coverage How much can I trust a tool? What kind of flaws can a tool find? Recall Discrimination What proportion of flaws can a tool find? How smart is a tool? Overlap How similar are unrelated tools? report report 25

26 4. Test Cases Static Analysis Tool Exposition (SATE) 26

27 Design of Test Cases Statistical significance #include <stdio.h> int main(){ 27

28 Design of Test Cases Statistical significance Relevance #include <stdio.h> int main(){ 28

29 Design of Test Cases Statistical significance Relevance Ground Truth #include #include <stdio.h> <stdio.h> int main(){ int main(){ 29

30 Design of Test Cases Statistical significance Relevance Ground Truth #include #include <stdio.h> <stdio.h> int main(){ int main(){ Types of Test Cases: Software with Common Vulnerability Enumeration (CVE) Production Software Synthetic Test Cases 30

31 Design of Test Cases Software w/ CVEs Statistical significance Relevance Ground Truth #include #include <stdio.h> <stdio.h> int main(){ int main(){ Types of Test Cases: Software with Common Vulnerability Enumeration (CVE) Production Software Synthetic Test Cases 31

32 Design of Test Cases Production Software Statistical significance Relevance Ground Truth #include #include <stdio.h> <stdio.h> int main(){ int main(){ Types of Test Cases: Software with Common Vulnerability Enumeration (CVE) Production Software Synthetic Test Cases 32

33 Design of Test Cases Synthetic Cases Statistical significance Ground Truth #include #include <stdio.h> <stdio.h> int main(){ int main(){ Types of Test Cases: Software with Common Vulnerability Enumeration (CVE) Production Software Synthetic Test Cases 33

34 Mapping Metrics to Data Question Production Software Software w/ CVEs Synthetic Test Cases Coverage Recall Precision Discrimination Overlap Applicable - Metric can be computed Limited - Some limitations with the calculation N/A - Not Applicable 34

35 5. Results 35

36 3,480,195 Warnings to analyze*! *from the SATE V experience 36

37 Coverage Spectrum per Tool For Synthetic Java 37

38 Recall per Tool For Synthetic Java 38

39 Precision per Tool For Synthetic Java 39

40 Discrimination per Tool For Synthetic Java 40

41 Combination of Tool Metrics 41

42 Findings Overlap 42

43 Code Complexity char * data; data = NULL; char mystring[] = "mystring" ; data = strdup(mystring); delete [] data; char * data; char * *dataptr1 = &data; char * *dataptr2 = &data; data = NULL; char * data = *dataptr1; char mystring[] = "mystring" ; data = strdup(mystring); *dataptr1 = data; { char * data = *dataptr2; delete [] data; } 43

44 Code Complexity char * data; data = NULL; char mystring[] = "mystring" ; data = strdup(mystring); delete [] data; char * data; char * *dataptr1 = &data; char * *dataptr2 = &data; data = NULL; char * data = *dataptr1; char mystring[] = "mystring" ; data = strdup(mystring); *dataptr1 = data; { char * data = *dataptr2; delete [] data; } CWE 762: Mismatched Memory Management Routines 44

45 Complexity vs. Tool Effectiveness char * data; data = NULL; char mystring[] = "mystring" ; data = strdup(mystring); delete [] data; char * data; char * *dataptr1 = &data; char * *dataptr2 = &data; data = NULL; char * data = *dataptr1; char mystring[] = "mystring" ; data = strdup(mystring); *dataptr1 = data; { char * data = *dataptr2; delete [] data; } Found by tool X Found by tool X Found by tool Y Missed by tool Y 45

46 Recall per Complexity For Synthetic C 46

47 Precision per Tool On Production Software vs. Synthetic Java 47

48 5. Conclusion 48

49 Conclusion Tools need evaluation! Test cases need improvement Testing procedure needs more metrics: Usability Integration Impact 49

50 Thanks! Any questions? Find us at: 50

51 51

52 SATE The Art of Collecting Data Tool Vendors Static Analysis Tool Test Cases SATE Format Converter Synth. Data CVE Data SATE Reports SATE Database Manual Sample Analysis Semi-Automated CVE matching SAMATE Automated Juliet Analysis 52

53 Evaluation Metrics Question What proportion of defects can a tool find? How noisy is a tool? How similar are unrelated tools? Metrics Recall / Coverage Precision / Discrimination Overlap 53

54 Complexity Different kinds of complexities in the Synthetic Test Cases None No complexity int main() { char buf[15]; cin >> buf; cout << "echo: " << buf << endl; return 0; } 54

55 Complexity Different kinds of complexities in the Synthetic Test Cases None Control Flow Control Flow complexity int main() { char buf[15] = "COUFLESS2015"; if (1) cin >> buf; cout << "echo: " << buf << endl; return 0; } 55

56 Complexity Different kinds of complexities in the Synthetic Test Cases None Control Flow Data Flow Data Flow complexity char *stringcopy(char *str1, char *str2) { while (*str2) *str1++ = *str2++; return str2; } int main(int argc, char **argv) { char *buffer = (char *)malloc(16 * sizeof(char)); stringcopy(buffer, argv[1]); printf("%s\n", buffer); return 0; } 56