Finding the Static Analysis Sweet Spot

Transcription

1 Finding the Static Analysis Sweet Spot UMSEC Summer Software Symposium 2012 Presented by: Paul Anderson, VP of Engineering, GrammaTech Inc. GrammaTech, Inc. 317 N Aurora St. Ithaca, NY Tel: info@grammatech.com

2 Key Takeaways Advanced Static Analysis Tools can and do find serious bugs Maximizing ROI is non-obvious Truth is subjective Tools and code bases have very different characteristics Tools in default configurations are unlikely to be optimal Beware of instant gratification What feels best does not necessarily make sense Consider the long-term economics Page 2

3 Agenda What is Advanced Static Analysis? Warning Classifications The Business Case Case Studies Conclusions Page 3

4 Introduction to Static Analysis Infers information about software behavior based on a static representation In contrast to dynamic analysis, such as profiling, debugging, testing Analyzes code instead of executing it So no test cases are needed Is usually a two-phase process Extract semantic information from source code Use information to discover defects or other properties of interest This talk is about Advanced static analysis for Bug Finding As exemplified by CodeSonar Page 4

5 Early-generation Static Analysis Tools Tools whose primary purpose is stronger static type and style checking E.g., Lint Key Techniques Syntactic analysis using imprecise parsing Intraprocedural dataflow analysis Primitive ways to specify inter-procedural or inter-module information Some functionality absorbed by modern compilers Tools whose primary purpose is to find violations of coding rules E.g., Misra checkers Page 5

6 Risks of Blind Elimination of Violations Low correlation between Misra violations and actual defects Only 9/72 Misra rules were observed to be better than random at predicting defects C. Boogerd and L. Moonen. Assessing the Value of Coding Standards: An Empirical Study. In Proceedings of the 24th International Conference on Software Maintenance (ICSM), pages IEEE Computer Society Press, All changes risk introducing new defects Probability of introducing an error while making your code Misra-C compliant: 15% L. Hatton. Language Subsetting in an Industrial Context: A Comparison of Misra C 1998 and Misra C Information & Software Technology, 49(5): , Page 6

7 Advanced Static Analysis Tools Tools whose primary purpose is to find serious bugs Key Analysis Techniques High-precision program model Interprocedural Whole program Flow sensitive Context sensitive Path sensitive Highly customizable Page 7

8 Benefits of Advanced Static Analysis Benefits of advanced static analysis are based on Can examine far more execution paths than conventional testing Can be applied automatically, early in the development cycle Advanced static analysis Catches problems that test suites miss Catches bugs early, when they are less expensive to fix Pinpoints defects automatically, improving productivity Shows if code quality is improving over time Improves software quality (and security) Reduces time to market Page 8

9 Zune Bug: 1 Bad Line of Code Page 9

10 Zune Bug: 1 Bad Line of Code Page 10

11 Example Bug found by CodeSonar s malloc newloc B U F F E R \0 B U F F E R \0 Buffer Overrun!!!! Page 11

12 Some Checks Buffer Overrun Null-Pointer Dereference Divide by Zero Uninitialized Variable Free Non-Heap Variable Use After Free Double Free/Close Free Null Pointer Format String Vulnerability Race Conditions Unreachable Code Memory/Resource Leak Return Pointer to Local Mismatched Array New/Delete Invalid Parameter Missing Return Statement Dangerous Cast Inconsistent form User-Defined Checks Many More... Page 12

13 Agenda What is Advanced Static Analysis? Warning Classifications The Business Case Case Studies Conclusions Page 13

14 Definitions Recall The probability of finding a real bug Perfect recall => sound R = TT N = TT TT+FF Precision The probability that a result is a real bug Perfect precision => complete P = TT TT+FF Page 14

15 To: From: Dear Santa: I would like a tool that finds all bugs in my program. Thanks, Bob Page 15

16 Here is a static analysis tool has NO FALSE NEGATIVES!!! It is guaranteed to find every bug in your code. It s called the Code Assurance Tool: cat. Watch it list every line in your program that has a bug on it: % cat foo.c #include <stdio.h> It also can be used in a mode that reports ZERO FALSE POSITIVES! int main(char argc, char *argv[]) { printf( Hello world!\n ); return 0; } % % cat foo.c >/dev/null Page 16

17 All practical Static- Analysis Tools have false negatives and false positives Page 17

18 Warning Classifications Traditional technical interpretations True Positive The tool reports a real bug False Positive The tool reports a bug that doesn t exist False Negative The tool fails to report a real bug True Negative The tool succeeds in not reporting a non-existing bug Better interpretation True Positive Worth doing something about False Positive Not worth doing something about Page 18

19 Why not fix a (technical) True Positive? Every change has a cost Re-programming, re-testing, re-certification, etc. Every change to the code has a risk of introducing a new bug Estimated to be 15%! Cost of fixing might exceed cost of leaving it alone Page 19

20 Why fix a (technical) False Positive? False Positives imply tool confusion Tool confusion may lead to False Negatives Fixing False Positives may lead to fewer False Negatives If the compiler or the static analyzer gets confused, the code causing the confusion should be rewritten so that it becomes more trivially valid. Many have been caught in the assumption that a warning was likely invalid, only to realize much later that the report was in fact valid for less obvious reasons. Gerard J. Holzmann's "rule 10" from The Power of Ten Rules for Writing Safety Critical Code. Cost to not change the code may exceed cost to change it Page 20

21 Role of Tool User Code author Needs to churn out code fast. Hates distractions. Likes maintainability. Code reviewer Acts as a check on the author. Cares about consistency. Low time commitment. Tester Seeks out bugs. Concerned with testability. Safety engineer Checks for compliance Internal security analyst Looks for potential security vulnerabilities. Red Team/Attacker Looks for exploitable vulnerabilities. Page 21

22 Role influences Diagnosis Author Reviewer Tester Safety Engr Security Attacker Buffer overrun Y Y Y Y Y? Leak Y Y Y?? N Redundant condition N? Y N N N Null test after dereference Recursion? Y Y N N N N N N Y N N Function too long N N N Y N N Use of mktemp N N N N Y? Page 22

23 Other Influences on Diagnoses Platform-specific considerations Deployment environment Risk of code change Page 23

24 Buffer Underrun Example int pvm_pkstr(cp) char *cp; { int l = strlen(cp) + 1; int cc; Buffer Underrun reported Look at caller to see the real reason: p = getenv( PVM_EXPORT );... p = p 11;... pvm_pkstr(p); Suspicious code! This code does work p = xyz p 11 = PVM_EXPORT=xyz Page 24

25 getenv() issue char environ[] = USER=paul\0PVM_EXPORT=xyz\0PATH=/usr/bin... p - 11 p Code relies on the implementation of getenv(). This behavior is not specified (or precluded) by the specification of getenv(). Possible diagnoses: False positive because the target platform works this way? True positive because this may not port? True positive because it is confusing? Insignificant? Page 25

26 Truth is Subjective Page 26

27 Warning Triage Warnings from Tool Diagnosis False Positives True Positives Page 27

28 Which is better? Time wasted by Tool A Bugs missed by Tool B Tool A Tool B Page 28

29 Everyone Hates False Positives Page 29

30 Page 30

31 Page 31

32 Where s Waldo (Wally, Walter, Charlie, Hugo, ウォーリー, 威利 )? Page 32

33 You don t know what a real Waldo looks like Lots of other figures may resemble Waldo You don t know how many Waldos there are You WILL make misdiagnoses Page 33

34 Consequences of Misdiagnoses TP diagnosed as FP Warning report is suppressed in future reports Bug remains in the program FP diagnosed as TP Engineer may modify the code Change may be benign, or may introduce a new bug Risk of new bug strongly coupled to warning class Analysis is unlikely to notice the new bug Page 34

35 Probability of Misdiagnosis Waldo is easier to find among fewer non-waldoes More false positives => more misdiagnoses Page 35

36 100% 90% Probability of Correct Judgment 80% 70% 60% 50% 40% 30% 20% Good! All warnings are true positives, and all are judged correctly 10% 0% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Precision Bad! All warnings are false positives, so nobody uses the tool. Page 36

37 100% 90% Probability of Correct Judgment 80% 70% 60% 50% 40% 30% 20% 10% Linear Quadratic Cubic 0% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Precision Page 37

38 Bugs in Deployed Product Bugs not found by tool Bugs found by tool, but misjudged as False Positives Bugs introduced as a consequence of changes, and subsequently undetected Page 38

39 Which is better? Time wasted by Tool A Bugs missed by Tool B Tool A Tool B Page 39

40 Look at the Business Case Page 40

41 Agenda What is Advanced Static Analysis? Warning Classifications The Business Case Case Studies Conclusions Page 41

42 How to model the business case? Costs Engineering time Time to fix a bug early vs. late Time to inspect a warning Cost of bug incident Downtime Customer dissatisfaction Safety or security violation Risks Number of bugs in product Probability of bug triggering an incident Probability of new bug introduced by fix Consequences of warning misjudgment Tool variables Recall (probability of finding a real bug) Precision (probability of a warning being real) = 1-False positive rate Page 42

43 Inputs Number of bugs 100 Cost of bug incident $40,000 Probability of bug triggering an incident 5% Engineer hourly rate* (adjusted for inflation) $75 Hours to fix a bug early* 4 Hours to fix a bug late* 20 Time to inspect a warning 10 minutes Probability of FP misjudgment 0% Probability of necessary change introducing new undetected bug 0% Probability of un-necessary change introducing new undetected bug 0% * Source: The Economic Impacts of Inadequate Infrastructure for Software Testing. NIST Page 43

44 Tool Variables Tool Recall Precision Cost Tool A 60% 50% $20,000 Tool B 30% 80% $20,000 Tool C 95% 10% $20,000 No tool 0% n/a $0 Perfect tool 100% 100% $0 Page 44

45 Warning Results Bugs not detected by tool False positives True positives Tool A Tool B Tool C No tool Perfect tool Page 45

46 Remaining Bugs Linear Cubic Page 46

47 Return on Investment (cubic model) Linear Cubic Page 47

48 How to Compare Two tools, or two different configurations of the same tool Recall is impossible to quantify accurately And difficult to estimate But it is easy to count results Page 48

49 How to Compare Tools Let f(p) be the probability of true-positive misjudgment given precision p Tool 1 and 2 have the same value if: Or, N R 1 f(p 1 ) = N R 2 f(p 2 ) FF 2 = TT 2 f 1 ( TT 1 TT f( 1 )) TT 2 TT 1 + FF 1 TT 2 Page 49

50 Example Tool 1 TP 1 = 40, FP 1 = 10 Tool 2 TP 2 = 50 How many False Positives is it worth tolerating for those extra 10 real bugs? Linear model: 28 Cubic model (more realistic): 72 Page 50

51 Economics vs. Emotions Two factors dominate the economics: Cost of incident Consequence of misjudgments One factor dominates the emotions False positive rate Recall is impossible to quantify Consequently: Tools demonstrated with low FP rate feel better In production, tools are seldom configured optimally Page 51

52 Beware of Instant Gratification Page 52

53 Agenda What is Advanced Static Analysis? Warning Classifications The Business Case Case Studies Conclusions Page 53

54 Case Study: Software not Delivered Patient Monitoring Device: 900K LOC Buffer overflow found late in testing Bug found by developer fixing a different issue Bug fixed in such a way that a new bug was introduced Entire product shipment held up Loss of confidence led to code re-review, weeks of effort Static analysis found the issue immediately Also showed how the fix was not really a fix Analysis also identified additional issues that reviews and testing had missed Page 55 Page 55

55 NASA Satellite Network NASA s Tracking and Data Relay Satellite System (TDRSS) Nine geosynchronous satellites and three ground stations Launch support and on-orbit communications and tracking services for most NASA missions Striving for 99.9% availability (excluding scheduled down times) Detailed study of ROI of using CodeSonar on ground station software Conducted by NASA White Sands under the sponsorship of NASA Ames Research Center and the NASA IV&V Facility Found that tool price plus cost of labor to use was about equal to debugging labor cost However, tool use prevents outages, whereas reactive debugging does not Result of study: CodeSonar was adopted to reduce software errors Page 56

56 Agenda What is Advanced Static Analysis? Warning Classifications The Business Case Case Studies Conclusions Page 57

57 Conclusions and Recommendations Static-analysis tools have different characteristics All code bases have different characteristics Value of tool depends strongly on individual circumstances Best approach: Try them out on your own code But be careful to judge benefit rationally False-positive rate may not be the most important factor Consider customizations to find domain-specific issues Page 58

58 The End Me: Paper: Measuring the Value of Static Analysis Tool Deployments, Paul Anderson, IEEE Security and Privacy, May/June 2012 Product: CodeSonar Sign up for a fully-functional free evaluation on your code Web site: Page 59