SAMATE (Software Assurance Metrics And Tool Evaluation) Project Overview. Tim Boland NIST May 29,

Transcription

1 SAMATE (Software Assurance Metrics And Tool Evaluation) Project Overview Tim Boland NIST May 29,

2 NationaI Institute of Standards and Technology (NIST) NIST, an agency of the U.S. Department of Commerce, researches measurement science, standards, and technology 2

3 Presentation Order: Introduction to SAMATE/Software Assurance SAMATE Sub-Projects/Activities Present and Past Next Steps (Towards a Science of Weakness Measurement) Disclaimer any products/companies mentioned are for information only no endorsement implied 3

4 Background -Need for Software Assurance Society depends more and more on software Ad-hoc fixes are not enough Software must be engineered to have desired properties, and assurance is needed that software has those properties 4

5 Software Assurance Software assurance involves confidence that software is free from vulnerabilities and conforms to standards/policies/procedures Software assurance comes from three sources: process, analysis, and execution 5

6 Does This Violate Software Assurance? 6

7 SAMATE Goals improving software assurance by developing materials, specifications, and methods to test tools/techniques and measure their effectiveness supporting trustworthiness no vulnerabilities exist, either of malicious or unintentional origin supporting predictable execution -justifiable confidence that software, when executed, functions as intended 7

8 A SAMATE Approach Develop a specification Come up with test plan Set minimum bar for software 8

9 Relevant Definitions Failures: caused by defects (faults, bugs) in executing code, or by environmental conditions Vulnerability:result of one/more weaknesses in requirements, design, implementation, or operation whether a weakness is a vulnerability may depend on how software is used 9

10 Some Current SAMATE Sub- Projects/Activities Static Analysis Tool Exposition (SATE) SAMATE Reference Dataset (SRD) Precisely define some Common Weakness Enumeration (CWE) entries accurately (CWE Formalization) CWE compatibility/effectiveness studies 10

11 What is a Static Analysis Tool? 11

12 Definitions Relevant to Static Analysis Tools Warning:issue (usually, a weakness) identified by a tool (Tool) report: the output from a single run of a tool on a test case a tool report consists of warnings 12

13 Static Analysis Tool Exposition (SATE) Enable empirical research based on large test sets Encourage improvement of tools Speed adoption of tools by objectively demonstrating their use on real software NOT to choose the best tool Focus is on security issues (exploring the following characteristics of tools: relevance of warnings to security, their correctness, and prioritization ) Four SATEs: 2008, 2009, 2010, and 2011/12 13

14 SATE Steps Hold organizing workshop Test sets with security implications chosen by NIST and provided to participants Participants run tools on these test sets and return reports NIST analyzes returned tool reports (also experts analyze one program) Experience/Observations Shared at Workshop Final report/data released afterwards 14

15 Current SATE Activities Languages are C/C++, Java, and PHP Common Vulnerability Enumeration (CVE) -based programs chosen for analysis - Warning Subset Analysis Introduction of Synthetic Test Cases Reporting Format SATE 2008, 2009, 2010 reports available SATE IV is being completed 15

16 Future SATE Activities Planning for Next SATE (what to change, what to include) Mining large SATE data set to extract more useful information Soliciting more tool vendors to participate 16

17 SATE IV Information Page 17

18 SAMATE Reference Dataset (SRD) Public repository for software test cases in C, C++, Java, Python, C# User can locate test cases by language, weakness, code construct, etc. Contributions from Fortify, Defence R&D Canada, Klocwork, MIT Lincoln Laboratory, Praxis, Secure Software, and others Weaknesses/complexities known in advance 18

19 More SRD Information SRD includes test suites (the most recent being the Juliet test suite) SRD may be used by static analysis and binary analysis tools, among other uses 19

20 Juliet Test Suite 59,943 small C/C++ and Java programs C/C++ programs cover 116 different CWEs, and Java programs cover 106 different CWEs In dozens of subtle variations With dozens of code complexities Synthetic test cases 20

21 SRD User Interface Example (Test Suite Link Highlighted) 21

22 Some SRD Functions Browse the complete test case repository or download test cases Find (search for) specific test cases Download test suites Information for submitting test cases Find other assurance tool test collections/datasets 22

23 SRD Search Page Example 23

24 SRD Test Case 1552 (tainted input allows arbitrary files to be read or written re: CWE-417) 24

25 Future SRD Activities Encourage inclusion of more test cases/test suites Characterize/sort test cases re: investigation of certain CWEs and capabilities of tools Investigate further uses of the test cases Promote use of existing test cases 25

26 CWE Formalization Weaknesses must be defined precisely/accurately (from description summary, white-box definition) Goal of accurate/precise CWEs: demonstrate a high-bar for definitions, find out how much can be formalized (e.g., correctly perfom, intended command ), and investigate approaches for validating the definitions Link to information: 26

27 CWEs May Be Hierarchical 27

28 Classes May Interact 28

29 MITRE s CWE Compatibility and Effectiveness Program Phase 1 declare compatibility Phase 2 verify mapping to CWEs (possible support for coverage claim representation) Phase 3 test cases show effectiveness (tool/service effectively locates CWEs, deals with code complexities) NIST will help develop test cases URL: 29

30 Other SAMATE-Related Activities Voting systems Web accessibility Miscellaneous software measurement/metrics/classification research (e.g., complexities, models..) Software testing research (workshop) 30

31 NIST Workshop on Metrics and Standards for Software Testing (MaSST) In conjunction with Software Security And Reliability (SERE) June 2012 at NIST in Gaithersburg, MD URL: 31

32 Past SAMATE Sub-Projects/Activities Web application scanners Software labels Software assurance studies Assurance case research 32

33 Next Steps (Toward a Science of Weakness Measurement) Linguistics/vocabulary issues Designating weaknesses Distinguishing weaknesses Locating weaknesses Potentially useful concepts for measurement 33

34 Linguistics/Vocabulary What is a word? Orthographic word (blank-separated), lexical item, citation form (e.g., go vs. went ) Abbreviations, acronyms, contractions, clipped forms, etc. Possible International Vocabulary of Metrology (VIM) applicability? 34

35 Designating a Weakness Warning may be trivially true (e.g., strncpy() does not null terminate) Warning may be true but insignificant (e.g., possible buffer overflow while reading configuration file) Consider ease, likelihood, or impact of exploit Some warnings are difficult to decide on (e.g., divide by 0) Quality or style weaknesses (e.g., dead code) 35

36 Distinguishing Weaknesses Only 1/8 to 1/3 of weaknesses are simple (from SATE report) Weakness Classes are Related (one weakness?) (e.g., integer overflow -> buffer overflow) Data/Control Flows may be intermingled (one weakness?) Number of Weaknesses in code is ill-defined (e.g., mistakes may be repeated..) 36

37 Example of Tangled Flows (2 sources, 2 sinks, 4 paths) (from Nagios) 37

38 Locating Weaknesses Data/control flow issues (interacting statements) Location of missing feature (code omission) (e.g. failure to invalidate credentials, not logging all transactions..) Which statement should be reported (not obvious source? sink?) 38

39 Possible Useful Concepts for Measurement of Weaknesses Source/sink, source-to-sink graph Location to fix Flow Weakness Instance/Class Vulnerability Attack Fault Error Data or Control Path 39

40 40

41 Other SAMATE Resources from Web sites and resources list Bibliography SAMATE publications list 41

42 How to Participate in SAMATE SAMATE mailing list web site is (please become a member) General samate@nist.gov Contributions/collaboration welcomed Technical Advisory Panel 42