Static Analysis versus Software Model Checking for bug finding

Transcription

1 Static Analysis versus Software Model Checking for bug finding Dawson Englers, Madanlal Musuvathi Stanford University, CA, USA Presented By: Prateek Agarwal ETH, Zürich

2 Agenda Goal Aim Scope Methodologies used Meta Compilation C Model Checker Case Studies Conclusions

3 Goal Checking validity of general perception Static Analysis is easy to apply, finds shallow bugs Model Checking is harder but strictly better

4 Aim Documentation of experience Set of case studies Comparison of two approaches, based on Bugs/Lines of Code False positives vs Actual Bugs Time & Effort required Types of Bugs: generic vs protocol specific Find as many bugs as possible

5 Scope Verification of system/protocol software Critical Hard to test/inspect/reason about manually Highly optimized Protocol codes follow event driven model

6 Meta Compilation (MC) Extension of compiler Used for Static Analysis & Model Checking Metal: Language for Slice specification by pattern matching Code translation State machines xg++ over g++ compiler

7 Meta Compilation: Key Features Code should be highly structured e.g. special MACRO/functions for read/write No source code annotation needed Emphasis on reducing false positives No Simulation of code No model of heap, tracking variable values Simulation tends to become model checking

8 Static Analysis using MC Patterns & Actions Small State machines for checking Example Rule: Sync before read... start: {WAIT_FOR_DB_FULL(addr);} ==> stop {MISCBUS_READ_DB(addr);} ==> {err( Buffer read not sync");};... Start wait_for_db_full misc_bus_read_db Stop Error Engler & Masuvathi [1]

9 Model Checking using MC Extraction & Translation protocol code MurΦ Model Eliminates handcrafted system models MurΦ Explicit state enumeration model checker Each reachable state visited once

10 Model Checking using MC Translation Pattern (Metal Printer) State Variable (Metal Slicer) Protocol Code (Implementation) Correctness Properties xg++ Protocol Model Legend: Automatically Generated Hand Crafted by User Program Unedited source code Hardware Model MurØ Start State Error List Lie, Chou, Engler, Dill [2]

11 Example: Model Checking using MC Metal Slicer:... /* Patterns for n/w and proc mesgs, which use length field */ pat sends = { NI_SEND (type, data, keep,swp, wait,nl);} { P1_SEND(type, data, keep,swp, wait,nl);}... Metal Printer... /* Automatically insert length assertion before send */ {N1_SEND(type, data, keep, swap, wait, null);} ==> Lie, Chou, Engler, Dill [2]... { if(mgk_int_cst(data)!= 0)a mgk_e( assert(nh.len = len_data); ); else mgk_e( assert (nh.len = len_nodata); ); mgk_e( ni_send(%t, %t, procnum,nh,type,swap);

12 Model Checking : C Model Checker Motivation xg++ user needs to know system intricacies MurΦ lacks many C constructs Eliminate need of higher level system model Use the implementation code itself Process scheduling & execution by CMC

13 CMC Key Features Searches for all transitions Complete system state saved and restored Heuristics & Optimizations for state space explosion Correctness properties Assertions from implementation code Memory leaks etc. in built User specified invariants Environment Model still handcrafted

14 Case Study 1: FLASH Cache coherence protocol for FLASH Multiprocessor architecture Code runs on each cache miss 5 FLASH protocols Code Size: 10K-18K John Hennessy [3]

15 Results Static Analysis Bugs Found: 33, False Positives: 28 Most rules on buffer management e.g. allocation before use, deallocation after use, not used after deallocation Model Checking 8 bugs e.g. queue overflows, no invalidation of an exclusive line Handwritten model lead to delays

16 Summary Static Analysis Advantage: Better for same set of properties Works best for code visible rules Model Checking Advantage: Can cover complex properties, invariants e.g. counter overflow missed by SA Model Checking Disadvantage: Simple models Less Coverage Detailed model Increased Time & Effort Erroneous model False Positives

17 Case Study 2: AODV Protocol Ad-hoc On-demand Distance Vector Protocol Routing protocol Simplified environment Input: route request, timer interrupt, packet receipt 3 publicly available implementations used Average 6 KLOC Model Checking using CMC

18 Results Model Checking 42 unique bugs, 1 bug in AODV specification Static Analysis: 34 bugs Both Model Checking Static Analysis Generic Properties Protocol Specific Total comparison of bug count classified by properties

19 Comparison Generic Properties: CMC found only one bug not found by SA SA found 13 bugs not found by CMC Protocol Specific Properties: Difficult to check statically 0 bugs SA hit more code but CMC hit more properties!

20 Summary Static Analysis Advantage: Wins when checking same set of properties (again) Model Checking Advantage: Simulates execution of 'entire system' Looks for actual errors (not 'causes' of error) Gives actual execution trace More checks more bugs CMC executed the code

21 Case Study 3: TCP TCP Code size ~ 10 * (AODV code size) Frequently audited, Heavily Tested code Only Model Checking done (using CMC) System tightly coupled with Environment System: TCP stack Environment: Linux Kernel, Kernel Modules Incremental refinement of model Start simple, keep adding complexity

22 Case Study 3: TCP External Functions can be included in System Advantage Model Checker will find errors in them No effort to create stubs Disadvantage Increase in state space Dependencies on other functions Essential Question Where to draw the line between system and environment?

23 System Separation Cut on narrowest interface TCP Kernel module Poorly documented Effectively simulating kernel Months of effort! Difficult to determine false positives New approach: Use well defined interface System calls interface Hardware abstraction layer Effectively running entire kernel in CMC!

24 Results 4 bugs found Metric to measure coverage: Line Coverage: Lines of code covered Protocol Coverage: %age of abstract protocol behavior covered Description Line Coverage Protocol Coverage Bugs Standard server, client 47.4 % 64.7 % 2 + Simultaneous connect 51.0 % 66.7% 0 + Partial Close 52.7 % 79.5% 2 + Message corruption 50.6% 84.3% 0 Combined Coverage 55.4% 92.1%

25 Conclusions No model is as good as implementation itself Simplification, omission missing critical errors Manual work scale badly for complex systems Use well defined interfaces More code coverage More bugs found

26 My Opinion Usage of live software project is a big plus Meta-Compilation & CMC give huge advantage Better comparison if same tools used across case studies Time & Effort not measured accurately No comparison with other existing tools

27 Future Scope How to increase code coverage in meta compilation and CMC? Analysis of class of properties checked

28 References 1)Experiences using static analysis & model checking for bug finding Dawson Engler and Madanlal Masuvathi 2)A simple method for extracting models from protocol code David Lie, Andy Chou, Dawson Engler, David L. Dill 3)Stanford FLASH Multiprocessor: Status, Some Lessons and Plans John Hennessy

29 Q & A