Abstract Interpretation

Transcription

1 Abstract Interpretation MATHE MATICAL PROGRAM CHE CKING

2 Overview High level mathematical tools Originally conceived to help give a theoretical grounding to program analysis Useful for other kinds of analyses too Can also be viewed as a very general algorithmic tool for ensuring termination of algorithms Its discoverer (Cousout) is somewhat famous for defining every other analysis or algorithm as an instantiation of abstract interpretation

3 Goal Programs run, but sometimes they do the wrong thing Can we write a program and know before we run it if it will do the right thing? Can we try and answer this question for every possible execution of a program? Can we construct algorithms to help us efficiently answer these questions? The answer to all is: sometimes

4 Mathematical properties Concrete vs abstract interpretation of a program Concrete is easy Write an interpreter Supply concrete values for input / unbound variables Evaluate the program Abstract is harder Concrete semantics are obvious What to do with abstract semantics? How do you know when to stop executing?

5 Abstract interpretation Map program semantics to an abstract machine Evaluate those semantics to produce approximations of all possible program input states Now we can check all possible program states for errors Perhaps For example, abstract integer variables can be represented by ranges instead of integers Abstract integer operations (negation, addition) operate on ranges Now we can prove whether or not a program will divide by 0 With some false positives Type checking is abstract interpretation

6 Abstract Interpretation to build tools Building tools on top of AI has benefits You can be sure that your tools analysis will finish and produce some answer You can be sure that this answer is sound and as accurate as can be expected There are cases where analysis produces no good answers of course The construction of the tool is a modular composition Semantics Data types Standard algorithms

7 Program facts and lattices We want to represent that a variable could contain some combination of values Each value was produced previously in the program However, it could be any combination including none, so, all values, some set of values, one value, or no values We can define a strict ordering between these sets Program facts can be mapped onto lattices The utility of this will be apparent later

8 Example fun foo(a, b) { var j = a + 2; if(b % 2 == 0) { j = j + b + 1; } return j; } b % 2, {a + 2} j + b + 1, {a + 2} {a + 2} {b % 2} j + b + 1, {b % 2} {j + b + 1} At each fact point in the program we assign a set variable that corresponds to one of the sets within the lattice

9 Data Flow Analysis DFA frameworks can construct facts about abstract program states Well understood algorithms and approaches Some implementations for binary programs (REIL) In DFA, analyses produce specific kind of facts Available expressions Live variables Each fact has an abstraction into semantics Each fact s semantics can be applied algorithmically to the program to discover facts about the program At each step, we can compute a gen and a kill set for facts gen fact generated kill fact destroyed

10 High-level DFA algorithm (backward may) 1. Initialize a work list of statements to all statements in the program 2. Set all known facts to empty 3. Iterate over the work list while the work list is not empty 4. Compute gen and kill sets for current statement 5. Are gen and kill for current statement identical to current known facts? If no, update current facts with newly generated facts Add current statement to work list An advantage of mapping facts onto lattices becomes apparent Our analysis algorithm must terminate, the worst case is we drive all facts to bottom

11 Example LUA bytecode LUA has a bytecode that the language is compiled to LUA bytecode interpreter decodes and dispatches Some example bytecode semantics R(1) := Assign the constant 4 to register 1 R(1) := R(2) + R(0) Read from register 2 and register 0, add those values, and store the result in register 1 R(0) := R(1) Read from register 1, assign result to register 0

12 Abstract semantics (gen/kill) for liveness In Ocaml (*[dst, src]*) Move (a,b) -> (* gen b *) let gens = insert b bvs in (* kill a *) let kills = insert a bvs in (gens, kills) After a statement with a move, the source of the move is still live, having been read, but the destination of the move is dead, because it was not read

13 DFA frameworks on binary code BinNavi has a DFA framework that can be applied to x86 programs Internally it defines core data structures Users define facts in a lattice Users provide gen/kill transfer functions Framework iteratively applies user supplied transfer functions until lattice facts converge Allows you to do neat things like liveness analysis on registers In practice, without a mature view of memory, this is limited Developing a mature view of memory is an open research problem

14 Generic DFA shortcomings Generally, DFA has been considered to operate on source languages that have concepts of variables When your only variables are registers, but you have memory locations to load/store, this is a problem Generic DFA considers load/store as killing all statements in the program, sending them to top An alternative requires a whole-program points-to analysis and this is hard Extraction of variables from programs aims to resolve this somewhat Programs that make heavy use of memory still problematic for traditional DFA on source code

15 Creating an Intermediate Representation PRINCIPLE D PROGRAM ANALYSIS

16 Goals As we ve seen before, analysis can be principled when the actions of a program are defined as operations on an abstract or virtual machine This is useful for both source and binary analysis Compiler intermediate languages refine input languages into an unambiguous syntax and semantics Compiler intermediate languages allow for optimizations that remove and simplify code Both of these are advantages to binary analysis There are precision tradeoffs, usually in how much of the abstract machine we define We want An expressive IR A virtual machine specification expressive enough to talk about programs we care about A translator or frontend that produces our IR

17 Know the Domain What does our real CPU look like? X86 ABI It s not the ABI we need, but the ABI we deserve Integer-value general purpose registers Floating point Memory key-value store Fixed width integer keys mapping to variable width integer values Program code also stored in memory Memory has page permissions Segment selectors Sometimes there are interrupts

18 Abstract Binary Domain This gets really complicated when we consider the full spectrum of X86 behavior What about multi-threaded memory semantics w.r.t the cache? Or HTM instructions? What about self-modifying code? Pick some subsets of the domain and implement them first, and work incrementally BAP has been under development for almost a decade, still has no floating point support Some domains don t allow the expression of self-modifying code through immutable code These are choices you make at the language and CPU abstraction level to decide how accurate your translations are Compromises admit some programs, disallow others If admitted programs are programs you are interested in, you win

19 Runtime and Operating System Domain Knowing stuff about the CPU is great, what about the OS? What are the semantics of system/library calls? malloc returns a pointer to new memory mprotect changes page permissions These platform semantics can be just as important as instruction semantics This kind of platform modeling is a problem when doing source code analysis

20 Language Languages define semantics to map to We create operators that are pure that define the bit-wise operations performed at the ISA level xor, shift, sub, etc The language operates on values defined within your CPU or abstract domain It also operates on memory locations defined in your CPU or abstract domain You can give properties to your language that make analysis easier or harder SSA Side-effect free Exceptions or lack therof

21 Translators Produce language from native instructions When abstractions really leak, they leak here first Translating basic operations like add, sub, are easy What about instructions like iret, sysenter, or in/out? This depends on the way you have defined the abstract machine In good designs, a translator just applies a mapping from native instructions into a language previously defined Usually, translators apply heuristics and hacks to make things make sense

22 Previous Intermediate Representations Compiler hackers and PL scientists approaching the binary analysis problem have had a lot of thoughts about this before Many existing tools for representing binary code as an intermediate language exist There are two big undertakings Defining a virtual machine with a language that programs the virtual machine Translating binary code into the language for this virtual machine Existing tools tend to do both BAP INSIGHT/BINCOA REIL/RREIL TSL/MTLK

23 BAP BIL Dawn Song and David Brumley, University of Berkeley and Carnegie Mellon University OCaml Translates native code into two different types of IL One is SSA, the other is not Analysis and optimization passes written on top of BIL Symbolic execution system written on top of BIL

24 INSIGHT Emmanuel Fleury at LaBRI in Bordeaux C++ Translates x86 and ARM into an intermediate representation On the backend, uses libopcodes and YACC to produce IR Performs control flow recovery using recursive descent

25 REIL/RREIL Zynamics / Google Bindings in Java and Python Produces variables Micro-instructions Uses IDA for CFG recovery Extended into RREIL by Simon RREIL produces 1-bit flag variables

26 TSL/MLTK TSL Transformer Specification Language, Tom Reps, only described in a tech report MLTK Open source, Axel Simon, Domain specific languages to describe semantics of native code abstractly MTLK also includes a blueprint for decoding instructions

27 Static Single Assignment (SSA) A code representation strategy where there are no variables Each value produced by computation is a defined-once value Values are never re-defined Usually, values are unique per function SSA still permits a view of memory into which you can load and store

28 SSA and Loops F-nodes If values are defined-once, how do we represent variables in loops? We create an instruction called F This instruction defines a value based off of control flow If the previous block was A, F produces a SSA, with F nodes, allows for some immediate and trivial analysis to determine loop-carried values SSA can also approximate data flow analysis SSA value def-use chains can answer questions about live variables and available expressions

29 LLVM A compiler mid-end Allows parsers to produce LLVM assembly code from an AST Analyzes and optimizes this assembly code in a pluggable and configurable system Supports code generation of the assembly code to native machine code Lots of canned analysis algorithms that you can throw at code Alias analysis Loop information Dominance and post-dominance frontiers

30 Representation Challenges A straight decoding will give an analysis some new information about machine code Registers used Control flow Memory cells accessed This is one step closer to a de-compilation, but there are some pieces of information missing Full control flow recovery is difficult in the general case Promoting memory cell accesses into variables is difficult Involves answering aliasing questions Your representation has some design choices Faithful to an evaluation at the expense of readability More expressive of inferred intent at the expense of soundness

31 Why be sound anyway? Soundness could be overrated, it depends on your goals What if your goal is program re-writing to remove bugs? Then soundness is very good and a lack of it is very disturbing You could re-write programs incorrectly On a good day, this means that you don t remove a bug from a program when you want to On a bad day, this means that the program you re-write no longer works Soundness errors in program reconstruction result in nonworking programs

32 What about bug identification? Very different An unsound program representation could tell you false things It could also tell you true things The true things it tells you could be information about vulnerabilities in the program If what the representation tells you is a lie though, this vulnerability is a false alarm What to do about false alarm vulnerabilities? Can always test the faulting input and see if it s real False positives are annoying, but can be verified or checked after the fact Many static analysis systems for security permit false positives Worst case scenario Abstraction lies about detail, creates false negatives, results in finding no bugs

33 Variable Recovery Some current best work Anand, EUROSYS Balakrishnan Value-Set Analysis (VSA) Linear Stack Accesses Implemented in IDA/HexRays Doesn t always work totally right Some inherent problems Array accesses Compilers can re-use stack locations

34 Type Inference and Recovery Closely related to variable recovery is type recovery Area of active research interest Some cases seem easy Common idiom: moving a low-valued, non-zero integer into a register means that register has type integer Testing a register against 0 could mean that register is a pointer Some actually are easy If a register is used in a memory expression alone, it is a pointer type What about the combination of two registers? Which is the base and which is the offset? Nobody has a good system that works all the time on this

35 Some Demos