DART: Directed Automated Random Testing. CUTE: Concolic Unit Testing Engine. Slide Source: Koushik Sen from Berkeley

Transcription

1 DAR: Directed Automated Random esting CUE: Concolic Unit esting Engine Slide Source: Koushik Sen from Berkeley

2 Verification and esting We would like to prove programs correct

3 Verification and esting We would like to prove programs correct World prefers to test and validate

4 Verification and esting We would like to prove programs correct World prefers to test and validate Evidence: 50%-80% software development cost goes into testing only Why?

5 Verification and esting We would like to prove programs correct World prefers to test and validate Evidence: 50%-80% software development cost goes into testing only Why? One Reason: esting can expose Unforeseen Behaviors

6 Goals of esting Generate test inputs Execute program on generated test inputs Catch assertion violations Problem: how to ensure that all reachable ments are executed Solution: Explore all feasible execution paths

7 of Programs All Possible Paths Binary tree Non- Conditional Statements Conditional Statements Computation tree Internal node conditional ment execution Edge execution of a sequence of nonconditional ments Each path in the tree represents an equivalence class of inputs

8 Concolic esting Combine random testing (concrete execution) and symbolic testing (symbolic execution) Concrete + Symbolic = Concolic

9 Running Example int double (int v) { return 2*v; void testme (int x, int y) { z = double (y); if (z == x) { if (x > y+10) { ERROR;

10 Running Example int double (int v) { return 2*v; void testme (int x, int y) { N 2*y == x Y z = double (y); if (z == x) { N x > y+10 Y if (x > y+10) { ERROR; ERROR

11 Concolic esting Approach int double (int v) { return 2*v; concrete Concrete symbolic Symbolic path condition void testme (int x, int y) { z = double (y); x = 22, y = 7 x = x 0, y = y 0 if (z == x) { if (x > y+10) { ERROR;

12 Concolic esting Approach int double (int v) { return 2*v; void testme (int x, int y) { concrete Concrete symbolic Symbolic path condition z = double (y); if (z == x) { if (x > y+10) { x = 22, y = 7, z = 14 x = x 0, y = y 0, z = 2*y 0 ERROR;

13 Concolic esting Approach int double (int v) { return 2*v; void testme (int x, int y) { z = double (y); concrete Concrete symbolic Symbolic path condition if (z == x) { 2*y 0!= x 0 if (x > y+10) { ERROR; x = 22, y = 7, z = 14 x = x 0, y = y 0, z = 2*y 0

14 Concolic esting Approach int double (int v) { return 2*v; concrete Concrete symbolic Symbolic path condition void testme (int x, int y) { z = double (y); Solve: 2*y 0 == x 0 Solution: x 0 = 2, y 0 = 1 if (z == x) { 2*y 0!= x 0 if (x > y+10) { ERROR; x = 22, y = 7, z = 14 x = x 0, y = y 0, z = 2*y 0

15 Concolic esting Approach int double (int v) { return 2*v; concrete Concrete symbolic Symbolic path condition void testme (int x, int y) { z = double (y); x = 2, y = 1 x = x 0, y = y 0 if (z == x) { if (x > y+10) { ERROR;

16 Concolic esting Approach int double (int v) { return 2*v; void testme (int x, int y) { concrete Concrete symbolic Symbolic path condition z = double (y); if (z == x) { if (x > y+10) { x = 2, y = 1, z = 2 x = x 0, y = y 0, z = 2*y 0 ERROR;

17 Concolic esting Approach int double (int v) { return 2*v; void testme (int x, int y) { z = double (y); concrete Concrete symbolic Symbolic path condition if (z == x) { if (x > y+10) { x = 2, y = 1, z = 2 x = x 0, y = y 0, z = 2*y 0 2*y 0 == x 0 ERROR;

18 Concolic esting Approach int double (int v) { return 2*v; void testme (int x, int y) { z = double (y); concrete Concrete symbolic Symbolic path condition if (z == x) { if (x > y+10) { 2*y 0 == x 0 x 0 y ERROR; x = 2, y = 1, z = 2 x = x 0, y = y 0, z = 2*y 0

19 Concolic esting Approach int double (int v) { return 2*v; concrete Concrete symbolic Symbolic path condition void testme (int x, int y) { z = double (y); Solve: (2*y 0 == x 0 ) Æ (x 0 > y ) Solution: x 0 = 30, y 0 = 15 if (z == x) { if (x > y+10) { 2*y 0 == x 0 x 0 y ERROR; x = 2, y = 1, z = 2 x = x 0, y = y 0, z = 2*y 0

20 Concolic esting Approach int double (int v) { return 2*v; concrete Concrete symbolic Symbolic path condition void testme (int x, int y) { z = double (y); x = 30, y = 15 x = x 0, y = y 0 if (z == x) { if (x > y+10) { ERROR;

21 Concolic esting Approach int double (int v) { return 2*v; concrete Concrete symbolic Symbolic path condition void testme (int x, int y) { z = double (y); Program Error if (z == x) { if (x > y+10) { 2*y 0 == x 0 x 0 > y ERROR; x = 30, y = 15 x = x 0, y = y 0

22 Explicit Path (not State) Model Checking raverse all execution paths one by one to detect errors assertion violations program crash uncaught exceptions combine with valgrind to discover memory errors

23 Explicit Path (not State) Model Checking raverse all execution paths one by one to detect errors assertion violations program crash uncaught exceptions combine with valgrind to discover memory errors

24 Explicit Path (not State) Model Checking raverse all execution paths one by one to detect errors assertion violations program crash uncaught exceptions combine with valgrind to discover memory errors

25 Explicit Path (not State) Model Checking raverse all execution paths one by one to detect errors assertion violations program crash uncaught exceptions combine with valgrind to discover memory errors

26 Explicit Path (not State) Model Checking raverse all execution paths one by one to detect errors assertion violations program crash uncaught exceptions combine with valgrind to discover memory errors

27 Explicit Path (not State) Model Checking raverse all execution paths one by one to detect errors assertion violations program crash uncaught exceptions combine with valgrind to discover memory errors

28 Directed Search: Summary Dynamic test generation to direct executions along alternative program paths collect symbolic constraints at branch points (whenever possible) negate one constraint at a branch point to take other branch (say b) call constraint solver with new path constraint to generate new test inputs next execution driven by these new test inputs to take alternative branch b check with dynamic instrumentation that branch b is indeed taken Repeat this process until all execution paths are covered May never terminate! Significantly improves code coverage vs. pure random testing

29 DAR for C: Implementation Details prgm.c CIL (Berkeley) dart (OCaml, C) test_driver.c prgm_instrumented.c dart.c C compiler prgm.exe Constraint solver(s) (e.g., lp_solve.so) 3 possible outcomes: Error found Complete coverage Run forever

30 DAR: Success Stories ested a C implementation of a security protocol (Needham- Schroeder) with a known attack About 400 lines of C code; experiments on a Linux 800Mz P-III machine DAR takes less than 2 seconds (664 runs) to discover a (partial) attack, with an unconstrained (possibilistic) intruder model DAR takes 18 minutes (328,459 runs) to discover a (full) attack, with a realistic (Dolev-Yao) intruder model DAR found a new bug in this C implementation of Lowe s fix to the NS protocol (after 22 minutes of search; bug confirmed by the code s author) In contrast, a systematic -space search of this program composed with a concurrent nondeterministic intruder model using VeriSoft (a sw model checker) does not find the attack

31 Limitations Path Space of a Large Program is Huge Path Explosion Problem Entire Computation ree

32 Limitations Path Space of a Large Program is Huge Path Explosion Problem Entire Computation ree Explored by Concolic esting

33 Limitations Path Space of a Large Program is Huge Path Explosion Problem Entire Computation ree How bad is this in Practice? or a simple parser (PL/0) concolic testing failed to generate a valid program within 24 hours. or the vi editor, CUE only tested 3 character long inputs Explored by Concolic esting