Debugging programs. Khoo Yit Phang October 24, 2012

Transcription

1 Debugging programs Khoo Yit Phang October 24,

2 Ideal week of a CS430 student Write code for assignment Compile code into program??? Profit! (Or get an A ) 2

3 More realistic week of a CS430 student Write code for assignment Compile code into program Run tests on program find a misbehavior track down cause of misbehavior make change to code and recompile Debugging can take a surprisingly large amount of time and effort! test and repeat until fixed (or time runs out) Maybe profit? (Maybe get an A?) 3

4 Here s a program #include <stdio.h> void echo(void) { char buffer[80]; int i = 0; while (1) { Read one character int c = getchar(); if (c == EOF) return 0; else if (c == '\n') { buffer[i] = 0; puts(buffer); i = 0; } else { buffer[i++] = c; } Quit if EOF Print buffer and reset if newline Buffer character otherwise } } Print Done int main(int c, char *v[]) { echo(); puts("done"); return 0; } 4

5 There s a bug %./myecho < loremipsum.txt Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. Suspendisse potenti. Etiam egestas est ut mauris blandit iaculis. Aliquam erat volutpat. Etiam a feugiat metus. Sed vitae tortor vitae metus molestie ornare. Phasellus eleifend pellentesque consectetur. Proin lacinia nisl ac sem bibendum vitae sodales elit venenatis. Segmentation fault % Bug report: I ran myecho on this file, and it terminated abruptly without printing Done! 5

6 How to find the bug? With limited tools (e.g., embedded systems): Stare at code Trace the code by hand With a compiler and I/O: printf print the internal state of the program assert check the internal state of the program 6

7 Using printf #include <stdio.h> void echo(void) { } } char buffer[80]; int i = 0; while (1) { int c = getchar(); printf("c=%d, i=%d\n", c, i); if (c == EOF) return 0; else if (c == '\n') { buffer[i] = 0; puts(buffer); i = 0; } else { buffer[i++] = c; } int main(int c, char *v[]) { echo(); puts("done"); return 0; } 7 Print every character and the value of i

8 Run with printf %./myecho < loremipsum.txt c=76, i=0 many lines c=10, i=72 Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod many lines c=10, i=152 i exceeded 80 (the buffer size)! cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. Suspendisse potenti. Etiam egestas est ut mauris blandit iaculis. many lines c=-1, i=0 Segmentation fault % 8

9 Using assert #include <stdio.h> #include <assert.h> void echo(void) { char buffer[80]; int i = 0; while (1) { int c = getchar(); assert(i < 80); if (c == EOF) return 0; else if (c == '\n') { buffer[i] = 0; puts(buffer); i = 0; } else { buffer[i++] = c; } } } int main(int c, char *v[]) { echo(); puts("done"); return 0; } 9 Make sure that i does not exceed the buffer

10 Run with assert %./myecho < loremipsum.txt Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat Assertion failed: (i < 80), function foo, file myecho.c, line 6. % Quit early due to failed assertion 10

11 Pros/cons of printf/assert Pros: Applicable to any language Easy to use Cons: Need to recompile Clutter code Affect performance, change program behavior printf generate too much data assert few kinds of properties can be checked easily 11

12 Interactive debugger Explore program execution without recompiling code Run program in a controlled environment Examples: GDB native binary (e.g., C/C++) ocamldebug Ocaml jdb Java debugger 12

13 GDB demo 13

14 GDB demo recap Compile code with debugging info/no optimization gcc -g -O0 -o myprog.c myprog GDB commands list loc show source code at loc run run the program from the beginning to the end step/next/continue run one line/function/until the end breakpoint loc stop when program reaches loc watchpoint var stop when program changes var print expr print value of expr disassemble/m loc show assembly code at loc 14

15 Debugger basics Control the program using ptrace API (Linux) pid = fork() if (pid > 0) { // child process ptrace(pt_traceme, 0, 0, 0); execv(program_name, program_args); } else { // parent process ptrace(ptrace_attach, pid, 0, 0); ptrace(ptrace_singlestep, pid, 0, 0); ptrace(ptrace_cont, pid, 0, 0); ptrace(ptrace_peekdata, pid, addr, 0); ptrace(ptrace_pokedata, pid, addr, data); } 15

16 Debugger basics cont d Software breakpoints/watchpoints (x86) Breakpoints poke int 3 instruction into program Watchpoints single-step and compare (very slow) Hardware breakpoints/watchpoints (x86) x86 debug registers (no overhead, but limited) 16

17 Scriptable debugging Write scripts to automate debugging tasks Program-specific: How many times is foo called? Is foo, bar, baz called in this sequence? Generic: Is free(addr) called more than once for any addr? Is the call stack consistent before and after a function call? Do any two threads access var without consistent locks? 17

18 Scripting GDB GDB supports scripting using Python Count calls to foo: Set breakpoint at foo foo = gdb.breakpoint("foo"); count = 0; more = True def stop_handler(evt): if isinstance(evt, gdb.breakpointevent) and foo in evt.breakpoints: global count; count += 1 def exit_handler(evt): global more; more = False gdb.events.stop.connect(stop_handler) gdb.events.exited.connect(exit_handler) gdb.execute("start") while more: gdb.execute("continue") gdb.events.exited.disconnect(exit_handler) gdb.events.stop.disconnect(stop_handler) foo.delete() Function to count when stopped at foo Function to detect when program exited Run the program to main Delete the breakpoint 18 Register handler functions with GDB Keep running until program exited Deregister handler functions from GDB

19 GDB scripting demo 19

20 Pros/cons of GDB scripting Pros: Automate complex debugging tasks Reusable Cons: Imperative, callback-style (event handlers) is tedious Not easy to compose multiple scripts: have to be careful to avoid conflicting event handlers Difficult to deal with non-determinism (e.g., random numbers or thread scheduling) 20

21 Time-travel debugging Time-travel debuggers that allow you to inspect past program state Examples: GDB record/replay experimental, very slow ocamldebug UndoDB commercial time-travel GDB extension 21

22 ocamldebug demo 22

23 ocamldebug demo recap Compile code with debugging info ocamlc -g -o myprog myprog.ml Ocamldebug commands: run/step/next run program forward reverse/backstep/previous - run program backwards goto time jump to time break loc stop when program reaches loc print var print the value of var 23

24 Pros/cons of time-travel debugging Pros: Able to explore the entire program execution Removes nondetermism (within a single execution) Cons: Even more state to look at Hard to script using imperative, callback-style 24

25 Time-travel implementation basics Omniscient debuggers Log the program state after every instruction Very slow to run, large log file, quick lookup using indexes Replay debuggers Log only non-deterministic instructions (e.g., system calls), and checkpoint program state occasionally Re-run the program, feeding it input from the log Much faster to run, small log file, slower lookup 25

26 Expositor: scriptable time-travel debugging with first-class traces Khoo Yit Phang, Jeffrey S. Foster, Michael Hicks Under submission to ICSE

27 Key ideas Treat program execution as if it were a timeindexed list of program state snapshots Use common list operations such as filter/map/scan Use laziness extensively for efficiency/ interactivity Trace lazy, sparse, immutable interval tree representing (projections) of the program execution Edit hash array-mapped-trie (EditHAMT) lazy, immutable set/map (multiset/multimap) 27

28 Simple Expositor example Implemented using Python in UndoDB/GDB Create a trace of calls to foo: foo = execution.breakpoints("foo") Global object representing program execution Count calls to foo: count = len(foo) Much simpler than the earlier Python GDB script Find the previous snapshot at foo before time 100: snapshot = foo.get_before(100) Create a trace of x values when foo is called: foo_x = foo.map(lambda s: s.read_var("x")) 28

29 Simple trace operations Queries: trace. len () number of items, called by len(trace) trace.get_at(t)/get_after(t)/get_before(t) item at/after/before time t Simple operations: trace. filter(p) new trace of items for which p returns true trace. map(f) new trace with f applied to all items trace. slice(t0, t1) new trace of items from time t0 to t1 No add/remove traces contents completely defined at construction 29

30 Trace data structure Dotted boxes are lazy This query traverses a lazy node for the first time, so compute the contents Lazy node since search ended 0 50 foo = execution.breakpoints("foo") 0 snapshot = foo.get_before(100) foo Nodes are annotated with time intervals 50.1 Lazy node since this part is not needed to answer the query 100 UndoDB found foo at time 50 Search UndoDB starting at time

31 Merging between traces tr0 tr1 f tr0.merge(f, tr1) tr0 tr1 tr0 tr1 None f f f f f f f f None tr0.trailing_merge(f, tr1) tr0.rev_trailing_merge(f, tr1) 31

32 Computing within traces scan is fold over prefixes tr tr acc f f f tr.scan(f, acc) tr f f f acc tr.rev_scan(f, acc) tr tscan: tree-scan using associative f f f f f f f f f tr.tscan(f) tr.rev_tscan(f) 32

33 Lazy trace operations E.g., count prior calls to foo up until foo(0): def count_nonzero_foo(lazy_acc, snapshot): if snapshot.read_var("x") > 0: return lazy_acc.force() + 1 else: return 0 nonzero_foo = foo.scan(count_nonzero_foo, 0) add 1 to accumulator when x!= 0 reset the accumulator to 0 otherwise For every succeeding item in foo with initial accumulator 33

34 Lazy trace operations, cont d E.g., count prior calls to foo up until foo(0): def count_nonzero_foo(lazy_acc, snapshot): if snapshot.read_var("x") > 0: return lazy_acc.force() + 1 else: The accumulator is lazy! Why? return 0 nonzero_foo = foo.scan(count_nonzero_foo, 0) By definition of scan: accn = f (inn, accn-1) If the accn-1 were not lazy, then every input must be computed, compromising laziness 34

35 Expositor demo 35

36 Using sets/maps in Expositor Many useful analysis require sets/maps What s wrong with built-in Python sets? def collect_foo_args(lazy_acc, snapshot): return lazy_acc.force() \.union(set([ snapshot.read_var("x") ])) foo_args = foo.scan(set(), collect_foo_args) Laziness is compromised: have to force accumulator to apply union! Also, each output must be a different set, but union does so by deep copying! 36

37 Using sets/maps in Expositor, cont d Need a lazy set/map where elements can be added/removed without knowing the contents of prior sets/maps! E.g., lazy linked-list: def collect_foo_args(lazy_acc, snapshot): return ( snapshot.read_var("x"), lazy_acc ) foo_args = foo.scan(none, collect_foo_args) 37 Tuple of x as head, unforced lazy_acc as tail Issues with lazy linked-list: inefficient O(n) for lookup how to handle removal?

38 EditHAMT Lazy, immutable set/map (multiset/multimap) Combination of: EditList lazy linked-list of set/map edit operations lazy variant of hash array-mapped-trie (HAMT) hashtable/tree hybrid set/map 38

39 EditList Lazy linked-list based set/map with removal lazily append nodes tagged with add / remove Dotted lines indicate lazy tail None "add", a 0 "remove", a 0 "add", a 1 False! find(a0) { a0 } { a1 } Space efficient: can be viewed as a succession of sets 39

40 HAMT Hashtable/tree hybrid Chained hashtable HAMT bucket array array mapped trie (AMT) a type of trie buckets (e.g., linked-list) nested HAMTs with different hash functions 40

41 AMT Trie-based map from integer keys to values AMT lookup, e.g., find(7) 7 = binary Lookup are O(1) :c Internal nodes are arrays Leaf nodes are bindings 5:a 7:b Found! 41

42 LazyAMT Lazy variant of AMT Nodes lazily created during lookup Internal nodes are lazy sparse arrays 50:c 5:a 7:b 57:d Add 57:d by creating a new LazyAMT that updates the old binding (57:None) 42 57:None update

43 EditList + lazy HAMT = EditHAMT Chained hashtable HAMT EditHAMT Bucket array AMT LazyAMT Buckets nested HAMTs EditLists 43

44 EditHAMT example Lookup are O(1) on average Dotted lines indicate lazy updates None None "add", a 0 "remove", a 0 "add", a 1 { a0 } { a1 } Space efficient: can be viewed as a succession of sets 44

45 Case study 1 Reverse engineering stack-smashing attack (buffer overflow exploit) all_call_rets.trailing_merge(compare_call_stack, all_call_rets) For every function call or return find the immediately preceding call or return and compare the call stacks to ensure consistency 45

46 Case study 2 Firefox data race Trace of garbage collector behavior Trace of memory allocation behavior (a) (b) (c) C: gc_call = execution.breakpoints("js_gc") R: gc_return = execution.breakpoints("js_gc", index=-1) C R C R C R C R C R M: mmap2 = execution.syscalls("mmap2") U: munmap = execution.syscalls("munmap") M 70 M U 70 U timer_trace = set_tracing(a=timer-create, R=timer-fire) A R A R A R (d) (e) chunkswaiting_trace = execution.watchpoints(gcchunkswaitingtoexpire-variable).map(read-gcchunkswaitingtoexpire) chunkswaiting_hb = one_lock(r=gcchunkswaitingtoexpire-read, W=gcChunksWaitingToExpire-write, locks, unlocks) R W t scroll1 t scroll2 t end Trace of data races Trace of timer events 46