Practical Record And Replay Debugging With rr. Robert O'Callahan

Transcription

1 Practical Record And Replay Debugging With rr Robert O'Callahan

2 Debugging nondeterminism

3 Deterministic hardware

4 Sources of nondeterminism

5 Record inputs

6 Replay execution

7 Old idea PinPlay Nirvana Jockey Chronomancer Scribe CLAP PANDA FlashBack ReTrace ReVirt ReSpec ODR Echo QuickRec

8 rr goals l l l l Easy to deploy: stock hardware, OS Low overhead Works on Firefox Small investment

9 rr design Linux process boundary System call results Signals Deterministic user-space CPU execution Observable effects Record and replay

10 No code instrumentation mov EDX, tls.ebp mov ECX, tls call MemReadCallback mov EAX, [EDX]

11 Use modern HW/OS features System call results ptrace Signals ptrace Shared memory data races Limit to single core Asynchronous event timing HW performance counters Trap on a subset of system calls seccomp-bpf Notification when system call blocks in the kernel DESCHED perf events Cheap block copies FIOCLONERANGE

12 ptrace rr before_syscall Kernel read() after_syscall... record results...

13 Use modern HW/OS features System call results ptrace Signals ptrace Shared memory data races Limit to single core Asynchronous event timing HW performance counters Trap on a subset of system calls seccomp-bpf Notification when system call blocks in the kernel DESCHED perf events Cheap block copies FIOCLONERANGE

14 CPU1 CPU0 Data races Memory

15 CPU0 Data races

16 Use modern HW/OS features System call results ptrace Signals ptrace Shared memory data races Limit to single core Asynchronous event timing HW performance counters Trap on a subset of system calls seccomp-bpf Notification when system call blocks in the kernel DESCHED perf events Cheap block copies FIOCLONERANGE

17 Event timing: HW perf counters alarm() Measure progress Instructions executed! SIGALRM Retired conditional branches (Intel) Zero overhead

18 Use modern HW/OS features System call results ptrace Signals ptrace Shared memory data races Limit to single core Asynchronous event timing HW performance counters Trap on a subset of system calls seccomp-bpf Notification when system call blocks in the kernel DESCHED perf events Cheap block copies FIOCLONERANGE

19 Accelerating system calls rr before_syscall Kernel read() after_syscall... record results...

20 Avoid context switches librrpreload.so shim_read() Kernel read()... record results... Suppress ptrace trap Use seccomp-bpf predicates

21 Blocking system calls librrpreload.so shim_read() Kernel read()... record results... Blocks?

22 Blocking system calls read()... kernel thread 1 rr DESCHED perf event thread 2

23 Other issues RDTSC RDRAND XBEGIN/XEND CPUID

24 Overhead relative to baseline rr Overhead Record Replay Single Core cp octane htmltest sambatest Workload

25 Reverse-execution Debugging

26 Lessons Replay performance matters Session-cloning performance matters Cloning processes via fork() seems cheaper than e.g. cloning VM state

27 Lessons In-process system-call interception is fragile applications make syscalls in strange states (bad TLS, insufficient stack, etc) in-process interception code could be accidentally or maliciously subverted move this part into kernel?

28 OS design implications Recording boundary should: be stable, simple, documented API boundary also be a boundary for hardware performance counter measurement Linux kernel/user boundary is this (mostly) Windows kernel/user boundary is not

29 ARM retry: LDREX r0,[addr] ADD r0,1 hardware interrupt??? STREX r1,r0,[addr] CMP r1,0 BNE retry Need hardware support to detect/compensate Or binary rewriting

30 Related work VM-level replay heavyweight ReVirt, VMWare, QEMU (PANDA), Xen Kernel-supported replay hard to maintain Scribe, dos, Arnold Pure user-space replay instrumentation, higher overhead PinPlay, idna, UndoDB Higher-level replay more limited scope Chronon, Dolos, Chakra, R2 Parallel replay more limited scope, higher overhead SMP-ReVirt, DoublePlay, ODR, Castor Hardware-supported parallel replay nonexistent hardware FDR, BugNet, DeLorean, QuickRec

31 Conclusions rr s approach delivers a lot of value More research needed for multicore approaches Lots of unexplored applications of record+replay

32