Playing with process tracing to instrument static function at runtime in EZTrace. Damien Martin-Guillerez SED

Transcription

1 Playing with process tracing to instrument static function at runtime in EZTrace Damien Martin-Guillerez SED CENTRE Inria BORDEAUX SUD-OUEST

2 INTRODUCTION EZTrace is a performance trace generator for parallel programs - 1

3 INTRODUCTION EZTrace is a performance trace generator for parallel programs It relies on the dynamic library preloading mechanism to intercept function calls and insert event logs - 1

4 INTRODUCTION EZTrace is a performance trace generator for parallel programs It relies on the dynamic library preloading mechanism to intercept function calls and insert event logs As a consequence, no instrumentation of static function, i.e. no trace for them - 1

5 INTRODUCTION EZTrace is a performance trace generator for parallel programs It relies on the dynamic library preloading mechanism to intercept function calls and insert event logs As a consequence, no instrumentation of static function, i.e. no trace for them PROCESS TRACING - 1

6 INTRODUCTION EZTrace is a performance trace generator for parallel programs It relies on the dynamic library preloading mechanism to intercept function calls and insert event logs As a consequence, no instrumentation of static function, i.e. no trace for them PROCESS TRACING - 1

7 OUTLINE 1. EZTrace insights 2. The ptrace() system call 3. Instrumentation of static functions in EZTrace 4. Technical details for the geek 5. Other architectures - 2

8 1EZTrace Insights - 3

9 EZTrace basics EZTrace is a trace generator: $ export EZTRACE_TRACE= mpi pthread $ mpirun -np 16 `eztrace -e mpiapp` or $ eztrace pthreadapp - 4

10 EZTrace basics EZTrace is a trace generator: Look out for some events during execution and record them in a file $ export EZTRACE_TRACE= mpi pthread $ mpirun -np 16 `eztrace -e mpiapp` or $ eztrace pthreadapp EZTrace log files (/tmp/<username>_eztrace_log_rank_<rank>) - 4

11 EZTrace basics EZTrace is a trace generator: Look out for some events during execution and record them in a file Convert them to a standard format $ export EZTRACE_TRACE= mpi pthread $ mpirun -np 16 `eztrace -e mpiapp` or $ eztrace pthreadapp EZTrace log files (/tmp/<username>_eztrace_log_rank_<rank>) $ eztrace_convert -o myfile.paje /tmp/<username>_eztrace_log* - 4

12 EZTrace basics EZTrace is a trace generator: Look out for some events during execution and record them in a file Convert them to a standard format $ export EZTRACE_TRACE= mpi pthread $ mpirun -np 16 `eztrace -e mpiapp` or $ eztrace pthreadapp EZTrace log files (/tmp/<username>_eztrace_log_rank_<rank>) $ eztrace_convert -o myfile.paje /tmp/<username>_eztrace_log* Paje File (or OTF) - 4

13 EZTrace basics EZTrace is a trace generator: Look out for some events during execution and record them in a file Convert them to a standard format View the events $ export EZTRACE_TRACE= mpi pthread $ mpirun -np 16 `eztrace -e mpiapp` or $ eztrace pthreadapp EZTrace log files (/tmp/<username>_eztrace_log_rank_<rank>) $ eztrace_convert -o myfile.paje /tmp/<username>_eztrace_log* Paje File (or OTF) Visualizer (ex.: ViTE) - 4

14 EZTrace basics EZTrace is a trace generator: Look out for some events during execution and record them in a file Convert them to a standard format View the events $ export EZTRACE_TRACE= mpi pthread $ mpirun -np 16 `eztrace -e mpiapp` or $ eztrace pthreadapp EZTrace log files (/tmp/<username>_eztrace_log_rank_<rank>) ON DYNAMIC LIBRARIES! $ eztrace_convert -o myfile.paje /tmp/<username>_eztrace_log* Paje File (or OTF) Visualizer (ex.: ViTE) - 4

15 EZTrace is good for your health You can use EZTrace to trace parallel programs in order to: Detect problems in message exchanges and in parallel algorithm Improve communication patterns Show beautiful traces of your software to the world - 5

16 EZTrace is good for your health You can use EZTrace to trace parallel programs in order to: Detect problems in message exchanges and in parallel algorithm Improve communication patterns Show beautiful traces of your software to the world You probably should not use EZTrace to: Detect general bugs like memory faults Trace non-parallel programs Show that you program has ugly traces - 5

17 EZTrace is good for your health You can use EZTrace to trace parallel programs in order to: Detect problems in message exchanges and in parallel algorithm Improve communication patterns Show beautiful traces of your software to the world You probably should not use EZTrace to: Detect general bugs like memory faults Trace non-parallel programs Show that you program has ugly traces You cannot (currently) use EZTrace to: Stop when a memory error occurs Trace your own library or non-packaged library Trace static libraries Make the coffee - 5

18 Dynamic library loading libmpi.so./toto MPI_Send() MPI_Send: - 6

19 Dynamic library preloading With an EZTrace module LD_PRELOAD= libeztrace_mpi.so./toto MPI_Send() libmpi.so MPI_Send: - 6

20 Dynamic library preloading With an EZTrace module LD_PRELOAD= libeztrace_mpi.so./toto MPI_Send() libmpi.so MPI_Send: libeztrace_mpi.so - 6

21 Dynamic library preloading With an EZTrace module LD_PRELOAD= libeztrace_mpi.so./toto MPI_Send() libmpi.so MPI_Send: libeztrace_mpi.so MPI_Send: - 6

22 Dynamic library preloading With an EZTrace module LD_PRELOAD= libeztrace_mpi.so./toto MPI_Send() libmpi.so MPI_Send: libeztrace_mpi.so MPI_Send: LOG_EVENT( In MPI_Send, ) orig_mpi_send( ) LOG_EVENT( Out MPI_Send, ) - 6

23 Dynamic library preloading With an EZTrace module LD_PRELOAD= libeztrace_mpi.so./toto MPI_Send() libmpi.so MPI_Send: libeztrace_mpi.so orig_mpi_send = MPI_Send: LOG_EVENT( In MPI_Send, ) orig_mpi_send( ) LOG_EVENT( Out MPI_Send, ) - 6

24 Dynamic library preloading With an EZTrace module LD_PRELOAD= libeztrace_mpi.so./toto MPI_Send() libmpi.so MPI_Send: libeztrace_mpi.so orig_lib = dlopen( libmpi.so, ) orig_mpi_send = dlsym(orig_lib, MPI_Send ) MPI_Send: LOG_EVENT( In MPI_Send, ) orig_mpi_send( ) LOG_EVENT( Out MPI_Send, ) - 6

25 EZTrace basics EZTrace is a trace generator: Look out for some events during execution and record them in a file Convert them to a standard format View the events $ export EZTRACE_TRACE= mpi pthread $ mpirun -np 16 `eztrace -e mpiapp` or $ eztrace pthreadapp EZTrace log files (/tmp/<username>_eztrace_log_rank_<rank>) ONLY ON DYNAMIC LIBRARIES! $ eztrace_convert -o myfile.paje /tmp/<username>_eztrace_log* Paje File (or OTF) Visualizer (ex.: ViTE) - 7

26 It is the story of a fool that says to another fool who says to another fool that - 8

27 2The ptrace() system call - 9

28 ptrace() stands for process tracing ptrace() is: A POSIX system call Extended in its Linux version A process tracing mechanism - Attach and detach a target process - Stop at each system call / each instruction of target process - Examine and modify target memory and registers #include <sys/ptrace.h> int ptrace(int requête, int pid, void* addr, void* data); man 2 ptrace - 10

29 ptrace() process attachment An attached process can be traced - Stop at each system call / each instruction of target process - Examine and modify target memory and registers Two methods: - 11

30 ptrace() process attachment An attached process can be traced - Stop at each system call / each instruction of target process - Examine and modify target memory and registers tracer pid = fork() target Two methods: - fork() and PTRACE_TRACEME - 11

31 ptrace() process attachment An attached process can be traced - Stop at each system call / each instruction of target process - Examine and modify target memory and registers tracer pid = fork() target Two methods: - fork() and PTRACE_TRACEME execve(./target, ) - 11

32 ptrace() process attachment An attached process can be traced - Stop at each system call / each instruction of target process - Examine and modify target memory and registers wait(null) tracer pid = fork() target Two methods: - fork() and PTRACE_TRACEME execve(./target, ) - 11

33 ptrace() process attachment An attached process can be traced - Stop at each system call / each instruction of target process - Examine and modify target memory and registers wait(null) tracer pid = fork() target ptrace(ptrace_traceme, 0,0,0) Two methods: - fork() and PTRACE_TRACEME execve(./target, ) - 11

34 ptrace() process attachment An attached process can be traced - Stop at each system call / each instruction of target process - Examine and modify target memory and registers wait(null) tracer pid = fork() target ptrace(ptrace_traceme, 0,0,0) Two methods: - fork() and PTRACE_TRACEME SIGTRAP execve(./target, ) - 11

35 ptrace() process attachment An attached process can be traced - Stop at each system call / each instruction of target process - Examine and modify target memory and registers wait(null) tracer pid = fork() target ptrace(ptrace_traceme, 0,0,0) Two methods: - fork() and PTRACE_TRACEME - PTRACE_ATTACH SIGTRAP execve(./target, ) tracer target ptrace(ptrace_attach, pid,0,0) - 11

36 ptrace() process attachment An attached process can be traced - Stop at each system call / each instruction of target process - Examine and modify target memory and registers wait(null) tracer pid = fork() target ptrace(ptrace_traceme, 0,0,0) Two methods: - fork() and PTRACE_TRACEME - PTRACE_ATTACH SIGTRAP execve(./target, ) Detaching using PTRACE_DETACH tracer target ptrace(ptrace_attach, pid,0,0) - 11

37 ptrace() stepping Continue until next signal ptrace(ptrace_cont, pid, NULL, NULL); ptrace(ptrace_cont, pid, NULL, (void*)_signal); - 12

38 ptrace() stepping Continue until next signal ptrace(ptrace_cont, pid, NULL, NULL); ptrace(ptrace_cont, pid, NULL, (void*)_signal); Continue until next system call - Stops at system call entry and exit ptrace(ptrace_syscall, pid, NULL, NULL); ptrace(ptrace_syscall, pid, NULL, (void*)_signal); - 12

39 ptrace() stepping Continue until next signal ptrace(ptrace_cont, pid, NULL, NULL); ptrace(ptrace_cont, pid, NULL, (void*)_signal); Continue until next system call - Stops at system call entry and exit ptrace(ptrace_syscall, pid, NULL, NULL); ptrace(ptrace_syscall, pid, NULL, (void*)_signal); Stop after next instruction ptrace(ptrace_singlestep, pid, NULL, NULL); ptrace(ptrace_singlestep, pid, NULL, (void*)_signal); - 12

40 ptrace() memory reading and altering Reading / writing target memory type_of_word data; data = ptrace(ptrace_peektext, pid, (void*)addr, NULL); ptrace(ptrace_poketext, pid, (void*)addr, (void*)data); - 13

41 ptrace() memory reading and altering Reading / writing target memory type_of_word data; data = ptrace(ptrace_peektext, pid, (void*)addr, NULL); ptrace(ptrace_poketext, pid, (void*)addr, (void*)data); Reading / writing target registers #include <sys/user.h> struct user_regs_struct regs; ptrace(ptrace_getregs, pid, NULL, (void*)(&regs)); ptrace(ptrace_setregs, pid, NULL, (void*)(&regs)); - 13

42 ptrace() memory reading and altering Reading / writing target memory type_of_word data; data = ptrace(ptrace_peektext, pid, (void*)addr, NULL); ptrace(ptrace_poketext, pid, (void*)addr, (void*)data); Reading / writing target registers #include <sys/user.h> struct user_regs_struct regs; ptrace(ptrace_getregs, pid, NULL, (void*)(&regs)); ptrace(ptrace_setregs, pid, NULL, (void*)(&regs)); Reading / writing target user data #include <sys/user.h> struct user u; ptrace(ptrace_peekuser, pid, NULL, (void*)(&u)); ptrace(ptrace_pokeuser, pid, NULL, (void*)(&u)); - 13

43 ptrace() memory reading and altering Reading / writing target memory type_of_word data; data = ptrace(ptrace_peektext, pid, (void*)addr, NULL); ptrace(ptrace_poketext, pid, (void*)addr, (void*)data); Reading / writing target registers #include <sys/user.h> struct user_regs_struct regs; ptrace(ptrace_getregs, pid, NULL, (void*)(&regs)); ptrace(ptrace_setregs, pid, NULL, (void*)(&regs)); Reading / writing target user data #include <sys/user.h> struct user u; ptrace(ptrace_peekuser, pid, NULL, (void*)(&u)); ptrace(ptrace_pokeuser, pid, NULL, (void*)(&u)); Reading target signal informations siginfo_t siginfo; data = ptrace(ptrace_getsiginfo, pid, NULL, (void*)(&siginfo)); - 13

44 3Instrumentation of static functions in EZTrace - 14

45 How to instrument static functions? Target program call MPI_Send MPI_Send: opcode1 opcode2 opcode3-15

46 How to instrument static functions? Target program call MPI_Send MPI_Send: opcode1 opcode2 opcode3 Prolog/epilog library prolog: epilog: - 15

47 How to instrument static functions? Target program call MPI_Send MPI_Send: opcode1 opcode2 opcode3 Insertion of the library in target memory using LD_PRELOAD Prolog/epilog library prolog: epilog: - 15

48 How to instrument static functions? Target program call MPI_Send MPI_Send: opcode1 opcode2 opcode3 trampoline call prolog replay call epilog Prolog/epilog library prolog: epilog: - 15

49 How to instrument static functions? Target program call MPI_Send MPI_Send: opcode1 opcode2 opcode3 Fetch prolog & epilog addresses by using the Binary File Descriptor library (libbfd) and by analyzing target system calls Prolog/epilog library trampoline call prolog replay call epilog prolog: epilog: - 15

50 How to instrument static functions? Target program call MPI_Send MPI_Send: opcode1 opcode2 opcode3 trampoline call prolog replay call epilog Allocate memory in the target process for the trampoline using the mmap(2) system call Prolog/epilog library prolog: epilog: - 15

51 How to instrument static functions? Target program call MPI_Send MPI_Send: opcode1 opcode2 opcode3 trampoline call prolog replay call epilog Prolog/epilog library prolog: epilog: Tune the stack to support arguments using the AMD64 Application Binary Interface - 15

52 How to instrument static functions? Target program call MPI_Send MPI_Send: opcode1 jmp trampoline opcode2 opcode3 trampoline call prolog replay call epilog Prolog/epilog library prolog: epilog: - 15

53 How to instrument static functions? Target program call MPI_Send MPI_Send: opcode1 jmp trampoline opcode2 opcode3 trampoline call prolog replay call epilog Prolog/epilog library prolog: epilog: MPI_Send address fetched using libbfd - 15

54 How to instrument static functions? Target program call MPI_Send MPI_Send: opcode1 jmp trampoline opcode2 opcode3 Replay the instructions overwritten by the base trampoline trampoline call prolog replay call epilog Prolog/epilog library prolog: epilog: - 15

55 How to instrument static functions? Target program call MPI_Send MPI_Send: opcode1 jmp trampoline opcode2 opcode3 trampoline call prolog call replay call epilog replay: opcode1 opcode2 jmp opcode3 Prolog/epilog library prolog: epilog: - 15

56 How to instrument static functions? Target program call MPI_Send MPI_Send: opcode1 jmp trampoline opcode2 opcode3 trampoline call prolog call replay call epilog replay: opcode1 opcode2 jmp opcode3 Prolog/epilog library prolog: epilog: - 15

57 How to instrument static functions? Target program call MPI_Send MPI_Send: opcode1 jmp trampoline opcode2 opcode3 trampoline call prolog call replay call epilog replay: opcode1 opcode2 jmp opcode3 Prolog/epilog library prolog: epilog: - 15

58 How to instrument static functions? Target program call MPI_Send MPI_Send: opcode1 jmp trampoline opcode2 opcode3 trampoline call prolog call replay call epilog replay: opcode1 opcode2 jmp opcode3 Prolog/epilog library prolog: epilog: Get the size of the opcodes overwritten by the base trampoline - 15

59 4Technical details for the geek - 16

60 Technical challenges left Allocate memory in the target process Get the address of the preloaded library - The offset of the prolog and epilog inside the library are fetched with libbfd - The library loading mechanisms is spyed to find the mmaping of the library Get the size of the overwritten opcodes - 17

61 The mmap() system call The mmap() system call allocate a memory page #include <sys/mman.h> void* mmap(void *addr, size_t len, int prot, int flags, int fd, off_t offset) man 2 mmap - 18

62 The mmap() system call The mmap() system call allocate a memory page #include <sys/mman.h> void* mmap(void *addr, size_t len, int prot, int flags, int fd, off_t offset) man 2 mmap Can be used to map a library in memory - 18

63 The mmap() system call The mmap() system call allocate a memory page #include <sys/mman.h> void* mmap(void *addr, size_t len, int prot, int flags, int fd, off_t offset) man 2 mmap Can be used to map a library in memory Can also be used to allocate an executable segment for the trampoline - 18

64 Calling a system call from the parent process To call mmap() to allocate memory for the trampoline, we need to: Stop the target process Replace the current instruction by an int 0x80 (syscall) Set the accumulator (rax) to the system call number Fill the other registers according to the ABI Executes two ptrace(ptrace_syscall, pid, NULL, NULL) Get the result from the registers Restores the registers and the current instruction - 19

65 Fetching the library base address The base address of a preloaded library is determined at runtime. The dlopen() mechanism do it by: - Opening the library file using the open() system call - Mapping it in memory using the mmap() system call Thus to fetch the base address: - Wait for open(«library.so», *) = fd - Wait for mmap(*, *, *, *, fd, *) = baseaddress A wait is performed using the PTRACE_SYSCALL method and by looking into registers - 20

66 Getting the overwritten opcode size Obtained by single stepping ip = 4007c6 4007c6: sub $0x8,%rsp 4007ca: mov %rsi,%rbx 4007cd: cmp $0x3,%edi 4007d0: jmp 4007fa 4007d4: - 21

67 Getting the overwritten opcode size Obtained by single stepping ip = 4007ca 4007c6: sub $0x8,%rsp 4007ca: mov %rsi,%rbx 4007cd: cmp $0x3,%edi 4007d0: jmp 4007fa 4007d4: - 21

68 Getting the overwritten opcode size Obtained by single stepping ip = 4007ca 4007c6: sub $0x8,%rsp 4007ca: mov %rsi,%rbx 4007cd: cmp $0x3,%edi 4007d0: jmp 4007fa 4007d4: ip += 4 [sizeof( sub $0x8, %rsp )] incr = 4 < sizeof( base trampoline ) - 21

69 Getting the overwritten opcode size Obtained by single stepping ip = 4007cd 4007c6: sub $0x8,%rsp 4007ca: mov %rsi,%rbx 4007cd: cmp $0x3,%edi 4007d0: jmp 4007fa 4007d4: ip += 3 [sizeof( mov %rsi, %rbx )] incr = 7 (+3) < sizeof( base trampoline ) - 21

70 Getting the overwritten opcode size Obtained by single stepping ip = 4007d0 4007c6: sub $0x8,%rsp 4007ca: mov %rsi,%rbx 4007cd: cmp $0x3,%edi 4007d0: jmp 4007fa 4007d4: ip += 3 [sizeof( cmp $0x3, %edi )] incr = 10 (+3) < sizeof( base trampoline ) - 21

71 Getting the overwritten opcode size Obtained by single stepping ip = 4007fa 4007c6: sub $0x8,%rsp 4007ca: mov %rsi,%rbx 4007cd: cmp $0x3,%edi 4007d0: jmp 4007fa 4007d4: ip += 26 [JUMP] > threshold - 21

72 Getting the overwritten opcode size Obtained by single stepping ip = 4007fa 4007c6: sub $0x8,%rsp 4007ca: mov %rsi,%rbx 4007cd: cmp $0x3,%edi 4007d0: jmp 4007fa 4007d4: ip += 26 [JUMP] > threshold JUMP DECODER - 21

73 Getting the overwritten opcode size Obtained by single stepping ip = 4007fa 4007c6: sub $0x8,%rsp 4007ca: mov %rsi,%rbx 4007cd: cmp $0x3,%edi 4007d0: jmp 4007fa 4007d4: ip += 26 [JUMP] > threshold JUMP DECODER sizeof( jmp 4007fa ) = 4-21

74 Getting the overwritten opcode size Obtained by single stepping ip = 4007d4 4007c6: sub $0x8,%rsp 4007ca: mov %rsi,%rbx 4007cd: cmp $0x3,%edi 4007d0: jmp 4007fa 4007d4: ip 4007d4 [4007d0 + 4] incr = 14 (+4) sizeof( base trampoline ) - 21

75 Getting the overwritten opcode size Obtained by single stepping ip = 4007d4 4007c6: sub $0x8,%rsp 4007ca: mov %rsi,%rbx 4007cd: cmp $0x3,%edi 4007d0: jmp 4007fa 4007d4: ip 4007d4 [4007d0 + 4] incr = 14 (+4) sizeof( base trampoline ) The size of instruction is now known! - 21

76 5Other architectures - 22

77 Other architectures Other CPUs 32-bit Intel Architecture is straighforward - Just a simple modification of the ABI RISC processors are easier - Simplified ABI - Fixed-size instructions And other operating system *BSD should not be a problem - The ptrace() system call changes but offers similar functionalities - The ABI compatibility needs to be verified MacOS X is problematic - 23

78 The MacOS X issue MacOS X ptrace() system call is outdated The task_for_pid() mechanism replaces it The ABI changes MacOS X has a complex capability scheme needed for tracing Even the LD_PRELOAD mechanism changes - 24

79 6Future - 25

80 In the future EZTrace will: - 26

81 In the future EZTrace will: instrument functions of static libraries, - 26

82 In the future EZTrace will: instrument functions of static libraries, instrument user-defined functions, - 26

83 In the future EZTrace will: instrument functions of static libraries, instrument user-defined functions, do it easily thanks to a module generator, BEGIN_MODULE NAME example_lib DESC "module for the example library" LANGUAGE C ID 42 int example_function2(int* array, int array_size) BEGIN RECORD_STATE("running example_function2") END END_MODULE - 26

84 In the future EZTrace will: instrument functions of static libraries, instrument user-defined functions, do it easily thanks to a module generator, and save the world BEGIN_MODULE NAME example_lib DESC "module for the example library" LANGUAGE C ID 42 int example_function2(int* array, int array_size) BEGIN RECORD_STATE("running example_function2") END END_MODULE - 26

85 In the future EZTrace will: instrument functions of static libraries, instrument user-defined functions, do it easily thanks to a module generator, and save the world BEGIN_MODULE NAME example_lib DESC "module for the example library" LANGUAGE C ID 42 int example_function2(int* array, int array_size) BEGIN RECORD_STATE("running example_function2") END END_MODULE and the ornithorhynchus - 26

86 Thank you!