uftrace: function graph tracer for C/C++

Transcription

1 uftrace: function graph tracer for C/C++ Namhyung Kim ( 김남형 ) namhyung@gmail.com namhyung.kim@lge.com Open Source Summit "Powered by Marp"

2 uftrace overview function tracer for C/C++ inspired by ftrace framework in the kernel analyze single process execution tracing library calls and kernel various ltering and features project homepage "Open Source Summit 2017" 2

3 Installation requirements OS: Linux Arch: x86_64, ARM(v6+), AArch64 Library: libelf (elfutils) Distro packaging Debian/Ubuntu: OK Fedora: WIP Archlinux: AUR OpenEmbedded: OK $ make $ sudo make install "Open Source Summit 2017" 3

4 Before running compiler support need to rebuild your program (ouch!) use -pg (or -finstrument-functions ) add calls to mcount for every function recent compilers provide advanced options $ gcc -pg <program> or $ gcc -finstrument-functions <program> "Open Source Summit 2017" 4

5 Running uftrace run your program with uftrace uftrace does fork+exec for you save tracing data to shmem and then les analyze the data $ uftrace <command> <program> "Open Source Summit 2017" 5

6 uftrace commands recording record, recv, live analysis misc replay, report, graph, script info, dump "Open Source Summit 2017" 6

7 uftrace live default command can be omitted (does record + replay) $ uftrace hello Hello World us [22715] monstartup(); us [22715] cxa_atexit(); [22715] main() { us [22715] printf(); us [22715] } /* main */ "Open Source Summit 2017" 7

8 uftrace record program execution fork + exec to run your program child process LD_PRELOAD=libmcount.so controlled by environment variables saves trace data into shmem communicates to uftrace via pipe uftrace create directory to save data (uftrace.data) write data in shmem to les additional information for system/process "Open Source Summit 2017" 8

9 uftrace replay show process execution timeline $ uftrace replay -f +time -d abc.data # DURATION TID TIMESTAMP FUNCTION [ 1892] main() { [ 1892] a() { [ 1892] b() { [ 1892] c() { us [ 1892] getpid() us [ 1892] } /* c */ us [ 1892] } /* b */ us [ 1892] } /* a */ us [ 1892] } /* main */ "Open Source Summit 2017" 9

10 uftrace replay "Open Source Summit 2017" 10

11 uftrace report show table of process runtime $ uftrace report -s self Total time Self time Calls Function ========== ========== ===== ========= us us 1 c us us 1 getpid us us 1 b us us 1 main us us 1 a "Open Source Summit 2017" 11

12 uftrace report --diff "Open Source Summit 2017" 12

13 uftrace graph show function call graph $ uftrace graph a... backtrace ===================================== backtrace #0: hit 1, time us [0] main (0x400512) [1] a (0x4006b2) calling functions ===================================== us : (1) a us : (1) b us : (1) c us : (1) getpid "Open Source Summit 2017" 13

14 uftrace graph "Open Source Summit 2017" 14

15 uftrace script run (python) script for each function $ cat count.py count = 0 def uftrace_begin(): pass def uftrace_entry(ctx): global count count += 1 def uftrace_exit(ctx): pass def uftrace_end(): print(count) $ uftrace script -S count.py 5 "Open Source Summit 2017" 15

16 uftrace script supported language python (2.7) context info addr, name, timestamp, tid, depth duration (exit), args (entry), retval (exit) options function lter: UFTRACE_FUNCS = [ "func1",... ] record option: # uftrace-option:... "Open Source Summit 2017" 16

17 uftrace dump dump le contents and convert format $ uftrace dump uftrace file header: magic = uftrace file header: version = 4 uftrace file header: header size = 40 uftrace file header: endian = 1 (little) uftrace file header: class = 2 (64 bit) uftrace file header: features = 0x63 (PLTHOOK...) uftrace file header: info = 0x3ff : [entry] main(400512) depth: : [entry] a(4006b2) depth: : [entry] b(4006a0) depth: : [entry] c(400686) depth: : [entry] getpid(4004b0) depth: 4... "Open Source Summit 2017" 17

18 uftrace info (1/2) show system information $ uftrace info # system information # ================== # program version : uftrace v0.8 # recorded on : Sun Jul 30 23:37: # cmdline : uftrace record abc # cpu info : Intel(R) Core(TM) i7-2640m CPU... # number of cpus : 4 / 4 (online / possible) # memory info : 11.3 / 15.5 GB (free / total) # system load : 0.44 / 0.34 / 0.29 (1 / 5 / 15 min) # kernel version : Linux ARCH # hostname : danjae # distro : "Arch Linux"... "Open Source Summit 2017" 18

19 uftrace info (2/2) and process information... # process information # =================== # number of tasks : 1 # task list : 1892 # exe image : /home/namhyung/tmp/uftrace/abc # build id : 5ba a40118e047353e7994fc... # exit status : exited with code: 0 # elapsed time : sec # cpu time : / sec (sys / user) # context switch : 1 / 1 (voluntary / involuntary) # max rss : 3252 KB # page fault : 0 / 189 (major / minor) # disk iops : 0 / 16 (read / write) "Open Source Summit 2017" 19

20 uftrace recv save data through network record command can send -H/--host option default port: 8090 useful for embedded systems lacks disk space $ uftrace recv --port 8090 $ uftrace record -H <server> "Open Source Summit 2017" 20

21 uftrace tips using lters lters, triggers, dynamic patch accessing arguments argument/return value speci cation corelate kernel execution Linux kernel function graph tracer event tracing SDT, kernel tracepoint, read-trigger, scheduler data visualization Chrome tracing, ame graph "Open Source Summit 2017" 21

22 Using function lters lter (inclusive) $ uftrace -F a abc [ 1892] a() { [ 1892] b() { [ 1892] c() { us [ 1892] getpid() us [ 1892] } /* c */ us [ 1892] } /* b */ us [ 1892] } /* a */ "Open Source Summit 2017" 22

23 Using function lters lter (exclusive) $ uftrace replay -N c [ 1892] main() { [ 1892] a() { us [ 1892] b(); us [ 1892] } /* a */ us [ 1892] } /* main */ "Open Source Summit 2017" 23

24 Using function lters lter (regex) $ uftrace -N ^get abc [ 1892] main() { [ 1892] a() { [ 1892] b() { us [ 1892] c(); us [ 1892] } /* b */ us [ 1892] } /* a */ us [ 1892] } /* main */ "Open Source Summit 2017" 24

25 Using depth lters lter $ uftrace -D 3 abc [ 1892] main() { [ 1892] a() { us [ 1892] b(); us [ 1892] } /* a */ us [ 1892] } /* main */ "Open Source Summit 2017" 25

26 Using lters time lter $ uftrace -t 3us abc [ 1892] main() { [ 1892] a() { [ 1892] b() { us [ 1892] c(); us [ 1892] } /* b */ us [ 1892] } /* a */ us [ 1892] } /* main */ "Open Source Summit 2017" 26

27 Using lters lter combinations $ uftrace -F main -D 3 -t 3us abc [ 1892] main() { [ 1892] a() { us [ 1892] b(); us [ 1892] } /* a */ us [ 1892] } /* main */ "Open Source Summit 2017" 27

28 Using lters task lter (not avaiable for record) $ uftrace replay --tid 1892 [ 1892] main() { [ 1892] a() { [ 1892] b() { [ 1892] c() { us [ 1892] getpid() us [ 1892] } /* c */ us [ 1892] } /* b */ us [ 1892] } /* a */ us [ 1892] } /* main */ "Open Source Summit 2017" 28

29 Using triggers trigger allows to override settings for each function depth trigger $ uftrace -T a@depth=1 abc [ 1892] main() { us [ 1892] a(); us [ 1892] } /* main */ "Open Source Summit 2017" 29

30 Using triggers time trigger $ uftrace -T b@time=4us abc [ 1892] main() { [ 1892] a() { us [ 1892] b(); us [ 1892] } /* a */ us [ 1892] } /* main */ "Open Source Summit 2017" 30

31 Using triggers trace-off trigger $ uftrace -T getpid@trace-off abc [ 1892] main() { [ 1892] a() { [ 1892] b() { [ 1892] c() { uftrace stopped tracing with remaining functions ================================================ task: 1892 [3] c [2] b [1] a [0] main "Open Source Summit 2017" 31

32 Using triggers trace-on trigger (used with --disable) $ uftrace --disable -T b@trace-on abc [ 1892] b() { [ 1892] c() { us [ 1892] getpid() us [ 1892] } /* c */ us [ 1892] } /* b */ us [ 1892] } /* a */ us [ 1892] } /* main */ "Open Source Summit 2017" 32

33 Using triggers color trigger (replay only) $ uftrace -T getpid@color=red abc [ 1892] main() { [ 1892] a() { [ 1892] b() { [ 1892] c() { us [ 1892] getpid() us [ 1892] } /* c */ us [ 1892] } /* b */ us [ 1892] } /* a */ us [ 1892] } /* main */ "Open Source Summit 2017" 33

34 Using triggers read trigger (record only) $ uftrace -T a@read=proc/statm abc [ 1892] main() { [ 1892] a() { [ 1892] /* read:proc/statm (size=100k,...) */ [ 1892] b() { [ 1892] c() { us [ 1892] getpid() us [ 1892] } /* c */ us [ 1892] } /* b */ us [ 1892] } /* a */ us [ 1892] } /* main */ "Open Source Summit 2017" 34

35 Accessing arguments basic usage $ uftrace -A printf@arg1 hello Hello World us [22715] monstartup(); us [22715] cxa_atexit(); [22715] main() { us [22715] printf(0x400746); us [22715] } /* main */ "Open Source Summit 2017" 35

36 Accessing arguments argument format $ uftrace -A printf@arg1/s hello Hello World us [22715] monstartup(); us [22715] cxa_atexit(); [22715] main() { us [22715] printf("hello %s\n"); us [22715] } /* main */ "Open Source Summit 2017" 36

37 Accessing arguments return value $ uftrace -R printf@retval hello Hello World us [22715] monstartup(); us [22715] cxa_atexit(); [22715] main() { us [22715] printf() = 12; us [22715] } /* main */ "Open Source Summit 2017" 37

38 Kernel tracing basic usage $ sudo uftrace -k hello Hello World us [22715] monstartup(); us [22715] cxa_atexit(); [22715] main() { [22715] printf() { us [22715] sys_newfstat(); us [22715] do_page_fault(); us [22715] } /* printf */ us [22715] } /* main */ "Open Source Summit 2017" 38

39 Kernel tracing function lter $ sudo uftrace -k -F do_page_fault@kernel hello Hello World us [22715] monstartup(); us [22715] cxa_atexit(); [22715] main() { [22715] printf() { us [22715] do_page_fault(); us [22715] } /* printf */ us [22715] } /* main */ "Open Source Summit 2017" 39

40 Kernel tracing depth lter $ sudo uftrace -F main -K 2 hello Hello World [28473] main() { [28473] printf() { [28473] sys_newfstat() { us [28473] vfs_statx_fd(); us [28473] cp_new_stat(); us [28473] } /* sys_newfstat */ [28473] do_page_fault() { us [28473] down_read_trylock(); us [28473] find_vma(); us [28473] handle_mm_fault(); us [28473] up_read(); us [28473] } /* do_page_fault */ us [28473] } /* printf */ us [28473] } /* main */ "Open Source Summit 2017" 40

41 Library call tracing by hooking PLT (no need to rebuild) $ uftrace --force /usr/bin/pwd /home/namhyung/tmp/uftrace/tutorial us [22715] cxa_atexit(); us [22715] getopt_long(); us [22715] getcwd(); us [22715] puts(); us [22715] free();... "Open Source Summit 2017" 41

42 Library call tracing accessing argument $ uftrace --force -R getcwd@retval/s /usr/bin/pwd /home/namhyung/tmp/uftrace/tutorial us [22715] cxa_atexit(); us [22715] getopt_long(); us [22715] getcwd() = "/home/namhyung/tmp/uftrace/tutoria us [22715] puts(); us [22715] free();... "Open Source Summit 2017" 42

43 Library call tracing tracing nested library calls $ uftrace --force --nest-libcall -t 1us /usr/bin/ps PID TTY TIME CMD 1761 pts/3 00:00:05 bash 6633 pts/3 00:00:00 uftrace 6634 pts/3 00:00:00 less 6635 pts/3 00:00:00 ps us [ 6635] setlocale(); us [ 6635] bindtextdomain(); [ 6635] look_up_our_self() { us [ 6635] sprintf_chk(); us [ 6635] open(); us [ 6635] read(); us [ 6635] close(); us [ 6635] sscanf(); us [ 6635] } /* look_up_our_self */... "Open Source Summit 2017" 43

44 Library call tracing not tracing library calls $ uftrace --no-libcall hello Hello World us [22715] main(); "Open Source Summit 2017" 44

45 Dynamic tracing need (recent) compiler support add a nop-sled when compile convert it to jump at runtime (load-time precisely) gcc has -mnop-mcount $ gcc -pg -mfentry -mnop-mcount -o abc abc.c clang has -fxray-instrument $ clang -fxray-instrument -fxray-instruction-threshold=1 abc.c "Open Source Summit 2017" 45

46 Dynamic tracing $ uftrace abc uftrace: /home/namhyung/project/uftrace/cmd-record.c:1449:check_bin ERROR: Can't find 'mcount' symbol in the 'abc'. It seems not to be compiled with -pg or -finstrument-functi which generates traceable code. Please check your binary fi $ uftrace -P b --no-libcall abc us [ 1890] b(); $ uftrace -P b abc [ 1892] b() { us [ 1892] getpid() us [ 1892] } /* b */ "Open Source Summit 2017" 46

47 Visualization Chrome tracing tool uftrace record + dump run Chrome browser open chrome://tracing load the JSON data le $ uftrace record <program> $ uftrace dump --chrome > abc.json "Open Source Summit 2017" 47

48 Visualization "Open Source Summit 2017" 48

49 Visualization Flame graph uftrace record + dump convert to Flame graph format (SVG) open in a Web browser $ uftrace record <program> $ uftrace dump --flame-graph flamegraph.pl > abc.svg $ firefox abc.svg "Open Source Summit 2017" 49

50 Visualization "Open Source Summit 2017" 50

51 Event tracing built-in event read trigger (proc/statm, page-fault) scheduler event using PERF_RECORD_SWITCH (v4.3+) user event systemtap SDT v3, x86_64 only (for now) kernel event tracepoint + libtraceevent "Open Source Summit 2017" 51

52 Event tracing scheduler event $ uftrace --force -t 1ms -E linux:schedule /usr/bin/sleep 0.1 [15755] nanosleep() { ms [15755] /* linux:schedule */ ms [15755] } /* nanosleep */ kernel event $ sudo uftrace --force -t 1ms -E sched_switch@kernel /usr/bin/sleep [17937] nanosleep() { [17937] /* sched:sched_switch (prev_comm=sleep...) [17937] /* sched:sched_switch (prev_comm=swapper/ ms [17937] } /* nanosleep */ "Open Source Summit 2017" 52

53 Future works debug info (DWARF) automatic access to arguments and return value user interface make it easier to use script enhancement more useful scripts dynamic tracing without compiler support performance improvement "Open Source Summit 2017" 53

54 Thanks uftrace homepage tutorial chat "Open Source Summit 2017" 54