DTrace integration and quick start

Transcription

1 DTrace integration and quick start rubsd Dec 2013 Veniamin Gvozdikov

2 What is DTrace? Dynamic tracing framework Works on Solaris/FreeBSD/OSX/QNX/Linux 2

3 Who uses it? Apple s Xcode Instruments Cloud Analytics in Joyent SmartDataCenter Oracle s ZFS Storage Appliance analytics Brendan s Flame Graphs Command line DTraceToolkit 3

4 Xcode Instruments 4

5 Cloud Analytics 5

6 Oracle s ZFS-SA analytics 6

7 Flame Graphs 7

8 DTraceToolkit 8

9 Terminology probe provider module 9

10 How it works? dtrace (1) libproc libdtrace /dev/dtrace user space DTrace kernel space 10

11 Providers dtrace: tcp: kernel space fbt: dtmalloc: io: lockstat: and others: syscall: profile: 11

12 dtrace(1) Command line frontend Shows available probes/providers Enable probes and attach to processes D Language compiler Works via libdtrace and libproc ETC 12

13 Available probes To see all probes on the system you need to run: # dtrace -l [ many lines ] To see probes by dtrace provider: # dtrace -l -n 'dtrace:::' ID PROVIDER MODULE FUNCTION NAME 1 dtrace BEGIN 2 dtrace END 3 dtrace ERROR 13

14 D program structure D language is like C and AWK: provider:module:function:probe / predicates / { action; 14

15 First D program # dtrace -q -n ' > syscall::open:entry { > printf("%s\t%s\n", execname, copyinstr(arg0)); > > profile:::tick-1s { > exit(0); > > ' tmux /dev/null sh /etc/libmap.conf sh /var/run/ld-elf.so.hints sh /lib/libedit.so.7 sh /lib/libncurses.so.8 sh /lib/libc.so.7 15

16 Built-in Variables (not all) args pid probeprov errno probefunc timestamp execname probemod vtimestamp 16

17 Aggregating Functions Function Name Arguments count sum avg min max lquantize quantize none scalar expression scalar expression scalar expression scalar expression scalar expression, lower bound, upper bound, step value scalar expression 17

18 Tracing applications Trace with pid provider Trace with USDT (Userland Statically Defined Traces) 18

19 DTrace integration examples Please clone repo (seeing code is required for next slides): 19

20 int length_fast(char *); int length_slow(char *); int main(void) { int counter = 0; int c_loop = 0; char *Lines[] = { "x", "tt", "Test", NULL ; char *line; printf("%10s %4s\n", "STRING", "LEN"); while (15 > c_loop) { while (NULL = Lines[counter]) { line = Lines[counter]; printf("%10s: %02d\n", line, length_fast(line)); printf("%10s: %02d\n", line, length_slow(line)); ++counter; ++c_loop; counter = 0; sleep(c_loop); return 0; 20

21 #00 pid provider 00_script.sh pid$target::length_*:entry { self->ts = timestamp; printf("string: %s\n", copyinstr(arg0)); 21

22 pid$target::length_*:return /self->ts/ = = sum(timestamp - = quantize(timestamp - self->ts); self->ts = 0; profile:::tick-10sec { exit(0); 22

23 END { printf("count of call functions:\n"); printa(@num); printf("time of call functions:\n"); printa(@sum); printf("distribution of times by functions:\n"); printa(@q); 23

24 result Count of call functions: length_fast 6 length_slow 6 Time of call functions: length_fast length_slow Distribution of times by functions: length_fast value Distribution count length_slow value Distribution count

25 #01 First USDT provider dtrace_demo.d: provider demo { probe start(); probe stop(); ; Generate header file: # dtrace -h -s dtrace_demo.d 25

26 #01 Adding probes while (1) { if (DEMO_START_ENABLED()) DEMO_START(); while (NULL = Lines[counter]) { [ lines ] counter = 0; sleep(1); if (DEMO_STOP_ENABLED()) DEMO_STOP(); 26

27 Compile applications Make object file: # clang -c main.c -DENABLE_DTRACE Run dtrace(1) on object file: # dtrace -G -s dtrace_demo.d main.o Link objects to application: # clang -lelf main.o dtrace_demo.o -o demo 27

28 #01 Trace application 01_script.sh #/usr/sbin/dtrace -Zs #pragma D option quiet demo*:::start { self->ts = timestamp; demo*:::stop { printf("time: %d\n", timestamp - self->ts); self->ts = 0; result # scripts/01_script.sh time: time: time: time: time: time: time: time: time: time:

29 #02 Probes with arguments dtrace_demo.d: provider demo { probe start(char *); probe stop(int); ; Generate header file: # dtrace -h -s dtrace_demo.d 29

30 #02 Adding probes int length_fast(char *string) { int i; if (DEMO_START_ENABLED()) DEMO_START(string); for (i = 0; '\0' = *string; ++i) { ++string; if (DEMO_STOP_ENABLED()) DEMO_STOP(i); return i; int length_slow(char *string) { int i; if (DEMO_START_ENABLED()) DEMO_START(string); 30 for (i = 0; '\0' = *string; ++i) { ++string; usleep(i*i); if (DEMO_STOP_ENABLED()) DEMO_STOP(i); return i;

31 #/usr/sbin/dtrace -Zs #pragma D option quiet BEGIN { self->start = timestamp; self->ts = 0; printf("start tracing...\n\n"); demo*:::start { self->ts = timestamp; printf("string: %10s ", copyinstr(arg0)); demo*:::stop /self->ts/ { printf("len: %2d time: %6d \n", arg0, timestamp - = quantize((timestamp - self->ts) / 1000); self->ts = 0; END { printf("time %d.%d sec\n", (timestamp - self->start)/1000/1000/1000, (timestamp - self->start)/1000/1000/100); 31

32 result # scripts/02_script.sh Start tracing... String: tt len: 2 time: String: Test len: 4 time: 5131 String: Test len: 4 time: String: x len: 1 time: String: x len: 1 time: String: tt len: 2 time: 4949 String: x len: 1 time: String: x len: 1 time: 8848 String: tt len: 2 time: 4430 String: tt len: 2 time: String: Test len: 4 time: 4561 String: Test len: 4 time: ^C Time sec [ skipped distributions ] 32

33 #03 Evolution interface dtrace_demo.d: typedef struct { int dummy; dline_t; typedef struct { int dummy; demo_t; provider demo { probe start(); probe stop(dline_t *dp) : (demo_t *dp); ; #pragma D attributes Evolving/Evolving/ISA provider demo provider #pragma D attributes Private/Private/Unknown provider demo module #pragma D attributes Private/Private/Unknown provider demo function #pragma D attributes Private/Private/ISA provider demo name #pragma D attributes Evolving/Evolving/ISA provider demo args 33

34 #03 Fill structure demo_provider.h: typedef struct { const char *mystr; int len; dline_t; int length_fast(char *string) { dline_t data; int i; if (DEMO_START_ENABLED()) { data.mystr = string; DEMO_START(); [ some action ] if (DEMO_STOP_ENABLED()) { data.len = i; DEMO_STOP(&data); return i; 34

35 #03 First DTrace library demo.d: typedef struct { char *mystr; int len; dline_t; copy demo.d to /usr/lib/dtrace or run: typedef struct { string str; int len; demo_t; #pragma D binding "1.7" translator translator demo_t < dline_t *p > { str = copyinstr(*(uintptr_t *)copyin((uintptr_t)&p->mystr, sizeof (char*))); len = *(uint64_t *)copyin((uintptr_t)&p->len, sizeof (p->len)); ; # dtrace -L /path/to/dtrace/libs some args 35

36 #03 Access to structure 03_script.sh: demo*:::stop /self->ts/ { printf("string: %10s len: %2d time: %6d\n", args[0]->str, args[0]->len, timestamp - = quantize((timestamp - self->ts) / 1000); self->ts = 0; 36

37 result # scripts/03_script.sh Start tracing... String: x len: 1 time: 5178 String: x len: 1 time: 3485 String: tt len: 2 time: 1415 String: tt len: 2 time: String: Test len: 4 time: 1673 String: Test len: 4 time: ^C Time 2.20 sec [ skipped distributions ] 37

38 libusdt Creates DTrace probes at runtime Bindings to Lua/Ruby/node.js/Perl 38

39 libusdt and tarantool Please clone repo recursively from dtrace branch: See README.DTrace for more details 39

40 lua usdt init.lua: provider = usdt.provider("init_script", "tarantool_box") probe = provider:probe("dt_hello", "entry", "char *", "int") provider:enable() function dt_hello(mystr) probe:fire(mystr, string.len(mystr)) end 40

41 Example # dtrace -l -n 'init_script*:::' ID PROVIDER MODULE FUNCTION NAME init_script18132 tarantool_box dt_hello entry localhost> for _, val in ipairs({"test", "tt", "xxx") do dt_hello(val) end # dtrace -q -n 'init_script*:::entry { printf("string %10s len: %02d\n", copyinstr(arg0), arg1) tick-2s {exit(0);' String Test len: 04 String tt len: 02 String xxx len: 03 41

42 References