Ftrace - What s new. Since my last explanation of ftrace (from v3.18) Steven Rostedt 25/10/ VMware Inc. All rights reserved.

Size: px

Start display at page:

Rhoda Cook
6 years ago
Views:

block, etc) triggers stack trace trace off / (and on) snapshot

2 What ftrace did (and still does) Function tracing Function graph tracing Snapshots tracing events (sched_switch, timers, interrupts, block, etc) triggers stack trace trace off / (and on) snapshot Latency tracing (interrupts, wakeup) Debugging trace_printk() ftrace_dump_on_oops

3 Function tracing Can filter on specific functions (set_ftrace_filter) Can remove function from being traced (set_ftrace_notrace) Can trace just a specific PID (set_ftrace_pid) Can trace children of those PIDS (options/function-fork) Can set triggers on a specific function stack trace snapshot trace off (and on) Enable/disable an event Can profile functions (see hit counts) Trace stack usage (biggest stack hog)

4 Function Graph Tracer Same filtering as function tracing Can graph a function (see only what a function calls) My disable tracing of interrupts (only care what the function calls) Can set a max depth Only trace the first instance (see where the kernel gets called) syscalls page faults Can see the time a function takes Singe functions Can profile on the times functions are executing

5 Snapshot Take a snapshot of the current live trace Can be done by user space (snapshot) Has instructions: cat snapshot # Snapshot commands: # echo 0 > snapshot : Clears and frees snapshot buffer # echo 1 > snapshot : Allocates snapshot buffer, if not already allocated. # Takes a snapshot of the main buffer. # echo 2 > snapshot : Clears snapshot buffer (but does not allocate or free) # (Doesn't have to be '2' works with any number that # is not a '0' or '1') Swaps the main buffer with the snapshot buffer

6 Trace Events Thousands of events exist today scheduling Interrupts Timers Hypervisors signals block paging

7 Triggers Types snapshot tracing off (and on) stacktrace enable/disable events Filtering <trigger> if <cond> condition on field, CPU, PID, comm etc if common_pid == $$

8 Latency Tracing Interrupts and/or preemption off times Gives the max time irqs and/or preemption was disabled Wake up tracer Traces max time of wakeup to scheduling in wakeup - traces the latency of all tasks (trying to get the highest priority task) wakeup_rt - only traces RT tasks (trying to get the highest priority task) wakeup_dl - only traces DEADLINE tasks Both are static tracers not much room for customization hard to look at just a single process

9 Debugging trace_printk() Like printk() but has no limits for context (NMI, irq, scheduler, etc) (well, it can t debug the tracing ring buffer) Optimized to be very fast ftrace_dump_on_oops dumps to the console on panic save the serial output make the buffers smaller, or you may be waiting for a long time

10 Added in 4.0 (What I tell people why Linux changed its main version from 3 to 4) Dynamic trampolines to ftrace function hooks Optimization Two tracers could trace two different functions and still have direct callback trampolines You can see how they are in enabled_functions # cd /sys/kernel/debug/tracing # echo schedule do_irq > set_ftrace_filter # echo function > current_tracer # cat enabled_functions schedule (1) do_irq (1) tramp: 0xffffffffc >function_trace_call+0x0/0x1b0 tramp: 0xffffffffc >function_trace_call+0x0/0x1b0 # mkdir instances/foo # echo schedule do_fork > instances/foo/set_ftrace_filter # echo function > instances/foo/current_tracer # cat enabled_functions do_fork (1) schedule (2) do_irq (1) ->ftrace_ops_list_func+0x0/0x1b0 ->ftrace_ops_list_func+0x0/0x1b0 tramp: 0xffffffffc >function_trace_call+0x0/0x1b0

11 Added in 4.0 NOT in filter logic Filters could only test for true cases Could not test for false cases Say you wanted all kmalloc events except for GFP_ZERO (0x8000) echo!(gfp_flags & 0x8000) > events/kmem/kmalloc/filter Can also handle clauses echo!(common_pid == 1234 && vector == 259) > events/irq_vectors/filter yes this is the same as (common_pid!= 1234) (vector!= 259) but still /* If not of not match is equal to not of not, then it is a match */ return!!match ==!op->not;

12 Added in 4.0 NOT in filter logic Filters could only test for true cases Could not test for false cases Say you wanted all kmalloc events except for GFP_ZERO (0x8000) echo!(gfp_flags & 0x8000) > events/kmem/kmalloc/filter Can also handle clauses echo!(common_pid == 1234 && vector == 259) > events/irq_vectors/filter yes this is the same as (common_pid!= 1234) (vector!= 259) but still /* If not of not match is equal to not of not, then it is a match */ return!!match ==!op->not;

13 Added in 4.0 NOT in filter logic Filters could only test for true cases Could not test for false cases Say you wanted all kmalloc events except for GFP_ZERO (0x8000) echo!(gfp_flags & 0x8000) > events/kmem/kmalloc/filter Can also handle clauses echo!(common_pid == 1234 && vector == 259) > events/irq_vectors/filter yes this is the same as (common_pid!= 1234) (vector!= 259) but still /* If not of not match is equal to not of not, then it is a match */ return!!match ==!op->not;

14 Added in 4.0 NOT in filter logic Filters could only test for true cases Could not test for false cases Say you wanted all kmalloc events except for GFP_ZERO (0x8000) echo!(gfp_flags & 0x8000) > events/kmem/kmalloc/filter Can also handle clauses echo!(common_pid == 1234 && vector == 259) > events/irq_vectors/filter yes this is the same as (common_pid!= 1234) (vector!= 259) but still /* If not of not match is equal to not of not, then it is a match */ return!!match ==!op->not;

15 Added in 4.0 NOT in filter logic Filters could only test for true cases Could not test for false cases Say you wanted all kmalloc events except for GFP_ZERO (0x8000) echo!(gfp_flags & 0x8000) > events/kmem/kmalloc/filter Can also handle clauses echo!(common_pid == 1234 && vector == 259) > events/irq_vectors/filter yes this is the same as (common_pid!= 1234) (vector!= 259) but still /* If not of not match is equal to not of not, then it is a match */ return!!match ==!op->not;

16 Added in 4.0 NOT in filter logic Filters could only test for true cases Could not test for false cases Say you wanted all kmalloc events except for GFP_ZERO (0x8000) echo!(gfp_flags & 0x8000) > events/kmem/kmalloc/filter Can also handle clauses echo!(common_pid == 1234 && vector == 259) > events/irq_vectors/filter yes this is the same as (common_pid!= 1234) (vector!= 259) but still /* If not of not match is equal to not of not, then it is a match */ return!!match ==!op->not;

17 Added in 4.0 NOT in filter logic Filters could only test for true cases Could not test for false cases Say you wanted all kmalloc events except for GFP_ZERO (0x8000) echo!(gfp_flags & 0x8000) > events/kmem/kmalloc/filter Can also handle clauses echo!(common_pid == 1234 && vector == 259) > events/irq_vectors/filter yes this is the same as (common_pid!= 1234) (vector!= 259) but still /* If not of not match is equal to not of not, then it is a match */ return!!match ==!op->not;

18 Added in 4.0 NOT in filter logic Filters could only test for true cases Could not test for false cases Say you wanted all kmalloc events except for GFP_ZERO (0x8000) echo!(gfp_flags & 0x8000) > events/kmem/kmalloc/filter Can also handle clauses echo!(common_pid == 1234 && vector == 259) > events/irq_vectors/filter yes this is the same as (common_pid!= 1234) (vector!= 259) but still /* If not of not match is equal to not of not, then it is a match */ return!!match ==!op->not;

19 Added in 4.0 NOT in filter logic Filters could only test for true cases Could not test for false cases Say you wanted all kmalloc events except for GFP_ZERO (0x8000) echo!(gfp_flags & 0x8000) > events/kmem/kmalloc/filter Can also handle clauses echo!(common_pid == 1234 && vector == 259) > events/irq_vectors/filter yes this is the same as (common_pid!= 1234) (vector!= 259) but still /* If not of not match is equal to not of not, then it is a match */ return!!match ==!op->not;

20 Added in 4.0 tp_printk kernel command line option USE AT YOUR OWN RISK! Used with trace_event command to enable specific events Sends all events to printk! Great if you machine locks up / crashes / reboots early in boot Should disable after boot is complete echo 0 > /proc/sys/kernel/tracepoint_printk This proc file has no effect without the kernel command line option set

21 Adding in 4.1 The new tracefs file system Now in /sys/kernel/tracing mount -t tracefs /sys/kernel/tracing tracefs automatically mounted in debugfs tracing directory (for backward compatibility) No longer requires debugfs compiled in

22 Added in 4.4 (since 4.1) mono_raw clock echo mono_raw > trace_clock Not affected by NTP Compare consistent use of CPU cycles Can be safely called from NMI Can be used at early boot up trace_clock=mono_raw on kernel command line tracing_thresh affects function profile (with function graph) echo 0 > options/sleep_time echo 100 > tracing_thresh echo 1 > function_profile_enabled

23 tracing_thresh on function_profile_enabled # echo 0 > options/sleep_time # echo 1000 > tracing_thresh # echo 1 > ftrace_tracing_enabled # cat trace_stat/function0 Function Hit Time Avg s^ schedule us us us default_idle_call us us us arch_cpu_idle us us us default_idle us us us SyS_read us us us vfs_read us us us vfs_read us us us tty_read us us us n_tty_read us us us wait_woken us us us schedule_timeout us us us SyS_select us us us core_sys_select us us us do_select us us us poll_schedule_timeout us us us schedule_hrtimeout_range us us us schedule_hrtimeout_range_clock us us us

24 Added in 4.4 (since 4.1) Tracepoint sched_waking Trace when a task gets set to running Not when it actually gets woken up logic changed such that a task can be woken up later than the waker executed the wake up CPU and COMM can be used to filter triggers # echo 'stacktrace if cpu == 1' > \ /sys/kernel/debug/tracing/events/raw_syscalls/sys_enter/trigger # echo 'stacktrace if comm == sshd' > \ /sys/kernel/debug/tracing/events/raw_syscalls/sys_enter/trigger

25 Added in 4.4 (since 4.1) Show all options in options/ directory Use to only show tracer options if tracer is active func_stack_trace very dangerous option to leave on 4.1 options # ls options/ annotate ftrace_preempt markers sleep-time userstacktrace bin function-trace overwrite stacktrace verbose block graph-time printk-msg-only sym-addr branch hex print-parent sym-offset context-info irq-info raw sym-userobj disable_on_free latency-format record-cmd trace_printk 4.4 options # ls options/ annotate funcgraph-irqs latency-format sym-offset bin funcgraph-overhead markers sym-userobj blk_classic funcgraph-overrun overwrite test_nop_accept block funcgraph-proc printk-msg-only test_nop_refuse context-info funcgraph-tail print-parent trace_printk disable_on_free func_stack_trace raw userstacktrace display-graph function-trace record-cmd verbose funcgraph-abstime graph-time sleep-time funcgraph-cpu hex stacktrace funcgraph-duration irq-info sym-addr

26 Added in 4.4 (since 4.1) Globing for module function trigger filter # echo '*alloc*:mod:*' > set_ftrace_filter # echo '!*alloc*:mod:ipv6' >> set_ftrace_filter # echo '*alloc*:mod:!ipv6' > set_ftrace_filter # echo '*alloc*:mod:!*' > set_ftrace_filter # echo '*:mod:!*' > set_ftrace_filter # echo ':mod:!*' > set_ftrace_filter or # echo '*:mod:*' > set_ftrace_filter # echo ':mod:' > set_ftrace_filter or # echo '*:mod:*snd*' > set_ftrace_filter

27 Added in 4.4 (since 4.1) set_event_pid Similar to set_ftrace_pid but for events echo a number into here, and only events with a PID of that number will be traced Use to have to add filters if common_pid==123 common_pid==234 common_pid==3432 That is very limited Add as many pids to set_event_pid as you would like

28 Added in 4.9 (since 4.4) event_fork option Allows processes in set_event_pid to add children when forking # echo $$ > set_event_pid # echo 1 > options/event-fork # cat set_event_pid # cat set_event_pid # cat set_event_pid

29 Added in 4.9 (since 4.4) Introduction of histogram trigger # echo 'hist:keys=common_pid,bytes_req:values=bytes_alloc:sort=bytes_req,common_pid' \ > events/kmem/kmalloc_node/trigger # cat events/kmem/kmalloc_node/hist # event histogram # # trigger info: hist:keys=common_pid,bytes_req:vals=hitcount,bytes_alloc:sort=bytes_req,common_pid:size=2048 [active] # { common_pid: 803, bytes_req: 8 } hitcount: 7 bytes_alloc: 56 { common_pid: 803, bytes_req: 32 } hitcount: 1 bytes_alloc: 32 { common_pid: 998, bytes_req: 32 } hitcount: 1 bytes_alloc: 32 { common_pid: 803, bytes_req: 64 } hitcount: 1 bytes_alloc: 64 { common_pid: 998, bytes_req: 64 } hitcount: 1 bytes_alloc: 64 { common_pid: 803, bytes_req: 96 } hitcount: 1 bytes_alloc: 96 { common_pid: 998, bytes_req: 96 } hitcount: 1 bytes_alloc: 96 { common_pid: 797, bytes_req: 2048 } hitcount: 6 bytes_alloc: Totals: Hits: 19 Entries: 8 Dropped: 0

30 Added in 4.14 (since 4.9) Full glob matching for filters Old way (4.9 and earlier) only allowed: Straight match: echo schedule > set_ftrace_filter Prefixed match: echo sched* > set_ftace_filter Postfix match: echo *lock > set_ftrace_filter Contains match: echo *sched* > set_ftrace_filter New way is much more flexible Allows inserted wild cards within matches Allows sets of characters with [ and ] Allows single character?

31 Added in 4.14 (since 4.9) # echo sched*group > set_ftrace_filter # cat set_ftrace_filter sched_free_group sched_change_group sched_create_group sched_online_group sched_destroy_group sched_offline_group # echo?malloc > set_ftrace_filter # cat set_ftrace_filter vmalloc kmalloc # echo [Ss]y[Ss]_* > set_ftrace_filter # cat set_ftrace_filter head SyS_arch_prctl sys_rt_sigreturn sys_ioperm SyS_iopl sys_modify_ldt SyS_mmap SyS_set_thread_area SyS_get_thread_area SyS_set_tid_address sys_fork

32 Added in 4.14 (since 4.9) trace_marker_raw Userspace can write raw data into trace_marker_raw (structs, etc) Reader must handle endianess and word size A bit faster than trace_marker boot clock like monotonic clock Keeps track of time across suspend and resume Allow comm filter to act as string # echo 'COMM ~ "systemd*"' > events/syscalls/filter

33 Added in 4.14 (since 4.9) Hardware Latency Detector Detects SMIs or other hardware related instances Runs loops under interrupt disable (looks for NMIs) # cat hwlat_detector/width # cat hwlat_detector/window # cat tracing_thresh 10 # echo hwlat > current_tracer # cat trace # tracer: hwlat # # _-----=> irqs-off # / _----=> need-resched # / _---=> hardirq/softirq # / _--=> preempt-depth # / delay # TASK-PID CPU# TIMESTAMP FUNCTION # <...>-970 [001] d : #1 inner/outer(us): 94/1015 ts: <...>-970 [002] d : #2 inner/outer(us): 486/60 ts: <...>-970 [003] d : #3 inner/outer(us): 125/737 ts: <...>-970 [000] d : #4 inner/outer(us): 121/3144 ts:

34 Added in 4.14 (since 4.9) function_fork Same as event_fork When set, tasks with PIDs in set_ftrace_pid will have children added when they fork Tracing of init functions Functions tagged with init were not traced Remember the e1000e bricking bug? Enable function tracing at boot, init functions will be traced Filter module functions before module is loaded echo :mod:kvm > set_ftrace_filter # echo ':mod:snd_hda_intel' > set_ftrace_filter # cat set_ftrace_filter :mod:snd_hda_intel # modprobe snd_hda_intel # cat set_ftrace_filter azx_intel_link_power [snd_hda_intel] azx_clear_irq_pending [snd_hda_intel] azx_via_get_position [snd_hda_intel] azx_runtime_idle [snd_hda_intel] azx_dev_disconnect [snd_hda_intel] pci_azx_writel [snd_hda_intel] pci_azx_readl [snd_hda_intel] [...]

35 What s coming More advanced histograms Full customization Pick specific fields to compare Trace only specific tasks Can do stack dumps Can display specific processes Synthetic events Can store custom fields based on other event historgrams Can also produce trace events and histograms Variables Store date by one event Read it from another event

36 What s coming irq / preempt disable events Tracing when irqs and / or preemption is disabled Tracing when irqs and / or preemption is enabled Will allow for the histograms to work at the irq / preempt level Gives all the features of trace events to these locations Module Init tracing Enabling tracing of module events before a module is loaded Already there for module init functions (v4.14) Passing in trace events to enabled when the module is loaded Seeing what trace events exist in a module via modinfo Better interleaved tracing between hosts and guests

37 Wish list Zero overhead of irq / preempt enable/disable events Zero overhead when trancing is disabled (obviously not when it s enabled) There s got to be a way to do this Use of jump_label infrastructure Requires jump_label functionality in assembly Zero overhead for lock events Currently requires lockdep Perhaps can also use jump labels Would be able to create lock histograms (longest held, etc) Have more interaction with ebpf and ftrace

38 Wish list Add tracing of function parameters Use dwarf, or some other mechanism Function graph to report return code of functions Function graph rewrite (it needs some loving) Filtering of functions via sections, files, groups Use linker magic to add mappings between functions and with what they belong to Already exists for modules, but the interface can be better Needs to not bloat the kernel (make it a loadable module option, like config.gz)

39 Wish list Converting trace.dat (from trace-cmd) to CTF May have someone to work on that soon UUENCODED ftrace_dump_on_opps to make trace.dat file from when kexec/kdump doesn t work Make perf ring buffer generic that ftrace tools can use it too ftrace ring buffer optimized for splice() not mmap trace-event and trace-cmd libraries

40 Thank You Steven Rostedt

Adding Inter-event Capabilities to the Linux Trace Event Subsystem

Adding Inter-event Capabilities to the Linux Trace Event Subsystem Tom Zanussi Intel Open Source Technology Center Safety Critical Systems ELC 2017, Feb 22 1 Trace Events Background Latency Example Design