Hardware Latencies How to flush them out (A use case) Steven Rostedt Red Hat

Transcription

1 Hardware Latencies How to flush them out (A use case) Steven Rostedt Red Hat

2 Here s a story, of a lovely lady... No this isn t the Brady Bunch Nor is it about a lovely lady But it probably could have been a Brady Bunch episode.

3 Here s a story, of an upset customer Who were seeing lots of latencies on their own The machine wasn t verified yet Real time requires not just a kernel Requires the entire spectrum Application Kernel Hard ware

4 Here s a story, of an upset customer Who were seeing lots of latencies on their own The machine wasn t verified yet Real time requires not just a kernel Requires the entire spectrum Application Kernel Hard ware

5 Verification of Hardware rteval A tool by Red Hat to stress the machine Measures jitter (using cyclictest) Was a large machine 40 CPUs For such a box, we expect no more than 200us jitter Like less, but we are lenient with large HW

6 Latencies Seeing 500us latencies!!!! May not sound big to you But it's huge for PREEMPT_RT Took a while to hit that Was it HW? SW? We control the app (rteval) Of course I blamed the HW ;-) Of course the HW vendor blamed SW

7 The Enemy 500 microsecond latency Function tracing Latency tracers HW Lat detector Event tracing trace_printk() The Weapons

8 Function Tracing echo function > current_tracer echo function_graph > current_tracer trace-cmd is nicer trace-cmd start -p function_graph trace-cmd stop trace-cmd extract trace-cmd report

9 rteval hackbench kernel builds cyclictest rteval --duration=100h

10 rteval Breaking it up rteval --onlyload --duration=100h Does not run cyclictest Run cyclictest separately

11 cyclictest cyclictest --numa -p95 -d0 -i100 -qm numa implies -a -t -n a - bind a task to each CPU t - thread per CPU n - use nanosleep() not signals p95 - set priority to 95 d0 - all threads run same interval i100 - sleep for 100 us q - quiet - don't show status during test m - mlockall memory

12 cyclictest cyclictest --numa -p95 -d0 -i100 -qm -b 200 b break after 200 us latency implies running function tracer Stops tracer on latency too Function tracing adds a lot of overhead! cyclictest --numa -p95 -d0 -i100 -qm -b 1000 increase breakpoint by a lot!

13 cyclictest Function tracing adds too much overhead cyclictest --numa -p95 -d0 -i100 -qm -b 200 -E E - uses event tracing instead of function Better with latencies Not as much info

14 cyclictest Limit function tracing with trace-cmd trace-cmd start -p function -n '*lock* trace-cmd start -p function -l '*sched*'' cyclictest --numa -p95 -d0 -i100 -qm -b 300

15 Latency Tracers wakeup-rt Ignore wakeup tracer preemptirqsoff Just ignore the: irqsoff preemptoff

16 Wakeup-rt trace-cmd start -p wakeup-rt Records the time of the highest -rt task Problems From wake up to schedule defaults running function tracer trace-cmd start -p wakeup-rt -d disables function tracing Not much info without functions trace-cmd start -p wakeup-rt -d -e all enables all events

17 Wakeup-rt Didn't help :-( Not enough info with events Function tracing caused latencies Hard to determine if latency was real or Heisenbug

18 preemptirqsoff trace-cmd start -p preemptirqsoff -e all -d Showed us issues with the scheduler Pointed to load balancing but that was a symptom not the cause

19 Modified cyclictest Changed to use function graph instead of function trace-cmd -p function -l load_balance <idle>-0 [000] : function: load_balance <idle>-0 [001] : function: load_balance <idle>-0 [000] : function: load_balance

20 Modified cyclictest trace-cmd -p function_graph -l load_balance Much more useful <idle>-0 [002] : us load_balance(); <idle>-0 [003] : us load_balance(); <idle>-0 [002] : us load_balance(); <idle>-0 [003] : us load_balance();

21 Latency without Load Balance? Hit a latency, and load balance wasn't called? PREEMPT_RT converts spinlocks to mutex except for raw_spin_locks! trace-cmd start -p function_graph \ -l '*raw_spin_lock*' <idle>-0 24dN : funcgraph_entry:! us _raw_spin_lock_irqsave();

22 Latency without Load Balance? Hit a latency, and load balance wasn't called? PREEMPT_RT converts spinlocks to mutex except for raw_spin_locks! trace-cmd start -p function_graph \ -l '*raw_spin_lock*' entry:! us _ra

23 Latency without Load Balance? Hit a latency, and load balance wasn't called? PREEMPT_RT converts spinlocks to mutex except for raw_spin_locks! trace-cmd start -p function_graph \ -l '*raw_spin_lock*' entry:! us _ra

24 Graph vs Function Tracing graph gives you time of function held function tracing can give you backtrace trace-cmd -p function -l 'raw_spin*' --func-stack trace-cmd-8725 [002] : function: trace-cmd-8725 [002] : kernel_stack: => schedule (ffffffff8146d08f) => schedule (ffffffff8146dd09) => do_nanosleep (ffffffff8146c7ec) => hrtimer_nanosleep (ffffffff8106eecb) => sys_nanosleep (ffffffff8106f00e) => system_call_fastpath (ffffffff ) _raw_spin_lock_irq <stack trace>

25 What to do? Keep function graph Add events All events added their own latencies Limit the events to trace trace-cmd start -p function_graph -l '*raw_spin_lock*' -e sched -e timer -e irq

26 Long story short Found latency rq lock contention in pull_rt_tasks 30 or more CPUs tried to take the same lock Between cache line bouncing and locking the bus, caused a large HW latency but you can still blame SW Fixed by doing IPIs instead

27 Pull RT Tasks CPU 0 CPU 1 CPU 2 CPU 40 cyclic test prio 90 cyclic test prio 90 cyclic test prio 90 cyclic test prio 90

28 Pull RT Tasks CPU 0 CPU 1 CPU 2 CPU 40 cyclic test prio 90 cyclic test prio 90 watchdog prio 99 cyclic test prio 90 cyclic test prio 90

29 Pull RT Tasks CPU 0 CPU 1 CPU 2 CPU 40 cyclic test prio 90 cyclic test prio 90 cyclic test prio 90 cyclic test prio 90

30 Pull RT Tasks CPU 0 CPU 1 CPU 2 CPU 40 cyclic test prio 90 cyclic test prio 90 cyclic test prio 90 cyclic test prio 90 irq thread prio 50

31 Pull RT Tasks CPU 0 CPU 1 CPU 2 CPU 40 <idle> <idle> cyclic test prio 90 <idle> irq thread prio 50

32 The Finding Nemo Seagull Effect! Mine

33 Pull RT Tasks The Finding Nemo Seagull Effect CPU 0 CPU 1 CPU 2 CPU 40 <idle> <idle> cyclic test prio 90 <idle> irq thread prio 50

34 Pull RT Tasks The Finding Nemo Seagull Effect CPU 0 CPU 1 CPU 2 CPU 40 <idle> <idle> cyclic test prio 90 <idle> Mine Mine irq thread prio 50 Mine

35 Pull RT Tasks IPI to push task CPU 0 CPU 1 CPU 2 CPU 40 <idle> <idle> cyclic test prio 90 <idle> irq thread prio 50

36 Pull RT Tasks IPI to push task CPU 0 CPU 1 CPU 2 CPU 40 <idle> <idle> cyclic test prio 90 <idle> IPI IPI irq thread prio 50 IPI

37 Pull RT Tasks IPI to push task CPU 0 CPU 1 CPU 2 CPU 40 <idle> irq thread prio 50 cyclic test prio 90 <idle>

38 The End? Looked like we found our bug! Started verification process Told everyone things will be verified shortly

39 Nope! Passed a 12 hour run Failed a 24 hour run Lets start again!

40 HW Lat Detector Hard ware latency detector Runs periodic stop machine Define a period and run run!= period system will lock up Spins looking for latency

41 HW Lat Issue While (now - start < perido) { tmp = timestamp(); now = timestamp(); diff = now - tmp; if (diff > thresh) record(); }

42 HW Lat Issue 80% While (now - start < perido) { tmp = timestamp(); now = timestamp(); diff = now - tmp; if (diff > thresh) record(); } 20%

43 HW Lat Issue Last = 0; While (now - start < perido) { tmp = timestamp(); now = timestamp(); if (last) { diff = tmp - last; if (diff > outer_thresh) record_outer(); } last = now; diff = now - tmp; if (diff > thresh) record(); }

44 HW Lat Detector Stop Machine Needs to run periodically Will lock up the system otherwise Has chance to miss latency again! Changed to a thread Thread takes up one of the CPUs Still needs to yield Locks up machine otherwise But yield is much smaller than periodic More likely to measure latency

45 But not good enough HW Lat Detector Worked! Vendor did not trust this code??? Had to use their code Did somewhat the same thing In userspace Could easily miss latencies

46 trace-cmd trace-cmd start -p function_graph -l raw_spin* -e all Modified cyclictest to use function graph Still limited to raw_spin* locks Wont disable the events started cyclictest will still stop the trace on latency

47 trace-cmd ksoftirqd/ [033] : timer_cancel: timer=0xffff88403f0ce520 ksoftirqd/ [033] : timer_expire_entry: timer=0xffff88403f0ce520 function=delayed_work ksoftirqd/ [033] : funcgraph_entry: us _raw_spin_lock_irqsave(); ksoftirqd/ [033] : funcgraph_entry: us _raw_spin_lock(); ksoftirqd/ [033] : sched_stat_sleep: comm=kworker/33:1 pid=1222 delay= ksoftirqd/ [033] : sched_wakeup: kworker/33:1:1222 [120] success=1 CPU:033 ksoftirqd/ [033] : timer_expire_exit: timer=0xffff88403f0ce520 ksoftirqd/ [033] : timer_cancel: timer=0xffff88403f0ce620 ksoftirqd/ [033] : timer_expire_entry: timer=0xffff88403f0ce620 function=delayed_work ksoftirqd/ [033] : funcgraph_entry: us _raw_spin_lock_irqsave(); ksoftirqd/ [033] : timer_expire_exit: timer=0xffff88403f0ce620 cyclictest-6110 [007] : funcgraph_entry: us _raw_spin_lock(); cyclictest-6110 [007] : funcgraph_entry: us _raw_spin_lock_irqsave(); cyclictest-6110 [007] : funcgraph_entry: us _raw_spin_lock_irqsave(); cyclictest-6113 [010] : funcgraph_entry: us _raw_spin_lock(); cyclictest-6110 [007] : funcgraph_entry: us _raw_spin_lock(); cyclictest-6110 [007] : sched_wakeup: cyclictest:6113 [4] success=1 CPU:010 cyclictest-6110 [007] : funcgraph_entry: us _raw_spin_lock_irqsave(); cyclictest-6113 [010] : funcgraph_entry: us _raw_spin_lock_irq(); cyclictest-6113 [010] : funcgraph_entry: us _raw_spin_lock_irqsave(); cyclictest-6113 [010] : funcgraph_entry: us _raw_spin_lock_irq(); cyclictest-6110 [007] : print: ffffffff810e5776 hit latency threshold (247 > 200)

48 trace-cmd trace-cmd report Lots of information Detailed information Great to analyze TOO MUCH INFO! Can not understand it all Hard to see the big picture KernelShark

49 kernelshark

50 kernelshark

51 kernelshark

52 kernelshark

53 kernelshark

54 kernelshark

55 kernelshark

56 kernelshark

57 Demo

58 Questions?

59 Questions? Yeah Right? Like we have time.