How Linux RT_PREEMPT Works

Transcription

1

2 How Linux RT_PREEMPT Works A common observation about real time systems is that the cost of the increased determinism of real time is decreased throughput and increased average latency. This presentation enumerates some of the design choices and implementation that enable Linux PREEMPT_RT_FULL real time and the resulting performance implications. Frank Rowand, Sony Network Entertainment September 20, _1647

3 The Trade-off The cost of the increased determinism of real time - Increased average latency - Decreased throughput

4 Some Random Data Compare the latency of an application on kernels built with: 1) CONFIG_PREEMPT_NONE 2) CONFIG_PREEMPT_RT_FULL

5 Some Random Data Test System: - ARM11 MPCore development system - 4 cpus Mhz processor clock - L1 cache 64 Kbyte per processor - L2 cache 1 Mbyte unified - Linux rt17

6

7 Latency (Response Time) Kernel without RT patchset: + smaller average + smaller minimum - larger maximum PREEMPT RT enabled: - larger average - larger minimum + smaller maximum + more consistent

8 Statistics Min Avg Max PREEMPT_NONE PREEMPT_RT_FULL

9 Latency (Response Time) Next graph shows an old kernel, circa 2009 Hardware configuration: unknown, server class

10 Red Hat Enterprise Linux Red Hat MRG tuned Messaging Workload source: Red Hat

11 The Trade-off The previous graphs illustrate real results on: - variety of kernel versions - range of hardware, from embedded to server

12 The Trade-off The cost of the increased determinism of real time - Increased average latency - Decreased throughput

13 Some Random Data Compare the throughput of an application on kernels built with: 1) CONFIG_PREEMPT_NONE 2) CONFIG_PREEMPT_RT_FULL

14 Some Random Data The workload I used for the throughput results - is not realistic - is not reasonable - violates real time application design rules - is stupid! - but was easy to implement...

15 Some Random Data Test System (same as first test system): - ARM11 MPCore development system - 4 cpus Mhz processor clock - L1 cache 64 Kbyte per processor - L2 cache 1 Mbyte unified - Linux rt17

16 Test Variables UP vs SMP SMP, maxcpus=4 vs SMP maxcpus=1 workload: SCHED_FIFO vs SCHED_NORMAL 1, 2, or 4 instances of the workload Permutations of variables results in 10 tests

17 Test Data The following graphs show the duration of each test. Longer duration means lower throughput, so larger duration is worse throughput.

18

19

20

21 The Trade-off The previous graphs illustrate real results on: - variety of test loads - range of hardware, UP and SMP

22 The Trade-off The cost of the increased determinism of real time - Increased average latency - Decreased throughput

23 What Latency Can I Expect? It depends...

24

25

26 Part 2 This presentation enumerates some of the design choices and implementation that enable Linux PREEMPT_RT_FULL real time and the resulting performance implications.

27 Real Time 001 Working definition of latency The duration of time from when an event occurs until real time application code begins to execute.

28 Real Time 001 Examples of events - hardware clock (timer expires) - switch closes - optical sensor triggers - changed data between video frames predicts contact of beer bottle by robotic hand

29 Real Time 001 Working definition of deadline Maximum allowable elapsed time after event for RT code to begin executing + RT code to respond to the event

30 Real Time 001 deadline = latency + RT execution (1) Max allowed latency = deadline - max RT execution (2) Max allowed RT execution = deadline - max latency (1) max RT executed is a given (2) max latency is a given

31 Real Time 001 deadline = latency + RT execution (1) Max allowed latency = deadline - max RT execution (2) Max allowed RT execution = deadline - max latency (1) is a kernel developer's view (2) is a real time app developer's view

32 Real Time 001 deadline = latency + RT execution (1) Max allowed latency = deadline - max RT execution (1) is a kernel developer's view This presentation is the kernel developer's view

33 Required for Real-Time - Deterministic latency small enough to meet deadline - Real-time is NOT real-fast But fast (small latency) is often required in order to meet real-time application deadline.

34 Components of Latency while kernel code is not allowed to process event - interrupts disabled

35 Components of Latency cost of kernel code path to process event - save processor state - decode and acknowledge IRQ - driver handles device and transfers data - wake up real time thread

36 Components of Latency while RT code is not allowed to run - interrupts disabled - preemption disabled

37 Components of Latency cost of kernel code path to schedule RT task

38 Components of Latency while kernel code is not allowed to process event - interrupts disabled cost of kernel code path to process event while RT code is not allowed to run - interrupts disabled - preemption disabled cost of kernel code path to schedule RT task

39 Critical Components of Latency while kernel code is not allowed to process event - interrupts disabled cost of kernel code path to process event while RT code is not allowed to run - interrupts disabled - preemption disabled cost of kernel code path to schedule RT task

40 Side track... Q. How can the developer of a real time system negatively impact latency? A. Invoke code that leads to: - interrupts disabled - preemption disabled - block on lock

41 Features enabling real-time Linux - preemptible kernel - locking - threaded interrupt handlers - threaded softirq

42 Non-Preemptible Kernel When a task invokes a system call, the system call must complete (or sleep due to blocking on a resource) before another task can be scheduled. Preemption can not occur during the execution of the system call.

43 Non-Preemptible Kernel Preemption can not occur during the execution of the system call. Scheduling may occur on: - completion of system call - system call sleeping

44 Non-Preemptible Kernel Problems of typical non-preemptible kernel: - kernel path lengths non-deterministic - longest kernel path has long duration - large variance in kernel path length

45 Non-Preemptible Kernel Next slide illustrates non-preemptible kernel.

46 syscall external syscall event completes RT task Normal task syscall IRQ handler wake RT task RT task runs

47 Non-Preemptible Kernel Next slide illustrates non-preemptible kernel. Adding some complexity: - 2 external events occur - lock (critical section) during syscall

48 critical section syscall external syscall events completes RT task Normal task syscall IRQ 1 handler IRQ 2 handler wake RT task RT task runs

49 Preemptible Kernel Mainline 2.6 and 3.0 kernel CONFIG_PREEMPT_NONE No forced kernel preemption CONFIG_PREEMPT_VOLUNTARY Explicit preemption points in kernel CONFIG_PREEMPT All kernel code (not in critical section) preemptible

50 Preemptible Kernel RT_PREEMPT patch renames config option: Vanilla 2.6 kernel CONFIG_PREEMPT RT_PREEMPT 2.6 kernel CONFIG_PREEMPT_DESKTOP

51 Preemptible Kernel RT_PREEMPT patch renames config option: Vanilla 3.0 kernel CONFIG_PREEMPT RT_PREEMPT 3.0 kernel CONFIG_PREEMPT_LL

52 Preemptible Kernel Mainline 2.6 and 3.0 kernel CONFIG_PREEMPT fully preemptible - except when preemption is explicitly disabled - except when interrupts are explicitly disabled - except when a lock is held ( in a critical section )

53 Preemptible Kernel Next slide illustrates preemptible kernel.

54 critical section syscall external syscall events completes RT task Normal task syscall IRQ 1 handler IRQ 2 handler wake RT task RT task runs

55 Score - added 0 schedule with context switch + shorter maximum wakeup latency

56 Preemptible Kernel RT_PREEMPT 2.6 kernel CONFIG_PREEMPT_RT fully preemptible - except when preemption is explicitly disabled - except when interrupts are explicitly disabled - except when a raw spinlock is held

57 Preemptible Kernel RT_PREEMPT 3.0 kernel CONFIG_PREEMPT_RT_FULL fully preemptible - except when preemption is explicitly disabled - except when interrupts are explicitly disabled - except when a raw spinlock is held

58 Preemptible Kernel CONFIG_PREEMPT_RT CONFIG_PREEMPT_RT_FULL Most kernel locks are converted to preemptible priority inheritance mutex. Some kernel locks are converted to non-preemptible raw spinlock.

59 Score: Spinlock vs Mutex Acquire, Release Spinlock shorter code path Mutex longer code path

60 Score: Spinlock vs Mutex Contention Spinlock spin until lock is available Mutex if owner is executing spin until lock is available else sleep, wake when lock is available (2 schedules)

61 Score: Spinlock vs Mutex Result with contention Spinlock is cheaper than mutex if: spinlock lock path + spinlock unlock path + lock contention time < mutex lock path + mutex unlock path + 2 schedules

62 Score: Spinlock vs Mutex Result with contention Previous slide ignores: - extra schedules due to priority inheritance - cache effects

63 Preemptible Kernel Next slide illustrates preemptible kernel with spinlocks converted to mutexes.

64 critical section (mutex) syscall external syscall events completes RT task Normal task syscall IRQ 1 handler IRQ 2 handler wake RT task RT task runs

65 Score - added 0 schedule with context switch + shorter maximum wakeup latency

66 Priority Inheritance Mutex - May result in more schedule events. - Avoids priority inversion. - Typically larger execution cost than spinlock. - Reader-Writer lock limited to one concurrent reader to minimize PI complexity. - Limits scalability of multiple readers.

67 non-pi Mutex Priority Inversion higher number is higher priority A p = 1 B p = 2 C p = 3 A gets lock B wakes C wakes C blocks on lock

68 PI Mutex higher number is higher priority A B C p = 1 p = 3 p = 1 p = 2 p = 3 A gets lock B wakes C wakes C blocks on lock, A boosted to 3 A releases lock, unboosted back to 1, C gets lock

69 Threaded Interrupt Handler

70 Overview Of Interrupt handling algorithm - Save context - Handle highest priority interrupt Interrupt handler executes in interrupt mode irq_exit() may process softirq or wake softirqd - Iterate over active interrupts (arch dependent) - Schedule - Restore context returning either to previous process or to newly scheduled process

71 Overview Of Threaded Interrupt handling algorithm - Save context - Handle highest priority interrupt Wake Interrupt handler thread. irq_exit() may process softirq or wake softirqd - Iterate over active interrupts (arch dependent) - Schedule - Restore context returning either to previous process or to newly scheduled process Interrupt handler thread executes when scheduled.

72 Threaded Interrupt Handler RT_PREEMPT patchset converts almost all drivers to threaded model. (Timer handler executes in interrupt context.)

73 Threaded Interrupt Handler Priorities must be properly set for: - interrupt handler threads - softirq threads - other kernel threads - real time application processes / threads Do not expect default priorities to be proper.

74 Preemptible Kernel Next slide illustrates preemptible kernel with interrupt threads.

75 critical section (mutex) syscall external syscall events completes RT task Normal task syscall IRQ 1 handler IRQ 2 handler wake RT task RT task runs

76 Score - added 2 schedule with context switch + shorter maximum wakeup latency

77 Other Interrupt Overhead

78 Other Interrupt Overhead CONFIG_PREEMPT_RT and CONFIG_PREEMPT_RT_FULL changes: to: irq_exit() may process softirq or wake ksoftirqd thread irq_exit() may wake ksoftirqd thread

79 syscall external syscall events completes RT task Normal task syscall IRQ 1 IRQ 2 softirq

80 syscall external syscall events completes RT task Normal task syscall IRQ 1 IRQ 2 softirq

81 Score - added 1 schedule with context switch + shorter maximum wakeup latency

82 Other Interrupt Overhead raise_softirq(), (softirq trigger) typically called from: irq context (timer softirq) interrupt thread but can be called from anywhere in the kernel. Previous slides show trigger from irq context.

83 Other Locking Overhead CONFIG_TREE_PREEMPT_RCU Evolving in the early 3.0 RT patches... Not analyzed in this presentation.

84 Other Locking Overhead local_lock get_local_var() uses migrate_disable() instead of preempt_disable(). Evolving in the early 3.0 RT patches... Not analyzed in this presentation.

85 Real Life Real systems are much more complicated than the previous diagrams. Other scenarios can generate different performance improvements or penalties.

86 Real Life The flow of control diagrams are not an exhaustive analysis of why and how much PREEMPT_RT_FULL impacts performance metrics. But instead provide scenarios illustrating the impacts.

87 Full Analysis of Overhead Would Include: Lock implementation overhead Reduced concurrency for Reader-Writer locks Increased lock contention, due to sleeping while holding lock Additional schedules Additional context switches

88 Recap: Enabling real-time Linux - preemptible kernel - locking - interrupt handlers - threaded softirq

89 Impact of Real-Time Features + Variance of real-time task latency decreased + Maximum real-time task latency decreased - Average real-time task latency may be increased - Throughput decreased

90 Questions?

91 How to get a copy of the slides 1) leave a business card with me 2) frank.rowand@am.sony.com

92