A Comparison of Scheduling Latency in Linux, PREEMPT_RT, and LITMUS RT. Felipe Cerqueira and Björn Brandenburg

A Comparison of Scheduling Latency in Linux, PREEMPT_RT, and LITMUS RT Felipe Cerqueira and Björn Brandenburg July 9th, 2013 1

Linux as a Real-Time OS 2

Linux as a Real-Time OS Optimizing system responsiveness 3

Linux as a Real-Time OS Optimizing system responsiveness PREEMPT_RT (Linux) 4

Linux as a Real-Time OS Optimizing system responsiveness Algorithmic changes based on real-time systems research PREEMPT_RT (Linux) 5

Linux as a Real-Time OS Optimizing system responsiveness Algorithmic changes based on real-time systems research PREEMPT_RT (Linux) 6

PREEMPT_RT Main real-time branch of Linux Goal: decrease scheduling latency through the use of low-level hacks Convert in-kernel spinlocks into (preemptable) mutexes Limit the extent of non-preemptable sections Commonly evaluated with cyclictest Single, easy-to-compare measure of scheduling latency as output 7

Testbed for applied real-time systems research Goal Allow implementation and evaluation of novel multiprocessor schedulers and synchronization protocols NOT to reduce scheduling latency Evaluated with Feather-Trace Flexible, fine-grained measurement of different overheads 8

Testbed for applied real-time systems research Goal How do LITMUS RT and PREEMPT_RT compare? Allow implementation and evaluation of novel multiprocessor schedulers and synchronization protocols NOT to reduce scheduling latency Evaluated with Feather-Trace Flexible, fine-grained measurement of different overheads 9

Testbed for applied real-time systems research Goal How do LITMUS RT and PREEMPT_RT compare? Allow implementation and evaluation of novel multiprocessor schedulers and synchronization protocols NOT to reduce scheduling latency It is not straightforward to Evaluated with Feather-Trace compare them! Flexible, fine-grained measurement of different overheads 10

Objective vs. PREEMPT_RT (Linux) Direct comparison of scheduling latency between LITMUS RT and PREEMPT_RT 11

Background 12

How is LITMUS RT evaluated? Evaluated with Lightweight tracing framework for measuring fine-grained overheads (e.g., IPI latency, context-switching overhead, etc.) Extensively used (20+ publications) Suitable for schedulability analysis Check if a task is going to miss a deadline 13

How is PREEMPT_RT evaluated? Evaluated with cyclictest Standard benchmark for assessing real-time responsiveness Creator: Thomas Gleixner Current maintainer: Clark Williams Reports scheduling latency as a single measure Treats hardware and OS as a black-box 14

Scheduling Latency Time until the highest-priority task is scheduled brake sensor HP task interrupt! ECU! ISR called 15

Scheduling Latency Time until the highest-priority task is scheduled interrupt! wake up task Z Z Z Z T T T! s ISR called scheduler invoked 16

Scheduling Latency Time until the highest-priority task is scheduled interrupt! wake up task T T T! s P ISR called scheduler invoked task picked 17

Scheduling Latency Time until the highest-priority task is scheduled interrupt! wake up task T T T Perform context switch! s P C ISR called scheduler invoked task picked switched 18

Scheduling Latency Time until the highest-priority task is scheduled interrupt! wake up task T T T Perform context switch scheduling latency! s P C ISR called scheduler invoked task picked switched 19

How does cyclictest measure Scheduling Latency? Init Measure Exit 20

How does cyclictest measure Scheduling Latency? POSIX sched_setscheduler( ) Init Measure Exit Measuring thread is granted real-time status. 21

How does cyclictest measure Scheduling Latency? Init Measure Exit 22

How does cyclictest measure Scheduling Latency? Init Measure Exit Periodically setup one-shot timers with nanosleep. Calculate delta between the instant the task starts executing and the instant the timer should have fired. 23

How does cyclictest measure Scheduling Latency? Init Measure Exit 24

How does cyclictest measure Scheduling Latency? POSIX sched_setscheduler( ) Init Measure Exit Measuring thread returns to best-effort status. 25

cyclictest on LITMUS RT Init Measure Exit LITMUS RT does not use POSIX API to setup real-time tasks! 26

cyclictest on LITMUS RT Init Measure Exit LITMUS RT does not use POSIX API to setup real-time tasks! cyclictest works, but does not measure what we expect... 27

Porting cyclictest to LITMUS RT POSIX sched_setscheduler( ) POSIX sched_setscheduler( ) Init Measure Exit LITMUS RT does not use POSIX API to setup real-time tasks! cyclictest works, but does not measure what we expect... 28

Porting cyclictest to LITMUS RT LITMUS RT API task_mode() LITMUS RT API task_mode() LITMUS RT Init Measure LITMUS RT Exit No changes in the measurement phase, no bias. 30

Study 31

Questions that We Address Stock Linux Userspace Scheduler + Dispatcher Linux (core) 32

The Cost of LITMUS RT LITMUS RT Stock Linux Userspace Scheduler + Dispatcher Linux (core) Userspace Scheduling Policy Plugins Dispatcher Linux (core) 33

The Cost of LITMUS RT LITMUS RT Stock Linux Userspace Scheduler + Dispatcher Linux (core) Userspace Scheduling Policy Plugins Dispatcher Linux (core) Question 1 How much latency does the scheduling policy interface add to the system? 34

LITMUS RT vs. PREEMPT_RT LITMUS RT Stock Linux Userspace Scheduler + Dispatcher Linux (core) Userspace Scheduling Policy Plugins Dispatcher Linux (core) PREEMPT_RT Userspace Scheduler + Dispatcher Linux (core) 35

LITMUS RT vs. PREEMPT_RT LITMUS RT Question 2 What is the penalty for LITMUS RT not being based on PREEMPT_RT? Userspace Scheduling Policy Plugins Dispatcher Linux (core) PREEMPT_RT Userspace Scheduler + Dispatcher Linux (core) 37

Evaluation Userspace Scheduler + Dispatcher Linux (core) 38

Evaluation cyclictest background Userspace + workload Scheduler + Dispatcher Linux (core) 39

Background Workloads NO background tasks CPU-bound background tasks I/O-bound background tasks 40

Experimental Setup Different kernels: 1.LITMUS RT (Linux 3.0) Partitioned Fixed Priority (P-FP), Partitioned EDF with synchronization support (PSN-EDF), Global EDF with synchronization support (GSN-EDF) 2.PREEMPT_RT (Linux 3.8.13) 3.Unpatched Linux 3.0 and Linux 3.8.13 } SCHED_FIFO 41

Experimental Setup 16-core Intel Xeon platform cyclictest s standard setup: one real-time task per processor periods: {1000, 1500, 2000,...} μs Duration: 20 minutes per experiment Almost 6 million samples for each case Results shown in microseconds 42

First Scenario NO background tasks 43

No Background Tasks Scheduling Latency (!s) 20 Average (99% conf.) Maximum 15 15.3 13.9 11.2 10 5 3.5 2.9 2.7 0 P-FP (LITMUS^RT) Linux 3.0 PREEMPT_RT 44

No Background Similar max. and Tasks avg. latency for Linux 3.0 and LITMUS RT. Scheduling Latency (!s) 20 Average (99% conf.) Maximum 15 15.3 13.9 11.2 10 5 3.5 2.9 2.7 0 P-FP (LITMUS^RT) Linux 3.0 PREEMPT_RT 45

No Background Similar max. and Tasks avg. latency for Linux 3.0 and LITMUS RT. Improved max. latency for PREEMPT_RT Scheduling Latency (!s) 20 Average (99% conf.) Maximum 15 15.3 13.9 11.2 10 5 3.5 2.9 2.7 0 P-FP (LITMUS^RT) Linux 3.0 PREEMPT_RT 46

Second Scenario CPU-bound background tasks Tasks running an infinite loop accessing memory (read/write) Working set larger than L2 cache size 47

Second Scenario CPU-bound background tasks Tasks running an infinite loop accessing memory (read/write) Working set larger than L2 cache size Generates memory traffic and cache contention! 48

CPU-bound Background Tasks Scheduling Latency (!s) 80 Average (99% conf.) 72.7 Maximum 60 47.6 40 20 0 17.4 5.2 4.2 3.4 P-FP (LITMUS^RT) Linux 3.0 PREEMPT_RT 49

CPU-bound Background Tasks Scheduling Latency (!s) Average (99% conf.) Maximum 80 72.7 60 40 47.6 Latency under PREEMPT_RT is significantly lower. 20 0 17.4 5.2 4.2 3.4 P-FP (LITMUS^RT) Linux 3.0 PREEMPT_RT 50

CPU-bound Background Tasks LITMUS RT s latency lower than on Linux 3.0? Scheduling Latency (!s) 80 60 47.6 Average (99% conf.) 72.7 Maximum Latency under PREEMPT_RT is significantly lower. 40 20 0 17.4 5.2 4.2 3.4 P-FP (LITMUS^RT) Linux 3.0 PREEMPT_RT 51

LITMUS RT vs. Linux 3.0:! CPU-bound Background Tasks log scale! number of samples 1e+07 1e+06 100000 10000 1000 100 10 1 Linux 3.0: scheduling latency (CPU-bound bg tasks) Linux 3.0 P-FP(LITMUS min=2.04us max=72.73us avg=4.22us median=3.86us stdev=1.37us RT ) avg=4.22!s max=72.73!s avg=5.17!s max=47.59!s samples: total=5854711 0 10 20 30 40 50 60 70 80 90 Scheduling Latency (!s) (Bin size = 1!s) overhead in microseconds (bin size = 1.00us) 52

LITMUS RT vs. Linux 3.0:! CPU-bound Background Tasks number of samples 1e+07 1e+06 100000 10000 1000 100 10 1 Linux 3.0: scheduling latency (CPU-bound bg tasks) Linux 3.0 P-FP(LITMUS min=2.04us max=72.73us avg=4.22us median=3.86us stdev=1.37us RT ) avg=4.22!s max=72.73!s avg=5.17!s max=47.59!s samples: total=5854711 3 samples out of ~6 million 0 10 20 30 40 50 60 70 80 90 Scheduling Latency (!s) (Bin size = 1!s) overhead in microseconds (bin size = 1.00us) 53

LITMUS RT vs. Linux 3.0:! CPU-bound Background Tasks number of samples 1e+07 1e+06 100000 10000 1000 100 10 1 Linux 3.0: scheduling latency (CPU-bound bg tasks) Linux 3.0 P-FP(LITMUS min=2.04us max=72.73us avg=4.22us median=3.86us stdev=1.37us RT ) avg=4.22!s max=72.73!s avg=5.17!s max=47.59!s Slightly worse latencies on average 0 10 20 30 40 50 60 70 80 90 Scheduling Latency (!s) (Bin size = 1!s) overhead in microseconds (bin size = 1.00us) samples: total=5854711 54

LITMUS RT vs. Linux 3.0:! CPU-bound Background Tasks number of samples 1e+07 1e+06 100000 10000 1000 100 10 1 Linux 3.0: scheduling latency (CPU-bound bg tasks) Linux 3.0 P-FP(LITMUS min=2.04us max=72.73us avg=4.22us median=3.86us stdev=1.37us RT ) avg=4.22!s max=72.73!s avg=5.17!s max=47.59!s 0 10 20 30 40 50 60 70 80 90 Scheduling Latency (!s) (Bin size = 1!s) overhead in microseconds (bin size = 1.00us) samples: total=5854711 extra spinlock lack of low-level optimizations 55

CPU-bound Background Tasks LITMUS RT incurs slightly Scheduling more latency Latency (!s) than 80Linux 3.0 on average. 60 47.6 Average (99% conf.) 72.7 Maximum Latency under PREEMPT_RT is significantly lower. 40 20 0 17.4 5.2 4.2 3.4 P-FP (LITMUS^RT) Linux 3.0 PREEMPT_RT 56

Third Scenario I/O-bound background tasks hackbench: Linux scheduler stress tool bonnie++: Disk and file system benchmark wget: Network activity 57

Third Scenario I/O-bound background tasks Causes a lot of system calls and interrupts hackbench: Linux scheduler stress tool bonnie++: Disk and file system benchmark wget: Network activity 58

I/O-bound Background Tasks log scale! Scheduling Latency (!s) 10,000 3956.5 Average (99% conf.) 4300.4 Maximum 1,000 100 44.2 10 6.6 6.4 4.1 1 P-FP (LITMUS^RT) Linux 3.0 PREEMPT_RT 59

Huge impact on I/O-bound Background scheduling Tasks latency under standard Linux. Scheduling Latency (!s) Average (99% conf.) Maximum 10,000 3956.5 4300.4 1,000 100 44.2 10 6.6 6.4 4.1 1 P-FP (LITMUS^RT) Linux 3.0 PREEMPT_RT 60

Huge impact on I/O-bound Background scheduling Tasks latency under standard Linux. Scheduling Latency (!s) Average (99% conf.) Maximum 10,000 3956.5 4300.4 1,000 PREEMPT_RT is not affected by the interrupt load! 100 44.2 10 6.6 6.4 4.1 1 P-FP (LITMUS^RT) Linux 3.0 PREEMPT_RT 61

Summary 1.Cost of the scheduling plugin layer 2.LITMUS RT vs. PREEMPT_RT 62

Summary 1.Cost of the scheduling plugin layer The overhead introduced by LITMUS RT is small 2.LITMUS RT vs. PREEMPT_RT 63

Summary 1.Cost of the scheduling plugin layer The overhead introduced by LITMUS RT is small 2.LITMUS RT vs. PREEMPT_RT PREEMPT_RT significantly decreases scheduling latency. 64

Importance of Feather-Trace cyclictest was ported to LITMUS RT. Should it become the standard tool for evaluating LITMUS RT? 65

Importance of Feather-Trace cyclictest was ported to LITMUS RT. Should it become the standard tool for evaluating LITMUS RT? NO! 66

Interference? OOPS... a higher priority task interrupt! wake up task T T T! s ISR called scheduler invoked 67

Interference? interrupt! wake up task T T T! s? ISR called scheduler invoked another task picked 68

Interference? interrupt! wake up task T T T! s?... P ISR called scheduler invoked another task picked task picked 69

Interference? interrupt! wake up task T This length of this interval depends on the execution of other tasks... T T! s?... P ISR called scheduler invoked another task picked task picked 70

Interference?..., which depends on other kinds of overhead, preemptions, context switches, etc. interrupt! wake up task T This length of this interval depends on the execution of other tasks... T T! s?... P ISR called scheduler invoked another task picked task picked 71

Interference?..., which depends on other kinds of overhead, preemptions, context switches, etc. interrupt! wake up task This length of this interval depends on the execution of other tasks... T T T Overhead-aware schedulability analysis is required!! s?... P ISR called scheduler invoked another task picked task picked 72

cyclictest or Feather-Trace? BOTH! cyclictest LITMUS RT /Feather-Trace Practical, easy-tounderstand measure Can easily compare responsiveness between kernels. For tasks other than the highest-priority ones, schedulability analysis is necessary. Only with Feather-Trace we obtain the data required for the analysis. 73

Conclusion LITMUS RT : small overheads in comparison with stock Linux PREEMPT_RT is highly necessary for Linux as a RTOS LITMUS RT will be ported to PREEMPT_RT soon Scheduling latency should not be used as the sole metric for quantifying real-time guarantees 74

Thank You! We also have a patch that implements Feather-Trace on top of standard Linux, enabling fine-grained measurements. 75

Appendix 76

Linux 3.0 vs. Linux 3.8.13 77

No Background Tasks Scheduling Latency (!s) 25 20 Average Maximum 19.7 15 13.9 10 5 2.9 2.9 0 Linux 3.0 Linux 3.8.13 78

No Background Tasks Scheduling Latency (!s) 25 Average Maximum 20 Similar averages 15 10 13.9 19.7 5 2.9 2.9 0 Linux 3.0 Linux 3.8.13 79

No Background Tasks number of samples number of samples 1e+07 1e+06 100000 10000 1000 100 10 1 1e+07 1e+06 100000 10000 1000 100 10 Linux 3.0 Linux 3.0: scheduling latency (no bg tasks) min=1.87us max=13.89us avg=2.89us median=2.77us stdev=0.51us avg=2.89!s max=13.89!s Linux 3.8.13 samples: total=5854779 0 10 20 30 40 50 60 70 80 90 Linux 3.8.13: scheduling latency (no bg tasks) min=1.52us overhead max=19.73us in microseconds avg=2.89us (bin median=2.58us size = 1.00us) stdev=0.69us avg=2.89!s max=19.73!s samples: total=5854801 Similar shapes 1 0 10 20 30 40 50 60 70 80 90 80

CPU-bound Background Tasks Scheduling Latency (!s) 80 72.7 60 Average Maximum 64.5 40 20 0 4.2 4.0 Linux 3.0 Linux 3.8.13 81

CPU-bound Background Tasks Scheduling Latency (!s) 80 72.7 60 Similar averages 40 Average Maximum 64.5 20 0 4.2 4.0 Linux 3.0 Linux 3.8.13 82

CPU-bound Background Tasks number of samples number of samples 1e+07 1e+06 100000 10000 1000 100 10 1 1e+07 1e+06 100000 10000 1000 100 10 Linux 3.0 Linux 3.0: scheduling latency (CPU-bound bg tasks) min=2.04us max=72.73us avg=4.22us median=3.86us stdev=1.37us avg=4.22!s max=72.73!s Linux 3.8.13 samples: total=5854711 0 10 20 30 40 50 60 70 80 90 Linux 3.8.13: scheduling latency (CPU-bound bg tasks) min=2.14us overhead max=64.47us in microseconds avg=4.02us (bin median=3.67us size = 1.00us) stdev=1.20us avg=4.02!s max=64.47!s samples: total=5854707 Similar shapes 1 0 10 20 30 40 50 60 70 80 90 83

I/O-bound Background Tasks log scale! Scheduling Latency (!s) 10,000 4300.4 Average Maximum 5464.1 1,000 100 10 6.4 6.2 1 Linux 3.0 Linux 3.8.13 84

I/O-bound Background Tasks Scheduling Latency (!s) 10,000 4300.4 1,000 Similar averages 100 Average Maximum 5464.1 10 6.4 6.2 1 Linux 3.0 Linux 3.8.13 85

I/O-bound Background Tasks number of samples number of samples 1e+07 1e+06 100000 10000 1000 100 10 1 1e+07 1e+06 100000 10000 1000 100 10 Linux 3.0 Linux 3.0: scheduling latency (IO-bound bg tasks) min=1.85us max=4300.43us avg=6.39us median=4.98us stdev=13.25us avg=6.39!s max=4300.43!s Linux 3.8.13 samples: total=5854674 0 150 300 450 600 750 900 1050 1200 1350 Linux 3.8.13: scheduling latency (IO-bound bg tasks) min=1.85us max=5464.07us overhead in microseconds avg=6.23us (bin median=4.60us size = 1.00us) stdev=15.91us avg=6.23!s max=5464.07!s samples: total=5854773 Similar shapes 1 0 150 300 450 600 750 900 1050 1200 1350 86

LITMUS RT s plugins 87

No Background Tasks Scheduling Latency (!s) 30 22.5 Average 26.2 Maximum 15 15.1 14.3 7.5 3.5 3.5 3.1 0 P-FP PSN-EDF GSN-EDF 88

No Background Tasks Similar shapes 1e+07 1e+06 100000 10000 1000 100 10 1 LITMUS^RT G-EDF: scheduling latency (no bg tasks) min=1.59us max=14.34us avg=3.06us median=2.56us stdev=1.18us 1e+07 samples: total=5854797 1e+06 0 10 20 30 40 50 60 70 80 90 number of samples LITMUS^RT P-EDF: scheduling latency (no bg tasks) min=1.76us max=26.17us avg=3.45us median=2.87us stdev=1.24us 100000 10000 1000 100 10 1 0 10 20 30 40 50 60 70 80 90 overhead in microseconds (bin size = 1.00us) overhead in microseconds (bin size = 1.00us) LITMUS^RT P-FP: scheduling P-FPlatency (no bg tasks) min=1.96us max=15.13us avg=3.45us median=3.10us stdev=1.03us number of samples GSN-EDF avg=3.06!s max=14.34!s 1e+07 1e+06 100000 10000 1000 100 10 1 avg=3.45!s max=15.13!s samples: total=5854818 0 10 20 30 40 50 60 70 80 90 overhead in microseconds (bin size = 1.00us) PSN-EDF avg=3.45!s max=26.17!s samples: total=5854783 89

No Background Tasks Similar shapes 1e+07 1e+06 100000 10000 1000 100 10 1 LITMUS^RT G-EDF: scheduling latency (no bg tasks) min=1.59us max=14.34us avg=3.06us median=2.56us stdev=1.18us 1e+07 samples: total=5854797 1e+06 number of samples 100000 10000 1000 100 10 1 LITMUS^RT P-EDF: scheduling latency (no bg tasks) min=1.76us max=26.17us avg=3.45us median=2.87us stdev=1.24us 0 10 20 30 40 50 60 70 80 90 0 10 20 30 40 50 60 70 80 90 overhead in microseconds (bin size = 1.00us) overhead in microseconds (bin size = 1.00us) LITMUS^RT P-FP: scheduling P-FPlatency (no bg tasks) min=1.96us max=15.13us avg=3.45us median=3.10us stdev=1.03us number of samples GSN-EDF avg=3.06!s max=14.34!s 1e+07 1e+06 100000 10000 1000 100 10 1 avg=3.45!s max=15.13!s samples: total=5854818 0 10 20 30 40 50 60 70 80 90 overhead in microseconds (bin size = 1.00us) PSN-EDF avg=3.45!s max=26.17!s 2 samples samples: total=5854783 90

CPU-bound Background Tasks Scheduling Latency (!s) 80 Average 73.3 Maximum 60 40 47.6 60.2 20 0 5.2 5.1 5.8 P-FP PSN-EDF GSN-EDF 91

CPU-bound Background Tasks 1e+07 1e+06 100000 LITMUS^RT G-EDF: scheduling latency (CPU-bound bg tasks) min=1.91us max=60.20us avg=5.81us median=5.39us stdev=2.51us samples: total=5854728 1e+07 1e+06 number of samples 100000 LITMUS^RT P-EDF: scheduling latency (CPU-bound bg tasks) min=2.40us max=73.27us avg=5.14us median=4.21us stdev=2.95us samples: total=5854739 10000 10000 1000 1000 100 100 10 10 1 1 0 10 20 30 40 50 60 70 80 90 0 10 20 30 40 50 60 70 80 90 overhead in microseconds (bin size = 1.00us) overhead in microseconds (bin size = 1.00us) LITMUS^RT P-FP: scheduling P-FP latency (CPU-bound bg tasks) min=2.10us max=47.59us avg=5.17us median=4.37us stdev=2.75us number of samples GSN-EDF avg=5.81!s max=60.20!s 1e+07 1e+06 100000 10000 1000 100 10 1 avg=5.17!s max=47.59!s 0 10 20 30 40 50 60 70 80 90 overhead in microseconds (bin size = 1.00us) samples: total=5854719 PSN-EDF avg=5.14!s max=73.27!s Similar shapes 92

CPU-bound Background Tasks 1e+07 1e+06 100000 LITMUS^RT G-EDF: scheduling latency (CPU-bound bg tasks) min=1.91us max=60.20us avg=5.81us median=5.39us stdev=2.51us samples: total=5854728 1e+07 1e+06 number of samples 100000 LITMUS^RT P-EDF: scheduling latency (CPU-bound bg tasks) min=2.40us max=73.27us avg=5.14us median=4.21us stdev=2.95us samples: total=5854739 10000 10000 1000 1000 100 100 5 samples 10 10 1 1 0 10 20 30 40 50 60 70 80 90 0 10 20 30 40 50 60 70 80 90 overhead in microseconds (bin size = 1.00us) overhead in microseconds (bin size = 1.00us) LITMUS^RT P-FP: scheduling P-FP latency (CPU-bound bg tasks) min=2.10us max=47.59us avg=5.17us median=4.37us stdev=2.75us number of samples GSN-EDF avg=5.81!s max=60.20!s 1e+07 1e+06 100000 10000 1000 100 10 1 avg=5.17!s max=47.59!s 0 10 20 30 40 50 60 70 80 90 overhead in microseconds (bin size = 1.00us) samples: total=5854719 PSN-EDF avg=5.14!s max=73.27!s Similar shapes 93

I/O-bound Background Tasks log scale! Scheduling Latency (!s) 10,000 3956.5 Average 3875.0 Maximum 3905.8 1,000 100 10 6.6 6.6 11.0 1 P-FP PSN-EDF GSN-EDF 94

I/O-bound Background Tasks Scheduling Latency (!s) 10,000 3956.5 Similar results 1,000for P-FP and PSN-EDF 100 Average 3875.0 Maximum 3905.8 10 6.6 6.6 11.0 1 P-FP PSN-EDF GSN-EDF 95

I/O-bound Background Tasks Scheduling Latency (!s) 10,000 3956.5 Similar results 1,000for P-FP and PSN-EDF 100 Average 3875.0 Maximum Higher average for 3905.8 GSN-EDF 10 6.6 6.6 11.0 1 P-FP PSN-EDF GSN-EDF 96

I/O-bound Background Tasks 1e+07 1e+06 100000 10000 1000 100 10 1 LITMUS^RT G-EDF: scheduling latency (IO-bound bg tasks) min=2.26us max=3905.79us avg=10.95us median=7.38us stdev=14.11us 1e+07 samples: total=5854793 1e+06 0 150 300 450 600 750 900 1050 1200 1350 number of samples 100000 10000 1000 100 10 1 LITMUS^RT P-EDF: scheduling latency (IO-bound bg tasks) min=1.98us max=3874.99us avg=6.56us median=5.11us stdev=12.66us 0 150 300 450 600 750 900 1050 1200 1350 overhead in microseconds (bin size = 1.00us) overhead in microseconds (bin size = 1.00us) LITMUS^RT P-FP: scheduling P-FP latency (IO-bound bg tasks) min=1.89us max=3956.48us avg=6.60us median=5.17us stdev=12.76us number of samples GSN-EDF avg=10.95!s max=3905.79!s 1e+07 1e+06 100000 10000 1000 100 10 1 avg=6.60!s max=3956.48!s 0 150 300 450 600 750 900 1050 1200 1350 overhead in microseconds (bin size = 1.00us) samples: total=5854660 PSN-EDF avg=6.56!s max=3874.99!s samples: total=5854606 Similar shapes 97

I/O-bound Background Tasks 1e+07 1e+06 100000 10000 1000 100 10 1 LITMUS^RT G-EDF: scheduling latency (IO-bound bg tasks) min=2.26us max=3905.79us avg=10.95us median=7.38us stdev=14.11us 1e+07 samples: total=5854793 1e+06 0 150 300 450 600 750 900 1050 1200 1350 number of samples 100000 10000 1000 100 10 1 LITMUS^RT P-EDF: scheduling latency (IO-bound bg tasks) min=1.98us max=3874.99us avg=6.56us median=5.11us stdev=12.66us 0 150 300 450 600 750 900 1050 1200 1350 overhead in microseconds (bin size = 1.00us) overhead in microseconds (bin size = 1.00us) LITMUS^RT P-FP: scheduling P-FP latency (IO-bound bg tasks) min=1.89us max=3956.48us avg=6.60us median=5.17us stdev=12.76us number of samples GSN-EDF avg=10.95!s max=3905.79!s Higher average 1e+07 1e+06 100000 10000 1000 100 10 1 avg=6.60!s max=3956.48!s 0 150 300 450 600 750 900 1050 1200 1350 overhead in microseconds (bin size = 1.00us) samples: total=5854660 PSN-EDF avg=6.56!s max=3874.99!s samples: total=5854606 Similar shapes 98

threadirqs in Linux 3.8.13 99

No Background Tasks number of samples number of samples Linux 3.8.13 threadirqs Linux 3.8.13: scheduling latency (no bg tasks) min=1.60us max=25.15us avg=2.82us median=2.65us stdev=0.47us avg=2.82!s max=25.15!s 1e+07 samples: total=5854778 1e+06 100000 10000 1000 100 10 1 0 10 20 30 40 50 60 70 80 90 Linux 3.8.13: scheduling Linux latency 3.8.13 (no bg tasks) min=1.52us overhead max=19.73us in microseconds avg=2.89us (bin median=2.58us size = 1.00us) stdev=0.69us avg=2.89!s max=19.73!s 1e+07 samples: total=5854801 1e+06 100000 10000 1000 100 10 1 0 10 20 30 40 50 60 70 80 90 100

CPU-bound Background Tasks number of samples number of samples 1e+07 1e+06 100000 10000 1000 100 10 1 1e+07 1e+06 100000 10000 1000 100 10 1 Linux 3.8.13 threadirqs Linux 3.8.13: scheduling latency (CPU-bound bg tasks) min=1.95us max=40.90us avg=3.67us median=3.29us stdev=1.19us avg=3.67!s max=40.90!s Linux 3.8.13 samples: total=5854684 0 10 20 30 40 50 60 70 80 90 Linux 3.8.13: scheduling latency (CPU-bound bg tasks) min=2.14us overhead max=64.47us in microseconds avg=4.02us (bin median=3.67us size = 1.00us) stdev=1.20us avg=4.02!s max=64.47!s samples: total=5854707 0 10 20 30 40 50 60 70 80 90 101

I/O-bound Background Tasks number of samples number of samples 1e+07 1e+06 100000 10000 1000 100 10 1 1e+07 1e+06 100000 10000 1000 100 10 1 Linux 3.8.13 threadirqs Linux 3.8.13: scheduling latency (IO-bound bg tasks) min=1.51us max=5203.38us avg=5.89us median=4.67us stdev=14.33us avg=5.89!s max=5203.38!s Linux 3.8.13 samples: total=5854724 0 150 300 450 600 750 900 1050 1200 1350 Linux 3.8.13: scheduling latency (IO-bound bg tasks) min=1.85us max=5464.07us overhead in microseconds avg=6.23us (bin median=4.60us size = 1.00us) stdev=15.91us avg=6.23!s max=5464.07!s samples: total=5854773 0 150 300 450 600 750 900 1050 1200 1350 102