Predictable Interrupt Management and Scheduling in the Composite Component-based System

Size: px

Start display at page:

Download "Predictable Interrupt Management and Scheduling in the Composite Component-based System"

Jared Glenn
5 years ago
Views:

1 Predictable Interrupt Management and Scheduling in the Composite Component-based System Gabriel Parmer and Richard West Computer Science Department Boston University Boston, MA {gabep1, December 2, 2008

2 Motivation, Goals, and Challenges Customizable/extensible base system upon which wide range of systems can be built with application-specific services and abstractions System dependability challenges code complexity complicate testing and verification focus on fault tolerance services provided by 3rd party, untrusted developers schedulers, synchronization policies,... Can we provide a canonical base system that is predictable and dependable? Parmer, West, BU CS Scheduling in Composite 2/33

3 The Composite Component-Based OS System policies and abstractions defined in separate components (schedulers, synchronization policies, etc...) at user-level in their own protection domains Constrain scope of impact of component faults Focus on application-specific system composition system-provided dependability Parmer, West, BU CS Scheduling in Composite 3/33

4 Simple Composite System Task 1 (T1) Task 2 (T2) Network (N) Demultiplexer (DM) Deferrable Server (DS) Fixed Priority, Round Robin (FPRR) Network Device Timer components related via dependency Parmer, West, BU CS Scheduling in Composite 4/33

5 User-Level Component-Based Scheduling Correctness of RT systems dependent on tasks temporal behaviors Goal: Define system scheduling policies in components control task and interrupt execution maintain accurate execution accountability must be efficient and predictable Challenge: increased overhead of scheduler invocation inter-protection domain communication Parmer, West, BU CS Scheduling in Composite 5/33

6 Composite Scheduler Components Composite kernel provides trusted communication between components scheduling external to kernel Composite includes functions to create a hierarchy of schedulers grant control of specific threads to certain schedulers Schedulers multiplex CPU via cos switch thread(thd id,...) Parmer, West, BU CS Scheduling in Composite 6/33

7 Component Invocations User Kernel Common design: threads in protection domains that communicate with each other via IPC each IPC includes scheduling decisions user-level component scheduler significant overhead, potentially increased WCET of a critical path Parmer, West, BU CS Scheduling in Composite 7/33

8 Component Invocations II A B User Kernel Composite: component invocation via migrating thread model a thread, τ executing in component A invokes B via the kernel and continues executing in B τ is charged for execution in A and B IPC does not require scheduling decision by design Parmer, West, BU CS Scheduling in Composite 8/33

9 Interrupt Execution Management Scheduling of interrupts interrupts promoted to threads run interrupt thread, or currently executing thread? scheduling decision needed interrupt thread finishes execution another scheduling decision needed Possibility of significant overhead with user-level scheduling lessen effective utilization increase interrupt response time Parmer, West, BU CS Scheduling in Composite 9/33

10 Interrupt Management/Scheduling in Composite Composite s mechanism for asynchronous upwards execution brands a path of components to guide the asynchronous execution a priority/urgency with which to schedule a corresponding upcall upcalls thread, associated with a brand conducts the asynchronous execution along path recorded in brand Interrupts are branded, which associates them with a brand, and attempts to execute the brand s upcall Parmer, West, BU CS Scheduling in Composite 10/33

11 Avoiding Scheduler Invocations Scheduling decision required when interrupt is branded shared data-structure between schedulers and kernel scheduler posts urgency/importance of threads with kernel kernel publishes CPU usage information to scheduler (cycles) when upcall attempted for a brand kernel compares urgency/importance of current thread with brand switch to upcall thread if higher urgency/importance Parmer, West, BU CS Scheduling in Composite 11/33

12 Avoiding Scheduler Invocations II When upcall completes execution common case: immediately switch back to preempted thread rare case: possibly when a scheduling decision involving upcall has been made Parmer, West, BU CS Scheduling in Composite 12/33

13 User-Level Scheduler Synchronization Schedulers must synchronize around critical sections mechanism must be efficient, predictable, policy neutral Kernel-provided semaphores, mutexes, locks??? but these primitives rely on a scheduler 1 τ 1 acquires lock to shared run-queues 2 τ 1 is preempted 3 τ 2 attempts to take lock: contention 4 kernel must know which thread to switch to 5 invoke scheduler that synchronizes around run-queues 6 livelock Allow schedulers to disable/enable interrupts??? not with untrusted/malicious/errant schedulers Parmer, West, BU CS Scheduling in Composite 13/33

14 User-Level Scheduler Synchronization II Uncontested critical section access user-level solution using lock-free synchronization and restartable atomic sections sections of assembly: if preempted in section, instruction pointer reset to start of section can model atomic instructions Contention for critical section library and kernel supplied wait-free synchronization higher-priority thread, H, helps lower priority thread through critical section, which then switches back to H Parmer, West, BU CS Scheduling in Composite 14/33

15 Microbenchmarks Thread migration efficient? Thread switching support efficient? Brands/upcall costs vs. scheduler invocations for interrupts Operation Cycles 1 Hardware RPC Costs (U/K transitions, page tbls) 1110 Linux Pipe RPC Composite Component Invocation 1620 Linux Thread Switch 1903 Composite Base Thread Switch 529 Composite Thread Switch w/ FPRR 976 Composite Brand w/ Upcall Execution 3442 Composite Brand w/ Scheduler Invocations Ghz Pentium IV, Linux Parmer, West, BU CS Scheduling in Composite 15/33

16 Case Study: Predictable Interrupt Scheduling Goal: predictable interrupt scheduling Predictable task execution control interrupt interference with tasks tasks have dependencies on interrupts! a NIC interrupt executes on behalf of task that will receive that packet Intelligent scheduling of interrupts demultiplexing component inspects packet contents brands specific to the contents of the packet interrupts for different tasks are scheduled differently Parmer, West, BU CS Scheduling in Composite 16/33

17 Linux-Style Interrupt Scheduling All interrupts executed at highest priority HW Level 0 Ints FP_RR Task 1 & 2 Interrupts Task 1 Task 2 Task 3 High Priority Low Priority italic nodes are schedulers dotted lines represent dependencies Parmer, West, BU CS Scheduling in Composite 17/33

18 Linux-Style Interrupt Scheduling II Task 1 & 2 interrupt (prio 0) Task 1 processing (prio 1) Task 2 processing (prio 2) Packets Processed Packets/Sec per Stream Sent Parmer, West, BU CS Scheduling in Composite 18/33

19 Priority Differentiation for Tasks and Interrupts I Interrupts are executed at the priority above their associated task, but below other tasks of higher priority HW Level 0 Ints FP_RR Task 1 Interrupts Task 1 Task 2 Interrupts Task 2 Task 3 High Priority Low Priority Parmer, West, BU CS Scheduling in Composite 19/33

20 Priority Differentiation for Tasks and Interrupts II Packets Processed Task 1 interrupt (prio 0) Task 1 processing (prio 1) Task 2 interrupt (prio 2) Task 2 processing (prio 3) Packets/Sec per Stream Sent Parmer, West, BU CS Scheduling in Composite 20/33

21 Threaded Interrupts with Deferrable Server Interrupts are executed at a higher priority than tasks, but in an aperiodic (deferrable) server HW Level 0 Ints FP_RR DS (7/20) Task 1 Task 2 Task 3 Task 1 & 2 Interrupts High Priority Low Priority Parmer, West, BU CS Scheduling in Composite 21/33

22 Threaded Interrupts with Deferrable Server II Task 1 & 2 interrupt (prio 0, ds 7/20) Task 1 processing (prio 1) Task 2 processing (prio 2) Packets Processed Packets/Sec per Stream Sent Parmer, West, BU CS Scheduling in Composite 22/33

23 Priority Differentiation and Deferrable Servers Interrupts are executed at the priority above their associated task, but below other tasks of higher priority, and are executed in deferrable servers with allocations commensurate with desired task progress HW Level 0 Ints FP_RR DS (5/20) Task 1 DS (2/20) Task 2 Task 3 Task 1 Interrupts Task 2 Interrupts High Priority Low Priority Parmer, West, BU CS Scheduling in Composite 23/33

24 Priority Differentiation and Deferrable Servers II Packets Processed Task 1 interrupt (prio 0, ds 5/20) Task 1 processing (prio 1) Task 2 interrupt (prio 2, ds 2/20) Task 2 processing (prio 3) Packets/Sec per Stream Sent Parmer, West, BU CS Scheduling in Composite 24/33

25 Overall Comparison Cumulative Packets Processed (x10e9) Task 1 (higher prio.) Task 2 (lower prio.) Total Highest Priority Interrupts Interrupt Thread Per-Task Prioritized Interrupts Differentiated Service Parmer, West, BU CS Scheduling in Composite 25/33

26 Related Work L4, Scheduler Activations (inc. K42), middleware-scheduling, CPU inheritance scheduling,... No previous work including all of the following 1 define complete scheduling behavior of all execution in system 2 schedulers don t have to be trusted 3 efficient even in presence of frequent interrupts Parmer, West, BU CS Scheduling in Composite 26/33

27 Conclusions Component-based schedulers enable application-specific system behavior high confidence system/fault tolerance User-level component-defined scheduling policies can precisely control the system s temporal behavior can maintain accurate accounting information even for interrupt execution Parmer, West, BU CS Scheduling in Composite 27/33

28 Questions? Parmer, West, BU CS Scheduling in Composite 28/33

29 Priority Differentiation and Deferrable Servers III Task 1 & 2 interrupt (prio 0, ds 7/20) Task 1 processing (prio 1) Task 2 processing (prio 2) Packets Processed (d) Packets/Sec in Stream 2 Sent, Stream 1 Constant at Parmer, West, BU CS Scheduling in Composite 29/33

30 Priority Differentiation and Deferrable Servers IV Packets Processed Task 1 interrupt (prio 0, ds 5/20) Task 1 processing (prio 1) Task 2 interrupt (prio 2, ds 2/20) Task 2 processing (prio 3) (e) Packets/Sec in Stream 2 Sent, Stream 1 Constant at Parmer, West, BU CS Scheduling in Composite 30/33

31 Composite Thread Migration Thread 0 Task 1 (T1) Network (N) Task 2 (T2) Deferrable Server (DS) Thread 0 s Invocation Stack T1 N Demultiplexer (DM) Fixed Priority, Round Robin (FPRR) FPRR Network Device Timer Parmer, West, BU CS Scheduling in Composite 31/33

32 Brand Creation 1 Thread 0 calls scheduler (FPRR) to create a brand to be executed from DM Thread 0 Task 1 (T1) Task 2 (T2) Deferrable Server (DS) Thread 0 s Invocation Stack T1 Network (N) N Demultiplexer (DM) Fixed Priority, Round Robin (FPRR) DM FPRR Network Device Timer Parmer, West, BU CS Scheduling in Composite 32/33

33 Brand Creation 1 Thread 0 calls scheduler (FPRR) to create a brand to be executed from DM 2 cos brand cntl(brand CREATE, DM) in FPRR Thread 0 Task 1 (T1) Task 2 (T2) Network (N) Demultiplexer (DM) Deferrable Server (DS) Fixed Priority, Round Robin (FPRR) Thread 0 s Invocation Stack T1 N DM FPRR Brand s Invocation Trace T1 N DM Network Device Timer Parmer, West, BU CS Scheduling in Composite 32/33

34 Upcall Execution 1 Thread 1 in the Demultiplexer wishes to cause an asynchronous upcall Thread 1 in DM Task 1 (T1) Task 2 (T2) Network (N) Demultiplexer (DM) Deferrable Server (DS) Fixed Priority, Round Robin (FPRR) Thread s 1 Invocation Stack DM Brand s Invocation Trace T1 N DM Network Device Timer Parmer, West, BU CS Scheduling in Composite 33/33

35 Upcall Execution 1 Thread 1 in the Demultiplexer wishes to cause an asynchronous upcall 2 Thread 1 executes cos brand upcall(brand, arg1, arg2) Thread 1 in DM Task 1 (T1) Network (N) Task 2 (T2) Deferrable Server (DS) Thread s 1 Invocation Stack DM Brand s Invocation Trace T1 N DM Demultiplexer (DM) Network Device Fixed Priority, Round Robin (FPRR) Timer Upcall s Invocation Stack N Parmer, West, BU CS Scheduling in Composite 33/33

Predictable Interrupt Management and Scheduling in the Composite Component-based System

Predictable Interrupt Management and Scheduling in the Composite Component-based System Gabriel Parmer and Richard West Computer Science Department oston University oston, MA 2215 {gabep1,richwest}@cs.bu.edu