A Synchronous IPC Protocol for Predictable Access to Shared Resources in Mixed-Criticality Systems

Transcription

1 A Synchronous IPC Protocol for Predictable Access to Shared Resources in Mixed-Criticality Systems RTSS 14 December 4, 2014 Björn B. Linux Testbed for Multiprocessor Scheduling in Real-Time Systems

2 How To Synchronize Access to Shared Resources? (such as data structures, OS services, I/O ports, ) Part 1 Locking is the wrong approach in mixed-criticality (MC) systems. Part 2 MC-IPC: Predictable IPC for Cross-Criticality Resource Sharing Part 3 A Case Study: Freedom-from-Interference Despite Failures, DoS, 2

3 Synchronization vs. Isolation in Mixed-Criticality Systems 3

4 Isolation: The Core MC Requirement Freedom from Interference High-criticality tasks not negatively affected by other (low-criticality) tasks. 4

5 Isolation: The Core MC Requirement Freedom from Interference High-criticality tasks not negatively affected by other (low-criticality) tasks. analyze & trust isolate & enforce 5

6 Isolation: The Core MC Requirement Freedom from Interference High-criticality tasks not negatively affected by other (low-criticality) tasks. analyze & trust Design Time Proof Obligation Show that a (lower-criticality) task does not cause interference. isolate & enforce Mandate Strict Isolation Ensures that a (lower-criticality) task cannot cause interference. At runtime Trust that other tasks do not cause unpredictable interference. At runtime Unpredictable interference impossible due to isolation. 6

7 Isolation: The Core MC Requirement The Problem with Analyze & Trust Freedom from Interference Need to establish absence of interference at the level of assurance High-criticality tasks not negatively affected by other (low-criticality) tasks. of the highest-criticality task that could be interfered with. we re back to applying the highest standards to all tasks analyze & trust Design Time Proof Obligation Show that a (lower-criticality) task does not cause interference. isolate & enforce Mandate Strict Isolation Ensures that a (lower-criticality) task cannot cause interference. At runtime Trust that other tasks do not cause unpredictable interference. At runtime Unpredictable interference impossible due to isolation. 7

8 Isolation: The Core MC Requirement Benefit of Isolation Freedom from Interference Need to apply highest standards High-criticality tasks not negatively affected by other (low-criticality) tasks. only when validating the isolation mechanism itself. But NOT to each isolated (lower-criticality) task. analyze & trust Design Time Proof Obligation Show that a (lower-criticality) task does not cause interference. isolate & enforce Mandate Strict Isolation Ensures that a (lower-criticality) task cannot cause interference. At runtime Trust that other tasks do not cause unpredictable interference. At runtime Unpredictable interference impossible due to isolation. 8

9 Isolation: The Core MC Requirement Freedom from Interference High-criticality tasks not negatively affected by other (low-criticality) tasks. logical isolation + temporal isolation 9

10 Locking Implies Trust Real-time locking protocols: PIP, PCP, MPCP, FMLP, OMLP Applicable across multiple criticalities? task (1) lock (3) unlock (2) read & write shared memory Lowcriticality Highcriticality task 10

11 Locking Implies Trust Real-time locking Could protocols: access without PIP, PCP, locking MPCP, first. FMLP, OMLP Applicable across multiple criticalities? Could fail to unlock (or hold lock for arbitrary duration). Could leave resource in (1) lock (3) unlock inconsistent state. (also with wait-free/lock-free) Lowcriticality Highcriticality task (2) read & write shared memory task 11

12 Better: Message Passing / IPC Isolate resource in resource server process to mediate access Use inter-process communication (IPC) to request service Canonical approach in µ-kernels task (1) IPC request (2) check & process request (3) IPC reply IPC Resource Server private memory Lowcriticality Highcriticality task 12

13 Untrusted client cannot occupy resource indefinitely. Better: Message-Passing / IPC Isolate resource in resource server process to mediate access Untrusted client cannot Untrusted client cannot leave Use inter-process communication (IPC) to request service bypass synchronization. resource in inconsistent state. Canonical approach in µ-kernels task (1) IPC request (2) check & process request (3) IPC reply IPC Resource Server private memory Lowcriticality Highcriticality task 13

14 Logical isolation: Must still validate and trust the server Better: Message-Passing / IPC at highest-criticality level of assurance Isolate resource in resource server process to mediate access Use inter-process communication (IPC) to request service Canonical but not approach all clients! in µ-kernels task (1) IPC request (2) check & process request (3) IPC reply IPC Resource Server private memory Lowcriticality Highcriticality task 14

15 Temporal Isolation: What is the maximum IPC ing delay? Better: Message-Passing / IPC and what happens if clients misbehave!? Isolate resource in resource server process to mediate access Use inter-process communication (IPC) to request service Canonical approach in µ-kernels task (1) IPC request (2) check & process request (3) IPC reply IPC Resource Server private memory Lowcriticality Highcriticality task 15

16 MC-IPC An IPC Protocol for Mixed-Criticality Systems 16

17 System Topology: Clustered Scheduling J 2 J 4 J 1 J 3 Q1 Q2 m = 4 Core 1 Core 2 Core 3 Core 4 K = 2 m1 = 2 L2 Cache L2 Cache m2 = 2 Main Memory Shared-Memory multiprocessor platform Organized into K clusters C1, C2,, CK mi processors in cluster Ci non-uniform clusters permitted 17

18 Partitioned Scheduling: mi = 1 in each cluster (= partition) System Topology: Clustered Scheduling Special case most common in practice (and used in evaluation). J 2 J 4 J 1 J 3 Q1 L2 Cache Q2 Core 1 Core 2 Core 3 Core 4 L2 Cache m = 4 K = 2 m1 = 2 m2 = 2 Main Memory Shared-Memory multiprocessor platform Organized into K clusters C1, C2,, CK mi processors in cluster Ci non-uniform clusters permitted 18

19 Temporal Isolation: Reservation-Based Scheduling per-cluster top-level scheduler R1 R2 T1 T2 T3 T? T? Reservation-Based Scheduling R1, R2, top-level scheduler chooses reservation reservation chooses task Each reservation has a current current budget one or more client tasks Specific type of reservation intentionally left undefined. Compatible examples: polling reservation table-driven reservation constant bandwidth server (CBS) 19

20 Reservation Assumptions per-cluster top-level scheduler R1 R2 T1 T2 T3 T? T? Assumptions A1: the of a reservation changes only when its budget is exhausted or replenished. A2: a reservation selected for service by the top-level scheduler consumes budget even if all clients are blocked on IPC (idling rule). 20

21 Resource Servers cluster Ca top-level scheduler cluster Cb top-level scheduler Rx Ry T? Ti T? T? T? Sq Tj low-criticality high-criticality synchronous IPC synchronous IPC Shared resource servers each shared resource q encapsulated in resource server Sq Logical Isolation OS: isolate server in private protection domain (private address space) server implementation: reject requests that are illegal / illformed / non-sensical Temporal Isolation server implementation: bounded maximum operation length IPC protocol must ensure bounded IPC delay (= bandwidth consumption) 21

22 What Could Violate Temporal Isolation? P1: The resource server may be preempted indefinitely Need to ensure timely IPC request completion P2: Clients may attempt to monopolize server Need to prevent starvation ( FIFO?) P3: There may be an unpredictable number of contending clients Need to respect of requesting clients (?) P4: A client may run out of budget while waiting for server Need to prevent backlog of stale clients P5: Best-effort background tasks may need to access server Some system services inherently shared (e.g., network stack) 22

23 What Could Violate Temporal Isolation? P1: The resource server may be preempted indefinitely Need to ensure timely IPC request completion No existing design is resilient to all of these issues. P2: Clients may attempt to monopolize server Need to prevent starvation ( FIFO?) P3: There may be an unpredictable number of contending clients Need to respect of requesting clients (?) P4: A client may run out of budget while waiting for server Need to prevent backlog of stale clients P5: Best-effort background tasks may need to access server Some system services inherently shared (e.g., network stack) 23

24 What Could Violate Temporal Isolation? P1: The resource server may be preempted indefinitely Need to ensure timely IPC request completion No existing design is resilient to all of these issues. P2: Clients may attempt to monopolize server Need to prevent starvation ( FIFO?) P3: There may be an unpredictable number of contending clients Need to respect of requesting clients (?) MC-IPC: we can tolerate all causes by combining P4: A client may run out of budget while waiting to contact server prior techniques and simple commonsense solutions Need to prevent backlog of stale clients in just the right way P5: Best-effort background tasks may need to access server Some system services inherently shared (e.g., network stack) 24

25 P1: Ensuring Server Progress Server preempted while replying to IPC = Lock-holder preempted while executing critical section. apply well-known techniques for ensuring lock-holder progress 25

26 P1: Ensuring Server Progress Server preempted while replying to IPC = Lock-holder preempted while executing critical section. Ensuring timely IPC completion: Bandwidth Inheritance let server execute on budget of any client (implicit in idling rule). When preempted, migrate server to processor of waiting client. Well-known solution Helping / timeslice donation in L4/Fiasco Hohmuth & Härtig, Multiprocessor Bandwidth Inheritance (MBWI) Faggioli et al., 2010 & M. Hohmuth and H. Hä rtig, Pragmatic nonblocking synchronization for real-time systems, in Proc. USENIX ATC 01, D. Faggioli, G. Lipari, and T. Cucinotta, Analysis and implementation of the multiprocessor bandwidth inheritance protocol, Real-Time Systems, vol. 48, no. 6, pp ,

27 P2 & P3: Dealing With Unknown Contention An unknown number of tasks may issue requests at arbitrary rates. 27

28 P2 & P3: Dealing With Unknown Contention An unknown number of tasks may issue requests at arbitrary rates. Need to respect priorities within each cluster, but also need to ensure fairness among clusters. can apply OMIP three-level structure [ECRTS 13] OMIP:, A Fully Preemptive Multiprocessor Semaphore Protocol for Latency-Sensitive Real-Time Applications, In Proc. ECRTS 13, July

29 P2 & P3: Dealing With Unknown Contention An unknown number of tasks may issue requests at arbitrary rates. Cluster 1 Cluster 2 Cluster K 29

30 Clients ordered by current reservation. P2 & P3: Dealing With Unknown Contention Priority-ordered w.r.t. top-level scheduler, An unknown number of tasks may issue requests at arbitrary rates. not reservation-internal priorities (if any): important for proof. Cluster 1 Cluster 2 Cluster K 30

31 P2 & P3: Dealing With Unknown Contention An unknown number of tasks may issue requests at arbitrary rates. Cluster 1 head (FIFO) Cluster 2 head (FIFO) Cluster K head (FIFO) 31

32 P2 & P3: Dealing With Unknown Contention Bounded length: at most mi - 1 jobs in each head. An unknown number of tasks may issue requests at arbitrary rates. (mi = number of cores in cluster) Cluster 1 head (FIFO) Cluster 2 head (FIFO) Cluster K head (FIFO) 32

33 P2 & P3: Dealing With Unknown Contention An unknown number of tasks may issue requests at arbitrary rates. Cluster 1 head (FIFO) Cluster 2 local head in each cluster head (FIFO) next to be served Cluster K head (FIFO) 33

34 P2 & P3: Dealing With Unknown Contention An unknown number of tasks may issue requests at arbitrary rates. Cluster 1 head (FIFO) global (FIFO) Cluster 2 head (FIFO) Cluster K head (FIFO) 34

35 P2 & P3: Dealing With Unknown Contention Bounded length: at most K - 1 jobs in global. An unknown number of tasks may issue requests at arbitrary rates. (K = number of clusters in the system) Cluster 1 head (FIFO) global (FIFO) Cluster 2 head (FIFO) Cluster K head (FIFO) 35

36 P2 & P3: Dealing With Unknown Contention An unknown number of tasks may issue requests at arbitrary rates. Cluster 1 client currently being served waiting for IPC reply head (FIFO) Cluster 2 Sq global (FIFO) server head (FIFO) Cluster K head (FIFO) 36

37 P4: Dealing With Budget Exhaustion Idling & bandwidth inheritance: a waiting client can run out of budget. 37

38 P4: Dealing With Budget Exhaustion Idling & bandwidth inheritance: a waiting client can run out of budget. With predictable IPC delay, a correct (high-criticality) task will have been provisioned to never exceed its budget. can prune clients from when budget exhausted & let them reissue requests after budget has been replenished 38

39 P4: Dealing With Budget Exhaustion Idling & bandwidth inheritance: a waiting client can run out of budget. Cluster 1 client currently being served waiting for IPC reply head (FIFO) Cluster 2 Sq global (FIFO) server head (FIFO) Cluster K head (FIFO) 39

40 P4: Dealing With Budget Exhaustion Easy case: client request not yet being served simply de. Idling & bandwidth inheritance: a waiting client can run out of budget. Cluster 1 client currently being served waiting for IPC reply head (FIFO) Cluster 2 Sq global (FIFO) server head (FIFO) Cluster K head (FIFO) 40

41 Tricky case: client request is being served. P4: propagate Dealing next task With from Budget head to Exhaustion local head, but do not add to global yet! (important for proof) Idling & bandwidth inheritance: a waiting client can run out of budget. Cluster 1 client currently being served waiting for IPC reply head (FIFO) Cluster 2 Sq global (FIFO) server head (FIFO) Cluster K head (FIFO) 41

42 P4: Dealing With Budget Exhaustion Idling & bandwidth inheritance: a waiting client can run out of budget. Cluster 1 client currently being served waiting for IPC reply head (FIFO) Cluster 2 Sq global (FIFO) server head (FIFO) Remove local stop flag only when IPC request is completed. Cluster K Intuition: create back-pressure without affecting other clusters. head (FIFO) 42

43 Added benefit: this mechanism allows us P4: Dealing With Budget Exhaustion to deal with budget-less best-effort tasks. Idling & bandwidth inheritance: a waiting client can run out of budget. (see paper for des) Cluster 1 client currently being served waiting for IPC reply head (FIFO) Cluster 2 Sq global (FIFO) server head (FIFO) Remove local stop flag only when IPC request is completed. Cluster K Intuition: create back-pressure without affecting other clusters. head (FIFO) 43

44 MC-IPC: Bandwidth Isolation Guarantee MC-IPC = MBWI + OMIP Queue + Pruning + Best-Effort Tasks 44

45 MC-IPC: Bandwidth Isolation Guarantee MC-IPC = MBWI + OMIP Queue + Pruning + Best-Effort Tasks If a client Ti does not exhaust its reservation s budget during a synchronous IPC invocation (= if not pruned), then Ti s IPC request to a server Sq is delayed by at most 2 mk K other requests ( delayed = forced to expend budget). mk number of cores in local cluster K number of clusters 45

46 MC-IPC: Bandwidth Strict Temporal Isolation Guarantee The per-request IPC delay bound is independent of any task parameters. MC-IPC = MBWI + OMIP Queue + Pruning + Best-Effort Tasks No trust implied w.r.t. the number of tasks, request frequencies, budgets of other tasks, which resources any other task accesses, etc. If a client Ti does not exhaust its reservation s budget during a synchronous IPC invocation (= if not pruned), then Ti s IPC request to a server Sq is delayed by at most 2 mk K other requests ( delayed = forced to expend budget). mk number of cores in local cluster K number of clusters 46

47 Evaluation: A Case Study PRIO-IPC vs. FIFO-IPC vs. MC-IPC 47

48 Case Study: Access To Signing Service measurements of a real implementation on a real multicore system in a plausible scenario 48

49 Case Study: Access To Signing Service sensor read sample Data collection with high integrity requirements multiple data sources (sensors, ) forward collected samples for further processing over untrusted, potentially compromised network data acquisition task send sample network 49

50 Case Study: Access To Signing Service sensor read sample Data collection with high integrity requirements multiple data sources (sensors, ) forward collected samples for further processing over untrusted, potentially compromised network data acquisition task send sample + signature To ensure integrity and to prevent playback attacks timestamp each sample assign a sequence number add cryptographic signature network 50

51 Case Study: Access To Signing Service sensor Data collection with high integrity requirements multiple data sources (sensors, ) read sample forward collected samples for further processing over untrusted, potentially compromised network data acquisition task IPC request: sample IPC reply: signature send sample + signature signature server To ensure integrity and to prevent playback attacks timestamp each sample assign a sequence number add cryptographic signature To prevent leakage of cryptographic key material key material accessible only to signature server network 51

52 The signature server is a shared resource. Case Study: Access To Signing Service If there are multiple data acquisition tasks, what is the maximum IPC delay? data acquisition task sensor read sample IPC request: sample IPC reply: signature send sample + signature signature server Data collection with high integrity requirements multiple data sources (sensors, ) forward collected samples for further processing over untrusted, potentially compromised network To ensure integrity and to prevent playback attacks timestamp each sample assign a sequence number add cryptographic signature To prevent leakage of cryptographic key material key material accessible only to signature server network 52

53 Case Study: Implementation Reservation-Based Scheduler implemented in LITMUS RT table-driven, sporadic, polling reservations with EDF and FP scheduling Linux Testbed for Multiprocessor Scheduling in Real-Time Systems IPC System Calls added to LITMUS RT Signature Server RSA w/ 2048bit keys max. request length: 2ms Platform 4 Xeon E (2.4 Ghz) cores 53

54 Case Study: Setup on a 4-Core System 4 high-criticality tasks provisioned with table-driven static reservations 10 low-criticality tasks provisioned with simple EDF-scheduled polling reservations 0ms 50ms 100ms core C 1 R H 1 idle core C 2 R H 2 R H 3 static schedule length = 100 ms core C 3 R H 4 idle core C 4 idle idle 54

55 Low-criticality polling reservations assigned to cores C3 and C4; Case executed Study: only Setup when table-driven on a reservation 4-Core is idle. System 4 high-criticality tasks provisioned with table-driven static reservations 10 low-criticality tasks provisioned with simple EDF-scheduled polling reservations 0ms 50ms 100ms core C 1 R H 1 idle core C 2 R H 2 R H 3 static schedule length = 100 ms core C 3 R H 4 idle core C 4 idle idle 55

56 Case Study: Setup on a 4-Core System 4 high-criticality tasks One high-criticality task each provisioned with table-driven static reservations with period = deadline =100ms 10 low-criticality tasks provisioned with simple EDF-scheduled polling reservations 0ms 50ms 100ms core C 1 R H 1 idle core C 2 R H 2 R H 3 static schedule length = 100 ms core C 3 R H 4 idle core C 4 idle idle 56

57 10 low-criticality tasks Case Study: Setup on a 4-Core System one per EDF-based low-criticality reservation 4 high-criticality five per core, tasks with budget = 20ms and periods provisioned = deadline with table-driven {100, 120, 250, static 500, reservations 1000}ms 10 low-criticality tasks provisioned with simple EDF-scheduled polling reservations 0ms 50ms 100ms core C 1 R H 1 idle core C 2 R H 2 R H 3 static schedule length = 100 ms core C 3 R H 4 R L 5,...,R L 9 core C 4 R L 10,...,R L 14 57

58 Case Study: Experiment 8 phases of execution, 60 seconds each = 480 seconds 1 phase = 60 seconds of normal execution 7 phases = 420 seconds of different failure modes measured: IPC delay experienced by each task 58

59 Case Study: Experiment 8 phases of execution, 60 seconds each = 480 seconds 1 phase = 60 seconds of normal execution 7 phases = 420 seconds of different failure modes measured: IPC delay experienced by each task Three IPC protocols MC-IPC (this paper) FIFO-IPC serve requests strictly in FIFO order like Multiprocessor Bandwidth Inheritance Protocol (MBWI) PRIO-IPC order requests by of reservation like most microkernels (e.g., L4/Fiasco) 10 runs each MBWI: D. Faggioli, G. Lipari, and T. Cucinotta, Analysis and implementation of the multiprocessor bandwidth inheritance protocol, Real-Time Systems, vol. 48, no. 6, pp ,

60 Case Study: Experiment 8 phases of execution, 60 seconds each = 480 seconds 1 phase = 60 seconds of normal execution 7 phases = 420 seconds of different failure modes measured: IPC delay experienced by each task 0ms 50ms 100ms core C 1 R H 1 idle Vantage Point core C 2 R H 2 R H 3 All data reported for core C 3 R H 4 R L 5,...,R L 9 high-criticality task on core 1. core C 4 R L 10,...,R L 14 60

61 Case Study: Results Overview MC-IPC (predicted bound: 18ms) FIFO-IPC (predicted bound: 28ms) PRIO-IPC (predicted bound: 8ms) 8 phases of execution, 60 seconds each = 480 seconds measured & shown: IPC delay experienced by jobs of task on core 1 Comparison: measured IPC delay vs. predicted maximum All three IPC protocols permit a priori IPC delay bounds but they react differently to task failures / unexpected behavior. 61

62 thick black line = predicted maximum delay Case Study: Results Overview Based on RT analysis & specified task set parameters. MC-IPC (predicted bound: 18ms) FIFO-IPC (predicted bound: 28ms) PRIO-IPC (predicted bound: 8ms) 8 phases of execution, 60 seconds each = 480 seconds measured & shown: IPC delay experienced by jobs of task on core 1 Comparison: measured IPC delay vs. predicted maximum All three IPC protocols permit a priori IPC delay bounds but they react differently to task failures / unexpected behavior. 62

63 Case Study: Results Overview MC-IPC (predicted bound: 18ms) FIFO-IPC (predicted bound: 28ms) PRIO-IPC (predicted bound: 8ms) Red samples = measured IPC delay One sample per job of task on core 1. 8 phases of execution, 60 seconds each = 480 seconds measured & shown: IPC delay experienced by jobs of task on core 1 Comparison: measured IPC delay vs. predicted maximum All three IPC protocols permit a priori IPC delay bounds but they react differently to task failures / unexpected behavior. 63

64 measured IPC delay > predicted maximum delay Case Study: Results Overview unpredictable IPC delays in case of failures! MC-IPC (predicted bound: 18ms) FIFO-IPC (predicted bound: 28ms) PRIO-IPC (predicted bound: 8ms) 8 phases of execution, 60 seconds each = 480 seconds measured & shown: IPC delay experienced by jobs of task on core 1 Comparison: measured IPC delay vs. predicted maximum All three IPC protocols permit a priori IPC delay bounds but they react differently to task failures / unexpected behavior. 64

65 Case Study: Results Overview MC-IPC (predicted bound: 18ms) FIFO-IPC (predicted bound: 28ms) PRIO-IPC (predicted bound: 8ms) 8 phases of execution, 60 seconds each = 480 seconds measured & shown: IPC delay experienced by jobs of task on core 1 Comparison: measured IPC delay vs. predicted maximum In this talk: 3 select phases. All three IPC protocols permit a priori IPC delay bounds but they react differently to task failures / unexpected behavior. (See paper for discussion of all 8 phases.) 65

66 Phase 1: Normal Operation MC-IPC (predicted bound: 18ms) FIFO-IPC (predicted bound: 28ms) PRIO-IPC (predicted bound: 8ms) Reality check: all tasks behave exactly as designed / as assumed. Tasks Operate Normally One request to signature server per data acquisition job Expected number of tasks All IPC delays (well) below upper bounds lowest bound under PRIO-IPC most pessimistic bound with FIFO-IPC 66

67 Phase 1: Normal Operation MC-IPC (predicted bound: 18ms) FIFO-IPC (predicted bound: 28ms) PRIO-IPC (predicted bound: 8ms) Tasks Operate Normally One request to signature server per data-acquisition job No unexpected tasks All IPC delays (well) below upper bounds lowest bound under PRIO-IPC (8ms) most pessimistic bound with FIFO-IPC (28ms) 67

68 Phase 3: Surge in Low-Criticality Tasks MC-IPC (predicted bound: 18ms) FIFO-IPC (predicted bound: 28ms) PRIO-IPC (predicted bound: 8ms) Deviation from original system model: additional low-criticality reservations with period = 3000ms are admitted on all cores. 0ms 50ms 100ms core C The system is flooded 1 R with additional 1 H idle low-criticality tasks Each task operates as expected but there core are unexpectedly many low-criticality tasks. C 2 R H 2 R H 3 16 additional lowcriticality tasks Schedulability of high-criticality tasks should not be affected table-driven core C reservations 3 R have 4 H statically R5 L,...,R higher L (on each 9 anyway. core.) FIFO-IPC problem: core C unexpected length 4 R10,...,R L 14 L With FIFO-IPC, need to place trust in the total number of tasks! 68

69 Phase 3: Surge in Low-Criticality Tasks MC-IPC (predicted bound: 18ms) FIFO-IPC (predicted bound: 28ms) PRIO-IPC (predicted bound: 8ms) The system is flooded with additional low-criticality tasks Each task operates as expected but there are unexpectedly many low-criticality tasks. Schedulability of high-criticality tasks should not be affected table-driven reservations have statically higher anyway. FIFO-IPC problem: unexpected length With FIFO-IPC, need to place trust in the total number of tasks! 69

70 measured IPC delay > predicted maximum delay Phase 3: Surge in Low-Criticality Tasks unpredictable IPC delays incurred by high-criticality MC-IPC (predicted bound: 18ms) FIFO-IPC (predicted bound: 28ms) PRIO-IPC (predicted bound: 8ms) The system is flooded with additional low-criticality tasks Each task operates as expected but there are unexpectedly many low-criticality tasks. Schedulability of high-criticality tasks should not be affected table-driven reservations have statically higher anyway. FIFO-IPC problem: unexpected length With FIFO-IPC, need to place trust in the total number of tasks! 70

71 MC-IPC: somewhat elevated IPC delays, Phase but below 3: predicted Surge bound! in Low-Criticality Tasks MC-IPC (predicted bound: 18ms) FIFO-IPC (predicted bound: 28ms) PRIO-IPC (predicted bound: 8ms) The system is flooded with additional low-criticality tasks Each task operates as expected but there are unexpectedly many low-criticality tasks. Schedulability of high-criticality tasks should not be affected table-driven reservations have statically higher anyway. FIFO-IPC problem: unexpected length With FIFO-IPC, need to place trust in the total number of tasks! 71

72 Phase 5: High-Criticality DoS MC-IPC (predicted bound: 18ms) FIFO-IPC (predicted bound: 28ms) PRIO-IPC (predicted bound: 8ms) Deviation from original system model: two high-criticality tasks malfunction, start invoking signature server as quickly as possible effectively a Denial-of-Service (DoS) attack. 0ms 50ms 100ms core C Two higher- 1 R tasks of equal 1 H idle criticality monopolize server PRIO-IPC permits starvation With Attempt PRIO-IPC, core to Cneed 2 to place R H trust in maximum request frequencies. 2 R3 H monopolize FIFO-IPC signature C predicted server. core bound and 3 actual R IPC 4 H delays independent R5 L,...,R9 L of request frequencies MC-IPC core C 4 R10,...,R L 14 L predicted bound accounts for arbitrary request frequencies 72

73 Phase 5: High-Criticality DoS MC-IPC (predicted bound: 18ms) FIFO-IPC (predicted bound: 28ms) PRIO-IPC (predicted bound: 8ms) Two higher- tasks of equal criticality monopolize server PRIO-IPC permits starvation With PRIO-IPC, need to place trust in maximum request frequencies. FIFO-IPC predicted bound and actual IPC delays independent of request frequencies MC-IPC predicted bound accounts for arbitrary request frequencies 73

74 measured IPC delay approaching 100ms (= period) Phase 5: High-Criticality DoS high-criticality task on core 1 starved completely. MC-IPC (predicted bound: 18ms) FIFO-IPC (predicted bound: 28ms) PRIO-IPC (predicted bound: 8ms) Two higher- tasks of equal criticality monopolize server PRIO-IPC permits starvation With PRIO-IPC, need to place trust in maximum request frequencies. FIFO-IPC predicted bound and actual IPC delays independent of request frequencies MC-IPC predicted bound accounts for arbitrary request frequencies 74

75 MC-IPC: provides freedom-from-interference even with respect to tasks Phase 5: High-Criticality DoS of equal or higher criticality (= fault containment). MC-IPC (predicted bound: 18ms) FIFO-IPC (predicted bound: 28ms) PRIO-IPC (predicted bound: 8ms) Two higher- tasks of equal criticality monopolize server PRIO-IPC permits starvation With PRIO-IPC, need to place trust in maximum request frequencies. FIFO-IPC predicted bound and actual IPC delays independent of request frequencies MC-IPC predicted bound accounts for arbitrary request frequencies 75

76 Only MC-IPC Ensures Predicted Worst Case MC-IPC (predicted bound: 18ms) FIFO-IPC (predicted bound: 28ms) PRIO-IPC (predicted bound: 8ms) FIFO-IPC and PRIO-IPC Vulnerable to deviations from analyzed system model. MC-IPC: does not require trusting the bound on the number of tasks (either high- or low-criticality). any bounds on maximum request frequencies. the behavior of best-effort or any real-time tasks. The trust rests solely with the server (must reject illegal requests). 76

77 Only MC-IPC Ensures Predicted Worst Case MC-IPC (predicted bound: 18ms) FIFO-IPC (predicted bound: 28ms) PRIO-IPC (predicted bound: 8ms) FIFO-IPC Analytical and PRIO-IPC Benefits: Integration with Vestal s Model Vulnerable to deviations from analyzed system model. Because the MC-IPC ensures strict isolation, it is simple to statically reclaim pessimism at lower criticalities MC-IPC: does not require trusting in the spirit of (and compatible with) Vestal s model. the bound on the number of tasks (either high- or low-criticality). Criticality-dependent any bounds on maximum IPC costs, request criticality-dependent frequencies. interference, the behavior of best-effort or any real-time tasks. The trust rests solely with See the paper server. for des. 77

78 Conclusion 78

79 Summary Locking and lock-free/wait-free synchronization are ill-suited for cross-criticality synchronization. Explicit synchronization implies trust and violates isolation. MC-IPC: Resources may be shared between high- and lowcriticality tasks without violating freedom-from-interference Other tasks do not need to be trusted, only the accessed server. Prototype and case study in LITMUS RT (Lack of) isolation easily observed in practical system 79

80 Limitations & Future Work Algorithmic challenge: nested IPC requests (server to server) Not supported by current analysis. Which to en in? Cluster of server? Cluster of client? Both? None? IPC Overheads Prototype in LITMUS RT not optimized, not comparable to highly tuned implementations like those found in the L4 family. Even if highly optimized, MC-IPC will likely still be more heavyweight due to additional operations. 80

81 MC-IPC: freedom-from-interference despite shared resources, despite untrusted tasks, even on multiprocessors. Linux Testbed for Multiprocessor Scheduling in Real-Time Systems Björn B.

82 Appendix 82

83 P5: Accommodating Best-Effort Tasks Sharing with best-effort tasks may be unavoidable for system resources. 83

84 P5: Accommodating Best-Effort Tasks Sharing with best-effort tasks may be unavoidable for system resources. A best-effort task without any guaranteed budget is no different than a real-time task that exhausted its budget just after it started being served. bandwidth inheritance takes care of such tasks 84

85 P5: Accommodating Best-Effort Tasks Sharing with best-effort tasks may be unavoidable for system resources. Cluster 1 client currently being served waiting for IPC reply head (FIFO) Cluster 2 Sq global (FIFO) server head (FIFO) Cluster K head (FIFO) 85

86 P5: Accommodating Best-Effort Tasks Sharing with best-effort tasks may be unavoidable for system resources. Cluster 1 client currently being served waiting for IPC reply head (FIFO) Cluster 2 Sq global (FIFO) server head (FIFO) best-effort served only if global RT is empty Cluster K head (FIFO) 86

87 A real-time task encounters at most one best-effort task. P5: Accommodating Best-Effort Tasks Analytically, no different than finding Sq stuck on Sharing with best-effort a real-time tasks task may with be an unavoidable exhausted for budget. system resources. Best-effort is global, can be ordered arbitrarily. Cluster 1 client currently being served waiting for IPC reply head (FIFO) Cluster 2 Sq global (FIFO) server head (FIFO) best-effort served only if global RT is empty Cluster K head (FIFO) 87

88 Phase 8: Flood of Best-Effort Tasks MC-IPC (predicted bound: 18ms) FIFO-IPC (predicted bound: 28ms) PRIO-IPC (predicted bound: 8ms) Deviation from original system model: additional low-criticality reservations with period = 3000ms are admitted on all cores. Best-effort tasks create contention for resource used by real-time tasks With FIFO-IPC, need to place trust in maximum number of background best-effort tasks. PRIO-IPC Best-effort tasks have lower no additional delays. MC-IPC Handles best-effort tasks explicitly analysis accounts for best-effort tasks. 88

89 Phase 8: Flood of Best-Effort Tasks MC-IPC (predicted bound: 18ms) FIFO-IPC (predicted bound: 28ms) PRIO-IPC (predicted bound: 8ms) Best-effort tasks create contention for resource used by real-time tasks With FIFO-IPC, need to place trust in maximum number of background best-effort tasks. PRIO-IPC Best-effort tasks have lower no additional delays. MC-IPC Handles best-effort tasks explicitly analysis accounts for best-effort tasks. 89

90 Phase 8: Flood of Best-Effort Tasks MC-IPC (predicted bound: 18ms) FIFO-IPC (predicted bound: 28ms) PRIO-IPC (predicted bound: 8ms) FIFO-IPC: cannot share resources with best-effort tasks (or must derive high-criticality assurance on maximum number of tasks). Best-effort tasks create contention for resource used by real-time tasks With FIFO-IPC, need to place trust in maximum number of background best-effort tasks. PRIO-IPC Best-effort tasks have lower no additional delays. MC-IPC Handles best-effort tasks explicitly analysis accounts for best-effort tasks. 90

91 MC-IPC: combines advantages of FIFO-IPC and PRIO-IPC. Phase 8: Flood of Best-Effort Tasks Like PRIO-IPC w.r.t. to lower-/lower-criticality tasks, like FIFO-IPC w.r.t. monopolization attempts and starvation effects. MC-IPC (predicted bound: 18ms) FIFO-IPC (predicted bound: 28ms) PRIO-IPC (predicted bound: 8ms) Best-effort tasks create contention for resource used by real-time tasks With FIFO-IPC, need to place trust in maximum number of background best-effort tasks. PRIO-IPC Best-effort tasks have lower no additional delays. MC-IPC Handles best-effort tasks explicitly analysis accounts for best-effort tasks. 91