Department of Computer Science Institute for System Architecture, Operating Systems Group REAL-TIME MICHAEL ROITZSCH OVERVIEW 2 SO FAR talked about in-kernel building blocks: threads memory IPC drivers enable access to a wide range of non-kernel resources need to manage resources 3
COURSE EIP Applications System Services Basic Abstractions 4 RESOURCES Disk Bandwidth TCP/IP Sessions Network I/O Semaphores Windows Files Memory Threads Communication Rights 5 TIME 6
COMPARISON Memory Time discrete, limited continuous, infinite hidden in the system user-perceivable managed by pager managed by scheduler page-granular partitions arbitrary granularity all pages are of equal value value depends on workload active policy decisions, passive enforcement hierarchical management active policy decisions, active enforcement Fiasco: flattened in-kernel view 7 HISTORY in the early years of computing: time coarsely managed by batch systems jobs receive the entire machine or a dedicated part of it for a given time accounting, budgeting no preemption no interaction, good utilization still prevalent in HPC systems 8 THREADS today: threads provide a CPU abstraction each thread should observe its own time as continuous if there are more threads than physical CPU cores, we have to multiplex enforced by preemption implemented with timer interrupt 9
PROGRESS thread progress wall-clock time 10 REAL-TIME 11 DEFINITION a real-time system denotes a system, whose correctness depends on the timely delivery of results it matters, when a result is produced real-time denotes a predictable relation between system progress and wall-clock time 12
INTUITION real-time is about predictability guarantees timeliness responsiveness real-time is not about being fast live calculations 13 game engines NON-EXAMPLE rendered frames are delivered at instances of wall-clock time, but there is no guaranteed (minimal) frame-rate rendering times are not even predictable guaranteed responsiveness? game engines are just fast (hopefully), but not real-time 14 EXAMPLES engine control in a car break-by-wire avionics railway control set-top box DVD player GSM-stack in your cell phone focused catastrophic failures benign failures complex 15
FLAVORS hard real-time firm real-time soft real-time missing some deadlines is tolerable a result delivered after its deadline is still useful 16 THEMES ➀ Predictability ➁ Guarantees ➂ Enforcement 17 PREDICTABILITY 18
ENEMIES gap between worst and average case memory caches, disk caches, TLBs smart hardware system management mode disk request reordering cross-talk from resource sharing servers showing O(n) behavior unpredictable external influences interrupts 19 CROSSCUTTING Applications Realtime System Services Kernel Hardware 20 CUSTOM RTOS small real-time executives tailor-made for specific applications fixed workload known a priori pre-calculated time-driven schedule used on small embedded controllers benign hardware 21
RTLINUX full Linux kernel and real-time processes run side-by-side small real-time executive underneath supports scheduling and IPC real-time processes implemented as kernel modules all of this runs in kernel mode no isolation 22 XNU the kernel used in Mac OS X implements four priority bands normal system high priority kernel threads real-time threads you tell the kernel: I need A out of the next B cycles, can be contiguous 23 XNU normal users can create real-time threads threads exceeding their quantum will be demoted with the first real-time thread, the kernel switches from using continuations to full preemptibility all drivers need to handle interrupts correctly 24
FIASCO static thread priorities bounded interrupt latency fully preemptible in kernel mode lock-free synchronization uses atomic operations wait-free synchronization locking with helping instead of blocking 25 SKEPTICISM architecture for those afraid of touching the OS example: Real-Time Java Applications Real-Time Middleware Non-Real-Time Kernel 26 RESOURCES a real-time kernel alone is not enough the microkernel solution: real-time kernel enables temporal isolation eliminates cross-talk and interrupt problems user-level servers on top act as resource managers implement real-time views on specific resources real-time is not only about CPU details in the resource management lecture 27
GUARANTEES 28 PROBLEM worst case execution time (WCET) largely exceeds average case offering guarantees for the worst case will waste lots of resources missing some deadlines can be tolerated with the firm and soft real-time scheme 29 MOTIVATION desktop real-time there are no hard real-time applications on desktops there is a lot of firm and soft real-time low-latency audio processing smooth video playback desktop effects user interface responsiveness 30
H.264 DECODING 15% 10% 5% 0% WCET 0 5 10 15 20 25 30 ms 31 KEY IDEA guarantees even slightly below 100% of WCET can dramatically reduce resource allocation unused reservations will be used by others at runtime use probabilistic planning to model the actual execution quality q: fraction of deadlines to be met 32 KEY IDEA WCET 33
l (which Figure can 4. Admission with 0 QAS vs. QRMS d i. onqrms of the disest complex- r In the admission, the reservation time is regarded as a emptible real number of 0 Consequently, tasks are independentd i during admission, an Figure constant 5. execution time. X i Y i Y i Y i RESERVATION Y i s of Y the ranhe scheduler Figure 5. overhead. Figure 4 illustrates the modified priority assign- i important advantage to drastically decrease the admission d priorities. So i J ment and the notion of reservation times for two tasks T 1, he type of remputation of To derive the reservation times, we consider preemptible T 2 with uniformr periods of length d. The approach uses the task model given in Definition 1. reasing costs resources first. We have toj use Equation (8). Due to the explodes for laws of probability calculus, we can compute the expected 03 and 510) value of A i as for a way to P(J does not run m longer i than r mission conree respects: J isea completed i = P(A until i its relative k), i = deadline) 1,..., n q (9) k=1 rvation time, 34 n algorithm. The number A i of completed parts of task T i within a ng llowed by an d overhead. ndon the exvor of applyic scheduling assignment vation times: s mandatory priority) acn reservation b) QRMS period does no longer depend on the reservation times of other tasks. Obviously, it holds (see Figure (5)): P(A i (r) k) = P(X i + k Y i r), k = 1,..., m i RESERVATION (10) Thus: r i = min(r R r i The = max(r final formula i, w i ) respects i = 1, the..., fact n that r i may be shorter than the WCET w i of the mandatory part and includes the constraint to fully understand that jobs are aborted this: at the end of their period: see real-time r i = systems max(r lecture i, w i ) i = 1,..., n (12) good We check for microkernel: r i d i for all reservation i in a first admission can be step. Then the calculated final admission by a userland test can be done service using the Liu/Laylandcriterion or time demand analysis [13]. In case of nonpreemptible kernel only resources, needs r to support static priorities i is computed as above, but the admission must include the WCET w O,i of an optional part: 1 m i m i k=1 P(X i +k Y i r) q i ). (11) r i = max(r i + w O,i, w i ) (13) 35 SCHEDULING scheduling = admission + dispatch admission evaluates the requests from clients, which follow some task model calculates task parameters for dispatcher verifies the feasibility of the guarantees can reject requests dispatch executes and enforces the schedule 36
ENFORCEMENT 37 DISPATCHER executes schedule enforces task parameters by preemption e.g. on deadline overrun picks the next thread static priorities (e.g. RMS, DMS) dynamic priorities (e.g. EDF) seems simple 38 PROBLEM high priority thread calls low priority service with a medium priority thread interfering: Priority Thread 1 Thread 2 Thread 3 1 waits for 3 3 waits for 2 = 1 waits for 2 Priority Inversion 39
SOLUTION priority inheritance, priority ceiling nice mechanism for this in Fiasco, NOVA: timeslice donation implemented by splitting thread control block execution context: holds CPU statue scheduling context: time and priority on IPC-caused thread switch, only the execution context is switched 40 DONATING CALL Priority Thread 1 Thread 2 Thread 3 IPC receiver runs on the sender s scheduling context priority inversion problem solved with priority inheritance 41 ACCOUNTING servers run on their clients time slice when the server executes on behalf of a client, the client pays with its own time this allows for servers with no scheduling context server has no time or priority on its own can only execute on client s time relieves dispatcher from dealing with servers 42
TIMELESS servers could be malicious, so you need timeouts to get your time back now, malicious clients can call the server with a very short timeout server s time will be withdrawn while the server is in the middle of request handling servers need to do session cleanup on whose time? open research problem 43 SMP timeslice donation does not work in multiprocessor case: server is running on a different CPU cross-cpu donation would be needed you cannot donate time between CPUs think of real-time guarantees if one CPU is 100% loaded, donating time to it will overload it servers need to migrate to client CPU? 44 OPTIMIZATION 45
SEMAPHORES Thread 1 down up Semaphore Thread Thread 2 down enqueue and block dequeue and unblock up IPC only in the contention case optimized for low contention bad for producer-consumer (high contention) 46 SCENARIO consumer request wakeup: set flag, enqueue block producer check for wakeup: test flag, dequeue send wakeup very short critical sections semaphores too expensive (2 IPCs) 47 IDEA allow threads to have short periods where they are never preempted like a very low cost global system lock delayed preemption threads can set don t preempt me flag in UTCB very low cost 48
PROBLEMS unbounded delay if preemption would occur, the kernel honors the delayed preemption flag only for a fixed maximum delay delay affects all threads problematic for real-time guarantees affected threads can be limited to those within a priority band other threads can still preempt any time 49 BEWARE system must be designed carefully threads outside the affected priority band must not interfere with protected data structures more problems unaffected threads might donate time to a thread within the priority band does not work cross-cpu 50 managing time is necessary SUMMARY we interact with the system based on time real-time is a cross-cutting concern hard real-time is hard, soft real-time is even harder priority inheritance by timeslice donation delayed preemption next week: names 51