Decoupling Cores, Kernels and Operating Systems

Transcription

1 Decoupling Cores, Kernels and Operating Systems Gerd Zellweger, Simon Gerber, Kornilios Kourtis, Timothy Roscoe Systems Group, ETH Zürich 10/6/2014 1

2 Outline Motivation Trends in hardware and software Booting and shutting down cores dynamically Decoupling the kernel state Evaluation Kernel updates, specialized kernels 10/6/2014 2

3 What s happening to hardware Constrained by power consumption Reconfigurable cores (dynamically changed behavior) DVFS, Turbo Boost, SMT Core Fusion [ISCA 07] Dark silicon [ISCA 10] Heterogeneous cores Fast and power hungry vs. slow and power efficient Asymmetric multiprocessing Conservation Cores [ASPLOS 10] 10/6/2014 3

4 What current operating systems look like Application Application Application Kernel 10/6/2014 4

5 What current operating systems look like Application Application Application Kernel 10/6/2014 4

6 What current operating systems look like Application Application Application Kernel 10/6/2014 4

7 What s happening to software OS needs to adapt to different workloads Adapting at build-, boot-, and run-time Debugging support: profiling, tracing etc. Real-time support On-the-fly kernel updates KSplice (Linux) [EuroSys 09] K42 [ATC 07] 10/6/2014 5

8 Multikernel [SOSP 09] Application Application Application Kernel Async. Messages Async. Messages Async. Messages Kernel Kernel Kernel 10/6/2014 6

9 Implementation Barrelfish OS Treating cores as pluggable devices Booting a core dynamically with boot drivers Shutting down a core Decoupling Cores, Kernels and the Operating System Externalizing kernel state 10/6/2014 7

10 Booting a core with boot drivers OS service for target core management Dynamically chooses kernel for core based on runtime information Boots any core with any suitable kernel Run any on any compatible core Implements boot, shutdown, reboot protocol 10/6/2014 8

11 Shutting down a core Harder than booting a core Need to deal with per-core state: Scheduler queues, memory pools, page-tables Takes time (and energy) However, we want to remove the core as fast as possible General approach (cf. Chameleon [ASPLOS 12]) Get state out of the way quickly Dismantle it later, lazily (if needed) 10/6/2014 9

12 Shutting down a core Application Application Application Kernel Kernel Kernel Kernel 10/6/

13 Shutting down a core Application Application Application Kernel Kernel Kernel 10/6/

14 Shutting down a core Application Application Application Kernel Kernel Kernel 10/6/

15 Shutting down a core Application Application Application Kernel Kernel Kernel Scheduler 10/6/

16 Shutting down a core Application Application Application Parking Kernel Kernel Kernel Scheduler 10/6/

17 Shutting down a core Application Application Application Parking Kernel Kernel Kernel Scheduler Highly scalable, only two cores involved 10/6/

18 What is the? : All state for a single core and kernel How do we capture this? Capabilities: Tracks all application state Tracks all OS state cf. sel4, EROS, KeyKOS KCB (Kernel control block) Hardware specific state Entry point to capability tree Represented as a capability itself Scheduler State Cap Derivation Tree Timer Offset IRQ State KCB... Frame Frame Frame Frame CNode PCB... Frame Frame Frame CNode CNode PCB... Null Frame Frame Frame 10/6/

19 Decoupling Cores, Kernels and Operating Systems State externalization & dynamic core booting is a much more general mechanism Kernel Kernel 10/6/

20 Decoupling Cores, Kernels and Operating Systems State externalization & dynamic core booting is a much more general mechanism Kernel Kernel 10/6/

21 Decoupling Cores, Kernels and Operating Systems State externalization & dynamic core booting is a much more general mechanism Kernel Kernel 10/6/

22 Decoupling Cores, Kernels and Operating Systems State externalization & dynamic core booting is a much more general mechanism Kernel 10/6/

23 Decoupling Cores, Kernels and Operating Systems State externalization & dynamic core booting is a much more general mechanism Kernel Kernel 10/6/

24 Evaluation Core management Adding and removing cores in the system Kernel updates Hot-swapping the kernel Specialized kernels e.g., eliminate OS jitter 10/6/

25 Core management (Haswell, 1x4 cores, no HT) Booting a core No Load Load Linux ms 20 ms Barrelfish/DC 7.5 ms 7.5 ms Removing a core No Load Load Linux ms 2542 ms Barrelfish/DC ms ms 10/6/

26 Use-case: Kernel Updates Shut-down target core Application Kernel 10/6/

27 Use-case: Kernel Updates Shut-down target core Reboot core with a new kernel image Application Kernel Kernel 10/6/

28 Use-case: Kernel Updates Shut-down target core Reboot core with a new kernel image Dispatch previous Application Application Kernel Kernel 10/6/

29 Kernel updates: PostgreSQL & TPC-H Hot-swapping the kernel 10/6/

30 Use-case: Temporary real time task A thread that needs to run with hard real time performance E.g., phone baseband stack, control application, robotics etc. A lot of effort spent to make this work in a general purpose OS Many real time OS for embedded systems (RTLinux, LynxOS, QNX, ) 10/6/

31 Use-case: Real time application OS OS OS OS OS OS OS OS 10/6/

32 Use-case: Real time application RT OS Kernel OS OS OS OS OS OS OS 10/6/

33 Use-case: Real time application RT OS Kernel OS OS OS OS 10/6/

34 Use-case: Specialized kernels Shut-down target core Application Kernel 10/6/

35 Use-case: Specialized kernels Shut-down target core Temporarily park the target Application Application Kernel 10/6/

36 Evaluation: PostgreSQL & TPC-H Parked (2 s per core) Move back to original core One per core 10/6/

37 Use-case: Specialized kernels Shut-down target core Temporarily park the target Boot simple real-time kernel that runs just one application Does not take interrupts No timers No scheduler Temporarily provides task with hard real time guarantees Application Application Kernel RT Application RT Kernel 10/6/

38 # cycles for 1k memory stores Almost all samples between 6-7k Outliers (OS jitter) No samples outside of 6-7k range 10/6/

39 Future Work & Applications Transfer s between power efficient and high performance cores Dynamic OS instrumentation Profiling, tracing kernels A/B kernel testing Specialized kernel to run applications in guest ring 0 cf. Arrakis 10/6/

40 Conclusion Decoupling the kernel state Result: highly dynamic OS architecture Kernels can be rebooted, updated and specialized Cores can be allocated and de-allocated arbitrarily For many versions of the dark silicon hardware, this may be the only way for system software 10/6/

41 Backup 25

42 Dealing with interrupts 1. Timers, etc. local to core and driver Handled internally to driver 2. Inter-processor interrupts (IPIs) Indirection table of s physical cores 3. Device interrupts Must be re-routed to new core 27

43 Device interrupts Device driver kernel 1 IRQh core 1 core 2 vectors IOAPIC 28

44 Device interrupts Device driver Kernel masks interrupts kernel 1 kernel 2 core 1 core 2 vectors IOAPIC 29

45 Device interrupts New kernel initializes Device driver IRQh kernel 2 core 1 core 2 vectors vectors IOAPIC 30

46 Device interrupts "Register interrupts" message Device driver IRQh kernel 2 core 1 core 2 vectors vectors IOAPIC 31

47 Device interrupts Device driver IRQh kernel 2 Interrupt rerouted core 1 core 2 vectors IOAPIC 32