Microkernel Design A walk through selected aspects of

Transcription

1 Microkernel Design A walk through selected aspects of kernel design and sel4

2 These slides are made distributed under the Creative Commons Attribution 3.0 License, unless otherwise noted on individual slides. You are free: to Share to copy, distribute and transmit the work to Remix to adapt the work Under the following conditions: Attribution You must attribute the work (but not in any way that suggests that the author endorses you or your use of the work) as follows: Courtesy of Kevin Elphinstone, UNSW The complete license text can be found at 2 Kevin Elphinstone. Distributed under Creative Commons Attribution License

3 Formal Verification - Proof Architecture Specification Proof C Code NICTA Copyright 2010 From imagination to impact 3

4 Proof Architecture Access Control Spec Confinement Specification Design Haskell Prototype C Code NICTA Copyright 2010 From imagination to impact 4

5 Verification Strategy An OS perspective simple is better complex system-wide invariants increase difficulty concurrency is very difficulty to reason about must consider every possible interleaving of execution 5 Kevin Elphinstone. Distributed under Creative Commons Attribution License

6 Fundamental Kernel Execution Abstractions support CPU running multiple activities Memory support (and protect) state associated with an activity 6 Kevin Elphinstone. Distributed under Creative Commons Attribution License

7 Execution Two-execution environments kernel level (in-kernel) and user-level (application execution) Covered execution models in detail earlier in the course Two common approaches Event-based smaller memory footprint, limited to smaller kernels Process-based larger memory footprint, programming model scales to larger kernels, though synchronisation adds complexity 7 Kevin Elphinstone. Distributed under Creative Commons Attribution License

8 sel4 Kernel Execution? For verifiability Event-based sequential execution from kernel mode entry to exit Context switch at kernel exit current process/thread control block switch as late as possible kernel c-code not re-entrant Interrupts disabled delivered on return to user-level, or polled during long running operations 8 Kevin Elphinstone. Distributed under Creative Commons Attribution License

9 Application Execution From kernel perspective, commonly two models single-threaded straight forward program execution potentially with another execution model layered on top (e.g. user-level threads) multi-threaded potentially with another execution model or user-level involvement m-n user-level threads scheduler activations 9 Kevin Elphinstone. Distributed under Creative Commons Attribution License

10 Virtualisation Introduces third application (guest OS) execution model Virtual CPU Has close parallels to a thread We ll distinguish them as follows Fixed set at boot time e.g. no create/delete CPUs by guest Hardware-like synchronisation no blocking synch primitives Hardware-like communication low-level notification (interrupts), no complex messaging handled via interrupt handler 10 Kevin Elphinstone. Distributed under Creative Commons Attribution License

11 Application Execution For verification single threaded execution still simplest event-based sequential code multithreaded problematic due to concurrency good to overlap I/O (blocking) with execution, and to utilise multiprocessors virtual CPU with interrupts disabled event-based sequential code interrupts enabled, problematic due potential number of instruction interleavings obviously good for replication normal CPU execution model for guest OS. 11 Kevin Elphinstone. Distributed under Creative Commons Attribution License

12 Multithreaded sel4 Application Execution? verified applications would be limited to a single thread Alternatives VCPUs verified applications have interrupts disabled 12 Kevin Elphinstone. Distributed under Creative Commons Attribution License

13 Memory Management Page-based virtual memory ubiquitous Applications expect a specific memory model Text, data, bss, stack Memory mapped files shared libraries, shared memory External pagers of memory objects Mach External control of mappings Virtualisation (hypercalls, shadow page tables) L4 13 Kevin Elphinstone. Distributed under Creative Commons Attribution License

14 Text, data,. Virtual Address Space Text Data BSS Stack Implications for kernel knowledge of executable format limits alternative e.g. guest OS, guest application at minimum, ability to load application and set up mappings also implies allocation of page tables and memory frames. implies some model for managing memory securely between applications also implies book keeping for de-allocation, i.e. resource attribution e.g. processes. 14 Kevin Elphinstone. Distributed under Creative Commons Attribution License

15 Memory Mapped Files/Objects Virtual Address Space Text Data BSS libc File Stack Implications for kernel similar to text, data, additionally adds file-like store to name data and retrive/store data adds mechanism for mapping vm region to file 15 Kevin Elphinstone. Distributed under Creative Commons Attribution License

16 External Pagers User-level server File System Server Text Data BSS libc File Stack Page faults propagated to user-level servers they supply data for page, kernel still manages memory (frames, page tables, etc..) Implications for kernel adds complexity of vm-region-based fault forwarding data provision mechanism removes complexity of supplying/storing data from the kernel (not in Mach s case) 16 Kevin Elphinstone. Distributed under Creative Commons Attribution License

17 Historical L4 Mapping Model 17 Kevin Elphinstone. Distributed under Creative Commons Attribution License

18 2002 Kevin Elphintone Address Spaces

19 Address Spaces map unmap grant 2002 Kevin Elphintone

20 Address Spaces map unmap grant 2002 Kevin Elphintone

21 Address Spaces map unmap grant 2002 Kevin Elphintone

22 Page Fault Handling "PF" msg Application Pager map msg 2002 Kevin Elphintone

23 Page Fault Handling PF IPC "PF" msg Application Pager res IPC map msg 2002 Kevin Elphintone

24 Address Spaces Physical Memory 2002 Kevin Elphintone

25 Address Spaces Initial AS Physical Memory 2002 Kevin Elphintone

26 Address Spaces Pager 1 Pager 2 Initial AS Physical Memory 2002 Kevin Elphintone

27 Address Spaces Pager 4 Pager 3 Pager 1 Pager 2 Initial AS Physical Memory 2002 Kevin Elphintone

28 Address Spaces Application Application Application Application Pager 4 Pager 3 Pager 1 Pager 2 Initial AS Physical Memory 2002 Kevin Elphintone

29 Address Spaces Application Application Application Application Pager 4 Driver Pager 3 Driver Pager 1 Pager 2 Initial AS Physical Memory 2002 Kevin Elphintone

30 Historical L4 Mapping Kernel only provides Model relatively simple mechanisms physical memory can be directly managed at userlevel page-tables still managed in kernel complexity of some memory management remains introduces complexity of tracking mapping relationships 30 Kevin Elphinstone. Distributed under Creative Commons Attribution License

31 Recursive mapping removed Single privilege syscall for Initial AS Pagers requested mapping from Initial AS Removed need to track mapping relationships from kernel Application Pager 1 Pager 2 Initial AS Physical Memory 31 Kevin Elphinstone. Distributed under Creative Commons Attribution License

32 Initial task removed Mapping operates preallocate physical memory partitions Removes need for user-level to proxy Adds partitioning policy in kernel, but not significant source of complexity page table management still in kernel some memory allocation remains Application Pager Physical Memory Pager Kevin Elphinstone. Distributed under Creative Commons Attribution License 32

33 Note parallels with Hypervisors Mapping operates on preallocated physical memory partitions hypercalls page table management still in kernel some memory allocation remains page table management becomes quite tricky when directly virtualising page tables without hardware assistance Application Guest OS Guest OS Physical Memory 33 Kevin Elphinstone. Distributed under Creative Commons Attribution License

34 Kernel Design for Isolation and Assurance of Physical Memory Dhammika Elkaduwe Philip Derrin Kevin Elphinstone

35 Embedded Systems Increasing functionality Increasing software complexity Millions of lines of code Mutually untrusted SW vendors Consolidate functionality Connectivity Attacks from outside No longer close systems Download SW IIES08/seL4 1

36 Embedded Systems Diverse applications Real-time Vs. best effort Tight resource budgets Mission/life- critical applications Sensitive information Reliability is paramount IIES08/seL4 2

37 Small Kernel Approach Smaller, more trustworthy foundation Hypervisor, microkernel, isolation kernel,.. Facilitate controlled integration and isolation Isolate: fault isolation, diversity Legacy Legacy Legacy App. App. App. Legacy App. Untrusted Trusted Sensitive Sensitive Sensitive App. Sensitive App. App. App. Integrate: performance Linux Server Device Driver Trusted Service Trusted Service Trusted Service Trusted Service Device Driver Device Driver Supervisor OS Small kernel (e.g. Microkernel) Hardware IIES08/seL4 3A

38 Small Kernel Approach Smaller, more trustworthy foundation Hypervisor, microkernel, isolation kernel,.. Facilitate controlled integration and isolation Isolate: fault isolation, diversity Legacy Legacy Legacy App. App. App. Legacy App. Untrusted Trusted Sensitive Sensitive Sensitive App. Sensitive App. App. App. Integrate: performance Linux Server Device Driver Trusted Service Trusted Service Trusted Service Trusted Service Device Driver Device Driver Microkernel should: Provide sufficient API Correct realisation of API Adhere to isolation/integration requirements of the system Supervisor OS Small kernel (e.g. Microkernel) Hardware IIES08/seL4 3B

39 Issue Kernel consumes resources Machine cycles Physical memory (kernel metadata) Example: threads thread control block, Legacy Legacy Legacy App. App. App. Legacy App. Untrusted Trusted Sensitive Sensitive Sensitive App. Sensitive App. App. App. address space page-tables bookkeeping to reclaim memory Linux Server Device Driver Trusted Service Trusted Service Trusted Service Trusted Service Device Driver Device Driver Supervisor OS Microkernel TCB PT TCB PT IIES08/seL4 4

40 Possible Approaches How do we manage kernel metadata? Cache like behaviour [EROS,Cache kernel, HiStar..] No predictability, limited RT applicability Static allocations Works for static systems Dynamic systems: overcommit or fail under heavy load Domain specific kernel modifications? Legacy Legacy Legacy App. App. App. Legacy App. Linux Server Untrusted Device Driver Trusted Supervisor OS Trusted Service Trusted Service Trusted Service Trusted Service Sensitive Sensitive Sensitive App. Sensitive App. App. App. Device Driver Device Driver Microkernel TCB PT TCB PT IIES08/seL4 5

41 Modified Verified L4.Verified project: Formally verify the implementation correctness of the kernel Properties: Isolation, information flow... Formal refinement Formally connect the properties with the kernel implementation Mathematically proven properties Abstract Model Property preserving refinement C Code HW IIES08/seL4 6A

42 Modified Verified L4.Verified project: Formally verify the implementation correctness of the kernel Properties: Isolation, information flow... Formal refinement Formally connect the properties with the kernel implementation Mathematically proven properties Abstract Model Property preserving refinement C Code HW IIES08/seL4 6B

43 Modified Verified L4.Verified project: Formally verify the implementation correctness of the kernel Properties: Isolation, information flow... Formal refinement Formally connect the properties with the kernel implementation Modifications invalidate refinement Verification is labour intensive 10K C-lines = 200K proof lines Memory management is core functionality Mathematically proven properties Abstract Model C Code Property preserving refinement HW IIES08/seL4 6C

44 Approach in a nutshell Truste d OS server Kernel heap Legacy OS server... supervisory OS sel4 Microkernel No implicit allocations within the kernel No heap, no slab allocation etc.. All abstractions are provided by first-class kernel objects Threads TCB object Address space Page table objects All objects are created upon explicit user request IIES08/seL4 7

45 Memory Management Model Trusted OS server Legacy OS server... No implicit allocations within the kernel Physical memory is divided into untyped objects Authority conferred via capabilities supervisory OS Untyped capability is sufficient authority to allocate kernel objects Kernel Code sel4 Microkernel untyped object1 untyped object2.. untyped object n All abstractions are provided via first class kernel objects Allocate on explicit user request Creator gets the full authority Distribute capabilities to allow other access the service IIES08/seL4 8A

46 Memory Management Model Trusted OS server Kernel Code TCB Legacy OS server supervisory OS... sel4 Microkernel TCB Kernel objects Untyped untyped object2 TCB (Thread Control Blocks) Capability tables (CT) Comm. ports..... untyped object n No implicit allocations within the kernel Physical memory is divided into untyped objects Authority conferred via capabilities Untyped capability is sufficient authority to allocate kernel objects All abstractions are provided via first class kernel objects Allocate on explicit user request Creator gets the full authority Distribute capabilities to allow other access the service IIES08/seL4 8B

47 Memory Management Model Trusted OS server Kernel Code TCB Legacy OS server... supervisory OS sel4 Microkernel TCB Kernel objects Untyped PT PT TCB (Thread Control Blocks) Capability tables (CT) Comm. ports... Objects are managed by user-level.. untyped object n No implicit allocations within the kernel Physical memory is divided into untyped objects Authority conferred via capabilities Untyped capability is sufficient authority to allocate kernel objects All abstractions are provided via first class kernel objects Allocate on explicit user request Creator gets the full authority Distribute capabilities to allow other access the service IIES08/seL4 8C

48 Memory Management Model... Trusted OS server Legacy OS server... Delegate authority Allow others to obtain services Delegate resource management Kernel Code TCB supervisory OS Microkernel TCB untyped object2.. untyped object n Memory management policy is completely in user-space Isolation of physical memory = Isolation of authority (capabilities) Capability dissemination is controlled by a Take-Grant like protection model IIES08/seL4 8D

49 Memory Management Model... Trusted OS server Legacy OS server... De-allocation upon explicit user request Call revoke on the Untyped capability Memory can be reused supervisory OS Kernel tracks capability derivations Kernel Code TCB sel4 Microkernel TCB untyped object2.. untyped object n Recorded in capability derivation tree (CDT) Need bookkeeping Doubly-linked list through capabilities untyped cap 1 Space allocated with capability tables CDT TCB TCB TCB copy 9

50 Capability Derivation Tree Trusted OS server Legacy OS server... For allocation: The untyped capability should not have any CDT children supervisory OS sel4 Microkernel Guarantees that there are no previously allocated objects Size of the object(s) must be small or equal to untyped object Kernel Code TCB TCB untyped object2.. untped object n untyped cap 1 CDT TCB TCB TCB copy IIES08/seL4 10

51 Evaluation Formal properties: Formalised the protection model in Isabelle/HOL Machine checked, abstract model of the kernel Formal, machine checked proof that mechanisms are sufficient for enforcing spatial partitioning Proof also identify the invariants the supervisory OS needs to enforce for isolation to hold... supervisory OS sel4 Microkernel IIES08/seL4 11A

52 Evaluation Formal properties: Formalised the protection model in Isabelle/HOL Machine checked, abstract model of the kernel Formal, machine checked proof that mechanisms are sufficient for enforcing spatial partitioning Proof also identify the invariants the supervisory OS needs to enforce for isolation to hold Can not share modifiable page/capability tables Can not share thread control blocks Can not have communication channels that allow capability propagation... supervisory OS sel4 Microkernel IIES08/seL4 11B

53 Evaluation... Performance Used paravirtualised Linux as an example Compared with L4/Wombat (Linux) for running LMBench Linux Driv ers... supervisory OS sel4 Microkernel Bench mark L4 ( s) sel4( s) Gain(%) fork exec shell page faults Null Syscall ctx Proxy via Iguana Linux Driv (Wom ers bat)... Iguana L4 Microkernel IIES08/seL4 12

54 Conclusion No implicit allocations within the kernel Users explicitly allocate kernel objects No heap, slab.. (no hidden bookkeeping) Authority confinement guarantees control of kernel memory All kernel memory management policy is outside the kernel Different isolation/integration configurations Support diverse, co-existing policies No modification to the kernel (remains verified) Hard guarantees on kernel memory consumption Facilitate formal reasoning of physical memory consumption Improve performance by controlled delegation Similar performance in other case IIES08/seL4 14

55 Virtual Memory & sel4 Implemented using 3 objects* Frames: An object corresponding to physical memory Page directory: An object corresponding to level 1 page table of a two-level page table. Page table: An object corresponding to level 2 page table of a two-level page table created from untyped memory (as directed by user-level) * currently actually 4 expect ASIDs will be removed 55 Kevin Elphinstone. Distributed under Creative Commons Attribution License

56 Virtual Memory & sel4 Broadly similar model to previous L4 kernels VM faults are propagated as IPC Introduce new page fault type missing page table To install a mapping, one needs: A cap to a page directory page table to be installed in page directory install requires cap to both PD and PT A cap to a frame of physical memory Thus, model allows creation of domain specific VM model using only authorised memory Revocation handled via CDT 56 Kevin Elphinstone. Distributed under Creative Commons Attribution License

57 Verification Perspective Complexity of memory management policy, and VM model pushed outside the kernel simple VM model implemented at user-level should also be verifiable unverified complex models also supported e.g. para-virtualised guest OS s CDT an additional complexity needed for revocation of caps anyway guarantees integrity (used to determine when memory has no references) 57 Kevin Elphinstone. Distributed under Creative Commons Attribution License

58 Quick Summary Basic abstractions Execution Memory Many alternative models sel4 uses subset that: is amenable to verification in-kernel should be amenable to verification at user-level 58 Kevin Elphinstone. Distributed under Creative Commons Attribution License

59 Inter-process Communication Enables system construction alternative is a monolithic server Processes cooperate to provide services Enables extensibility of the system 59 Kevin Elphinstone. Distributed under Creative Commons Attribution License

60 IPC Semantics Blocking versus Non-blocking Buffered versus Unbuffered Fixed versus Variable-size Direct versus Indirect 60 Kevin Elphinstone. Distributed under Creative Commons Attribution License

61 Blocking versus Nonblocking Blocking (termed synchronous) Send return control only after message is sent Receive returns control only after message is received Non-blocking (termed asynchronous) Send message always immediately copied or queued, and send returns Receive Issues: polls for new message Needs buffering buffering bounded 61 Kevin Elphinstone. Distributed under Creative Commons Attribution License

62 Buffered versus Unbuffered Buffered Requires at least extra copy to buffer Send may get ahead of receive matches differing processing rates Buffers are finite send eventually becomes blocking synchronisation and rendezvous occurs Unbuffered Rendezvous always Potential to copy message directly performance 62 Kevin Elphinstone. Distributed under Creative Commons Attribution License

63 Fixed versus Variable Size Fixed size simplifies buffering and marshalling Variable size needs receiver to wait on largest size message every time not really an issue except for large messages 63 Kevin Elphinstone. Distributed under Creative Commons Attribution License

64 Direct versus Indirect Direct send(dest, message) receive(var, message) Source Dest 64 Kevin Elphinstone. Distributed under Creative Commons Attribution License

65 Direct versus Indirect Indirect send(mailbox, message) receive(var, message) Comms path first class objects Source Mailbox Dest 65 Kevin Elphinstone. Distributed under Creative Commons Attribution License

66 sel4 IPC model 6 system calls send, nbsend, call, wait, reply, replywait 2 communication objects EndPoint, AsyncEndPoint 66 Kevin Elphinstone. Distributed under Creative Commons Attribution License

67 Kernel Calls are IPC IPC specifies a capability as the destination call -ing a cap, invokes the kernel identifies the object TCB, PD, PT specifies the method and arguments of call 67 Kevin Elphinstone. Distributed under Creative Commons Attribution License

68 Communications Objects EndPoint (EP) and AsyncEndPoint (AEP) acts as a mailbox (indirect comms) distinguished caps to EP and AEP have badges a word of bits used to determine authority or identity of sender 68 Kevin Elphinstone. Distributed under Creative Commons Attribution License

69 Call EndPoints sends message via EP unbuffered (at the moment) receiver receives message unforgeable badge a reply cap to sender allows caps to propagate in a usable way reply responds via reply cap 69 Kevin Elphinstone. Distributed under Creative Commons Attribution License

70 Call, EP, and extensible systems Call and EP enable kernel extensibility via user-level servers (Hydra) Calling a capability invokes a kernel implemented object TCB, PD, PT, etc. invokes a server implemented object Capability propagation is consistent for both kerneland user-level implemented objects authority confinement of kernel object applies to user-objects as well 70 Kevin Elphinstone. Distributed under Creative Commons Attribution License

71 AEP Used for signalling nbsend Badge is or -ed with word in AEP object can never block Receiving receives state of AEP word zeros work (atomically) Depending on encoding of badges, notification of 32 source events used in conjunction with shared memory. 71 Kevin Elphinstone. Distributed under Creative Commons Attribution License

72 IPC Importance

73 General IPC Algorithm Validate parameters Locate target thread if unavailable, deal with it Transfer message short data only long outlined or cap transfer Schedule target thread switch address space as necessary Wait for IPC

74 IPC - Implementation Short IPC

75 Short IPC (uniprocessor) system-call preamble (disable intr) identify dest thread or endpoint and check basically cap lookup ready-to-receive? analyze msg and transfer short: no action required switch to dest thread & address space system-call postamble The critical path

76 Short IPC (uniprocessor) call system-call pre (disable intr) identify dest thread or endpoint and check basically cap lookup ready-to-receive? running wait to receive analyze msg and transfer short: no action required switch to dest thread & address space system-call post wait to receive running

77 Short IPC (uniprocessor) send (eagerly) system-call pre (disable intr) identify dest thread or endpoint and check basically cap lookup ready-to-receive? running running analyze msg and transfer short: no action required switch to dest thread & address space system-call post wait to receive running Not common operation if send is signal

78 Short IPC (uniprocessor) send (lazily) system-call pre (disable intr) identify dest thread or endpoint and check basically cap lookup ready-to-receive? running running analyze msg and transfer short: no action required switch to dest thread & address space system-call post wait to receive running

79 IPC EAX ECX EDX EBX ESI EDI EBP ESP EFLAGS EIP CS SS DS ES FS GS

80 IPC EAX ECX EDX EBX ESI EDI EBP ESP EFLAGS EIP CS SS DS ES FS GS

81 IPC EAX ECX EDX EBX ESI EDI EBP ESP EFLAGS EIP CS SS DS ES FS GS

82 IPC EAX ECX EDX EBX ESI EDI EBP ESP EFLAGS EIP CS SS DS ES FS GS

83 IPC EAX ECX EDX EBX ESI EDI EBP ESP EFLAGS EIP CS SS DS ES FS GS

84 IPC EAX ECX EDX EBX ESI EDI EBP ESP EFLAGS EIP CS SS DS ES FS GS

85 IPC EAX Note payload from green thread ECX EDX EBX ESI EDI EBP ESP EFLAGS EIP CS SS DS ES FS GS

86 Implementation Goal Most frequent kernel op: short IPC thousands of invocations per second Performance is critical: structure IPC for speed structure entire kernel to support fast IPC What affects performance? cache line misses TLB misses memory references pipe stalls and flushes instruction scheduling

87 Fast Path Optimize for common cases write in assembler non-critical paths written in C/C++ but still fast as possible Avoid high-level language overhead: function call state preservation poor code optimizations We want every cycle possible!

88 IPC Attributes for Fast Path short message single runnable thread after IPC must be valid IPC call switch threads, originator blocks send phase: the target is waiting receive phase: the sender is not ready to couple, causing us to block

89 Avoid Memory References!!! Memory references are slow Microkernel should minimize indirect costs cache pollution TLB pollution memory bus

90 Optimized Memory Also: hard-wire TLB entries for kernel code and data. stack Single TLB entry. thread state UTCB cpu ID thread ID TCB state, grouped by cache lines.

91 Branch Elimination Common case: -1 slow = ~receiver->thread_state + (timeouts & 0xffff) + sender->resources + receiver->resources; if( slow ) enter_slow_path() Reduces branch prediction foot print. Avoids mispredicts & stalls & flushes. Increases latency for slow path Common case: 0

92 TCB Resources Resources bitfield 1 1 One bit per resource Fast path checks entire word if not 0, jump to resource handlers Debug registers Copy area

93 Message Transfer IBM PowerPC 750, 500 MHz, 32 registers up to 10 physical registers virtual register copy loop Many cycles wasted on pipe flushes for privileged instructions.

94 Slow Path vs. Fast Path L4Ka::Pistachio IPC performance Pentium cycles Inter C-Path Inter FastPath num ber m essage registers

95 Inter vs. Intra Address Space L4Ka::Pistachio IPC performance Pentium cycles Intra FastPath Inter FastPath num ber m essage registers

96 IPC - Implementation Long IPC

97 Long IPC (uniprocessor) system-call preamble (disable intr) identify dest and check ready-to-receive? analyze msg and transfer long/map: Preemptions possible! (end of timeslice, device interrupt ) Pagefaults possible! (in source and dest address space) transfer message switch to dest thread & address space system-call postamble

98 Long IPC (uniprocessor) system-call pre (disable intr) identify dest and check ready-to-receive? analyze msg and transfer long/map: lock both partners Preemptions possible! (end of timeslice, device interrupt ) Pagefaults possible! (in source and dest address space) transfer message unlock both partners switch to dest thread & address space system-call post

99 Long IPC (uniprocessor) system-call pre (disable intr) identify dest and check ready-to-receive? analyze msg and transfer long/map: lock both partners enable intr transfer message disable intr unlock both partners switch to dest thread & address space system-call post Preemptions possible! (end of timeslice, device interrupt ) Pagefaults possible! (in source and dest address space)

100 Long IPC (uniprocessor) system-call pre (disable intr) identify dest thread and check running locked running wait to receive same chief ready-to-receive? analyze msg and transfer long/map: lock both partners enable intr transfer message disable intr unlock both partners switch to dest thread & address space system-call post wait locked wait running

101 IPC - mem copy Why is it needed? Why not share? Security Need own copy Granularity Object small than a page or not aligned

102 copy in - copy out copy into kernel buffer

103 copy in - copy out copy into kernel buffer switch spaces

104 copy in - copy out copy into kernel buffer switch spaces copy out of kernel buffer costs for n words 2 2n r/w operations 3 n/8 cache lines 1 n/8 overhead cache misses (small n) 4 n/8 cache misses (large n)

105 temporary mapping

106 temporary mapping select dest area (4+4 M)

107 temporary mapping select dest area (4+4 M) map into source AS (kernel)

108 temporary mapping select dest area (4+4 M) map into source AS (kernel) copy data

109 temporary mapping select dest area (4+4 M) map into source AS (kernel) copy data switch to dest space

110 temporary mapping

111 temporary mapping problems multiple threads per AS mappings might change while message is copied How long to keep PTE? What about TLB? current AS

112 temporary mapping invalidate PTE flush TLB when leaving curr thread during ipc? current AS

113 temporary mapping invalidate PTE flush TLB when leaving curr thread during ipc: current AS

114 temporary mapping when returning to thread during ipc: current AS

115 temporary mapping Reestablishing temp mapping requires to store partner id and dest area address in the sender s tcb. Note: receiver s page mappings might have changed! when returning to thread during ipc: current AS

116 Cost estimates Copy in - copy out Temporary mapping R/W operations Cache lines Small n overhead cache misses Large n cache misses Overhead TLB misses Startup instructions 2 2n 2n 3 n/8 2 n/8 n/8 0 5 n/8 3 n/8 0 n / words per page 0 50

117 486 IPC costs Mach: copy in/out L4: temp mapping [µs] Mach L4 + cache flush L4 raw copy msg len

118 Summary Small messages buffering costs a little mapping more so ideally, direct copy between two pinned message areas needs to be synchronous Large messages mapping is more efficient especially with outlined messages startup costs high (cost of setup amortised) implementation complexity high Shared memory and notification similar to buffering in terms of performance copy-in copy-out if mutually distrusting implementation complexity out of kernel 118 Kevin Elphinstone. Distributed under Creative Commons Attribution License

119 EndPoint sel4 unbuffered, synchronous, small message to preallocated pinned buffer used for call AsyncEndPoint or - ed notification used for notification (shared memory buffers) Expect long copied messages to be avoided if possible via shared memory 119 Kevin Elphinstone. Distributed under Creative Commons Attribution License

120 FPU Context Switching Strict switching Thread switch: Store current thread s FPU state Load new thread s FPU state Extremely expensive IA-32 s full SSE2 state is 512 Bytes IA-64 s floating point state is ~1.5KB May not even be required Threads do not always use FPU

121 Lazy FPU switching Kernel FPU Lock FPU on thread switch Unlock at first use exception handled by kernel Unlock FPU If fpu_owner!= current Save current state to fpu_owner Load new state from current fpu_owner := current current fpu_owner locked pacman() finit fld fcos fst finit fld