Pre-virtualization internals

Size: px

Start display at page:

Download "Pre-virtualization internals"

Scarlett Shavonne Conley
6 years ago
Views:

1 Pre-virtualization internals Joshua LeVasseur 3 March 2006 L4Ka.org Universität Karlsruhe (TH)

2 Compile time overview Compiler C code Assembler code OS source code Hand-written assembler Afterburner Assembler Virtualization aware binary 2006 Universität Karlsruhe Pre-virtualization internals Joshua LeVasseur Page 2 of 30

3 Afterburner assembler parser Built using an Antlr grammar 2006 Universität Karlsruhe Pre-virtualization internals Joshua LeVasseur Page 3 of 30

4 Instruction padding Virtualization aware binary movl %eax, %fs Afterburner.L_sensitive_6: movl %eax, %fs nop nop nop nop nop nop.l_sensitive_7:.pushsection.afterburn.balign 4.long.L_sensitive_6.long.L_sensitive_7.popsection 2006 Universität Karlsruhe Pre-virtualization internals Joshua LeVasseur Page 4 of 30

Address space Xen x86 0 4 GB Guest application Guest kernel IPVMM Hypervisor 0 Ring 1

5 Address space Xen x GB Guest application Guest kernel IPVMM Hypervisor 0 Ring 1 Ring Universität Karlsruhe Pre-virtualization internals Joshua LeVasseur Page 5 of 30

6 Address space Xen x86 L4, Linux-on-Linux (generic) 0 4 GB Guest application Guest kernel IPVMM Hypervisor 0 Ring 1 Ring 3 Guest kernel IPVMM Hypervisor Guest application Hypervisor 2006 Universität Karlsruhe Pre-virtualization internals Joshua LeVasseur Page 6 of 30

7 Loading the guest kernel Virtualization aware binary Guest kernel Guest kernel IPVMM IPVMM IPVMM Hypervisor 2006 Universität Karlsruhe Pre-virtualization internals Joshua LeVasseur Page 7 of 30

8 In-Place VMM mov eax, cr3 sti cli popf hlt frontend Virtual CPU IPVMM backend Map, unmap, TLB, timer, threads Page faults, traps, system calls, interrupts Hypervisor 2006 Universität Karlsruhe Pre-virtualization internals Joshua LeVasseur Page 8 of 30

9 Constraints Code expansion: Timing? Interrupts? Simple state machine? Efficiency? 2006 Universität Karlsruhe Pre-virtualization internals Joshua LeVasseur Page 9 of 30

10 Constraints Code expansion: Timing? Interrupts? Simple state machine? Efficiency? Guest kernel is a sequential process Important: forward progress Unimportant: rate of forward progress We provide a virtual CPU Illusion of continuous time 2006 Universität Karlsruhe Pre-virtualization internals Joshua LeVasseur Page 10 of 30

11 Constraints Code expansion: Timing? Interrupts? Simple state machine? Efficiency? Guest kernel is a sequential process Important: forward progress Unimportant: rate of forward progress We provide a virtual CPU Illusion of continuous time Delay delivery Synchronous (sti, popf) Optimize common case Asynchronous Avoid IPVMM reentrance 2006 Universität Karlsruhe Pre-virtualization internals Joshua LeVasseur Page 11 of 30

12 Constraints Code expansion: Timing? Interrupts? Simple state machine? Efficiency? Thread model VM thread Interrupt thread Guest kernel is a sequential process Important: forward progress Unimportant: rate of forward progress We provide a virtual CPU Illusion of continuous time Delay delivery Synchronous (sti, popf) Optimize common case Asynchronous Avoid IPVMM reentrance 2006 Universität Karlsruhe Pre-virtualization internals Joshua LeVasseur Page 12 of 30

13 Threads mov eax, cr3 sti cli popf hlt frontend backend VM thread Page faults, traps, system calls Hypervisor 2006 Universität Karlsruhe Pre-virtualization internals Joshua LeVasseur Page 13 of 30

14 Threads mov eax, cr3 sti cli popf hlt frontend IRQ thread backend VM thread Interrupts Page faults, traps, system calls Hypervisor 2006 Universität Karlsruhe Pre-virtualization internals Joshua LeVasseur Page 14 of 30

15 Frontend entry mov eax, cr3 mov eax, cr3 sti cli popf hlt Assembler trampoline frontend Establish C calling conventions. Future: eliminate by Afterburning the IPVMM. IRQ thread backend VM thread C++ frontend, thread Interrupts Page faults, traps, system calls Hypervisor 2006 Universität Karlsruhe Pre-virtualization internals Joshua LeVasseur Page 15 of 30

16 Boundary transitions Guest kernel invlpg vaddr pushl lea call popl %eax vaddr, %eax burn_invlpg %eax 2006 Universität Karlsruhe Pre-virtualization internals Joshua LeVasseur Page 16 of 30

17 Boundary transitions (generic) Guest kernel IPVMM trampoline invlpg vaddr pushl %eax lea vaddr, %eax call burn_invlpg popl %eax burn_invlpg: // Preserve C clobbers. pushl %eax pushl %ecx pushl %edx // Build burn_clobbers_frame_t parameter. pushl %esp subl $8, 0(%esp) call afterburn_cpu_invlpg_ext popl %edx ; popl %ecx ; popl %eax ret 2006 Universität Karlsruhe Pre-virtualization internals Joshua LeVasseur Page 17 of 30

18 Boundary transitions (generic) Guest kernel invlpg vaddr pushl %eax lea vaddr, %eax call burn_invlpg popl %eax IPVMM trampoline burn_invlpg: // Preserve C clobbers. pushl %eax pushl %ecx pushl %edx // Build burn_clobbers_frame_t parameter. pushl %esp subl $8, 0(%esp) call afterburn_cpu_invlpg_ext popl %edx ; popl %ecx ; popl %eax ret IPVMM C++ frontend struct burn_clobbers_frame_t { word_t burn_ret_address; word_t frame_pointer; word_t edx; word_t ecx; word_t eax; word_t guest_ret_address; word_t params[0]; }; extern "C" void afterburn_cpu_invlpg_ext( burn_clobbers_frame_t *frame ) { backend_flush_vaddr( frame->eax ); } 2006 Universität Karlsruhe Pre-virtualization internals Joshua LeVasseur Page 18 of 30

19 Atomic instructions mov eax, cr3 VM thread C++ code Time Interrupts During IPVMM code? During a hypercall? 2006 Universität Karlsruhe Pre-virtualization internals Joshua LeVasseur Page 19 of 30

20 Thread reschedule mov eax, cr3 VM thread C++ code ret Time VM thread C++ code: IRQ dispatch 2006 Universität Karlsruhe Pre-virtualization internals Joshua LeVasseur Page 20 of 30

21 Race conditions VM thread Time iret if( IRQ pending ) reschedule(); else iret(); 2006 Universität Karlsruhe Pre-virtualization internals Joshua LeVasseur Page 21 of 30

22 Race conditions Time iret VM thread iret_restart: if( IRQ pending ) reschedule(); else iret(); IRQ thread if( VM thread in iret ) rollback( iret_restart ); Universität Karlsruhe Pre-virtualization internals Joshua LeVasseur Page 22 of 30

Modules Guest kernel Interrupt delivery to guest Memory paging Instruction set emulation Port accesses Device memory accesses Device switch Network RTC Synchronous

23 Modules Guest kernel Interrupt delivery to guest Memory paging Instruction set emulation Port accesses Device memory accesses Device switch Network RTC Synchronous events Asynchronous events Disk PIT Serial port PCI Interrupt dispatch Hypervisor PIC 2006 Universität Karlsruhe Pre-virtualization internals Joshua LeVasseur Page 23 of 30

24 Adding a device model 1. Define a device class 2. Define its interfaces: Port accesses Memory-mapped registers 3. Define its PCI registers: Static structure created at compile-time 4. Raise interrupts with the intlogic_t class 5. In some cases, define a virtual IRQ handler 2006 Universität Karlsruhe Pre-virtualization internals Joshua LeVasseur Page 24 of 30

25 Productivity-focused coding Minimize errors: Simple code Avoid purely quantitative increase in work Use assertions Avoid reentrancy C++ Modularity (quickly understood code) Compile-time features only Fast (world's fastest kernel, L4Ka::Pistachio, is written in C++) No dynamic memory allocation Construct data structures at compile time Ex: big switch() statement for port access, PCI device access Heavily typed 2006 Universität Karlsruhe Pre-virtualization internals Joshua LeVasseur Page 25 of 30

26 Productivity-focused coding Code maintenance: Minimize C Preprocessor We want structured code Use CML2 configuration system Feature management Code reuse: Abstract frontends and backends Reuse frontend across multiple backends 2006 Universität Karlsruhe Pre-virtualization internals Joshua LeVasseur Page 26 of 30

27 Legacy devices Keyboard UART 1 3, 4 CPU INTR XT-PIC primary 2 INTR XT-PIC secondary 8 0 RTC Channel 0 PIT Channel 2 PC speaker 2006 Universität Karlsruhe Pre-virtualization internals Joshua LeVasseur Page 27 of 30

28 Legacy devices: IDT vector Keyboard UART 1 3, 4 CPU INTR XT-PIC primary 2 INTR XT-PIC secondary 8 0 RTC Channel 0 ICW2 PIT Channel 2 PC speaker XT-PIC IRQ + IDT vector 2006 Universität Karlsruhe Pre-virtualization internals Joshua LeVasseur Page 28 of 30

29 Legacy devices: Ports 3F8h, 2F8h, 3E8h, 2E8h 60h, 64h Keyboard UART 20h, 21h 1 3, 4 CPU INTR XT-PIC primary 2 INTR XT-PIC secondary 8 0 A0h, A1h 70h, 71h RTC 40h-43h Channel 0 ICW2 PIT Channel 2 PC speaker XT-PIC IRQ + IDT vector 61h 2006 Universität Karlsruhe Pre-virtualization internals Joshua LeVasseur Page 29 of 30

30 Further information Publications: Full paper: Pre-Virtualization: Slashing the Cost of Virtualization Quick read: Pre-Virtualization: Uniting Two Worlds Quick read: White paper 2006 Universität Karlsruhe Pre-virtualization internals Joshua LeVasseur Page 30 of 30

Homework / Exam. Return and Review Exam #1 Reading. Machine Projects. Labs. S&S Extracts , PIC Data Sheet. Start on mp3 (Due Class 19)

Homework / Exam. Return and Review Exam #1 Reading. Machine Projects. Labs. S&S Extracts , PIC Data Sheet. Start on mp3 (Due Class 19) Homework / Exam Return and Review Exam #1 Reading S&S Extracts 385-393, PIC Data Sheet Machine Projects Start on mp3 (Due Class 19) Labs Continue in labs with your assigned section 1 Interrupts An interrupt