Xen is not just paravirtualization

Size: px

Start display at page:

Download "Xen is not just paravirtualization"

Nigel Todd
6 years ago
Views:

1 Xen is not just paravirtualization Dongli Zhang Oracle Asia Research and Development Centers (Beijing) December 16, 2016 Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

2 Plan Virtualization Xen Virtualization Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

3 Plan Virtualization Xen Virtualization When discussing virtualizatin 1) CPU Virtualization? 2) Memory Virtualization? 3) Device Virtualization? Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

4 What is virtualization A virtual machine is taken to be an efficient, isolated duplicate of the real machine (by Formal Requirements for Virtualizable Third Generation Architectures, Gerald J.Popek and Rebert P. Goldberg, 1974) Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

Virtualizable Third Generation Architectures, Gerald J.Popek and Rebert P.

5 What is virtualization A virtual machine is taken to be an efficient, isolated duplicate of the real machine (by Formal Requirements for Virtualizable Third Generation Architectures, Gerald J.Popek and Rebert P. Goldberg, 1974) Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

Trap and Emulate Virtual Machine (Guest) at Unprivileged Mode Virtual Machine Monitor (Host or Hypervisor) at Priviledged Mode Guest OS + Applications Unprivileged Page Fault

6 Trap and Emulate Virtual Machine (Guest) at Unprivileged Mode Virtual Machine Monitor (Host or Hypervisor) at Priviledged Mode Guest OS + Applications Unprivileged Page Fault Privileged Instruction virq MMU Emulation CPU Emulation IRQ Emulation Privileged Virtual Machine Monitor Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

7 x86 is NOT virtualizable Virtualizable Architecture: all sensitive instructions must also be privileged instructions (by Gerald J.Popek and Rebert P. Goldberg) critical instructions = sensitive instructions privileged instructions Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

8 x86 is NOT virtualizable Virtualizable Architecture: all sensitive instructions must also be privileged instructions (by Gerald J.Popek and Rebert P. Goldberg) critical instructions = sensitive instructions privileged instructions 18 critical instructions on x86 (Analysis of the Intel Pentium s Ability to Support a Secure Virtual Machine Monitor. USENIX Security 2000): SGDT/SIDT/SLDT, SMSW, PUSHF/POPF LAR/LSL, VERR/VERW, POP/PUSH CALL, JMP, INT n, RET STR, MOV Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

9 x86 is NOT virtualizable Virtualizable Architecture: all sensitive instructions must also be privileged instructions (by Gerald J.Popek and Rebert P. Goldberg) critical instructions = sensitive instructions privileged instructions 18 critical instructions on x86 (Analysis of the Intel Pentium s Ability to Support a Secure Virtual Machine Monitor. USENIX Security 2000): SGDT/SIDT/SLDT, SMSW, PUSHF/POPF LAR/LSL, VERR/VERW, POP/PUSH CALL, JMP, INT n, RET STR, MOV Solutions: Binary Translation (QEMU, VMWare) Paravirtualization (Xen) Hardware-Assisted Virtualization (Xen, KVM, VMWare based on Intel-VT and AMD-V) Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

10 Solution 1/3: Binary Translation philosophy: rewrite critical instructions Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

11 Solution 2/3: Hardware Virtualization (Intel VT) philosophy: instroduce new privileged mode Ring 3 Ring 3 Ring 0 Ring 0 Non-Root Mode (Guest) VM Entry VM Entry Non-Root Mode (Guest) VM Exit Ring 3 VM Exit VMXON Ring 0 Root Mode (VMM) VMOFF Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

12 KVM (Kernel-based Virtual Machine) CPU hardware virtualization extensions (Intel VT or AMD-V) Loadable kernel module (kvm.ko, kvm-intel.ko/kvm-amd.ko) QEMU as userspace emulator Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

13 Solution 3/3: Paravirtualization philosophy: replace critical instructions with hypercalls A hypercall is a software trap from a domain to the hypervisor, just as a syscall is a software trap from an application to the kernel x86 32: int 0x82 x86 64: syscall instruction x86 Intel-VT vmcall instruction user ring 3 user System call via syscall ring 3 user non-root ring 3 kernel Hypercall via int 0x82 ring 1 kernel Hypercall via syscall ring 3 kernel Hypercall via vmcall non-root ring 0 xen ring 0 xen Xen hypervisor will checks in which mode the syscall instruction is triggered ring 0 xen root ring 0 x86 32-bit pvm x86 64-bit pvm x86 vt-x hvm/pvhvm Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

14 State of the Art Virtualization Binary Translation (QEMU, Bochs, VMWare) Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

15 State of the Art Virtualization Binary Translation (QEMU, Bochs, VMWare) Paravirtualization (Xen) Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

16 State of the Art Virtualization Binary Translation (QEMU, Bochs, VMWare) Paravirtualization (Xen) Hardware-assisted Virtualization (KVM, Xen, VMware) Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

17 State of the Art Virtualization Binary Translation (QEMU, Bochs, VMWare) Paravirtualization (Xen) Hardware-assisted Virtualization (KVM, Xen, VMware) OS-level Virtualization (Linux Container) Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

18 State of the Art Virtualization Binary Translation (QEMU, Bochs, VMWare) Paravirtualization (Xen) Hardware-assisted Virtualization (KVM, Xen, VMware) OS-level Virtualization (Linux Container) Programming Language Virtualization (Java,.NET CLR) Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

19 State of the Art Virtualization Binary Translation (QEMU, Bochs, VMWare) Paravirtualization (Xen) Hardware-assisted Virtualization (KVM, Xen, VMware) OS-level Virtualization (Linux Container) Programming Language Virtualization (Java,.NET CLR) Library Virtualization (Wine, Cygwin) Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

20 What is Xen Wikipedia Xen Project is a hypervisor using a microkernel design, providing services that allow multiple computer operating systems to execute on the same computer hardware concurrently. Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

21 What is Xen Wikipedia Xen Project is a hypervisor using a microkernel design, providing services that allow multiple computer operating systems to execute on the same computer hardware concurrently. SOSP 2003: Xen and the Art of Virtualization This paper presents Xen, an x86 virtual machine monitor which allows multiple commodity operating systems to share conventional hardware in a safe and resource managed fashion, but without sacrificing either performance or functionality. Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

22 What is Xen Wikipedia Xen Project is a hypervisor using a microkernel design, providing services that allow multiple computer operating systems to execute on the same computer hardware concurrently. SOSP 2003: Xen and the Art of Virtualization This paper presents Xen, an x86 virtual machine monitor which allows multiple commodity operating systems to share conventional hardware in a safe and resource managed fashion, but without sacrificing either performance or functionality. Basic Idea of Paravirtualization Actively inform the hypervisor with the action guest is going to taken via hypercall Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

23 Xen Framework 1/2 xen hypervisor (microkernel): dictator scheduling, memory management, interrupt and device control per-domain and per-vcpu info management Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

24 Xen Framework 1/2 xen hypervisor (microkernel): dictator scheduling, memory management, interrupt and device control per-domain and per-vcpu info management dom0 (host): privileged admin xm/xend/xl (libxc) pygrub/hvmloader xenstored qemu and paravirtual driver backend native device driver Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

25 Xen Framework 1/2 xen hypervisor (microkernel): dictator scheduling, memory management, interrupt and device control per-domain and per-vcpu info management dom0 (host): privileged admin xm/xend/xl (libxc) pygrub/hvmloader xenstored qemu and paravirtual driver backend native device driver domu (guest): non-privileged user paravirtual driver frontend Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

26 Xen Framework 2/2 Domain 0 PVM HVM PVHVM xm xl xenstore xend QEMUs privcmd driver Legacy Device Drivers Backend PV Drivers Frontend PV Drivers Legacy Device Drivers Frontend PV Drivers Xen Hypervisor Memory Management CPU Virtualization Timer Virtualization Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

27 Convert Linux to Paravirtual Dom0/DomU ELF notes (Linux) or xen guest section (MiniOS) in kernel image Enable xen features in.config when building kernel Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

28 PV, HVM or PVHVM Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

29 Xen CPU Virtualization vcpu task struct domain container or process group xen schedules vcpu user 2. system call ring 3 user 1. system call ring 3 user 1. system call non-root ring 3 kernel 3. Trap to and handled in guest kernel ring 1 kernel 3. Handled in guest kernel ring 3 kernel 2. Trap to and handled In guest kernel directly non-root ring 0 xen 1. set a per-domain system call handler when the domain gets scheduled ring 0 xen 2. Route to guest kernel system call handler ring 0 xen root ring 0 x86 32-bit pvm x86 64-bit pvm x86 vt-x hvm/pvhvm Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

30 Xen Interrupt Virtualization: Event Channel 1/2 Event Channel Types Interdomain Event Virtual IRQ Event Physical IRQ Event IPI Event Registration PVM registers event channel handler to Xen via register callback(callbacktype event, xen hypervisor callback) PVHVM sets HYPERVISOR CALLBACK VECTOR via HYPERVISOR hvm op(hvmop set param, &a) Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

31 Xen Interrupt Virtualization: Event Channel 2/2 Domain 0 PVM HVM PVHVM xen_evtchn_do_upcall will traverse and handle each pending event xen_evtchn_do_upcall will traverse and handle each pending event Guest will handle interrupt as native machine xen_evtchn_do_upcall will traverse and handle each pending event IRQ handler for vector 0xf3 is called vcpu vcpu vcpu vcpu vcpu vcpu vcpu vcpu Global Event Channel Info Global Event Channel Mask Global Event Channel Mask Global Event Channel Mask Per-vcpu Event Channel Info Per-vcpu Event Channel Mask Per-vcpu Event Channel Mask Per-vcpu Event Channel Mask set eip to xen_hypervisor_callback during scheduling if vcpu has pending event set eip to xen_hypervisor_callback during scheduling if vcpu has pending event Intel-vt based interrupt injection and one vector for each irq Intel-vt based interrupt injection and vector 0xf3 for each event Xen Hypervisor Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

32 Xen Memory Virtualization 1/2 Address Types GVA (Guest Virtual Address) GPA (Guest Physical Address) or GFN (Guest page Frame Number) HPA (Host Physical Address) or MFN (Machine page Frame Number) Hardware-assisted Memory Virtualization (Method 1/3): Second-Level Page Table : Intel: Extended Page Table (EPT) : AMD: Nested Page Table (NPT) Guest CR3 Register Non-Root Mode Root Mode Host EPTP Register Guest Virtual Address Guest Page Tables Guest Physical Address Host Page Tables Host Physical Address Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

33 Xen Memory Virtualization 2/2 Direct Paging (Method 2/3): guest manage the (GVA, HPA) page table directly Shadow Paging (Method 3/3): xen hypervisor maintains a shadow (GVA, HPA) page table which is not awared by guest PFN MFN... P2m Table is mapped to guest by hypervisor MFN PFN Guest OS MFN MFN Guest OS PFN PFN PFN Shadow Table Xen Hypervisor Xen Hypervisor PFN PFN MMU MFN MFN MFN MMU MFN MFN MFN Direct Paging (MMU Paravirtualization) Shadow Page Table Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

34 Xen Device Virtualization HVM emulated legacy device (QEMU) Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

35 Xen Device Virtualization HVM emulated legacy device (QEMU) Paravirtual (PV) drivers Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

36 Xen Device Virtualization HVM emulated legacy device (QEMU) Paravirtual (PV) drivers Device Passthrough (vt-d) Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

37 Xen Device Virtualization HVM emulated legacy device (QEMU) Paravirtual (PV) drivers Device Passthrough (vt-d) Virtual Function (vt-d) Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

(vt-d) Virtual Function (vt-d) Dongli Zhang (Oracle)

38 Xen Device Virtualization HVM emulated legacy device (QEMU) Paravirtual (PV) drivers Device Passthrough (vt-d) Virtual Function (vt-d) Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

39 PV driver vs. PCI driver device abstraction device discovery device configuration data flow shared memory interrupt PCI driver pci device, pci driver PCI Tree PCI Config Space (IO/MMIO) DMA Ring Buffer N/A or IOMMU IOAPIC, MSI, MSI-X PV driver Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

40 PV driver vs. PCI driver PCI driver PV driver device abstraction pci device, pci driver xenbus device, xenbus driver device discovery PCI Tree device configuration PCI Config Space (IO/MMIO) data flow DMA Ring Buffer shared memory N/A or IOMMU interrupt IOAPIC, MSI, MSI-X Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

41 PV driver vs. PCI driver PCI driver PV driver device abstraction pci device, pci driver xenbus device, xenbus driver device discovery PCI Tree Xenstore device configuration PCI Config Space (IO/MMIO) data flow DMA Ring Buffer shared memory N/A or IOMMU interrupt IOAPIC, MSI, MSI-X Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

42 PV driver vs. PCI driver PCI driver PV driver device abstraction pci device, pci driver xenbus device, xenbus driver device discovery PCI Tree Xenstore device configuration PCI Config Space (IO/MMIO) Xenstore data flow DMA Ring Buffer shared memory N/A or IOMMU interrupt IOAPIC, MSI, MSI-X Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

43 PV driver vs. PCI driver PCI driver PV driver device abstraction pci device, pci driver xenbus device, xenbus driver device discovery PCI Tree Xenstore device configuration PCI Config Space (IO/MMIO) Xenstore data flow DMA Ring Buffer Memory Ring Buffer shared memory N/A or IOMMU interrupt IOAPIC, MSI, MSI-X Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

44 PV driver vs. PCI driver PCI driver PV driver device abstraction pci device, pci driver xenbus device, xenbus driver device discovery PCI Tree Xenstore device configuration PCI Config Space (IO/MMIO) Xenstore data flow DMA Ring Buffer Memory Ring Buffer shared memory N/A or IOMMU Grant Table interrupt IOAPIC, MSI, MSI-X Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

device discovery PCI Tree Xenstore device configuration PCI Config Space (IO/MMIO) Xenstore data flow

45 PV driver vs. PCI driver PCI driver PV driver device abstraction pci device, pci driver xenbus device, xenbus driver device discovery PCI Tree Xenstore device configuration PCI Config Space (IO/MMIO) Xenstore data flow DMA Ring Buffer Memory Ring Buffer shared memory N/A or IOMMU Grant Table interrupt IOAPIC, MSI, MSI-X Event Channel Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

46 Xenstore/Xenbus xm / xl write VM config to xenstore: * device info * memory hotplug... xenstore monitor changes in xenstore with xenwatch xenbus monitor changes in xenstore with xenwatch Domain 0 xenbus Domain U Xen Hypervisor Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

Grant Table Domain 0 pfn 1024 xen-netback 3.

xen-netfront Domain 1 4. Can I map (copy) ref 19 to my memory space? 2.

Domain 0 can map or copy from this page 1.

Table for Domain 1 5. You are allowed to access ref 19.

47 Grant Table Domain 0 pfn 1024 xen-netback 3. Share ref 19 to domain 0 via xenstore or other ways Network Packets xen-netfront Domain 1 4. Can I map (copy) ref 19 to my memory space? 2. I want to share pfn 1024 as grant table reference 19 to Domain 0. Domain 0 can map or copy from this page 1. Pick up a free grant table reference 19 Grant Table for Domain 0 Grant Table for Domain 1 5. You are allowed to access ref 19. I will map or copy the data to your memory space Xen Hypervisor Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

48 I/O Ring Buffer Usually put grant ref (not data) in ring Grant ref of ring pages are shared via xenstore Usually one ring buffer for each device queue One or more pages for each ring Producer and Consumer (barrier) Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

49 Xen Paravirtual Networking Framework Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

50 VM Creation Workflow vm.cfg xm create xl create (libxc) XML-RPC via socket xend (libxc) Extract kernel and ramdisk from vdisk via pygrub for PVM Ask xen hypervisor to create a VM, initiate vcpu, p2m, etc. Write VM device info to xenstore xen hypervisor xenstore Boot PVM into protected mode Boot HVM/PVHVM into real mode via hvmloader DomU Guest Dom0 Watching at xenstore. Initiate device driver at frontend Watching at xenstore. Initiate device driver at backend xensore xenstore Ask userspace hotplug script to help configure backend udev on Dom0 Bridging vif to bridge or obtain major/minor number of VM disk image file hotplug script Synchronize with each other via xenstore and finish! Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

51 Selected Xen Projects COLO - Coarse Grain Lock Stepping Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

52 Selected Xen Projects COLO - Coarse Grain Lock Stepping LivePatch Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

53 Selected Xen Projects COLO - Coarse Grain Lock Stepping LivePatch Stealthy monitoring with Xen altp2m Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

54 Selected Xen Projects COLO - Coarse Grain Lock Stepping LivePatch Stealthy monitoring with Xen altp2m Real-Time-Deferrable-Server(RTDS) CPU Scheduler Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

55 Selected Xen Projects COLO - Coarse Grain Lock Stepping LivePatch Stealthy monitoring with Xen altp2m Real-Time-Deferrable-Server(RTDS) CPU Scheduler Windows PV Receive Side Scaling Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

Windows PV Receive Side Scaling More at Xen Summit and xen-devel Dongli

56 Selected Xen Projects COLO - Coarse Grain Lock Stepping LivePatch Stealthy monitoring with Xen altp2m Real-Time-Deferrable-Server(RTDS) CPU Scheduler Windows PV Receive Side Scaling More at Xen Summit and xen-devel Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

57 Reference Publications Xen and the art of virtualization. Paul Barham, Boris Dragovic, Keir Fraser, Steven Hand, Tim Harris, Alex Ho, Rolf Neugebauer, Ian Pratt, and Andrew Warfield. SOSP 2003 The Definitive Guide to the Xen Hypervisor. David Chisnall Intel 64 and IA-32 Architectures Software Developer Manuals Various system & security research paper and presentation Miscellaneous Xen Project Developer Summit Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

58 Take-Home Message What is virtualization Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

59 Take-Home Message What is virtualization Paravirtualization and Hardware-assisted Virtualization Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

60 Take-Home Message What is virtualization Paravirtualization and Hardware-assisted Virtualization Xen vs. KVM Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

61 Take-Home Message What is virtualization Paravirtualization and Hardware-assisted Virtualization Xen vs. KVM Grant Table, Event Channel, Paravirtual Drivers Dongli Zhang (Oracle) Xen is not just paravirtualization December 16, / 30

Introduction to Oracle VM (Xen) Networking

Introduction to Oracle VM (Xen) Networking Dongli Zhang Oracle Asia Research and Development Centers (Beijing) dongli.zhang@oracle.com May 30, 2017 Dongli Zhang (Oracle) Introduction to Oracle VM (Xen)