Large Systems: Design + Implementation: Virtualization. Image (c) Facebook

Transcription

1 Large Systems: Design + Implementation: Image (c) Facebook Virtualization

2 Virtualization What is Virtualization "a technique for hiding the physical characteristics of computing resources from the way in which other systems, applications, or end users interact with those resources." Virtualization is the process of making things more abstract in order to make them easier to use. 2

3 Examples? 3

4 Storage virtualization Files Linear sequence of bytes Instead of blocks on a disk (or magnetic particles) Disk partitioning LBA RAID - redundant array of independent disks Logical Volume management Combines disks and partitions into logical disks. 4

5 Network virtualization VLAN Channel bonding multiple links combined offered a single, higher-bandwidth link Computer clusters Multiple logical networks on same physical wires multiple discrete computers into larger metacomputers e.g. Hadoop Virtual NICs and bridges for VM communication 5

6 Resource virtualization Multiprogramming Each process thinks it has CPU to itself Virtual memory Present linear address space composed of non-consecutive blocks of: Physical memory Disk space 6

7 7 Virtual Machines

8 Credits Slides largely based on: Virtual Machines: Versatile Platforms for Systems and Processes James E. Smith Ravi Nair Morgan Kaufmann Publishers 2005 Or see: DS3, 3.2 8

9 Why Virtual Machines? LS context: Basis for cloud computing Resource utilization Spin up capacity on demand Server consolidation Facilitate maintenance 9

10 Why Virtual Machines? Spin up capacity on demand Isolate applications in separate VMs Sandbox applications for security Support different OSes concurrently Legacy applications on legacy OSes Application testing using VMs with known state Testing OS upgrades, training OS development 10

11 12 Computer Architecture Recap

12 13 Instruction Set Architecture Software ISA Hardware E.g. x86_64

13 14 User + System ISA Application Programs System ISA Operating System Hardware User ISA

14 15 User ISA = For doing computations Simple Memory Instructions Move data from memory to registers and v.v. Integer Instructions Floating-Point Instructions Branch instructions Jump to address Jump to address if... Part of SHA1 in assembly: addl movl xorl andl xorl %esi, %e; %c, %esi; %d, %esi; %b, %esi; %d, %esi;

15 System ISA (1/2) = Management of system resources System Resources: Management: Main memory Storage Other I/O devices Fair allocation between user programs Prevent concurrent/unauthorized access Role of the Operating System 16

16 17 System ISA (2/2) OS requires special privileges over user programs OS runs in CPU Kernel mode Apps run in CPU User Mode User Mode x86: Implemented via 2 privilege levels / rings Kernel Mode Ring

17 System ISA Instructions Processor Management Memory Management Manage page table, TLB: virtual memory I/O Management Change to user mode + run application Timer interrupt gives control back to OS load and store to/from device Traps Change to kernel mode from application On purpose (system call) or on exception 18

18 19 System Call Instruction Application Programs Operating System System call Handler I/O instructions Hardware Disk System call: read(file) E.g. syscall on x86 or svc on ARM.

19 System Call = Mechanism for User Mode code to request services from Kernel Mode What services? Read/write to files and devices Create processes i.e. use Operating System abstractions: File abstraction for storing blocks on disk Process abstraction for running different code in parallel 20

20 21 System Call Interface (1/2) Application Programs System calls Operating System ISA Hardware

21 System Call Interface (2/2) Read/write files or devices: CreateFile(...) ReadFile(...) WriteFile(...) SetConsoleMode(...) Manipulate processes open(filename, ) read(fd,data, ) write(fd,data, ) ioctl() fork() exit() More... CreateProcess( ) ExitProcess( ) 22

22 23 System calls via libraries Application Programs Libraries System calls Operating System Hardware E.g. libc

23 Architecture Model Application Programs Run Kernel Mode Libraries System calls Operating System Interrupts, Traps, faults Privileged instructions Hardware 24 User Mode

24 25 Class 1 Recap

25 Recap 1 What is Distribution Transparency? What are the 2 main problems in achieving distribution transparency? How much time does it take to send a packet Over a LAN? Over the Atlantic? What are the 5 Scaling Techniques? 26

26 Recap 2 What is the user part of an ISA? What is the system part of an ISA? What functionality do they provide? What is virtual memory? 27

27 28 Recap 3 Application Programs 1 4 Libraries Operating System 3 2 Hardware Arrows? What runs in User / Kernel Mode?

28 29 Virtual Machines

29 Virtualize the machine? What is the machine? Machine is defined by an interface 3 interfaces that can be virtualized: 1. Instruction Set Architecture (ISA) 2. Application Binary Interface (ABI) 3. Application Programming Interface (API) 30

30 31 Interface 1: ISA Application Programs Libraries System calls Operating System ISA Hardware Virtualize a complete machine, that can run an OS, supporting multiple processes = System VM

31 32 Interface 2: ABI Application Programs Libraries System calls Operating System ABI = Hardware Virtualize the environment of a single process (binary) = Process VM System calls + user ISA

32 33 Interface 3: API Application Programs Libraries System calls Operating System API = Hardware Virtualization provides same interface at programming language (source) level. Libraries + user ISA

33 Example Virtualizing ISA Support a machine's complete ISA VM/370 Xen* KVM* 34

34 Example Virtualizing ABI Run binaries unmodified on different platform Common: Run Win32-x86 binaries on: Unix-x86 (Wine) Solaris-SPARC (Sun WABI) Win32-Alpha (Digital FX!32) 35

35 Example Virtualizing API Recompile applications from source Runs on any platform with same API E.g. Linux-x86 and Linux-ARM (Assuming platform-independent code) 36

36 VM Implementations Application Programs Application Programs Libraries Guest OS Virtual Machine Monitor Hardware (a) System VM Virtual Machine Monitor Host OS Hardware (b) Process VM 37

37 What ISA? Same or different Same: Run Win32-x86 on Linux-x86 Diff: Run Linux-ARM on Win32-x86 Application Programs Libraries OS Source ISA Virtual Machine Monitor Hardware Target ISA 38

38 39 Taxonomy Process VMs System VMs Same ISA Different ISA Same ISA Different ISA Multiprogrammed Systems Emulators/ Translators Classic-System VMs Whole-System VMs High-level Language VMs Hosted VMs Codesigned VMs

39 Example: Windows Multiprogramming Win32 native Win32 native 40 Process Win32 API Windows OS One CPU, illusion of processes running in parallel X86 Hardware Process VM / Same ISA

40 41 Example: Android Java High-level Language VM (HLL-VM) Java VM + Libs Linux OS ARM Hardware Different ISA: Java vs. ARM Process VM / Diff ISA

41 42 Example: Android Emulation Java Java VM Win32 native Linux OS WholeSystem VM ARM VM Runtime Process Win32 API Windows OS X86 Hardware System VM / Diff ISA

42 43 Example: Android Emulation on Xen Java Java VM Linux OS ARM VM Runtime Win32 Native Linux Native Windows OS Xen domain Linux Native Linux OS Xen Hypervisor X86 Hardware = Hosted VM

43 44 Example: VMWare Applications VMApp Guest OS VM Driver VMMonitor Host OS X86 Hardware = Hosted VM System VM / Same ISA

44 45 Example: AS/400 Application Programs Higher level ISA OS Source ISA Virtual Machine Monitor Allow evolution of hardware ISA Hardware Target ISA = Co-designed VM System VM / Diff ISA

45 46 Taxonomy Examples Process VMs Same ISA UNIX Multiprogrammed Systems Different ISA Wine, WABI Emulators/ Translators Java VM, MS CLR High-level Language VMs System VMs Same ISA Different ISA VM/370 ARM VM runtime Classic-System VMs Whole-System VMs VMware, Xen, Docker* Hosted VMs AS/400 Codesigned VMs

46 47 Crossword!

47 49 Implementing Virtual Machines with Different ISAs

48 VM implementation: Emulation Emulation = implement interface of one system on another system with a different interface Example: x86 instruction addl %edx,4(%eax) Emulated via PowerPC instructions: lwz addi lwzx lwz add stw r4,0(r1) r5,r4,4 r5,r2,r5 r4,12(r1) r5,r4,r5 r5,12(r1) 50

49 LS 2018 The slides on how to handle different ISAs are not part of the exam. 51

50 52 Emulation Model e Sa Sb Source Executing instruction e changes state of Source machine from Sa to Sb

51 53 Emulation Model e Sa Sb Source S'b Target e' S'a Real machine has corresponding state S'a Performs e by means of instruction(s) e'

52 Recall: Registers + Memory r0 r1 r2 Cache rn PC CPU MEM 54

53 55 Example: Change registers e CPU MEM CPU MEM e' CPU MEM CPU MEM Source Target

54 56 Example: Change registers e CPU MEM CPU MEM e' CPU MEM CPU MEM CPU state of Source may be kept in Target memory, not registers! Source Target

55 Emulation Performance + Methods Can be slow because of mapping Source to Target! Range of emulation methods: Interpretation: Interpretation Binary translation Decode a single source instruction and execute using target instructions Binary translation: Translate a block of source instructions once and reuse 57

56 Interpretation Source instruction is a series of bytes Different formats RISC: clean and simple CISC: complex with legacy Non-hardware: Java bytecodes Complexity of format influences interpretation performance! 58

57 59 Example Formats x86: Prefixes Opcode 0-4 bytes Opcode ModR/M SIB optional optional optional Displace- Immediate ment 0,1,2,4 0,1,2,4 bytes Software developer's manual: 3796 pages! Java: Opcode Index Opcode Index1 Index2 Opcode Data1 Data2 Java VM Specification: 604 pages

58 60 x86 Format Prefixes 0-4 bytes Opcode Opcode ModR/M SIB optional optional optional Displace- Immediate ment 0,1,2,4 0,1,2,4 bytes Prefixes: Repetition for strings, overrides for address and operand sizes ModR/M: addressing mode and which register SIB: base register, index register, index scale Displacement: offset to be added to address Immediate: variable length operand

59 Binary Translation Per-instruction interpretation slow Alternative: Especially when complex Translate blocks of source instructions once Reuse cf. Just-in-Time compilers Hard 61

60 Performance Tradeoff E(n) = time needed to execute an instruction n times Formula: E(n) = S + n*t S = startup time T = time required per emulation of the instruction Interpretation: S low, T high Binary translation S high, T low 62

61 63 Performance Tradeoff Interpretation E Binary Translation n In practice: automatic profiling, often used code is binary translated

62 OS Emulation For Process VMs have to emulate whole ABI User ISA System call interface System call instructions (e.g. sysenter) emulated Translate call from Source OS to Target OS Same OS: straightforward to hard Diff OS: straightfoward to impossible No guarantees that Target OS has same features as Source! E.g. fsync() 64

63 65 Implementing Virtual Machines with the Same ISA

64 66 Same ISA VMs Emulation needed for different ISA VMs For same ISA: Theoretically, source instructions can be executed directly on target Fastest Application Programs Does this work for all Libraries instructions? Guest OS Virtual Machine Monitor Hardware

65 Same ISA VMs (cont'd) No: System ISA instructions need to be controlled When? System VMs with a guest OS How? Guest OS runs in CPU User Mode System ISA instructions called in User Mode activate Kernel Mode i.e., cause a Trap VMM in Kernel Mode then emulates system instruction 67

66 Example: App Scheduling Applications User Mode Run Kernel Mode Operating System Interrupts, Traps, faults Privileged instructions Hardware 68

67 Example: App Scheduling Applications User Mode 2. Run app in User Mode Kernel Mode Operating System 3. Interrupt 1. Set interval timer Hardware 69

68 Example: App Scheduling Applications 4. Run app in User Mode User Mode Operating System 3. Run OS in User Mode 1. Set interval timer t Kernel Mode Virtual Machine Monitor 5. Interrupt 2. Set interval timer t' Hardware 70

69 Example: App Scheduling 71 Applications 4. Run app in User Mode User Mode Operating System 3. Run OS in User Mode 1. Set interval timer t Kernel Mode Virtual Machine Monitor 5. Interrupt 2. Set interval timer t' Hardware t = requested by OS t' = granted by VMM for fair scheduling of multiple VMs

70 Example: App Scheduling Applications Guest OS schedules Apps 4. Run app in User Mode Operating System 3. Run OS in User Mode 1. Set interval timer t Virtual Machine Monitor VMM schedules VMs 5. Interrupt 2. Set interval timer t' Hardware 72

71 x86 Same ISA Problems (1/3) Normal: Apps VM: OS no longer in kernel mode! OS + Apps OS Ring 0: Kernel Mode VMM OS Ring 3: User Mode 73

72 x86 Same ISA Problems (2/3) In x86, not all system instructions in User Mode activate Kernel Mode! When Guest OS runs in User Mode, not all system instruction calls observed by VMM Old solution: patch all binary code! Replace these critical instructions with explicit traps to Kernel Mode 74

73 75 x86 Same ISA Problems (3/3) New solution: Intel VT-x Allows Guest OS to run in Kernel Mode (Ring 0) Shared resources still controlled by VMM Using extra mode: VMX VMX Root for VMM VMX Non-root for Guest OS Ring -1 Also hardware support for VM context switch Apps OS OS VMM

74 77 Native and Hosted VMs Apps Apps Guest OS Guest OS VMM VMM Hardware (a) Native VM Host OS Hardware (b) Dual-Mode Hosted VM User Mode Kernel Mode

75 VMWare Workstation Install on top of existing host OS Easy to use Can use myriad of device drivers available in host OS 78

76 VMWare Architecture Applications VMApp Guest OS VM Driver VMMonitor Host OS X86 Hardware 79

77 VMWare Workstation Adds 3 components VMMonitor VMApp VMM in Kernel Mode, alongside Host OS User Process for translating VMM requests into system calls to host OS VMDriver Extension of the host OS Support switching between Host OS and VMM Enable VMM VMApp communication 80

78 81 VMWare I/O Applications VMApp Guest OS VMMonitor Device Driver Direct support, e.g. disk access via IDE Use host support, e.g. CD, sound, serial port VM Driver Device Driver Host OS X86 Hardware Device Driver

79 82 VMWare: New Capabilities Applications VMApp Guest OS VMMonitor VM Driver Device Driver Host OS X86 Hardware Device Driver COW Golden Image

80 83 Virtualization II Recap

81 Recap: App Scheduling Applications Operating System Virtual Machine Monitor Hardware 84

82 Recap 2 What is Intel VT-x? What problem does it solve? 85

83 86 Recap 3 Applications VMApp Guest OS VMMonitor VM Driver Device Driver Host OS X86 Hardware Device Driver COW Golden Image

84 87 Operating System Support for Virtualization

85 Native, Hosted, Paravirtualized VMs Apps Apps Apps Guest OS Guest OS VMM Guest OS Host OS VMM VMM Hardware Hardware Hardware Modify the Guest OS! 88

86 Paravirtualization System VMs can be faster when Guest OS can be modified for virtualization Showcased in Xen Project Modified Linux Windows XP Near native performance! 89

87 90 Paravirtualizing I/O Operating System Operating System Real Device Driver PV Device Driver I/O instructions Reverse Device Driver Hypercalls Network Handler Network Handler VMM Hardware VMM Hardware

88 Xen Evolution Problems: only open-source OSes can be modified Xen implementation tricks not on x86-64 New approach: Start from Full virtualization with Hardware Support (e.g. VT-x) Apply Paravirtualization in areas where speed can be gained: 1. Disk and network I/O 2. Interrupts and timers 3. Emulated motherboard, legacy boot 4. Privileged instructions, page tables 91

89 Xen Mode: HVM Source: Lars Kurth,

90 Xen Mode: PV 93

91 Xen Mode: HVM + PV Drivers 94

92 Xen Mode: PVHVM Drivers 95

93 Xen Mode: PVH 96

94 97 KVM KVM P

95 98 Xen Architecture Toolstack Applications Applications Domain 0 Host OS Drivers Guest OS (Modified) Guest OS (Modified) PV front PV front Xen Hypervisor virtual x86 Scheduler CPU X86 Hardware MMU Timers DomU = Unprivil eged

96 99 Xen Architecture Toolstack Applications Applications Domain 0 Host OS Guest OS (Modified) Guest OS (Modified) Driver MUX PV front PV front Xen Hypervisor virtual x86 Scheduler CPU X86 Hardware MMU Timers Domain

97 100 Xen Virtual Devices Toolstack Applications Domain 0 Guest OS Host OS vif6.0 eno1 eth0 xenbr0 Xen Hypervisor virtual x86 Scheduler CPU X86 Hardware MMU Timers

98 101 New: Xen PV Calls Toolstack Applications Domain 0 socket() Guest OS Host OS eno1 socket() backend frontend Xen Hypervisor virtual x86 Scheduler CPU X86 Hardware MMU Timers

99 Operating-System Level Virtualization In between System VM and Process VM Not System VM: Not Process VM: Cannot choose OS Multiple processes, not isolated As if multiple instances of the same OS are running on the same machine Example: Linux Containers cf. Docker 102

100 103 Linux Containers Applications OS View Namespaces Applications OS View Linux Host OS X86 Hardware Applications OS View CGroups

101 Linux Containers: Namespaces Linux kernel has a configuration + state Controlled via many files and outside input Idea: allow a configuration per process (group) E.g. for process A the hostname is X, for process B the hostname is Y cf. chroot Now: configuration is set of 6 namespaces Source: Rami Rosen, Linux Kernel Networking, APress. Michael Kerrisk, 104

102 6 Namespaces of Linux uts (hostname) mnt (mount points, filesystems) pid (processes) user (UIDs) ipc (inter process communication IDs) net (network stack) (plans to add more) 105

103 106 UTS Namespace (1/3) UNIX time sharing?! Contains 6 strings: sysname Operating system name (e.g., "Linux") nodename Name within "some implementation -defined network" release OS release (e.g., "2.6.28") version OS version machine Hardware identifier domainname NIS or YP domain name i.e., control the names of the container Source: uname(2)

104 UTS Namespaces (2/3) The old implementation of gethostname(): asmlinkage long sys_gethostname(char user *name, int len) {... if (copy_to_user(name, system_utsname.nodename, I)) errno = -EFAULT;... } system_utsname is a global variable 107

105 UTS Namespaces (3/3) The new implementation of gethostname(): static inline struct new_utsname *utsname(void) { return &current->nsproxy->uts_ns->name; } SYSCALL_DEFINE2(gethostname, char user *, name, int, len) { struct new_utsname *u;... u = utsname(); if (copy_to_user(name, u->nodename, i)) errno = -EFAULT; 108

106 109 MNT Namespace View of which filesystems are mounted New mounts only visible in current mnt namespace Unless special flags are used: mount make-shared / (root) /mnt sdb1 / (root) /home sda3 /mnt mnt ns1 /home sdc1 mnt ns2

107 PID Namespace Processes in different PID namespaces can have the same process ID. When creating the first process in a new namespace, its PID is 1. Hierarchy of PID namespaces: PIDs visible to parent namespace Nested upto 32 levels deep 110

108 User namespace New namespace = new set of User IDs and Group IDs Existing UIDs are mapped into new space E.g. user os3 has user ID 1023, group students2017 has group E.g. UID 1000 becomes UID 0 in new space First process in the new space has root Only for namespaces inside the new space! Create container: 1. Create new user namespace 2. Create new UTS, MNT, PID, etc. namespaces from that namespace Outside: permissions of parent UID 111

109 NET Namespace (1/3) A network namespace is logically another copy of the network stack: own routes, own firewall rules, own network devices. A network device belongs to exactly 1 network namespace A socket belongs to exactly 1 network namespace 112

110 NET Namespace (2/3) The initial network namespace includes: loopback device all physical devices, networking tables, etc. New network namespace includes only the loopback device Real devices can be moved into NS Virtual devices can be added

111 NET Namespace (3/3) Control via ip netns command And e.g. /etc/netns/<nsname>/hosts 114

112 Containers via namespaces Create a container: 1. Create a user namespace 2. Create a PID and UTS namespace inside 3. Create a MNT namespace to get your own filesystem 4. Mount container disk image 5. Create NET namespace, add virtual devices 6. Connect virtual devices to real network via e.g. virtual bridges 115

113 CGroups Namespaces can give groups of processes: Same view of the OS Illusion there are no other groups Control Groups is a mechanism for resource management for groups of processes: Set limits, e.g. on memory usage (main + FS cache) Set priorities (CPU or disk bandwidth) Accounting Checkpointing E.g. stop set of processes for resuming later 116

114 117 Episode MMXVI: The Empire Strikes Back

115 Virtualization on Windows Hypervisor: Hyper V Containers: Windows Server Containers Hyper V Containers Docker integration 119

116 Windows (cont'd) 120 Source: Wikipedia

117 Windows (cont') Windows Server Containers: 121 provide application isolation through process and namespace isolation technology. shares a kernel with the container host and all containers running on the host. Hyper-V Containers: like Server Containers but expand on the isolation by running each container in a highly optimized virtual machine. kernel of the container host is not shared with the Hyper-V Containers. Source: ew

118 Windows Container Types 122 Source: McCabe, Friis - Introduction to Windows Containers

119 Fedora Container Coloring Book 123

120 124 Virtualization III Recap

121 Full virtualization How can we improve performance? What are the implications of the solutions? 125

122 126 Recap 3 Application Programs Libraries Operating System Hardware In OS-level virtualization, where is the VMM? What are the risks of OS-level virtualization? What are the benefits?

123 130 VMs without Operating Systems

124 Unikernels Optimize VM for one application: Strip unused parts of OS and libraries Link directly with application Advantages: No context switching: No User-to-Kernel and vv. Less memory usage Apply application-specific OS optimizations More secure? More info see: 131

125 132 Unikernels Apps User Mode Kernel Mode Guest OS Apps VMM Guest + Host OS Hardware Hardware App Essential Guest OS VMM Hardware (a) Paravirtualization (b) Containers (c) Unikernels Unikernel

126 Video The Next Generation Cloud: Unleashing the Power of the Unikernel Russell Pavlicek, Xen Project Evangelist Large Installation System Administration Conference (LISA) entation/pavlicek (Thanks to Niels) 133