Virtual Machine Security

Size: px

Start display at page:

Download "Virtual Machine Security"

Egbert Lloyd
6 years ago
Views:

1 Virtual Machine Security CSE443 - Spring 2012 Introduction to Computer and Network Security Professor Jaeger 1

2 Operating System Quandary Q: What is the primary goal of system security? OS enables multiple users/programs to share resources on a physical device Q: What happens when we try to enforce Mandatory Access Control policies on UNIX systems Think SELinux policies What can we to do to simplify? 2

Virtual Machines Instead of using system software to enable sharing, use system software to enable isolation Virtualization a technique for hiding the physical characteristics of computing

3 Virtual Machines Instead of using system software to enable sharing, use system software to enable isolation Virtualization a technique for hiding the physical characteristics of computing resources from the way in which others systems, applications, and end users interact with those resources Virtual Machines Single physical resource can appear as multiple logical resources 3

4 Virtual Machine Architectures Full system simulation CPU can be simulated Paravirtualization (Xen) VM has a special API Requires OS changes Native virtualization (VMWare) Simulate enough HW to run OS OS is for same CPU Application virtualization (JVM) Application API 4

5 Virtual Machine Types Type I Lowest layer of software is VMM E.g., Xen, VAX VMM, etc. Type II Runs on a host operating system E.g., VMWare, JVM, etc. Q: What are the trust model issues with Type II compared to Type I? 5

6 Virtual Machine Types Type 2 VMM! Hybrid VMM! Type 1 VMM! App! App! Guest OS 1! Guest OS 2! App! App! App! App! VMM! Guest OS 1! Guest OS 2! Guest OS 1! Guest OS 2! Host OS! Host OS! VMM! VMM! Hardware! Hardware! Hardware! JVM! CLR! VMware Workstation! MS Virtual Server! KVM! VMware ESX! Xen! MS Hyper-V! 6

7 VM Security Isolation of VM computing Like a separate machine Partitioned Resources VM Guest OS VM Guest OS Device Requests Virtual Machine Monitor Physical Device Controls 7

8 VAX VMM System First system design to examine virtualization in the context of information flow security! Virtualization mechanisms necessary to implement a reference validation mechanism that satisfies the reference monitor concept! Assure system design and implementation to the highest level A1 level per the Orange Book! Control all system information flows according to MLS and Biba integrity policies (modulo exceptions in privileges )! Also, covert channel countermeasures were produced, approximating noninterference! System was piloted, but not released commercially! 8

9 VAX VMM System Key design tasks of secure VMM! Virtualize processor! All security-sensitive instructions must be mediated by VMM! VMM protection ring! VMM must be deployed in a more privileged protection ring than the VMs! I/O emulation! Privileged I/O tasks must be executed in VMM or trusted VM! Self-virtualizable! OS must not detect when running on a VMM (or VMMs)! 9

10 Virtualizing Instructions Security-Sensitive Instructions! Instructions that read or modify privileged system state! Privileged Instructions! Instructions that cause a trap when executed in a nonprivileged ring! All security-sensitive instructions must be privileged to enable the VMM to manage privileged system state (rather than individual VMs)! This requirement was not met by VAX hardware nor x86 originally! 10

11 I/O Emulation Access to devices is expected by each operating system, but this access is security-sensitive! Thus, devices are virtualized! Access to devices must be directed to the party with physical device access! Memory-mapped I/O uses unprivileged instructions! VAX VMM adds a layer of indirection! I/O interface that causes a trap! OS must be modified to use that interface (paravirtualize)! 11

12 Other Issues Driver management! In VAX VMM, all drivers were in the VMM kernel! This was for assurance, but added code to VMM! Drivers are outside the VMM in most systems! DMA! Devices can use this mechanism to write to physical memory, but under guidance of untrusted VMs! VAX VMM trusted drivers, but not practical today! Performance E.g., page table lookups! 12

13 VAX VMM System Applications (Top Secret) Applications (Secret) Applications (Unclassified) Ultrix OS VMS OS VMS OS VMM Security Kernel Memory Device Disk Device Print Device Display Device... 13

14 NetTop Isolated networks of VMs Alternative to air gap security VM: Secret VM: Public VM: Secret VM: Public Guest OS Guest OS Guest OS Guest OS VMWare MLS SELinux Host OS VMWare MLS SELinux Host OS 14

15 Xen Privileged VM VM: DomU Guest OS Partitioned Resources VM Services Dom 0 Host OS Drivers VM: DomU Guest OS Device Requests Xen Hypervisor 15

16 Xen shype Controlled information flows among VMs VM: DomU Guest OS Partitioned Resources VM Services Dom 0 Host OS Drivers VM: DomU Guest OS Device Requests Xen Hypervisor Ref Mon 16

17 Xen shype Policies Type Enforcement over VM communications VM labels are subjects VM labels are objects How do VMs communicate in Xen? Grant tables: pass pages between VMs Event channels: notifications (e.g., when to pass pages) shype controls these Q: What about VM communication across systems? 17

18 Xen Security Modules Comprehensive Reference Monitor interface for Xen Based on LSM ideas Includes about 57 hooks (more expected) Supports shype hooks Plus, hooks for VM management, resource partitioning Another aim: Decompose domain 0 Specialize kernel for privileged operations E.g., Remove drivers 18

19 IOMMU Role in the System RAM! System! Software! Penn State Systems and Internet Infrastructure Security Lab Page 19

20 IOMMU Role in the System RAM! System! Software! Penn State Systems and Internet Infrastructure Security Lab Page 20

21 IOMMU Role in the System MMU! RAM! System! Software! control Penn State Systems and Internet Infrastructure Security Lab Page 21

22 IOMMU Role in the System Peripheral! MMU! RAM! Peripheral! System! Software! Peripheral! control Penn State Systems and Internet Infrastructure Security Lab Page 22

23 IOMMU Role in the System Peripheral! MMU! RAM! IOMMU! Peripheral! System! Software! Peripheral! control 23

24 I/O Device Assignment Process! Process! VM Guest 1! OS! VM 1! Peripheral! VM Guest 2! VM Guest 3! MMU! RAM! IOMMU! Peripheral! Parent! VM 0! Hypervisor! Peripheral! control! Penn State Systems and Internet Infrastructure Security Lab Page 24

25 VM Security Status Aim is simplicity Are we achieving this? Do we care what happens in the VMs? When might we care? Trusted computing base How does this compare to traditional OS? 25

26 Virtual Machine Threats How does the insertion of a virtual machine layer change the threats against the system? CSE443 Introduction to Computer and Network Security - Spring Professor Jaeger Page 26

27 Virtual Machine Rootkit Rootkit Malicious software installed by an attacker on a system Enable it to run on each boot OS Rootkits Kernel module, signal handler,... When the kernel is booted, the module is installed and intercepts user process requests, interrupts, etc. E.g., keylogger VM Rootkit Research project from Michigan and Microsoft If security service runs in VM, then a rootkit in VMM can evade security E.g., Can continue to run even if the system appears to be off CSE443 Introduction to Computer and Network Security - Spring Professor Jaeger Page 27

28 Take Away VM systems focus on isolation Enable reuse, but limited by security requirements Enable limited communication The policies are not trivial, but refer to coarser-grained objects 28

CSE543 - Computer and Network Security Module: Virtualization

CSE543 - Computer and Network Security Module: Virtualization Professor Trent Jaeger CSE543 - Introduction to Computer and Network Security 1 Operating System Quandary Q: What is the primary goal of system