VirtuOS: an operating system with kernel virtualization

Transcription

1 VirtOS: an operating system with kernel virtalization Rslan Nikolaev, Godmar Back SOSP '13 Proceedings of the Twenty-Forth ACM Symposim on Oper ating Systems Principles 이영석, 신현호, 박재완

2 Index Motivation Design & Implementation Evalation Conclsion

3 Motivation System critical data of crrent operating system resides in the kernel s address space where it cannot be directly accessed by applications. This isolation falls short of protecting the system from failing kernel components. Failre of jst one kernel component generally cases the entire syste m to crash. device drivers 65~83% of all crashes ( in Windows XP )

4 Motivation Architectral approaches for increasing the reliability of kernel soft ware and redcing the impact of falts often rely on decomposition. Reliability? Decomposition

5 VirtOS s goal to explore opportnities for improved isolation of kernel components in virtalized containers withot significant compromises in performance.

6 Design & Implementation Microkernel Microkernel-based system design moves device drivers and other system critical code from the kernel into sep arate ser space processes. Bt they reqire carefl engineering to achieve good IP C performance. Extensive emlation interface layers mst be impleme nted Mach 1995 L Sawmill

7 Design & Implementation Virtal Machines also se hardware isolation to create strongly isolated domains in which to separate software components. to safely share a machine s resorces while maintaining application and kernel compatibility with existing systems.

8 Design & Implementation Microkernel Virtal Machines Dissimilar Motivations Bt the same larger design space

9 Design & Implementation

10 Design & Implementation Failre Model Primary Domain A single, primary domain is dedicated to core system tasks sch as process management, schedling, ser memory management, and IPC. Service Domain do not rn any ser processes other than for bootstrapping and to perform any necessary system management tasks related to a domain s fnction. is to handle reqests coming from the ser processes managed by the prima ry domain.

11 Design & Implementation System Call Design Remote system call system calls destined for a service domain Local system call directly handled by the primary domain A modified C library contains all necessary infrastrctre to transparently demltiplex local and remote system calls and forward remote calls to service domains.

12 Design & Implementation System Call Design Traditional system call implementations has Mode Switch : transition the processor from a less privileged ser mo de to kernel mode Cache polltion : Cache and TLB polltion cased by the different w orking sets of ser and kernel code. So this implementation was discarded, the Exception-less system calls is selected.

13 Design & Implementation System Call Design Exceptionless System Call( FlexSC: Flexible System Call, 2008 ) a ser-level library places system call reqests into a bffer that is sh ared with kernel worker threads that execte the system call on the t ask s behalf, withot a mode switch. Effective exceptionless system call handling assmes that kernel wor ker threads rn on different cores from the ser threads they serv e.

14 Design & Implementation System Call Design

15 Design & Implementation System Call Design impossible direct access in VirtOS The kernel worker threads reside in a different virtal machine. Or implementation addresses these differences which reqires the primary domain to commnicate essential inform ation abot rnning processes to the service domains

16 Design & Implementation System Call Design Each process create two sch shared areas for each service domain: (1) One area to hold the reqest qee for otstanding system call reqests (2) An area sed as a temporary bffer for system calls that transfer ser data.

17 Design & Implementation System Call Design Copy Strategy - the ser process copies data into or ot of a Temporary bffer of memory shared with the service domain. - A shared memory region is mapped in a continos virtal address space region in the ser program and in the service domain.

18 Design & Implementation Thread Management User-level Thread Schedling When a system call reqest is placed into a service domain s reqeset qe e, the issing ser-level thread incl des a pointer to its thread control b lock in the system call entry.

19 Design & Implementation Thread Management User-level Thread Schedling User-level Thread User-level Thread Ready Qee Ready Qee 1. isse a system call

20 Design & Implementation Thread Management User-level Thread Schedling Context Switch Case 1. other threads are ready to execte User-level Thread User-level Thread ready to execte Ready Qee Ready Qee If other ser-level threads are ready to execte, the crrent ser-level thread blocks and performs a low-overhead context switch to the next ready ser-level thread.

21 Design & Implementation Thread Management User-level Thread Schedling Spinning Case 2. there are no ready threads User-level Thread User-level Thread Ready Qee Ready Qee If there are no ready ser-level threads after a system call reqest is issed, a ser-level thread spins for a fixed amont of time. Otherwise, it blocks the nderlying kernel thread via a local system call

22 Design & Implementation Thread Management Worker Thread Schedling Create worker threads on demand as system call reqests are issed, by always maintain on spare worker thread per process. Service Domain Worker Thread Worker Thread Worker Thread Worker Thread Reqest qee Reqest qee

23 Design & Implementation Thread Management Worker Thread Schedling Once created, a worker thread remains dedicated to a particlar process. This fixed assignment allows s to set p the thread s process specific dat a strctres only once. Service Domain Worker Thread Worker Thread Worker Thread Worker Thread Reqest qee Reqest qee

24 Design & Implementation Thread Management Worker Thread Schedling All worker threads cooperate in checking for new reqests sing the following strategy. When a worker thread has completed servicing system call, it checks the req est qees of all other processes for incoming reqests and wakes p worker t hreads for any processes whose reqest qee has pending reqests. Finally, it checks its own process s reqest qee and handles any pending req ests. If no reqest is pending in any qee, the worker thread will contine to chec k those qees for a fixed spinning threshold. If the threshold is exceeded, the worker thread will block.

25 Design & Implementation Service Domain complete! Worker Thread Worker Thread Worker Thread Worker Thread Reqest qee Reqest qee

26 Design & Implementation Service Domain Worker Thread Worker Thread Worker Thread Worker Thread Reqest qee Reqest qee other process s reqest qee check if it has pending reqests

27 Design & Implementation Service Domain wake p Worker Thread Worker Thread Worker Thread Worker Thread Reqest qee Reqest qee

28 Design & Implementation Service Domain Worker Thread Worker Thread Worker Thread Worker Thread Reqest qee Reqest qee check its own reqest qee if pending reqest exists - If no reqest in qee, thread is spinning - If the threshold is exceeded, the worker thread will block.

29 Evalation System Configration

30 Evalation Overhead System Call Dispatch & Spinning For a single fcntl(2) call, at least 45 times iteration is needed for the system call to complete withot blocking. Withot blocking, the achieved throghpt was 0.7x that of the native case. If the process is blocked, the throghpt slowdowns p to 14x. They fond to need a mch larger spinning threshold (1000 iterations) to achi eve the best performance. They se the same vale for all benchmarks : 1000 iterations.

31 Evalation Overhead Copying writes 16MB in chnks of 32, 64, p to 2MB to a file created in a tmpfs filesystem

32 Evalation Overhead Process Coordination A microbenchmark that forks N concrrent processes, then waits f or all of them.

33 Evalation Performance Mltithreaded programs OLTP/SysBench to evalate the performance of VirtOS s network domains. a mysql server receives and responds to reqests, each comprising of 10 selection qeries with otpt ordering. FileIO/SysBench to evalate the performance of VirtOS s storage domain generates 128 files with 1GB of total data and perform random reads with a block size of 16KB.

34 Evalation Performance Mltithreaded programs OLTP/SysBench

35 Evalation Performance Mltithreaded programs FileIO/SysBench with and withot disk access

36 Evalation Failre Recovery VirtOS spports failre recovery for any falts occrring in service domains, inclding memory access violations, interrpt handling rotine failre, deadlocks.

37 Conclsion VirtOS is a falt-resilent operating system design which provides isolation for kernel components by rnning them in virtalized se rvice domains. VirtOS is the first system to se virtal machines for system call dispatch and to apply exceptionless commnication across virtal machines.

38 Corey : An Operat ing System for Ma nycores Silas Boyd-Wickizer, Haibo Chen, Rong Chen, Yandong Mao, Frans Kaashoek, Robert Mrris, Aleksey Pesterev, Lex Stein MingW Yeha Dai Yang Zhang, Zheng Zhang Silas Boyd-Wickizer et al., 2008 Proceedings of the 8th USENIX conference on operating systems design and implementation 이영석, 신현호, 박재완 38

39 Previos OS 39

40 Processor trends 40

41 Performance Scalability of a Mlti-core Web Server Length of code path may increase with nmber of cores Examples Waiting longer for spin locks Traversing larger data strctres Increases instrctions per cycle (IPC) In fact, IPC is decreasing 41

42 Overview Paper presents a techniqe allowing mlticore architectres to overcome memory access bottlenecks Key idea is that applications shold control sharing of memory and kernel resorces Make them private by defalt Let each application specify which resorces it want to share 42

43 Introdction Throghpt of microbenchmark actally decreases with nmber of cores Problem cased by cache coherence protocol Each iteration reslts in a cache miss Resolving the miss reqires access to a shared data strctre protected by spin locks Increasing the nmber of threads attempting to pdate the table introdces qeing delays Throghpt of the file descriptor dp and close microbenchmark on Linx 43

44 Why throghpt drops? Load fd_table data to C0 from L1 in 3 cycles. Load fd table data to C1 from C2 sl1 in 121 cycles AMD 16-core system Sixteen cores on for chips Each core has a 64-KB L1 512-KB L2 cache Each chip has a 2-MB shared L3 cache

45 The bottleneck is shared OS data strctres Contention on shared data strctres is costly: serialization moving data between caches Why does the OS need shared data strctres? OS semantics reqires it Simplifies resorce management An example Shared FD table is a bottleneck A lock serializes pdates to fd_table 45

46 Application spend time in kernel Even applications implemented with mlticore MapRedce spend time in kernel 30% of time spent in OS on 16 cores Fraction of time in OS increases with the nmber of cores OS becomes a bottleneck 46

47 Performance isses Linx spin locks Repeatedly access a shared lock variable MCS locks (Mellor-Crmmey and Scott, 1991) Process reqesting the lock inserts itself in a possibly empty qee Waiting processes do not interfere with each other Spinlocks are better at low contention rates Reqire three instrctions to acqire and release a lock MCS locks reqire fifteen instrctions MCS locks are mch better at higher contention rates Less synchronization overhead 47

48 Performance isses Time reqired to acqire and release a lock on a 16-coreAMD machine when varying nmber of cores contend for the lock. The two lines show Linx kernel spin locks and MCS locks (on Corey). A spin lock with one core takes abot 11 nanoseconds; an MCS lock ab ot 26 nanoseconds. 48

49 08 s practices for scaling OS Avoiding shared data strctres altogether Redesign kernel sbsystem Fine- grain locking Waith-free primitives RCU 49

50 Corey s soltion Applications don't always need to share all the data strctres that existing interfaces share We propose three interface changes shares, address ranges, kernel cores Implemented in Corey OS Partially implemented in Linx 50

51 New OS interfaces Shares control the kernel data sed to resolve application references. Address ranges control page tables and the kernel data sed to manage them. Kernel cores allow applications to dedicate cores to rnning particlar kernel fnctions. Improve scalability of some applications by avoiding kernel bottlenecks 51

52 Address ranges - The problem Two options for mltiprocessor Shared address space Private address space 52

53 Address ranges - The problem Two options for mltiprocessor Core 0 Core 1 Core 0 Core 1 mm_strct mm_strct mm_strct Stack 0 array Stack 1 Stack 0 array Stack 1 Shared address spaces Private address spaces 53

54 Address ranges - The problem Two options for mltiprocessor Core 0 pgtable Core 1 pgtable Core 0 Core 1 pgtable mm_strct mm_strct mm_strct Stack 0 array Stack 1 Stack 0 array Stack 1 Shared address spaces Private address spaces mm_strct : memory management strct pgtable : page table 54

55 Address ranges - The problem Neither option accrately represents how the application is sing kernel data strctres: shared address spaces the mm_strct is global contention nnecessary for private memory private address spaces the mm_strct is private extra soft page falts, becase no PTE sharing 55

56 Corey s soltion : address ranges Address ranges provide benefits of both shared and private address spaces avoid contention for private memory share PTEs for shared memory Core 0 pglevel1 pglevel1 Core 1 pglevel0 ar_strct ar_strct ar_strct Stack 0 array Stack 1 56 ar_strct : address range strct

57 Kernel Cores - The problem Application code invokes a system call, kernel code is exected on the same core Uses kernel data strctres Acqires locks When the kernel data strctres are again accessed from a different core, reslt in cache invalidation 57

58 Corey s soltion : Kernel Cores Application can dedicate cores to kernel fnctions and data 58

59 Corey s soltion : Shares Allow applications to dynamically create lookp tables and decide how these tables are shared Applications specify when they need sharing, for example: shared FDs allocated in shared table private FDs allocated in private table Corey kernel ses shares for all lookp tables 59

60 Shares - Linx FD share example Cores maniplate FDs withot contending for kernel data strctres Core B fdtable 0 Core R FD goo fdtable 1 Core B FD goo Core R 1 fd2 = open( goo ); 3 fdtable1 = share_alloc(); 4 fd0 = open( foo, share1); 5 write(fd0, bf, 128, share1); 2 write(fd2, bf, 128); 60

61 System services cfork cfork(core_id) is an extension of UNIX fork() that creates a new process (pcore) on core core_id Application can specify mltiple levels of sharing between parent and child Defalt is copy-on-write Network Applications can decide to rn Mltiple network stacks A single shared network stack Bffer cache Shared bffer like reglar UNIX bffer cache Three modifications A lock-free tree allows mltiple cores to locate cached blocks w/o contention A write scheme tries to minimize contention 61 A scalable read/write lock

62 Applications MapRedce applications 62

63 Applications Inverted index with MapRedce Apple Banana Map Core 0 Core 1 <key, val> bckets Apple, 0 Banana, 1025 Core 0 Core 1 Apple, 0 Banana, 1025 Redce 63

64 Applications in Address Range Inverted index with MapRedce Core 0 pglevel1 pglevel1 Core 1 pglevel0 ar_strct ar_strct ar_strct Apple, 0 Banana, 1025 Apple, 0 Banana, 1025 Avoids contention when growing Dring Redce PTEs are shared the address space dring 64 Map

65 Implementation Corey rns on AMD Opteron and Intel Xeon processors Implementation simplified by sing 64-bit virtal address space Implementation Low- level- Corey objects lines C and 150 lines assembly Architectre specific fnctions and device drivers High-level- UNIX like environment lines of C/C++ Bffer cache, cfork, and TCP/IP stack interface as well as the Corey-specific gle for the Cibc C standard library, 1wlp, and the Streamflow dynamic memory allocator 65

66 Evalation - Experimental setp AMD 16-core system 64GB memory AMD hardware event conter Nmber of cache misses Average of latency of cache misses OS: Debian Kernel patched with perfctr to allow application access to hardware even conters 66

67 Evalation - Address ranges To be investigated: Contention costs of maniplating mappings for private memory Soft page-falt costs for memory that is sed on mltiple cores Expectation: Corey has low costs for both sitations Other system have low cost for only one type of sharing, bt not both 67

68 Evalation - Address ranges Two micro benchmark: Memclone : has each allocate its own 100MB array and modify each page of the array Mempass : Allocates a single 100MB array on one of the clones, toches each bffer page and passes it to the next core which repeats the process 68 Throghpt L3 cache misses

69 Evalation - Kernel Cores Two server configrations to be compared: Dedicated : It ses a kernel core for all network processing Polling : It ses a kernel core only to poll for packet notifications and transmit completions 69 Throghpt L3 cache misses

70 Evalation - Shares Two microbenchmark: Add a per-core segment to a global share with share_addobj and Remove segment from the share with share_delobj Same bt per core segment is added to a local share Throghpt L3 cache misses

71 Evalation - Applications - MapRedce Linx Corey Cores share a single address space Each core maps the memory segments holding intermediate reslts sing per-core shared address ranges 71 Corey and Linx performance Corey improvement over Linx

72 Discssion and Ftre Work Corey lacks many featres of commodify operation systems, sch as Linx, which inflences experimental reslts both positively and negatively 72

73 Optimizing Commnication Bottlenecks in Mltiprocessor Operating System Kernels If a process creates a thread with the CLONE_FILES flag, the new thread has its own space of file descriptors; this can redce sharing and the associated contention OpLog Design OpLog optimizations : Batching pdates, Absorbing pdates, Allocating logs 73

74 Tiled-MapRedce: Optimizing Resorce Usages of Data-parallel Applications on Mlticore with Tiling Mlticore Research Corey proposes three new abstractions (address ranges, shares and kernel cores), to scale a MapRedce application (i.e., Word Revert Index) rnning 74

75 The Mltikernel: A New OS Architectre for Scalable Mlticore Systems Shared address spaces the former is typically more efficient, however the latter may redce crossprocessor TLB invalidations 75

76 Factored Operating Systems (fos): The Case for a Scalable Operating System for Mlticores Related work work has been done to investigate operating systems for mlticore processors. One example is Corey which focses on allowing applications to direct how shared memory data is shared between cores 76

77 Efficient System-Enforced Deterministic Parallelism Race-free system namespaces This principle ensres that naming a resorce reveals no shared state information other than what the application itself provided. Since implicitly shared namespaces often case mltiprocessor contention, designing system APIs to avoid this implicit sharing may be synergistic with recent mlticore scalability work 77

78 Q & A 78