OS PMM - More space: if you give an OS more memory

Transcription

1 Speaker Notes Page 1 of 7 Transcendent Memory Update (XenSummit 2010) Agenda: In the first few minutes, I'm going to quickly review the motivation and review some background about how physical memory is used today in a physical system and in a virtualized system. This first part will be a condensed repeat of my presentation from last year's Xen Summit, so for those of you who didn't see that presentation, I may be zooming a bit too fast, but for those who did, I don't want you to go to sleep so bear with me for a few minutes and consider this a quick refresher course in Transcendent Memory concepts. Then I'll cover the considerable forward progress in tmem since last year's Xen Summit at Oracle. Finally, I'll show some very interesting performance analysis I've been doing and hopefully have some time for Q&A. Motivation: Many Xen users have consolidated their data centers, but find that their CPUs are still spending a lot of time idle. Sometimes this is because the real bottleneck is that there isn t enough RAM in their systems. One solution is to add more memory to all of their systems but that can be very expensive and we d like to first ensure that the memory we do have is being efficiently utilized, not wasted. But it's often not easy to recognize the symptoms of inefficiently utilized memory in a bare metal OS, and it's even harder in a virtualized system. Challenge: So before we try to solve the memory problem, let s try to clearly identify what it is we re trying to do. Our deceptively simple objective is to (read, rephrase, and wing it). Agenda - Overview: take a look at a few points on how physical memory is managed and utilized, first in an operating system, and then in a virtual machine monitor like Xen. OS PMM: first thing to note is that OS s are memory hogs. Why? Most OS s were written many years ago when memory was a scarce and expensive resource and every bit of memory had to be put to what the OS thinks is a good use. So as a result, OS PMM - More space: if you give an OS more memory OS PMM - Used up all space: it s going to essentially grow fat and use up whatever memory you give it. So it's not very easy to tell if an OS needs more memory or not and similarly it s not very easy to tell whether it's using the memory it does have efficiently or not. So let s dig a little deeper and see what an OS is really doing with its memory. OS PMM - What does it do: (just flash picture for a few seconds and move on) Transcendent Memory Update (Xen Summit 2010) -- Speaker Notes Page 1 of 7

2 Speaker Notes Page 2 of 7 Transcendent Memory Update (XenSummit 2010) OS PMM - Page Cache crystal ball: Most OS's use a lot of RAM for their page cache. What's that? Well, because disks are slow, an OS will often keep a copy of pages that its read from disk, so as to avoid having to read that page all over again. The OS is essentially trying to predict the future in order to make disk reads less frequent. It stores these pages in the page cache. And although it s far from perfect at predicting the future, the OS guesses right a lot of the time, which is why the page cache concept exists in all modern operating systems. If the OS attempts to predict the future and it does guess right, we ll call that a good page cache page, and we ll represent the concept of good page cache pages with this crystal ball. OK? Now I said the OS guesses right a lot of the time but that means it also guesses the future wrong a lot of the time. OS PMM - Page cache waste: so that means that a lot of the pages in the page cache have data that is NOT going to be used in the future and so is wasting valuable memory space. We ll represent wasted space with this symbol. Now if you watch over time how memory is being used, when an OS is really busy with a lot of applications running, the page cache is fairly small. OS PMM - Page cache waste: But when the workload gets light again, much or most of memory is used for the page cache. So whenever you see a pig, I want you to be thinking that at any given point in time some, maybe most, maybe nearly all of the memory used by an OS is wasted. Agenda - Overview VMM: OK, let s switch from the OS to Xen. VMM PMM: Xen reserves some memory for itself. And it reserves some memory for domain0. When a guest is started, a certain fixed amount of physical memory is specified in its configuration file. Xen partitions memory and any memory that s not used is left lying around in case you want to launch more guests. But unless you are truly a miser about your megabytes, there's almost always still a lot of memory left lying around. We re going to call that FALLOW memory. And that fallow memory is completely unused, completely wasted. Although there are times when you really do want to have fallow memory... VMM PMM - Migration: Migration is one of the coolest features of virtualization, but one of its constraints is unfortunately that every migrating guest temporarily needs its whole physical memory allocation both on the source machine and on the target machine so, for a short time, it is using double its physical memory allocation. So if you don't have a good amount of fallow memory in your data center at all times, migration will be limited or impossible. VMM PMM - Ballooning: Another widely used technique for managing memory is ballooning. A balloon driver allows the hypervisor to redistribute physical memory from Transcendent Memory Update (Xen Summit 2010) -- Speaker Notes Page 2 of 7

3 Speaker Notes Page 3 of 7 Transcendent Memory Update (XenSummit 2010) one guest to another, but it has some real issues. One is that it works great when you have fallow memory that you want to give to a guest OS. But for several reasons... VMM PMM - Ballooning issues: Ballooning doesn't work so great the other way around... if you want use ballooning to take memory away. First, ballooning isn t instantaneous. You can t just take memory away from a guest with ballooning, you have to ask for it. You often won t get it fast, and sometimes you may not get it at all. This is because, as we ve previously seen, every OS is always using all of its memory for something that it thinks is useful. Second, when the balloon driver wants some of that memory, the socalled donating OS has to decide what memory it needs the least. And that s going to be page cache pages. And as we ve seen, while some of those pages are really waste anyway and OK to throw away, some of them are actually valuable and throwing them away will lead to extra disk reads! Third, and worst, ballooning decisions made by Xen or domain0 have very little guidance on how much memory is safe to balloon or how fast it can be taken away. If the guess is wrong, bad things happen. VMM PMM Migration and Ballooning : And things get worse if you try to combine ballooning AND migration. Suppose, despite the problems I just mentioned, you do your best to use ballooning to dynamically distribute all RAM to your guests. Now, at any given time, there is probably not enough fallow memory for incoming migrations! Why IS hard! Summary: Now recall the problem we are trying to solve. We ve got a pretty good list of reasons why this is a hard problem (quickly go over list). So it should be fairly obvious that the situation we have today for managing physical memory isn t really working and we need to do something new. That leads us to... Agenda - Tmem Overview: Transcendent Memory! Now I ll do a very quick overview of what transcendent memory is all about, and then we'll be done with the review and can get into the new material. Tmem pool step 1: What we are going to do is surprisingly simple. Step 1, we are going to reclaim ALL wasted memory in the system from the hypervisor and from the guests and we re going to collect it in a pool and Tmem pool step 2: Step 2, we provide indirect access to that pool of memory, through a carefully crafted API, which is controlled by a set of restrictions imposed by the hypervisor. That s it. But the magic is in the characteristics of that API and the semantics of the restrictions imposed by the hypervisor. Tmem API: First, the API does require awareness by the OS and so requires guest kernel changes, or paravirtualization. But the changes required are surprisingly small. The API is Transcendent Memory Update (Xen Summit 2010) -- Speaker Notes Page 3 of 7

4 Speaker Notes Page 4 of 7 Transcendent Memory Update (XenSummit 2010) narrow in that the Linux interface is specified using a small handful of function pointers and these funnel down to a single Xen hypercall. There s a spec available today. It works primarily by synchronously copying whole pages between the guest and the hypervisor. We're already doing several different things with the API and, the API is extensible, if we find more uses in the future. Tmem four pool types: Transcendent memory, that huge pool of previously wasted memory, is dynamically subdivided into subpools. And when these subpools are created, they are given certain attributes. Ephemeral means that any data put into the pool may disappear at any time. This gives the hypervisor complete flexibility over whether the data in the pool stays around long enough so that it can be used if the guest needs it again. Or, for example, if the space is needed for an inbound migration, all of the data can be thrown away and all the memory space can be immediately used to accommodate the incoming guest. Persistent of course means the opposite. Any data put in a persistent pool will stay there, but ONLY until the guest that put it there dies. So it can t be used in place of a disk. There's also private and shared pools, but in the interest of time, I'm going to skip those for this presentation. Agenda: Back to the agenda, and now here's the new material so anybody who was bored by the review material can wake up now. Tmem update: As of earlier this month, tmem is now released in Xen It is disabled by default, but can be enabled with a Xen boot option. A lot of effort was put into a good locking strategy that provides a high level of concurrency so not only can multiple guests simultaneously access tmem, but different VCPUs in the same guest can also simultaenously access tmem. And there were some interesting challenges that I could talk a long time about, but you can ask me later. Tmem has complete save/restore support as well as support for live migration. This support is in the hypervisor and in the mainstream xend tools... I don't think it's in libxl yet. An interesting new feature is page dedup, and I'll talk about that more in a minute. The Linux kernel patch now provides support for the most frequently used filesystems and a nice clean interface to expose some important tmemrelated statistics. And tmem is starting to find its way into released Xen and Linux distros, including Oracle VM, which is Oracle's Xen-based virtualization solution. Tmem page deduplication: So what is this page deduplication stuff? First, it just got added to xen-unstable and will be back-ported to the 4.0 line and is enabled with another Xen boot option. If you have heard of "page sharing", it's a lot like that: Say you have fifty pages that have been put into tmem and their content is identical. Instead of taking up fifty pages of physical memory, you instead keep one copy in one physical page in memory and Transcendent Memory Update (Xen Summit 2010) -- Speaker Notes Page 4 of 7

5 Speaker Notes Page 5 of 7 Transcendent Memory Update (XenSummit 2010) have fifty pointers to it. It's completely transparent to guests that are using tmem. The implementation is fairly simple and differs dramatically from the other page sharing code that went into Xen in 4.0, in part because of tmem's concurrency requirements make it harder, but also because the way tmem works, no copy-on-write or swapping is required, so in many ways it is also much easier... adding dedup support to tmem was less than 500 lines of code. Dedup is currently restricted to only ephemeral pools for some fairly subtle reasons. Dedup can be combined with compression to get, as we will see, some pretty impressive results. We've also been experimenting with something we call "trailing zero elimination" which is kind of a poor-man's compression when there's a workload with a lot of small files being used that are zero-filled. Last, tmem provides a very extensive set of counters and statistics now including some for dedup, so we can measure how many duplicate pages are being shared and how much time is being used. All very useful stuff. Issue: Fragmentation: Every silver lining has a cloud, and for tmem, there's a fragmentation issue that Jan Beulich discovered shortly before 4.0 went out. Basically there's a few Xen operations that require internal data structures that must be dynamically allocated and that are larger than one page and, if the multi-page data structure allocation fails, the operations fail also. Well, any dynamic memory technique, even ballooning, will eventually churn through all the physical memory in the system and fragment it so that multi-page allocations will eventually fail. Tmem just does this very quickly, especially when aggressive ballooning is enabled. So, under certain circumstances, we ran into some of these fragmentation issues. We found a kind of hack-y workaround that we got into 4.0, and it seems to be holding up fairly well, but this is a problem that really needs to be fixed properly. Agenda: OK, I've got some interesting benchmark results to show and then we will finish up with any questions. Test workload: So, here's a test workload. We've got a two-core processor which has only 2GB of memory. We're going to launch four PV guests and to make them all it they each get 384MB of memory. Then we're going to pound on them all simultaneously with a parallel kernel compile workload. Measurement methodology: We're going to run that workload five times for each of several configurations, collect information from xentop and tmem, and record a few important statistics. Here's the configurations we are measuring. Transcendent Memory Update (Xen Summit 2010) -- Speaker Notes Page 5 of 7

6 Speaker Notes Page 6 of 7 Transcendent Memory Update (XenSummit 2010) Self-ballooning recap: Since four of those five configurations include self-ballooning, let me digress for a minute to review exactly what that is. Self-ballooning, as presented in Xen Summit 2008, is a feedback-directed ballooning technique which uses information from within a guest OS to dynamically resize the balloon, giving away physical memory when the guest OS is quiet and asking for it back when the guest OS has a workload that needs lots of memory. This is done aggressively so that a guest OS uses as little memory as it can; frequently so that it can respond to sudden changes in workload; independently of other guest OS's, and lots of parameters can be configured to adjust how self-ballooning works. On Linux, there's a value called "Committed-A-S" that is a reasonable guesstimate of how much memory is needed and we use that to provide the feedback, and the balloon driver itself ensures that not TOO much memory is ballooned away. At Xen Summit 2008, selfballooning was done with a userland daemon and the source for this has been in the Xen source tree for a couple of years now. But, for these tests, we have self-ballooning implemented in the kernel itself. Unchanged vs Self-ballooning (time graphs): So let's see how self-ballooning does by itself. On this workload, we get a small increase in wallclock time. This is the elapsed time from when the workload starts until its complete. Also an increase in total VCPU time... that's the sum of all the VCPU seconds reported by xentop, including dom0. Unchanged vs Self-ballooning (VBD graphs): How about disk activity? We're reporting here millions of sectors read and written as reported by xentop again. Here the VBD reads have gone up roughly 10% and writes are slightly up but essentially unchanged. Sigh, performance hit: What's the explanation for this performance hit? Well, what we're seeing is all those "good" page cache pages being refetched from the disk after being squeezed out of the guest's memory by self-ballooning. And although this workload doesn't show it because the load is relatively constant, if the guest OS needs memory NOW, you're going to get some swapping to disk, or maybe even some out-of-memory conditions. So that loss of performance is to be expected. In fact, if you go back and look at the Xen Summit 2008 paper, this performance degradation actually gets much worse under certain conditions. Self-ballooning AND tmem: Now if you think about it, those problems with selfballooning that cause a loss in performance are exactly those things that I told you tmem is good for! So what if we combine self-ballooning with tmem? Well, instead of copying pages to and from a disk, we are copying pages to and from transcendent memory. Will that help? Let's look at the results for the rest of the configurations... Transcendent Memory Update (Xen Summit 2010) -- Speaker Notes Page 6 of 7

7 Speaker Notes Page 7 of 7 Transcendent Memory Update (XenSummit 2010) Self AND tmem (time graphs): First wallclock time... tmem not only makes up for self-ballooning but is even faster than the unchanged OS. And when combined with dedup and compression, it really rocks. 5% to 8% faster time to completion. What about VCPU time? Well, there's a small hit for tmem by itself and a rather significant hit with dedup and compression enabled. How important is that? Well, as we've highlighted in the slide, although total VCPU time is higher, all we've really done is utilize the CPUs more effectively. We'll talk about that more in a second, but first let's look at disk stuff. Self AND tmem (VBD graphs): With tmem we see that not only do we recover from the hit in disk reads due to refaults, but we reduce it by 31% for tmem by itself and over 50% for tmem with dedup and compression. Again no noticeable change in writes. Why is tmem so good?: So this all looks pretty impressive. Why is it so good? Well tmem is basically acting as a big shared page cache, shared across all of the guest OS's. If you know a branch of statistics called queuing theory, you'll understand that this is mathematically better than a set of smaller page caches. Then when dedup and compression are turned on, the size of this single shared page cache is effectively quadrupled for this workload. Further, any swapping is also sharing the large pool of memory. What about that VCPU seconds increase though? Well, virtualization was first conceived as a way to more effectively utilize CPU cycles. Tmem does exactly that. It IS not only making better use of underutilized physical RAM, but it is also making use of previously unutilized CPU cycles! And... remember that this workload has all four guests pounding on the CPU and disk simultaenously. If those guests instead utilize the machine more sparsely, tmem may see even better results. Summary: So to summarize, tmem's primary objective is to make memory a more flexible resource and to do it better than ballooning and other memory utilization technologies with fewer disadvantages. It certainly works well on this workload by showing a dramatic reduction in disk reads, and a faster time-to-completion, while utilizing the CPU more effectively. Disadvantages? Well, the OS must be made smart in a few places by adding a few hooks that help it deal better with page cache evictions and swapping. And one could argue that by using the CPU more effectively, we're costing some power consumption. Those are the tradeoffs... you decide! Acknowledgements: Before I conclude, I'd like to thank Jeremy for some great ideas that have been applied to the Linux-side code for tmem. For more information: And there's a URL for more information and feel free to me or talk to me later. Transcendent Memory Update (Xen Summit 2010) -- Speaker Notes Page 7 of 7