Transcendent Memory on Linux

Transcription

1 <Insert Picture Here> Transcendent Memory on Linux Speaker: Dan Magenheimer Oracle Corporation

2 Agenda Motivation and Challenge Overview of Physical Memory Management Transcendent Memory ( tmem ) Overview Transcendent Memory in Action Status, Futures, etc.

3 Motivation Memory is increasingly becoming a bottleneck in virtualized system Existing mechanisms have major holes Four underutilized 2-cpu virtual servers each with 1GB RAM X memory overcommitment One 4-CPU physical server w/4gb RAM X page sharing ballooning

4 The Virtualized Physical Memory Resource Optimization Challenge Optimize, across time, the distribution of machine memory among a maximal set of virtual machines by: measuring the current and future memory need of each running VM and reclaiming memory from those VMs that have an excess of memory and either: providing it to VMs that need more memory or using it to provision additional new VMs. without suffering a significant performance penalty

5 The Virtualized Physical Memory Resource Optimization Challenge Optimize, across time, the distribution of machine memory among a maximal set of virtual machines by: measuring the current and future memory need of each running VM and reclaiming memory from those VMs that have an excess of memory and either: providing it to VMs that need more memory or using it to provision additional new VMs. without suffering a significant performance penalty..why is this a hard problem?

6 Agenda Motivation and Challenge Overview of Physical Memory Management in an operating system in a virtual machine monitor (Xen) Transcendent Memory Overview Transcendent Memory In Action Status, Futures, etc.

7 OS Physical Memory Management OS Operating systems are memory hogs! Memory constraint

8 OS Physical Memory Management OS Operating systems are memory hogs! If you give an operating system more memory.. New larger memory constraint

9 OS Physical Memory Management Operating systems are memory hogs! My name is Linux and I am a memory hog it uses up any memory you give it! Memory constraint

10 OS Physical Memory Management What does an OS do with all that memory? Kernel code and data Kernel code User code and data Kernel data Page tables User data User code Page cache Page cache! Everything else

11 OS Physical Memory Management page cache What does an OS do with all that memory? Page cache attempts to predict future needs of pages from the disk sometimes it gets it right good pages

12 OS Physical Memory Management page cache What does an OS do with all that memory? Page cache attempts to predict future needs of pages from the disk sometimes it gets it wrong wasted pages

13 OS Physical Memory Management page cache What does an OS do with all that memory? much of the time mostly page cache some of which will be useful in the future and some of which is wasted Everything else

14 Agenda Motivation and Challenge Overview of Physical Memory Management in an operating system in a virtual machine monitor (Xen) Transcendent Memory Overview Transcendent Memory In Action Status, Futures, etc.

15 VMM Physical Memory Management Xen partitions memory hypervisor memory dom0 memory memory leftover (in case needed later) Dom0 is special

16 VMM Physical Memory Management? Xen partitions memory Xen memory dom0 memory 1 memory 2 memory whatever s left over: fallow memory fallow, adj., land left without a crop for one or more years

17 VMM Physical Memory Management fallow fallow fallow Xen partitions memory Xen memory dom0 memory 1 memory 2 memory whatever s left over: fallow memory fallow, adj., land left without a crop for one or more years

18 VMM Physical Memory Management fallow fallow gues t fallow Xen partitions memory among more s Xen memory dom0 memory 1 memory 2 memory 3 BUT still fallow memory leftover

19 VMM Physical Memory Management in the presence of migration fallow migration requires fallow memory in the target machine leaves behind fallow memory in the originating machine Physical machine B Physical machine A fallow fallow gue st fallow gue st fallow gues t fallow gues t

20 VMM Physical Memory Management in the presence of ballooning fallow gues t fallow Use ballooning to allow memory size to grow? Goal: fill fallow memory fallow

21 VMM Physical Memory Management in the presence of ballooning fallow Look! No more fallow memory! gue st gue st But.

22 VMM Physical Memory Management in the presence of ballooning fallow Look! No more fallow memory! gue st gue st But.

23 VMM Physical Memory Management in the presence of ballooning gue st gue st fallow Look! No more fallow memory! But. And but

24 VMM Physical Memory Management in the presence of ballooning fallow Ballooning down to handle incoming migrations NASTY issues! To make room for a new, we have to starve existing s!

25 VMM Physical Memory Management in the presence of ballooning Using ballooning to take memory away: not instantaneous (memory inertia) can t predict future needs good pages are evicted along with the bad don t know how much/fast to balloon Too much or too fast thrashing or the dreaded OOM killer OOM OOM OOM OOM

26 The Virtualized Physical Memory Resource Optimization Challenge Optimize, across time, the distribution of machine memory among a maximal set of virtual machines by: measuring the current and future memory need of each running VM and reclaiming memory from those VMs that have an excess of memory and either: providing it to VMs that need more memory or using it to provision additional new VMs. without suffering a significant performance penalty..this IS a hard problem!!!

27 Why this IS a hard problem! Summary OS s use as much memory as they are given but cannot predict the future so often guess wrong and often much memory owned by an OS is wasted Xen leaves large amounts of memory fallow fixed partitioning results in fragmentation migration requires fallow memory to succeed Ballooning helps but: can t predict future memory needs of s memory has inertia the price of incorrect guesses can be dire NEED A NEW APPROACH TO VIRTUALIZED PHYSICAL MEMORY MANAGEMENT!!

28 Agenda Motivation and Challenge Overview of Physical Memory Management Transcendent Memory Overview Transcendent Memory In Action Status, Futures, etc.

29 Transcendent memory creating the transcendent memory pool fallow Step 1a: reclaim fallow memory fallow Step 1b: reclaim wasted memory (e.g. via ballooning) Step 1c: collect it all into a pool fallow Transcendent memory pool

30 Transcendent memory creating the transcendent memory pool data data Transcendent memory pool data Step 2: provide indirect access, strictly controlled by the hypervisor and dom0 control data control

31 Transcendent memory API characteristics Transcendent memory pool Transcendent memory API OS changes (small) narrow well-specified operations are: synchronous page-oriented (one page per op) copy-based multi-faceted extensible

32 Transcendent memory Linux tmem_ops API int (*new_pool)(struct tmem_pool_uuid uuid, u32 flags); int (*put_page)(u32 pool_id, u64 object, u32 index, unsigned long pfn); int (*get_page)(u32 pool_id, u64 object, u32 index, unsigned long pfn); int (*flush_page)(u32 pool_id, u64 object, u32 index); int (*flush_object)(u32 pool_id, u64 object); int (*destroy_pool)(u32 pool_id);

33 Transcendent memory Xen hypercall API int xen_tmem_op( u32 tmem_cmd, u32 tmem_pool, u64 object, u32 index, unsigned long gmfn, u32 tmem_offset, u32 pfn_offset, u32 len); int xen_tmem_new_pool( struct tmem_pool_uuid uuid, u32 flags);

34 Transcendent memory four different subpool types four different uses Legend: flags private shared ephemeral second-chance clean-page cache!! cleancache server-side cluster filesystem cache shared cleancache persistent Fast swap device!! frontswap inter- shared memory? Implemented and working today (Linux + Xen) Now in patch Under investigation eph-em-er-al, adj., transitory, existing only briefly, short-lived (i.e. NOT persistent)

35 Transcendent memory caveats Requirements OS kernel must be tmem-aware 64-bit Xen hypervisor and CPU (32-bit s OK) Workload: should exert memory pressure in at least one memory pressure in multiple s should vary across time For best results: dom0 should be configured with a fixed memory size should have a (virtual) swap disk configured Complementary to: feedback-directed ballooning (e.g. self-ballooning ) transparent content-based page sharing

36 Transcendent memory Linux core tmem patch diffstat Documentation/transcendent-memory.txt include/linux/tmem.h mm/kconfig 10 mm/makefile 1 mm/tmem.c files changed, 372 insertions(+) New files in green Changed files in black [PATCH 1/4] Declares the tmem_ops accessors and initializes them as no-ops (foundation layer for cleancache and frontswap ) (see for patchset)

37 Agenda Motivation and Challenge Overview of Physical Memory Management Transcendent Memory Overview Transcendent Memory In Action private-ephemeral pool cleancache shared-ephemeral pool shared cleancache private-persistent pool frontswap performance analysis Status, Future, etc.

38 cleancache* put Transcendent memory pool (private+ephemeral) a second-chance clean page cache for a put clean pages only get only valuable pages pages eventually are evicted coherency managed by exclusive cache semantics * previously called precache get private Transcendent Memory Pool types ephemeral second-chance clean-page cache!! cleancache persistent Fast swap device!! frontswap shared server-side cluster filesystem cache? shared cleancache inter- shared memory?

39 cleancache (with compression) Compression Optional (per-) nominally doubles available memory performance-space tradeoff some fragmentation issues

40 cleancache (multiple s) second-chance page cache for multiple s private ephemeral tmem pool #1 Need memory scheduler : global admission/eviction policy: LRU queue weight balanced private ephemeral tmem pool #2

41 shared cleancache (for clustering) Clustered filesystem SHARED ephemeral tmem pool s sharing a clustered filesystem (ocfs2) non-exclusive LFU instead of LRU compression optional a server-side disk cache! security issues being worked resolved for private Transcendent Memory Pool types ephemeral second-chance clean-page cache!! cleancache persistent Fast swap device!! frontswap shared server-side cluster filesystem cache? shared cleancache inter- shared memory?

42 Transcendent memory Linux cleancache API extern void cleancache_init(struct super_block *sb); extern int cleancache_get(struct address_space *mapping, unsigned long index, struct page *empty_page); extern int cleancache_put(struct address_space *mapping, unsigned long index, struct page *page); extern int cleancache_flush(struct address_space *mapping, unsigned long index); extern int cleancache_flush_inode( struct address_space *mapping); extern int cleancache_flush_filesystem( struct super_block *sb);

43 Transcendent memory Linux cleancache patch diffstat fs/buffer.c 5 fs/ext3/super.c 2 fs/btrfs/extent_io.c 9 + Small fs/btrfs/super.c 2 handful of fs/ext4/super.c 2 hooks fs/ocfs2/super.c 2 fs/super.c 5 needed to include/linux/fs.h 7 + utilize mm/truncate.c 10 + cleancache mm/filemap.c 11 + mm/kconfig 8 + API mm/makefile 1 include/linux/cleancache.h mm/cleancache.c files changed, 310 insertions(+) New files in green Changed files in black [PATCH 2/4] (see for patchset)

44 frontswap* over-ballooned s experiencing unexpected memory pressure have an emergency swap disk much faster than swapping persistent ( dirty ) pages OK prioritized higher than cleancache limited by s maxmem * previously called preswap Transcendent Memory Pool types ephemeral persistent OOM OOM OOM OOM private shared second-chance clean-page cache!! cleancache server-side cluster filesystem cache? shared cleancache Fast swap device!! frontswap inter- shared memory?

45 Transcendent memory Linux frontswap API extern void frontswap_init(unsigned type); extern int frontswap_put(struct page *page); extern int frontswap_get(struct page *page); extern void frontswap_flush(unsigned type, unsigned long offset); extern void frontswap_flush_area(unsigned type); extern void frontswap_shrink(unsigned long target_pages); extern int frontswap_test(struct swap_info_struct *sis, unsigned long offset); (Note: type is Linux term for which swap disk, an index into swap_info_struct array)

46 Transcendent memory Linux frontswap patch diffstat include/linux/swap.h mm/swapfile.c include/linux/sysctl.h 1 New files in green kernel/sysctl.c 12 Changed files in black mm/kconfig 8 mm/makefile 1 mm/page_io.c 12 mm/frontswap.c files changed, 441 insertions(+), 6 deletions(-) [PATCH 3/4] Somewhat more invasive than cleancache due to new frontswap_map array and partial swapoff (see for patchset)

47 Transcendent memory Linux tmem-xen glue patch diffstat arch/x86/include/asm/xen/hypercall.h 8 + drivers/xen/makefile 1 drivers/xen/tmem.c include/xen/interface/tmem.h include/xen/interface/xen.h files changed, 171 insertions(+) New files in green Changed files in black [PATCH 4/4] Changes in Xen-specific code to layer Linux tmem API on xen hypercall (see for patchset)

48 Transcendent memory Linux tmem patch diffstat showing changed existing files only fs/buffer.c 5 ++ fs/ext3/super.c 2 fs/btrfs/extent_io.c fs/btrfs/super.c 2 fs/ext4/super.c 2 fs/ocfs2/super.c 2 fs/mpage.c 8 ++ fs/super.c 6 ++ mm/filemap.c mm/page_io.c mm/swapfile.c mm/truncate.c kernel/sysctl.c include/linux/fs.h 7 ++ include/linux/swap.h include/linux/sysctl.h 1 mm/kconfig mm/makefile files changed, 204 insertions(+), 7 deletions(-) (see for patchset)

49 Agenda Motivation and Challenge Overview of Physical Memory Management Transcendent Memory Overview Transcendent Memory In Action private-ephemeral pool cleancache shared-ephemeral pool shared cleancache private-persistent pool frontswap performance analysis Status, Future, etc.

50 performance analysis: screenshot

51 performance analysis: workload Eight VM s, each running: # service xenballoond start # cd src/linux # while true; do make clean; make j4; done

52 performance analysis: data collection

53 performance analysis: data overview Quad-core physical CPU, single socket, dualthread 8 s with 384MB and 2 vcpus each Staggered starts, some running for nearly a virtual hour, some just started

54 performance analysis: memory pressure Aggressive self-ballooning launched as an initd service feedback-driven using CommittedAS from /proc/meminfo considering moving in-kernel See: Magenheimer, D., Memory Overcommit without the Commitment, Xen Summit 2008

55 performance analysis: kernel data normal /proc/vmstat data from each VM

56 performance analysis: tmem data Important per-vm tmem statistics

57 performance analysis: cleancache cleancache benefit reduced page-in s from external storage: by 30-40% by /sec (note, cleancache does not reduce page-out s pgout shown only for reference)

58 performance analysis: frontswap frontswap benefit eliminated all physical swapping reduced I/Os to external storage: by 128K writes by 80K reads read+write: ~70/sec

59 performance analysis: CPU cost Tmem total cost instrumented in hypervisor approximation only CPU overhead: 0.08%-0.15%

60 performance analysis: summary Bottom line: ~0.1% of one core saves ~300 io/sec (depends on workload, your mileage may vary, objects appear smaller in the mirror, etc., etc.)

61 Agenda Motivation and Challenge Overview of Physical Memory Management Transcendent Memory Overview Transcendent Memory In Action Status, Future, etc.

62 Current Status Cleancache, frontswap, and shared cleancache patches for posted to LKML now includes support for btrfs and ext4 most useful statistics available via sysfs Tmem to be released in Xen 4.0 (target release Feb 2010) Linux tmem patch checked-in in xen tree save/restore and live migration support fully functional Tmem already released in Oracle VM 2.2 (Oct 2009) technology preview

63 Future Work possible integration with ramzswap and FS-cache (Nitin Gupta investigating) combine tmem with page-sharing in-kernel self-ballooning shared-persistent pool investigation inter- communication? real world performance measurement/analysis identify any tuning opportunities, rinse, repeat tmem for: linux containers? l? KVM? fully-virtualized s? embedded?

64 Acknowledgements Chris Mason (Oracle) Linux vfs changes for cleancache Btrfs help Zhigang Wang (Oracle) Xen tools (xm + libxc) code Kurt Hackel, Dave McCracken (Oracle) design and debugging help Sunil Mushran, Joel Becker (Oracle) ocfs2 help for shared cleancache Jeremy Fitzhardinge, Ian Pratt, Keir Fraser, Jan Beulich hard questions and good advice Excellent feedback on lkml and xen-devel

65 With apologies to the late, great Bob Hope Thanks for the memory I really could use more My throughput s on the floor My balloon is flat, my swap disk s fat I ve O-O-M s in store Overcommitted so much

66 And Thank YOU so much! FOR MORE INFORMATION:

67 <Insert Picture Here> Transcendent Memory on Linux Speaker: Dan Magenheimer Oracle Corporation

68 Backup Slides