Memory Management Techniques for Large-Scale Persistent-Main-Memory Systems [VLDB 2017]

Transcription

1 Memory Management Techniques for Large-Scale Persistent-Main-Memory Systems [VLDB 2017] Ismail Oukid, Daniel Booss, Adrien Lespinasse, Wolfgang Lehner, Thomas Willhalm, Grégoire Gomes PUBLIC Non-Volatile Memories Workshop March 12, 2018

2 Motivation NVM can replace both main memory and storage single-level database storage architecture without I/O Fail-safe persistent NVM memory management is conditio sine qua non for enabling this novel architecture paradigm Existing persistent allocators are general-purpose and do not address the versatile needs of database systems We present PAllocator, a highly scalable fail-safe persistent allocator 2

3 Outlook What is a persistent allocator? PAllocator s design decisions Experimental evaluation Conclusion 3

4 What Characterizes a Persistent Allocator? Application A persistent allocator must: 1. Provide a recoverable addressing scheme 2. Avoid persistent memory leaks Transient allocator application address space Persistent allocator Virtual memory subsystem DRAM NVM 4

5 1. Recoverable Addressing Scheme Start address Program root at known offset Offset Volatile pointer = File start address + Offset (mmap) Virtual Address Space PPtr: {File ID, Offset} NVM File 5

6 2. Preventing Memory Leaks pptr = allocate(size); persist(&pptr); Traditional interface has a blind spot Reference passing allocate(pptr &pptr, size_t allocsize) pptr is owned by the data structure 6

7 PAllocator Design We explore the following design dimensions 1. Pool structure (single file vs. multiple files) 2. Allocation strategies 3. Concurrency Handling 4. Persistent fragmentation We do not consider garbage collection We assume hardware-managed wear-leveling 7

8 1. Pool Structure: Single Vs. Multiple Files Pool as Single File Pros 8-byte persistent pointers possible Easier to implement Cons Hard to shrink Huge block allocation a problem Pool as Multiple Files Pros Easier to grow and shrink Easy, fragmentation-free huge allocation handling Cons 16-byte persistent pointers Multiple files better suited for database systems 8

9 2. Allocation Strategies Three allocation strategies One file per allocation Segregated-fit for small blocks (e.g., < 4 KB) Best-fit for medium and large blocks (e.g., [4 KB, 16 MB)) One file per allocation not realistic Significant overhead and wasted memory for small blocks Filesystem might struggle to handle huge number of files except for huge blocks! Fragmentation handling pushed to filesystem 9

10 Segregated-Fit Allocation Strategy Fixed-size memory chunk, e.g., 8 KB, divided into fixed-size blocks Bitmap One allocation == one bit flip! Allocated block Free block Multiple class sizes Reduced fragmentation with moderate number of class sizes Not suitable for larger block allocations 10

11 2. Allocation Strategies: Best-Fit Allocation Strategy Allocate multiple of a predetermined size (e.g., system page size) Allocation Free blocks index sorted by block size DRAM NVM Inner nodes Inner nodes Coalescing Global block index sorted by block offset Indexes implemented with the FPTree, a hybrid NVM- DRAM B+-Tree [SIGMOD 16] Suitable for large blocks Segment (e.g., 128 MB) Prone to fragmentation 11

12 3. Concurrency Handling Thread-local allocation One allocator object per thread The standard in general-purpose allocators Used for small block allocations Local allocator requests chunks from global pool Need to be merged with global pool when thread terminates Does not scale under high concurrency Frequent chunk requests to the global pool 12

13 3. Concurrency Handling Core-local allocation One allocator object per physical core Local allocators request large files from global pool Socket 1 C1 C2 Alloc Alloc QPI Socket 2 C1 C2 Alloc Alloc Robust performance under high concurrency Stable local allocators Greedy 13

14 4. Persistent Fragmentation Restart is a last resort, but valid way of defragmenting volatile memory does not apply to NVM File system solutions do not apply to NVM - File systems benefit from an additional indirection layer - NVM is directly accessed with load/store instructions Need new defragmentation mechanisms 14

15 4. Persistent Fragmentation Most file systems have support for sparse files Defragmentation idea: Punch holes in free blocks Iterate until target size reached Used Free Hole Find largest free block Punch hole using fallocate Used Free Hole Free Used Must keep file size unchanged to maintain validity of offsets 15

16 PAllocator: Architecture Overview Allocator Objects Small Alloc 1 Small Alloc n Big Alloc 1 Big Alloc n Huge Alloc Persistent Allocators Small Block Allocator Big Block Allocator Huge Block Allocator Segment Manager Shared list of free segments Failure-Atomic Segment Provider Segment ownership map NVM-aware Filesystem File creation, deletion, memory mapping 16

17 Throughput [op/thread/sec] PAllocator Performance Evaluation Random-Size Allocation/Deallocation (64 B KB) Threads 1.7x 7.6x PAllocator NVML jemalloc PAllocator scales nearly linearly 17

18 KOPS/S Allocator Performance Impact on the FPTree % Insert 1.4x % Find, 50% Insert 1.2x PAllocator NVML 0 PAllocator NVML Persistent allocators do impact database performance 18

19 Time [sec] Allocator Recovery Time x 1 0,1 0, x 4.6x PAllocator NVML Makalu nvm_malloc 0,001 0, Allocated Data Size [MB] 1 TB PAllocator (0.75s), NVML (3.5s), Makalu (394.5s), nvm_malloc (22.5s) 19

20 Conclusion NVM has the potential to disrupt database storage architecture Memory management is a necessary building block We presented PAllocator: Designed for large NVM systems Highly scalable Fast recovery Defragmentation capability Allocator Objects Persistent Allocators Segment Manager NVM-aware Filesystem Small Alloc 1 Small Big Big Alloc n Alloc 1 Huge Alloc n Alloc Small Block Allocator Shared list of free segments Big Block Allocator Failure-Atomic Segment Provider File creation, deletion, memory mapping Huge Block Allocator Segment ownership map 20

21 State-of-the-Art Allocator Purpose Pool structure Allocation strategies Concurrency handling Garbage collection Defragmentation Source Mnemosyne General Multiple files Segregated-fit + best-fit Thread-local for small blocks Yes No ASPLOS 11 NV-Heaps General Single file Undefined Thread-local Yes No ASPLOS 11 nvm_malloc General Single file Segregated-fit + best-fit NVML General Single file Segregated-fit + best-fit Makalu General Single file Segregated-fit + best-fit Thread-local for small blocks Thread-local for small blocks Thread-local for small blocks No No ADMS 15 No No o/nvml/ Yes (offline) No OOPSLA 16 PAllocator Large systems Multiple files Segregated-fit + best-fit + file Core-local No Yes VLDB 17 Salient differences in design decisions For completeness: NVMalloc and Walloc focus on wear-leveling 21