Mnemosyne Lightweight Persistent Memory

Transcription

1 Mnemosyne Lightweight Persistent Memory Haris Volos Andres Jaan Tack, Michael M. Swift University of Wisconsin Madison

2 Executive Summary Storage-Class Memory (SCM) enables memory-like storage Persistent Memory is an abstraction that enables direct access to SCM Durable memory transactions allow consistent in-place updates 2

3 Technology Trends: SCM Features Memory-like interface (load/store) Short access time (1000x faster than flash) Non-volatile Technologies Phase Change Memory (PCM) Spin Torque Transfer RAM (STT-RAM) Memristors Flash-backed DRAM (+ supercapacitor) 3

4 Application Trends: Low-latency storage Web applications Amazon Facebook Desktop applications Firefox Other Distributed agreement protocols High-frequency trading 4

5 GOAL: Flexible Low-latency Storage Idea 1: File System Layering overheads Idea 2: Persistent Object Stores Single data-storage model Our idea: Persistent Memory +Flexible and fine-grain +Low-latency 5

6 Persistent Memory Memory Controller DRAM 6 SCM

7 Persistent Memory Address Space DRAM 7 SCM

8 Persistent Memory Address Space DRAM 8 SCM

9 Persistent Memory Address Space DRAM 9 SCM

10 Persistent Memory Address Space DRAM 10 SCM

11 Persistent Memory Address Space DRAM 11 SCM

12 Persistent Memory Address Space DRAM 12 SCM

13 Persistent Memory Address Space DRAM 13 SCM

14 Persistent Memory Address Space Inconsistent state (missing pointer) 14

15 Persistent Memory Address Space Consistent Updates prevent inconsistent state 15

16 In-vivo Persistence In-vitro vs. In-vivo persistence Logic-tier Caching-tier Storage-tier 16

17 In-vivo Persistence In-vitro vs. In-vivo persistence Logic-tier Caching-tier Storage-tier In-vitro persistence 17

18 In-vivo Persistence In-vitro vs. In-vivo persistence Logic-tier Caching-tier In-vivo persistence 18

19 In-vivo Persistence In-vitro vs. In-vivo persistence Logic-tier Caching-tier Example: OpenLDAP Handle Request In-memory Directory 19 Berkeley DB

20 In-vivo Persistence In-vitro vs. In-vivo persistence Logic-tier Caching-tier Example: OpenLDAP Handle Request In-memory Directory 20

21 Outline Motivation Persistent Memory Persistent regions Consistent updates Durable Memory Transactions Evaluation 21

22 Persistent Regions Virtual memory segments stored in SCM Enable programming flexibility Persist across restarts Two region types Persistent region type Static Dynamic (persistent heap) API pstatic var pmalloc 22

23 Persistent Regions Virtual memory segments stored in SCM Enable programming flexibility Persist across restarts Two region types Persistent region type Static Dynamic (persistent heap) API pstatic var pmalloc Persistent data New allocation 23

24 Persistent Regions Virtual memory segments stored in SCM Enable programming flexibility Persist across restarts Two region types Persistent region type Static Dynamic (persistent heap) API pstatic var pmalloc Persistent data New allocation 24

25 Persistent Regions Virtual memory segments stored in SCM Enable programming flexibility Persist across restarts Two region types Persistent region type Static Dynamic (persistent heap) API pstatic var pmalloc Persistent data mynamedroot New allocation 25

26 Consistent Updates Support updating data without risking correctness after a failure R.value = 0xC0FFEE R.valid = 1 value valid Write-back cache value valid SCM 0xDEADBEEF 0 26

27 Consistent Updates Support updating data without risking correctness after a failure R.value = 0xC0FFEE R.valid = 1 value valid Write-back cache 0xC0FFEE value valid SCM 0xDEADBEEF 0 27

28 Consistent Updates Support updating data without risking correctness after a failure R.value = 0xC0FFEE R.valid = 1 value Write-back cache 0xC0FFEE valid 1 value valid SCM 0xDEADBEEF 0 28

29 Consistent Updates Support updating data without risking correctness after a failure R.value = 0xC0FFEE R.valid = 1 value valid Write-back cache 0xC0FFEE value valid SCM 0xDEADBEEF 01 29

30 Consistent Updates Support updating data without risking correctness after a failure R.value = 0xC0FFEE R.valid = 1 value valid Write-back cache 0xC0FFEE value valid SCM 0xDEADBEEF 01 30

31 Consistent Updates Support updating data without risking correctness after a failure R.value = 0xC0FFEE R.valid = 1 value valid Write-back cache value valid SCM 0xDEADBEEF 01 31

32 Consistent Updates Support updating data without risking correctness after a failure Rely on conventional CPU primitives for ordering Flush Fence value Write-back cache value SCM 0xDEADBEEF valid valid 0 32

33 Consistent Updates Support updating data without risking correctness after a failure Rely on conventional CPU primitives for ordering Flush Fence R.value 0xC0FFEE value valid Write-back cache 0xC0FFEE value valid SCM 0xDEADBEEF 0 33

34 Consistent Updates Support updating data without risking correctness after a failure Rely on conventional CPU primitives for ordering Flush Fence Write-back cache value SCM 0xDEADBEEF 0xC0FFEE R.value 0xC0FFEE value FLUSH(&R.value) 0 valid valid 34

35 Consistent Updates Support updating data without risking correctness after a failure Rely on conventional CPU primitives for ordering Flush Fence R.value 0xC0FFEE FLUSH(&R.value) FENCE R.valid 1 value Write-back cache valid 1 35 value valid SCM 0xDEADBEEF 0xC0FFEE 0

36 Consistent Updates Support updating data without risking correctness after a failure Rely on conventional CPU primitives for ordering Flush Fence R.value 0xC0FFEE FLUSH(&R.value) FENCE R.valid 1 value Write-back cache valid 1 36 value valid SCM 0xDEADBEEF 0xC0FFEE 0

37 Consistent Updates Support updating data without risking correctness after a failure Rely on conventional CPU primitives for ordering Flush Fence R.value 0xC0FFEE FLUSH(&R.value) FENCE R.valid 1 value valid 37 Write-back cache value valid SCM 0xDEADBEEF 0xC0FFEE 0

38 Outline Motivation Persistent Memory Persistent regions Consistent updates Durable Memory Transactions Evaluation 38

39 Durable Memory Transactions Compiler instruments atomic blocks atomic { R.value = 0xC0F; R.valid = 1; } 39

40 Durable Memory Transactions Compiler instruments atomic blocks atomic { R.value = 0xC0F; R.valid = 1; } begin_transaction(); commit_transaction(); 40

41 Durable Memory Transactions Compiler instruments atomic blocks atomic { R.value = 0xC0F; R.valid = 1; } begin_transaction(); stm_store(&r.value, 0xC0F); stm_store(&r.valid, 1); commit_transaction(); 41

42 Durable Memory Transactions Compiler instruments atomic blocks atomic { R.value = 0xC0F; R.valid = 1; } begin_transaction(); stm_store(&r.value, 0xC0F); stm_store(&r.valid, 1); commit_transaction(); Runtime supports ACID transactions Based on TinySTM 42

43 Persistent Heap Hash Table Example pstatic htroot = NULL; htroot Static 43

44 Hash Table Example pstatic htroot = NULL; htroot 0x0000CAFE main() { if (!htroot) pmalloc(n*sizeof(*bucket), &htroot); } Static Persistent Heap 44

45 Hash Table Example pstatic htroot = NULL; htroot 0x0000CAFE main() { if (!htroot) pmalloc(n*sizeof(*bucket), &htroot); } Static Persistent Heap 45

46 Hash Table Example pstatic htroot = NULL; htroot 0x0000CAFE main() { if (!htroot) pmalloc(n*sizeof(*bucket), &htroot); } update_hash(key, value) { atomic { pmalloc(sizeof(*bucket), &bucket)); bucket->key = key; bucket->value = value; insert(hash, bucket); } } Static Persistent Heap 46

47 Hash Table Example pstatic htroot = NULL; htroot 0x0000CAFE main() { if (!htroot) pmalloc(n*sizeof(*bucket), &htroot); } update_hash(key, value) { atomic { pmalloc(sizeof(*bucket), &bucket)); bucket->key = key; bucket->value = value; insert(hash, bucket); } } Static Persistent Heap 47

48 Hash Table Example pstatic htroot = NULL; htroot 0x0000CAFE main() { if (!htroot) pmalloc(n*sizeof(*bucket), &htroot); } update_hash(key, value) { atomic { pmalloc(sizeof(*bucket), &bucket)); bucket->key = key; bucket->value = value; insert(hash, bucket); } } key value Static Persistent Heap 48

49 Hash Table Example pstatic htroot = NULL; htroot 0x0000CAFE main() { if (!htroot) pmalloc(n*sizeof(*bucket), &htroot); } update_hash(key, value) { atomic { pmalloc(sizeof(*bucket), &bucket)); bucket->key = key; bucket->value = value; insert(hash, bucket); } } key value Static Persistent Heap 49

50 Hash Table Example pstatic htroot = NULL; htroot 0x0000CAFE main() { if (!htroot) pmalloc(n*sizeof(*bucket), &htroot); } update_hash(key, value) { atomic { pmalloc(sizeof(*bucket), &bucket)); bucket->key = key; bucket->value = value; insert(hash, bucket); } } Static Persistent Heap 50

51 Hash Table Example pstatic htroot = NULL; htroot 0x0000CAFE main() { if (!htroot) pmalloc(n*sizeof(*bucket), &htroot); } update_hash(key, value) { atomic { pmalloc(sizeof(*bucket), &bucket)); bucket->key = key; bucket->value = value; insert(hash, bucket); } } Static Persistent Heap 51

52 Outline Motivation Persistent Memory Persistent regions Consistent updates Durable Memory Transactions Evaluation 52

53 Evaluation Is it easy to use? Applications Is it fast? Microbenchmarks Applications 53

54 Applications TokyoCabinet: B-tree based key-value store Original: syncs B-tree to a memory-mapped file Modified: keeps B-tree in persistent memory OpenLDAP: Lightweight Directory Original: stores dir-entries in Berkeley DB Modified: keeps dir-entries in persistent memory 54

55 Performance Evaluation Platform Intel Core 2 2.5GHz (4 cores) x86-64 Linux Intel STM C/C++ compiler Performance Model Writes: DRAM + extra fixed delay (150ns) Reads: DRAM Configurations PCM-disk: ext2fs + RAM-disk Mnemosyne: Persistent memory 55

56 Latency (μs) Microbenchmark Hash Table Berkeley DB (PCM-disk) vs Mnemosyne BDB-1 thread BDB-2 threads BDB-4 threads MNE-1 thread MNE-2 threads MNE-4 threads x 4x value size (bytes)

57 Update Throughput (10 3 ops/s ) Latency (μs) Microbenchmark Hash Table Berkeley DB (PCM-disk) vs Mnemosyne BDB-1 thread BDB-2 threads BDB-4 threads MNE-1 thread MNE-2 threads MNE-4 threads value size (bytes)

58 Update Throughput (10 3 ops/s ) Latency (μs) Microbenchmark Hash Table Berkeley DB (PCM-disk) vs Mnemosyne BDB-1 thread BDB-2 threads BDB-4 threads MNE-1 thread MNE-2 threads MNE-4 threads value size (bytes)

59 Update Throughput (10 3 ops/s ) Latency (μs) Microbenchmark Hash Table Berkeley DB (PCM-disk) vs Mnemosyne BDB-1 thread BDB-2 threads BDB-4 threads MNE-1 thread MNE-2 threads MNE-4 threads value size (bytes)

60 Update throughput relative to PCM-disk Application Performance Write-mostly workloads TokyoCabinet: 1024-byte ins/del queries OpenLDAP: template-based update queries 15 30, Original (PCM-disk) Mnemosyne 0 2,044 5,428 7,350 TokyoCabinet OpenLDAP 60

61 Conclusion: Mnemosyne Exposes SCM directly to programmers as persistent memory Relies on conventional CPU primitives for ordering Enables low-latency, consistent, in-place updates via durable memory transactions 61