Soft Updates Made Simple and Fast on Non-volatile Memory

Transcription

1 Soft Updates Made Simple and Fast on Non-volatile Memory Mingkai Dong, Haibo Chen Institute of Parallel and Distributed Systems, Shanghai Jiao Tong NVMW 18

2 Non-volatile Memory (NVM) ü Non-volatile ü Byte-addressable ü High throughput and low latency 2

3 NVM File Systems (NVMFS) Existing NVMFS use journaling or copy-on-write for crash consistency Synchronous cache flushes are necessary Cache flushes are expensive! Other options for crash consistency? A A File System Metadata Journal Area B C C D E E 3

4 NVM File Systems (NVMFS) Existing NVMFS use journaling or copy-on-write for crash consistency Synchronous cache flushes are necessary Cache flushes are expensive! Other options for crash consistency? A A File System Metadata Journal Area B C C D E E 4

5 Soft Updates Latest metadata in DRAM Updated in DRAM with dependency tracked ü DRAM performance ü No synchronous disk writes Consistent metadata in disks Persisted to disks with dependency enforced ü Always consistent ü Immediately usable after crash DRAM (Page cache) DISK Traditional Soft Updates 5

6 Soft Updates Update dependencies E.g., allocating a new data block 1. Allocate in bitmap 2. Fill data in the block 3. Update pointer to the block block bitmap new data block 6

7 Soft Updates Is Complicated Delayed disk writes Auxiliary structures for each update More complex dependencies block bitmap new data block Figures from Soft Updates: A Technique for Eliminating Most Synchronous Writes in the Fast Filesystem, ATC 99 7

8 Soft Updates Is Complicated Delayed disk writes Auxiliary structures for each operation More complex dependencies Cyclic dependencies Rolling back/forward Block #4 #5 Inode #6 Inode #7 Block <A, #4> <--, #0> <E, #7> block (in page cache) #4 Rollback #6 #5 #6 #7 block #4 #5 #6 #7 Flush block to disks block #4 #5 #6 #7 Rollforward #6 block #4 #5 #6 #7 8

9 Soft Updates Is Complicated Delayed disk writes Auxiliary structures for each operation More complex dependencies Cyclic dependencies Rolling back/forward The mismatch between per-pointer-based dependency tracking and block-based interface of traditional disks 9

10 Soft Updates Meets NVM Soft Updates ü No synchronous cache flushes ü Immediately usable after crash NVM: byte-addressable and fast ü ect write to NVM without delays ü No false sharing => no rolling back/forward ü Simple dependency tracking/enforcement 10

11 SoupFS A simple and fast NVMFS derived from soft updates Hashtable-based directories No false sharing Pointer-based dual views No synchronous cache flushes Semantic-aware dependency tracking/enforcement Simple dependency tracking/enforcement Get the best of both Soft Updates and NVM 11

12 Overview Background Design & Implementation Hashtable-based directories Pointer-based dual views Semantic-aware dependency tracking/enforcement Evaluation Conclusion 12

13 Overview Background Design & Implementation Hashtable-based directories Pointer-based dual views Semantic-aware dependency tracking/enforcement Evaluation Conclusion 13

14 Block-based ectories Block-based file systems usually use block-based directories False sharing Cyclic dependency Rolling back/forward Slow access Linear scan indirect block 1.TxT 32.TxT 38 fs-long-lon g.exe l+f.dir 12 14

15 Hashtable-based ectories Optimized for cache lines ü No false sharing Buckets ü No cyclic dependency Efficient access ü No linear scan Filename Pointer Pointer Latest Consistent Hash Len Filename 15

16 Overview Background Design & Implementation ü Hashtable-based directories Pointer-based dual views Semantic-aware dependency tracking/enforcement Evaluation Conclusion 16

17 Dual Views Latest view in page cache Consistent view in disks DRAM (Page cache) Dual views Eliminate synchronous writes Provide usability after crash DISK Traditional Soft Updates 17

18 Dual Views Latest view in page cache Consistent view in disks NVM Latest view? DRAM (Page cache) Another copy of metadata in DRAM Double writes Double storage overhead Unnecessary synchronizations DISK NVM Challenge: How to present latest view efficiently? Soft Updates on NVM 18

19 Pointer-based Dual Views Reuse data structures in both views Distinguish views by different pointers/structures DRAM NVM Soft Updates on NVM 19

20 Pointer-based Dual Views Reuse data structures in both views Distinguish views by different pointers/structures Data Structures In Consistent View In Latest View SoupFS VFS dentry consistent next pointer latest next pointer hash table bucket latest bucket if exists B-tree root/height in SoupFS root/height in VFS 20

21 Pointer-based Dual Views Buckets Volatile in DRAM Updates to NVM w/o persistence guarantee Persisted in NVM VFS D Filename Latest Consistent C B A Latest View Consistent View 21

22 Pointer-based Dual Views Buckets Volatile in DRAM Updates to NVM w/o persistence guarantee Persisted in NVM VFS D Filename Latest Consistent C B A Latest View Ø create E Consistent View File E 22

23 Pointer-based Dual Views Buckets Volatile in DRAM Updates to NVM w/o persistence guarantee Persisted in NVM VFS D Filename Latest Consistent C B A Latest View Ø create E Latest Buckets File E Consistent View 23

24 Pointer-based Dual Views Buckets Volatile in DRAM Updates to NVM w/o persistence guarantee Persisted in NVM VFS D Filename Latest Consistent C B A E Latest View Ø create E Latest Buckets File E Consistent View 24

25 Pointer-based Dual Views Buckets Volatile in DRAM Updates to NVM w/o persistence guarantee Persisted in NVM VFS D Filename Latest Consistent C B A E Latest View Ø create E Latest Buckets 3 Consistent View File E 25

26 Pointer-based Dual Views Buckets Volatile in DRAM Updates to NVM w/o persistence guarantee Persisted in NVM VFS D Filename Latest Consistent C B A E Latest View create E Latest Buckets 3 Consistent View File E 26

27 Pointer-based Dual Views Buckets Volatile in DRAM Updates to NVM w/o persistence guarantee Persisted in NVM VFS D Filename Latest Consistent C B A E Latest View create E Ø unlink B Latest Buckets 3 Consistent View File E 27

28 Pointer-based Dual Views Buckets Volatile in DRAM Updates to NVM w/o persistence guarantee Persisted in NVM VFS D Filename Latest Consistent C B A E Latest View create E Ø unlink B Latest Buckets 3 Consistent View File E 28

29 Pointer-based Dual Views Buckets Volatile in DRAM Updates to NVM w/o persistence guarantee Persisted in NVM VFS D Filename Latest Consistent C B A E Latest View create E unlink B Latest Buckets 3 Consistent View File E 29

30 Pointer-based Dual Views Buckets Volatile in DRAM Updates to NVM w/o persistence guarantee Persisted in NVM VFS D Filename Latest Consistent C B A E Latest View create E unlink B Latest Buckets 3 Consistent View File E 30

31 Pointer-based Dual Views Buckets Volatile in DRAM Updates to NVM w/o persistence guarantee Persisted in NVM VFS D Filename Latest Consistent C B A E Latest View create E unlink B Latest Buckets 3 Consistent View File E 31

32 Pointer-based Dual Views Buckets Volatile in DRAM Updates to NVM w/o persistence guarantee Persisted in NVM VFS D Filename Latest Consistent C B A E Latest View create E unlink B Latest Buckets 3 Consistent View File E 32

33 Pointer-based Dual Views Buckets Volatile in DRAM Updates to NVM w/o persistence guarantee Persisted in NVM VFS D Filename Latest Consistent C B A E Latest View create E unlink B Latest Buckets 3 Consistent View File E 33

34 Pointer-based Dual Views Buckets Volatile in DRAM Updates to NVM w/o persistence guarantee Persisted in NVM VFS D Filename Latest Consistent C B A E Latest View create E unlink B Latest Buckets 3 Consistent View File E File E 34

35 Pointer-based Dual Views Buckets Volatile in DRAM Updates to NVM w/o persistence guarantee Persisted in NVM VFS D Filename Latest Consistent C B A E Latest View create E unlink B Latest Buckets 3 Consistent View File E File E 35

36 Pointer-based Dual Views Buckets Volatile in DRAM Updates to NVM w/o persistence guarantee Persisted in NVM VFS D Filename Latest Consistent C B A E Latest View create E unlink B Latest Buckets 3 Consistent View File E File E 36

37 Pointer-based Dual Views Buckets Volatile in DRAM Updates to NVM w/o persistence guarantee Persisted in NVM VFS D Filename Latest Consistent C A E Latest View create E unlink B Latest Buckets 3 File A File C File D Consistent View File E File A File C File D File E 37

38 Pointer-based Dual Views Reuse data structures in both views Distinguish views by different pointers/structures DRAM ü Eliminate synchronous writes ü Provide usability after crash NVM ü No double write ü Little space overhead Soft Updates on NVM 38

39 Overview Background Design & Implementation ü Hashtable-based directories ü Pointer-based dual views Semantic-aware dependency tracking/enforcement Evaluation Conclusion 39

40 Dependency Tracking Auxiliary structures for each updates 40

41 Dependency Tracking Auxiliary structures for each updates The semantic gap between the page cache (where enforcement happens) and the file system (where tracking happens) After removing page cache, SoupFS involves semantics in dependency tracking/enforcement 41

42 Semantic-aware Dependency Tracking Track semantic operations with complementary information Enough for dependency enforcement Operation Type diradd dirrem sizechg attrchg Complementary Information (pointers/integers) added dentry, source directory, overwritten removed dentry, destination directory the old and new file size nothing Information tagged with is for rename operation. 42

43 Semantic-aware Dependency Tracking Track semantic operations with complementary information Enough for dependency enforcement Operations are stored in operation list of each VFS dirty list VFS VFS VFS list next operation list list next operation list list next operation list list next operation type Complimentary information 43

44 Semantic-aware Dependency Enforcement Persister daemons traverse the dirty list in background persist each operation from the latest view to the consistent view with respect to update dependencies dirty list VFS VFS VFS list next operation list list next operation list list next operation list list next operation type Complimentary information 44

45 Overview Background Design & Implementation ü Hashtable-based directories ü Pointer-based dual views ü Semantic-aware dependency tracking/enforcement Evaluation Conclusion 45

46 Evaluation Setup Platform Intel Xeon E5 server with two 8-core processors 48 GB DRAM and 64 GB NVDIMM File Systems SoupFS, PMFS, NOVA, Ext4-DAX, Ext4 NVM Write Delay Simulation ndelay() after clflush Benchmarks Micro-benchmarks: 100 iterations of 10 4 create/unlink/mkdir/rmdir Filebench and Postmark 46

47 Micro-benchmark Latency Latency (us/op) Inefficient Organization Ext4 Ext4-DAX PMFS NOVA SoupFS CDF EXT4 Ext4-DAX PMFS NOVA SoupFS 0 create unlink mkdir rmdir Latency (us) Lowest Latency 47

48 Sensitivity to NVM Write Delay Latency(us) PMFS Create Latency (us) PMFS NOVA SoupFS Unlink ~250% 10 5 NOVA SoupFS ~200% Delay (ns) 0 No effect Delay (ns) 48

49 Postmark & Filebench Throughput (MB/s) Ext4 Ext4-DAX PMFS NOVA SoupFS Postmark ~50% Throughput (x1000 ops/s) Fileserver-1K Ext4 Ext4-DAX PMFS NOVA SoupFS 0 Read Write Threads

50 Overview Background Design & Implementation ü Hashtable-based directories ü Pointer-based dual views ü Semantic-aware dependency tracking/enforcement Evaluation Conclusion 50

51 Conclusion Soft updates is complicated due to the mismatch between per-pointer-based dependency tracking and block-based interface of traditional disks We design and implement SoupFS ü Hashtable-based directories ü Pointer-based dual views ü Semantic-aware dependency tracking/enforcement Soft updates can be made simple and fast on NVM Thanks & Questions? ;-) 51

52 52