Strata: A Cross Media File System. Youngjin Kwon, Henrique Fingler, Tyler Hunt, Simon Peter, Emmett Witchel, Thomas Anderson

Transcription

1 A Cross Media File System Youngjin Kwon, Henrique Fingler, Tyler Hunt, Simon Peter, Emmett Witchel, Thomas Anderson 1

2 Let s build a fast server NoSQL store, Database, File server, Mail server Requirements Small updates (1 Kbytes) dominate Dataset scales up to 10 TB Updates must be crash consistent 2

3 Storage diversification Latency $/GB DRAM 100 ns 8.6 NVM (soon) 300 ns 4.0 SSD 10 us 0.25 HDD 10 ms 0.02 Better performance Higher capacity 3

4 Storage diversification Byte-addressable: cache-line granularity IO Latency $/GB DRAM 100 ns 8.6 NVM (soon) 300 ns 4.0 SSD 10 us 0.25 HDD 10 ms 0.02 Better performance Higher capacity 3

5 Storage diversification Byte-addressable: cache-line granularity IO Latency $/GB DRAM 100 ns 8.6 NVM (soon) 300 ns 4.0 SSD 10 us 0.25 HDD 10 ms 0.02 Better performance Higher capacity Large erasure blocks need to be sequentially written Random writes: 5~6x slowdown due to GC [FAST 15] 3

6 A fast server on today s file system Small updates (1 Kbytes) dominate Dataset scales up to 10TB Updates must be crash consistent Application Kernel file system 91% NVM Kernel file system: NOVA [FAST 16, SOSP 17] 4

7 A fast server on today s file system Small updates (1 Kbytes) dominate Dataset scales up to 10TB Updates must be crash consistent Application Small, random IO is slow! Kernel file system 91% NVM Kernel file system: NOVA [FAST 16, SOSP 17] 4

8 A fast server on today s file system Small updates (1 Kbytes) dominate Dataset scales up to 10TB Updates must be crash consistent Application Small, random IO is slow! Kernel file system 1 KB Write to device 91% Kernel code 91% NVM IO latency (us) Kernel file system: NOVA [FAST 16, SOSP 17] 4

9 A fast server on today s file system Small updates (1 Kbytes) dominate Dataset scales up to 10TB Updates must be crash consistent Application Small, random IO is slow! Kernel file system 1 KB Write to device 91% Kernel code 91% NVM IO latency (us) Kernel file system: NOVA NVM is so fast that kernel is the bottleneck [FAST 16, SOSP 17] 4

10 A fast server on today s file system Small updates (1 Kbytes) dominate Dataset scales up to 10TB Updates must be crash consistent Application Kernel file system NVM 5

11 A fast server on today s file system Small updates (1 Kbytes) dominate Dataset scales up to 10TB Updates must be crash consistent Application Need huge capacity, but NVM alone is too expensive! ($40K for 10TB) Kernel file system NVM 5

12 A fast server on today s file system Small updates (1 Kbytes) dominate Dataset scales up to 10TB Updates must be crash consistent Application Need huge capacity, but NVM alone is too expensive! ($40K for 10TB) Kernel file system NVM For low-cost capacity with high performance, must leverage multiple device types 5

13 A fast server on today s file system Small updates (1 Kbytes) dominate Dataset scales up to 10TB Updates must be crash consistent Application Kernel file system Block-level caching NVM SSD HDD 6

14 A fast server on today s file system Small updates (1 Kbytes) dominate Dataset scales up to 10TB Updates must be crash consistent Application Block-level caching manages data in blocks, but NVM is byte-addressable Extra level of indirection Kernel file system Block-level caching NVM SSD HDD 6

15 A fast server on today s file system Small updates (1 Kbytes) dominate Dataset scales up to 10TB Updates must be crash consistent Application Block-level caching manages data in blocks, but NVM is byte-addressable Extra level of indirection Kernel file system Block-level caching NOVA Block-level caching NVM SSD HDD 1 KB IO latency (us) Better 6

16 A fast server on today s file system Small updates (1 Kbytes) dominate Dataset scales up to 10TB Updates must be crash consistent Application Block-level caching manages data in blocks, but NVM is byte-addressable Extra level of indirection Kernel file system Block-level caching NOVA Block-level caching 1 KB NVM SSD HDD Block-level caching IO is latency too (us) slow Better 6

17 A fast server on today s file system Small updates (1 Kbytes) dominate Dataset scales up to 10TB Updates must be crash consistent 7

18 A fast server on today s file system Small updates (1 Kbytes) dominate Dataset scales up to 10TB Updates must be crash consistent SQLite HDFS ZooKeeper LevelDB HSQLDB Mercurial Git Crash vulnerabilities Pillai et al., OSDI

19 A fast server on today s file system Small updates (1 Kbytes) dominate Dataset scales up to 10TB Updates must be crash consistent SQLite HDFS ZooKeeper LevelDB HSQLDB Mercurial Git Crash vulnerabilities Pillai et al., OSDI 2014 Applications struggle for crash consistency 7

20 Problems in today s file systems Kernel mediates every operation NVM is so fast that kernel is the bottleneck Tied to a single type of device For low-cost capacity with high performance, must leverage multiple device types NVM (soon), SSD, HDD Aggressive caching in DRAM, write to device only when you must (fsync) Applications struggle for crash consistency 8

21 A Cross Media File System Performance: especially small, random IO Fast user-level device access Low-cost capacity: leverage NVM, SSD & HDD Transparent data migration across different storage media Efficiently handle device IO properties Simplicity: intuitive crash consistency model In-order, synchronous IO No fsync() required 9

22 main design principle Log operations to NVM at user-level Digest and migrate data in kernel 10

23 main design principle Log operations to NVM at user-level Performance: Kernel bypass, but private Digest and migrate data in kernel 10

24 main design principle Log operations to NVM at user-level Performance: Kernel bypass, but private Simplicity: Intuitive crash consistency Digest and migrate data in kernel 10

25 main design principle Log operations to NVM at user-level Performance: Kernel bypass, but private Simplicity: Intuitive crash consistency Digest and migrate data in kernel Coordinate multi-process accesses 10

26 main design principle Log operations to NVM at user-level Performance: Kernel bypass, but private Simplicity: Intuitive crash consistency Digest and migrate data in kernel Apply log operations to shared data Coordinate multi-process accesses 10

27 main design principle LibFS Log operations to NVM at user-level Performance: Kernel bypass, but private Simplicity: Intuitive crash consistency KernelFS Digest and migrate data in kernel Apply log operations to shared data Coordinate multi-process accesses 10

28 Outline LibFS: Log operations to NVM at user-level Fast user-level access In-order, synchronous IO KernelFS: Digest and migrate data in kernel Asynchronous digest Transparent data migration Shared file access Evaluation 11

29 Log operations to NVM at user-level unmodified application POSIX API LibFS NVM Private operation log creat write rename File operations (data & metadata) 12

30 Log operations to NVM at user-level NVM unmodified application Kernelbypass POSIX API LibFS Private operation log Fast writes Directly access fast NVM Sequentially append data Cache-line granularity Blind writes creat write rename File operations (data & metadata) 12

31 Log operations to NVM at user-level NVM unmodified application Kernelbypass POSIX API LibFS Private operation log creat write rename Fast writes Directly access fast NVM Sequentially append data Cache-line granularity Blind writes Crash consistency On crash, kernel replays log File operations (data & metadata) 12

32 Intuitive crash consistency unmodified application POSIX API LibFS Kernelbypass Synchronous IO NVM Private operation log 13

33 Intuitive crash consistency NVM Kernelbypass unmodified application POSIX API LibFS Private operation log Synchronous IO When each system call returns: Data/metadata is durable In-order update Atomic write Limited size (log size) 13

34 Intuitive crash consistency NVM Kernelbypass unmodified application POSIX API LibFS Private operation log Synchronous IO When each system call returns: Data/metadata is durable In-order update Atomic write Limited size (log size) fsync() is no-op 13

35 Intuitive crash consistency NVM Kernelbypass unmodified application POSIX API LibFS Private operation log Synchronous IO When each system call returns: Data/metadata is durable In-order update Atomic write Limited size (log size) fsync() is no-op Fast synchronous IO: NVM and kernel-bypass 13

36 Outline LibFS: Log operations to NVM at user-level Fast user-level access In-order, synchronous IO KernelFS: Digest and migrate data in kernel Asynchronous digest Transparent data migration Shared file access Evaluation 14

37 Digest data in kernel Application POSIX API LibFS KernelFS NVM Private operation log NVM Shared area 15

38 Digest data in kernel Application POSIX API LibFS Write KernelFS NVM Private operation log NVM Shared area 15

39 Digest data in kernel Application POSIX API LibFS Write KernelFS NVM Private operation log NVM Shared area Digest 15

40 Digest data in kernel NVM Application POSIX API LibFS Write Private operation log Digest KernelFS NVM Shared area Visibility: make private log visible to other applications Data layout: turn write-optimized to read-optimized format (extent tree) Large, batched IO Coalesce log 15

41 Digest optimization: Log coalescing SQLite, Mail server: crash consistent update using write ahead logging Application LibFS Digest eliminates unneeded work Private operation log Remove temporary durable writes KernelFS NVM Shared area.. 16

42 Digest optimization: Log coalescing SQLite, Mail server: crash consistent update using write ahead logging Application LibFS Digest eliminates unneeded work Private operation log Remove temporary durable writes KernelFS NVM Shared area Create journal file Write data to journal file Write data to database file Delete journal file.. 16

43 Digest optimization: Log coalescing SQLite, Mail server: crash consistent update using write ahead logging Application LibFS Digest eliminates unneeded work Private operation log Remove temporary durable writes KernelFS NVM Shared area Create journal file Write data to journal file Write data to database file Delete journal file. Write data to database file. 16

44 Digest optimization: Log coalescing SQLite, Mail server: crash consistent update using write ahead logging Application LibFS Digest eliminates unneeded work Remove temporary durable writes KernelFS Private operation log Create journal file Write data to journal file Write data to database file Delete journal file NVM Shared area Write data to database file. Throughput optimization:. Log coalescing saves IO while digesting 16

45 Digest and migrate data in kernel Application LibFS KernelFS Private operation log NVM Shared area 17

46 Digest and migrate data in kernel Application LibFS Private operation log KernelFS NVM Shared area Low-cost capacity KernelFS migrates cold data to lower layers SSD Shared area HDD Shared area 18

47 Digest and migrate data in kernel Application LibFS Private operation log KernelFS NVM Shared area NVM data SSD Shared area Low-cost capacity KernelFS migrates cold data to lower layers HDD Shared area 18

48 Digest and migrate data in kernel Application LibFS Private operation log Logs Digest KernelFS NVM Shared area NVM data SSD Shared area Low-cost capacity KernelFS migrates cold data to lower layers HDD Shared area 18

49 Digest and migrate data in kernel Application LibFS Private operation log Logs Digest KernelFS NVM Shared area NVM data SSD Shared area Low-cost capacity KernelFS migrates cold data to lower layers HDD Shared area 18

50 Digest and migrate data in kernel Application LibFS Private operation log KernelFS NVM Shared area NVM data SSD Shared area Low-cost capacity KernelFS migrates cold data to lower layers HDD Shared area 18

51 Digest and migrate data in kernel Application LibFS Private operation log KernelFS NVM Shared area NVM data SSD Shared area HDD Shared area Low-cost capacity KernelFS migrates cold data to lower layers Handle device IO properties Migrate 1 GB blocks Avoid SSD garbage collection overhead 18

52 Digest and migrate data in kernel Application LibFS Private operation log KernelFS NVM Shared area NVM data SSD Shared area HDD Shared area Low-cost capacity KernelFS migrates cold data to lower layers Handle device IO properties Migrate 1 GB blocks Avoid SSD garbage collection overhead Resembles log-structured merge (LSM) tree 18

53 Read: hierarchical search Application LibFS Search order KernelFS 1 Private OP log 2 NVM Shared area Log data NVM data 3 SSD data SSD Shared area 4 HDD data HDD Shared area 19

54 Shared file access Leases grant access rights to applications [SOSP 89] Required for files and directories Function like lock, but revocable Exclusive writer, shared readers 20

55 Shared file access Leases grant access rights to applications [SOSP 89] Required for files and directories Function like lock, but revocable Exclusive writer, shared readers On revocation, LibFS digests leased data Private data made public before losing lease Leases serialize concurrent updates 20

56 Outline LibFS: Log operations to NVM at user-level Fast user-level access In-order, synchronous IO KernelFS: Digest and migrate data in kernel Asynchronous digest Transparent data migration Shared file access Evaluation 21

57 Experimental setup 2x Intel Xeon E CPU, 64 GB DRAM 400 GB NVMe SSD, 1 TB HDD Ubuntu LTS, Linux kernel Emulated NVM Use 40 GB of DRAM Performance model [Y. Zhang et al. MSST 2015] Throttle latency & throughput in software 22

58 Evaluation questions Latency: Does Strata efficiently support small, random writes? Does asynchronous digest have an impact on latency? Throughput: Strata writes data twice (logging and digesting). Can Strata sustain high throughput? How well does Strata perform when managing data across storage layers? 23

59 Related work NVM file systems PMFS[EuroSys 14]: In-place update file system NOVA[FAST 16]: log-structured file system EXT4-DAX: NVM support for EXT4 SSD file system F2FS[FAST 15]: log-structured file system 24

60 Microbenchmark: write latency Strata logs to NVM Compare to NVM kernel file systems: PMFS, NOVA, EXT4-DAX Strata, NOVA In-order, synchronous IO Atomic write PMFS, EXT4-DAX No atomic write Latency (us) Better Strata PMFS NOVA EXT4-DAX 128 B 1 KB 4 KB 16 KB IO size 25

61 Microbenchmark: write latency Strata logs to NVM Compare to NVM kernel file systems: PMFS, NOVA, EXT4-DAX Strata, NOVA In-order, synchronous IO Atomic write PMFS, EXT4-DAX No atomic write Avg.: 26% better Tail : 43% better Latency (us) Better Strata NOVA 128 B 1 KB 4 KB 16 KB IO size PMFS EXT4-DAX

62 Latency: persistent RPC Foundation of most servers Persist RPC data before sending ACK to client 60 Strata NOVA No persist PMFS EXT4-DAX 98 RPC over RDMA 40 Gb/s Infiniband NIC Latency (us) Better For small IO (1 KB) 25% slower than No persist 35% faster than PMFS 7x faster than EXT4-DAX KB 4 KB 64 KB RPC size (IO size) 26

63 Latency: persistent RPC Foundation of most servers Persist RPC data before sending ACK to client 60 Strata NOVA No persist PMFS EXT4-DAX 98 RPC over RDMA 40 Gb/s Infiniband NIC Latency (us) Better For small IO (1 KB) 25% slower than No persist 35% faster than PMFS 7x faster than EXT4-DAX 15 35% better KB 4 KB 64 KB RPC size (IO size)

64 Latency: LevelDB LevelDB (NVM) Key size: 16 B Value size: 1 KB 300,000 objects Workload causes asynchronous digests Fast user-level logging Latency (us) Better Strata PMFS NOVA EXT4-DAX Random write 25% better than PMFS Random read Tied with PMFS 0 Write sync. Write seq. Write rand. Overwrite Read rand. 27

65 Latency: LevelDB LevelDB (NVM) Key size: 16 B Value size: 1 KB 300,000 objects Workload causes asynchronous digests Fast user-level logging Latency (us) Better Strata PMFS NOVA EXT4-DAX % better Tied Random write 25% better than PMFS Random read Tied with PMFS 0 Write sync. Write seq. Write rand. Overwrite Read rand. 27

66 Latency: LevelDB LevelDB (NVM) Key size: 16 B Value size: 1 KB 300,000 objects Workload causes asynchronous digests Fast user-level logging Random write 25% better than PMFS Latency (us) Better Write sync. Strata NOVA Write seq. Write rand. PMFS EXT4-DAX % better IO latency not impacted by asynchronous digest Random read Tied with PMFS Overwrite Tied Read rand. 27

67 Evaluation questions Latency: Does Strata efficiently support small, random writes? Does asynchronous digest have an impact on latency? Throughput: Strata writes data twice (logging and digesting). Can Strata sustain high throughput? How well does Strata perform when managing data across storage layers? 28

68 Throughput: Varmail Mail server workload from Filebench Using only NVM files Read/Write ratio is 1:1 Write-ahead logging Log coalescing Application LibFS Create journal file Write data to journal Write data to database file Delete journal file Digest eliminates unneeded work Removes temporary durable writes KernelFS Write data to database file 29

69 Throughput: Varmail Mail server workload from Filebench Using only NVM files Read/Write ratio is 1:1 Write-ahead logging Log coalescing Application LibFS Create journal file Write data to journal Write data to database file Delete journal file Digest eliminates unneeded work Removes temporary durable writes KernelFS Write data to database file Strata PMFS NOVA EXT4-DAX Better 0K 100K 200K 300K 400K Throughput (op/s) 29% better 29

70 Throughput: Varmail Mail server workload from Filebench Using only NVM files Read/Write ratio is 1:1 Write-ahead logging Log coalescing Application LibFS Create journal file Write data to journal Write data to database file Delete journal file Digest eliminates unneeded work Removes temporary durable writes KernelFS Write data to database file Strata PMFS NOVA EXT4-DAX Better 0K 100K 200K 300K 400K Throughput (op/s) 29% better Log coalescing eliminates 86% of log entries, saving 14 GB of IO 29

71 Throughput: Varmail Mail server workload from Filebench Using only NVM No kernel file system has both low latency and high throughput: Log coalescing PMFS: better latency Application Digest eliminates unneeded work NOVA: better files throughput KernelFS Read/Write ratio is 1:1 Write-ahead logging Create journal file Write data to journal Write data to database file Delete journal file Removes temporary durable writes Strata achieves both low latency and high throughput LibFS Write data to database file Strata PMFS NOVA EXT4-DAX Better 0K 100K 200K 300K 400K Throughput (op/s) 29% better Log coalescing eliminates 86% of log entries, saving 14 GB of IO 29

72 Throughput: data migration File server workload from Filebench Working set starts at NVM, grows to SSD, HDD Read/Write ratio is 1:2 30

73 Throughput: data migration File server workload from Filebench Working set starts at NVM, grows to SSD, HDD Read/Write ratio is 1:2 User-level migration LRU: whole file granularity Treat each file system as a black-box NVM: NOVA, SSD: F2FS, HDD: EXT4 30

74 Throughput: data migration File server workload from Filebench Working set starts at NVM, grows to SSD, HDD Read/Write ratio is 1:2 User-level migration LRU: whole file granularity Treat each file system as a black-box NVM: NOVA, SSD: F2FS, HDD: EXT4 Block-level caching Linux LVM cache, formatted with F2FS 30

75 Throughput: data migration File server workload from Filebench Working set starts at NVM, grows to SSD, HDD Read/Write ratio is 1:2 User-level migration LRU: whole file granularity Treat each file system as a black-box NVM: NOVA, SSD: F2FS, HDD: EXT4 Block-level caching Linux LVM cache, formatted with F2FS Strata User-level migration Block-level caching 2x faster 0K 2K 4K 6K 8K 10K Avg. throughput (ops/s) 30

76 Throughput: data migration File server workload from Filebench Working set starts at NVM, grows to SSD, HDD Read/Write ratio is 1:2 User-level migration LRU: whole file granularity Treat each file system as a black-box NVM: NOVA, SSD: F2FS, HDD: EXT4 Block-level caching Linux LVM cache, formatted with F2FS 22% faster than user-level migration Cross layer optimization: placing hot metadata in faster layers Strata User-level migration Block-level caching 2x faster 0K 2K 4K 6K 8K 10K Avg. throughput (ops/s) 30

77 Conclusion Server applications need fast, small random IO on vast datasets with intuitive crash consistency Strata, a cross media file system, addresses these concerns Performance: low latency, high throughput Novel split of LibFS, KernelFS Fast user-level access Low-cost capacity: leverage NVM, SSD & HDD Asynchronous digest Transparent data migration with large, sequential IO Simplicity: intuitive crash consistency model In-order, synchronous IO Source code is available at 31

78 Backup 32

79 Device management overhead SSD, HDD prefer large sequential IO For example, SSD Random write: SSD Throughput (MB/s) MB 128 MB 256 MB 512 MB 1024 MB x difference by hardware GC SSD utilization Sequential writes avoid management overhead 33