BetrFS: A Right-Optimized Write-Optimized File System

Transcription

1 BetrFS: A Right-Optimized Write-Optimized File System Amogh Akshintala, Michael Bender, Kanchan Chandnani, Pooja Deo, Martin Farach-Colton, William Jannen, Rob Johnson, Zardosht Kasheff, Bradley C. Kuszmaul, Prashant Pandey, Donald E. Porter, Leif Walsh, Jun Yuan, Yang Zhan Facebook, Farmingdale College, MIT & Oracle, Rutgers, Stony Brook, Two Sigma, UNC, Williams College

2 General-purpose file-systems strive to perform well on a wide variety of applications Sequential reads Sequential writes Random writes File/directory renames File deletes Recursive scans Metadata updates

3 Achieving good performance on all these operations is a long-standing challenge Sequential reads Sequential writes Random writes File/directory renames File deletes Recursive scans Metadata updates Example: ext4

4 Achieving good performance on all these operations is a long-standing challenge Sequential reads Sequential writes Random writes File/directory renames File deletes Recursive scans Metadata updates Example: log-based file systems Logging updates is fast, but logged data can have little locality

5 Some operations seem to require a trade-off sequential reads update-in-place renames inodes random writes log structured directory scans full-path indexing

6 Main idea of this talk BetrFS

7 Write-Optimized Data Structures (WODS) New class of data structures discovered in 90's LSM trees [O'Neil, Cheng, Gawlick, & O'Neil '96] BetrFS uses ε B -trees ε B -trees [Brodal & Fagerberg '03] COLAs [Bender, Farach-Colton, Fineman, Fogel, Kuszmaul, & Nelson '07] xdicts [Brodal, Demaine, Fineman, Iacono, Langerman, & Munro '10] WODS perform inserts/updates/deletes orders-ofmagnitude-faster than in a B-tree WODS queries are asymptotically no slower than in a B-tree

8 The Disk-Access Machine (DAM) model [Aggarwal & Vitter '88] How computation works: Data is transferred in blocks between RAM and disk. The number of block transfers dominates the running time. Goal: Minimize # of block transfers Performance bounds are parameterized by block size B, memory size M, data size N. B RAM M B Disk

9 Example: B-trees B. B B B children. B... Insert cost: O ( log B N ) O (log B N ) B Lookup cost: O ( log B N )

10 B -trees (ε=1/2) ε buffer space B B B B. Bchildren. Range queries can run at disk bandwidth faster than in a B-tree Inserts are B B B orders-of-magnitude B Insert cost: O (... B log N =O log B N B B B B ) ( ) B B B Point queries are B N ) O (log asymptotically as fast as in a B-tree B Lookup cost: Range query cost: O( log B N )=O( log B N ) O( log B N + k / B )

11 The search-insert asymmetry Inserts are orders-of-magnitude faster than point queries Point query: O ( log B N ) Insert: O ( log B N B ) But many updates require querying the old value first e.g. Add $10 to rob's account balance OldBalance = query(rob) NewBalance = oldbalance + $10 insert(newbalance) 11

12 Upserts: read-modify-write as fast as an insert rob: $15 B B B B B B B rob: $5 B rob:add $10. Bchildren buffer space.... B B B B O (log B N )

13 Bε-tree performance summary Point query O ( log B N ) Insert/delete/upsert log B N O B Range query ( As fast as a B-tree ) O ( log B N +k / B ) Very fast (10K-100K per second) Near disk bandwidth To get the best possible performance, we want to do Inserts, deletes, upserts, and range queries, and avoid point queries.

14 The BetrFS schema (version 0.1) Maintain two separate Bε-tree indexes: metadata index: full path > struct stat data index: (full path, blk#) > data[4096] Implications: Fast directory scans Data blocks are laid out sequentially

15 Mapping file-system operations to key-value operations Operation Implementation read range query write insert/upsert metadata update upsert readdir range query mkdir/rmdir insert/delete unlink rename Fast atime updates Fast directory traversals Do not map onto *delete each block single key-value *delete then reinsert each blockoperations

16 Small, random, unaligned writes are an order-of-magnitude faster Random 4 byte writes 1 GiB file, random data 1,000 random 4-byte writes fsync() at end Log scale Time (s) sec vs > 10sec BetrFS btrfs ext4 xfs zfs BetrFS random writes ε 0.1 *lower is better benefit from B -tree insertion performance

17 Small file creates are an order-of-magnitude faster Create 3 million files and write Small File Creation bytes to each Files/second Log scale Balanced directory tree with fanout BetrFS btrfs ext4 xfs zfs 1000 BetrFS file creates M 2M Files Created *higher is better 3M ε benefit from B -tree insertion performance

18 Sequential I/O 1GiB Sequential I/O 4K-blocks at a time Sequentially read data back MiB/s Write random data to file, 10 BetrFS btrfs ext4 xfs zfs BetrFS sequential reads 0 from Bε-tree range benefit read query performance Operation *higher is better write Mostly overhead of full-data journaling (we'll fix this later in the talk)

19 Recursive directory traversals GNU Find Recursive scans from root of grep r Linux source About 3-8x faster than other file systems BetrFS Time (s) Time (s) 15 GNU find scans file metadata grep r scans file contents btrfs 40 ext4 xfs zfs 5 20 BetrFS directory traversals 0 0 Lower is better ε benefit from B -tree range-query performance

20 File deletion BetrFS Delete Scaling Write random data to file, 300 fsync() it Delete file Time (s) 200 BetrFS File Size Lower is better BetrFS deletes require O(n) key-value operations

21 Directory rename Renames are orders-of-magnitude slower than in ext4 Lower is better BetrFS renames require O(n) key-value operations

22 BetrFS (version 0.1) performance summary Sequential reads Sequential writes Random writes File/directory renames File deletes Recursive scans Metadata updates Let's fix these problems

23 Accelerating rename without slowing down directory traversals

24 Full-path indexing yields fast directory scans home Example: grep -r key /home/rob/doc/ rob disk head a.t e ex b.t x 2.jpg p4 1.m r.c ba late x vid eo local c do.. /home/rob/doc /home/rob/doc/latex /home/rob/doc/latex/a.tex /home/rob/doc/latex/b.tex /home/rob/doc/bar.c /home/rob/local. Directory Tree (logical) Disk (physical)

25 Rename is expensive when using full-path indexing home Example: mv /home/rob/doc/latex /home/rob rob a.t e ex b.t x 2.jpg p4 1.m r.c ba late x late x vid eo local c do.. /home/rob/doc /home/rob/doc/latex /home/rob/doc/latex/a.tex /home/rob/doc/latex/b.tex /home/rob/doc/bar.c /home/rob/local. /home/rob/latex /home/rob/latex/a.tex /home/rob/latex/b.tex Directory Tree (logical) Disk (physical)

26 Zoning:The balancing indirection and tension between fast rename andlocality fast scan Scan cost Rename cost Indirection Zones Locality

27 BetrFS v0.2 rethinking the schema Zone: a subtree of the directory hierarchy home rob Partition file system into zones Use full-path indexing within zones Use inodes between zones a.t e x 2.jpg ex b.t Zone 0 Zone 1 p4 1.m r.c ba late x vid eo local c do Zone 2 Implication: Recursive directory scans only perform seeks when crossing zones

28 BetrFS v0.2 rethinking the schema Moving the root of a zone is cheap home Example: mv /home/rob/video/1.mp4 /home/rob/doc rob late x a.t ex p4 Zone 0 Zone 1 p4 1.m 2.jpg r.c ba ex b.t 1. m vid eo local c do Zone 2

29 BetrFS v0.2 rethinking the schema Renaming a subtree of a zone requires copying home Example: mv /home/rob/doc/latex /home/rob/latex rob lat ex late x a.t ex 2.jpg r.c ba ex b.t 1. m vid eo local c do p4 Zone 0 Zone 1 Zone 2

30 BetrFS v0.2 rethinking the schema Managing zone sizes home Large zones fast directory scans Small zones fast renames rob lat ex a.t ex ex b.t 2.jpg r.c ba 1. m vid eo local c do p4 Zone 0 Zone 1 We can keep zone sizes in a sweet spot by splitting large zones and merging small zones Zone 2

31 How big should zones be? Cost of renaming via copy Cost of renaming root of a zone BetrFS-0.2 uses 512KB zones to balance rename and scan performance

32 BetrFS 0.2 rename performance 21 seconds Rename linux source tree Rename performance is comparable to other file systems

33 Performing sequential writes at disk bandwidth and with full data journaling semantics

34 BetrFS version 0.1 writes everything twice root root (k2, a v2 ) b b (k1, time insert(k1, v1 ) v1 ) Log insert(k2, v2 ) 34

35 BetrFS version 0.2: late-binding journal root root (k2, v2 a ) b b unbound(k1, v1 ) time Unboundinsert(k1) insert(k2, v2 Log )... bind(, ) 35

36 Why don't we use late-binding for small writes? Reason 1: Late-binding requires writing out a large (e.g. 4MB) node For small writes, this is huge write-amplification It's more efficient to make small writes durable by simply logging them Reason 2: Small, random writes get written to disk several times as they get flushed down the Bε-tree So writing them to the log is not a big extra cost

37 Late-binding journal: performance evaluation Sequential write Fast sequential writes with full data journaling

38 File deletions require inserting a single message and enable efficient garbage collection Rangecast delete Delete /foo/* Garbage collected x o/ /a o /fo /fo / /bar /goo /a

39 Rangecast delete performance evaluation File delete Constant latency, about 30% faster than ext4

40 Is BetrFS still fast at other operations?

41 BetrFS still has metadata updates almost 100x faster than ext4 BetrFS still performs random writes orders of magnitude faster than other file systems Zone splits

42 What about real application performance?

43 Macrobenchmark: git git clone Performance comparable to other file systems git diff Recursive scan performance pays off in real applications

44 Macrobenchmark: dovecot imap maildir workload Payoff of improved delete and rename performance

45 BetrFS (version 0.2) performance summary Sequential reads Sequential writes Random writes File/directory renames File deletes Recursive scans Metadata updates

46 Conclusion Write-optimized data structures can enable us to overcome long-standing file-system design trade-offs Write-optimized file systems can offer across-theboard top-of-the-line performance Write-optimization creates a need/opportunity to revisit many file system design issues Code available at betrfs.org

47 SSD performance preview 6x speedup Still work to do