File Systems Fated for Senescence? Nonsense, Says Science!

Transcription

1 File Systems Fated for Senescence? Nonsense, Says Science! Alex Conway, Ainesh Bakshi, Yizheng Jiao, Yang Zhan, Michael A. Bender, William Jannen, Rob Johnson, Bradley C. Kuszmaul, Donald E. Porter, Jun Yuan and Martin Farach-Colton Rutgers University, The University of North Carolina at Chapel Hill, Stony Brook University, Oracle Corporation and Massachusetts Institute of Technology, Farmingdale State College of SUNY

2 File Systems Fated for Senescence? Nonsense, Says Science; The Essence of Semperjuvenescense is Coalescence!

3 File Systems Fated for Senescence? Nonsense, Says Science; old age The Essence of Semperjuvenescense is Coalescence! merging together being young forever

4 File System Aging Aging is fragmentation over time Performance

5 In this talk Do file systems age? What can we do about it?

6 Is aging a problem?

7 Is aging a problem?

8 Is aging a problem? Chris Hoffman at howtogeek.com says: Linux s ext2, ext3, and ext4 file systems [are] designed to avoid fragmentation in normal use. If you do have problems with fragmentation on Linux, you probably need a larger hard disk.

9 Is aging a problem? Chris Hoffman at howtogeek.com says: Linux s ext2, ext3, and ext4 file systems [are] designed to avoid fragmentation in normal use. If you do have problems with fragmentation on Linux, you probably need a larger hard disk. Modern Linux filesystems keep fragmentation at a minimum Therefore it is not necessary to worry about fragmentation in a Linux system.

10 Is aging a problem? Chris Hoffman at howtogeek.com says: Linux s ext2, ext3, and ext4 file systems [are] designed to avoid fragmentation in normal use. If you do have problems with fragmentation on Linux, you probably need a larger hard disk. Nope Modern Linux filesystems keep fragmentation at a minimum Therefore it is not necessary to worry about fragmentation in a Linux system.

11 Is aging a problem?

12 Is aging a problem? Aging happens in real filesystems Smith and Seltzer ( 97) Benchmarks should incorporate aging Zhu, Chen and Chiueh ( 05) Agrawal, A. Arpaci-Dusseau and R. Arpaci-Dusseau ( 09) Yep

13 Is aging a problem? Nope Yep

14 Let s do some science!

15 Inducing Aging We use three different workloads Developer workload Server workload Synthetic workloads

16 Inducing Aging We use three different workloads Developer workload Server workload See the paper Synthetic workloads

17 Simulating a Developer

18 Simulating a Developer get coffee

19 Simulating a Developer get coffee git pull git pull

20 Simulating a Developer get coffee git pull make make git pull

21 Simulating a Developer get coffee git pull make get coffee make git pull

22 Simulating a Developer get coffee git pull make get coffee git pull make git pull

23 Simulating a Developer get coffee git pull make get coffee git pull git pull add awesome features

24 Simulating a Developer get coffee git pull make get coffee git pull git pull add awesome features get coffee

25 Simulating a Developer get coffee git pull make get coffee git pull git pull add awesome features get coffee git pull

26 Simulating a Developer get coffee git pull make get coffee git pull git pull add awesome features get coffee git pull fix bugs

27 Simulating a Developer get coffee git pull make get coffee git pull git pull add awesome features get coffee git pull fix bugs...

28 Simulating a Developer get coffee git pull make get coffee git pull git pull add awesome features get coffee git pull fix bugs... We can simulate a developer by replaying Git histories

29 Simulating a Developer

30 Simulating a Developer Use the Linux kernel repo from github.com Do 100 git pulls Measure Performance

31 Measuring Aging time grep -r random_string /path/to/filesystem dir file1 file2 file3 file4

32 Measuring Aging time grep -r random_string /path/to/filesystem dir file1 file2 file3 file4

33 Measuring Aging time grep -r random_string /path/to/filesystem dir file1 file2 file3 file4

34 Measuring Aging time grep -r random_string /path/to/filesystem dir file1 file2 file3 file4

35 Measuring Aging time grep -r random_string /path/to/filesystem dir file1 file2 file3 file4

36 Measuring Aging time grep -r random_string /path/to/filesystem dir file1 file2 file3 file4 Intrafile Fragmentation

37 Measuring Aging time grep -r random_string /path/to/filesystem dir file1 file2 file3 file4 Intrafile Fragmentation

38 Measuring Aging time grep -r random_string /path/to/filesystem dir file1 file2 file3 file4 Intrafile Fragmentation

39 Measuring Aging time grep -r random_string /path/to/filesystem dir file1 file2 file3 file4 Intrafile Fragmentation Interfile Fragmentation

40 Measuring Aging time grep -r random_string /path/to/filesystem dir file1 file2 file3 file4 Intrafile Fragmentation Interfile Fragmentation

41 Measuring Aging time grep -r random_string /path/to/filesystem dir file1 file2 file3 file4 Intrafile Fragmentation Interfile Fragmentation Then normalize per gigabyte read

42 Do modern file systems age?

43 Git Workload on ext4 on HDD 800 Lower is better Time in seconds / GiB x Git pulls performed Our Setup: Cold Cache, 3.4 GHz Quad Core, 4GiB RAM, 20 GiB HDD partition - SATA 7200 RPM

44 Git Workload on ext4 on HDD 800 Lower is better Time in seconds / GiB x 2x slowdown Git pulls performed Our Setup: Cold Cache, 3.4 GHz Quad Core, 4GiB RAM, 20 GiB HDD partition - SATA 7200 RPM

45 Git Workload on ext4 on HDD 800 Lower is better Time in seconds / GiB x slowdown 14.3x 2x slowdown Git pulls performed Our Setup: Cold Cache, 3.4 GHz Quad Core, 4GiB RAM, 20 GiB HDD partition - SATA 7200 RPM

46 Git Workload on ext4 on HDD 800 Lower is better Time in seconds / GiB minutes to grep 1.2GiB 14.3x Git pulls performed Our Setup: Cold Cache, 3.4 GHz Quad Core, 4GiB RAM, 20 GiB HDD partition - SATA 7200 RPM

47 How can we be sure this slowdown is due to aging?

48 How can we be sure this slowdown is due to aging? I m not old. My directory structure is different!

49 File System Rejuvenation Idea: Copy same logical state to a new file system After each 100 pulls Compare grep cost

50 Aging ext4 with Git on HDD 800 Lower is better Time in seconds / GiB Aged 8.8x Unaged Git pulls performed

51 Aging ext4 with Git on HDD 800 Lower is better Aged Time in seconds / GiB Smaller average file size makes the unaged 60% slower 8.8x Unaged Git pulls performed

52 Is this specific to ext4?

53 Aging other file systems with Git on HDD Btrfs F2FS x 22.4x Lower is better weird unaged behavior on XFS XFS ZFS x x

54 Will SSDs save us?

55 Git Workload on XFS on SSD 30 Lower is better Aged Time in seconds / GiB x Unaged Git pulls performed

56 Git Workload on SSD Btrfs ext x Lower is better F2FS ZFS x

57 Git Workload on SSD Btrfs ext x Lower is better ZFS and ext4 slow down with smaller average file size F2FS ZFS x

58 Git Workload on SSD Btrfs ext x Lower is better ZFS and ext4 slow down with smaller average file size F2FS ZFS x Told ya! 0 0

59 Aging is real Btrfs, ext4, F2FS, XFS, ZFS all age Up to 22x on HDD Up to 2x on SSD Git lets us replay a real development history Induce aging by simulating years of use Takes between 5 hours and 2 days Download these scripts from betrfs.org

60 How can we prevent aging?

61 Design goals to address fragmentation Intrafile Fragmentation: Avoid breaking large files into small fragments

62 Design goals to address fragmentation Intrafile Fragmentation: Avoid breaking large files into small fragments Interfile Fragmentation: Cluster logically related small files

63 Design goals to address fragmentation Intrafile Fragmentation: Avoid breaking large files into small fragments Interfile Fragmentation: Cluster logically related small files What do we mean by small?

64 I/O Size vs Effective Bandwidth Read Length vs Bandwidth 1000 Higher is better Bandwidth in MiB/sec HDD 0.1 4KiB 8KiB 16KiB 32KiB 64KiB 128KiB 256KiB 512KiB 1MiB 2MiB 4MiB Sequential Read Length 8MiB 16MiB 32MiB 64MiB 128MiB 256MiB

65 I/O Size vs Effective Bandwidth Read Length vs Bandwidth 1000 Higher is better SSD Bandwidth in MiB/sec HDD 0.1 4KiB 8KiB 16KiB 32KiB 64KiB 128KiB 256KiB 512KiB 1MiB 2MiB 4MiB Sequential Read Length 8MiB 16MiB 32MiB 64MiB 128MiB 256MiB

66 I/O Size vs Effective Bandwidth Read Length vs Bandwidth 1000 Higher is better SSD Bandwidth in MiB/sec HDD 0.1 4KiB 8KiB 16KiB 32KiB 64KiB 128KiB 256KiB 512KiB 1MiB 2MiB 4MiB Sequential Read Length 8MiB 16MiB 32MiB 64MiB 128MiB 256MiB

67 Design goals to address fragmentation Intrafile Fragmentation: Avoid breaking large files into small fragments Interfile Fragmentation: Cluster logically related small files Prediction: 4MiB chunks will substantially reduce aging

68 Testing this with Btrfs

69 Btrfs: Larger leaves = less aging? Metadata and small files are stored in a B-tree Large files get written elsewhere Big File Bigger File Large File

70 Btrfs Leaf Size Performance Btrfs allows leaf size to be configured between 4KiB and 64KiB. 600 Bigger leaves does mean 4k less aging! Time in seconds / GiB k 16k 32k 64k lower is better Git pulls performed

71 Cost of large leaves Why don t B-tree usually have big leaves? Because making small changes to big leaves causes a lot of writing

72 Btrfs Leaf Size Writing Btrfs allows leaf size to be configured between 4KiB and 64KiB. 300 Bigger leaves do mean Blocks Written in Thousands more writing lower is better 64k 32k 16k 8k 4k Git pulls performed

73 B-Tree Performance Tradeoff Small Leaves Large Leaves More Aging Less Aging Less Writing More Writing

74 B-Tree Performance Tradeoff Small Leaves Large Leaves More Aging Less Aging Less Writing More Writing This tradeoff is inherent to B-trees

75 Other File System Types Must other types of file systems age? Update-in-place See the paper Log-structured Write-Optimized BεtrFS

76 BεtrFS BεtrFS packs small logically related data in a B ε -tree with 4MiB nodes.

77 BεtrFS BεtrFS packs small logically related data in a B ε -tree with 4MiB nodes.

78 BεtrFS BεtrFS packs small logically related data in a B ε -tree with 4MiB nodes. B ε -trees batch updates which allows leaves to be big without increasing the amount of writing

79 Git on BetrFS on HDD 800 Lower is better ZFS Time in seconds / GiB F2FS ext4 XFS Btrfs XFS ext4/f2fs/zfs Btrfs BetrFS Git pulls performed Aged Unaged

80 Git on BetrFS on HDD 80 Lower is better ext4 F2FS Time in seconds / GiB ZFS Btrfs BetrFS Git pulls performed Aged Unaged

81 Git on BetrFS on HDD 80 Lower is better ext4 F2FS Time in seconds / GiB ZFS Btrfs BetrFS Git pulls performed Aged Unaged

82 And SSDs?

83 Git on BetrFS on SSD 30 Lower is better ZFS Time in seconds / GiB ZFS ext4 F2FS/XFS/ext4 Btrfs F2FS XFS Btrfs BetrFS Git pulls performed Aged Unaged

84 How to prevent aging Rewrite to keep related data in large blocks Batch updates to avoid too much writing

85 Conclusion It s easy to age file systems quickly and substantially Aging is avoidable

86 Thank you! Alex Conway betrfs.org