A Study of Linux File System Evolution

Transcription

1 A Study of Linux File System Evolution Lanyue Lu Andrea C. Arpaci-Dusseau Remzi H. Arpaci-Dusseau Shan Lu University of Wisconsin - Madison

2 Local File Systems Are Important

3 Local File Systems Are Important Windows Mac Linux

4 Local File Systems Are Important Windows Mac Linux

5 Local File Systems Are Important Hadoop DFS Google GFS Windows Mac Linux

6 Local File Systems Are Important Hadoop DFS Google GFS Windows Mac Linux

7 Local File Systems Are Important Hadoop DFS Android iphone Google GFS Windows Mac Linux

8 Local File Systems Are Important Hadoop DFS Android iphone Google GFS Windows Mac Linux

9 Why Study Is Useful?

10 Why Study Is Useful? Study drives system designs previous work focuses on measurements little emphasis on system evolution

11 Why Study Is Useful? Study drives system designs previous work focuses on measurements little emphasis on system evolution Answer important questions complexity of file systems dominant bug types performance optimizations reliability enhancements similarities across file systems

12 Who Is This Study Useful To?

13 Who Is This Study Useful To? File system developers avoid same mistakes improve existing design and implementation

14 Who Is This Study Useful To? File system developers avoid same mistakes improve existing design and implementation System researchers identify problems that plague existing systems match research to reality

15 Who Is This Study Useful To? File system developers avoid same mistakes improve existing design and implementation System researchers identify problems that plague existing systems match research to reality Tool builders large-scale statistical bug patterns effective bug-finding tools realistic fault injection

16 How We Studied?

17 How We Studied? File systems are evolving code base is not static new features, bug-fixings performance and reliability improvement

18 How We Studied? File systems are evolving code base is not static new features, bug-fixings performance and reliability improvement Patches describe evolution how one version transforms to the next every patch is available system archeology

19 How We Studied? File systems are evolving code base is not static new features, bug-fixings performance and reliability improvement Patches describe evolution how one version transforms to the next every patch is available system archeology Study with other rich information source code, design documents forum, mailing lists

20 What We Did?

21 What We Did? Manual patch inspection XFS, Ext4, Btrfs, Ext3, Reiserfs, JFS Linux 2.6 series 5079 patches, multiple passes

22 What We Did? Manual patch inspection XFS, Ext4, Btrfs, Ext3, Reiserfs, JFS Linux 2.6 series 5079 patches, multiple passes Quantitatively analyze in various aspects patch types, bug patterns and consequence performance and reliability techniques

23 What We Did? Manual patch inspection XFS, Ext4, Btrfs, Ext3, Reiserfs, JFS Linux 2.6 series 5079 patches, multiple passes Quantitatively analyze in various aspects patch types, bug patterns and consequence performance and reliability techniques Provide an annotated dataset rich data for further analysis

24 Major Results Preview

25 Major Results Preview Bugs are prevalent

26 Major Results Preview Bugs are prevalent Semantic bugs dominate

27 Major Results Preview Bugs are prevalent Semantic bugs dominate Bugs are constant

28 Major Results Preview Bugs are prevalent Semantic bugs dominate Bugs are constant Corruption and crash are most common

29 Major Results Preview Bugs are prevalent Semantic bugs dominate Bugs are constant Corruption and crash are most common Metadata management has high bug density

30 Major Results Preview Bugs are prevalent Semantic bugs dominate Bugs are constant Corruption and crash are most common Metadata management has high bug density Failure paths are error-prone

31 Major Results Preview Bugs are prevalent Semantic bugs dominate Bugs are constant Corruption and crash are most common Metadata management has high bug density Failure paths are error-prone Various performance techniques are used

32 Outline Introduction Methodology Study Results

33 Methodology

34 Methodology Diverse: XFS Ext4 Btrfs Ext3 Reiser JFS

35 Methodology Diverse: XFS Ext4 Btrfs Ext3 Reiser JFS Complete: Linux to Dec Patches May. 2011

36 Methodology Diverse: XFS Ext4 Btrfs Ext3 Reiser JFS Complete: Linux to Dec Patches May Header Comprehensive: Patch Description Code

37 1 Patch Header [PATCH] fix possible NULL pointer in ext3/super.c.

38 2 Patch Description In fs/ext3/super.c::ext3_get_journal() at line 1675, `journal' can be NULL, but it is not handled right (detect by Coverity's checker).

39 3 Related Code --- /fs/ext3/super.c +++ /fs/ext3/super.c -1675, ,7 journal_t *ext3_get_journal() 1 if (!journal) { 2 printk(kern_err "EXT3: Could not load"); 3 iput(journal_inode); 4 } 5 journal->j_private = sb;

40 3 Related Code --- /fs/ext3/super.c +++ /fs/ext3/super.c -1675, ,7 journal_t *ext3_get_journal() 1 if (!journal) { 2 printk(kern_err "EXT3: Could not load"); 3 iput(journal_inode); 4 } 5 journal->j_private = sb;

41 3 Related Code --- /fs/ext3/super.c +++ /fs/ext3/super.c -1675, ,7 journal_t *ext3_get_journal() 1 if (!journal) { 2 printk(kern_err "EXT3: Could not load"); 3 iput(journal_inode); return NULL; 4 } 5 journal->j_private = sb;

42 Classifications

43 Classifications Patch overview type: bug size: 1

44 Classifications Patch overview type: bug size: 1 Bug analysis pattern: memory (nullptr) consequence: crash data structure: super tool: Coverity

45 Classifications Patch overview type: bug size: 1 Bug analysis pattern: memory (nullptr) consequence: crash data structure: super tool: Coverity Performance and reliability pattern location

46 Limitations

47 Limitations Only six popular file systems many other file systems

48 Limitations Only six popular file systems many other file systems Only Linux 2.6 major versions omit earlier versions

49 Limitations Only six popular file systems many other file systems Only Linux 2.6 major versions omit earlier versions Only reported bugs existing, but unknown bugs

50 Outline Introduction Methodology Study Results

51 Questions to Answer What do patches do? What do bugs look like? Do bugs diminish over time? What consequences do bugs have? Where does complexity of file systems lie? Do bugs occur on normal paths? What performance techniques are used?

52 Q1: What do patches do?

53 Patch Overview

54 Patch Overview Type Description Bug Fix existing bugs Performance Propose efficient design or implementation Reliability Improve robustness Feature Add new functionality Maintenance Maintain the code and documentation

55 Patch Overview 100% % 60% 40% 20% 0% XFS Ext4 Btrfs Ext3 Reiser JFS All

56 Patch Overview 100% % 60% 40% 20% 0% XFS Ext4 Btrfs Ext3 Reiser JFS All

57 Patch Overview 100% % 60% 40% 20% 0% XFS Ext4 Btrfs Ext3 Reiser JFS All

58 Patch Overview 100% % 60% 40% 20% 0% XFS Ext4 Btrfs Ext3 Reiser JFS All

59 Patch Overview 100% % 60% 40% 20% 0% XFS Ext4 Btrfs Ext3 Reiser JFS All

60 Patch Overview 100% % 60% 40% 20% Bug 0% XFS Ext4 Btrfs Ext3 Reiser JFS All

61 Patch Overview 100% % 60% 40% 20% Performance Bug 0% XFS Ext4 Btrfs Ext3 Reiser JFS All

62 Patch Overview 100% % 60% 40% 20% Reliability Performance Bug 0% XFS Ext4 Btrfs Ext3 Reiser JFS All

63 Patch Overview 100% % 60% 40% 20% Feature Reliability Performance Bug 0% XFS Ext4 Btrfs Ext3 Reiser JFS All

64 Patch Overview 100% % 60% 40% 20% Maintenance Feature Reliability Performance Bug 0% XFS Ext4 Btrfs Ext3 Reiser JFS All

65

66 45% of patches are for maintenance

67 45% of patches are for maintenance 35% of patches are bug fixing

68 45% of patches are for maintenance 35% of patches are bug fixing

69 Q2: What do bugs look like?

70 Bug Pattern

71 Bug Pattern Type Description Semantic Concurrency Memory Error Code Incorrect design or implementation (e.g. incorrect state update, wrong design) Incorrect concurrent behavior (e.g. miss unlock, deadlock) Incorrect handling of memory objects (e.g. resource leak, null dereference) Missing or wrong error code handling (e.g. return wrong error code)

72 Semantic Bug Example ext3/ialloc.c, find_group_other(...){ group = parent_group + 1; for (i = 2; i < ngroups; i++) { } }......

73 Semantic Bug Example ext3/ialloc.c, find_group_other(...){ group = parent_group; for (i = 0; i < ngroups; i++) { } }......

74 Bug Pattern 100% % 60% 40% 20% Semantic 0% XFS Ext4 Btrfs Ext3 Reiser JFS All

75 Bug Pattern 100% % 60% 40% 20% Semantic 0% XFS Ext4 Btrfs Ext3 Reiser JFS All

76 Bug Pattern 100% % 60% 40% 20% Semantic 0% XFS Ext4 Btrfs Ext3 Reiser JFS All

77 Concurrency Bug Example ext4/extents.c, ext4_ext_put_in_cache(...){ cex = &EXT4_I(inode)->i_cached_extent; 2 cex->ec_foo = FOO; }

78 Concurrency Bug Example ext4/extents.c, ext4_ext_put_in_cache(...){ spin_lock(i_br_lock); 1 cex = &EXT4_I(inode)->i_cached_extent; 2 cex->ec_foo = FOO; } spin_unlock(i_br_lock);

79 Bug Pattern 100% % 60% 40% 20% Concurrency Semantic 0% XFS Ext4 Btrfs Ext3 Reiser JFS All

80 Memory Bug Example btrfs/inode, btrfs_new_inode(...){ 1 inode = new_inode(...); 2 ret = btrfs_set_inode_index(...); 3 if (ret){ 4 return ERR_PTY(ret); } }

81 Memory Bug Example btrfs/inode, btrfs_new_inode(...){ 1 inode = new_inode(...); 2 ret = btrfs_set_inode_index(...); 3 if (ret){ 4 return ERR_PTY(ret); } }

82 Memory Bug Example btrfs/inode, btrfs_new_inode(...){ 1 inode = new_inode(...); 2 ret = btrfs_set_inode_index(...); 3 if (ret){ 4 return ERR_PTY(ret); } }

83 Memory Bug Example btrfs/inode, btrfs_new_inode(...){ 1 inode = new_inode(...); 2 ret = btrfs_set_inode_index(...); 3 if (ret){ iput(inode); 4 return ERR_PTY(ret); } }

84 Bug Pattern 100% % 60% Memory 40% 20% Concurrency Semantic 0% XFS Ext4 Btrfs Ext3 Reiser JFS All

85 Error Code Example reiserfs/xattr_acl.c, reiserfs_get_acl(...) { acl = posix_acl_from_disk(...); 2 *p_acl = posix_acl_dup(acl); }

86 Error Code Example reiserfs/xattr_acl.c, reiserfs_get_acl(...) { acl = posix_acl_from_disk(...); if (!IS_ERR(acl)) 2 *p_acl *p_acl = posix_acl_dup(acl); = posix_acl_dup(acl); }

87 Bug Pattern 100% % 60% Error Code Memory 40% 20% Concurrency Semantic 0% XFS Ext4 Btrfs Ext3 Reiser JFS All

88 55% of file-system bugs are semantic bugs

89 Q3: Do bugs diminish over time?

90 Ext3 Bug Trend 15 Number of Bugs Linux Versions

91 Ext3 Bug Trend : block reservation : xttra in inode Number of Bugs Linux Versions

92 Ext3 Bug Trend : multiple block allocation Number of Bugs Linux Versions

93 Ext3 Bug Trend : miss error handling Number of Bugs Linux Versions

94 Bug Trend 40 XFS Ext4 80 Btrfs Number of Bugs Ext ReiserFS JFS Linux Version

95 Bug Trend 40 XFS Ext4 80 Btrfs Number of Bugs Ext ReiserFS JFS Linux Version

96 Bug Trend 40 XFS Ext4 80 Btrfs Number of Bugs Ext ReiserFS JFS Linux Version

97 Bug Trend 40 XFS Ext4 80 Btrfs Number of Bugs Ext ReiserFS JFS Linux Version

98 Semantic Concurrency Memory Error Code 40 XFS Ext4 80 Btrfs Number of Bugs Ext ReiserFS JFS Linux Version

99 Semantic Concurrency Memory Error Code Number of Bugs 40 XFS Ext Ext Btrfs : remove BKL 40 ReiserFS JFS Linux Version

100 Bug-fixing is a Constant in a file system s lifetime

101 Q4: What consequences do file-system bugs have?

102 Bug Consequence

103 Bug Consequence Type Description Corruption On-disk or in-memory data is corrupted Crash File system becomes unusable Error Unexpected operation failure or error code Deadlock Wait for resources in circular chain Hang File system makes no progress Leak Resources are not freed properly Wrong Diverts from expectation (exclude above)

104 Bug Consequence 100% % 60% 40% 20% 0% XFS Ext4 Btrfs Ext3 Reiser JFS All Corruption

105 Bug Consequence 100% % 60% 40% 20% 0% XFS Ext4 Btrfs Ext3 Reiser JFS All Crash Corruption

106 Bug Consequence 100% % 60% 40% 20% 0% XFS Ext4 Btrfs Ext3 Reiser JFS All Error Crash Corruption

107 Bug Consequence 100% % 60% 40% 20% 0% XFS Ext4 Btrfs Ext3 Reiser JFS All Deadlock Error Crash Corruption

108 Bug Consequence 100% % 60% 40% 20% 0% XFS Ext4 Btrfs Ext3 Reiser JFS All Hang Deadlock Error Crash Corruption

109 Bug Consequence 100% % 60% 40% 20% 0% XFS Ext4 Btrfs Ext3 Reiser JFS All Leak Hang Deadlock Error Crash Corruption

110 Bug Consequence 100% % 60% 40% 20% 0% XFS Ext4 Btrfs Ext3 Reiser JFS All Wrong Leak Hang Deadlock Error Crash Corruption

111 Corruption and Crash are most common

112 Q5: Does each logical component have an equal degree of complexity?

113 Components

114 Components Type Description balloc Data block allocation and deallocation dir Directory management extent Contiguous physical blocks mapping file File read and write operations inode Inode-related metadata management transaction Journaling or other transactional support super Superblock-related metadata management tree Generic tree structure procedures other Other supporting components (e.g., xattr)

115 Correlation 0.4 XFS 0.4 Ext4 0.4 Btrfs Percentage of Bugs Ext ReiserFS JFS Percentage of Code

116 Correlation file inode super 0.4 XFS 0.4 Ext4 0.4 Btrfs Percentage of Bugs Ext ReiserFS JFS Percentage of Code

117 Correlation file inode super trans 0.4 XFS 0.4 Ext4 0.4 Btrfs Percentage of Bugs Ext ReiserFS JFS Percentage of Code

118 Correlation file inode super trans tree 0.4 XFS 0.4 Ext4 0.4 Btrfs Percentage of Bugs Ext ReiserFS JFS Percentage of Code

119 Correlation file balloc inode dir super extent trans other tree 0.4 XFS 0.4 Ext4 0.4 Btrfs Percentage of Bugs Ext ReiserFS JFS Percentage of Code

120

121 Metadata management has high bug density

122 Metadata management has high bug density Tree related code is not particularly error-prone

123 Q6: Do bugs occur at normal paths or failure paths?

124 Failure Path

125 Failure Path A wide range of failures resource allocation failure I/O operation failure silent data corruption incorrect system states

126 Failure Path A wide range of failures resource allocation failure I/O operation failure silent data corruption incorrect system states Unique code style goto statement error code propagation

127 A Semantic Bug on Failure Path ext4/resize.c, ext4_group_extend( ) { if (count!= ext4_blocks_count(es)){ 2 ext4_warning("multiple resizers run on filesystem!"); 3 err = -EBUSY; 4 goto exit_put; } }

128 A Semantic Bug on Failure Path ext4/resize.c, ext4_group_extend( ) { if (count!= ext4_blocks_count(es)){ 2 ext4_warning("multiple resizers run on filesystem!"); 3 err = -EBUSY; 4 goto exit_put; } }

129 A Semantic Bug on Failure Path ext4/resize.c, ext4_group_extend( ) { if (count!= ext4_blocks_count(es)){ 2 ext4_warning("multiple resizers run on filesystem!"); 3 err = -EBUSY; 4 goto exit_put; } } ext4_journal_stop(handle);

130 A Memory Bug on Failure Path ext4/inode.c, ext4_read_inode(struct inode * inode) { if (inode_is_bad) { 2 goto bad_inode; } }

131 A Memory Bug on Failure Path ext4/inode.c, ext4_read_inode(struct inode * inode) { if (inode_is_bad) { 2 goto bad_inode; } }

132 A Memory Bug on Failure Path ext4/inode.c, ext4_read_inode(struct inode * inode) { if (inode_is_bad) { brelse(bh); 2 goto bad_inode; } }

133 Bugs on Failure Paths Percentage of total bugs 100% 80% 60% 40% 20% % XFS Ext4 Btrfs Ext3 Reiser JFS All

134 38% of bugs are on failure paths

135 Q7: What performance techniques are used by file systems?

136 Performance

137 Performance Type Description Synchronization Improve synchronization efficiency Access Optimization Apply smarter access strategies Scheduling Improve I/O operations scheduling Scalability Scale on-disk and in-memory structures Locality Overcome sub-optimal block allocation Other Other performance improvement (e.g., reducing function stack usage)

138 Synchronization Example ext4/extents.c, ext4_fiemap(...){ 1 down_write(&ext4_i(inode)->sem); 2 error = ext4_ext_walk_space(...); 3 up_write(&ext4_i(inode)->sem); }

139 Synchronization Example ext4/extents.c, ext4_fiemap(...){ 1 down_write(&ext4_i(inode)->sem); 2 error = ext4_ext_walk_space(...); 3 up_write(&ext4_i(inode)->sem); }

140 Synchronization Example ext4/extents.c, ext4_fiemap(...){ 1 down_write(&ext4_i(inode)->sem); down_read(&ext4_i(inode)->sem); 2 error = ext4_ext_walk_space(...); 3 up_write(&ext4_i(inode)->sem); up_read(&ext4_i(inode)->sem); }

141 Performance 100% % 60% 40% 20% 0% XFS Ext4 Btrfs Ext3 Reiser JFS All Sync

142 Access Optimization Example btrfs/free-space-cache.c, btrfs_find_space_cluster( ) /* start to search for blocks */

143 Access Optimization Example btrfs/free-space-cache.c, btrfs_find_space_cluster( ) if (free_space < min_bytes) { spin_unlock(&tree_lock); return -ENOSPC; } /* start to search for blocks */

144 Performance 100% % 60% 40% 20% 0% XFS Ext4 Btrfs Ext3 Reiser JFS All Access Opt Sync

145 Performance 100% % 60% 40% 20% 0% XFS Ext4 Btrfs Ext3 Reiser JFS All Scheduling Access Opt Sync

146 Performance 100% % 60% Scalability 40% 20% 0% XFS Ext4 Btrfs Ext3 Reiser JFS All Scheduling Access Opt Sync

147 Performance 100% % Locality 60% Scalability 40% 20% 0% XFS Ext4 Btrfs Ext3 Reiser JFS All Scheduling Access Opt Sync

148 Performance 100% 80% Other Locality 60% Scalability 40% 20% 0% XFS Ext4 Btrfs Ext3 Reiser JFS All Scheduling Access Opt Sync

149 A wide variety of same performance techniques exist in all file systems

150 Results Summary

151 Results Summary Bugs are prevalent Semantic bugs dominate Bugs are constant Corruption and crash are common Metadata management is complex Failure paths are error-prone Diverse performance techniques

152 Lessons Learned

153 Lessons Learned A large-scale study is feasible and valuable time consuming, but still manageable similar study for other OS components new research opportunities

154 Lessons Learned A large-scale study is feasible and valuable time consuming, but still manageable similar study for other OS components new research opportunities Research should match reality new tools for semantic bugs more attention for failure paths

155 Lessons Learned A large-scale study is feasible and valuable time consuming, but still manageable similar study for other OS components new research opportunities Research should match reality new tools for semantic bugs more attention for failure paths History repeats itself similar mistakes similar performance and reliability techniques learn from history for a better future

156 Resources

157 Resources More information in paper detailed bug patterns reliability patches common patches across file systems

158 Resources More information in paper detailed bug patterns reliability patches common patches across file systems Even more information in dataset fix-on-fix detailed bug consequences tools used

159 Resources More information in paper detailed bug patterns reliability patches common patches across file systems Even more information in dataset fix-on-fix detailed bug consequences tools used Our dataset is released