Optimistic Crash Consistency. Vijay Chidambaram Thanumalayan Sankaranarayana Pillai Andrea Arpaci-Dusseau Remzi Arpaci-Dusseau

Transcription

1 Optimistic Crash Consistency Vijay Chidambaram Thanumalayan Sankaranarayana Pillai ndrea rpaci-dusseau Remzi rpaci-dusseau

2 Crash Consistency Problem Single file-system operation updates multiple on-disk data structures System may crash in middle of updates File-system is partially (incorrectly) updated 2

3 Performance OR Consistency 3

4 Performance OR Consistency Crash-consistency solutions degrade performance Users forced to choose between high performance and strong consistency - Performance differs by 10x for some workloads Many users choose performance - ext3 default configuration did not guarantee crash consistency for many years - Mac OSX fsync() does not ensure data is safe The Fast drives out the Slow even if the Fast is wrong - Kahan 3

5 Ordering and Durability Crash consistency is built upon ordered writes File systems conflate ordering and durability - Ideal: {, } -> {C} (made durable later) - Current scenario {, } durable {C} durable Inefficient when only ordering is required 4

6 Can a file system provide both high performance and strong consistency? Is there a middle ground between: high performance but no consistency strong consistency but low performance? 5

7 Our solution Optimistic File System (OptFS) Journaling file system that provides performance and consistency by decoupling ordering and durability Such decoupling allows OptFS to trade freshness for performance while maintaining crash consistency 6

8 Results Techniques: checksums, delayed writes, etc. OptFS provides strong consistency - Equivalent to ext4 data journaling OptFS improves performance significantly - 10x better than ext4 on some workloads New primitive osync() provides ordering among writes at high performance 7

9 Outline Introduction Ordering and Durability in Journaling Optimistic File System Results Conclusion 8

10 Outline Introduction Ordering and Durability in Journaling - Journaling Overview - Realizing Ordering on s - Journaling without Ordering Optimistic File System Results Conclusion 9

11 Journaling Overview efore updating file system, write note describing update Make sure note is safely on disk Once note is safe, update file system - If interrupted, read note and redo updates 10

12 Journaling Overview Workload: Creating and writing to a file Journaling protocol (ordered journaling) DT METDT PPLICTION FILE SYSTEM DISK Journal 11

13 Workload: Creating and writing to a file Journaling protocol (ordered journaling) - Data write (D) Journaling Overview DT METDT PPLICTION FILE SYSTEM DISK Journal 11

14 Workload: Creating and writing to a file Journaling protocol (ordered journaling) - Data write (D) Journaling Overview DT METDT PPLICTION FILE SYSTEM D DISK Journal 11

15 Workload: Creating and writing to a file Journaling protocol (ordered journaling) - Data write (D) Journaling Overview DT METDT PPLICTION FILE SYSTEM DISK D Journal 11

16 Journaling Overview Workload: Creating and writing to a file Journaling protocol (ordered journaling) - Data write (D) - Logging Metadata (JM) DT METDT PPLICTION FILE SYSTEM DISK D Journal 11

17 Journaling Overview Workload: Creating and writing to a file Journaling protocol (ordered journaling) - Data write (D) - Logging Metadata (JM) DT METDT PPLICTION FILE SYSTEM JM DISK D Journal 11

18 Journaling Overview Workload: Creating and writing to a file Journaling protocol (ordered journaling) - Data write (D) - Logging Metadata (JM) DT METDT PPLICTION FILE SYSTEM DISK D JM Journal 11

19 Journaling Overview Workload: Creating and writing to a file Journaling protocol (ordered journaling) - Data write (D) - Logging Metadata (JM) - Logging Commit (JC) DT METDT PPLICTION FILE SYSTEM DISK D JM Journal 11

20 Journaling Overview Workload: Creating and writing to a file Journaling protocol (ordered journaling) - Data write (D) - Logging Metadata (JM) - Logging Commit (JC) DT METDT PPLICTION FILE SYSTEM JC DISK D JM Journal 11

21 Journaling Overview Workload: Creating and writing to a file Journaling protocol (ordered journaling) - Data write (D) - Logging Metadata (JM) - Logging Commit (JC) DT METDT PPLICTION FILE SYSTEM DISK D JM JC Journal 11

22 Journaling Overview Workload: Creating and writing to a file Journaling protocol (ordered journaling) - Data write (D) - Logging Metadata (JM) - Logging Commit (JC) - Checkpointing (M) PPLICTION DT METDT FILE SYSTEM DISK D JM JC Journal 11

23 Journaling Overview Workload: Creating and writing to a file Journaling protocol (ordered journaling) - Data write (D) - Logging Metadata (JM) - Logging Commit (JC) - Checkpointing (M) PPLICTION DT METDT FILE SYSTEM M DISK D JM JC Journal 11

24 Journaling Overview Workload: Creating and writing to a file Journaling protocol (ordered journaling) - Data write (D) - Logging Metadata (JM) - Logging Commit (JC) - Checkpointing (M) PPLICTION DT METDT FILE SYSTEM DISK D M JM JC Journal 11

25 Outline Introduction Ordering and Durability in Journaling - Journaling Overview - Realizing Ordering on s - Journaling without Ordering Optimistic File System Results Conclusion 12

26 How Writes are Ordered Original s 13

27 How Writes are Ordered Original s 13

28 How Writes are Ordered Original s 13

29 How Writes are Ordered Original s 13

30 How Writes are Ordered Original s 13

31 How Writes are Ordered Original s 13

32 How Writes are Ordered Original s s with Write uffers Cache Platter 13

33 How Writes are Ordered Original s s with Write uffers Cache Platter 13

34 How Writes are Ordered Original s s with Write uffers Cache Platter 13

35 How Writes are Ordered Original s s with Write uffers Cache Platter 13

36 How Writes are Ordered Original s s with Write uffers Cache Platter 13

37 How Writes are Ordered Original s s with Write uffers Cache Platter 13

38 How Writes are Ordered Original s s with Write uffers Cache Platter 13

39 How Writes are Ordered Original s s with Write uffers Cache Platter 13

40 How Writes are Ordered Original s s with Write uffers Cache Platter 13

41 How Writes are Ordered Original s s with Write uffers Cache Platter 13

42 How Writes are Ordered Original s s with Write uffers Cache Platter 13

43 How Writes are Ordered Original s s with Write uffers Cache Platter 13

44 How Writes are Ordered Original s s with Write uffers Flush Cache Platter 13

45 How Writes are Ordered Original s s with Write uffers Flush Cache Platter 13

46 How Writes are Ordered Original s s with Write uffers Flush Cache Platter 13

47 Journaling with Flushes PPLICTION FILE SYSTEM DISK CCHE DISK PLTTER Journal 14

48 Journaling with Flushes Journaling protocol - Data write (D) DT METDT PPLICTION FILE SYSTEM D DISK CCHE DISK PLTTER Journal 14

49 Journaling with Flushes Journaling protocol - Data write (D) DT METDT PPLICTION FILE SYSTEM DISK CCHE D DISK PLTTER Journal 14

50 Journaling with Flushes Journaling protocol - Data write (D) - Logging Metadata (JM) DT METDT PPLICTION FILE SYSTEM DISK CCHE D DISK PLTTER Journal 14

51 Journaling with Flushes Journaling protocol - Data write (D) - Logging Metadata (JM) DT METDT PPLICTION FILE SYSTEM JM DISK CCHE D DISK PLTTER Journal 14

52 Journaling with Flushes Journaling protocol - Data write (D) - Logging Metadata (JM) DT METDT PPLICTION FILE SYSTEM DISK CCHE D JM DISK PLTTER Journal 14

53 Journaling with Flushes Journaling protocol - Data write (D) - Logging Metadata (JM) DT METDT PPLICTION FILE SYSTEM DISK CCHE D JM FLUSH DISK PLTTER Journal 14

54 Journaling with Flushes Journaling protocol - Data write (D) - Logging Metadata (JM) DT METDT PPLICTION FILE SYSTEM DISK CCHE FLUSH DISK PLTTER D JM Journal 14

55 Journaling with Flushes Journaling protocol - Data write (D) - Logging Metadata (JM) - Logging Commit (JC) DT METDT PPLICTION FILE SYSTEM DISK CCHE FLUSH DISK PLTTER D JM Journal 14

56 Journaling with Flushes Journaling protocol - Data write (D) - Logging Metadata (JM) - Logging Commit (JC) DT METDT PPLICTION FILE SYSTEM JC DISK CCHE FLUSH DISK PLTTER D JM Journal 14

57 Journaling with Flushes Journaling protocol - Data write (D) - Logging Metadata (JM) - Logging Commit (JC) DT METDT PPLICTION FILE SYSTEM DISK CCHE FLUSH JC DISK PLTTER D JM Journal 14

58 Journaling with Flushes Journaling protocol - Data write (D) - Logging Metadata (JM) - Logging Commit (JC) DT METDT PPLICTION FILE SYSTEM DISK CCHE FLUSH JC FLUSH DISK PLTTER D JM Journal 14

59 Journaling with Flushes Journaling protocol - Data write (D) - Logging Metadata (JM) - Logging Commit (JC) DT METDT PPLICTION FILE SYSTEM DISK CCHE FLUSH FLUSH DISK PLTTER D JM JC Journal 14

60 Journaling with Flushes Journaling protocol - Data write (D) - Logging Metadata (JM) - Logging Commit (JC) - Checkpointing (M) PPLICTION DT METDT FILE SYSTEM DISK CCHE FLUSH FLUSH DISK PLTTER D JM JC Journal 14

61 Journaling with Flushes Journaling protocol - Data write (D) - Logging Metadata (JM) - Logging Commit (JC) - Checkpointing (M) PPLICTION DT METDT FILE SYSTEM M DISK CCHE FLUSH FLUSH DISK PLTTER D JM JC Journal 14

62 Journaling with Flushes Journaling protocol - Data write (D) - Logging Metadata (JM) - Logging Commit (JC) - Checkpointing (M) PPLICTION DT METDT FILE SYSTEM DISK CCHE FLUSH FLUSH M DISK PLTTER D JM JC Journal 14

63 Journaling with Flushes Journaling protocol - Data write (D) - Logging Metadata (JM) - Logging Commit (JC) - Checkpointing (M) PPLICTION DT METDT FILE SYSTEM DISK CCHE FLUSH FLUSH DISK PLTTER D M JM JC Journal 14

64 Outline Introduction Ordering and Durability in Journaling - Journaling Overview - Realizing Ordering on s - Journaling without Ordering Optimistic File System Results Conclusion 15

65 Journaling without Ordering Practitioners turn off flushes due to performance degradation - E.g., ext3 by default did not enable flushes for many years Observe crashes do not cause inconsistency for some workloads We term this probabilistic crash consistency - Studied in detail 16

66 Probabilistic Crash Consistency p-inconsistency for different workloads - Read-heavy workloads have low p-inconsistency - Database workloads have high p-inconsistency See paper for detailed study - Factors that affect p-inconsistency Turning off flushing provides performance, but does not ensure consistency dditional techniques required to obtain both performance and consistency 17

67 Outline Introduction Ordering and Durability in Journaling Optimistic File System - Overview - Handling Re-Ordering - New File-system Primitives Results Conclusion 18

68 Optimistic File System chieves both performance and consistency by trading on new axis Freshness indicates how up-to-date state is after a crash OptFS provides strong consistency while trading freshness for increased performance X State 1 State 2 State 3 State 4 OptFS ext4 19

69 Optimistic File System Eliminates flushes in the common case locks may be re-ordered without flushes Optimistic Crash Consistency handles re-orderings with different techniques - Some re-orderings are detected after crash - Some re-orderings are prevented from occurring 20

70 Modified Interface synchronous Durability Notifications (DN) signal when block is made durable Cache Platter 21

71 Modified Interface synchronous Durability Notifications (DN) signal when block is made durable Cache Platter 21

72 Modified Interface synchronous Durability Notifications (DN) signal when block is made durable Cache Platter 21

73 Modified Interface synchronous Durability Notifications (DN) signal when block is made durable Cache Platter 21

74 Modified Interface synchronous Durability Notifications (DN) signal when block is made durable Cache Platter 21

75 Modified Interface synchronous Durability Notifications (DN) signal when block is made durable Cache Platter 21

76 Modified Interface synchronous Durability Notifications (DN) signal when block is made durable Cache Platter 21

77 Modified Interface DNs increase disk freedom - locks can be destaged in any order - locks can be destaged at any time - Only requirement is to inform upper layer OptFS uses DNs to control what blocks are dirty at the same time in disk cache - Re-ordering can only happen among these blocks 22

78 Outline Introduction Ordering and Durability in Journaling Optimistic File System - Overview - Handling Re-Ordering - New File-system Primitives Results Conclusion 23

79 Optimistic Journaling Checksums and Delayed Writes handle reordering from removing flushes DT METDT PPLICTION FILE SYSTEM D JM JC M DISK CCHE FLUSH FLUSH DISK PLTTER Journal 24

80 Optimistic Journaling Checksums and Delayed Writes handle reordering from removing flushes DT METDT PPLICTION FILE SYSTEM D JM JC M DISK CCHE FLUSH DISK PLTTER Journal 24

81 Optimistic Journaling Checksums and Delayed Writes handle reordering from removing flushes DT METDT PPLICTION FILE SYSTEM D JM JC M DISK CCHE FLUSH DISK PLTTER Journal 24

82 Optimistic Journaling Checksums and Delayed Writes handle reordering from removing flushes DT METDT PPLICTION FILE SYSTEM D JM JC M DISK CCHE FLUSH DISK PLTTER Journal 24

83 Optimistic Journaling Checksums and Delayed Writes handle reordering from removing flushes DT METDT PPLICTION FILE SYSTEM M DISK CCHE D JM JC FLUSH DISK PLTTER Journal 24

84 Optimistic Journaling Checksums and Delayed Writes handle reordering from removing flushes DT METDT PPLICTION FILE SYSTEM M DISK CCHE D JM JC DISK PLTTER Journal 24

85 Optimistic Journaling Checksums and Delayed Writes handle reordering from removing flushes DT METDT PPLICTION FILE SYSTEM M DISK CCHE D JC JM DISK PLTTER JM Journal 24

86 Optimistic Journaling Checksums and Delayed Writes handle reordering from removing flushes DT METDT PPLICTION FILE SYSTEM M DISK CCHE D JM JC DISK PLTTER JM JC Journal 24

87 Optimistic Journaling Checksums and Delayed Writes handle reordering from removing flushes DT METDT PPLICTION FILE SYSTEM M DISK CCHE D JM JC DISK PLTTER D JM JC Journal 24

88 Optimistic Journaling Checksums and Delayed Writes handle reordering from removing flushes DT METDT PPLICTION FILE SYSTEM DISK CCHE D JM JC M DISK PLTTER D JM JC Journal 24

89 Optimistic Journaling Checksums and Delayed Writes handle reordering from removing flushes DT METDT PPLICTION FILE SYSTEM DISK CCHE D JM JC DISK PLTTER D M JM JC Journal 24

90 Optimistic Techniques Other Techniques - In-order journal recovery and release - Reuse after notification - Selective data journaling See paper for more details 25

91 Outline Introduction Ordering and Durability in Journaling Optimistic File System - Overview - Handling Re-Ordering - New File-system Primitives Results Conclusion 26

92 File-system Primitives fsync() provides ordering and durability OptFS splits fsync() - osync() for only ordering and high performance - dsync() for durability Primitives can increase performance - Ex: SQLite write(log) fsync(log) write(header) fsync(header) 27

93 File-system Primitives fsync() provides ordering and durability OptFS splits fsync() - osync() for only ordering and high performance - dsync() for durability Primitives can increase performance - Ex: SQLite write(log) fsync(log) write(header) fsync(header) write(log) osync(log) write(header) dsync(header) 27

94 File-system Primitives fsync() provides ordering and durability OptFS splits fsync() - osync() for only ordering and high performance - dsync() for durability Primitives can increase performance - Ex: SQLite write(log) fsync(log) write(header) fsync(header) write(log) osync(log) write(header) dsync(header) 27

95 Outline Introduction Ordering and Durability in Journaling Optimistic File System Results Conclusion 28

96 Evaluation Does OptFS preserve file-system consistency after crashes? - OptFS consistent after 400 random crashes How does OptFS perform? - OptFS 4-10x better than ext4 with flushes Can meaningful application-level consistency be built on top of OptFS? - SQLite provides CI semantics at 10x performance 29

97 Outline Introduction Ordering and Durability in Journaling Optimistic File System Results Conclusion 30

98 Summary Problem: providing both performance and consistency Solution: decoupling ordering and durability in OptFS OptFS maintains consistency while trading freshness for increased performance osync() provides a cheap primitive to order application writes 31

99 Conclusion Storage-stack layers are increasing - 18 layers between application and storage [Thereska13] - Interfaces that provide freedom to each layer are the way forward First impulse: trade consistency for performance - Trade-off not required in distributed systems [Escriva12] - y trading freshness, we can obtain both consistency and high performance 32

100 Thank You Source code Questions?