Knockoff: Cheap versions in the cloud. Xianzheng Dou, Peter M. Chen, Jason Flinn

Transcription

1 Knockoff: Cheap versions in the cloud Xianzheng Dou, Peter M. Chen, Jason Flinn

2 Cloud-based storage Google Drive Dropbox Pros: Ease-of-management Reliability Microsoft OneDrive Xianzheng Dou 1

3 Cloud-based storage Google Drive Dropbox Challenges: Storage costs Communication costs Microsoft OneDrive Xianzheng Dou 2

4 Versioning increases costs Google Drive Dropbox Pros: Recovery of lost data Auditing Troubleshooting Versioning Microsoft OneDrive Xianzheng Dou 3

5 Reducing costs: a new direction Established methods exploit similarities in data Chunk-based deduplication Delta compression Greater work for incremental gains Our goal: explore an orthogonal new dimension Deterministically recompute data in lieu of communication, storage Xianzheng Dou 4

6 File: data or computation? Computation File data Xianzheng Dou 5

7 File: data or computation? Computation File data Xianzheng Dou 5

8 File: data or computation? Computation File data Xianzheng Dou 6

9 File: data or computation? Computation File data Xianzheng Dou 6

10 File: data or computation? Computation File data Xianzheng Dou 6

11 File: data or computation? Computation File data Xianzheng Dou 6

12 File: data or computation? Computation File data How can we address non-determinism? Different output data Xianzheng Dou 6

13 File: data or computation? Deterministic record and replay How to use X-ray RECORD Record Log PLAY Latency Metric Mail Scope Request Logs of nondeterminism debug_peer_list List of root causes: Xianzheng mydomain.com 1) Dou 7 queue_directory 2)

14 File: data or computation? Deterministic record and replay How to use X-ray RECORD Record Log PLAY Latency Metric Mail Scope Request Logs of nondeterminism debug_peer_list List of root causes: Xianzheng mydomain.com 1) Dou 7 queue_directory 2)

15 File: data or computation? Deterministic record and replay How to use X-ray RECORD Record Log PLAY Latency Metric Mail Scope Request Logs of nondeterminism debug_peer_list List of root causes: Xianzheng mydomain.com 1) Dou 7 queue_directory 2)

16 File: data or computation? Deterministic record and replay RECORD How to use X-ray How to use X-ray Record Log RECORD PLAY Log PLAY Replay Latency Metric Mail Scope Request Logs of nondeterminism Latency Metric Mail Scope Request debug_peer_list List of root causes: Xianzheng mydomain.com Dou debug_peer_list List of root causes: 1) 7 queue_directory 2) mydomain.com 1)

17 Knockoff Selectively substitutes computation for data Benefits Reduction compared to chunk-based deduplication Communication costs: 21% Storage costs: 19% Benefits increases as we retain versions more frequently A new fined-grained versioning policy Xianzheng Dou 8

18 Outline Introduction Writing files Storing files Evaluation Xianzheng Dou 9

19 Knockoff Knockoff selectively represents a file as: Normal file data (by value) Logs of the nondeterminism needed to recompute the file (by operation) File Xianzheng Dou 10

20 Knockoff Knockoff selectively represents a file as: Normal file data (by value) Logs of the nondeterminism needed to recompute the file (by operation) File Xianzheng Dou 10

21 Knockoff Knockoff selectively represents a file as: Normal file data (by value) Logs of the nondeterminism needed to recompute the file (by operation) OR File Xianzheng Dou 10

22 Knockoff Knockoff selectively represents a file as: Normal file data (by value) Logs of the nondeterminism needed to recompute the file (by operation) OR File OR Xianzheng Dou 10

23 An example log for compilation Log entry Values 1 open rc=3 2 mmap rc=<addr>,file=<id,version> 3 gettimeofday rc=0,time=<time> 4 pthread_lock rc=0 5 SIGCHILD Xianzheng Dou 11

24 An example log for compilation Log entry Values 1 open rc=3 2 mmap rc=<addr>,file=<id,version> Return values from syscalls 3 gettimeofday rc=0,time=<time> 4 pthread_lock rc=0 Ordering of thread synchronization 5 SIGCHILD Signals Xianzheng Dou 11

25 An example log for compilation Log entry Values 1 open rc=3 2 mmap rc=<addr>,file=<id,version> 3 gettimeofday rc=0,time=<time> 4 pthread_lock rc=0 5 SIGCHILD Xianzheng Dou 11

26 An example log for compilation Log entry Values 1 open rc=3 2 mmap rc=<addr>,file=<id,version> 3 gettimeofday rc=0,time=<time> 4 pthread_lock rc=0 5 SIGCHILD Xianzheng Dou 11

27 Writing files By value By operation Xianzheng Dou 13

28 Writing files By value By operation Xianzheng Dou 13

29 Writing files By value By operation Xianzheng Dou 13

30 Writing files By value photo editing By operation Xianzheng Dou 14

31 Writing files By value cryptographic key generation By operation Xianzheng Dou 15

32 Outline Introduction Writing files Storing files Evaluation Xianzheng Dou 17

33 Storing files Store files by value or by operation?? A tradeoff between latency and costs Current versions: by value Past versions: by value or by operation Xianzheng Dou 18

34 Storing past versions Maximum materialization delay Time bound for reconstructing any version Regeneration time = 20s < Materialization delay = 60s Xianzheng Dou 19

35 Storing past versions Maximum materialization delay Time bound for reconstructing any version Regeneration time = 20s < Materialization delay = 60s Xianzheng Dou 19

36 Storing past versions Maximum materialization delay Time bound for reconstructing any version Regeneration time = 100s > Materialization delay = 60s Xianzheng Dou 20

37 Storing past versions Maximum materialization delay Time bound for reconstructing any version Regeneration time = 100s > Materialization delay = 60s Xianzheng Dou 20

38 Storing past versions Maximum materialization delay Time bound for reconstructing any version Longest path > materialization delay 20s Total regeneration time = 20s < Materialization delay = 60s Xianzheng Dou 21

39 Storing past versions Maximum materialization delay Time bound for reconstructing any version Longest path > materialization delay 30s 20s Total regeneration time = 50s < Materialization delay = 60s Xianzheng Dou 22

40 Storing past versions 30s Maximum materialization delay Time bound for reconstructing any version Longest path > materialization delay 30s 20s Total regeneration time = 80s > Materialization delay = 60s Xianzheng Dou 23

41 Storing past versions 30s Maximum materialization delay Time bound for reconstructing any version Longest path > materialization delay 30s 20s Total regeneration time = 80s > Materialization delay = 60s Xianzheng Dou 24

42 Storing past versions 30s Maximum materialization delay Time bound for reconstructing any version Longest path > materialization delay 30s 20s Total regeneration time = 80s > Materialization delay = 60s Xianzheng Dou 24

43 Storing past versions Maximum materialization delay Time bound for reconstructing any version Longest path > materialization delay A greedy algorithm Materialization delay = 60s Xianzheng Dou 25

44 Storing past versions: versioning policies Frequency of versioning Xianzheng Dou 26

45 Storing past versions: versioning policies Frequency of versioning No versioning Version on close Version on write Eidetic versioning Xianzheng Dou 26

46 Storing past versions: versioning policies Frequency of versioning No versioning Version on close Version on write Eidetic versioning Memory-mapped files Any past transient state in memory? Xianzheng Dou 26

47 Optimization: log compression Chunk-based deduplication is effective for file data Executions of the same application have similar patterns Can it also be applied to computation (logs of nondeterminism)? Delta compression Xianzheng Dou 28

48 Optimization: log compression Problem: a smattering of values differ in each log Xianzheng Dou 29

49 Optimization: log compression Problem: a smattering of values differ in each log Delta compression: 42% reduction Xianzheng Dou 29

50 Outline Introduction Writing files Storing files Evaluation Xianzheng Dou 30

51 Evaluation How much does Knockoff reduce bandwidth usage? How much does Knockoff reduce storage costs? What is Knockoff s performance overhead? For more experimental results, please refer to our paper Xianzheng Dou 31

52 Experimental setup User study 8 participants performed several simple tasks in one hour 20-day study A single-user longitudinal study A variety of programs used Various Linux utilities, text editors and programming languages Xianzheng Dou 32

53 Bandwidth usage: user study Xianzheng Dou 33

54 Data sent to the server (MB) Bandwidth usage: user study Already achieve 80%-85% reduction No versioning Version on close Version on write Eidetic Chunk-based deduplication Knockoff Xianzheng Dou 33

55 Data sent to the server (MB) Bandwidth usage: user study Already achieve 80%-85% reduction No versioning Version on close Version on write Eidetic Chunk-based deduplication Knockoff Xianzheng Dou 33

56 Data sent to the server (MB) Bandwidth usage: user study % No versioning Version on close Version on write Eidetic Chunk-based deduplication Knockoff Xianzheng Dou 33

57 Data sent to the server (MB) Bandwidth usage: user study % 25% 47% No versioning Version on close Version on write Eidetic Chunk-based deduplication Knockoff Xianzheng Dou 33

58 Data sent to the server (MB) Bandwidth usage: user study % 25% 47% No versioning Version on close Version on write Eidetic Chunk-based deduplication Knockoff Xianzheng Dou 33

59 Bandwidth savings(%) Variances across applications Linux utility Graph editing Libreoffice Programming Text editing Web browsing Total Version on close Xianzheng Dou 35

60 Bandwidth savings(%) Variances across applications Linux utility Graph editing Libreoffice Programming Text editing Web browsing Total Version on close Xianzheng Dou 35

61 Bandwidth savings(%) Variances across users Knockoff A B C D E F G Version on close Xianzheng Dou 36

62 Bandwidth savings(%) Variances across users Knockoff A B C D E F G Version on close Xianzheng Dou 36

63 Relative storage cost Relative storage costs for past versions Version on close Version on write Eidetic Chunk-based deduplication Knockoff Xianzheng Dou 37

64 Relative storage cost Relative storage costs for past versions % 1 19% Version on close Version on write Eidetic Chunk-based deduplication Knockoff Xianzheng Dou 37

65 Relative storage cost Relative storage costs for past versions % 1 19% Version on close Version on write Eidetic Chunk-based deduplication Knockoff Xianzheng Dou 37

66 Performance overheads 7-8% performance overheads Baseline No Versioning Version on close Version on write Eidetic 38

67 Conclusion A new dimension for reducing costs Selectively substitute computation for data A general-purpose system for deterministic recomputation Reduces storage and communication costs for existing versioning policies Enables eidetic versioning Xianzheng Dou 39

68 Thank you! Xianzheng Dou 40

69 Varying the materialization delay Xianzheng Dou 41

70 Monetary costs Xianzheng Dou 42

71 Workload characteristics Xianzheng Dou 43