*-Box (star-box) Towards Reliability and Consistency in Dropbox-like File Synchronization Services

Transcription

1 *-Box (star-box) Towards Reliability and Consistency in -like File Synchronization Services Yupu Zhang, Chris Dragga, Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau University of Wisconsin - Madison 6/27/2013 1

2 Cloud-Based File Synchronization Services Exploding in popularity Numerous providers:, Google Drive, SkyDrive Large user base: has more than 100 million users Key benefit Automatic synchronization across clients/devices Reliable data storage on the server through replication your stuff is safe in and will never be lost - tour, page 1 6/27/2013 2

3 Is Your Data Really Safe? Data corruption Uploaded from local machine to cloud Propagated to other devices/clients 6/27/2013 3

4 Is Your Data Really Safe? Crash consistency Inconsistent data ends up everywhere Out-of-sync synchronization after reboot sync client thinks everything is in sync 1 6/27/2013 4

5 Your Data is NOT Really Safe False sense of safety Many copies do NOT always make your data safe Why? Semantic gap between local file system and cloud Separately designed and loosely linked 6/27/2013 5

6 Project *-Box (star-box) Goal Close the gap between local file system and cloud Provide * without too much infrastructure changes * represents desired properties e.g., reliable, consistent, fast, private Currently focus on two properties Reliable: Data corruption Consistent: Crash consistency 6/27/2013 6

7 Outline Introduction Data Corruption Crash Consistency Current Status Conclusion 6/27/2013 7

8 Corruption Problem Data corruption is not uncommon Comes from disk media, firmware, controllers [Bairavasundaram07, Anderson03] Remains local w/o synchronization With synchronization Corruption may propagate and pollute other copies Synchronization is a double-edged sword Make sure synchronized data is good 6/27/2013 8

9 Background Changed Multiple Times Server foo [v0] [v1] (4MB data chunks) C1 C2 C3 C1 Client File System Local Database Name Attributes foo I I read foo (chunk by chunk) foo was modified Inotify Dedup NOT duplicate a NO offline changes I overwrite D2 D2 Rsync C1 C2 C3 D2 Disk I1 I D1 D2 6/27/2013 foo: inode 4KB data blocks 9

10 Inject Corruption Server foo [v0] (4MB data chunks) C1 C2 C3 File System Inotify Corrupt D1 Disk M1 I D1 D2 6/27/2013 foo: inode 4KB data blocks 10

11 Start Client Server foo [v0] (4MB data chunks) C1 C2 C3 Client Local Database Name Attributes foo I Dedup NO offline changes Rsync File System Inotify Disk I D1 D2 6/27/2013 foo: inode 4KB data blocks 11

12 Data Write Server foo [v0] [v1] (4MB data chunks) C1 C2 C3 C1 Client File System Local Database Name Attributes foo I I read foo (chunk by chunk) foo was modified Inotify Dedup NOT a duplicate C1 I I D1 overwrite D2 D2 Rsync D2 Disk I D1 D2 6/27/2013 foo: inode 4KB data blocks 12

13 Metadata Change Server foo [v0] [v1] (4MB data chunks) C1 C2 C3 C1 Client File System Local Database Name Attributes foo I I read foo foo s metadata (chunk by chunk) was changed Inotify Dedup NOT a duplicate C1 I I D1 touch -m D2 Rsync Disk I D1 D2 6/27/2013 foo: inode 4KB data blocks 13

14 More Results FS Service Data Writes ext4 (Linux) HFS+ (Mac OS X) L: Local corruption G: Global corruption Metadata Changes mtime ctime atime L G L G L G L owncloud L G L G L L FileRock L G L G L L L G L G L L owncloud L G L G L L GoogleDrive L G L G L L SugarSync L G L L L Syncplicity L G L G L L Corruption is propagated when there is a change to file data Even if there is no data change, corruption may still be uploaded 6/27/

15 Summary Bad bits are promoted to resilient bad bits ALL copies polluted Cloud copies protected by checksum Fundamental problem, not implementation bugs FS monitoring services only provide file-level notification Sync clients cannot tell legitimate changes from corruption Redundant data on the cloud is not fully utilized If corruption can be detected, local FS can recover from corruption using cloud copies 6/27/

16 Outline Introduction Data Corruption Crash Consistency Current Status Conclusion 6/27/

17 Crash Recovery Techniques Copy-on-write (e.g., ZFS, btrfs) Always roll back to a consistent version Journaling (e.g., ext4) Data journaling mode Both data and metadata are logged Provide data consistency Ordered journaling mode Only journal metadata Data blocks are written before metadata is logged Cannot guarantee data consistency 6/27/

18 Ordered Mode 1. Write dirty data blocks to home locations 2. Write metadata blocks to journal 3. Write journal commit block to the journal 4. Checkpoint journaled metadata blocks to home locations overwrite D1 &D2 File System I I D1 D Disk C I D1 D2 6/27/2013 Journal / Log foo: inode 4KB data blocks 18

19 Crash in Ordered Mode Crash during step 1 Write dirty data blocks to home locations overwrite D1 &D2 File System I I D1 D2 Inconsistent Consistent Data 1 1 Disk I D1 D2 6/27/2013 foo: inode 4KB data blocks 19

20 Ext4 Ordered Mode + Case 1 Inconsistent data is propagated Case 2 Consistent data is NOT synchronized 6/27/

21 Case 1 Crash NOT fully updated Server foo [v0] (4MB data chunks) C1 C2 C3 Client Local Database Name Attributes foo I I Dedup Rsync File System foo was modified Inotify I I overwrite D1 &D2 D1 crash AFTER database is changed D2 Disk I D1 D2 6/27/2013 inconsistent data foo: inode 4KB on data disk blocks 21

22 Case 1 Reboot Client NOT fully updated Local Database Name Attributes foo I Server foo [v0] [v1] (4MB data chunks) Dedup C1 C1 Sync! C2 Rsync C3 inconsistent data D1 D2 on cloud File System Inotify Disk 6/27/2013 I D1 D2 inconsistent data foo: inode 4KB on data disk blocks 22

23 Case 2 Crash Server foo [v0] (4MB data chunks) C1 C2 C3 Client Local Database Name Attributes foo I Dedup Rsync File System Inotify I I crash BEFORE database is changed overwrite D1 &D2 (O_SYNC) D1 D2 Disk I D1 D2 6/27/2013 consistent data foo: inode 4KB on data disk blocks 23

24 Case 2 Reboot Client Local Database Name foo Attributes I Server Dedup NO offline changes foo [v0] (4MB data chunks) C1 C2 Rsync C3 Server and other devices still have v0 NO sync! File System Inotify This machine has v1 Disk 6/27/2013 I D1 D2 consistent data foo: inode 4KB on data disk blocks 24

25 Recover using Data on Cloud? Data on server does not always reflect a consistent state on disk uploads data asynchronously reorders file uploading Actively modified files may get delayed When crash occurs, files on server could be inconsistent with respect to disk 6/27/

26 Summary Inconsistent content gets propagated Out-of-sync files may exist Different client/devices see different versions of the same file Need in-depth communication between local FS and cloud Cloud has very weak sense of actual FS state 6/27/

27 Outline Introduction Data Corruption Crash Consistency Current Status Conclusion 6/27/

28 Current Status of *-Box Finished Data checksumming in ext4 -aided corruption recovery in ext4 Fine-grained inotify in Linux Add a ranged file-update notification Adapt owncloud (an open-source sync service) to use it In-progress Use in-memory snapshot to facilitate crash recovery in ext4 ordered mode 6/27/

29 Outline Introduction Data Corruption Crash Consistency Possible Solutions Conclusion 6/27/

30 Conclusion Many copies do NOT always make your data safe Propagation of corrupt data and inconsistent state Synchronized files are out-of-sync Propose *-Box project Solve problems by reducing the semantic gap between existing local FS and cloud storage Ultimately may need a cohesive system that provides capabilities unachievable in isolation 6/27/

31 Thank you! Questions? Advanced Systems Lab (ADSL) University of Wisconsin-Madison Wisconsin Institute on Software-defined Datacenters in Madison 6/27/