Synchronisation solutions,

Transcription

1 Synchronisation solutions, NDS2 Cryptobox Kamil Guryn, BIAMAN, Maciej Brzeźniak, PSNC, & NDS2 project partners:

2 Presentation agenda Background Features of good synchronisation mechanisms Discussion of features, example scenarios Comparison of existing solutions Cryptobox: Motivation: why yet another solution? Features of Cryptobox Live demo

3 Background Some NRENs offer data sync & share solutions Some NRENs consider providing them: Purchase of hosted services (e.g. NorduNET: box.com) Purchase of licenses/support for private/community cloud services: owncloud PowerFolder Other initiatives We need to understand technical aspects of sync solutions The aim of this presentation is to: Discuss selected features of synchronisation algorithms Present algorithms presented in Cryptobox of NDS2project

4 Steps of the typical synchronisation process: How it works: filespace enumeration detection = comparison of current and past state reconciliation planning propagation conflicts resolution propagation

5 Synchronization problems - overview track changes on the stores: file identifiers support not present everywhere async events reporting mechanism: inotify or FSwatcher anchor-based sync needs specialized datastore: e.g. simple SFTP server not capable need to deal with concurrency during sync which can break anchor-based logic; consistency, missing changes, complexity conflicts detection and resolving unresolved items may lead to sync endless loop concurrency of data access and synchronisation locking for sync can interfere with user activities concurrent file activities should be supported unreliable networks, sync interrupts & errors sync endless loops, statedb loss or corruption and many more

6 Data synchronisation methods and algorithms What is needed to synchronise two data stores: change detection: client and/or server-side rename/move detection: id-based vs hash-based relative ordering of asynchronous events, e.g.: rename within same dir detected move outside dir = delete + create combination of id/hash and fs events (e.g. Dropbox) full-enumeration vs anchor-based change detection client-side state database needed for change detection data transmission proprietary vs open protocols

7 Important features of sync solution Intelligent name change / move detection on both sides Concurrency resistant during detection and synchronization Conflicts resolving Large datastores indexing efficiency Support for advanced scenarios: graceful cancellation of running sync preview mode of the incremental sync operation without committing changes to the replicas support for sync with partially equal file hierarchies Security and privacy: client-side cryptography Wuala and Spideroak does it client-side Dropbox/Box does it server-side (PRISM.) owncloud does not plan supporting it Cryptobox does it client-side

8 Intelligent rename/move detection - synchronization clue Example scenario: directory folderx contains 10GB of data user on machine A moves folderx to folderz changes synchronized to server as move operation in some solutions: changes badly interpreted by sync client on machine B this may result in unnecessary download of the data instead of name change / move propagation while acceptable for small files, may be a killer for big data volumes A B

9 Intelligent change detection explained: FileID-based vs hash-based FileID-based - lightweight: no need for files content analysis does not require a lot of I/O does not load CPU Hash-based - resource consuming: needs files contents analysis I/O needed to read files content load on CPU for calculation hashes

10 Intelligent change detection explained: FileID-based FileID must be provided/supported by (file)system Works in: o o o Windows: NTFS, ReFS Linux: EXT2/3/4 OS X: HFS+

11 Sync issues: FS events analysis (1): FS events reporting (inotify, Fswatcher) is helper in detecting changes BUT reliable interpretation of users activity is not trivial: Common issues: FS events are reported asynchronously (unordered)! all events have to be analysed to ensure consistency reliable interpretations of events not trivial sync on concurrent environment (opposite concurrent file activities events, opposite folder hierarchical events) Client-side issues: Buffer overflows when many operations in short period of time may lead to missed events Server-side issues: needs specialized server functionality server-client events notify mechanisms needed to avoid costly full namespace scans

12 Sync issues: FS events analysis (2): operation event logged rename folder\a.txt -> folder\b.txt move folder\a.txt -> folder2\a.txt copy a.txt to local sync folder Client side: copy a.txt to foldera\folderb\folderc Sever-side: rename foldera to folderx rename delete folder\a.txt & create folder2\a.txt create (there is no event about relasing file lock - close a file) copy a.txt to foldera\folderb\folderc is not valid anymore, should be propagated in correct order (plus concurrency resistance ) Example scenario where event logging fails to provide consistent state

13 Concurrency during detection and synchronization Scenario: 1. File A.txt is synchronized across replicas 2. Bob updates A.txt on machine1 (client) 3. Alice updates A.txt on server (by updating it on machine2 and performing sync), while Bob changes are currently pending This should not lead to lost of Alice changes.

14 Conflicts resolving Scenario: 1. Bob creates file A.txt on local machine1 2. Alice also creates A.txt on server (by creating it locally on machine2 and performing sync) while Bob changes are currently pending This leads to name conflict and should be automatically resolved with a no-data loss policy

15 Security and privacy: client-side cryptography Support for client-side encryption desired these days: o People concerned about privacy o o o o Some organisation ;) known to analyse our data AES-256 CTR encryption AES-256 recommended by NIST to 2031 CTR mode recommended by Niels Ferguson and Bruce Schneier AES-NI supported on increasing number of CPU platforms File names to be encrypted too! as they may be meaningfull Integrity control Lets users verify consistency of the data Enables detecting failures and intended tweaks SHA-512 considered to be safe beyond 2012 (SHA-1 only to 2012) Fast implementations possible using regular CPUs

16 Synchronisation solution comparison * Feature NDS Cryptobox owncloud BOX.com Dropbox Spider Oak Power Folder Client & server-side move detection YES NO YES YES YES NO Real-time change detection YES NO (client side only) YES YES YES YES Files under user control YES YES NO NO NO YES Client-side encryption YES NO NO NO YES NO Concurrency resistant on detection and propagation YES NO (may lead to chaos) YES (but goes through temp) YES??? YES Sync files of any size YES YES (in multiple parts, then merge) NO YES YES YES Synchronisation of any folder YES YES NO NO YES YES NO file-locking policy YES (drobbox way) NO NO (sync goes through temp) YES??? NO Preview mode YES NO NO NO NO NO Live-sync NO NO NO YES NO NO * Comparison made to our best knowledge. Based on documentation analysis and tests. If you notice any inconsistency, please contact us.

17 What is NDSCryptobox? (1) Motivation: we were not happy with existing solutions: services, tools, applications and libraries (we tested most of them) Issues: Simplistic / unreliable sync algorithms Lack of client&server-side move/rename discovery Lack of client-side encryption Slow detection and propagation of changes Reliability vs concurrency changes issues File-locking policy makes local sync folder harder to use Some solutions not able to pass even simple tests: e.g. multiple folder rename while uploading leads to chaos on local computer concurrent changes occur on one replica during detection and propagation No way to benefit from special features of our NDS2 system: inefficient enumeration makes sync too expensive for storage backend fast & lightweight metadata enumeration needed

18 What is NDSCryptobox? (2) About NDS2 project Secure Storage Cloud with efficient and easy data access More information: nds.psnc.pl NDS2 paper published in the TNC2013 NDSCryptobox in NDS2 architecture one of client applications accessing the system using SFTP + server-side mechanisms: deltas, enumeration etc.

19 Key NDS2 features: Security, client-side cryptography AES 256 CBC encryption using per-file keys for data privacy: Key hierarchy makes security high while keeping system easy to use File keys stored in the file header, protected with user s private RSA key File names encryption using per-folder keys SHA-512 digests for integrity check: Digests calculated per 64-byte logical block Stored with the data, encrypted Data encrypted with AES-256 CTR Plus key mgmt and protection mechanisms to prevent data loss More information: NDS2 paper submitted to TNC2013

20 Some tricks of NDSCryptobox Efficient enumeration thanks to integration of sync client with NDS2 system meta-data mechanisms find on elements takes: s with required inode lookup over NFS Tested on Bluearc Mercury 100 NAS Filer 3.017s! by speaking directly to NDS2 meta-data; DB Select * on elements No overload in NDS2 system even for many concurrent clients enumerating files (we are curious how others can scale ) other approaches e.g. propfind with infinite on WebDAV may cause high overload on filesystem and on Apache/PHP stack request/response XML/HTTP overhead may lead to bad performance on-the-wire, timeouts, etc.

21 Other features of NDS Cryptobox May talk to regular SFTP server: easy to deploy in user s infrastructure login/pass or keys can be used for authorisation easy to integrate with existing auth systems: LDAP, etc. others support: owncloud: WebDAV (with hooks) proprietary protocols: SpiderOak, PowerFolder, (Drop)box Virtual-IO: enables developing drivers for other backends currently: SFTP, NDS2 in future: WebDAV, WebDAV/ownCloud, S3, Mega Supports multiple directories sync SugarSync does, many others don t

22 Cryptobox limitations It s not yet resolving file vs folder name clash. This causes sync endless loop (operation won t be resolved on other replica)

23 Plans/roadmap Linux/OSX support (currently only Windows) Further optimalization on sync algorithm Support for read-only files (Dropbox way) Support for file vs folder name-clash Improvements in reliability Internal recycle bin for sftp mode (auto backup of replaced/overwritten/deleted files) Dropbox-like icon overlays Shell integration (share/export/publish a file option) Secure data sharing support NDS2 feature Data import/export NDS2 s FileSender-like feature Data publication NDS2 publication feature Console version

24 Cryptobox recipe Did research on existing solutions Got frustrated ;) Took SyncSharp: GNU GPL v3 Adopted: Interface GUI with minor modifications synchronisation logic - with major modifications Added: SyncSharp is originaly developed for syncing local folders FileID support Concurrency, errors and interrupts resistance Graceful cancellation of running sync Client-side encryption Virtual IO, SFTP client Autosync feature, real-time change detection And many more

25 Cryptobox backup mode In addition to synchronisation Cryptobox supports backup mode similarly to e.g. Wuala CryptoBox supports so-called contribute mode: add changes from the left folder to the right folder. New and updated files are copied left to right. No deletions! Folder creates and folder updates on the left are repeated on the right

26 CryptoBox GUI: simple to use with drag&drop support Many sessions/connection to many SFTP servers Support for multiple synchronisation tasks (local folders vs servers) On-demand synchronisation Preview mode Auto sync Folders to be sync d can be dragged here System tray integration

27 CryptoBox GUI - Sessions Sessions window lets you configure the server access

28 CryptoBox GUI - Sync tasks In sync tasks window you may configure client- and server-side folders to be synchronised and decide if client-side encryption should be used

29 CryptoBox GUI: Preview mode Shows actions to be performed during synchronisation process In the example below propagations of create / rename actions and transfer of small file is planned

30 Cryptobox s auto-detect server capabilities function Cryptobox can use SFTP servers for sync It may also speak to NDS2 servers supporting advanced features Client detects type of the server and behaves accordingly This is a regular server that does not support versioning, link share, share & collaborate, geo-replication etc.

31 Cryptobox change tracking compatibility FileID-based: client-side mechanisms: file-id based change detection on (NTFS,EXT,HFS+) server-side: with NDS2 servers: make use of file-id support and fast and lightweight enumeration with regular SFTP servers: uses shell access for fast change detection Hash-based: Used where FileID-base is not possible e.g. clients with FAT32 file system

32 Conclusions (general) Reliable, secure & fast sync of large data is not trivial Even popular solutions use simplistic sync algorithms: Lack of intelligent change detection (both side rename/move discovery) prevents them for scaling to really large data volumes Only part of solutions uses encryption: user s data privacy? Some solutions use server-side encryption only (e.g. Dropbox) Authors of some other solution assume that data on the usercontrolled server are safe (what about hackers?) In our opinion end-to-end encryption is the only solution! We evaluated most popular solutions: General results in the table, details available on request We consider also writing a report to be shared with larger public Co-authors welcome! This work can be continued within TF-Storage We will start with putting our results online (TF-Storage wiki) Feedback and discussion welcome! Pilots to be run in collaboration with other NRENs within TF-Storage?

33 Conclusions (NDS Cryptobox) In NDS2 project we implemented Cryptobox: open source solution for secure and reliable data sync working with standard and specialised SFTP servers able to exploit NDS2 features for fast & leightweigt enumeration able to cope with concurrency / conflict scenarios supporting client-side encryption! Results of this work: Developed alpha version of the sync client for Windows Gained experience & knowledge about sync & share problems We are open to share this knowledge with TF-Storage

34 Synchronisation solutions, NDS2 Cryptobox THANK YOU!!!