Advanced Technologies for Cloud Storage. Liwei Ren, Ph.D Data Security Research, Trend Micro August, 2012, Dalian, China

Size: px

Start display at page:

Download "Advanced Technologies for Cloud Storage. Liwei Ren, Ph.D Data Security Research, Trend Micro August, 2012, Dalian, China"

Adam Ball
6 years ago
Views:

2 Backgrounds Liwei Ren, Ph.D, Data Security Research, Trend Micro Mathematician, algorithm scientist & entrepreneur Significant works include 10 academic papers, 12 patents, and a successful DLP startup company. Specialized in data loss prevention (DLP), differential compression, practical algorithms. Interested in data de-duplication, data transfer protocols, and fully homomorphic encryption Trend Micro Global security software company with headquarter in Tokyo, and R&D centers in Nanjing, Taipei and Silicon Valley. Pioneer in cloud security with its product lines Deep Security, SecureCloud An online storage service provider with SafeSync Classification 8/2/2013 Copyright 2011 Trend Micro Inc. 2

4 Cloud Storage and Challenges Cloud Storage Landscape Primary cloud storage Amazon S3 EMC Atoms Online Storage Services Online file storage and sharing DropBox Box.net SafeSync (owned by Trend Micro) SugarSync Online data backup Mozy (owned by EMC) Carbonite Evault Classification 8/2/2013 Copyright 2011 Trend Micro Inc. 4

5 Cloud Storage and Challenges Many challenges for Cloud Storage Big data Data security Data availability Performance due to latency constraints Interoperability/ protocol translation Manageability Regulatory compliances... Classification 8/2/2013 Copyright 2011 Trend Micro Inc. 5

7 Cloud Storage and Challenges Amazon S3 First 1 TB / month for $0.140 per GB Next 49 TB /month for $0.125 per GB Over 5000 TB/month for $0.055 per GB Dropbox Basic : 2GB for free Pro 50: 50GB for $9.99 monthly Pro 100: 100GB for $19.99 monthly Storage space is about cost! Classification 8/2/2013 Copyright 2011 Trend Micro Inc. 7

8 Cloud Storage and Challenges Amazon EC2 for Data Transferring First 1 GB/month $0.000 per GB Up to 10 TB/month $0.120 per GB $1200 Next 40 TB/month $0.090 per GB $3600 AT&T ipad2 3G Data plans: 250MB data $14.99/month 2GB data $25.00/month Bandwidth is a cost! Classification 8/2/2013 Copyright 2011 Trend Micro Inc. 8

9 Cloud Storage and Challenges Three technologies for big data Data De-duplication to reduce cloud storage cost File Sync Protocols to reduce traffic Between clouds Between users and clouds High speed file transport protocols Trend Micro data security research team at Silicon Valley has evaluated and prototyped these technologies Classification 8/2/2013 Copyright 2011 Trend Micro Inc. 9

10 Data De-duplication What is de-duplication? Data de-duplication is a specialized data compression technique for eliminating redundant data, typically to improve storage utilization and network bandwidth De-duplication technology is not new in storage space: It has been successfully applied to D2D data backup for years For example, DataDomain ( acquired by EMC) was a pioneer. There are practices for applying it into the primary storage Classification 8/2/2013 Copyright 2011 Trend Micro Inc. 10

11 Data De-duplication Data de-duplication can be categorized as follows: De-dupe Granularity De-dupe method Pros Cons aka File-level Remove redundant file instances Easy implementation, good performance Trivial compression rate SIS Chunk-level Split a file into chunks and remove redundant file chunks Excellent compression rate CPU intensive Sub-file level Byte-level Compare two files and only store 1 st file & the differences Best compression rate Only suited to data backup Differential Compression FS block Remove redundant FS blocks Easy implementation, fair performance Classification 8/2/2013 Copyright 2011 Trend Micro Inc. 11 File system dependent

14 Chunk Level De-duplication Solution Highlights: Chunking algorithm It splits a file into non-overlapping variable-length chunks. Sliding window rolling hash H(x) The key is to identify the cutting points so that near duplicate files have high probability to have many common chunks For example: set H(x)=0 mod p De-dupe engine Split files into chunks with cutting points identified by chunking algorithm Generate a MD5 hash or other advanced hash from a chunk as a key Store chunks by indexing file references, keys and chunks Restore engine: Restore files from the indices & chunks Copyright 2011 Trend Micro Inc. 14

15 Data De-duplication Business opportunities: Cloud storage cost saving De-dupe based file system VM Image backup in data centers Dropbox applies de-duplication to save cost according to a blog written by Michael Woloszynowicz: The Economics of Dropbox Copyright 2011 Trend Micro Inc. 15

Cloud File Transfer Primary Data center Data Transfer Secondary

17 File Transfer Protocols Big data requires better file transfer protocols Existing File Transfer Protocol FTP and others are slow There are two orthogonal approaches to speed up file transfer process: To reduce data size To accelerate the traffic Classification 8/2/2013 Copyright 2011 Trend Micro Inc. 17

18 File Synchronization Protocols File sync protocols are designed to reduce data size What is a file sync protocol? It is a client server file synchronization algorithm that allows the contents of two files to be synchronized by communicating only the differences between them. Also known as remote differential compression Classification 8/2/2013 Copyright 2011 Trend Micro Inc. 18

19 File Synchronization Protocols Some File Sync Protocols Protocol IP Ownership Authors RSYNC Academic work Dr. Andrew Tridgell RDC Microsoft TSYNC Trend Micro Dr. Liwei Ren( 任力伟博士 ) & Jian-Ying Chen Copyright 2011 Trend Micro Inc. 19

20 File Synchronization Protocols The key of a file sync protocol is to send the differences instead of the whole file: All three protocols RSYNC/TSYNC/RDC are implemented on top of TCP sockets Copyright 2011 Trend Micro Inc. 20

21 File Synchronization Protocols All three protocols RSYNC/TSYNC/RDC are designed based on a same Diff model: The difference between two files is a sequence of COPY/ADD string edit operations that can transfer the reference file to the target file. Copyright 2011 Trend Micro Inc. 21

23 High Speed File Transfer Protocols TCP based file transfer is slow for transferring large files For massive file transferring across WAN, UDP based protocols are much better UDP based high speed file protocols: Protocol IP Ownership Authors UDT Academic work Dr. Yunhong Gu ( 谷云洪博士 ) FASP MTP Aspera Data Expedition Copyright 2011 Trend Micro Inc. 23

25 Synergized File Transfer Protocols We have discussed two categories of orthogonal transfer protocols File sync protocols such as RSYNC and TSYNC High speed file transfer protocols such as UDT and FASP How about integrating them together as iftp? For example, iftp = RSYNC + UDT: Copyright 2011 Trend Micro Inc. 25

27 Summary and Vision To conclude, lets summarize the speech. We have discussed three technologies for cloud storage : Data de-duplication Integrated file transfer protocol (iftp) File sync protocols UDP based high speed file transfer protocols Classification 8/2/2013 Copyright 2011 Trend Micro Inc. 27

28 Summary and Vision What is the next? Can we do better? I would propose a total solution: Cloud Distributed File System (CDFS) on top of both data deduplication engine and iftp. Just read that the open source XtreamFS does not have deduplication capability yet! Classification 8/2/2013 Copyright 2011 Trend Micro Inc. 28

Taxonomy of Differential Compression

Taxonomy of Differential Compression Liwei Ren, Ph.D, Trend Micro SNIA SDC 2015, Santa Clara, California, Sept, 2015 Copyright 2011 Trend Micro Inc. 1 Background Liwei Ren, Ph.D Research interests Data