HYDRAstor: a Scalable Secondary Storage

Transcription

1 HYDRAstor: a Scalable Secondary Storage 7th TF-Storage Meeting September 9 th 00 Łukasz Heldt

2 Largest Japanese IT company $4 Billion in annual revenue 4,000 staff Polish R&D company 50 engineers and scientists Owns & sells R&D of critical backend component Scalable disk based storage for backup with global deduplication Started in 00 in NEC Labs by Cezary Dubnicki 007 Product of the year award by SearchStorage.com 008 Product innovation award by Network Products Guide 009/00 FAST conference publication in San Jose Sold in US and Japan since 007 Will be sold in Poland in 0 by 9LivesData in coop. with NEC

3 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC Backup storage Tapes are most common, despite: Sensitive environment requirements Unreliable restore Low performance Manual labor or expensive robots Problematic replication

4 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC 4 Backup storage size Usual backup policy 4-+ full backups 7-0+ incremental Majority of data does not change Data compression : Secondary storage size: 5x-0x more than primary storage Includes many copies of the same data Each data chunk stored 5-0+ times

5 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC 5 Backup storage size Usual backup policy 4-+ full backups 7-0+ incremental Majority of data does not change Data compression : Secondary storage size: 5x-0x more than primary storage Includes many copies of the same data Each data chunk stored 5-0+ times High potential for the deduplication technology.

6 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC 6 Deduplication Save disk space by eliminating duplicates Sample reduction ratio 0: (depends on backup policy) Lowers price of gigabyte Sub-file level deduplication File A File B A A B C D E Only unique blocks are stored Stored blocks A B C D E File A A B C

7 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC 7 Global deduplication Prevent silos of deduped data One system to manage Global vs. siloed dedup

8 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC 8 HYDRAstor product Provides global deduplication using DataRedux performance, storage scalability and data resiliency using Distributed Resilient Data

9 HYDRAstor deployment 9 Interface: CIFS, NFS, Symantec OST Marker filtering for: Tivoli, Netbackup, Networker, CommVault

10 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC 0 HYDRAstor architecture Accelerator Nodes realize performance Storage Nodes realize capacity Accelerator Nodes NFS / CIFS / OST over Ethernet Storage Nodes Internal Network

11 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC HYDRAstor architecture Accelerator Nodes realize performance Storage Nodes realize capacity Accelerator Nodes NFS / CIFS / OST over Ethernet Internal Network Non-disruptive grid expansion Storage Nodes

12 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC HYDRAstor scalability MiniHYDRA single server Storage: TB 40 TB* Performance:. TB / hour AN 4SN Storage: 48 TB 960 TB* Performance:.6 TB / hour 0AN 40SN (4 racks) Storage: 480 TB 9600 TB* Performance: 6 TB / hour * - assuming 0x data reduction through DataRedux

13 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC HYDRAstor scalability Slide from Curtis Preston presentation Curtis Preston is a famous storage analyst owning independent consulting company

14 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC 4 HYDRAstor other features Fully automatic/non-disruptive mgmt Recovery of lost data resiliency Periodic data scrubbing Machine and disk failure recovery Configurable redundancy level erasure coding better than RAID6 Optimized replication Smart resource management

15 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC 5 HYDRAstor backend design Details of the design:

16 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC 6 Programming Model Repository of blocks Content-addressed Immutable Variable-sized hash=0..0

17 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC 7 Programming Model Repository of blocks Content-addressed Immutable Variable-sized Exposed pointers to other blocks E 0..0 hash=0..0

18 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC 8 Programming Model Repository of blocks Content-addressed hash=00.. Root E Immutable Variable-sized Exposed pointers to other blocks E Trees of blocks E 0..0 hash=0..0

19 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC 9 Programming Model Repository of blocks Content-addressed hash=00.. Root E Root E Immutable hash=0..0 Variable-sized Exposed pointers to other blocks Trees of blocks DAGs due to deduplication No cycles possible E E hash=0..0

20 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC 0 Programming Model Repository of blocks Content-addressed hash=00.. Root E Root E Immutable hash=0..0 Variable-sized Exposed pointers to other blocks Trees of blocks DAGs due to deduplication No cycles possible E E Deletion of whole trees hash=0..0

21 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC Programming Model Repository of blocks Content-addressed hash=00.. Root E Root E Immutable hash=0..0 Variable-sized Exposed pointers to other blocks Trees of blocks DAGs due to deduplication No cycles possible E E Deletion of whole trees hash=0..0

22 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC Programming Model Repository of blocks Content-addressed hash=00.. Root E Root E Immutable hash=0..0 Variable-sized Exposed pointers to other blocks Trees of blocks DAGs due to deduplication No cycles possible E E Deletion of whole trees hash=0..0

23 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC Programming Model Repository of blocks Content-addressed Root E Immutable hash=0..0 Variable-sized Exposed pointers to other blocks Trees of blocks DAGs due to deduplication No cycles possible E Deletion of whole trees hash=0..0

24 Decode HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC 4 Failure tolerance: erasure coding Example: N=8, m=5 Original block Encode Redundant Fragments Original Fragments Any fragments can be lost

25 Decode HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC 5 Failure tolerance: erasure coding Example: N=8, m=5 Original block Encode Redundant Fragments Original Fragments Any fragments can be lost Assuming disks array Mirror -copy RAID6 Erasure coding Resiliency Overhead 00% 00% 0% 0% %

26 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC 6 Scalability with DHT: data placement Block location: DHT with prefix routing empty prefix

27 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC 7 Scalability with DHT: data placement Block location: DHT with prefix routing Block mapped to hash prefix hash=0..0 empty prefix Block

28 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC 8 Scalability with DHT: data placement Block location: DHT with prefix routing Block mapped to hash prefix Prefix components Hosted on SNs empty prefix 0 hash=0..0 Block N=4 N components per prefix Node Node Node Node Node 5 Node 6 0

29 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC 9 Scalability with DHT: data placement Block location: DHT with prefix routing Block mapped to hash prefix Prefix components Hosted on SNs empty prefix 0 hash=0..0 Block N=4 N components per prefix Node Store fragments Node Node Node Node 5 Node 6 0

30 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC 0 Scalability with DHT: data placement Block location: DHT with prefix routing Block mapped to hash prefix Prefix components empty prefix hash=0..0 Block Hosted on SNs 0 N=4 N components per prefix Node Store fragments Node Distributed consensus Node Node 4 Node Node 6 0

31 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC Scalability with DHT: data placement Block location: DHT with prefix routing Block mapped to hash prefix Prefix components empty prefix hash=0..0 Block Hosted on SNs 0 N=4 N components per prefix Node Store fragments Node Distributed consensus Node Node 4 Node Node 6 0

32 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC Scalability with DHT: data placement Block location: DHT with prefix routing Block mapped to hash prefix Prefix components empty prefix hash=0..0 Block Hosted on SNs 0 N=4 N components per prefix Node Store fragments Node Distributed consensus Node Node 4 Node Node 6 0

33 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC Scalability with DHT: data placement Block location: DHT with prefix routing Block mapped to hash prefix Prefix components empty prefix hash=0..0 Block Hosted on SNs 0 N=4 N components per prefix Node Store fragments Node Distributed consensus Node Node 4 Node Node 6 0

34 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC 4 Scalability with DHT: data placement Block location: DHT with prefix routing Block mapped to hash prefix Prefix components empty prefix hash=0..0 Block Hosted on SNs 0 N=4 N components per prefix Node Store fragments Node Distributed consensus Load balancing Node Node 4 Node 5 Node

35 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC 5 Data organization: synchrun chains A B C D E F G Data stream split to blocks

36 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC 6 Data organization: synchrun chains A B C D E F G Data stream split to blocks es of blocks computed

37 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC 7 Data organization: synchrun chains A B C D E F G Data stream split to blocks es of blocks computed Routing through DHT

38 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC 8 Data organization: synchrun chains A B C D E F G 00 0 Prefix Data stream split to blocks es of blocks computed Routing through DHT

39 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC 9 Data organization: synchrun chains A B C D E F G 00 0 Prefix Compression 0 00 Data stream split to blocks es of blocks computed Routing through DHT Erasure Coding

40 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC 40 Data organization: synchrun chains 0 A B C D E F G 00 Prefix Compression Erasure Coding 0 00 Data stream split to blocks es of blocks computed Routing through DHT Erasure-coded fragments stored by components

41 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC 4 Data organization: synchrun chains A B C D E F G 00 Prefix Compression 0 00 Data stream split to blocks es of blocks computed Routing through DHT 0 Erasure Coding A D F Erasure-coded fragments stored by components A D F A D F A D F

42 Synchrun HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC 4 Data organization: synchrun chains A B C D E F G 00 Prefix Compression 0 00 Data stream split to blocks es of blocks computed Routing through DHT 0 Erasure Coding Synchrun Synchrun Synchrun A D F Erasure-coded fragments stored by components Grouped into synchruns A D F A D F A D F

43 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC 4 Data organization: synchrun chains A B C D E F G 00 Prefix Compression 0 00 Data stream split to blocks es of blocks computed Routing through DHT 0 Erasure Coding Synchrun Synchrun Synchrun A D F Erasure-coded fragments stored by components Grouped into synchruns A D F A D F Containers stored on disks Fragment metadata separately from data A D F Container Synchrun

44 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC 44 Data organization: synchrun chains A B C D E F G 00 Prefix Compression 0 00 Data stream split to blocks es of blocks computed Routing through DHT 0 Erasure Coding Synchrun Synchrun Synchrun A D F Erasure-coded fragments stored by components Grouped into synchruns A D F A D F A D F Containers stored on disks Fragment metadata separately from data Ordered synchrun chains Preserve order & locality Container Synchrun Manageable

45 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC 45 Data Services: Identification of data resiliency level Missing fragments 0:0 0: 0: 0:

46 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC 46 Data Services: Identification of data resiliency level 0:0 0: 0: 0: Chain scanning

47 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC 47 Data Services: Identification of data resiliency level 0:0 0: 0: 0: Chain scanning

48 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC 48 Data Services: Identification of data resiliency level 0:0 0: 0: 0: Chain scanning

49 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC 49 Data Services: Identification of data resiliency level 0:0 0: 0: 0: Chain scanning

50 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC 50 Data services: reconstruction 0:0 0: 0: 0: Sequential read/write of entire Containers Erasure decoding and re-encoding

51 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC 5 Data services: reconstruction 0:0 0: 0: 0: Sequential read/write of entire Containers Erasure decoding and re-encoding

52 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC 5 Data services: reconstruction 0:0 0: 0: 0: Sequential read/write of entire Containers Erasure decoding and re-encoding

53 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC 5 Data services: fast data transfer 0:0 0: 0: 0: Old component 0: Location of new node (DHT)

54 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC 54 Data services: fast data transfer 0:0 0: 0: 0: Data transfer Old component 0:

55 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC 55 Data services: fast data transfer 0:0 0: 0: 0: Data transfer Old component 0:

56 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC 56 Data services: fast data transfer 0:0 0: 0: 0: Data transfer Old component 0:

57 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC 57 Data services: fast data transfer 0:0 0: 0: 0: Old component 0:

58 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC 58 Data services for deduplication hash=0.. Block 0:0 Choose complete chain 0: 0: 0: Completeness: definitely not a duplicate Deletion interaction: wasn't the block scheduled for deletion?

59 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC 59 Data services for deduplication hash=0.. Block 0:0 0: Query 0: 0:

60 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC 60 Data services for deduplication hash=0.. Block 0:0 0: 0: 0: Local candidate found

61 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC 6 Data services for deduplication hash=0.. Block 0:0 0: 0: Successful dedup 0: Candidate verification

62 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC 6 On-demand data deletion Distributed garbage collection Per-block reference counter stored perfragment Failure-tolerant Block reference counter calculated independently on peer Container chains Interference with duplicate elimination: duplicates resurrection after garbage collection space reclamation in background

63 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC 6 Resource management Configurable load balancing between: backup/restore background tasks (reconstruction, transfer, etc.) garbage collection Shares depend on system state Assigns priority of tasks automatically e.g. reconstruction before transfer or space reclamation Maximizes resources utilization

64 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC 64 Topics for further discussion Features and technical details of HYDRAstor Sales of HYDRAstor in Poland Cooperation with 9LivesData on other projects

65 HYDRAstor: a Scalable Secondary Storage. 9LivesData, LLC 65 Questions? Contact: heldt@9livesdata.com