ECS ARCHITECTURE DEEP DIVE #EMCECS. Copyright 2015 EMC Corporation. All rights reserved.

Transcription

1 ECS ARCHITECTURE DEEP DIVE 1

2 BOOKING HOTEL IN

3 TODAY! 3

4 MODERN APPS ARE CHANGING INDUSTRIES 4

5 @scale unstructured global accessible web many devices 5

6 MODERN APPS ARE CHANGING INDUSTRIES 6

7 MODERN APPS ARE BREAKING TRADITIONAL STORAGE NOT DESIGNED FOR MODERN APPLICATIONS AND HYPERSCALE Architecture is too complex Locking, replication, High Availability, geodistribution is complex Not Web or firewall friendly Distributed (WAN) access is complex Storage silos impede development Different hardware for every data type and access protocol 7

8 CUSTOMER PAIN POINTS Help Me Reduce Costs Give Me Freedom of Choice I Want a Path to the Cloud 8

9 ECS CLOUD STORAGE PLATFORM GEO DISTRIBUTED ACTIVE-ACTIVE ARCHITECTURE 9

10 ECS PLATFORM ADVANTAGES Global consistency No Single Point of Failure Scalability 10

11 ECS DESIGN PHILOSOPHY Software Defined Storage 11

12 Sys Admin Namespace Admin Object User STORAGE AS A SERVICE Namespace 1 Namespace 2 Replication Group VDC 1 Storage Pool Storage Pool VDC 2 12

13 OPTIMIZED STORAGE OVERHEAD # of Data Centers Overhead x x x x x x x x Low Storage Overhead: ~1.8x replication over head across 4 sites WAN Optimization: All node and disk failures are repaired within the site, without any WAN traffic. Increasing the number of data centers lowers storage overhead and increases protection Business Value: Hyperscale economics to leverage key assets and compete with public cloud IaaS 13

14 BUILD YOUR OWN ECS (DIY) HW & OS agnostic Self-contained Docker containers Timeline Dev/Test (now) Production (EOY 2015) 14

15 DEEP DIVE 15

16 STORAGE ARCHITECTURE 16

17 CHUNK Acks Node Chunk Writer Nodes 17

18 DATA PATH 18

19 LAYERS IN SYSTEM 19

20 20

21 SYSTEM INFORMATION STORE - VNEST Paxos based replicated store Ability to dynamically change the group membership without downtime Used to store the information about the cluster itself Not used by data storage path 21

22 MULTI PROTOCOL ACCESS 22

23 MULTIPLE PROTOCOL SUPPORT Support for object protocols (S3, Swift, Atmos, CAS), and HDFS. NFS support coming soon. Ability to access the same data through different protocols at the same time. 23

24 INTERNAL STRUCTURE Core engine implementation is superset of file and object features. Internal structure supports hierarchy of elements. The hierarchy is represented using key-value in the distributed index. Example of hierarchy- Tenant > Bucket > Object Fileshare > Subdir > Subdir > > File 24

25 GEO REPLICATION 25

26 DISASTER RECOVERY Site-1 Site-2 A A D E Site-3 D 26

27 GLOBAL STORAGE OVERHEAD OPTIMIZED DATA ACCESS, PROTECTION AND EFFICIENCY Overhead # of Data Centers 27

28 GLOBAL CONSISTENCY SEMANTICS 28

29 STRONG CONSISTENCY 29

30 TEMPORARY SITE FAILOVER 30

31 Better Multi-Site Performance/Availability New Monitoring Diagnostics ECS FREE download 31

32 Better Multi-Site Performance - Increased Availability *Small Objects: write 6X faster - read 2X faster *Large Objects: write 5X faster - read 9X faster Geo Caching at secondary sites Dynamic cache sizing on all nodes Improved failover scheme Reduced application latency Access to data during a temporary outage Improved RPO granularity *Internal EMC benchmark with comparable object platforms 32

33 Monitoring/Diagnostics Improved multi-tenant/bucket views Trend charts, system health, usage Easily create/delete modify quotas Simplified management Increased visibility Comprehensive metering and reporting 33

34 FREE ECS Software 2.0 Free Unlimited capacity Free Unlimited time Go from application prototype to production faster Container-ized download available on emc.com Community driven support 34

35 ECS USE CASES Global Content Repository Cloud Modern/ Apps Analytics Archive Internet of Things Sensor data Global Namespace SITE 1 SITE 2 SITE 3 35

36 ECS ECOSYSTEM Enterprise Information Archiving Enterprise Content Management Protocols Analytics CAS Analytics Sync & Share Cloud Gateways Migration 36

37 ECS IN ACTION Analytics: Data in Place Analysis Required ECS created a multi-protocol data lake that is geo-distributed and can accept huge volumes of data from a variety of sources Global Content Repository Massive Scale Required ECS is being used to Digitize 40 million pages, and deliver the content to anyone, anywhere on any device Analytics - Ability to add dense storage without more compute ECS scales compute and storage independently to ensure that the data in the HDFS archive is accessible to analytics applications Tensor Company LTD Modern Applications: Multi protocol object support ECS delivers multi protocol object support to offer customized tax software services using S3 and Swift *NDA Only Not public references 37

38