Can Enterprise Storage Fix Hadoop? PRESENTATION TITLE GOES HERE John Webster Senior Partner Evaluator Group

Transcription

1 Can Enterprise Storage Fix Hadoop? PRESENTATIN TITLE GES HERE John Webster Senior Partner Evaluator Group

2 Agenda What is the Internet Data Center and how is it different from Enterprise Data Center? How is the Apache Software Foundation (ASF) addressing the issues? What needs fixing from the perspective of Enterprise Storage vendors and the Enterprise Storage world? What are the proposed fixes? Can Hadoop fix Enterprise Storage? Can the Internet Data Center/Enterprise Data Center Chasm be Crossed? FYI: I will use vendor names and products as examples only no explicit or implied endorsements 4/13/

3 The Data Center Chasm Internet Data Center Enterpris e Data Center 4/13/

4 Defining the Data Center Chasm Internet Data Center Embraces open source Automates IT Comfortable with systems that run in failure mode Cheep and deep hardware inefficiency not an obvious issue More willing to build their own systems and self-support Manages storage (often JBD) from a systems perspective Enterprise Data Center Prefers proprietary but learning open source Approaches IT automation conservatively Doesn t get failure mode Hardware efficiency-conscious More willing to buy from proprietary vendors and deal with them for support Sees value in storage environment as a place for data and storage management 4 4/13/2014 4

5 4/13/

6 What has the ASF Fixed in HDFS? NameNode SPF NameNode active/standby failover support Snapshot Read-only Copy on Write (CW) included in latest v2 Beta (2.1.0) NFS support Support for NFSv3 in latest v2 Beta (2.1.0) DR Support Distributed Copy (distcp) 4/13/

7 What Needs to be Fixed the Enterprise Storage Vendor Perspective Hadoop NameNode is a single point of failure in V1. Manual failover in v2 (Beta). JobTracker is also a single point of failure For data integrity and protection, HDFS creates three full clone copies of data 3x the storage for each file slow and inefficient If all three copies are corrupted, you re still hosed (reload and start over) 60% of Enterprise Hadoop projects fail or are put on hold Steep learning curve six months is not uncommon for those that actually go from pilot to production No storage tiering Limited (if any) ways to respond to corporate security and data governance policies Difficult to move between cloud and data center Fundamentally a batch process Data in/out processes can take longer than the actual query process Inability to dis-aggregate storage from compute so that the two can be scaled independently Dearth of applications built on top Dearth of people available in the job market to run this beast and the ones that can go for big bucks.and more leading some analysts to believe that Big Data has entered the trough of disillusion 7

8 What Needs to be Fixed the Enterprise Storage Vendor Perspective Hadoop NameNode is a single point of failure JobTracker is also a single point of failure For data integrity and protection, HDFS creates three full clone copies of data 3x the storage for each file slow and inefficient If all three copies are corrupted, you re still hosed (reload and start over) 60% of Enterprise Hadoop projects fail or are put on hold Steep learning curve six months is not uncommon for those that actually go from pilot to production No storage tiering Limited (if any) ways to respond to corporate security and data governance policies Difficult to move between cloud and data center Fundamentally a batch process Data in/out processes can take longer than the actual query process Inability to dis-aggregate storage from compute so that the two can be scaled independently Dearth of applications built on top Dearth of people available in the job market to run this beast and the ones that can go for big bucks.and more leading some analysts to believe that Big Data has entered the trough of disillusion 8

9 Storage in Shared Nothing Network Layer 1-10 Gb Ethernet B8GMR3 nly the Ethernet network is shared 1 Link 2 3 Link 4 5 Link 6 7 Link 8 Pwr Console Active Active Active Active Active Compute Layer Commodity Servers Storage Layer 6-12 disks in each server typically JBD C N T R L DAS N D E 1 N D E 2 DAS DAS DAS DAS N D E 3 Scale to thousands of nodes N D E n In Hadoop, Control = Name Node; Node 1,2 = Data Node 9

10 Hadoop External Storage EMC Isilon Example Shared storage replaces nodelevel DAS HDFS implemented as over the wire protocol on nefs Isilon cluster nodes emulate NameNodes and DataNodes NameNode SPF eliminated Decoupled storage and compute layers Data services, data protection, and DR by nefs Analytics on data in place i.e. minimal if any data moving 4/13/

11 Hadoop External Storage NetApp Example MAPREDUCE Hadoop 10GbE Namenode Secondary Namenode 10GbE Datanodes/ Tasktracke rs FAS Series Data NTAP NFS Metadata Store E-Series Data Stores 4 separate, sharednothing partitions per chassis 6Gb SAS, Direct Connect Source: NetApp Preserves shared nothing architecture and HDFS Decouple compute and storage Hardware RAID: reduction in copies from 3 to 2 NameNode metadata in separate array for faster NameNode recovery DataNode drive failures do not blacklist the DataNode Apply built-in enterprise data and storage management functionality to Hadoop data 4/13/

12 Shared Storage as Secondary Storage Network Layer B8GMR3 1 Link 2 3 Link 4 5 Link 6 7 Link 8 Active Active Active Active Pwr Active Console Compute Layer C N T R L N D E N D E N D E N D E Primary Storage Layer 1 n Secondar y Storage Layer SAN/NAS Secondary Storage Layer Data mirrored or migrated from primary to secondary storage Storage services also live here 12

13 Progression of Yahoo! Source: Yahoo! 4/13/

14 Can Hadoop Fix Enterprise Storage? 4/13/

15 Can Hadoop Fix Enterprise Storage? Modern Enterprise Storage Issues Inflexible and non-elastic Siloed Proprietary pposite of Cheap and deep Bound to three-year product life cycles Developed for the traditional enterprise data center environment Doesn t offer performance at scale and low cost and all at the same time 4/13/

16 Is Hadoop a new Storage Platform? No Yes It s a distributed computing platform for analytics HDFS - Embedded, distributed file system (like scale-out NAS) Data protection and management built-in (like Enterprise Storage) Storage performance at scale and low cost and with native intelligence and all at the same time Growing use case as data repository for existing enterprise BI and Data Warehousing apps the Data Lake 16

17 What Does the Enterprise Want from Big Data? If we could harness all of our data, we would be a much stronger business. * * From CompTIA survey where two thirds of respondents either agreed or strongly agreed with the statement 4/13/

18 Can the Chasm Be Crossed? Internet Data Center Enterpris e Data Center 4/13/

19 Is a new computing paradigm lurking behind the Hadoop hype? 19

20 Summary Hadoop is crossing the chasm A more pragmatic approach to integrating shared storage with Hadoop is emerging The Hadoop Holy Grail: perational (transactional) processing with real time analytics 20