Building a Multi-protocol, analytics-enabled, Data Lake with Isilon

Transcription

1 Building a Multi-protocol, analytics-enabled, Data Lake with Isilon Ahmad #EMCForum 1

2 Trends 2

3 3

4 Big Data X in T 4

5 Unstructured Data Growth 67% 74% 80% EB 71 EB 133 EB Total Capacity Shipped, Worldwide Unstructured Data Source: IDC 5

6 Big Data V, V and V and more Vs 6

7 7

8 8

9 Shopping The Old Way! 9

10 Wednesday The night of 21 st of March 10

11 11

12 Shopping It is personal! Welcome back to MegaMall Ahmad! 12

13 Special Gifts for our Special Shoppers All that I am, or hope to Paradise be, I owe to lies my at angel the feet of mother. mothers Abraham Prophet Mohammed Lincoln (pbuh) 13

14 Discounts on Razors 20% Off for MACH3 Turbo 14

15 Happy Shopping 15

16 C1- Data Warehouse and Data Mining Traditional Data Sources Existing EDW ERP Store CRM New Data Sources/Formats ETL (DWH) BI / Analytical Tools Machine 16

17 C2- The Four Storage Architectures Clustered Scale Up & Down TYPE 1 Tightly Coupled Scale Out TYPE 2 Loosely Coupled Scale Out TYPE 3 Distributed Share Nothing TYPE 4 Dual Controller Shared Meta-Data Transactional Commits General Purpose Storage Multi-Controller Grid Shared Meta-Data Transactional Commits Mission Critical Workloads Independent Controllers Distributed Meta-Data Transactional Commits Large Content Repositories Independent Controllers Distributed Meta-Data Non-Transactional Commits 3 rd Platform Applications 17

18 C3- Hadoop 18

19 EMC Data Lake Storage Foundation EMC Isilon Outstanding in Critical Capabilities for Scale-Out File 19

20 How Do You Leverage All of this Data? 20

21 TRADITIONAL WORKLOADS NEXT-GEN WORKLOADS NAS DAS SAN CLOUD TAPE OBJECT 21

22 TRADITIONAL WORKLOADS NEXT-GEN WORKLOADS NAS DAS SAN Isilon Scale-Out Data Lake CLOUD TAPE OBJECT 22

23 Next-Gen Access Methods FILE Data 23

24 Enterprise-Grade Features DATA PROTECTION Isilon Scale-Out Data Lake DATA MANAGEMENT DATA SECURITY PERFORMANCE MANAGEMENT 24

25 OneFS: Scale-Out Architecture Single Volume/ File System Simplicity & Ease of Use High Performance Linear Scalability Unmatched Efficiency Easy Growth 25

26 Hadoop on the Scale-Out Data Lake SMB Node reply NameNode Data NFS SMB HTTP SMB, NFS, HTTP, FTP name node name node name node name node data node HDFS 26

27 Largest and Most Scalable File System More scalable than traditional storage systems OneFS scales from 18 TB to 20 PB in a single file system, single volume Scale in minutes with no downtime 27

28 Gain New Levels of Efficiency Isilon AutoBalance BALANCED EMPTY FULL BALANCED EMPTY FULL BALANCED EMPTY FULL BALANCED EMPTY FULL BALANCED EMPTY AutoBalance Automated data balancing across nodes reduces costs, complexity, and risks for scaling storage AutoBalance automatically moves content to new storage nodes while system is online and in production Eliminates hot spots Enables unmatched storage utilization of more than 80% 28

29 Optimize Resources: Automated Tiering Isilon SmartPools Single point of management Single file system/single volume Multiple performance tiers Automatic data movement Policy-based tiering management Transparent reallocation NO application changes Optimize storage resources Automatically match storage resources with data requirements Eliminate data migration Files Reduced cost/tb S-Series Performance X-Series Collaboration NL-Series Active archives 29

30 Isilon Scale-Out Data Lake Single file system, single volume, global namespace More than 80% storage utilization and automated storage tiering Scales from 18 TB to over 20 PB in a single file system Efficient backup and recovery, reliable disaster recovery, and N+1 thru N+4 redundancy Roles-based administration; Authentication Zones; WORM data protection Integrated support for industry-standard protocols including NFS, SMB, HTTP, FTP & HDFS 30

31

32 Internal Data Social Data Trend Data Internet Data Open Data Any Data Real-time Ingestion / CEP Pivotal HD GemFire Structured Data (MPP Database) Advanced Analytics BI, Reporting & Visualization 32