Webtalk Storage Trends

Transcription

1 Sepp Stieger- Technology Watch Fujitsu Webtalk Storage Trends Webtalk Channel TechCommunity New Storage Technologies which ones are hype and reality? How soon will these new trends impact your data center? 0 Fujitsu 2014

2 Storage Trends Trends / Disruptions Storage System Challenges Key Requirements Fujitsus ETERNUS CD10000 Technology OpenStack 1 Fujitsu 2014

3 Storage Industry Distruption Driving Transformation on Premise Flash/ Software Defined Cloud Services commercialon premise Convergence e.g. Mobile Phones Distributed Architectures Source: IDC 04/14 2 Fujitsu 2014

4 IDC 3 rd Platform: Opportunities at the intersection of Mobile, Cloud, Social and Big Data Source: IDC 12/12 From 2013 through 2020, 90% of IT industry growth will be driven by 3rd Platform technologies that, today, represent just 22% of ICT spending Services will be build on innovative mashups of cloud, mobile devices/apps, social technologies, big data/analytics, and more Data Center Transforming Converged systems will account for over 1/3 of enterprise cloud deployments by 2016 Software-defined networks will penetrate 35% of Ethernet switching in the data center Growing importance of mega DC, Service 3 Fujitsu 2014

5 And the others? This page is intentionally left blank. Please listen. 4 Fujitsu 2014

6 Key requirements for future storage architectures Modular scalability of capacity and performance up to 100s of PB Zero planned or unplanned downtime Technology refreshes without system downtime or migration Self-optimizing Efficient management of exponential data growth Lower total costs of capacity The new unified data access (Block, File, Object, future access formats) 7 Fujitsu 2014

7 ETERNUS CD10000 Technology 10 Fujitsu 2014

8 Next Generation Storage from Fujitsu Zero Downtime Immortal System Unlimited Scalability ETERNUS CD10000 The new unified Open Standards Cost optimized ETERNUS CD Fujitsu 2014

9 Immortal System + + Node1 Node2 Node(n) Adding nodes Adding nodes with new generation of hardware Non-disruptive add / remove / exchange of hardware (disks and nodes) Mix of nodes of different generations, online technology refresh Very long lifecycle reduces migration efforts and costs 12 Fujitsu 2014

10 Zero downtime Node1 Node2 Node2 Node(n) Data is automatically distributed over disks and nodes in a self optimizing way 2,3,4.. data copies (replicas) protect against disk and node failures (instead of RAID) In case of a disk or node failure lost data copies are automatically rebuilt A very fast node interconnect enables a fast rebuild Maintenance without any interruption 13 Fujitsu 2014

11 Unlimited Scalability Basic node 12 TB Performance node 35 TB Capacity node 252 TB Cluster of storage nodes Capacity scales by adding storage nodes Each nodes has integrated compute power adding also performance Three node types differing in capacity and performance attributes any mix possible Theoretically scalable to exabytes of data 1 st version of CD10000 is released for max. 224 nodes enabling a max. capacity of 50 Petabyte 14 Fujitsu 2014

12 How Nature Overcomes the Central Access Bottleneck Swarm of birds or fish Source: wikipedia 15 Fujitsu 2014

13 How ETERNUS CD10000 Overcomes the Central Access Bottleneck Disks assume the role of fish in the storage swarm every disk is represented by an OSD (Object Storage Device) Clients can directly access the OSDs (disks, fish ) without a central instance 16 Fujitsu 2014

14 From Scalability to Reliability VMs CLIENT 1 VMs CLIENT 2 Block Redundant Client Interconnect (IP based) Redundant Cluster Interconnect (IP based) VMs CLIENT 3 VMs CLIENT 4 Block Object Block Distributed Redundant Storage Intelligent data Distribution across all nodes and spindles = wide striping (64KB 16MB) Redundancy with replica = 2, 3 8 Thin provisioning Fast distributed rebuild Availability, Fault tolerance Disk, Node, Interconnect Automatic rebuild Distributed HotSpare Space Transparent Block, File access Reliability and Consistency Scalable Performance Pure PCIe- for extreme Transaction processing 19 Fujitsu 2014

15 From Reliability Back To Scalability VMs CLIENT 1 VMs CLIENT 2 Block Block VMs Object CLIENT 3 Redundant Client Interconnect (IP based) Rebalance PCI Rebalance VMs CLIENT 4 Block Distributed Redundant Storage Add nodes without reconfiguration For more Performance For more Capacity Automatic rebalancing avoids hot spots Rebalance Rebalance Redundant Cluster Interconnect (IP based) 20 Fujitsu 2014

16 Seamless Technology Hardware Refresh VMs CLIENT 1 VMs CLIENT 2 VMs CLIENT 3 VMs CLIENT 4 Block Block Object Redundant Client Interconnect (IP based) Rebalance Block Add storage nodes based on new technology without reconfiguration Seamlessly phase out old technology storage nodes PCI Rebalance Restore redundancy Restore redundancy Redundant Cluster Interconnect (IP based) 21 Fujitsu 2014

17 DRS HA/DR Design, up to 80km VMs CLIENT 1 VMs CLIENT 2 VMs CLIENT 3 VMs CLIENT n Block Object Block Object Redundant Client Interconnect (IP) Redundant Client Interconnect (IP) 2 replica will be placed locally The 3 rd /4 th replica will go to the remote site On a site failure, all data will be available at the remote site to minimize the RTO Recovery Time Objective DR Node (at 3 rd site) Redundant Cluster Interconnect (IP) Redundant Cluster Interconnect (IP) 22 Fujitsu 2014

18 DRS HA/DR Design, up to 80km VMs CLIENT 1 VMs CLIENT 2 VMs CLIENT 3 VMs CLIENT n Block Object Block Object Redundant Client Interconnect (IP) Redundant Client Interconnect (IP) 16 Storage Nodes 16 Storage Nodes DR Node (at 3 rd site) Redundant Cluster Interconnect (IP) Redundant Cluster Interconnect (IP) 23 Fujitsu 2014

19 Unified Storage Mgmt. for Block, Object, & File App App Host / VM Client Object Virtual Disk Files & Dirs Librados A library allowing apps to directly access RADOS, with support for C, C++, Java, Python, Ruby, and PHP Ceph Object Gateway (RGW) A bucket-based REST gateway, compatible with S3 and Swift Ceph Block Device (RBD) A reliable and fullydistributed block device, with a Linux kernel client and a QEMU/KVM driver Ceph File System (CephFS) A POSIX-compliant distributed file system, with a Linux kernel client and support for FUSE Ceph Storage Cluster (RADOS) A reliable, autonomous, distributed object store comprised of self-healing, self-managing, intelligent storage nodes 24 Fujitsu 2014

20 Software Defined Storage ETERNUS CD10000 Application Areas 33 Fujitsu 2014

21 Application Areas for Scale-Out Storage Cloud Services Service Providers Telcos Scientific Computing Universities Research Institutions Life Sciences, Chemical Industry, Automotive, Oil & Gas, Multi Media Video Surveillance Broadcasting 34 Fujitsu 2014

22 Cloud Cloud Services Focus on IaaS virtualization platforms with OpenStack / KVM Sync & Share S3 compatible Object Store Cloud service / Telco provider with fast growing expansion strategy based on Open Souce software Enterprise companies with in-house cloud service and additional focus on providing their own services for external customers 35 Fujitsu 2014

23 Scientific Computing Universities, Research Institutions, Industries depending on scientic computing (Life Sciences (e.g. Genomic Sequencing), Weather Forecasting, Climate Research, Automotive, Seismic Analysis, Oil & Gas, ) High performance storage for HPC Mid to long term storage of larger data quantities (historical data) addressing 500TB+ capacities file system object storage 36 Fujitsu 2014

24 Multi Media Video Surveillance Media and Broadcasting companies Fast online streaming services of PB`s of video and audio data Realtime rendering and streaming process of high quality video data Fast growing data environment and secure of consistent availability Secure handling of PB data volumes - always available HA- no downtime scenario 37 Fujitsu 2014

25 Cloud Services Public cloud offering with high degree of standardisation Private cloud offering with significant portions of Open Source / Linux Customers who are open for new infrastructure concepts (OpenStack, Ethernet based storage access) Customers interested in storing mid to long term data via an object oriented interface (e.g. Amazon S3) Scientific Computing / Multi Media larger quantities of unstructured data significant portions of inactive data high throughput for sequential data 38 Fujitsu 2014

26 OpenStack 39 Fujitsu 2014

27 What is OpenStack? Open Source Cloud Platform Simple to implement powerful implementation frameworks Massively scalable Fastest growing cloud platform Developers Biggest industry support 180+ Participating Organisations Platin Gold Corporate 40 Fujitsu 2014

28 OpenStack Architecture Billing Portal OpenStack Cloud Layers Dashboard (Horizon) Metering (Ceilometer) OAM -dhcp -Deploy -LCM Compute (Nova) Hypervisor KVM (ESXi, Hyper-V) Images (Glance) Volume (Cinder) Network (Neutron) + plugins Block (RBD) Object (Swift) S3 (Rados-GW) RADOS Base Operating System (Linux) OpenStack Cloud APIs Authentication (Keystone) EC2 API Physical Server (CPU, Memory,, HDD) and Network 41 Fujitsu 2014

29 OpenStack and ETERNUS CD10000 Billing Portal OpenStack Cloud Layers Dashboard (Horizon) Metering (Ceilometer) OAM -dhcp -Deploy -LCM Compute (Nova) Hypervisor KVM (ESXi, Hyper-V) Images (Glance) Volume (Cinder) Network (Neutron) + plugins Block (RBD) Object (Swift) S3 Fujitsu (Rados-GW) Storage ETERNUS CD10000 RADOS Base Operating System (Linux) OpenStack Cloud APIs Authentication (Keystone) EC2 API Physical Server (CPU, Memory,, HDD) and Network 42 Fujitsu 2014

30 43 Fujitsu 2014

31 Scale out storage customer profiles 44 Fujitsu 2014

32 Typical usage areas Data volumes 500TB and beyond Exponential and unknown data growth Typical usage areas (Cloud) service providers Telecommunications providers Financial institutions R&D, public institutions, universities Companies with large R&D activities Public institutions with huge document repositories Media, broadcasting / streaming companies Business analytics tasks needing fast access to large scale (historical) data 45 Fujitsu 2014

33 Typical usage scenarios (Cloud) service providers IaaS providers Collaboration, , Sharepoint, DropBox services (National) telco providers who are expanding in IT services Large enterprises companies with in-house cloud services IT organizations extending their DC usage in order to provide services for external customers as well Providers with open source platform strategy (KVM, OpenStack, etc.) 46 Fujitsu 2014

34 Typical usage scenarios Companies, public institutions, universities etc. with large online archives Need online fast access to large amounts of current and historical data, e.g. research or legal documents, contracts Use object data for efficient archiving Reliable online storage for 500TB of data and beyond when data migration to tape is not applicable 47 Fujitsu 2014

35 Typical usage scenarios Industry specific service providers (e.g. midsize banking, insurance companies) High availability and a fast online store Disaster recovery site for storage Fast access for customer data and profiles Zero downtime requirements 48 Fujitsu 2014

36 Typical usage scenarios Media and broadcasting/streaming companies Fast online streaming services with petabytes of video and audio data / mediastores Real-time rendering and streaming process of high quality video data Creation of fast copies for media with high access frequency Fast growing data environment; secure consistent availability Secure handling of petabyte data volumes Zero downtime requirements 49 Fujitsu 2014

37 Typical usage scenarios Big data/business analytics Fast access to high volumes of historical data High I/O performance to upload data into analytical servers for number crunching Reduction of storage costs by using highly distributed data on disks instead of using s Using one storage system for production and analytics enabling so that current data can be analyzed without slowing down productive environments 50 Fujitsu 2014

38 Summary: typical usage scenarios ETERNUS CD10000 for enterprise environments High capacity demand Unpredictable data growth Keep high volumes even of historical data online 51 Fujitsu 2014

39 ETERNUS CD10000 Example Configurations OpenStack and VDI Video Surveillance and Broadcasting DropBox, Synch&Share OpenStack, VDI, DropBox, Synch&Share 52 Fujitsu 2014

40 ETERNUS CD10000 example configuration (1) VMs Client VMs Client VMs Client VMs Client OpenStack compute, VDI librbd librbd librbd librbd Perf Node Redundant Client Interconnect (IP based) Perf Node Perf Node Perf Node Perf Node Perf Node Perf Node Perf Node 8 Performance nodes Raw: 8 6TB PCI- = 48TB Raw: 8 48TB = 384TB as Cache pool with replica = 3 ~16TB usable 2.5 as Tiered pool with EC ~275TB usable Bandwidth IOPS Redundant Cluster Interconnect (IP based) Cache Pool Tiered Pool Capacity Pool 53 Fujitsu 2014

41 ETERNUS CD10000 example configuration (3) Apps Client Apps Client Apps Client Dropbox, Sync & Share Object Object Object Cap.Node Redundant Client Interconnect (IP based) Cap.Node Cap.Node Cap.Node 4 Capacity nodes Raw: 4 0.8TB PCIe- Journal Raw: 4 240TB SATA = 960TB 3,5 HDD as Capacity pool with EC ~680TB usable NL- NL- NL- NL- Bandwidth IOPS NL- NL- NL- NL- NL- NL- NL- NL- NL- NL- NL- NL- Redundant Cluster Interconnect (IP based) Cache Pool Tiered Pool Capacity Pool 54 Fujitsu 2014

42 ETERNUS CD10000 example configuration (4) VMs librbd Client VMs librbd Client Apps File Client Apps File Client OpenStack compute, VDI Dropbox, Sync & Share Perf Node Redundant Client Interconnect (IP based) Perf Node Perf Node Perf Node CapaNode Cap.Node Cap.Node Cap.Node NL- NL- NL- NL- NL- SA NL- TA NL- NL- 4 Performance, 4 Capacity nodes Raw: 8 6TB PCI- = 48TB Raw: 4 48TB = 192TB Raw: 4 240TB SATA = 960TB as Cache pool with replica = 3 ~16TB usable 2.5 as Tiered pool with EC ~135TB usable 3,5 HDD as Capacity pool with EC ~680TB usable NL- NL- NL- NL- Redundant Cluster Interconnect (IP based) NL- NL- NL- NL- Bandwidth IOPS Cache Pool Tiered Pool Capacity Pool 55 Fujitsu 2014

43 Next Generation Storage from Fujitsu Zero Downtime Immortal System Unlimited Scalability ETERNUS CD10000 The new unified Open Standards Cost optimized ETERNUS CD Fujitsu 2014

44 Unlimited Scalability Basic node 12 TB Performance node 35 TB Capacity node 252 TB Cluster of storage nodes Capacity scales by adding storage nodes Each nodes has integrated compute power adding also performance Three node types differing in capacity and performance attributes any mix possible Theoretically scalable to exabytes of data 1 st version of CD10000 is released for max. 224 nodes enabling a max. capacity of 50 Petabyte 57 Fujitsu 2014