IBM CAUDIT/RDSI Briefing

Transcription

1 IBM CAUDIT/RDSI Briefing 13 th August 2013 Presented by: Daniel Richards HPC Architect Glen Lance - System Networking Specialist April Neoh HPC Sales Lead Page 1

2 Agenda Introduction GPFS Storage Server (GSS) CINDER Support on IBM Storage Linear Tape File System (LTFS) SONAS Software Defined Networking (SDN) Page 2

3 GPFS Storage Server (GSS) Page 3

4 What is GPFS? GPFS is a mature product with established market presence. It has been generally available since 1998 with research development starting in GPFS Storage HPC IO Checklist: Massive Namespace Central Administration Seamless Growth Linux, AIX, Windows All features included Page 4

5 GPFS History and Evolution We are now at version GPFS 3.3, which introduced administration/policy/performance improvements, additional supported platforms (RedHat 5.x, Windows Server 2008 ) and a new licensing model. Video Streaming Tiger Shark -Real Time -Streaming -Read Perf -Wide Stripe First Called GPFS High Performance Computing GPFS General File Serving -Standards -POSIX Semantics -Large Block -Directory and -Small File perf -Data Management VTS Virtual Tape Server Linux Clusters (Multiple Architectures) AIX Loose Clusters GPFS HPC Research Visualization Digital Media Seismic Weather Exploration Life Sciences 32 bit /64 bit Inter-op (AIX & Linux) GPFS Multi-Cluster GPFS Over WAN Large Scale Clusters 1000 s of nodes GPFS 3.1 ILM - Storage Pools - Filesets - Policy Engine Ease of Administration Multiplenetworks/ RDMA Distributed Token Management GPFS 3.2 SLES/xen Red Hat 5/xen Windows ( ) Faster Failover Multiple NSD servers NFS v4 Support Small File Performance GPFS 3.3 Restricted Admin Functions Improved installation New license model Improved snapshot and Backup Improved ILM policy engine Page 5

6 DARPA HPCS File System and Storage HPCS file system requirements (a subset) Balanced capacity and performance (100 PB file system, 6 TB/s file I/O) Reliability in the presence of localized failures Support for full-up PERCS system (~64K nodes) One trillion files to a single file system 32K file creates per second Streaming I/O at 30GB/s full duplex (for data capture) Storage Requirements Reasonable cost % of system cost Large number of disk drives makes this difficult to achieve Metadata performance requires substantial amount of expensive NVRAM or SSD Reliability - system must continue to be available in spite of component failures One or more drives continually in rebuild Hard error rate between and Silent data corruption- errors after write require check on read Productivity - non-disruptive repair and rebuild Goal: rebuild overhead in the 2-3% range (normal RAID 30%) Parallel file system with wide striping: x% hit on one LUN causes same x% hit to entire file system Page 6

7 nic nic nic nic nic nic nic nic nic nic nic nic nic nic nic ioa ioa nic ioa nic ioa ioa ioa nic ioa nic ioa IBM Presentation to CAUDIT Perseus Software RAID - Motivation Reduce storage subsystem cost! Expect storage to be 10-20% of system cost... difficult to achieve with conventional external RAID controllers and dedicated storage network The insight: leverage existing resources to lower storage cost HFI switch, 500TB/s bisection bandwidth P7 storage node (32 cores) SAS-attached JBOD SSD fast write cache (vs. battery-backed RAM) Dense storage packaging using removable multi-disk carriers 30-40% reduction in storage subsystem cost Cluster Client Nodes I/O NSD nodes I/O NSD nodes SAN Storage controller SAS Fabric JBOD drawers Traditional Storage Controller Page 7 SBOD drawers PERSEUS with dual-ported SBODs

8 As long as we re implementing our own RAID Reduce impact of RAID rebuild on performance HPC file systems stripe data across all IO nodes, performance is only as good as slowest controller when rebuilding At petascale, the system is always rebuilding!! * Perseus Declustered RAID reduces the impact of RAID rebuild on performance Improve data integrity with stronger RAID codes 1-fault-tolerant RAID-5 (8+P): MTTDL about 4 days 2-fault-tolerant RAID-6 (8+2P): MTTDL about 100 years Assuming max PERCS system, 600-GB SAS drives with MTTF=600khrs, hard error rate = 1E-15, and uncorrelated failures, MTTDL ~ 100 years 3-fault-tolerant RAID (8+3P): MTTDL is 6 orders of magnitude better Same assumptions above, MTTDL ~ 130 million years Eliminate silent data corruption Occasionally you don t get back what you wrote! Disks exhibit silent errors, RAID firmware bugs, etc. Perseus on-disk checksums provide end-to-end error detection Page 8 * 100,000 disks * 24 hr/600,000 hr MTBF = 4 failures/day

9 Ok, enough of the Intro, what is GSS Page 9

10 Declustered RAID example RAID 1 21 stripes (42 strips) 7 stripes per group (2 strips per stripe) 49 strips 3 1-fault-tolerant mirrored groups (RAID1) Page 10 3 groups 6 disks spare disk 7 spare strips 7 disks

11 Declustered RAID Rebuild failed disk failed disk time time Rd Wr Rd-Wr Rebuild activity confined to just a few disks slow rebuild, disrupts user programs Rebuild activity spread across many disks, less disruption to user programs Rebuild overhead reduced by 3.5x Page 11

12 Declustered RAID6 Example 14 physical disks / 3 traditional RAID6 arrays / 2 spares 14 physical disks / 1 declustered RAID6 array / 2 spares Decluster data, parity and spare failed disks Number of faults per stripe failed disks Red Green Blue Number of stripes with 2 faults = 7 failed disks Number of faults per stripe failed disks Red Green Blue Number of stripes with 2 faults = 1 Page 12

13 GNR Data Integrity Page 13

14 GNR Disk Hospital Page 14

15 Model 26 Logical layout Page 15

16 GSS Summary Better performance Better Resilience 3 + P End to Check-sum Intelligent rebuild Disk hospital Cheaper Page 16

17 OpenStack CINDER Support Page 17

18 GPFS As the Enterprise Storage Layer for OpenStack Nova / Glance (Compute) GPFS Placement Driver Cinder (Volumes) GPFS Volume Driver Swift (Objects) GPFS Object Driver GPFS: A Reliable, Scalable, POSIX-Compliant Enterprise File System that stores Compute images, Volumes and Objects A common storage layer for images, volumes and objects Avoids data copy Local access of data Adds enterprise storage management features to OpenStack Rapid volume and virtual machine provisioning using file system level copy-on-write function Scale out IO performance through support for shared nothing clusters Resilient volumes through transparent pipeline replication Page 18

19 IBM Storage Systems support for OpenStack Storwize family including and SVC Supported since Folsom release Upgraded for Grizzly iscsi or Fibre Channel Supports advanced Storewize features such as Real-time Compression and Easy Tier, Software defined placement via OpenStack filter scheduler XIV Supported since Folsom release Page 19

20 Self-provisioning of Storwize volumes via OpenStack dashboard Page 20

21 Self-provisioning of Storwize volumes via OpenStack dashboard Page 21

22 Linear Tape File System (LTFS) Page 22

23 What is LTFS A self-describing tape file system LTFS can write files directly to tape media by using operating system commands and without any additional application. The tape drive appears on the operating system as though it is a USB-attached disk drive. IBM created but open. Single drive, Library and EE Page 23

24 Tape Partitioning LTO 5: 1.5TB, 1280 tracks, 16 tracks per wrap, 80 wraps in total. TS1140: 4TB, 2560 tracks, 32 tracks per wrap, 80 wraps in total. Page 24

25 The Solution Tiered Network Storage Single file system view C:/user defined namespace Data Protection Operational High use data, databases, , etc Smaller Easier to protect Faster Time to recovery Smaller backup footprint Time critical applications/data Policy Based Tier Migration LTFS Static data, rich media, unstructured, archive LTFS Scalable LTFS LTFS LTFS Lower cost, scalable storage Data types/uses for tape Static data, rich media, etc. Replication backup strategies Page 25

26 Glues Data Protection and Backup File Storage Tiers File Namespace C:/ Subdir 1 One.txt Video1.mpg Database.db2 Dept0A Memo.txt Movie1.mpg Manuf Inv02.txt Repair.mpg Sales.db2 Page 26 Single file system view C:/ C:/user defined namespace Subdir 1 One.txt Video1.mpg Move files older than X Database.db2 Dept0A Move Memo.txt Policy Based files named Tier Migration *.mpg Movie1.mpg Manuf Never move files named *.db2 Inv02.txt Repair.mpg Move files opened for modification Sales.db2 LTFS Pool 1 One.txt Memo.txt Inv02.txt LTFS Pool 2 Video1.mpg Movie1.mpg Repair.mpg LTFS LTFS LTFS One.txt Memo.txt Inv02.txt Video1.mpg Movie1.mpg Repair.mpg Database.db2 Sales.db2

27 SONAS Page 27

28 SONAS Architecture HTTP Clients NFS Clients CIFS Clients FTP Clients Other Clients This is the internal GPFS parallel file system global cluster IP Network Global Namespace All nodes share workload equally in global SONAS cluster Nodes provide high availability dynamic failover for each other Virtual Management Node Interface... Node Interface Node... Interface Node Global cluster can be dynamically upgraded / changed by adding or removing nodes and/or storage, file system stays up while doing this Massive scalability: max interface nodes = 30, max storage nodes = 60 IP Mgmt. Network Storage Pod Storage Node Storage Node Infiniband Data Network Storage Pod Storage Node Storage Node External Storage All interconnect components in SONAS global cluster are duplexed Storage controller & disk Storage Expansion Storage controller & disk Storage Expansion... Storage controller & disk Storage Expansion Storage controller & disk Storage Expansion Page 28

29 Manage more with less effort with new SONAS GUI Allows for immediate monitoring during environmental changes Monitor and analyze status including performance Proactively plan for more capacity, performance growth as needed Common GUI across IBM storage portfolio: IBM XIV, Storwize V7000 Unified, DS8000 and SONAS Page 29

30 Software Defined Networks (SDN) Page 30

31 What is - Software Defined Networks? Software Defined managing complex hardware interactions with software The ultimate goal is, to make your networks have the properties of virtualised software systems as far as innovation, provisioning speed, and upgrade speed You should want your networks to be as flexible and as agile as compute is IBM Software Defined Network for Virtual Environments (IBM SDN VE) is distributed virtual network overlay software that virtualises a physical data centre network. IBM SDN VE is a multi-hypervisor, high-availability server-centric solution comprising multiple connections that overlay virtual networks onto any physical network that provides IP connectivity It can be best described as VMware for data centre networking. 31 Page 31

32 System Networking Speeds application provisioning Cut network provisioning from days to hours Secure VM mobility across Layer 2 networks and data centers Existing Network Manual physical network configuration change Time consuming network provisioning Server virtualization gated by network provisioning Network with IBM SDN VE-Enabled Servers No physical network configuration change Automated network provisioning Server virtualization not gated by physical network Existing Network Existing Network Physical network change for virtual workload connectivity Yellow Virtual Network RED Virtual Network IBM SDN VE Network VM VM VM VM VM VM VM VM 32

33 System Networking A hypervisor for the network Based on IBM s Distributed Overlay Virtual Ethernet (DOVE) technology Uses existing IP infrastructure No change to existing physical network Provides server-based connectivity for virtual workloads Enables secure and scalable multi-tenancy IBM SDN VE Connectivity Server Virtual Appliance IBM SDN VE Management Console Virtual Appliance APIs RESTful, Quantum OpenStack Cloud/DC Provisioning 16,000 Virtual Networks Today 16 Million Virtual Networks Future Applications IBM SDN VE Virtualized Network V M V M IBM SDN VE vswitch Hypervisor V M Virtual Network 1 V M V M IBM SDN VE vswitch Hypervisor Virtual Network 2 V M V M V M IBM SDN VE vswitch Hypervisor V M V M V M V M Virtual Network 1 Virtual Network 2 Virtual Network 3 Virtual Network 3 IBM SDN VE Gateway Existing IP Network Existing IP Network Virtual Network 3 End Station End Station 33

34 Questions?? Page 34