IBM HPC/HPDA/AI Solutions Albert Valls Badia IBM Client Technical Architect IBM Systems Hardware

Transcription

1 IBM HPC/HPDA/AI Solutions Albert Valls Badia IBM Client Technical Architect IBM Systems Hardware June 15th, 2017

2 New Drivers and Directions Datacentric Data Volumes are Exploding Especially Unstructured Data Data Needs to e Colle ted, Ma aged, a d Digested Deriving Insight and Information from the Data requires: A variety of pro essi g steps i a Workflo A variety of processing optimizations Many Analytics Steps can make use of Large In Memory Solvers Energy Efficiency requires: Processing Elements that are Optimized to the task Energy and Data aware Workflow Management The Open Power Foundation provides innovation opportunities to a variety of Partners Making innovations like A elerators Co su a le is critical Price/Performance Workflow Dependency Graph Technology and Processors Firmware / OS Accelerators Software Storage Network Full system stack innovation required 2

3 OpenPOWER and Innovation (strategy started in 2014) OpenPOWER: Bringing Partner Innovation to Power Systems IBM Stack Research And Innovation IBM Mellanox OpenPower Open Innovation Google NVIDIA TYAN 5 initial members 200+ members 24 countries

4 OpenPOWER Innovation Pervasive in System Design (21 TFlops/node) NVIDIA: Tesla P100 GPU with NVLink NVLink Interface Ubuntu by Canonical: Launch OS supporting NVLink and Page Migration Engine Wistron: Platform co-design Mellanox: InfiniBand/Ethernet Connectivity in and out of server Samsung: 2.5 SSDs HGST: Optional NVMe Adapters Hynix, Samsung, Micron: DDR4 IBM: POWER8 CPU 4

5 POWER8: Leadership performance - designed for Memory Intensive Workloads DRAM Chips Memory Buffer 12 cores 96 threads 4 cache levels Up to 1/2 TB per socket Up to 230 GB/s sustained Consistent speed POWER8 Faster cores 8 Threads per Core Bigger cache Accelerator direct links 3x higher memory bandwidth, 1 TB/Socket 5

6 Differentiated Acceleration - CAPI and NVLink CAPI-attached Accelerators NVIDIA Tesla GPU with NVLink POWER GB/s Coherence Bus Graphics Memory Graphics Memory CAPP 80 GB/s Peak* System Memory New Ecosystems with CAPI PSL FPGA or ASIC Partners innovate, add value, gain revenue together w/ibm Technical and programming ease: virtual addressing, cache coherence Accelerator is hardware peer POWER8 with NVLink Future, Innovative Systems with NVLink Faster GPU-GPU communication Breaks down barriers between CPU-GPU New system architectures 6

7 IBM Power Accelerated Computing Roadmap Mellanox Interconnect Technology ConnectX-4 EDR Infiniband PCIe Gen3 ConnectX-4 EDR Infiniband CAPI over PCIe Gen3 HDR Infiniband Enhanced CAPI over PCIe Gen4 TBD NVIDIA GPUs Kepler PCIe Gen3 Pascal NVLink Volta Enhanced NVLink TBD POWER9 IBM CPUs POWER8 Enhanced CAPI & NVLink POWER8 with NVLink NVLink CAPI Interface POWER Server System Name TBD S822LC Firesto e S8 LC for HPC Mi sk Witherspoo

8 FLOPS are not the only PKI in HPC: example workflow in seismic analysis. Read from storage Memory load Preporcessing Realtime algorithm execution Visualization and Insight Simulation and modeling 7/3/2017 Every step in the workflow takes advantage of different hardware capabilities. Therefore the need for a balanced system design. 8

9 IBM Data Centric Computing Strategy: HPC->HPDA

10 Introducing IBM Spectrum Scale Highly scalable high-performance unified storage for files and objects with integrated analytics Remove data-related bottlenecks with a parallel, scale-out solution Enable global collaboration with unified storage and global namespace Optimize cost and performance with automated data placement POSIX NFS SMB Swift/S3 Spectrum Scale Ensure data availability, integrity and security with erasure coding, replication, snapshots, and encryption Unified Scale-out Data Lake File In/Out, Object In/Out; Analytics on demand. High-performance native protocols Single Management Plane Cluster replication & global namespace Enterprise storage features across file, object & HDFS HDFS Compression Encryption SSNR SSD Disk Fast Disk Slow Disk Tape

11 11

12 IBM Spectrum Scale: Parallel Architecture No Hot Spots NSD Client All NSD servers export to all clients in active-active mode Spectrum Scale stripes files across NSD servers and NSDs in units of file-system block-size NSD Servers File-system load spread evenly Storage Easy to scale file-system capacity and performance while keeping the architecture balanced 12 Storage NSD Client does real-time parallel I/O to all the NSD servers and storage volumes/nsds

13 Spectrum Scale Cluster Overview NFS Clients SMB Clients OpenStack Swift Clients Servers use Disk Volumes/LUNs NFS exports SMB Shares HTTP GET/PUT (Swift) Spectrum Scale Protocol Nodes NFS, SMB, OpenStack Clustered Swift Failover Application Nodes (Oracle,ERP, HPC Cluster) Spectrum Scale Clients /file_systema /file_systema Up to 16 (SMB) or 32 (NFS) servers Spectrum Scale NSD Protocol Ethernet Network (TCP/IP) or Low Latency Network (Infiniband) Spectrum Scale File Servers Commodity Servers (x86_64 or Power) Spectrum Scale Native RAID Controllers /file_systema JBODs JBODs JBODs IBM Elastic Storage Solution Block Storage Block Storage Heterogeneous Block Storage File-system load spread evenly across all the servers. No Hot Spots Data is stripped across servers in block-size No single-server bottleneck Can share access to data with NFS, SMB and Swift S3 Easy to scale while keeping the architecture balanced Can add capacity and performance

14 Spectrum Scale Architecture Highlights: Scalability Data scalability Capacity: Large number of disks/luns in a single file system Throughput: wide striping, large block size Capacity efficient (data in i-node, fragments) Multiple nodes write in parallel (even within single file) Metadata scalability Wide striping of all metadata (inodes, indirect blocks, directories, allocation maps...) Scalable data structures: Segmented allocation map, Extensible hashing for directories Highly scalable, distributed lock manager: After o tai i g lo k toke, each node can cache metadata, update locally, write back directly Fine-grain locking, when necessary: shared inode write locks, byte-range locks lock directory entries by name (hash) Dy a i ally ele ted metanode collects inode, ind block & directory updates

15 Speed and simplicity: Graphical user interface Reduce administration overhead Graphical User Interface for common tasks Performance monitoring Problem determination Easy to adopt Common IBM Storage UI Framework Integrated into Spectrum Control Storage portfolio visibility Consolidated management Multiple clusters

16 Spectrum Scale Built-in Tiering (ILM) Challenge Data growth is outpacing budget Low-cost archive is another storage silo Flash is u der utilized e ause it is t shared Lo ally atta hed disk a t e used ith e tralized storage Migration overhead is preventing storage upgrades Automated data placement Span entire storage portfolio, including DAS, with a single namespace Policy driven data placement & data migration Share storage, even low-latency flash Automatic failover and seamless file-system recovery Lower TCO Powerful policy engine Information Lifecycle Management Fast etadata s a i g a d data o e e t Automated data migration to based on threshold Users not affected by data migration Example: Online storage reaches 90% full then move all 1GB or larger files that are 60 days old to offline to free up space Automation Small files last accessed > 30 days System pool (Flash) accessed today and file size is <1G last accessed > 60days Gold pool (SSD) Silver pool is >60% full Drain it to 20% Silver pool ( NL SAS)

17 Spectrum Scale HDFS Transparency Challenge Separate storage systems for ingest, analysis, results HDFS requires locality aware storage (namenode) Data transfer slows time to results Different frameworks & analytics tools use data differently HDFS Transparency Map/Reduce on shared, or shared nothing storage No waiting for data transfer between storage systems Immediately share results Si gle Data Lake for all appli atio s Enterprise data management Archive and Analysis in-place Traditional Analytics Solution Data ingest Existing System A A Analytics System A Export result In-place Analytics Solution Analytics System Existing System A A A HDFS Transparency Spectrum Scale File System File Object

18 Spectrum Scale Compression Transparent compression for HDFS transparency, Object, NFS, SMB and POSIX interface. Improved storage efficiency Typically 2x improvement in storage efficiency Improved I/O bandwidth Read/write compressed data reduces load on storage Improved client side caching Caching compressed data increases apparent cache size 18 Per file compression Use policies Compress cold data Data not being used/accessed

19 Spectrum Scale Encryption Native Encryption of data at rest Files are encrypted before they are stored on disk Keys are never written to disk No data leakage in case disks are stolen or improperly decommissioned Secure deletion Ability to destroy arbitrarily large subsets of a file system No digital shredding, no overwriting: Security deletion is a cryptographic operation Use Spectrum Scale Policy to encrypted (or exclude) files in fileset or file system Generally < 5% performance impact

20 Performance Feature Spectrum Scale Local Read-Only Cache (LROC) Application Nodes Benefits Expands local node file cache (Pagepool) Leverages fast local storage Can reduce load on central storage Transparent to applications Can use inexpensive local devices Where to use it Protocol Node Virtual Machine storage Large Memory Analytics LROC LROC Easy to enable NSD Type localcache Define only this node as NSD server

21 Performance Feature Spectrum Scale Highly Available Write Cache (HAWC) Benefits Speeds-up small writes Used by IBM Elastic Storage Server Application Nodes Local Storage Shared Storage Where to use it Logs handle small writes Any storage architecture Shared Disk Shared Nothing (Use replication) IO Sizes up to 64KiB Flash Easy to enable Create a system.log pool Enable write-cache on the file system

22 Spectrum Scale Multicluster: cross-cluster sharing Cross-mounting file systems between Spectrum Scale clusters Separate clusters = separate administration domains When connection is established, all nodes are interconnected All nodes in both clusters must be within same IP network segment / VLAN Channel can be encrypted (openssl) 22

23 Synchronous Replication & Stretched Cluster Performed synchronously by the node who writes to disk Synchronous replication happens within Spectrum Scale cluster I/O it does not return to the application until both copies are written Active/Active data access Read from fastest source DR with automatic failover and seamless file-system recovery If replication between sites -> Spectrum Scale Stretched Cluster Application Whichever is fastest 23 Synchronous replication

24 Spectrum Scale Active File Management (AFM) An asynchronous, cross-cluster, data-sharing utility Functions well over unreliable and high latency networks Extends global name space between multiple WAN dispersed locations to share and exchange data asynchronously Ca hes lo al opies of data distri uted to o e or read and write performance ore lusters to i pro e lo al As data is written or modified at one location, all other locations see that same data 24

25 Spectrum Scale AFM Main Concepts Home - Where the information lives. Owner of the data in a cache relationship Cache - Fileset in a remote cluster that points to home The relationship between a Cache and Home is one to one Cache knows about its Home. Home does not know a cache exists Data is copied to the cache when requested or data written at the cache is copied back to home as fast as possible 25

26 IBM Elastic Storage Server (ESS) is a Software Defined Solution Clients Clients FDR IB 10/40 GbE Commodity file servers FDR IB 10/40 GbE Spectrum Scale Server 1 Spectrum Scale Server 1 Spectrum Scale Server 2 Custom dedicated Disk Controllers Spectrum Scale RAID Migrate RAID and disk management to commodity file servers! JBOD Disk enclosures Spectrum Scale Server 2 Spectrum Scale RAID Commodity file servers with RAID and disk management JBOD Disk enclosures Spectrum Scale Native RAID is a software implementation of storage RAID technologies within Spectrum Scale. It requires special Licensing It is only approved for pre-certified architectures such as Lenovo-GSS, IBM-ESS (Elastic Storage Server)

27 Advantages of Spectrum Scale RAID Use of standard and i e pe sive disk drives Erasure Code software implemented in Spectrum Scale Data is declustered and distributed to all disk drives with selected RAID protection 3-way, 4-way, RAID6 8+2P, RAID6 8+3P Faster rebuild times As data is declustered, more disks are involved during rebuild Approx. 3.5 times faster than RAID-5 Minimal impact of rebuild on system performance Rebuild is done by many disks Rebuilds can be deferred with sufficient protection Better fault tolerance End to end checksum Much higher mean-time-to-data-loss (MTTDL) Spectrum Scale RAID JBODs

28 RAID algorithm Two types of RAID: 3 or 4 way replication or 3 way parity 2-fault and 3-fault tolera t odes RAID-D2, RAID-D3 2-fault tolerant codes 3-fault tolerant codes 8 + 2p Reed Solomon 3-way Replication (1+2) 8 + 3p Reed Solomon 4-way Replication (1+3) 8 strips (GPFS block) 2 or 3 redundancy strips 1 strip (GPFS block) 2 or 3 replicated strips

29 Rebuild overhead reduction example 31

30 Declustered RAID6 example

31 Critical Rebuild Performance on GL6 8+2p Spectrum Scale RAID 6 minutes for a critical rebuild As one can see during the critical rebuild impact on workload was high, but as soon as we were back to a single parity protection the impact to the customers workload was <2% Data Integrity Manager prioritizes tasks: Rebuild, Rebalance, Data scrubbing and proactive correction JBODs

32 End-to-end checksum True end-to-end checksum fro Repairs soft/latent read errors Repairs lost/missing writes. disk surfa e to lie t s Spe tru S ale i terfa e Checksums are maintained on disk and in memory and are transmitted to/from client. Checksum is stored in a 64-byte trailer of 32-KiB buffers 8-byte checksum and 56 bytes of ID and version info Sequence number used to detect lost/missing writes. 8 data strips 32-KiB buffer 64B trailer ¼ to 2-KiB terminus 3 parity strips

33 IBM Elastic Storage Server family GS odels use Support drives: GL odels use Support drives: U. JBODs or SSDs. TB,.8TB SAS, GB, 8 GB,. TB SSD. U. JBODs 4TB,6TB,8TB NL-SAS. HDDs Supported NICs: 10GbE, 40GbE Ethernet and FDR or EDR Infiniband FC FC 5887 Net Capacity 4TB = 327TB 6TB = 491TB 8TB = 655TB FC 5887 FC 5887 Net Capacity 400GB = 13TB 800GB = 26TB 1.6TB = 53TB 1.2TB = 35TB 1.6TB = 53TB Net Capacity 4TB = 673TB 6TB = 1PB 8TB = 1.3PB FC FC 5887 Net Capacity 400GB = 6TB 800GB = 13TB 1.6TB = 26TB Net Capacity 1.2TB = 121TB 1.6TB = 182TB Net Capacity 400GB = 28TB 800GB = 57TB 1.6TB = 115TB 1.2TB = 78TB 1.6TB = 117TB Net Capacity 4TB = 1PB 6TB = 1.5PB 8TB = 2PB FC 5887 FC 5887 FC FC FC 5887 FC 5887 FC Model GS1 Model GS2 Model GS4 Model GS6 24 SSD 46 SAS + 2 SSD or 48 SSD Drives 94 SAS + 2 SSD or 96 SSD Drives 142 SAS + 2 SSD 2 GB/Sec SAS 12 GB/Sec SSD 5 GB/Sec SAS 16 GB/Sec SSD 6 GB/Sec 7 GB/Sec Model GL2 Analytics Focused Model GL4 Analytics and Cloud Model GL6 PetaScale Storage 2 Enclosures, 12U 116 NL-SAS, 2 SSD 4 Enclosures, 20U 232 NL-SAS, 2 SSD 6 Enclosures, 28U 348 NL-SAS, 2 SSD 5-8 GB/Sec 10 to 16 GB/Sec 10 to 25 GB/sec

34 ESS New Models Performance and Capacity 23 GB/s 17 GB/s 11 GB/s 8 GB/s Model GL2: 2 Enclosures, 12U 116 NL-SAS, 2 SSD Model GL4: 4 Enclosures, 20U 232 NL-SAS, 2 SSD New! Model GL2S: 2 Enclosures, 14U 166 NL-SAS, 2 SSD New! Model GL4S: 4 Enclosures, 24U 334 NL-SAS, 2 SSD 34 GB/s 25 GB/s Max: 1.6PB raw Net Capacity 4TB = 508PB 8TB = 1PB 10TB = 1.27PB ESS 5U84 Storage ESS 5U84 Storage ESS 5U84 Storage ESS 5U84 Storage ESS 5U84 Storage ESS 5U84 Storage ESS 5U84 Storage Max: 1.8PB raw ESS 5U84 Storage ESS 5U84 Storage ESS 5U84 Storage ESS 5U84 Storage Max: 3.3PB raw Net Capacity 4TB = 1PB 8TB = 2PB 10TB = 2.5PB Spectrum Scale ESS Model GL6: 6 Enclosures, 28U 348 NL-SAS, 2 SSD ESS 5U84 Storage Max:.9PB raw New! Model GL6S: 6 Enclosures, 34U 502 NL-SAS, 2 SSD Max: 2.8PB raw Max: 5PB raw Net Capacity 4TB = 1.5PB 8TB = 3.1PB 10TB = 3.9PB

35 Sequential throughput vs. Capacity

36 Software Defined Compute: IBM Platform Computing Delivering a highly utilized shared services environment optimized for time to results Traditional Application Examples Simulation Analysis Design Big data Software Defined Application Simulation & Modeling Big Data / Hadoop Analytics Long Running Services Prioritized matching of supply with demand IBM Platform Computing IT constrained Long wait times Low utilization IT Sprawl Repeated for many apps and groups Clusters Grid Cloud Faster results Make lots of computers look like one Benefits High utilization Throughput Performance Prioritization Reduced cost Fewer resources

37 Overall Artificial Intelligence (AI) Space Cognitive / ML/DL Human Intelligence Exhibited by Machines New class of applications Machine Learing & Training Pattern matching Image Real-time decision support Complex workflows Data Lakes Machine Learning Human Trained using large amounts of data & ability to learn how to perform the task Deep Learning IT Systems break tasks into Artificial Neural Networks New Data Sources: NoSQL, Hadoop & Analytics Extend Enterprise applications Finance: Fraud detection / prevention Retail: shopping advisors Healthcare: Diagnostics and treatment Supply chain and logistics Extend Predictive Analytics to Advance Analytics with AI Growing across Compute, Middleware, and Storage 39

38 PowerAI Platform Deep Learning Frameworks Supporting Libraries Accelerated Servers and Infrastructure for Scaling 40 Caffe NVCaffe IBMCaffe TensorFlow DL4J Theano OpenBLAS Bazel Cluster of NVLink Servers Distributed Frameworks NCCL Spectrum Scale: High-Speed Parallel File System Torch DIGITS Scale to Cloud Coming Soon

39 Where to start? IBM Power System S822LC Ej. Universidad Carlos III Barcelona Supercomputing Center Parallel Computing The Deep Learning Server + Ej. Molecular dynamics. Centro de Biología Molecular GPU development And optimisation Machine Learning Deep Learning 20 x POWER8 cores with NvLink Hasta 1TB DDR4 Mem with NvLink Hasta Tesla P s. cores 20 Core POWER GB + 1 GPU Nvidia Volta Starting at IVA

40 Questions? 42