Brent Gorda. General Manager, High Performance Data Division

Transcription

1 Brent Gorda General Manager, High Performance Data Division

2 Legal Disclaimer Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked "reserved" or "undefined". Intel reserves these for future definition and shall have no responsibility whatsoever for conflicts or incompatibilities arising from future changes to them. The information here is subject to change without notice. Do not finalize a design with this information. The products described in this document may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel technologies features and benefits depend on system configuration and may require enabled hardware, software or service activation. Performance varies depending on system configuration. No computer system can be absolutely secure. Check with your system manufacturer or retailer or learn more at intel.com. Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. Tests document performance of components on a particular test, in specific systems. Differences in hardware, software, or configuration will affect actual performance. Consult other sources of information to evaluate performance as you consider your purchase. For more complete information about performance and benchmark results, visit Intel does not control or audit the design or implementation of third party benchmark data or Web sites referenced in this document. Intel encourages all of its customers to visit the referenced Web sites or others where similar performance benchmark data are reported and confirm whether the referenced benchmark data are accurate and reflect performance of systems available for purchase. Intel, Xeon, Xeon Phi and the Intel logo are trademarks of Intel Corporation in the U.S. and other countries. *Other names and brands may be claimed as the property of others. Copyright 2016 Intel Corporation. All rights reserved. 2

3 Today: ML Data Movement with Lustre* Terabytes of Data Petabytes of Data 100 s GB/s of Bandwtih Network Infrastructure DATA CENTER/ CLOUD Lustre Parallel F/S Collection: Remote Ruggedized 100 s MB/s of IO Data Ingress Asynchronous upload Formatting / Authentication Security Upto GB/s of IO Data = Memories CPUs = Brain More Data== Better Results More CPU = Faster Results Machine Learning 20GB/s+ per PB 10 s 100 s of PB Data Growth > 2 PB Month Rack scale compute 3

4 Lustre with OpenStack is a growing area 4

5 HPE Scalable Storage with Intel Enterprise Edition for Lustre* High Performance Storage Solution Meets Demanding I/O requirements Performance measured for an Apollo 4520 building block Up to 17 GB/s Read/15 GB/s Writes with EDR 1 Up to 16 GB/s Reads and Writes with OPA 1 Up to 21GB/s Reads and 15GB/s Writes with all SSD s² Designed for PB-Scale Data Sets Density Optimized Design For Scale Dense Storage Design Translates to Lower $/GB Limitless Scale Capability Solution Grows Linearly in Appliance Like Delivery Model Built on Apollo 4520 HPC Clients Apollo 6000 Pre-Configured Flexible Solution Deployment Services for Installation Simplified Wizard Driven Management thru Intel Manager for Lustre Innovative Software Features Leading Edge Yet Enterprise Ready Solution ZFS software RAID provides Snapshot, Compression & Error Correction ZFS reduces hardware costs with uncompromised performance Rigorously Tested for Stability & Efficiency HPE Scalable Storage with Intel EE for Lustre* DL360 + MSA 2040 OmniPath DL360 (Intel Management Server) 1: Different Conditions & Workloads can affect the Performance 2: 24 x1.6tb MU SSD s in A4520 no JBOD

6

7 Frequency of Access Byte-addressable Persistent Memory Cold Warm Hot ` Disk Drives Random Access Time (Gap) SSD Disk Drives SSD (PCIe) SSD NVMe SSD NVMe Disk Drives Disk Drives Disk Drives NVDIMM NVMe Rotational NAND 3D-Xpoint 7

8 Disruptive Technologies 3D Xpoint + Omni-Path Byte granular Ultra low latency ~0.01 µs storage latency + ~1 µs network latency = ~1 µs hardware latency Conventional Storage Stack Block/page granular and locking High overhead ~1 µs kernel/user context switch + ~10 µs communications software + ~100 µs filesystem & block I/O stack = ~100 µs software latency Entirely masks HW capabilities!!! ~ = order of magnitude µs = micro seconds 8

9 Disruptive Technologies 3D Xpoint + Omni-Path Byte granular Ultra low latency ~0.01 µs storage latency + ~1 µs network latency = ~1 µs hardware latency Conventional Exascale NEW Storage Stack Arbitrary alignment and granularity Ultra low overhead OS bypass comms + storage Shared nothing Deliver HW performance!!! 100x/1000x increase in data velocity!!! ~ = order of magnitude µs = micro seconds 9

10 Lightweight Storage Stack End-to-end OS bypass Mercury userspace function shipping MPI equivalent communications latency Built over libfabric Applications link directly with DAOS lib Direct call, no context switch No locking, caching or data copy Userspace DAOS server Mmap non-volatile memory (NVML) NVMe access through SPDK* User-level thread with Argobots** FPGA offload * ** HPC Application DAOS library Bulk transfers Mercury/libfabric NVRAM NVMe DAOS Server Storage Server 10

11 Version = epoch Ultra-fine grained I/O Index Mix of storage technologies NVRAM (3D Xpoint DIMMs) DAOS metadata & application metadata Byte-granular application data NVMe (NAND, 3D NAND, 3D Xpoint) Cheaper storage for bulk data Multi-KB I/Os are logged & inserted into persistent index All I/O operations tagged/indexed by version Non-destructive write: log blob@version Consistent read: blob@version No alignment constraints read@v3 v3 v2 v1 Being written Committed NVRAM NVMe Extents 11

12 Extreme Scale-out Scalable I/O Lockless, no read-modify-write Producers not blocked by consumers And vice-versa Conflict resolution in I/O middleware No system-imposed, worst case serialization Ad hoc concurrency control mechanism Scalable communications Track process groups/jobs not individual compute node Tree-based broadcast Scalable metadata Collective open/close Tree-based caching, refcount, open handles, Distributed/global transactions No object metadata maintained by default Shared-nothing distribution schema Algorithmic placement Scales with # storage nodes Data-driven placement Scales with volume of data Performance domains 12

13 DAOS Ecosystem Legion HPC NetCDF HPC HPC application, storage service/engine, machine learning HDF5 + Extensions HPC POSIX HPC MPI-IO HPC KV Store Varia USD Computer animation HDFS/ Spark Analytics Cloud Integra tion DAOS Apache

14 DAOS Roadmap ESSIO (Q Q2 17) Alpha quality N-way replication Online rebuild Multi-tier prototype Follow-on projects DAOS Productization Next gen HW support System integration Future Progressive layout Erasure code Security Model

15 DAOS Resources (Apache 2.0) Public git repository git clone Browsable source code: Mirror on github: Released under Open source Apache 2.0 License Leveraging other open source projects DoE: Mercury, Argobots Intel: Libfabric, PMIx, NVML, SPDK, ISA-L 15

16

17 High IOPS Lustre Configurations with Intel SSDs Utilize single port SSDs to build high performance Lustre scratch file system Design with SATA SSDs for Object store target while NVMe SSD for Metadata targets. All flash configuration at Intel Endeavor cluster on 350 SSDs deliver remarkable performance for parallel access needs LFS09 Intel SSD DC S3500 Series Based Lustre* System 1x Meta Data Server (MDS) 2 x Intel Xeon Processor E GB RAM + 2x SATA RAID0 MDT FDR In niband* (56Gb/s) 8x Storage Server (OSS) 2 x Intel Xeon Processor E GB RAM 1x Intel SSD 320 Series for OS 3x RAID controllers with 8 SAS/SATA targets each 6x OST per server, each target = 4x Intel SSD DC S3500 SSDs over-provisioned to 75% for 450GB usable Software Stack: RedHat* Enterprise Linux* Lustre* *Other names and brands may be claimed as the property of others. + Intel SSD DC S3500 Series SSD 4x Base 4x <than commercial HDD solution Continued interleaved scaling after 32x clients Iozone* 17

18 Emerging trends Vast majority of storage objects are tiny Kilobytes and smaller Trending to larger proportion of system capacity High performance => Billions of lowest possible latency > Resilience => Replication improves acceptable storage overhead Poorly supported by today s filesystems but vast majority of space is used by large storage objects Megabytes and larger High performance => efficient Terabytes per second Resilience => Erasure codes required for space efficiency & limit system cost