NVMe Takes It All, SCSI Has To Fall. Brave New Storage World. Lugano April Alexander Ruebensaal

Transcription

1 Lugano April 2018 NVMe Takes It All, SCSI Has To Fall freely adapted from ABBA Brave New Storage World Alexander Ruebensaal 1

2 Design, Implementation, Support & Operating of optimized IT Infrastructures ABC Systems AG - HA & HP - allowing for fail-safe Transportation of the Applications since

3 In the Year 2012 NVMe PCIe SSD 64GB/s 3

4 Six Years After Non-Volatile Memory Express NVMe SSD NVMe is an innovative Host Controller Interface to use SSD natively over PCIe. Mainly, it allows for acceleration due to parallelism resulting in reduced I/O overhead and latency. M.2 PCIe NGSFF Next Generation SFF [ M.3 ] U EDSFF Enterprise & DataCenter SFF [ Ruler ]

5 Why NGSFF and EDSFF? U.2 [ 2.5 ] No costly drive cages with failure points No cables to SSDs Eliminate the backplane with cooling holes Simplified thermal implementation NGSFF Next Generation SFF [ M.3 ] Less complicated chassis Reduced component cost per SSD Simple hot swap with high density capabiltites 5 EDSFF Enterprise & DataCenter SFF [ Ruler ]

6 INTEL OPTANE NVMe PCIe AIC U

7 NVMe SSD against SAS SSD 10x NVMe 1U 24x NVMe 2U 48x NVMe 2U 7

8 NVMe Storage Protocol is designed to take full Advantage of Flash NVMe supports 64K commands per queue (SAS 256, SATA 32) and up to 64K queues. These queues are designed such that I/O commands and responses to those commands operate on the same processor core and can take advantage of the parallel processing capabilities of multi-core processors. Each application or thread can have its own independent queue, so no I/O locking is required. NVMe also supports MSI-X and interrupt steering, which prevents bottlenecking at the CPU level and enables massive scalability as systems expand. NVMe has a streamlined and simple command set that uses less than half the number of CPU instructions to process an I/O request that SAS or SATA does, providing higher IOPS per CPU instruction cycle and lower I/O latency in the host software stack. NVMe also supports enterprise features such as reservations and client features such as power management, extending the improved efficiency beyond just I/O. Text & Graphics from 8

9 PCIe 3.0 Bus - 64GB/s A Single NVMe 9

10 NVMe uses CPU Lanes directly CPU Bus NVMe Flash or CPU - Bus FC-HBA Switche(s) FC-HBA RAID Ctrl SAS Enclosure Disk 8x SAS NVMe already saturate the SAS-Bus -> Effect is reduced to Electronics vs Mechanics! 10 1 NVMe uses 4 CPU Lanes Broadwell 40 Lanes Skylake 48 Lanes EPYC 128 Lanes

11 NVMe new Level of Performance 11

12 Are the NVMe too strong, are the CPU too weak Lanes: 1 CPU 48 1 NVMe 4 Intel Xeon Scalable Processors F (With OmniPath) Single on-package OmniPath interface Incremental to existing 48 PCIe Lanes Single cable connection to QSFP I/O module Same socket for Skylake & Skylake-F processors 12

13 AMD EPYC CPU 8~32 Zen Cores TDP 120W~180W 8 Memory Channels Up to 2TB per CPU Dedicated Security Engine Lanes of High Bandwidth I/O 128 How to use them? 1x NVMe 4 Lanes < 3 500MB/S / 3 938MB/s 89% 1x SATA SSD 1 Lane max. 32 directly supported by CPU < 540MB/s / 985MB/s 55% 1x 100GbE 16 Lanes < MB/s / MB/s 79% 2x 25GbE 8 Lanes < 6 250MB/s / 7 877MB/s 79% 2x 10GbE 8 Lanes [standard Interface for comparison] < 2 500MB/s / 7 877MB/s 32% 13

14 Storage Centric Solution Design Don t waste Lanes! Balanced Designs for Multi-Socket Server Solutions, regardless of CPU Vendor, is a huge Optimization Challenge! Component Parameter K IOPS Random GB/s Sequential C A P A C I T Y I O P S T H R O U G H P U T Reads Writes Reads Writes 112 TB IOPS GB/s Gbps 112 TB Mio. GB/s Gbps 112 TB IOPS GB/s Gbps NVDIMM 32GB NVMe SSD 4 Lane 11TB SATA SSD max. 32/CPU 1 Lane 8TB GbE 16 Lane full bandwidth GbE 4 Lane 8 Lane for 2x 25GbE 16 Lane 2x PCIe Slots 48 8 lane 2x Total R-world performance is, of course, application, workload and file system depend Assumption: 112 net Lanes availabe of 128 1x NVMe 4 Lanes < 3 500MB/S / 3 938MB/s 89% 1x SATA SSD 1 Lane max. 32 directly supported by CPU < 540MB/s / 985MB/s 55% 1x 100GbE 16 Lanes < MB/s / MB/s 79% 2x 25GbE 8 Lanes < 6 250MB/s / 7 877MB/s 79% 2x 10GbE 8 Lanes [standard Interface for comparison] < 2 500MB/s / 7 877MB/s 32% 14

15 Conceptual NVMe-Server Design Universel Department Store Servers might be over- or wrong-sized for SDS Purpose-built Servers for Efficiency cost, performance, power, space etc. effective allow for lean Architecture HPC, s in biggest DCs e.g. choose from > 100 NVMe Servers AMD and INTEL CPUs 15

16 From Storage-Server to Server-Storage < 10 million IOPS 1U 1U 36x NVMe NGSFF 2x Intel Xeon Scalable CPU 3x UPI, <10.4GT/s 24x DIMM up to 3TB 2x PCIe x16, 1x PCIe x8 2x 10GBase-T 32x NVMe U.2 2x Intel Xeon Scalable CPU 3x UPI, <10.4GT/s 24x DIMM up to 3TB 2x PCIe x16 2x 10GBase-T 32x NVMe EDSFF 2x Intel Xeon Scalable CPU 3x UPI, <10.4GT/s 24x DIMM up to 3TB 2x PCIe x16 2x 10GBase-T NVMe < 576TB < 352TB < 1 080TB 16

17 From Storage-Server to Server-Storage 2U 2U 48x NVMe U.2 Dual Port 2x Nodes: 2x Intel Xeon E5-2600v4 CPU 2x QPI, <9.6GT/s 16x DIMM up to 2TB 1x PCIe x16, 1x PCIe x8 2x 10GBase-T, SIOM 24x NVMe U.2 4x Nodes: 2x Intel Xeon Scalable CPU 3x UPI, <10.4GT/s 24x DIMM up to 3TB 2x PCIe x16 2x 10GBase-T 48x NVMe U.2 2x Intel Xeon E5-2600v4 CPU 2x QPI, <9.6GT/s 24x DIMM up to 3TB 2x PCIe x16, 1x PCIe x8, SIOM SIOM (e.g. 2x 25GbE, 2x 10GbE) NVMe < 528TB < 264TB < 528TB 17

18 From Storage-Server to Server-Storage 4x V100, P100, P40, M U All-NVMe & GPU Server on ABC booth NVMe-oF - NVMEe over Fabric JBOF Just a Bunch of Flash 20x NVMe U.2 7mm 2x Intel Xeon Scalable CPU 3x UP up 10.4GT/s 24x DIMM up to 3TB 2x PCIe x8 2x 25GBe Storage changes to - SERVER-CENTRIC - SOFTWARE-DEFINED RAID Protection HCI Hyper Converged Infrastructure < 80TB Holisitc Data Management 18

19 NVMe-oF - NVMe over Fabric The goal of NVMe over Fabrics is to provide distance connectivity to NVMe devices with no more than 10 micro-seconds (µs) of additional latency over a native NVMe device inside a server. Low Latency Networking Use Cases - A storage system comprised of many NVMe devices, using NVMe over Fabrics with either an RDMA or Fibre Channel interface, making a complete end-to-end NVMe storage solution. This system would provide extremely high performance while maintaining the very low latency available via NVMe. - Usage of NVMe over Fabrics to achieve the low latency while connected to a storage subsystem that uses more traditional protocols internally to handle I/O to each of the SSDs in that system. This would gain the benefits of the simplified host software stack and lower latency over the wire, while taking advantage of existing storage subsystem technology. FC NVMe-oF FC-HBA RDMA Storage Accelerations, leveraging hardware offloads for NVMe - ConnectX adapters support NVMe-oF <100Gbps - BlueField (SoC) Smart NIC 2x 25GbE combines ConnectX5 with ARM CPU IBM Spectrum Scale Acceleration Achieved with 24x NVMe: The only sub-millisecond overall response time at 0.69ms ORT! 2.5x more builds than other Spectrum Scale storage options. Higher IOPS and throughput than all other SPEC SFS2014_swbuild results. Soltuion available as Appliance or Software only. NVMesh Reference Architecture - near server-local performance in a linear scale-out remote standard NVMe solution. NVMesh RA provides the flexibility to create and manage a single, centralized pool of storage, create right-sized logical volumes, and even share storage resources with existing compute resources. Also supporting existing applications without changes. Text & Graphics from 19

20 All-NVMe JBOF 1U 32-bay JBOF Just a Bunch Of Flash NVMe SSD U2 hot-swap NVMe SSD EDSFF hot-swap Sync. Mirror 4x PCIe-Bus Extension PCIe 3.0 x16 64, 128TB Capacity Drives Cache 256TB 512 TB 1 024TB 10x more Performance with 3D XPoint OPTANE Technology than NAND via PCIe* NVMe* 64 GB/s > 36 Mio. IOPS Cluster Interconnect 2x EDR IB 100Gbps PCIe 3.0 x16 Client Network 2x EDR IB 100Gbps / 100/40GbE PCIe 3.0 x16 20

21 Supermicro JBOF 32x NVMe SSD U.2 4 Mini-SAS HD x16 ports, 2 PCI-E 3.0 x16 Slots, 2 IPMI ports The Supermicro JBOF supports up to 12 direct attached hosts, making this the go-to storage platform for any high-performance computing application. Alternatively, the dual PCI-E 3.0 x16 slots can support dual NVMe-oF add-on-cards to enable additional deployment scenarios. 21

22 RAID Protection Flash Technology More reliable, less Replacements Higher Throughput, faster Rebuilds With a 3.2x lower Annualized Failure Rate (AFR) - SASTA SSD - compared to HDD, IT departments will spend less time and expense replacing or upgrading storage devices. RAID Approach Hardware-Defined (RAID-Controller) Software-Defined (SDS) Hybrid: Intel VROC Virtual RAID on CPU 22

23 Intel Virtual RAID on CPU VROC RAID Function in VMD Volume Management Device 23

24 HCI Hyper Converged Infrastructure 8x 2U 4-Node Server. Dual CPU. 3UPI <10.4GT/s. 24x DIMM. 6x NVMe U.2. 1x PCIe Extension x16. 2x 10GbE 1U JBOF 32x NVMe. 4x Mini-SAS HD x16 ports. 2x PCI-E 3.0 x16 Slots

25 Data Management not only Data Storing Conceptual Optimization LTO LTFS NVMe 25

26 NVMe Takes It All, SCSI Has To Fall Storage-Technology Parameter Mainstream Directions of Movements up to TB /drive TB in 1U Server or JBOF TB in 4U Server or JBOD RAM Volatile Non-Volatile EDSFF [Ruler] NGSFF [M.3] NVMe U.2 2.5" 11 Flash M.2 2 AIC Add-in Card 8 SSD 2.5" SAS 8 SATA 11 Disk Tape SAS 2.5" NL SAS SATA LTO 15K IBM TS1150 [Jaguar] 10K " 2 3.5" Gotthardpost, 1873 Johann Rudolf Koller, Change Horses, add Horses or use the Gotthard Tunnel 26

27 NVMe Takes It All, SCSI Has To Fall The PCIe Bus is in the Server NVMe is the Protocol for Flash Flat screens TB NVMe PCIe 4.0 Hot Data Somewhat Hot Data NVMe SATA SSD NL SAS SATA SAS FC vs Displays Lukewarm Data LTO [ LTFS ] Cold Data 27

28 simplify and win with us and our Partners Headquarter Zurich Branch Office Berne Ruetistrasse 28 Giessereiweg 9 CH-8952 Schlieren CH-3007 Bern Spectrum Scale Spectrum Protect Tel Tel SAN@ABCsystems.ch Alexander Ruebensaal AR@ABCsystems.ch Other names and brands may be claimed as property of others. 28