Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitme

Transcription

1 SER2413BU NVMe: What Is It? An Interface? A Protocol? A New Drive Technology? An Industry Revolution? VMworld 2017 Content: Not for publication Sudhanshu (Suds) Jain & Adrian Marinescu #VMworld #SER2413BU

2 Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitment from VMware to deliver these features in any generally available product. Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery. Pricing and packaging for any new technologies or features discussed or presented have not been determined. #SER2413BU CONFIDENTIAL 2

3 Agenda 1 Why do we need faster storage? 2 Where is the holdup? 3 What is NVMe? 4 NVMe in vsphere 5 Future Evolution #SER2413BU CONFIDENTIAL 3

4 Deliver risk and opportunity for future Rise of the Third Platform Transforming the World of Applications Need to deliver business value in real-time Artificial Intelligence Predictive Analytics Real-time analytics Business Intelligence Transactional Processing Machine Learning Highly parallel processing on very large set of data Low latency for mission critical transactions Hyper-scale Datacenter $5.3 trillion by 2020: IDC #SER2413BU CONFIDENTIAL 4

5 EXPLOSIVE GROWTH OF DATA Key Infrastructure Challenges NEAR REALTIME VMworld 2017 Content: Not for HIGHLY DISTRIBUTED publication Towards Data-centric Future! #SER2413BU CONFIDENTIAL 5

6 Explosive Growth of Data Stressing your storage and network benchmarks Cloudera Zookeeper Source: Intel IDF 2016 #SER2413BU CONFIDENTIAL 6

7 Business Value of Data Diminish Fast Time = $$$ What we do when data is changing Act fast to gain business value Source: #SER2413BU CONFIDENTIAL 7

8 Highly Distributed Build, deploy and manage interconnected, collaborative workloads Big Data Cloud-Native App Multi-tier App VMworld 2017 Content: Not for Deploy multiple workloads with strong demand for inter-vm traffic publication Optimize data delivery to applications Adaptive storage tier critical for improved application performance Distributed Storage Virtual SAN #SER2413BU CONFIDENTIAL 8

9 Key Technology Trends Future is here now CPU densities continues to increase Hi-Density Flash and NVDIMM reducing cost of data access High speed interconnect can deliver any to any fast storage or distribution Technology Trends Application demands for data and key technology trends are disrupting the entire SAN & DAS ecosystem! Source: #SER2413BU CONFIDENTIAL 9

10 Composable Infrastructure Disaggregation of Infrastructure All infrastructure delivered as a service Hyper-scalable to keep up with Performance business Curevdemands Resources automatically tuned to application workloads Source: #SER2413BU CONFIDENTIAL 10

11 Agenda 1 Why do we need faster storage? 2 Where is the holdup? 3 What is NVMe? 4 NVMe in vsphere 5 Future Evolution #SER2413BU CONFIDENTIAL 11

12 Anatomy of Data Access Where Is the Holdup? For long period of time, innovation in silicon (processor and memory) industry far outpaced the disk I/O speedup * - Typical/Worst case consideration Source: #SER2413BU CONFIDENTIAL 12

13 Anatomy of Data Access Let s Look Closer on the Various Delays Various Delays? VMworld 2017 Content: Not for How does various Technology perform? publication Source: #SER2413BU CONFIDENTIAL 13

14 Agenda 1 Why do we need faster storage? 2 Where is the holdup? 3 What is NVMe? 4 NVMe in vsphere 5 Future Evolution #SER2413BU CONFIDENTIAL 14

15 Non Volatile Memory Express: A New Protocol NVMe is architected for NVM NVM Express defines a host controller interface and command set for use with non-volatile memory (NVM) Key differentiators Streamlined Commands; lighter weight I/O Stack Direct access to CPU; lower latency Massive parallelism; higher throughput Faster media Scalable performance Key Innovations VMworld Source: Content: Not for publication #SER2413BU CONFIDENTIAL 15

16 NVMe: A New Drive Technology NVMe need new controller and faster media to deliver on its promise! HDD SSD NVMe The head has to move to the right place and wait for the right block to come around Ah, no need to wait for the blocks to come to the right place picking blocks when you need them, one at a time Getting multiple blocks of information all at once! It is fast Source: #SER2413BU CONFIDENTIAL 16

17 NVM Express, Inc. Roadmap NVM Express NVMe-MI NVMe over Fabrics Future direction Q1 Q2 Q3 Q4 NVMe 1.2 Nov 14 Namespace Management Controller Memory Buffer Host Memory Buffer Live Firmware Update Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 NVMe May 16 NVMe-oF 1.0 May 16 Transport and protocol RDMA binding NVMe 1.3 May 17 Sanitize Streams Virtualization NVMe-MI 1.0 Nov 15 NVMe-MI 1.1* Out-of-band management SES Device discovery NVMe-MI In-band Health & temp monitoring Native Enclosure Mgmt Firmware Update NVMe (next)* IO Determinism Persistent Cntlr Mem Buffer Multipathing NVMe-oF (next)* Enhanced Discovery Authentication TCP Transport Released NVMe Planned NVMe Specification releases * Subject to change

18 NVMe: Revolutionizing the SAN Architecture NVMe Over Fabric Scalability Bandwidth NVMf VMworld 2017 NVMe IOPS Latency Unlocking the Reach of Data The value of data is based on how fast it can be accessed and processed Faster storage access enables cost reduction through consolidation Independently scale storage and compute infrastructures Content: Not for publication Data resilience Mitigate the server stranded storage (with DAS storage) problem Optimized data delivery to applications Efficiency of shared storage with no compromise #SER2413BU CONFIDENTIAL 18

19 Storage Industry is Pivoting Towards NVMe NVMe Adoption and Considerations PCIe or NVMe Flash 48% Drivers to NVMe Adoption Better scalability to meet our performance requirements over time Better cost/performance ($/IOPS) Higher throughput or bandwidth Increased storage density (TB/U) Currently using Planning to use within 12 months Lower latency Not using and no plans 0% 10% 20% 30% 40% 50% 60% #SER2413BU CONFIDENTIAL 19

20 Market Trends High Growth Rate for NVMe It is not a coincidence! Component Revenues: NVMe, HDD, SSD $11.8B $7.3B $16.2B NVMe Drive Revenues growing at 47% CAGR; expected to hit $16.2B by 2020 Overall revenue from NVMe and NVMe related product and services is many fold Key factors driving adaption Higher capacity drive and lower cost Availability of higher endurance flash media Migration from SLC to MLC NVMe HDD SSD Source: snia.org & IDC Overall revenue of flash media has already exceeded HDD; NVMe is the main growth engine #SER2413BU CONFIDENTIAL 20

21 Agenda 1 Why do we need faster storage? 2 Where is the holdup? 3 What is NVMe? 4 NVMe in vsphere 5 Future Evolution #SER2413BU CONFIDENTIAL 21

22 Software-Defined Infrastructure NVMe/Flash is Key Focus Area vsphere Flash Use-Cases: (KB ) Host swap cache Regular Datastore vsphere Flash Read Cache (aka Virtual Flash) VSphere ESXi Boot Disk VSphere ESXi Coredump device VSphere ESXi Logging device Virtual SAN (VSAN) Flash and NVMe Open Source Effort to build for faster innovation and the broad ecosystem Open ecosystem for certification and support #SER2413BU CONFIDENTIAL 22

23 vsphere Driver Architecture Evolution Random, IPMI SW FCoE FCoE Driver RDMA RoCE, iwarp Feature focus area VMK API Native SAS/FC Driver Native NIC Driver Native Graphics Driver Additional Partner Native Drivers VMK API VMK API SATA Library SATA Driver vmklinux SW FCoE FCoE Driver SAS/FC Driver Legacy support NIC Driver 23

24 vsphere Storage I/O Stack Built for Best of the Breed Fabric Solutions! T C P / I P S T A C K ESXi Monitor NIOC Guest Physical Hardware SIOC Vmxnet3 Driver vmxnet3 VDS I/O Driver NIC LSI Driver LSI PVSCSI vnvme Virtual SAN FCoE OS Stack SAS/SCSI PCSCI Driver iscsi Virtual Volumes FC/SAS Native NVMe Driver NVMe VMFS Para Virtualized RDMA NVMeOverFabric RDMA vrdma NFS Scheduler Memory Allocator S C S I S T A C K N V M e S T A C K CPU DIMM/NVDIMM NIC HBA HDD SSD PCI SSD PCIe NVMe #SER2413BU CONFIDENTIAL 24

25 NVM Express Evolution and vsphere Journey towards All NVMe Stack NVMe Technical Work Begins NVMe 1. 0 Released March 1, 2011 Queueing Interface NVM Command Set Admin Command Set End- to- end Protection (DIF /DIX ) Security Physical Region Pages ( PRPs ) NVMe 1. 1 Released October 11, 2012 General Scatter Gather Lists (SGLs) Multi-Path I/ O & Namespace Sharing Reservations Autonomous Power Transitions During Idle VMworld 2017 Content: Not for NVMe 1. 2 Released November 3, 2014 Implementation and Reporting Refinements Name Space Management Controller Memory Buffer Host Memory Buffer Ground Work for NVMe Management NVMe Over Fabric 1. 0 Released June 5, 2016 Defines extension to NVMe, for non PCI Primary focus on RDMA Compatible to FC-NVMe (INCITS 540) Host Memory Buffer Ground Work for NVMe Management publication vsphere 5.5 vsphere 6.0 vsphere 6.5 Future Direction Introduce first async NVMe driver 1.0e Launch IOVP cert program for NVMe Introduce first inbox NVMe driver Bring broader ecosystem support vnvme Optimized performance for NVMe driver Multiple name spaces, Queues NVMe Over Fabric End-to-end NVMe Stack 25

26 NVMe Driver Ecosystem Available as part of base ESXi image from vsphere 6.0 onwards Faster innovation with async release of VMware NVMe driver VMware led vsphere NVMe Open Source Driver project to encourage ecosystem to innovate Broad NVMe Ecosystem on VMware NVMe Driver Close to 300 third party NVMe devices certified on VMware NVMe driver Also available for download (async) VMware ESXi 5.5 nvme vmw NVMe Driver for PCI Express based Solid-State Drives #SER2413BU CONFIDENTIAL 26

27 NVMe Build Your Infrastructure with Confidence! Driver Core Stack vsphere 6.5 Boot (UEFI) Firmware Update End-to-end protection Deallocate/TRIM/Unmap 4K SMART, Planned hot-remove Reduced serialization Locality improvements vnvme Adaption layer Multiple completion worlds support in NVMe Tech Preview Performance enhancements Extended CLI/UI Name space management Async event error handling Enhance diagnostic logs Optimized stack - Highly parallel execution for single path local NVMe devices Reach target of 90%+ performance of device spec Future Direction NVMe Over Fabric Multiple fabric option SR-IOV Sanitize I/O Determinism Next Generation Storage Stack with ultra-high IOPS End-to-end NVMe Stack NVMe Multi-pathing Virtual Devices NVMe 1.0e spec Hot-plug support VM orchestration Performance improvements Async mode support unmap support Rev the specification Parallel 4K Support Scatter-gather support Interrupt coalescing 27

28 Key Challenges for NVMe Based Architecture NVMe Require Fresh Approach for Software Stack Designs Parallel Execution Elevator / C-Scan forced stack to be serialized File systems are also not optimized for concurrent NUMA architectures pose additional challenges for locality Interrupts distribution and queue selection Performance More processing power needed to saturate NVMe device IOPS increase by almost order of magnitude IO overhead reduced by 2-3x Queuing == increases latency IO scheduling #SER2413BU CONFIDENTIAL 28

29 Typical Storage Stack Application Application Application Block storage File System Driver IO Scheduler Implied serialization Processing resources 100 KIOPS SSD Source: #SER2413BU CONFIDENTIAL 29

30 NVMe Build Your Infrastructure with Confidence! Driver Core Stack vsphere 6.5 Boot (UEFI) Firmware Update End-to-end protection Deallocate/TRIM/Unmap 4K SMART, Planned hot-remove Reduced serialization Locality improvements vnvme Adaption layer Multiple completion worlds support in NVMe Tech Preview Performance enhancements Extended CLI Name space management Async event error handling Enhance diagnostic logs Optimized stack - Highly parallel execution for single path local NVMe devices Reach target of 90%+ performance of device spec Future Direction NVMe Over Fabric Multiple fabric option SR-IOV Next Generation Storage Stack with ultra-high IOPS End-to-end NVMe Stack NVMe Multi-pathing Virtual Devices NVMe 1.0e spec Hot-plug support VM orchestration Performance improvements Async mode support unmap support Rev the specification Parallel 4K Support Scatter-gather support Interrupt coalescing #SER2413BU CONFIDENTIAL 30

31 ESXi Storage Stack Enhancements for NVMe Performance Boost Throughput (IOPS) 1,200,000 1,000, , , , ,000 0 vsphere today vsphere prototype # Workers Hardware: Intel Xeon E5-2687W (10 cores + HT) 64 GB RAM NVM Express* 1M 4K Reads Software: vsphere* 6.0U2 vs. Future prototype 1 VM, 8 VCPU, Windows* 2012, 4 VMDK eager-zeroed IOMeter: 4K seq reads, 64 OIOs per worker, even distribution of workers to VMDK Source: SDC Conference SD_Performance_and_Scale_Out_Software.pdf #SER2413BU CONFIDENTIAL 31

32 vsphere Virtualized I/O Stack vsphere Offers Rich Set of I/O Virtualization T C P / I P S T A C K Guest I/O Device Driver D i Monitor r vmxnet3 e c t P a VDS s s NIOC - t h SIOC r ESXi o u g Physical Hardware h I/O Driver P C I NIC NIC Driver LSI PVSCSI vnvme Virtual SAN FCoE OS Stack SAS/SCSI PCSCI Driver iscsi Virtual Volumes FC/SAS Native NVMe Driver NVMe Device Emulation VMFS Para Virtualized RDMA NVMeOverFabric RDMA vrdma NFS Scheduler Para- Virtualized Device Memory Allocator S C S I S T A C K N V M e S T A C K NSX Datapath Interconnect like PCIe NIC GPU/GPGPU NVMe SSD PCI SSD FPGA IB and OPA HDD Hardware Accelerations e.g QAT vsan Datapath #SER2413BU CONFIDENTIAL 32

33 Virtual NVMe High Performance Guest Block I/O VMworld 2017 New in vsphere 6.5 Feature: NVMe 1.0e Device Emulation Hot add/remove support Integrated with VM orchestration Multi-Q support 16Queues, total 4K depth Benefits: Improved application performance, better IOPS and latency numbers Leverage native NVMe stack in various Guest OS including Linux and Windows Reduced in-guest I/O stack overhead Content: Not for publication #SER2413BU CONFIDENTIAL 33

34 Virtual NVMe Rich Set of Features & Potential! Implemented #SER2413BU CONFIDENTIAL 34

35 NVMe Build Your Infrastructure with Confidence! Driver Core Stack vsphere 6.5 Boot (UEFI) Firmware Update End-to-end protection Deallocate/TRIM/Unmap 4K SMART, Planned hot-remove Reduced serialization Locality improvements vnvme Adaption layer Multiple completion worlds support in NVMe Tech Preview Performance enhancements Extended CLI/UI Name space management Async event error handling Enhance diagnostic logs Optimized stack - Highly parallel execution for single path local NVMe devices Reach target of 90%+ performance of device spec Future Direction NVMe Over Fabric Multiple fabric option SR-IOV Sanitize I/O Determinism Next Generation Storage Stack with ultra-high IOPS End-to-end NVMe Stack NVMe Multi-pathing Virtual Devices NVMe 1.0e spec Hot-plug support VM orchestration Performance improvements Async mode support unmap support Rev the specification Parallel 4K Support Scatter-gather support Interrupt coalescing #SER2413BU CONFIDENTIAL 35

36 NVMe Summary Driver Core Stack vsphere 6.5 Boot (UEFI) Firmware Update End-to-end protection Deallocate/TRIM/Unmap 4K SMART, Planned hot-remove Reduced serialization Locality improvements vnvme Adaption layer Multiple completion worlds support in NVMe Tech Preview Performance enhancements Extended CLI/UI Name space management Async event error handling Enhance diagnostic logs Optimized stack - Highly parallel execution for single path local NVMe devices Reach target of 90%+ performance of device spec Future Direction NVMe Over Fabric Multiple fabric option SR-IOV Sanitize I/O Determinism Next Generation Storage Stack with ultra-high IOPS End-to-end NVMe Stack NVMe Multi-pathing Virtual Devices NVMe 1.0e spec Hot-plug support VM orchestration Performance improvements Async mode support unmap support Rev the specification Parallel 4K Support Scatter-gather support Interrupt coalescing 36

37 Agenda 1 Why do we need faster storage? 2 Where is the holdup? 3 What is NVMe? 4 NVMe in vsphere 5 Future Evolution #SER2413BU CONFIDENTIAL 37

38 Storage Evolutionary Path & Key Drivers Building the Stack for Future System Block storage SCSI Storage Architecture SCSI Driver Applications File System NVMe Storage Architecture NVMe/NVMeoF Driver NVDIMM Storage Byte addressable Flash evolution in cost, endurance and capacity NVMe as a replacement of SCSI Composable Infrastructure NVMe Over Fabric as primary way to access storage SAN Evolution towards Server Attached Storage #SER2413BU CONFIDENTIAL 38

39 Next-Generation Hardware Evolution NVMe as Caching Tier & Performance Tier SSD NVMe NVDIMM Caching Tier Persistence Tier NVMe NVMe NVDIMM Today s Hardware Evolution of Storage Tier #SER2413BU CONFIDENTIAL

40 NVMe Over Fabric: Demo Next Generation SAN is Here! POC 40

41 Key Takeaways NVMe is the choice of technology for your storage needs vsphere is ready to power tour NVMe infrastructure vsphere and its NVMe ecosystem is constantly innovating Foundation to deploy Software Defined Infrastructure #SER2413BU CONFIDENTIAL 41

42

43