NVM Express (NVMe) & Solid State Storage: What s Next? David L. Black, Ph.D. Distinguished Engineer, Office of the CTO

Transcription

1 NVM Express (NVMe) & Solid State Storage: What s Next? David L. Black, Ph.D. Distinguished Engineer, Office of the CTO

2 Roadmap Information Disclaimer Dell EMC makes no representation and undertakes no obligations with regard to product planning information, anticipated product characteristics, performance specifications, or anticipated release dates (collectively, Roadmap Information ). Roadmap Information is provided by Dell EMC as an accommodation to the recipient solely for purposes of discussion and without intending to be bound thereby. Roadmap information is Dell EMC Restricted Confidential and is provided under the terms, conditions and restrictions defined in the Dell EMC Non- Disclosure Agreement in place with your organization. 2

3 Business drivers - Need for speedy outcomes Enterprise value Knowledge, insight, decisions, beneficial outcomes Faster decision speeds - Low latency data access/analysis Data set centricity - From few/very large and growing to very many/relatively small Client/commercial value Enhanced platform performance demands Higher capacity local data sets Balance between performance and power 3

4 Overview 1. NVM Express (NVMe) interface Fixed block read/write I/O interface to SSDs (drives) and external storage systems Applies to both current flash (NAND) and emerging (new) low-latency NV Media Lower latency, higher concurrency than SAS or SATA: Essential for SSDs that use low-latency NV Media 2. New low-latency non-volatile media (New NV Media) New class of persistent storage media 3D XPoint TM is best known example, but there are others Faster than flash, better endurance (# of writes) Slower and larger than DRAM, at least initially. Presentation uses NV Media term for clarity instead of NVM (Non-Volatile Memory) NOTE: This is a technology session Product specifics are covered in other sessions 4

5 Terminology NVMe = NVM Express: I/O interface to PCIe SSDs NVMe over Fabrics: NVMe for networked fabrics (e.g., InfiniBand, RDMA over Ethernet) NV Media = Non-Volatile Media: Persistently stores data, e.g., Flash (NAND), 3D XPoint New NV Media = New faster-than-flash persistent media, e.g., 3D XPoint, NRAM (carbon nanotube) NV Media packaging Drives (SSDs = Solid State Drives): I/O interface (read/write blocks), e.g., NVMe Memory (NVDIMMs = Non-Volatile DIMMs): Memory interface (load/store words, cache lines), e.g., DDR4 DIMM = Dual In-line Memory Module: Industry-standard DRAM memory packaging, DDR (e.g., DDR4) interface Additional Memory Terms: Non Volatile Memory (NVM): General term for any NV Media Storage Class Memory (SCM) & Persistent Memory (PM): New faster-than-flash NV Media Memory word often implies DIMM packaging for load/store access 5

6 NVM Express and I/O Latency Latency (us) 10, SSD NAND technology offers ~100X reduction in latency versus HDD NVMe eliminates 20 µs of latency today 3D XPoint technology reduces latency: ~10x reduction in latency versus NAND SSD 0 HDD +SAS/SATA SSD NAND +SAS/SATA SSD NAND +NVMe 3D XPoint TM SSD + NVMe Drive Latency Controller Latency (ie. SAS HBA) Software Latency 6 Slide credit: Intel and NVM Express Source: Storage Technologies Group, Intel. Comparisons between memory technologies based on in-market product specifications and internal Intel specifications.

7 Server Architecture Classic CPU DDR DRAM Two types of data storage: DRAM memory fast, volatile, byte level access HDD disk slow, nonvolatile, block access (spinning rust ) Data moves from HDD disk to DRAM memory Fetched by CPU CPU writes results to DRAM memory Drive Controller SAS or SATA HDD Data stored back to HDD disk for future use. 7 Slide credit: Amnon Izhar, Dell EMC

8 Server Architecture Evolved CPU Drive Controller DDR DDR NVMe over PCIe SAS or SATA SAS or SATA DRAM DRAM* NVDIMM NVMe SSD SSD HDD NV Media HDD NV Media HDD NV Media Low latency High BW High Endurance Low $/GB High Capacity NV Media (initially NAND flash) reduces response time First adoption through SATA (and SAS) Same software HDD interface Followed by PCIe/NVMe SSDs: Exploit multi-core CPUs Avoid separate Drive Controller NVDIMM DRAM for speed Data destaged to slower NV Media DRAM: battery (or capacitor) backed *New low-latency NV Media may not need DRAM in NVDIMMs 8 Slide credit: Amnon Izhar, Dell EMC

9 NVM Express (NVMe)

10 What is NVM Express (NVMe)? Specification for SSD access via PCI Express (PCIe) Extended to Fabrics (e.g., InfiniBand, RDMA/Ethernet) Designed to scale to New NV Media, agnostic to type of NV Media Design target: high parallelism and low latency SSD access Does not rely on SCSI (SAS/FC) or ATA (SATA), new host driver & I/O stack Common interface for Enterprise & Client drives/systems (reuse & leverage engineering investments) New modern command set 64-byte commands (vs. typical 16 bytes for SCSI) Administrative vs. I/O command separation Small set of commands: Small fast host and storage implementations Standard development by NVMe working group. NVMe working group incorporated as NVM Express in 2014 >100 companies across the industry 10

11 Why NVM Express (NVMe)? Removing data movement bottleneck 4K RANDOM WORKLOADS, QD= ,000 SEQUENTIAL WORKLOADS, QD=128 3,000 PCIe/NVMe SAS 12 Gb/s dual port SATA 6 Gb/s 400,000 2,500 IOPS 300, , ,000 MB/s 2,000 1,500 1, % Read 70% Read 0% Read 0 100% Read 0% Read NVM Express delivers vs. leadership SAS/SATA products Random workloads > 2X performance of SAS 12 Gb/s 4-6X performance of SATA 6 Gb/s For sequential workloads, realize close to 3 Gb/s reads Sequential workloads > 2X performance of SAS 12 Gb/s 4-6X performance of SATA 6 Gb/s Source: Intel, Flash Memory Summit Slide credit: Intel and NVM Express

12 Why NVMe/PCIe SSDs? Scalable performance ~1GB/s per lane (PCIe gen3). More lanes increased performance Low latency direct CPU connection (via PCIe) Reduced software overhead New operating systems drivers and I/O stacks Parallelism matched to multi-core CPUs Compared to SAS and SATA: No Drive Controller Lower $ cost Lower latency 12 Slide credit: Amnon Izhar, Dell EMC

13 NVMe: Technical Basics Supports deep memory-based queues (up to 64K commands per queue, up to 64K queues) Typical product limits are much smaller Streamlined and simple command set (13 required commands) Command completion interface optimized for success (common case) Optional features to address target segment requirements Data Center: Reservations, etc. Client: Power features, etc. 13 Slide credit: Amnon Izhar, Dell EMC

14 The Life of an NVMe Command 2 1 PCIe PCIe PCIe PCIe PCIe Submission Queue Tail Doorbell PCIe Completion Queue Head Doorbell 8 1)Queue Command(s) 2)Ring Doorbell (New Tail) 3)Fetch Command(s) 4)Process Command 5)Queue Completion(s) 6)Generate Interrupt 7)Process Completion 8)Ring Doorbell (New Head) 14 Slide credit: Intel and NVM Express

15 Important NVMe Functionality Namespaces: Analogous to SCSI logical units (host can access many) Flat 32-bit namespace numbering (not SCSI hierarchy) Commands to create/delete, attach/detach namespaces End-to-end data protection Same format as SCSI Protection Information (aka DIF, Data Integrity Field) Support for multiple ports on drives and subsystems Most current NVMe drives (SSDs) are single port TCG data-at-rest encryption (self-encrypting drives) Opalite: TCG profile defined for NVMe drives Pyrite: Opalite access control without data encryption Other TCG profiles can be used (e.g., full Opal) 15

16 NVMe SSD Form Factors PCIe SSD first incarnation was primarily as PCIe add in card NVMe now offered in multiple form factors PCIe AIC U.2 SSD SFF-8639 M.2 BGA 16 Slide credit: Amnon Izhar, Dell EMC

17 NVMe Adoption Industry NVMe seen as disruptive technology to SAS and SATA SSDs PCIe NAND Flash SSDs, primarily inside servers. NVMe displacing SAS and SATA in server/pc markets Lower latency NV Media (e.g., 3D XPoint) SSDs - NVMe-only NVMe ecosystem and recognition growing quickly Growing number of servers offer NVMe slots - different server configs and form factors Startups already shipping NVMe and NVMeoF solutions Storage class NVMe SSDs emerging will enable HA storage arrays Dual port support, enterprise level features, e.g., Data Integrity Field and TCG stored data encryption NVMe over fabrics emerging as a solution to limited scale of PCIe as a fabric Expanding ecosystem, e.g., NVMe over Fabrics adapters, Analyzers, PCIe host adapters 17 Slide credit: Amnon Izhar, Dell EMC

18 NVMe: Will Take <108 Years to Happen 2016 Public Domain Image Chicago Cubs 1908 World Series Champions 18 Image Credit: Flickr user apardavila

19 NVMe over Fabrics

20 NVMe Over Fabrics Use NVMe end-to-end to get the simplicity, efficiency and low latency benefits NVMe over Fabrics is a thin encapsulation of the base NVMe protocol across a fabric No translation to another protocol (e.g. SCSI) First target is RDMA enabling use with Ethernet, InfiniBand and other RDMA fabrics NVMe Host Software Host Side Transport Abstraction Fibre Channel InfiniBand * RoCE Controller Side Transport Abstraction NVMe SSD iwarp Next Gen Fabrics 20 Slide credit: Intel and NVM Express

21 NVMe in Fabric Environments A primary use case for NVMe PCIe SSDs is in an all flash appliance Hundreds or more SSDs may be attached too many for PCIe based attach scale-out Concern: Today remote SSD scaleout over a fabric attach uses SCSI based protocols: Requires protocol translation Desire best performance and latency from NVMe SSD investment over fabrics like Ethernet with RDMA (iwarp,roce) and InfiniBand 21 March 15 18, 2015 #OFADevWorkshop - slide credit: Dave Minturn (Intel)

22 End-to-End NVMe Over Fabrics Front-end Fabric Back-end Fabric Extend efficiency of NVMe over Front and Back-end Fabrics Enables efficient NVMe end-to-end model (Host<->NVMe PCIe SSD) 22 March 15 18, 2015 #OFADevWorkshop - slide credit: Dave Minturn (Intel)

23 NVMe Usage Scenarios: Servers and Storage NVMe over PCIe (now) 1. Drives in servers 2. Server access to external storage 3. Drives in storage systems NVMe over Fabrics (soon) 4. Server access to external storage 5. Internal to storage systems 23 Slide credit: Erin Bournival, Dell EMC

24 NVMe for Clients (e.g., laptops, tablets) PCIe NVMe devices enable Client platforms to address local data storage in a balanced approach between performance, storage capacity, physical footprint and energy consumption. Client NVMe Considerations: High performance I/O Best I/O performance where maximum capacity is not a primary consideration. Storage acceleration Small PCIe NVMe SSD as cache for larger SATA HDD. Mobility thin and light balance of portability and performance via PCIe NVMe SSDs - M.2 or BGA packaging. NVMe Usage Scenarios (present and future examples): Fast system boot, rapid hibernate & recovery Highly mobile commercial laptops, micro desktops Mobile gaming 24 Slide credit: Lee Zaretsky, Dell EMC

25 NVMe Standards (Specifications)

26 NVMe Specifications Current Specifications (Mid-April) Base NVMe spec (NVMe/PCIe) v1.2.1: June 5, 2016 NVMe over Fabrics v1.0: June 5, 2016 (Fabrics additions to NVMe 1.2.1) Specifications available from: NVMe Technical WG has updated base NVMe spec to v1.3 Complete and in member review Expected public release: late April 2017 Highlights: Sanitize (secure erase), Virtualization (PCIe SR-IOV), Para-virtualization (for hypervisors) : NVMe v1.4 and NVMe-over-Fabrics v1.1 NVMe 1.4 highlight: I/O Determinism (next slide) NVMe over Fabrics 1.1: Significant attention to discovery and security 26

27 The I/O Determinism Problem Customers are demanding max (tail) latency controls (e.g., 99.99%) with NAND Flash today Low latency New NV Media makes this much more demanding To meet current and future needs, need host awareness of units of parallelism Avoid I/O collisions inside SSD Also need to minimize and control SSD background operations Can t get 99.99% tail latency control without this Yahtzee Effect: Statistical Clumping When rolling 6 dice with 6 faces, on average only 4 of the 6 values will come up (Odds of all 6 values appearing: ~1.5%) 27 Slide credit: Intel and NVM Express

28 NVMe On One Slide Background Specification: SSD access via PCIe Command Set: New NV Memory Goals: High Parallelism & Low Latency Direct: Host End Device Fabric: Fabric Extensions Industry Standards SFF Product Options Memory Types 3D NAND Drivers Drive Classes Enterprise Client Product Ecosystem Drive Form Factors AIC M.2 JBODs All Flash Arrays Servers U.2 BGA 28 Slide credit: Gary Kotzur, Dell EMC

29 New NV Media And you thought flash was disruptive

30 New demands Proc Mem Low Capacity Low Latency Volatile memory Existing platform limitations Choke points, technology limits, DRAM capacity, Memory BW, IO latency, Enhanced value Higher Capacity: Larger data low latency In-memory Persistence: Fast recovery from power loss and resets Mitigated disruptions Lower Latency: Data and processing closer to each other Faster outcomes Persistent Mem (NV Media) Storage NVMe NV Media SSD Storage NVMe Flash SSD Higher Capacity Low Latency Non-Volatile Memory High Capacity Lower Latency Durable Storage Higher Capacity Higher Latency Durable Storage 30

31 New NV Media enables Large memory-access capacities Addresses latency, capacity, $/GB gap between DRAM and flash Addition of persistence to load/store memory Fast system recovery times Storage and network system/device options Low latency block IO devices NV Media NVMe (e.g. Intel 3D XPoint SSD) New and updated platforms Scalable persistent memory pools 31

32 New NV Media impact Addresses memory-storage latency-capacity gap New NV Media Domain High Speed Storage SSD $ (Flash) < $ << $ Low Speed Storage HDD Relative Capacity (not to scale) $$$$ Processor SRAM $$$ $$$ DRAM Much DRAM Faster NV Media Capacity/latency gap fill $$ Faster NV Media SSD (NAND Flash) NV Media HDD (Lower RPM) Access Latency 1ns 10ns 100ns 1us 10us 100us 1ms 10ms Memory access semantics IO block access semantics 32

33 New NV Media and related technology stack Value Proposition Enabling Technologies It s about all three! Platforms/Applications NV Media Programming Model Connection Technologies New NV Media Technologies 33

34 New NV Media and related technology stack Value Proposition Platforms/Applications NV Media Programming Model Enabling Technologies Connection Technologies New NV Media Technologies 34

35 NV Media technology candidates year span 3D XPoint? STT-MRAM Included as reference Stack of Carbon Nanotubes Latency > DRAM NRAM (CNT) Latency ~ DRAM Latency > DRAM Mechanism: Opening and closing of paths through CNT stack with bias voltage Latency ~ DRAM Latency > DRAM 35

36 Faster NV Media candidate summary Item DRAM (ref) 3D XPoint ReRAM Capacity (density) Baseline Higher Higher Endurance (write cycles) Unlimited Retention (data lifetime) None 5 years Years Latency 10 s ns 100 s ns 1 s us w/controller 100 s ns 1 s us w/controller DDRx compatibility Yes Slower, requires DDR changes Slower, requires DDR changes Comments 2017 NVMe SSD; 2018 DIMM Large vendor investments Still developing st appearance? Large vendor investments Comparable to 3D XPoint 36

37 Much Faster NV Media candidate summary Item DRAM (ref) STT-MRAM NRAM Capacity (density) Baseline Significantly lower Same or potentially higher Endurance (write cycles) Unlimited >10 10 (potentially unlimited) >10 12 (potentially unlimited) Retention (data lifetime) None Months Effectively unlimited Latency 10 s ns ~ Same ~ Same (capacity impact) DDRx Compatibility Yes Yes Yes (target) Comments Available now in low capacities Lower cap, higher $/GB vs DRAM Same or higher capacity vs DRAM Lower $/GB vs DRAM Industry-standard server motherboard/chipset support for Much Faster NV Media DIMMs? STAY TUNED 37

38 New NV Media and related technology stack Value Proposition Platforms/Applications NV Media Programming Model Enabling Technologies Connection Technologies New NV Media Technologies 38

39 Connection technologies Possible configurations Processor (or offload device) expanded local memory DRAM + NV Media Lower latency NV Media SSDs via NVMe(oF) Rack level fabric NV Media capacity and BW scale independent of processor memory Proc Mem Proc P NV Media NV Media Persistent Mem (NV Media DIMM) Storage NVMe NV Media SSD P NV Media NV Media Storage NVMe Flash SSD P NV Media NV Media Scalable Memory Fabric 39

40 NV Media implementations DIMM form factor NVDIMM-x defined devices (from JEDEC) DIMM form factor, DDRx interface NVDIMM-N DRAM backed up by flash Available now (lower cap) Available SW support Good starting point NVDIMM-F Flash SSD on DIMM Available now Available SW support Particular vendors NVDIMM-P Multi-function - Upcoming Support via in-progress JEDEC DDR4/DDR5 extensions NVDIMM-N NVDIMM-F NVDIMM-P Byte access Block access Multi-function 40

41 Connection technologies Possible configurations Processor (or offload device) expanded local memory DRAM + NV Media Lower latency NV Media SSDs via NVMe(oF) Rack level fabric NV Media capacity and BW scale independent of processor memory Proc Mem Proc P NV Media NV Media Persistent Mem (NV Media DIMM) Storage NVMe NV Media SSD P NV Media NV Media Storage NVMe Flash SSD P NV Media NV Media Scalable Memory Fabric 41

42 Connection technologies Consortia activity GenZ Memory semantic-based scalable fabric Plus storage and network gateway connectivity Announced Fall 2016 Large membership Dell-EMC high profile (president) CCIX (C6) Offload engine connection fabric with support of memory and coherency Overlapping membership with GenZ OpenCAPI Open version of IBM CAPI (Coherent Accelerator Processor Interface) Overlapping membership with GenZ Activities continuing Commentary Considered complementary to a degree but still needs harmonization Industry-standard server motherboard/chipset support? STAY TUNED 42

43 New NV Media and related technology stack Value Proposition Platforms/Applications NV Media Programming Model Enabling Technologies Connection Technologies New NV Media Technologies 43

44 NV Media programming model SNIA Open Programming Model support Version 1.2 expected early 2017 Spec v1.1 developed, approved by SNIA voting members and published Legacy block mode Legacy file mode PM-aware block mode PM-aware file mode 44 Slide credit: SNIA

45 NV Media programming model SNIA model/ecosystem Block, file, direct access Libraries, tool chains Management Modified SW needs NV Media (PM)-aware file systems Base OS E.g. NVDIMM Firmware/BIOS E.g. NVDIMM Remote access New security concerns for NVDIMMs Data persists when NVDIMM removed from server 45 Slide credit: SNIA and Intel

46 Summary NV Media and related technologies 1) Expanded memory capacities with persistence (vs. DRAM) 2) Scalable memory-accessible pools 3) Enhanced storage capabilities 4) Supporting ecosystem 46

47 Credits I only steal (reuse ideas and material) from the best Jim Espy, Distinguished Engineer NV Media (responsible for most of the NV Media slides) Stuart Berke, Distinguished Engineer NV Media Erin Bournival Storage Amnon Izhar SSDs Gary Kotzur, Distinguished Engineer Server Randy Shain Programming Model Patrick Weiler NV Media Lee Zaretsky, Distinguished Engineer Client Amber Huffman (Intel), Fellow NVMe Dave Minturn (Intel), Principal Engineer NVMe over Fabrics 47

48