Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitme

Transcription

1 FUT3040BU Storage at Memory Speed: Finally, Nonvolatile Memory Is Here Rajesh Venkatasubramanian, VMware, Inc Richard A Brunner, VMware, Inc #VMworld #FUT3040BU

2 Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitment from VMware to deliver these features in any generally available product. Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery. Pricing and packaging for any new technologies or features discussed or presented have not been determined. #FUT3040BU CONFIDENTIAL 2

3 Agenda 1 Persistent Memory: Storage at Memory Speed 2 VMware and the Amazing Future of Virtual Non-Volatile Memory 3 Summary #FUT3040BU CONFIDENTIAL 3

4 Persistent Memory: Storage at Memory Speed Rich Brunner VMworld 2017 Content: Not for publication

5 Business Agility 101: The Speed of Analysis Today: Find the needle in the haystack Tomorrow: Find it faster in bigger haystacks at lower cost #FUT3040BU CONFIDENTIAL 5

6 Speed of Analysis Translates to Cost Savings and Strategic Advantage Speed of Analysis is dependent on Storage and Database Access Latency. A key component is local storage latency. Current solutions can not compete with the latency of DRAM. #FUT3040BU CONFIDENTIAL 6

7 Problem: Local Storage Latency Speed of Analysis is dependent on Storage and Database Access Latency. A key component is local storage latency. Current solutions can not compete with the latency of DRAM. What if you could move storage closer to where the analysis is being done? So close that the data can be accessed by a processor as if it were DRAM-like. VMworld 2017 With reduced latency and byte-granular access. Content: Not for publication #FUT3040BU CONFIDENTIAL 7

8 Problem: Local Storage Latency What if you could move storage closer to where the analysis is being done? So close that the data can be accessed by a processor as if it were DRAM-like. With reduced latency and byte-granular access. You can with Byte-Addressable Persistent Memory (PMEM). PMEM is a fundamental change in Storage & Database architecture. This year PMEM solutions will offer high-capacity and performance. VMworld 2017 Future VMware vsphere will bring the agility benefits of PMEM to the data center. Content: Not for publication #FUT3040BU CONFIDENTIAL 8

9 What is PMEM (Persistent Memory)? Block-Mode Access NVMe NVM Express: NVM block-mode attached via PCIe. NVDIMM-F NVM block-mode implemented in DIMM form factor. Non-Volatile Memory NVM Random Access (byte/word addressable) NVRAM NVM with Random-Access Memory. PMEM Persistent Memory: NVRAM that is direct-mapped & addressable by CPU Load/Store Instructions from non-privileged Apps. NVDIMM-N,P PMEM implemented in DIMM form factor. Byte-Addressable Persistent Memory (PMEM) is Storage with these characteristics: DRAM latency and bandwidth DRAM granularity and access DRAM model for App software DRAM model for OS Memory Mapping A few hundred nanoseconds or less on average byte-level access in the normal system memory map regular, non-privileged, load/store CPU instructions paged/mapped by OS just like DRAM A future version of VMware vsphere intends to enable PMEM by supporting a virtualized NVDIMM device. #FUT3040BU CONFIDENTIAL 9

10 How Does PMEM Change & Benefit the Data Access Model? Read-Modify-Write (RMW) Byte Example (Greatly Simplified*) 64-Byte Cacheline 4-KByte block External Device Block Storage Model Processor Package DRAM (RMW duration can be a few hundred microseconds SSD Storage Controller & Media Byte-Addressable PMEM Model Processor Package PMEM (RMW duration can be a few hundred nanoseconds) Updating storage at finer-granularity can reduce storage overhead. PMEM benefits persistent workloads that need reduced latency & more flexible data access: Traditional Database: In-Memory Database: Enterprise Storage: High-Performance Computing: Log Acceleration by caching and combining data writes Journaling, Logging, Reduced Recovery time Fast-Caching Layer In-memory check-pointing * = Storage optimization can remove some of the penalty, but the basic flow is still the same. Note CPU Cache skipped due to complexity of example. #FUT3040BU CONFIDENTIAL 10

11 How VMware vsphere Can Work with PMEM Solutions PMEM Native OS Application OS Drivers Disk DRAM Future Version of VMware vsphere PMEM Application OS Drivers Disk DRAM Legacy OS & Application Usage Native: Can use PMEM as block storage device with special driver. Virtualized: Can use PMEM as block storage device with special driver in VM. With a future version of vsphere, no special driver is required in the VM. Guest Storage can be mapped to PMEM outside of VM. Server Future vsphere (Hypervisor) PMEM Disk Server DRAM #FUT3040BU CONFIDENTIAL 11

12 How VMware vsphere Can Work with PMEM Solutions Application PMEM.IO DAX/SCM Windows 2016 Drivers PMEM Disk Future Version of VMware vsphere DRAM Future vsphere (Hypervisor) Application PMEM.IO DAX PMEM Linux Kernel Drivers Disk DRAM New OS & Application Usage Native & Virtualized Can use a direct load/store model with little OS overhead All the benefits of VMware vsphere Virtualization can be available: Multiple workloads using PMEM Live VM Migration across servers Check-pointing Boost for Legacy VMs/Workloads And More PMEM Disk DRAM Server #FUT3040BU CONFIDENTIAL 12

13 Key Characteristics of PMEM Solutions (1) 1. The persistence technology, latency, and reliability. Note: in some cases access is to the DRAM image of the data, thus latency is that of DRAM. DRAM caches can also be used with certain technologies. Note: in some cases writes only happen to the media on shutdown, thus significantly reducing P/E cycles. 2. The capacity of the solution. Per DIMM can span from 8 GiB to 100s of GiB 3. The software access model. Can it support both legacy block-mode and new load/store access? 4. The right metric for assessing Total Cost of Ownership for your workloads. A traditional Cost / GB metric may not take into account the significantly reduced latency and bandwidth of PMEM. Technology Program/Erase Cycles (P/E) HDD NAND Flash SLC 10 5 NAND Flash MLC D-XPoint [1] 10 8 PCRAM [2] 10 8 ReRAM [2] HDD DRAM Program/Erase cycle: when data is written to a cell, erased, and re-written The limits to the number of P/E cycles supported before the cell fails can measure Durability 1. VMware's estimate on Intel 3D XPoint endurance. Estimate is based on Intel's claim that 3D XPoint is capable of up to 1000 times greater endurance than NAND. 2. (Bernstein Research), #FUT3040BU CONFIDENTIAL 13

14 PMEM Technology (1) NVDIMM-N Controller DRAM DRAM Host Memory-Mapped DDR4 DRAM, backup to NAND Flash on Shutdown & Power Fail DRAM Backed to NVMe DRAM DRAM NAND NAND External NVMe Memory-Mapped DDR4 DRAM, backup to NVMe devices on Shutdown & Power Fail Capacity: Latency: Endurance: Access: Cost per GiB: Energy Source: Availability: VMware: Capacity: Latency: Endurance: Access: Cost per GiB: Energy Source: Availability: VMware: 8-32 GiB per DIMM DRAM (10 s of ns) NAND (10^3 to 10^5 PE cyc) Load/Store ~ $80/GiB [1] Battery or equivalent Now Intend to support HPE NVDIMM-N RDIMM (128 GiB per RDIMM) DRAM (10 s of ns) NVMe Media Load/Store TBD Less than $80/GIB?? Battery or equivalent This year Intend to support HPE Scalable PMEM Examples HPE, Micron, Netlist, Viking, etc HPE Scalable PMEM 1.Based on 8-GiB RDIMM - and 8-GiB NVDIMM-N - #FUT3040BU CONFIDENTIAL 14

15 PMEM Technology (2) Intel DIMMs Based on 3D XPoint Technology Controller 3D XPoint Memory Memory-Mapped DDR4-extended Storage, No backup needed NVDIMM-P / Netlist HybriDIMM DRAM Cache Event Xfer Controller NAND NAND is logically mapped to memory, DRAM acts as Cache when needed, backup to NAND Est. Capacity: Est. Latency: Est. Endurance: Est. Access: Est. Cost per GiB: Est. Energy Source: VMware: Capacity: Latency: Endurance: Access: Cost per GiB: Energy Source: 256 GiB 512 GiB [1] 100 s of ns [2] 1000 x NAND [4] Load/Store (Also Block) < DRAM Light-Weight Energy Source Intend to support Intel 3D Xpoint [3] (Today 128 GiB RDIMM is ~$13/GiB) 100 s of GiB 100 s of ns NAND Load/Store, and Block-mode TBD Battery or equivalent Examples Intel Netlist HybriDIMM 1.VMware's estimate on Intel 3D XPoint capacity. Estimate is based on Intel's claim that 3D XPoint is ~10x capacity of DRAM 2.VMware's estimate on Intel 3D XPoint performance. Estimate is based on Intel's claim that 3D XPoint is ~10x latency of DRAM 3.Intel Developer Forum 2015 San Francisco, Diane Bryant Keynote 4.VMware's estimate on Intel 3D XPoint endurance. Estimate is based on Intel's claim that 3D XPoint is capable of up to 1000 times greater endurance than NAND. #FUT3040BU CONFIDENTIAL 15

16 Key Characteristics of PMEM Solutions (2) 5. How persistence is ensured during a power fail. All PMEM solutions need a way of ensuring that the last set of updates have made it. Persistence Domain size depends on SW requirements, platform power budget, and PMEM energy source. Some PMEM solutions can ensure from the CPU cache outward; others from just DRAM. 6. The Server hardware & firmware support of PMEM. Correct and tested UEFI/ACPI interfaces are critical to Operating Systems & Hypervisors. Well-tested error and power-fail handling are critical. From VMware s POV, this is so important that we will only certify a PMEM solution in combination with the platform that supports it. PCIe*3.0 IMC DMI IMC Core LLC Core LLC.. Core LLC Core LLC Intel UPI Processor DRAM Controller The HPE Smart Storage Battery Supercap NAND Flash HPE Integrated design eliminates thermal and connection Issues associated with alternative backup power sources. Pre-wired to memory slots to support NVDIMMs No extra cabling required #FUT3040BU CONFIDENTIAL 16

17 HPE 16GB NVDIMM & PMEM Solutions HPE 16GB NVDIMMs are flash-backed DIMMs delivering the performance of memory with the persistence of storage. HPE Scalable Persistent Memory is an integrated storage solution that runs at memory speeds with terabyte capacity. VMworld 2017 Learn more: 3 HPE Internal lab testing. Data gathered on pre-release hardware and software, final results may differ, Nov Comparing 8GB NVDIMM to 16GB NVDIMM equals 2X capacity increase, September Key Features: DRAM-level performance for fastest performing persistent memory Up to 192GB total capacity in a single server HPE Smart Storage Battery for backup power 2x the capacity of 1 st generation HPE NVDIMMs 4 Up to 50% reduction in database licensing costs using the fastest tier of storage on HPE servers 3 Key Features: DRAM-level performance for fastest performing persistent memory Flexible capacity points with capacity up to 1TB Complete solution using DRAM for application performance, flash tier for persistence and backup power for moving data to persistent store Up to 13x reduction in database restart time preserving maximum uptime 1 Up to 27x faster checkpoint operations enabling significantly faster business operations 2 Content: Not for publication 1 HPE Internal Labs test. HPE Scalable Persistent Memory, restarting 1TB Database with MSFT SQL Hekaton prototype is as fast as restarting 200GB database, September, Up to 20x with SAS SSDs. 2 TPC-C Benchmark Throughput with Checkpoint (trans/sec). Calculated Time to Checkpoint and Restore a Docker Container running MySQL, compare Persistent Memory vs. SSD, March 31, 2017 #FUT3040BU CONFIDENTIAL 17

18 VMware and the Amazing Future of Virtual Non-Volatile Memory Rajesh Venkatasubramanian VMworld 2017 Content: Not for publication

19 Concept: VMware vsphere Persistent Memory Support Virtualize and manage NVDIMMs as the hardware hits the market Accelerate legacy and modified applications running in VMs Virtual disks stored on PMem No change to the guest OS or application stack - just a simple change to the VM Byte-addressable virtual hardware for virtual NVDIMMs Applications using direct-access mode (DAX mode) gain all the benefits of VMware virtualization AND gain direct access to real hardware NVDIMMs 19

20 Where We are Headed PMEM.IO Volatile RAM vnvdimm PMEM.IO vpmemdisk Mem Scheduler Future Version of VMware vsphere Goal: Your applications whether modified to use NVDIMMs or legacy can take advantage of PMem on VMware vsphere! PMEM Application DAX/SCM Windows 2016 Drivers Persistent Memory Datastore nvdimm-0 DRAM Disk Future vsphere (Hypervisor) Application DAX Disk Linux Kernel vscsi Drivers pmemdsk-0 dsk1.vmdk DRAM PMEM Disk VM PMEM Disk DRAM Server DRAM NVDIMMs Disks 20

21 vsphere Support For Persistent Memory PMemDS vcenter & DRS NVDIMMs NVDIMM NVDIMMs NVDIMM NVDIMMs

22 vsphere Support For Persistent Memory (2) PMemDS Enter maintenance mode (vacate powered off VMs also) vcenter & DRS NVDIMMs NVDIMM NVDIMMs NVDIMM NVDIMMs

23 Create VM Workflow Mockup 23

24 Mockup for Upgrading All VM Disks to vpmemdiks 24

25 Migrate VM Workflow Mockup 25

26 Applications Using Persistent Memory

27 Modes of Operation NVMe SSD FS vnvme Emulation Storage stack NVMe vpmemdisk FS vnvme Emulation PMem vnvdimm vnvdimm DAX PMem FS PMem 27

28 Performance - Single Thread Fedora VM - FIO

29 Performance - Four Threads Bandwidth (GBPS) KB Rand Read 4KB Rand Write 512KB Rand Read 512KB Rand Write vnvdimm 29

30 A Tryst with Redis Why Redis? A key-value store; about 40K lines of well written C, open source, easy to understand Keeps all data structures in memory, saves periodically to persistent media In-memory format is not the same as the layout in persistent media Format conversion happens when data is written back to disk and re-read into memory Serialization and de-serialization are CPU hungry operations Time for in-memory image warm up is proportional to the database size Persistence guarantees Are configurable, but expensive when you choose fine grained Background thread that saves the in-memory data needs to be given a high priority 30

31 Redis Using Virtual NVDIMMs Runs in a Linux VM on top of the modified VMware vsphere ESXi/VMKernel Keeps the database in virtual NVDIMMs Keys and values are volatile only while still in the network buffers Data is written in a journal before in-memory data structures are updated No background save thread any more Large reduction in software complexity No need to provision part of the system a priori for this activity CPU is freed from format conversion activities Instant restart after a crash - warm up time is not proportional to the size of the data set Makes better use the expensive hardware, significantly lowered software complexity 31

32 PMem-Aware Redis 32

33

34