Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitme

Transcription

1 SER2734BU Extreme Performance Series: Byte-Addressable Nonvolatile Memory in vsphere VMworld 2017 Content: Not for publication Qasim Ali and Praveen Yedlapalli #VMworld #SER2734BU

2 Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitment from VMware to deliver these features in any generally available product. Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery. Pricing and packaging for any new technologies or features discussed or presented have not been determined. 2

3 Disclaimer All the performance results and experiments were performed on a prototype version of vsphere and hardware Results may vary on a future vsphere release and hardware VMworld 2017 Content: Not for publication 3

4 Convergence of Memory and Storage Load-Store Data Low Latency Volatile Block-Store Data Long Latency Non-Volatile CPU Cache DRAM Flash SDD Magnetic HDD Persistent Memory (PMEM) Byte Addressable Latency: A few hundred nanoseconds or less on average High durability Low power consumption than DRAM 4

5 Different Types of PMEM/NVM Block-Mode Access NVMe NVM Express: NVM block-mode attached via PCIe. Non-Volatile Memory NVM VMworld 2017 Random Access (byte/word addressable) NVRAM NVM with Random-Access Memory. PMEM NVRAM that is direct-mapped & addressable by CPU Load/Store Instructions Content: Not for publication NVDIMM-N,P PMEM implemented in DIMM form factor. A future version of VMware vsphere intends to enable PMEM by supporting a virtualized NVDIMM device. 6

6 Outline Persistent Memory (PMEM) in vsphere Host level configuration VM level configuration How to use PMEM in vsphere VMs PMEM Performance Evaluation Microbenchmarks Application benchmarks vmotion of VMs with PMEM 7

7 How to use PMEM in vsphere

8 Host level PMEM Configuration 9

9 Host level PMEM Configuration Exposed as host local datastore Vmfs as the namespace filesystem 10

10 How PMEM is exposed to vsphere VMs vdisk Disk-0.vmdk Datastore PCIe/SAS vscsi vpmemdisk vpmem pmemdisk-0 Memory Bus PMEM Datastore nvdimm-0 vram VM ESX Host Memory Memory Bus 11

11 vpmemdisk Virtual Persistent Memory Disk (vpmemdisk) Guest will access a regular SCSI/NVMe device (block device) Virtual disk (vmdk) stored on PMEM datastore backed using PMEM Create a new vpmemdisk or Clone vpmemdisk from an existing regular vmdk 12

12 vpmemdisk Configuration in vsphere 13

13 vpmem Virtual Persistent Memory (vpmem) Exposed to the guest via new virtual device called NVDIMM VM can use it as a block device or byte-addressable/direct access (DAX) mode. Create NVDIMM controller and Add vpmem/nvdimm to a VM 14

14 vpmem Configuration 15

15 Software Evolution to Take Advantage of PMem Case 1: Legacy application on legacy guest OS Use vpmemdisk as ultra-fast disk Case 2: Legacy applications on modern guest OS Fast access to vpmem via file read/write Direct-access (DAX) file system gets rid of volatile buffer cache e.g., Windows Server 2016 and Linux Kernel 4.0+ Case 3: PMEM-optimized applications on modern guest OS Access memory-mapped vpmem via load/store instructions File Read/Write/fsync EXT4/NTFS vpmemdisk File Read/Write/fsync Modified EXT4/NTFS vpmem Load/Store/Commit e.g., Microsoft SQL Server 2016, REDIS-PMEM* Modified EXT4/NTFS * VMware s modified REDIS for PMEM vpmem 16

16 Performance Evaluation of PMEM in vsphere VMworld 2017 Content: Not for publication

17 PMEM Configurations for Performance Evaluation Configuration Guest IO mode SSD Block Block vpmemdisk Block Block vpmem Block/Byte Byte VMworld 2017 Virtual Device IO mode Content: Not for publication 18

18 Workloads Application Database VMworld 2017 VCbench (with Postgres DB) MS Exchange vmotion MySQL with sysbench MS SQL Server with HammerDB* Redis* Content: Not for publication Filesystem fio in Linux * PMEM-aware applications 19

19 Hardware Configuration HPE s ProLiant DL380 Gen9 prototype System 2 socket, 20 cores/40 threads 512GB DDR4 RAM 200GB prototype Scalable PMEM 1.6TB Intel P3600 NVMe SSD VMworld 2017 Content: Not for HPE s ProLiant DL380 Gen10 prototype System 2 socket, 28 cores/56 threads 1TB DDR4 RAM 1TB Scalable PMEM publication HP s NVMe SSDs (MO0800KEFHP) 20

20 FIO Config Details Fedora 25 VM 4 vcpus 8 GB vram 16GB PMEM disks 16GB SSD disks Fio Random access pattern libaio ioengine 4KB/512KB IO 100% Read and 100% Write 21

21 FIO Results w/ Fedora VM Bandwidth (GBytes/sec) KB Read 4KB Write 512KB Read 512KB Write VMworld 2017 Single Thread - FIO SSD vpmemdisk vpmem 4KB IO 99 %-ile Latency (microsec) SSD vpmemdisk vpmem Rand Read Rand Write Content: Not for publication Up to 127x improvement in latency with vpmem 22

22 FIO Scaling vpmem with 4 threads Bandwidth (GBytes/sec) Up to 32x performance improvement with vpmem over SSD Memory Bus Limit KB Read 4KB Write 512KB Read 512KB Write 23

23 Native vs VM (Fedora FIO Throughput Test) VM to native ratio Virtualization Overhead with vpmem is less than 5% 0 4KB Read 4KB Write 512KB Read 512KB Write 24

24 Database/Application Performance

25 Sysbench with MySQL Fedora 25 VM 10 vcpu 16GB RAM 16 GB SSD disk 16 GB PMEM disks Sysbench 3 tables 8GB of DB and Logs 9 threads Read-Write (80-20)

26 Sysbench Result Throughput (Trans/sec) Up to 1.8x better throughput Up to 2.3x better latency SSD vpmemdisk vpmem TPS Latency (95 %-ile) ms Latency (ms) 27

27 vcenter (VC) Performance using VCBench VC VM 32 vcpu 64 GB RAM DB & DBLOG, 25 GB each Postgres DB Infrastructure Managed 2K hosts (simulated) 25K VMs 28

28 vcenter (VC) Performance using VCBench Operations per minute % 14% 0 SSD vpmemdisk vpmem

29 VCBench IOPs IOPs Up to 40% increase in DB IOPs SSD vpmemdisk DB IOPS DBLOG IOPS 30

30 Performance of Microsoft Tier-1 Applications VMworld 2017 Content: Not for publication

31 Microsoft Exchange 2016 Jetstress 2013: Disk Subsystem Throughput test 16vCPU Windows 2016 VM with 8GB vram 16 DBs Total size = 300GB; 3 copies of each DB 16 threads driving the DBs 32

32 Microsoft Exchange DB Performance (Throughput) Exchange Transactional IOPs Exchange DB I/O Performance 1.7x increase in Transactional IOPs SSD vpmem DB Reads/Sec DB Writes/Sec Log Writes/Sec 33

33 Microsoft Exchange DB Performance (Latency in msecs) DB Reads Avg Latency (msec) DB Writes Avg Latency (msec) Log Writes Avg Latency (msec) SSD vpmem Speed Up x x x 34

34 HammerDB Opensource database load testing and benchmarking tool The tools implements an OLTP-type workload (60/40 Read/Write) with 8KB random reads and writes (during checkpoint) and 64KB sequential writes to the log file Used TPC-C Profile We used Microsoft SQL Server 2016 SP1 for the Database 35

35 HammerDB: Configuration Details Windows 2016 Server VM with MS SQL vCPU and 128GB vram DB size = 710GB, Logs = 160GB 2x HP NVMe 800GB SSD drives. Various configurations for performance evaluation DB and Logs on SSDs DB and Logs on vpmem Application/SQL specific optimization - Tail-of-the-Log on DAX volume 36

36 Normalized Throughput w.r.t SSD Hammer-DB Results SSD vpmemdisk vpmem 37

37 MSSQL Tail-of-the-Log Caching 1. Copy log records into buffer, building up block 2. Close log block once commit arrives 3. Schedule I/O to persist block on SSD 4. Complete transaction when I/O completes Memory (DRAM) Log Buffers * Reference: Microsoft Ignite 2016 SSD (Block) 38

38 MSSQL Tail-of-the-Log Caching on PMEM With ToL on PMEM: 1. Copy log records into buffer, building up block, persistently in PMEM 2. Complete transaction when commit arrives 3. Close log block when full 4. Schedule I/O to persist full block on SSD PMEM (Byte) Log Buffers * Reference: Microsoft Ignite 2016 SSD (Block) 39

39 Normalized Throughput w.r.t SSD SQL Application Change DAX Mode w/ 20MB TOL SSD SSD-TOL-vPMEM 40

40 vmotion Support for PMEM

41 vmotion Support for VMs with PMEM Location of data disks e.g., DB and Logs: Shared storage (SAN): Compute vmotion Direct attached Storage: Compute and Storage vmotion e.g., SSDs vpmemdisk: Compute and Storage vmotion vpmem: Similar to Compute vmotion vsphere Compute vmotion vsphere 42

42 vmotion of VMs with PMEM Two HPE Proliant DL 380 s Gen10 with 25GbE NICs vmotion Hammer-DB VM w/ ~1TB Database Performance Considerations for vmotion Duration of vmotion Performance during vmotion Does the performance recover after vmotion? 43

43 vmotion Time for 1TB VM vmotion Time (mins) VM Config: 128GB vram, ~900 GB DB&Logs SAN SSD vpmem 44

44 vmotion of Hammer-DB VM with ~1TB vpmem Normalized Throughput w.r.t SSD vmotion Begin SSD vpmem vmotion End Time (5 second interval) 45

45 PMEM-Aware Redis

46 PMEM-Aware Redis Redis is a high-performance, in-memory, key-value store (no-sql) Vanilla Redis Caching Layer PMEM-Aware Redis Primary Database Better Throughput & Latency Instant Startup/Recovery

47 PMEM-Aware Redis Configs Performance Results Memtier benchmark with various R/W ratios Throughput Mode (4 threads, 50 clients per thread, pipeline depth = 30) Latency Mode (1 thread, 1 client, pipeline depth = 1) Startup/Crash Recovery 16 GB Database 48

48 Redis Throughput Millions of Ops/sec Up to 1.9x throughput Improvement Read Intensive (20 % SETs) Balanced (50% SETs) Write Intensive (80% SETs) 1.6 SSD vpmemdisk vpmem (DAX)

49 Redis Latency Latency (us) Up to 11x latency Improvement SSD vpmemdisk vpmem (DAX) Read Intensive (20 % SETs) Balanced (50% SETs) Write Intensive (80% SETs) 50

50 Redis Startup/Crash Recovery Time 16 GB DB SSD vpmemdisk vpmem Recovery time (secs) Recovery time for SSD/vPMEMDisk scales with DB size Instant recovery with vpmem ~0 51

51 Conclusions A future version of vsphere intends to enable PMEM Both legacy and new applications can take advantage of vsphere PMEM solution vmotion enabled for VMs with PMEM Performance Improvements with PMEM on vsphere are impressive with little to no virtualization overhead 52

52 Extreme Performance Series - Hand on Labs Don t miss these popular Extreme Performance labs: HOL SDC: vsphere 6.5 Performance Diagnostics & Benchmarking Each module dives deep into vsphere performance best practices, diagnostics, and optimizations using various interfaces and benchmarking tools. HOL CHG: vsphere Challenge Lab Each module places you in a different fictional scenario to fix common vsphere operational and performance problems. 53

53 Extreme Performance Series Las Vegas SER2724BU Performance Best Practices SER2723BU Benchmarking 101 SER2343BU vsphere Compute & Memory Schedulers SER1504BU vcenter Performance Deep Dive SER2849BU Predictive DRS Performance & Best Practices SER1494BU Encrypted vmotion Architecture, Performance, & Futures STO1515BU vsan Performance Troubleshooting VIRT1445BU Fast Virtualized Hadoop and Spark on All-Flash Disks VIRT1397BU Optimize & Increase Performance Using VMware NSX VIRT2550BU Reducing Latency in Enterprise Applications with VMware NSX VIRT1052BU Monster VM Database Performance VIRT1983BU Cycle Stealing from the VDI Estate for Financial Modeling VIRT1997BU Machine Learning and Deep Learning on VMware vsphere FUT2020BU Wringing Max Perf from vsphere for Extremely Demanding Workloads FUT2761BU Sharing High Performance Interconnects across Multiple VMs 54

54 Extreme Performance Series Barcelona SER2724BE Performance Best Practices SER2343BE vsphere Compute & Memory Schedulers SER1504BE vcenter Performance Deep Dive SER2849BE Predictive DRS Performance & Best Practices VIRT1445BE Fast Virtualized Hadoop and Spark on All-Flash Disks VMworld 2017 VIRT1397BE Optimize & Increase Performance Using VMware NSX VIRT1052BE Monster VM Database Performance FUT2020BE Wringing Max Perf from vsphere for Extremely Demanding Workloads Content: Not for publication 55

55 Extreme Performance Series - Hand on Labs Don t miss these popular Extreme Performance labs: HOL SDC: vsphere 6.5 Performance Diagnostics & Benchmarking Each module dives deep into vsphere performance best practices, diagnostics, and optimizations using various interfaces and benchmarking tools. HOL CHG: vsphere Challenge Lab Each module places you in a different fictional scenario to fix common vsphere operational and performance problems. 56

56 Performance Survey The VMware Performance Engineering team is always looking for feedback about your experience with the performance of our products, our various tools, interfaces and where we can improve. Scan this QR code to access a short survey and provide us direct feedback. Alternatively: Thank you! VMworld 2017 Content: Not for publication 57

57

58