SER1494BU Encrypted vmotion in vsphere 6.5: Architecture, Performance and Futures Sreekanth Setty Arunachalam Ramanathan #VMworld #SER1494BU
Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitment from VMware to deliver these features in any generally available product. Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery. Pricing and packaging for any new technologies or features discussed or presented have not been determined. #SER1494BU CONFIDENTIAL 2
Agenda 1 Encrypted vmotion 2 vmotion over 100 GbE 3 vmotion Futures #SER1494BU CONFIDENTIAL 3
Encrypted vmotion Architecture
Why Encrypted vmotion? vmotion network prior to vsphere 6.5 VM s data transmitted in plain text Memory pages Device state Disks for Cross vmotion For secure transfer Separate, isolated network Non-routable Challenges Security vmotion across datacenters L3 network VMworld 2017 Content: Not for publication Long Distance vmotion VM Encryption On-prem to VMC on AWS Software defined datacenter vmotion to public cloud #SER1494BU CONFIDENTIAL 5
How to Encrypt vmotion? Application Layer SSL Library vmotion Kernel User Space Transport Layer TCP Internet Protocol IP Sec UDP Performance? Can t saturate 40/100 GbE Configuration? Only IPv6 Support in ESX Configure every vmknic No per VM config vmotion Encryption Protocol Implementation? Design a protocol that meets all requirements #SER1494BU CONFIDENTIAL 6
vmotion Encryption Protocol vmkernel crypto library vmkernel API for symmetric encryption Direct import of BoringSSL, a fork of OpenSSL FIPS certified NIST 800-38D Standard Key is a secret, IV is unique per key 1. 256 bit key 2. 96 bit IV 3. 128 bit authentication tag vmotion Encryption Protocol Symmetric Key Encryption VC trusted third party No public key encryption problems 1. Complex math - Slow 2. No certificate management AES GCM AES - Advanced Encryption Standard GCM - Galois Counter Mode 1. Performance advantage 2. Confidentiality and integrity in one operation #SER1494BU CONFIDENTIAL 7
Encrypted vmotion Encryption Key Management VC generates a new key for every vmotion Key is destroyed after vmotion Symmetric Key Encryption Confidentiality : AES Integrity : GCM Authentication : VC Encrypt vmotion network traffic Transparent to VM s OS and applications Per VM configuration Encrypted vcenter Server Migration Spec Encrypted vmotion ESX ESX Encrypted traffic vmotion Network Encryption Key Nonce #SER1494BU CONFIDENTIAL 8
How to Use Encrypted vmotion? Per VM configuration Disabled Do not use Encrypted vmotion Even when its available Opportunistic Default Option Encrypted vmotion if ESX hosts are capable Unencrypted vmotion if ESX hosts not capable Required Allow only Encrypted vmotion Block vmotion if ESX hosts or VC is not capable #SER1494BU CONFIDENTIAL 9
What Does Encrypted vmotion Buy? Protects Data in transit Eavesdrop vmotion traffic Replay of vmotion traffic Tamper with vmotion traffic Doesn t Protect Data at rest/use Compromised ESX host Compromised VC Malicious access to memory dump/swap file Compromised VC to ESX/VC communication channel Doesn t Protect Doesn t Denial of Protect service Modify a single byte in traffic vmotion Failure Injecting random traffic vmotion Failure #SER1494BU CONFIDENTIAL 10
vmotion Protocol Is Encrypted Encryption Header IV - Unique for every encrypted message Authentication tag Provides data integrity Replay attack counter One per data stream 96 bit IV 128 bit tag 8 byte counter Initialization Vector Authentication Tag 36 byte encryption header Replay attack counter vmotion message is encrypted Consists of message header and data Separate encryption header for message header and message data IP Header Encryption Header TCP Header Encrypted vmotion Header Regular vmotion message vmotion Header Encryption Header Encrypted vmotion message TCP Payload vmotion Data Encrypted vmotion Data #SER1494BU CONFIDENTIAL 11
Designed for Parallelism Regular vmotion stream 2 Threads: Prepare and transmit vmotion Header Prepare Thread vmotion Data vmotion Message Transmit Thread Encrypted vmotion stream Parallelism: Split data into segments 3 Threads: Prepare, encrypt, transmit vmotion Header Prepare Thread S1.. S8 Encrypt Thread Encrypt segment Encryption Encrypted Header Segment 1 Transmit Thread Encryption Header Enc Msg Encrypt segment and Transmit full message Encrypted Segment 2 Line rate on 40 GbE network Network Line rate on 40 GbE network with encryption Network #SER1494BU CONFIDENTIAL 12
Encrypted vmotion Performance
How to Gauge vmotion Performance Migration Time (memory, disk, total) Impact on guest applications Application latency and throughput during vmotion Time to resume to normal level of performance Switch-over Time vmotion CPU Usage #SER1494BU CONFIDENTIAL 14
Encrypted vmotion Performance Testbed Configuration Hosts: Dell PowerEdge R730 w/ Intel Broadwell @ 3 GHz, 24 cores, 1 TB NICs: 4 * Intel 10 GbE Encrypted vmotion CPU Requirement: support for AES New Instruction Set Memory intensive micro-benchmarks OLTP Database workloads In-memory Database workloads Test Scenarios Encrypted vmotion (with typical customer deployment scenarios) Long Distance migrations Workloads #SER1494BU CONFIDENTIAL 15
Encrypted vmotion Performance VM : 8 vcpus, 150 GB memory, SLES x64 Workload: Redis (in-memory key/value datastore) Redis config: 4 instances of Redis, each with keyspace of 100 million keys Test Scenario GET Operations (per second) SET Operations (per second) Idle 0 0 100% GET / 0% SET 200,000 0 75% GET / 25% SET 150,000 50,000 50% GET / 50% SET 100,000 100,000 25% GET / 50% SET 50,000 150,000 VMworld 2017 Content: Not for 0% GET / 100% SET 0 200,000 Duration (in seconds) 60 50 40 30 20 10 0 vmotion Performance: Encryption On Vs Off (Active Redis VM migration over 40GbE) publication Idle 100% GET 75% GET / 25% SET Duration Switchover Time 50% GET / 50% SET 25% GET / 75% SET Duration (Encryption) 100% SET Switchover Time (Encryption) 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 Switchover Time (in seconds) No impact on Active VM vmotions when encryption is on #SER1494BU CONFIDENTIAL 16
Encrypted vmotion Peak Network Throughput 40 35 30 25 20 15 10 5 0 10 20 30 40 Available Network Bandwidth Peak Network Bandwidth Usage () Encrypted vmotion Peak Network Bandwidth Usage () No impact on vmotion peak throughput when encryption is on #SER1494BU CONFIDENTIAL 17
Encrypted vmotion CPU Cost 40 30 20 10 Encrypted vmotion CPU Overhead (Source host) 0 1 2 3 4 5 6 7 CPU cores 40 30 20 10 Encrypted vmotion CPU Overhead (Destination host) 0 1 2 3 4 5 6 7 CPU cores Workload: Memhog micro-benchmark CPU usage captured during steady-state CONFIDENTIAL 18
Encrypted vmotion CPU Cost 40 30 20 10 Encrypted vmotion CPU Overhead (Source host) 0 1 2 3 4 5 6 7 CPU cores vmotion requires more CPU on destination host compared to source host 40 30 20 10 Encrypted vmotion CPU Overhead (Destination host) 0 1 2 3 4 5 6 7 CPU cores CONFIDENTIAL 19
Encrypted vmotion CPU Cost 40 30 20 10 Encrypted vmotion CPU Overhead (Source host) 0 1 2 3 4 5 6 7 CPU cores vmotion requires more CPU on destination host compared to source host Encryption is more compute intensive than Decryption 40 30 20 10 Encrypted vmotion CPU Overhead (Destination host) 0 1 2 3 4 5 6 7 CPU cores CONFIDENTIAL 20
Encrypted vmotion CPU Cost 40 30 20 10 Encrypted vmotion CPU Overhead (Source host) 0 1 2 3 4 5 6 7 CPU cores vmotion requires more CPU on destination host compared to source host Encryption is more compute intensive than Decryption 0 1 2 3 4 5 6 7 CPU overhead: approx. 0.8 core on source host and 0.4 core on destination host (per 10 vmotion traffic) 40 30 20 10 Encrypted vmotion CPU Overhead (Destination host) CPU cores CONFIDENTIAL 21
Long Distance Encrypted vmotion Performance VMworld 2017 Content: Not for publication
Testbed Logical Layout Site A VM migration across the vcenter Servers No shared storage between the two sites vmotion traffic encrypted Site B #SER1494BU CONFIDENTIAL 23
Testbed Physical Layout Hosts: Dell PowerEdge R730 w/ Intel Broadwell @ 3 GHz, 24 cores, 1 TB Storage: Intel NVMe disks NICs: Intel 1 GbE vmotion Network: latency injected through Maxwell appliance ESX 6.5 1 GbE link (Encrypted vmotion traffic) 1 GbE link (Encrypted vmotion traffic) vsphere Source Host Maxwell Appliance vsphere Destination Host #SER1494BU CONFIDENTIAL 24
Long Distance vmotion Performance in Database Environment Workload Open source DVD Store 2 database OLTP workload Performance Metric: orders per second (1-second granularity) Test Scenario VM configuration: 8 VCPUs, 24 GB memory, 40GB Sys disk, 30GB Db disk, Windows Server 2012 MS SQL Server Database size: 12 GB Benchmark load: 5 DVD store users with no think-time CONFIDENTIAL #SER1494BU CONFIDENTIAL 25
Long Distance vmotion Performance in Database Environment Total Duration (in seconds) 800 700 600 500 400 300 200 100 0 vmotion Duration (with varying RTT, Encrypted traffic ) <1ms 5 ms 25 ms 50 ms 75 ms 100 ms 125ms 150ms Round-trip latency Duration not impacted by round-trip latency when migrating an active VM #SER1494BU CONFIDENTIAL 26
Long Distance vmotion Performance in Database Environment Orders Per Second 300 250 200 150 100 50 0 Source Datacenter Long Distance vmotion (150ms RTT latency, Encrypted traffic) Encrypted vmotion over Long distance 1 22 43 64 85 106 127 148 169 190 211 232 253 274 295 316 337 358 379 400 421 442 463 484 505 526 547 568 589 610 631 652 673 694 715 736 757 778 799 820 841 862 883 904 925 946 967 Time (in seconds) Switch-over Destination Datacenter Minimal impact on guest even when migrating across cross-continental distance! #SER1494BU CONFIDENTIAL 27
Agenda 1 Encrypted vmotion 2 vmotion over 100 GbE 3 vmotion Futures #SER1494BU CONFIDENTIAL 28
vmotion over 100 GbE Architecture
vmotion Performance Timeline v1.0 v4.1 vmotion 1 GbE Introduce vmotion Stream v5.0 v6.0 v6.5 Multi-NIC vmotion Streams New Stream Architecture Encrypted vmotion Stream Architecure 10 GbE 2 x 10 GbE 40 GbE 40 GbE Prepare Transmit Prepare Encrypt Transmit Next? Whats new? 100 GbE vmotion stream Multiple 10 GbE NIC New vmotion stream Encrypted vmotion stream 100 GbE NIC #SER1494BU CONFIDENTIAL 30
vmotion Stream Architecture vmotion stream 3 Threads per stream for parallelism vmotion vmkernel nic Default: 1 vmotion stream per vmknic Physical Nic hardware queue VMworld 2017 Content: Not for Prepare Thread Encrypt Thread vmotion single stream vmotion vmkernel nic Physical NIC Transmit Thread Software TX/RX Dispatch queue publication Hardware TX/RX Dispatch queue #SER1494BU CONFIDENTIAL 31
vmotion over 100 GbE Performance: Monster VM migrations
ESX 6.5 vmotion over 100 GbE vmotion Stream Helper Software RX Dispatch queue Hardware RX Dispatch queue vmotion vmknic vmotion Stack Vmkernel network Stack vmotion Stream Helper Software RX Dispatch queue Hardware RX Dispatch queue vmotion vmknic vmotion Stack Vmkernel network Stack CPU 100% busy 10 Gbe NIC 100 Gbe NIC 33 #SER1494BU CONFIDENTIAL 33
ESX 6.5 vmotion over 100 GbE Instantiate more vmotion streams VMworld 2017 vmotion Stream Helpers Software RX Dispatch queue Hardware RX Dispatch queue vmotion vmknic vmotion Stack Vmkernel network Stack Content: Not for publication Rx Queue bottleneck 100 Gbe NIC #SER1494BU CONFIDENTIAL 34
6.5 vmotion over 100 GbE Instantiate more vmotion streams Create more vmknics on a 100GbE NIC VMworld 2017 vmotion Stream Helpers vmotion vmknic Hardware RX Dispatch queues vmotion vmknic vmotion Stack Vmkernel network Stack Content: Not for publication 100 Gbe NIC #SER1494BU CONFIDENTIAL 35
ESX 6.5 vmotion Performance over 100GbE NIC Testbed Configuration Hosts: Dell PowerEdge R930 w/ Intel Xeon E7-8890 @ 2.2 GHz, 96 cores, 4 TB NIC: Mellanox 100GbE ESX 6.5 Test Scenario VM configured w/ 12 VCPUs, 1 TB memory, 300GB sys disk, SLES 11 as guest OS Workload: Memhog micro benchmark (extremely memory intensive) Needs nearly 6 vmknics to reach high throughput (due to current design) Encrypted vmotion scalability limited (as encryption is very compute intensive) 90 80 70 60 50 40 30 20 10 0 or 1 vmknic 2 vmknics 4 vmknics 6 vmknics distribution Peak Network Bandwidth Usage () Encrypted vmotion Peak Network Bandwidth Usage () #SER1494BU CONFIDENTIAL 36
vmotion over 100 GbE (Upcoming Performance Optimizations) Instantiate more vmotion streams Leverage pnic Receive Side Scaling (RSS) VMworld 2017 vmotion Stream Helpers Software RX Dispatch queue Hardware RX Dispatch queues vmotion vmknic vmotion Stack Vmkernel network Stack Content: Not for publication 100 Gbe NIC #SER1494BU CONFIDENTIAL 37
vmotion over 100 GbE (Upcoming Performance Optimizations) VMworld 2017 vmotion Stream Helpers Software RX Dispatch queue Hardware RX Dispatch queues vmotion vmknic vmotion Stack Vmkernel network Stack Rx Queue bottleneck Content: Not for publication 100 Gbe NIC 38 #SER1494BU CONFIDENTIAL 38
vmotion over 100 GbE (Upcoming Performance Optimizations) Use multiple Software Rx Queues 39 VMworld 2017 vmotion Stream Helpers Hardware RX Dispatch queues vmotion vmknic vmotion Stack Vmkernel network Stack Content: Not for publication 100 Gbe NIC #SER1494BU CONFIDENTIAL 39
vmotion over 100 GbE (Upcoming Performance Improvements) vmotion Traffic Few Rx Queues With Receive Side Scaling Physical NIC Logic to place packets to Rx Queues Q u e u e Q u e u e Q u e u e CPU utilization per core vmotion Traffic More Rx Queues With Dynamic Receive Side Scaling Q u e u e Physical NIC Logic to place packets to Rx Queues Q u e u e Q u e u e Q u e u e Q u e u e CPU utilization per core Q u e u e Core 1 Core 2 Core 3 Core 4 Core 5 Core N Core 1 Core 2 Core 3 Core 4 Core 5 Core N #SER1494BU CONFIDENTIAL 40
vmotion over 100 GbE (Upcoming Performance Optimizations) Goal: Instantiate optimal # vmotion streams Goal: Create optimal # Software vmknic Rx queues Goal: Leverage RX queues in the pnic 41 VMworld 2017 vmotion Stream Helpers Software RX Dispatch queues Hardware RX Dispatch queues vmotion vmknic vmotion Stack Vmkernel network Stack Content: Not for publication 100 Gbe NIC #SER1494BU CONFIDENTIAL 41
vmotion over 100 GbE (Upcoming Performance Improvements) 90 85 80 70 60 50 40 30 20 10 Single vmknic vmotion Peak Throughput over 100 GbE 19 0 Before After #SER1494BU CONFIDENTIAL 42
Monster VM vmotion Performance Testbed Configuration Hosts: Dell PowerEdge R730 w/ Intel Broadwell @ 3 GHz, 24 cores, 1 TB NICs: Mellanox 40GbE and 4 * Intel 10 GbE (total 80 GbE bandwidth) Storage: Intel NVMe disks ESX 6.5 (vmotion traffic encrypted) Test Scenario VM configured w/ 24 VCPUs, 512 GB memory, 300GB sys disk, SLES 11 as guest OS Application: open-source Redis 3.0.3 (In-memory key/value store) Redis-Benchmark Load: 160 clients Operation: SET 1 byte (with a randomly generated key in a space of 2 billion keys) 8 Redis server instances (serving approx. 2 million requests/sec) #SER1494BU CONFIDENTIAL 43
Monster VM vmotion Performance Idle VM 1 redis instances / 20 clients 2 redis instances / 40 clients 4 redis instances / 80 clients 8 redis instances / 160 clients vmotion Peak Network Throughput Usage 0 10 20 30 40 50 60 70 80 Gbps Mileage (network bandwidth usage) varies based on workload 30 Gbps peak usage even during Idle VM migration #SER1494BU CONFIDENTIAL 44
Dissection of Monster VM Migration Network Bandwidth (Gbps) 80 70 60 50 40 30 20 10 0 Time Test Scenario: 24 VCPUs, 512 GB memory, 300GB Sys disk, 8 Redis instances Mileage also varies in different phases of vmotion #SER1494BU CONFIDENTIAL 45
Dissection of Monster VM Migration Network Bandwidth (Gbps) 80 70 60 50 40 30 20 10 0 Disk Copy Time Network bandwidth usage typically limited by Storage throughput Transmit rate of disk copy increased gradually to avoid overwhelming the destination CONFIDENTIAL #SER1494BU CONFIDENTIAL 46
Dissection of Monster VM Migration Network Bandwidth (Gbps) 80 70 60 50 40 30 20 10 0 Time Precopy - 1 35 Gbps average network bandwidth usage (limited by memory allocation on destination host) CONFIDENTIAL #SER1494BU CONFIDENTIAL 47
Dissection of Monster VM Migration Network Bandwidth (Gbps) 80 70 60 50 40 30 20 10 0 Time Bandwidth usage reaching line speed during Precopy-2 Precopy - 2 CONFIDENTIAL #SER1494BU CONFIDENTIAL 48
CONFIDENTIAL 49
Agenda 1 Encrypted vmotion 2 vmotion over 100 GbE 3 vmotion Futures #SER1494BU CONFIDENTIAL 50
vmotion Futures What s next for vmotion?
1 2 3 4 Snowball + vmotion Order snowball Copy & ship snowball Data Transfer to S3 Data Transfer to SDDC vcenter Server ESX Hybrid Linked Mode vmotion L2 VM Network L3 vmotion Network Snowball vcenter Server ESX 5 Cross VC vmotion Snowball AWS S3 #SER1494BU CONFIDENTIAL 52
vmotion Futures Active / Passive Storage Replication Leverage broad partner ecosystem VVol required to reverse replication direction after vmotion Support next generation hardware advancements Non Volatile Memory (NVM) RDMA over Ethernet (RoCE) Page modification logging (PML) 4Kn disks VM1 Active Passive Replication Broad partner ecosystem #SER1494BU CONFIDENTIAL 53
Summary vmotion is continually innovating in pace with technology trends and customer requirements Encrypted vmotion : All vmotion network traffic is encrypted Securely cross geographical and management boundaries vmotion Performance Encrypted vmotion : Identical performance as unencrypted vmotion Monster VM vmotion : Achieve near line rate on 100 GbE What s next for vmotion? Snowball + vmotion : Hybrid cloud Mobility between on prem and vsphere on AWS Active/Passive replication : Optimize disk copy Hardware advancements : NVM, 4Kn, RDMA over Ethernet, PML #SER1494BU CONFIDENTIAL 54
Extreme Performance Series Las Vegas SER2724BU Performance Best Practices SER2723BU Benchmarking 101 SER2343BU vsphere Compute & Memory Schedulers SER1504BU vcenter Performance Deep Dive SER2734BU Byte Addressable Non-Volatile Memory in vsphere SER2849BU Predictive DRS Performance & Best Practices SER1494BU Encrypted vmotion Architecture, Performance, & Futures STO1515BU vsan Performance Troubleshooting VIRT1445BU Fast Virtualized Hadoop and Spark on All-Flash Disks VIRT1397BU Optimize & Increase Performance Using VMware NSX VIRT2550BU Reducing Latency in Enterprise Applications with VMware NSX VIRT1052BU Monster VM Database Performance VIRT1983BU Cycle Stealing from the VDI Estate for Financial Modeling VIRT1997BU Machine Learning and Deep Learning on VMware vsphere FUT2020BU Wringing Max Perf from vsphere for Extremely Demanding Workloads FUT2761BU Sharing High Performance Interconnects across Multiple VMs #SER1494BU CONFIDENTIAL 55
Extreme Performance Series - Hand on Labs Don t miss these popular Extreme Performance labs: HOL-1804-01-SDC: vsphere 6.5 Performance Diagnostics & Benchmarking Each module dives deep into vsphere performance best practices, diagnostics, and optimizations using various interfaces and benchmarking tools. HOL-1804-02-CHG: vsphere Challenge Lab Each module places you in a different fictional scenario to fix common vsphere operational and performance problems. #SER1494BU CONFIDENTIAL 56
Performance Survey The VMware Performance Engineering team is always looking for feedback about your experience with the performance of our products, our various tools, interfaces and where we can improve. Scan this QR code to access a short survey and provide us direct feedback. Alternatively: www.vmware.com/go/perf Thank you! VMworld 2017 Content: Not for publication #SER1494BU CONFIDENTIAL 57