Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitme

Size: px

Start display at page:

Download "Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitme"

Allen Stephens
6 years ago
Views:

1 SER1494BU Encrypted vmotion in vsphere 6.5: Architecture, Performance and Futures Sreekanth Setty Arunachalam Ramanathan #VMworld #SER1494BU

2 Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitment from VMware to deliver these features in any generally available product. Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery. Pricing and packaging for any new technologies or features discussed or presented have not been determined. #SER1494BU CONFIDENTIAL 2

3 Agenda 1 Encrypted vmotion 2 vmotion over 100 GbE 3 vmotion Futures #SER1494BU CONFIDENTIAL 3

4 Encrypted vmotion Architecture

5 VM s data transmitted in plain text Memory pages

transfer Separate, isolated network Non-routable

network VMworld 2017 Content: Not for publication Long

5 Why Encrypted vmotion? vmotion network prior to vsphere 6.5 VM s data transmitted in plain text Memory pages Device state Disks for Cross vmotion For secure transfer Separate, isolated network Non-routable Challenges Security vmotion across datacenters L3 network VMworld 2017 Content: Not for publication Long Distance vmotion VM Encryption On-prem to VMC on AWS Software defined datacenter vmotion to public cloud #SER1494BU CONFIDENTIAL 5

Only IPv6 Support in ESX Configure every vmknic No per VM

6 How to Encrypt vmotion? Application Layer SSL Library vmotion Kernel User Space Transport Layer TCP Internet Protocol IP Sec UDP Performance? Can t saturate 40/100 GbE Configuration? Only IPv6 Support in ESX Configure every vmknic No per VM config vmotion Encryption Protocol Implementation? Design a protocol that meets all requirements #SER1494BU CONFIDENTIAL 6

vmotion Encryption Protocol vmkernel crypto library vmkernel API for

certified NIST 800-38D Standard Key is a secret, IV is unique per key 1.

128 bit authentication tag vmotion Encryption Protocol Symmetric Key

7 vmotion Encryption Protocol vmkernel crypto library vmkernel API for symmetric encryption Direct import of BoringSSL, a fork of OpenSSL FIPS certified NIST D Standard Key is a secret, IV is unique per key bit key bit IV bit authentication tag vmotion Encryption Protocol Symmetric Key Encryption VC trusted third party No public key encryption problems 1. Complex math - Slow 2. No certificate management AES GCM AES - Advanced Encryption Standard GCM - Galois Counter Mode 1. Performance advantage 2. Confidentiality and integrity in one operation #SER1494BU CONFIDENTIAL 7

network traffic Transparent to VM s OS and applications Per VM configuration Encrypted vcenter Server

8 Encrypted vmotion Encryption Key Management VC generates a new key for every vmotion Key is destroyed after vmotion Symmetric Key Encryption Confidentiality : AES Integrity : GCM Authentication : VC Encrypt vmotion network traffic Transparent to VM s OS and applications Per VM configuration Encrypted vcenter Server Migration Spec Encrypted vmotion ESX ESX Encrypted traffic vmotion Network Encryption Key Nonce #SER1494BU CONFIDENTIAL 8

Opportunistic Default Option Encrypted vmotion if ESX hosts are capable Unencrypted

9 How to Use Encrypted vmotion? Per VM configuration Disabled Do not use Encrypted vmotion Even when its available Opportunistic Default Option Encrypted vmotion if ESX hosts are capable Unencrypted vmotion if ESX hosts not capable Required Allow only Encrypted vmotion Block vmotion if ESX hosts or VC is not capable #SER1494BU CONFIDENTIAL 9

10 What Does Encrypted vmotion Buy? Protects Data in transit Eavesdrop vmotion traffic Replay of vmotion traffic Tamper with vmotion traffic Doesn t Protect Data at rest/use Compromised ESX host Compromised VC Malicious access to memory dump/swap file Compromised VC to ESX/VC communication channel Doesn t Protect Doesn t Denial of Protect service Modify a single byte in traffic vmotion Failure Injecting random traffic vmotion Failure #SER1494BU CONFIDENTIAL 10

11 vmotion Protocol Is Encrypted Encryption Header IV - Unique for every encrypted message Authentication tag Provides data integrity Replay attack counter One per data stream 96 bit IV 128 bit tag 8 byte counter Initialization Vector Authentication Tag 36 byte encryption header Replay attack counter vmotion message is encrypted Consists of message header and data Separate encryption header for message header and message data IP Header Encryption Header TCP Header Encrypted vmotion Header Regular vmotion message vmotion Header Encryption Header Encrypted vmotion message TCP Payload vmotion Data Encrypted vmotion Data #SER1494BU CONFIDENTIAL 11

12 Designed for Parallelism Regular vmotion stream 2 Threads: Prepare and transmit vmotion Header Prepare Thread vmotion Data vmotion Message Transmit Thread Encrypted vmotion stream Parallelism: Split data into segments 3 Threads: Prepare, encrypt, transmit vmotion Header Prepare Thread S1.. S8 Encrypt Thread Encrypt segment Encryption Encrypted Header Segment 1 Transmit Thread Encryption Header Enc Msg Encrypt segment and Transmit full message Encrypted Segment 2 Line rate on 40 GbE network Network Line rate on 40 GbE network with encryption Network #SER1494BU CONFIDENTIAL 12

13 Encrypted vmotion Performance

14 How to Gauge vmotion Performance Migration Time (memory, disk, total) Impact on guest applications Application latency and throughput during vmotion Time to resume to normal level of performance Switch-over Time vmotion CPU Usage #SER1494BU CONFIDENTIAL 14

Encrypted vmotion Performance Testbed Configuration Hosts: Dell PowerEdge

Encrypted vmotion CPU Requirement: support for AES New Instruction Set

Database workloads Test Scenarios Encrypted vmotion (with typical

15 Encrypted vmotion Performance Testbed Configuration Hosts: Dell PowerEdge R730 w/ Intel 3 GHz, 24 cores, 1 TB NICs: 4 * Intel 10 GbE Encrypted vmotion CPU Requirement: support for AES New Instruction Set Memory intensive micro-benchmarks OLTP Database workloads In-memory Database workloads Test Scenarios Encrypted vmotion (with typical customer deployment scenarios) Long Distance migrations Workloads #SER1494BU CONFIDENTIAL 15

Encrypted vmotion Performance VM : 8 vcpus, 150 GB memory, SLES x64 Workload: Redis (in-memory key/value datastore) Redis config: 4 instances of Redis, each with keyspace of 100 million keys Test

150,000 VMworld 2017 Content: Not for 0% GET / 100% SET 0 200,000 Duration (in seconds) 60 50 40 30 20 10 0 vmotion Performance: Encryption On Vs Off (Active Redis VM migration over 40GbE)

16 Encrypted vmotion Performance VM : 8 vcpus, 150 GB memory, SLES x64 Workload: Redis (in-memory key/value datastore) Redis config: 4 instances of Redis, each with keyspace of 100 million keys Test Scenario GET Operations (per second) SET Operations (per second) Idle % GET / 0% SET 200, % GET / 25% SET 150,000 50,000 50% GET / 50% SET 100, ,000 25% GET / 50% SET 50, ,000 VMworld 2017 Content: Not for 0% GET / 100% SET 0 200,000 Duration (in seconds) vmotion Performance: Encryption On Vs Off (Active Redis VM migration over 40GbE) publication Idle 100% GET 75% GET / 25% SET Duration Switchover Time 50% GET / 50% SET 25% GET / 75% SET Duration (Encryption) 100% SET Switchover Time (Encryption) Switchover Time (in seconds) No impact on Active VM vmotions when encryption is on #SER1494BU CONFIDENTIAL 16

17 Encrypted vmotion Peak Network Throughput Available Network Bandwidth Peak Network Bandwidth Usage () Encrypted vmotion Peak Network Bandwidth Usage () No impact on vmotion peak throughput when encryption is on #SER1494BU CONFIDENTIAL 17

18 Encrypted vmotion CPU Cost Encrypted vmotion CPU Overhead (Source host) CPU cores Encrypted vmotion CPU Overhead (Destination host) CPU cores Workload: Memhog micro-benchmark CPU usage captured during steady-state CONFIDENTIAL 18

19 Encrypted vmotion CPU Cost Encrypted vmotion CPU Overhead (Source host) CPU cores vmotion requires more CPU on destination host compared to source host Encrypted vmotion CPU Overhead (Destination host) CPU cores CONFIDENTIAL 19

20 Encrypted vmotion CPU Cost Encrypted vmotion CPU Overhead (Source host) CPU cores vmotion requires more CPU on destination host compared to source host Encryption is more compute intensive than Decryption Encrypted vmotion CPU Overhead (Destination host) CPU cores CONFIDENTIAL 20

21 Encrypted vmotion CPU Cost Encrypted vmotion CPU Overhead (Source host) CPU cores vmotion requires more CPU on destination host compared to source host Encryption is more compute intensive than Decryption CPU overhead: approx. 0.8 core on source host and 0.4 core on destination host (per 10 vmotion traffic) Encrypted vmotion CPU Overhead (Destination host) CPU cores CONFIDENTIAL 21

22 Long Distance Encrypted vmotion Performance VMworld 2017 Content: Not for publication

23 Testbed Logical Layout Site A VM migration across the vcenter Servers No shared storage between the two sites vmotion traffic encrypted Site B #SER1494BU CONFIDENTIAL 23

Testbed Physical Layout Hosts: Dell PowerEdge R730 w/ Intel Broadwell @ 3 GHz, 24 cores, 1 TB Storage: Intel NVMe disks NICs: Intel 1 GbE vmotion Network: latency injected through

24 Testbed Physical Layout Hosts: Dell PowerEdge R730 w/ Intel 3 GHz, 24 cores, 1 TB Storage: Intel NVMe disks NICs: Intel 1 GbE vmotion Network: latency injected through Maxwell appliance ESX GbE link (Encrypted vmotion traffic) 1 GbE link (Encrypted vmotion traffic) vsphere Source Host Maxwell Appliance vsphere Destination Host #SER1494BU CONFIDENTIAL 24

Long Distance vmotion Performance in Database Environment Workload Open source DVD Store 2 database OLTP workload Performance Metric: orders per second (1-second granularity) Test Scenario VM

25 Long Distance vmotion Performance in Database Environment Workload Open source DVD Store 2 database OLTP workload Performance Metric: orders per second (1-second granularity) Test Scenario VM configuration: 8 VCPUs, 24 GB memory, 40GB Sys disk, 30GB Db disk, Windows Server 2012 MS SQL Server Database size: 12 GB Benchmark load: 5 DVD store users with no think-time CONFIDENTIAL #SER1494BU CONFIDENTIAL 25

26 Long Distance vmotion Performance in Database Environment Total Duration (in seconds) vmotion Duration (with varying RTT, Encrypted traffic ) <1ms 5 ms 25 ms 50 ms 75 ms 100 ms 125ms 150ms Round-trip latency Duration not impacted by round-trip latency when migrating an active VM #SER1494BU CONFIDENTIAL 26

27 Long Distance vmotion Performance in Database Environment Orders Per Second Source Datacenter Long Distance vmotion (150ms RTT latency, Encrypted traffic) Encrypted vmotion over Long distance Time (in seconds) Switch-over Destination Datacenter Minimal impact on guest even when migrating across cross-continental distance! #SER1494BU CONFIDENTIAL 27

28 Agenda 1 Encrypted vmotion 2 vmotion over 100 GbE 3 vmotion Futures #SER1494BU CONFIDENTIAL 28

29 vmotion over 100 GbE Architecture

30 vmotion Performance Timeline v1.0 v4.1 vmotion 1 GbE Introduce vmotion Stream v5.0 v6.0 v6.5 Multi-NIC vmotion Streams New Stream Architecture Encrypted vmotion Stream Architecure 10 GbE 2 x 10 GbE 40 GbE 40 GbE Prepare Transmit Prepare Encrypt Transmit Next? Whats new? 100 GbE vmotion stream Multiple 10 GbE NIC New vmotion stream Encrypted vmotion stream 100 GbE NIC #SER1494BU CONFIDENTIAL 30

31 vmotion Stream Architecture vmotion stream 3 Threads per stream for parallelism vmotion vmkernel nic Default: 1 vmotion stream per vmknic Physical Nic hardware queue VMworld 2017 Content: Not for Prepare Thread Encrypt Thread vmotion single stream vmotion vmkernel nic Physical NIC Transmit Thread Software TX/RX Dispatch queue publication Hardware TX/RX Dispatch queue #SER1494BU CONFIDENTIAL 31

32 vmotion over 100 GbE Performance: Monster VM migrations

33 ESX 6.5 vmotion over 100 GbE vmotion Stream Helper Software RX Dispatch queue Hardware RX Dispatch queue vmotion vmknic vmotion Stack Vmkernel network Stack vmotion Stream Helper Software RX Dispatch queue Hardware RX Dispatch queue vmotion vmknic vmotion Stack Vmkernel network Stack CPU 100% busy 10 Gbe NIC 100 Gbe NIC 33 #SER1494BU CONFIDENTIAL 33

34 ESX 6.5 vmotion over 100 GbE Instantiate more vmotion streams VMworld 2017 vmotion Stream Helpers Software RX Dispatch queue Hardware RX Dispatch queue vmotion vmknic vmotion Stack Vmkernel network Stack Content: Not for publication Rx Queue bottleneck 100 Gbe NIC #SER1494BU CONFIDENTIAL 34

35 6.5 vmotion over 100 GbE Instantiate more vmotion streams Create more vmknics on a 100GbE NIC VMworld 2017 vmotion Stream Helpers vmotion vmknic Hardware RX Dispatch queues vmotion vmknic vmotion Stack Vmkernel network Stack Content: Not for publication 100 Gbe NIC #SER1494BU CONFIDENTIAL 35

ESX 6.5 vmotion Performance over 100GbE NIC Testbed Configuration Hosts: Dell PowerEdge R930 w/ Intel Xeon E7-8890 @ 2.

5 Test Scenario VM configured w/ 12 VCPUs, 1 TB memory, 300GB sys disk, SLES 11 as guest OS Workload: Memhog micro benchmark (extremely memory intensive) Needs nearly

36 ESX 6.5 vmotion Performance over 100GbE NIC Testbed Configuration Hosts: Dell PowerEdge R930 w/ Intel Xeon 2.2 GHz, 96 cores, 4 TB NIC: Mellanox 100GbE ESX 6.5 Test Scenario VM configured w/ 12 VCPUs, 1 TB memory, 300GB sys disk, SLES 11 as guest OS Workload: Memhog micro benchmark (extremely memory intensive) Needs nearly 6 vmknics to reach high throughput (due to current design) Encrypted vmotion scalability limited (as encryption is very compute intensive) or 1 vmknic 2 vmknics 4 vmknics 6 vmknics distribution Peak Network Bandwidth Usage () Encrypted vmotion Peak Network Bandwidth Usage () #SER1494BU CONFIDENTIAL 36

vmotion over 100 GbE (Upcoming Performance Optimizations) Instantiate more vmotion streams Leverage pnic Receive Side Scaling (RSS) VMworld 2017 vmotion Stream Helpers

37 vmotion over 100 GbE (Upcoming Performance Optimizations) Instantiate more vmotion streams Leverage pnic Receive Side Scaling (RSS) VMworld 2017 vmotion Stream Helpers Software RX Dispatch queue Hardware RX Dispatch queues vmotion vmknic vmotion Stack Vmkernel network Stack Content: Not for publication 100 Gbe NIC #SER1494BU CONFIDENTIAL 37

vmotion over 100 GbE (Upcoming Performance Optimizations) VMworld 2017 vmotion Stream Helpers Software RX Dispatch queue Hardware RX Dispatch

38 vmotion over 100 GbE (Upcoming Performance Optimizations) VMworld 2017 vmotion Stream Helpers Software RX Dispatch queue Hardware RX Dispatch queues vmotion vmknic vmotion Stack Vmkernel network Stack Rx Queue bottleneck Content: Not for publication 100 Gbe NIC 38 #SER1494BU CONFIDENTIAL 38

39 vmotion over 100 GbE (Upcoming Performance Optimizations) Use multiple Software Rx Queues 39 VMworld 2017 vmotion Stream Helpers Hardware RX Dispatch queues vmotion vmknic vmotion Stack Vmkernel network Stack Content: Not for publication 100 Gbe NIC #SER1494BU CONFIDENTIAL 39

40 vmotion over 100 GbE (Upcoming Performance Improvements) vmotion Traffic Few Rx Queues With Receive Side Scaling Physical NIC Logic to place packets to Rx Queues Q u e u e Q u e u e Q u e u e CPU utilization per core vmotion Traffic More Rx Queues With Dynamic Receive Side Scaling Q u e u e Physical NIC Logic to place packets to Rx Queues Q u e u e Q u e u e Q u e u e Q u e u e CPU utilization per core Q u e u e Core 1 Core 2 Core 3 Core 4 Core 5 Core N Core 1 Core 2 Core 3 Core 4 Core 5 Core N #SER1494BU CONFIDENTIAL 40

vmotion over 100 GbE (Upcoming Performance Optimizations) Goal: Instantiate optimal # vmotion streams Goal: Create optimal # Software vmknic Rx queues Goal: Leverage RX queues in the pnic 41 VMworld

41 vmotion over 100 GbE (Upcoming Performance Optimizations) Goal: Instantiate optimal # vmotion streams Goal: Create optimal # Software vmknic Rx queues Goal: Leverage RX queues in the pnic 41 VMworld 2017 vmotion Stream Helpers Software RX Dispatch queues Hardware RX Dispatch queues vmotion vmknic vmotion Stack Vmkernel network Stack Content: Not for publication 100 Gbe NIC #SER1494BU CONFIDENTIAL 41

42 vmotion over 100 GbE (Upcoming Performance Improvements) Single vmknic vmotion Peak Throughput over 100 GbE 19 0 Before After #SER1494BU CONFIDENTIAL 42

Monster VM vmotion Performance Testbed Configuration Hosts: Dell PowerEdge R730 w/ Intel Broadwell @ 3

SLES 11 as guest OS Application: open-source Redis 3.0.

43 Monster VM vmotion Performance Testbed Configuration Hosts: Dell PowerEdge R730 w/ Intel 3 GHz, 24 cores, 1 TB NICs: Mellanox 40GbE and 4 * Intel 10 GbE (total 80 GbE bandwidth) Storage: Intel NVMe disks ESX 6.5 (vmotion traffic encrypted) Test Scenario VM configured w/ 24 VCPUs, 512 GB memory, 300GB sys disk, SLES 11 as guest OS Application: open-source Redis (In-memory key/value store) Redis-Benchmark Load: 160 clients Operation: SET 1 byte (with a randomly generated key in a space of 2 billion keys) 8 Redis server instances (serving approx. 2 million requests/sec) #SER1494BU CONFIDENTIAL 43

44 Monster VM vmotion Performance Idle VM 1 redis instances / 20 clients 2 redis instances / 40 clients 4 redis instances / 80 clients 8 redis instances / 160 clients vmotion Peak Network Throughput Usage Gbps Mileage (network bandwidth usage) varies based on workload 30 Gbps peak usage even during Idle VM migration #SER1494BU CONFIDENTIAL 44

45 Dissection of Monster VM Migration Network Bandwidth (Gbps) Time Test Scenario: 24 VCPUs, 512 GB memory, 300GB Sys disk, 8 Redis instances Mileage also varies in different phases of vmotion #SER1494BU CONFIDENTIAL 45

46 Dissection of Monster VM Migration Network Bandwidth (Gbps) Disk Copy Time Network bandwidth usage typically limited by Storage throughput Transmit rate of disk copy increased gradually to avoid overwhelming the destination CONFIDENTIAL #SER1494BU CONFIDENTIAL 46

47 Dissection of Monster VM Migration Network Bandwidth (Gbps) Time Precopy Gbps average network bandwidth usage (limited by memory allocation on destination host) CONFIDENTIAL #SER1494BU CONFIDENTIAL 47

48 Dissection of Monster VM Migration Network Bandwidth (Gbps) Time Bandwidth usage reaching line speed during Precopy-2 Precopy - 2 CONFIDENTIAL #SER1494BU CONFIDENTIAL 48

49 CONFIDENTIAL 49

50 Agenda 1 Encrypted vmotion 2 vmotion over 100 GbE 3 vmotion Futures #SER1494BU CONFIDENTIAL 50

51 vmotion Futures What s next for vmotion?

52 Snowball + vmotion Order snowball Copy & ship snowball Data Transfer to S3 Data Transfer to SDDC vcenter Server ESX Hybrid Linked Mode vmotion L2 VM Network L3 vmotion Network Snowball vcenter Server ESX 5 Cross VC vmotion Snowball AWS S3 #SER1494BU CONFIDENTIAL 52

vmotion Futures Active / Passive Storage Replication Leverage

direction after vmotion Support next generation hardware

(RoCE) Page modification logging (PML) 4Kn disks VM1 Active

53 vmotion Futures Active / Passive Storage Replication Leverage broad partner ecosystem VVol required to reverse replication direction after vmotion Support next generation hardware advancements Non Volatile Memory (NVM) RDMA over Ethernet (RoCE) Page modification logging (PML) 4Kn disks VM1 Active Passive Replication Broad partner ecosystem #SER1494BU CONFIDENTIAL 53

54 Summary vmotion is continually innovating in pace with technology trends and customer requirements Encrypted vmotion : All vmotion network traffic is encrypted Securely cross geographical and management boundaries vmotion Performance Encrypted vmotion : Identical performance as unencrypted vmotion Monster VM vmotion : Achieve near line rate on 100 GbE What s next for vmotion? Snowball + vmotion : Hybrid cloud Mobility between on prem and vsphere on AWS Active/Passive replication : Optimize disk copy Hardware advancements : NVM, 4Kn, RDMA over Ethernet, PML #SER1494BU CONFIDENTIAL 54

55 Extreme Performance Series Las Vegas SER2724BU Performance Best Practices SER2723BU Benchmarking 101 SER2343BU vsphere Compute & Memory Schedulers SER1504BU vcenter Performance Deep Dive SER2734BU Byte Addressable Non-Volatile Memory in vsphere SER2849BU Predictive DRS Performance & Best Practices SER1494BU Encrypted vmotion Architecture, Performance, & Futures STO1515BU vsan Performance Troubleshooting VIRT1445BU Fast Virtualized Hadoop and Spark on All-Flash Disks VIRT1397BU Optimize & Increase Performance Using VMware NSX VIRT2550BU Reducing Latency in Enterprise Applications with VMware NSX VIRT1052BU Monster VM Database Performance VIRT1983BU Cycle Stealing from the VDI Estate for Financial Modeling VIRT1997BU Machine Learning and Deep Learning on VMware vsphere FUT2020BU Wringing Max Perf from vsphere for Extremely Demanding Workloads FUT2761BU Sharing High Performance Interconnects across Multiple VMs #SER1494BU CONFIDENTIAL 55

56 Extreme Performance Series - Hand on Labs Don t miss these popular Extreme Performance labs: HOL SDC: vsphere 6.5 Performance Diagnostics & Benchmarking Each module dives deep into vsphere performance best practices, diagnostics, and optimizations using various interfaces and benchmarking tools. HOL CHG: vsphere Challenge Lab Each module places you in a different fictional scenario to fix common vsphere operational and performance problems. #SER1494BU CONFIDENTIAL 56

Performance Survey The VMware Performance Engineering team is always looking for feedback about your experience with the performance of our products, our various tools, interfaces and where we can

57 Performance Survey The VMware Performance Engineering team is always looking for feedback about your experience with the performance of our products, our various tools, interfaces and where we can improve. Scan this QR code to access a short survey and provide us direct feedback. Alternatively: Thank you! VMworld 2017 Content: Not for publication #SER1494BU CONFIDENTIAL 57

Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitme

Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitme SER2734BU Extreme Performance Series: Byte-Addressable Nonvolatile Memory in vsphere VMworld 2017 Content: Not for publication Qasim Ali and Praveen Yedlapalli #VMworld #SER2734BU Disclaimer This presentation