Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitme

Size: px

Start display at page:

Download "Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitme"

Scott Gibbs
6 years ago
Views:

1 VIRT1983BU Making the Complicated Simple: Cycle Harvesting from the Virtual Desktop Infrastructure Estate for Financial Modeling and Simulation VMworld 2017 Content: Not for publication #VMworld #VIRT1983BU

2 Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitment from VMware to deliver these features in any generally available product. Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery. Pricing and packaging for any new technologies or features discussed or presented have not been determined. CONFIDENTIAL 2

3 VIRT1983BU Making the Complicated Simple: Cycle Harvesting from the Virtual Desktop Infrastructure Estate for Financial Modeling and Simulation Wayne Longcore, VP IT Client Services, Jackson National Life Josh Simons, Chief Technologist for HPC, VMware #VMworld #VIRT1983BU

4 Agenda 1 Virtualized HPC 2 JNL Environment 3 Cycle Harvesting at JNL 4 Performance 5 Benefits 6 Futures #VIRT1983BU CONFIDENTIAL 4

5 Virtualized HPC Science, Research, Engineering, & Financial Applications on vsphere VMworld 2017 Content: Not for publication

6 Our Goal and Approach Increase agility and decrease time to discovery for researchers, scientists, and engineers Provide IT with the ability to efficiently provision, allocate, manage and ensure compliance of research compute infrastructure across an increasingly broad range of technical and business requirements By leveraging VMware s proven, enterprise-class virtualization and cloud technologies to meet the performance requirements of research computing and HPC workloads, and Bringing novel capabilities to bear to enable new capabilities not available in traditional HPC environments #VIRT1983BU CONFIDENTIAL 6

HPC Workloads Scientific or technical workloads Often floating-point intensive Often storage intensive Often parallel Throughput Jobs VMworld 2017 Content: Not for HPC Cluster Mechanical

7 HPC Workloads Scientific or technical workloads Often floating-point intensive Often storage intensive Often parallel Throughput Jobs VMworld 2017 Content: Not for HPC Cluster Mechanical Design/Drafting Chemical Engineering Economics/Financial Weather Electronic Design Automation (EDA) Geosciences Defense Computer-Aided Engineering (CAE) Bioscience Government Lab University/Academic publication MPI Jobs Message Passing Interface #VIRT1983BU CONFIDENTIAL 7

8 Virtual Machine Benefits for HPC hardware App hypervisor hardware OS VM Virtual Machines offer: Heterogeneity Multi-tenant data security Fault isolation Reproducibility Fault resiliency Dynamic load balancing Performance #VIRT1983BU CONFIDENTIAL 8

9 Virtualized HPC Performance Representative throughput and MPI examples (performance ratios higher is better) Performance Ratio Run 1 Run 2 Run 3 Run 4 performance Ratios Monte Carlo simulation Science & engineering applications FLUENT GROMACS LAMMPS LS-DYNA NAMD OpenFOAM #VIRT1983BU CONFIDENTIAL 9

10 Introducing vsphere Scale-Out for Big Data and HPC Workloads New package that provides all the core features required for scale-out workloads at an attractive price point Features Packaging Hypervisor, vmotion, vshield Endpoint, Storage vmotion, Storage APIs, Big Data Extensions, Distributed Switch, I/O Controls & SR-IOV, Host Profiles / Auto Deploy and more Sold in Packs of 8 CPU at a cost-effective price point Licensing EULA enforced for use w/ Big Data/HPC workloads only 10

11 Cycle Harvesting Long history of harvesting spare cycles in HPC and more broadly Performance On average, about 980 TFLOPs 104,000 active users 156,000 active hosts #VIRT1983BU CONFIDENTIAL 11

Cycle Harvesting Berkeley Open Infrastructure for Network Computing (BOINC) 100 projects

University of Wisconsin Well-established HPC distributed resource manager and

adding software infrastructure and complexity to achieve cycle harvesting Is there another

12 Cycle Harvesting Berkeley Open Infrastructure for Network Computing (BOINC) 100 projects On average, about 110 PetaFLOPs 203,000 active users 1,109,000 active hosts HTCondor University of Wisconsin Well-established HPC distributed resource manager and opportunistic scheduler Support for VMware virtual machines These approaches require adding software infrastructure and complexity to achieve cycle harvesting Is there another way for VMware customers? Stats: BOINC: #VIRT1983BU CONFIDENTIAL 12

13 Jackson National Life Environment Making the complicated awesomely simple

14 Architectural Imperatives Measure everything based on user experience Stay current (Win10 & Office 365 Pro Plus, etc.) Focus on user productivity (uptime & responsiveness) Shared Nothing Architecture Reduce fault domains to fewer than 30 users Increase throughput via parallelization High core count for peak usage Subpar performance is unacceptable even 1% of the day Do not believe in expensive Overstuffed Rackmounts with limited cores and overcommitted buses Extremely limited memory overcommit, even in a failover situation Spread departments and functions across data centers Re-leveling Scripts based on sizing factors that match our chargeback factors Charge based on size, not based on use (like a laptop) #VIRT1983BU CONFIDENTIAL 14

15 User Specific Virtual Desktops Low Cost High Performance Use low cost components with no bottlenecks to memory and disk $255/year AVG Per VDI for hardware, including all storage & GPUs (a laptop per year?) Differentiated service levels Treat each Virtual Desktop like a different laptop Model Bronze Desktop Silver Desktop Gold Desktop Platinum Desktop Diamond Desktop Concierge Desktop I 2017 Description of Config 2 cores, 5 GB RAM, Windows 10, 100Gig HD, 10,000-15,000 IOPs 2 cores, 8 GB RAM, Windows 10, 100Gig HD, 10,000-15,000 IOPs 4 cores, 8 GB RAM, Windows 10, 128Gig HD, 10,000-15,000 IOPs 6 cores, 10GB RAM, GPU, Windows 10, Gig HD, 10,000-15,000 IOPs 8 cores, 12GB RAM, GPU, Windows 10, 100+ Gig HD, 40,000-70,000 IOPs 8 Cores, 16GB RAM, GPU, Windows 10, 100+ Gig HD, 40, ,000 IOPs Concierge Desktop II Concierge Desktop III Concierge Desktop IV 14 Cores, 32GB RAM, GPU, Windows 10, 100+ Gig HD, 40, ,000 IOPs 14 Cores, 64GB RAM, GPU, Windows 10, 100+ Gig HD, 40, ,000 IOPs 28 Cores, 96GB RAM, GPU, Windows 10, 100+ Gig HD, 40, ,000 IOPs #VIRT1983BU CONFIDENTIAL 15

16 Blade Design NFS VMDK exposed only via NFS mount vsphere Replication to SSD #2 on host 6001c SSD 1: 1.6 TB single VMFS partition VMworld 2017 NFS VMDK exposed only via NFS mount SSD 2: 1.6 TB single VMFS partition Blade: 2 CPU, each from 8 to 14 cores Hostname: 6001a vsphere Replication from SSD #1 on host 6001c Content: Not for publication #VIRT1983BU CONFIDENTIAL 16

17 VDI Design vcpu Allocation VDI VMs NFS VM Cores CPUs Various Sizes, up to 5:1 CPU oversubscription 2 vcpu Typical Blade: 2 CPU, 8 14 cores each #VIRT1983BU CONFIDENTIAL 17

Data Center Design: Active Active vsphere 6.0 Update 2, transitioning to vsphere 6.

18 Data Center Design: Active Active vsphere 6.0 Update 2, transitioning to vsphere Blades Data Center A 13 Enclosures + 16 Rack Mounts VMworld 2017 Multiple 10GB Fiber Links Identical to Data Center A Content: Not for publication Data Center C #VIRT1983BU CONFIDENTIAL 18

19 Environmental Summary 444 Hosts being cycle harvested 10,468 Cores 71TB of RAM being used by cycle harvesters in 155TB physical RAM Total capacity of 71.6 Million IOs Per Second (4K RW 50%) Total throughput of 4.022Tb/s to disk 6768 VDI desktops 280TB of SSD being used out of 2007TB physical local SSD 888 Cycle harvesting desktop VMs #VIRT1983BU CONFIDENTIAL 19

20 Cycle Harvesting Harvesting unused VDI compute cycles for Risk Analysis

21 Cycle Harvesting Virtual Machine Overlay Cycle Harvesting VMs VDI VMs NFS VM Cores CPUs Equal to # cores in a CPU Various Sizes, up to 5:1 CPU oversubscription 2 vcpu Equal to # cores in a CPU Typical Blade: 2 CPU, 8 14 cores each #VIRT1983BU CONFIDENTIAL 21

Cycle Harvesting VM Configuration Virtual CPUs Virtual RAM Virtual Disk Equal to number of cores per socket 7 GB RAM per core 30 GB per core of scratch 60 GB for Boot Boot Disk Drive C:/ MG Alfa,

22 Cycle Harvesting VM Configuration Virtual CPUs Virtual RAM Virtual Disk Equal to number of cores per socket 7 GB RAM per core 30 GB per core of scratch 60 GB for Boot Boot Disk Drive C:/ MG Alfa, etc. Windows 10 Drive D:/ MSFT DFS for job data Scratch Disk: GB NFS mount (eventually will be VMDK on local SSD s VMFS) VMDK - sits directly on local SSD s VMFS #VIRT1983BU CONFIDENTIAL 22

23 Cycle Harvesting Virtual Disk Overlay NFS VMDK exposed only via NFS mount vsphere Replication to SSD #2 on host 6001c Not replicated Cycle harvesting VM 1 VMDK SSD 1: 1.6 TB single VMFS partition VMworld 2017 NFS VMDK exposed only via NFS mount Blade: 2 CPU, each from 8 to 14 cores Hostname: 6001a vsphere Replication from SSD #1 on host 6001c Not replicated Cycle harvesting VM 2 VMDK Content: Not for publication SSD 2: 1.6 TB single VMFS partition #VIRT1983BU CONFIDENTIAL 23

24 Cycle Harvesting Job Submission Windows HPC Server HPC Job Slots Cycle Harvesting VMs VDI VMs NFS VM Shares set to 100 per vcpu Shares set to 1000 per vcpu Shares set to 1000 per vcpu 2 vcpu Shares set to 100 per vcpu Cores #VIRT1983BU CONFIDENTIAL 24

25 Performance Analysis Representative performance details from a single host

26 VDI VM Activity VDI VM CPU %USED #VIRT1983BU CONFIDENTIAL 26

27 VDI and Harvester CPU Utilization #VIRT1983BU CONFIDENTIAL 27

28 All VMs on a 28-core Host Peak utilization = 100% * (28 cores) * (1.25 HT factor) = 3500 #VIRT1983BU CONFIDENTIAL 28

29 Harvesters and VDI VMs #VIRT1983BU CONFIDENTIAL 29

30 Harvesters and VDI VMs Detail #VIRT1983BU CONFIDENTIAL 30

31 A Further Note on Licensing vsphere Desktop, the vsphere edition that underpins VMware Horizon, is restricted by EULA to run desktop VMs Jackson harvester VMs satisfy this requirement by Running a desktop OS (Windows 10) Running the Horizon View Agent Being tied to users within Jackson s Named User limit #VIRT1983BU CONFIDENTIAL 31

32 Benefits

33 Primary Benefits Cost savings due to avoided cloud costs No new skills required to administer solution Cycle harvesting is transparent to both VDI and HPC users Increased virtual infrastructure reliability Better VDI experience for desktop users Enables end-users (actuaries) to think beyond just batch scheduling #VIRT1983BU CONFIDENTIAL 33

34 Futures

35 Future Directions Harvesting GPU compute cycles Testing OpenCL-based approaches 1M stream processors online by Q K Xeon core-equivalent capacity #VIRT1983BU CONFIDENTIAL 35

36 Extreme Performance Series Las Vegas SER2724BU Performance Best Practices SER2723BU Benchmarking 101 SER2343BU vsphere Compute & Memory Schedulers SER1504BU vcenter Performance Deep Dive SER2734BU Byte Addressable Non-Volatile Memory in vsphere SER2849BU Predictive DRS Performance & Best Practices SER1494BU Encrypted vmotion Architecture, Performance, & Futures STO1515BU vsan Performance Troubleshooting VIRT1445BU Fast Virtualized Hadoop and Spark on All-Flash Disks VIRT1397BU Optimize & Increase Performance Using VMware NSX VIRT2550BU Reducing Latency in Enterprise Applications with VMware NSX VIRT1052BU Monster VM Database Performance VIRT1983BU Cycle Harvesting from the VDI Estate for Financial Modeling VIRT1997BU Machine Learning and Deep Learning on VMware vsphere FUT2020BU Wringing Max Perf from vsphere for Extremely Demanding Workloads FUT2761BU Sharing High Performance Interconnects across Multiple VMs #VIRT1983BU CONFIDENTIAL 36

37 Extreme Performance Series Hand on Labs Don t miss these popular Extreme Performance labs: HOL SDC: vsphere 6.5 Performance Diagnostics & Benchmarking Each module dives deep into vsphere performance best practices, diagnostics, and optimizations using various interfaces and benchmarking tools. HOL CHG: vsphere Challenge Lab Each module places you in a different fictional scenario to fix common vsphere operational and performance problems. #VIRT1983BU CONFIDENTIAL 37

Performance Survey The VMware Performance Engineering team is always looking for feedback about your experience with the performance of our products, our various tools, interfaces and where we can

38 Performance Survey The VMware Performance Engineering team is always looking for feedback about your experience with the performance of our products, our various tools, interfaces and where we can improve. Scan this QR code to access a short survey and provide us direct feedback. Alternatively: Thank you! VMworld 2017 Content: Not for publication #VIRT1983BU CONFIDENTIAL 38

Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitme

STO1515BU Extreme Performance Series: vsan Performance Troubleshooting Amitabha Banerjee and Suraj Kasi #VMworld #STO1515BU Disclaimer This presentation may contain product features that are currently