Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitme

Size: px

Start display at page:

Download "Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitme"

Crystal Patrick
6 years ago
Views:

1 VIRT1997BU Machine Learning on VMware vsphere with NVIDIA s #VMworld #VIRT1997BU

2 Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitment from VMware to deliver these features in any generally available product. Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery. Pricing and packaging for any new technologies or features discussed or presented have not been determined. CONFIDENTIAL 2

3 VIRT1997BU Machine Learning on VMware vsphere with NVIDIA s Uday Kurkure, VMware Performance Engineering Ziv Kalmanovich, VMware vsphere Product Management #VMworld #SER3052BU

4 Overview Defining Machine Learning Accelerator Infrastructure Topologies Benchmarks DirectPath IO vs Mediated Passthrough Scalability Mixed Workloads VMware Vision for ML/DL Infrastructure on vsphere CONFIDENTIAL 4

5 Universal App Platform Test / Dev / Tier 2/3 Business-Critical Apps Desktop Virtualization 3D Graphics Deep Learning with VMworld 2017 Content: Not for Big Data SAP HANA vsphere Integrated Containers Cloud-Native Applications publication 5

6 Machine Learning, Industry Perspective Financial Biometrics Messaging Calendars Social Media Medical Big Data VMworld 2017 Machine Learning Deep Learning HPC High Performance Computing Thermodynamics 3D Modeling Military Chemistry Content: Not for publication E-Commerce Automotive Sensors Phone Network Biology Financial

7 Machine Learning, Application Perspective Big Data VMworld 2017 Machine Learning Deep Learning HPC High Performance Computing Content: Not for publication

8 Machine Learning, Infrastructure Perspective FPGA NVM Distributed CPU Graphcore IPU TM Big Data VMworld 2017 Machine Learning Deep Learning HPC High Performance Computing FPGA RoCE Content: Not for publication Inifiniband Graphcore IPU TM Google TPU TM Google TPU TM

9 The End of The One-Size Fits All VM NIC SRIOV Simulation Engineer SRIOV IB v Machine Learning Expert Data Scientist NVM Developer Intel QAT FPG A SRIO V NIC App Admin IB Websites Dev & Test, Staging LoB Applications Small to mid-size DB SAP, Oracle,SharePoint Cluster computing Enterprise applications vcpu Mem vnic VMFS or VSAN VI Admin

Virtual, Virtualized, Virtualized VM Virtual Shared Acceleration

Device Virtualization VM HW Virtualization aka Mediated

F IHV Host Driver xpu X86 Virtualization, no device

10 Virtual, Virtualized, Virtualized VM Virtual Shared Acceleration ESXi VMW Guest Driver VMW Virtual Device IHV Host Driver xpu PCIe Device Virtualization VM HW Virtualization aka Mediated Passthrough NVIDIA GRID,SRIOV ESXi Native Vendor Device Driver V F IHV Host Driver xpu X86 Virtualization, no device virtualization VM Direct Path I/O (Passthrough) ESXi Native Vendor Device Driver xpu 10

11 Compute on vsphere with DirectPath IO Benefits VM vsphere VM level QoS Workload Isolation Reproducibility vsphere CONFIDENTIAL HW Isolation 13

12 Compute on vsphere Mediated Passthrough Benefits VM vsphere Workload Isolation Reproducibility VM level QoS VF VF VF VF VF VF VF VF VF VF VF VF VF Accelerator level QoS Vendor HW Virtualization Driver VF CONFIDENTIAL HW Isolation Accelerator Sharing 14

13 Compute on vsphere with DirectPath IO Machine Learning CUDA Developer VM vsphere CUDA Developer 1:1 Data Scientist Data Scientist VMworld 2017 Content: Not for vsphere X:1 publication CONFIDENTIAL 15

14 Compute on vsphere with DirectPath IO HPC and Big Data vsphere Worker Worker Worker VMworld 2017 Content: Not for vsphere Researchers Master node Worker publication Worker CONFIDENTIAL 16

vsphere with Mediated Pass-through SRIOV or NVIDIA GRID CUDA Developer VM v CUDA Developer v Data Scientist v VMware vsphere VDI User v CUDA Developer VM v CUDA Developer v v v v v VDI User v Data

15 vsphere with Mediated Pass-through SRIOV or NVIDIA GRID CUDA Developer VM v CUDA Developer v Data Scientist v VMware vsphere VDI User v CUDA Developer VM v CUDA Developer v v v v v VDI User v Data Scientist VMworld 2017 Content: Not for v v v v v v v Vendor HW Virtualization Driver publication v v The information in this presentation is intended to outline our general product direction and should not be relied on in making a purchasing decision. It is for informational purposes only and may not be incorporated into any contract. 17

16 Exploring Machine Learning Workload Performance Scalability Performance Mixed Workloads Performance: Native vs. Virtual, vs. CPU Scalability: scale # of VMs and scale # of s Mixed Workloads: ML, 3D workloads 18

17 Workloads, ML Framework Scalability Performance Mixed Workloads 19

18 ML Framework, Workloads and Neural Networks ML Framework: Google s TensorFlow Nvidia CUDA 7.5, Nvidia cudnn Different Workloads : Language Modeling on Penn Tree Bank Handwriting Recognition with MNIST Image Classifier with CIFAR-10 2 Different Neural Network Architectures: Convolutional Neural Network Recurrent Neural Network 20

19 Performance: Native vs Virtual Scalability Performance Mixed Workloads 21

Testbed Configurations for Native vs Virtual Comparison VMware Test-bed for NVIDIA GRID on Horizon View Native Configuration TensorFlow 0.10 CentOS 7.

20 Testbed Configurations for Native vs Virtual Comparison VMware Test-bed for NVIDIA GRID on Horizon View Native Configuration TensorFlow 0.10 CentOS 7.2 VMworld 2017 Content: Not for Virtual Configuration TensorFlow 0.10 CentOS 7.2 ESX 6.X publication Dell R730 Intel Haswell CPUs + 2 x NVidia GRID M60 24 cores (2 x 12-core socket) E V3 768 GB RAM Dell R730 Intel Haswell CPUs + 2 x NVidia GRID M60 24 cores (2 x 12-core socket) E V3 768 GB GB RAM 22

21 Workload for Native vs Virtualized Workload Complex Language Modelling Given history of words, predicts next word Neural Network Type: Recurrent Neural Network Large Model 1500 LSTM units /layer Penn Tree Bank (PTB) Database: 929K training words 73K validation words 82K test words 10K vocabulary 23

22 Performance: Training Times on native vs virtualized 4% of overhead for both GRID v & DirectPath I/O compared to native Normalized Training Times Lower is better Language Modelling with RNN on PTB Native GRID v DirectPathIO 24

23 Performance: s vs CPUs 25

24 Performance: vs. CPU on a virtualized server Two VM Configurations: 1 VM with 1 v (M60-8q) vs 1 VM without a Each VM has 12 vcpus, 60GB memory, 96GB of SSD storage, CentOS 7.2 Workload : Handwriting Recognition Dataset: MNIST database of handwritten digits Training set: 60,000 examples Test set: 10,000 examples Neural Net: CNN Workload: Language Modelling on PTB dataset using RNN 26

25 Training Times with vs. without on a virtualized server Normalized Training Time Lower is better Handwriting Recognition with CNN on MNIST VMworld 2017 Normalized Training Time Lower is better Language Modeling with RNN on PTB Content: Not for publication v No 0 1 v No 27

26 Performance: DirectPath IO vs GRID v Scalability Performance Mixed Workloads 28

27 Performance: DirectPath I/O and GRID v Normalized Training Times Lower is better Normalized training times for Handwriting Recognition & Language Modelling DirectPath I/O GRID v Handwriting CNN with Recognition MNIST (CNN on MNIST) Language RNN with Modelling PTB (RNN on PTB) 29

28 Scalability- VMs Scalability Performance Mixed Workloads 30

29 Scalability: Scaling number of VMs/server The number of VMs with ML workload per server: limited by #of vs supported by One VM with one v Four VMs with one v each

30 Scalability: Image Classifier(CIFAR-10) Workload Workload: Image Classifier(CIFAR-10): 10 classes 60K images 50K training and 10K Test images Convolutional Neural Network ~ One Million learning parameters ~19 million multiply-add to compute inference on a single image 32

Scalability VMs Workload: Image Classifier (CNN on CIFAR-10) Performance Scales almost linearly with number of VMs Performance of DirectPath I/O and GRID v are

31 Scalability VMs Workload: Image Classifier (CNN on CIFAR-10) Performance Scales almost linearly with number of VMs Performance of DirectPath I/O and GRID v are comparable Normalized Images Processed / second Higher is better Normalized Images Processed / second DirectPath I/O 0.9 GRID v VM 4 VMs

32 Scalability - s Scalability Performance Mixed Workloads 34

33 Scalability (DirectPath IO) How a ML application performs as increasing number of s up to maximum available on the server VM with 1 VM with 2 s VM VM VM with 4 s VM 35

34 Scalability (DirectPath I/O) Workload: Image Classifier (CNN on CIFAR-10) Scale almost linearly with number of s Only DirectPath I/O supports multiple s per VM Normalized Images Processed per second Higher is better Normalized images processed /sec with CNN on CIFAR s 4 s

35 Resource Utilization 37

36 CPU Utilization for Machine Learning Workloads with s CPU Utilization (%) Lower is better Handwriting Recognition with CNN on MNIST 100% 80% 60% 40% 20% 0% 3% 78% VMworld v No CPU Utilization (%) Lower is better Language Modelling with RNN on PTB 50% 40% 30% Content: Not for publication 20% 10% 0% 8% 43% 1 v No Off-loading Compute to : CPU Resources become available 38

37 Benefits of Off-loading Compute to s More CPU resources are available Mix and non-gpu workloads on the same server Have more users / VMs per server

38 Mixed Workloads (Graphics+ML) Scalability Performance Mixed Workloads 40

39 vsphere with Mediated Pass-through NVIDIA GRID CUDA Developer VM v CUDA Developer v Data Scientist v VMware vsphere VDI User v CUDA Developer VM v CUDA Developer v v v v v VDI User v Data Scientist VMworld 2017 Content: Not for v v v v v v v Vendor HW Virtualization Driver publication v v The information in this presentation is intended to outline our general product direction and should not be relied on in making a purchasing decision. It is for informational purposes only and may not be incorporated into any contract. 41

40 Nvidia GRID v Profiles for Maxwell and Pascal virtual type physical board GRAPHICS CUDA maximum virtual s per physical GRID M60-0q Tesla M60 yes no 16 GRID M60-1q Tesla M60 yes no 8 GRID M60-2q Tesla M60 yes no 4 GRID M60-4q Tesla M60 yes no 2 GRID M60-8q Tesla M60 yes yes 1 GRID P40-1q Tesla P40 yes yes 24 GRID P40-2q Tesla P40 yes yes 12 GRID P40-3q Tesla P40 yes yes 8 GRID P40-4q Tesla P40 yes yes 6 GRID P40-6q Tesla P40 yes yes 4 GRID P40-8q Tesla P40 yes yes 3 GRID P40-12q Tesla P40 yes yes 2 GRID P40-24q Tesla P40 yes yes 1 42

Mixed Workloads Goal: Quantify Performance and Resource

Recognition (mnist) 3D-CAD: SPECapc for 3ds Max 2015 CentOS

10 12vCPU, 16 GB RAM, 96GB HD Content: Not for publication

41 Mixed Workloads Goal: Quantify Performance and Resource Utilization of running 3D CAD & ML on the same server concurrently. VMworld 2017 ML on M60-8Q Benchmarks: ML: Handwriting Recognition (mnist) 3D-CAD: SPECapc for 3ds Max 2015 CentOS 7.2, TensorFlow vCPU, 16 GB RAM, 96GB HD Content: Not for publication Dell R730 Intel Haswell CPUs + 2 x NVidia GRID M60 24 cores (2 x 12-core socket) E V3 768 GB GB RAM Win 7, x64, 4 vcpu, 16 GB RAM, 120 GB HD Autodesk 3ds Max

Mixed Workloads Experiment Design 1) First set of runs : Run 3D-CAD Workload ONLY (SPECapc) 2) Second set of runs: Run ML workload ONLY (MNIST) 3) Third set of runs: Run 3D-CAD and ML Concurrently

42 Mixed Workloads Experiment Design 1) First set of runs : Run 3D-CAD Workload ONLY (SPECapc) 2) Second set of runs: Run ML workload ONLY (MNIST) 3) Third set of runs: Run 3D-CAD and ML Concurrently Perf. Metrics: 3D-CAD (SPECapc): Geo. Mean of run-times ML (Handwriting Recognition): Training Times VMworld 2017 Note: 3D-CAD (SPECapc) runs on all v profiles ML runs only on M60-8Q profile Content: Not for publication 44

43 Mixed Workloads - Results 3D-CAD (SPEC) < 5% penalty in M60-2Q, M60-4Q, M60-8Q ML (MNIST) < 15% in M60-2Q, M60-4Q, M60-8Q % Increase [Lower is better] % Increase in run-time for Mixed Workloads 4.86 SPEC 3.4 MNIST M60-2Q (13 VMs) (7 VMs) M60-4Q M60-8Q (4 VMs) (12 CAD VMs+ 1 ML VM) (6 CAD VMs+ 1 ML VM) (3 CAD VMs+ 1 ML VM) v Profile 45

44 Mixed Workloads - Results 3D-CAD (SPEC) < 5% penalty in M60-2Q, M60-4Q, M60-8Q ML (MNIST) < 15% in M60-2Q, M60-4Q, M60-8Q % Increase [Lower is better] % Increase in run-time for Mixed Workloads M60-1Q (25 VMs) M60-2Q (13 VMs) M60-4Q (7 VMs) M60-8Q (4 VMs) (24 CAD VMs+ 1 ML VM) v Profile SPEC MNIST 46

45 Mixed Workloads Results: Concurrent is always better. Normalized Execution time Concurrent Sequential (13 Vms) M60-1Q M60-2Q M60-4Q M60-8Q v Profile (25 VMs) (7 VMs) (4 VMs) 47

46 Advantages of Running Mixed Workloads Concurrent Running times are always shorter than sequential times. Better Resource Utilization Higher Consolidation Ratios

47 Key Performance Takeaways Virtualized s deliver near bare metal Performance for ML workloads s can be used in two modes on vsphere: Direct Path IO and NVidia GRID v For CUDA/ML workloads, GRID V requires 1 v per VM For Multi- ML workloads, use Direct Path IO mode For more consolidation of -based workloads, use GRID v GRID v combines performance of s and datacenter management benefits of VMware vsphere 49

48 Vision - Extend vsphere Benefits to DirectPath IO (passthrough) and SR-IOV Capable Accelerators Suspend &Resume vsphere Snapshots Shared Resources vmotion DRS CPU Mem xpu xpu xpu vsphere HA DRS Placement xp U VF xpu VF xp U VF Roadmap CONFIDENTIAL The information in this presentation is intended to outline our general product direction and should not be relied on in making a purchasing decision. It is for informational purposes only and may not be incorporated into any contract. 50

Roadmap Vision Extend all vsphere Benefits to NVIDIA GRID v Suspend &Resume vsphere CPU Snapshots Shared Resources Mem vmotion DRS v v v v NVIDIA GRID tm See @booths Cloud Platform - New Workloads

49 Roadmap Vision Extend all vsphere Benefits to NVIDIA GRID v Suspend &Resume vsphere CPU Snapshots Shared Resources Mem vmotion DRS v v v v NVIDIA GRID tm Cloud Platform - New Workloads EUC 3D Experience NVIDIA GRID The information in this presentation is intended to outline our general product direction and should not be relied on in making a purchasing decision. It is for informational purposes only and may not be incorporated into any contract. 51

Overview - Considered Milestones for HW Accelerators in vsphere Virtual PC Suspend&Resume For NVIDIA GRID Tech Preview See @booths VMW Cloud Platform - New Workloads Snapshots for NVIDIA GRID Roadmap

and SRIOV VFs Roadmap Machine Learning and Analytics vsphere DRS for DirectPath IO and SRIOV VFs Roadmap VMW EUC 3D Experience NVIDIA GRID VMW EUC 3D Experience NVIDIA GRID The information in this

50 Overview - Considered Milestones for HW Accelerators in vsphere Virtual PC Suspend&Resume For NVIDIA GRID Tech Preview VMW Cloud Platform - New Workloads Snapshots for NVIDIA GRID Roadmap Virtual Workstation vsphere vmotion for NVIDIA GRID Roadmap VMW Cloud Platform - New Workloads High Performance Computing vsphere DRS for NVIDIA GRID Roadmap vsphere HA for DirectPath IO and SRIOV VFs Roadmap Machine Learning and Analytics vsphere DRS for DirectPath IO and SRIOV VFs Roadmap VMW EUC 3D Experience NVIDIA GRID VMW EUC 3D Experience NVIDIA GRID The information in this presentation is intended to outline our general product direction and should not be relied on in making a purchasing decision. It is for informational purposes only and may not be incorporated into any contract. 52

51 Introducing vsphere Scale-Out for Big Data and HPC Workloads New package that provides all the core features required for scale-out workloads at an attractive price point Features Packaging Hypervisor, vmotion, vshield Endpoint, Storage vmotion, Storage APIs, Distributed Switch, I/O Controls & SR-IOV, Host Profiles / Auto Deploy and more Sold in Packs of 8 CPU at a cost-effective price point Licensing EULA enforced for use w/ Big Data/HPC/ML workloads only 53

Value of vsphere Scale-Out for Big Data and HPC Flexibility & Agility Infrastructure on demand Iterate faster Scale out more rapidly Multi-tenancy enables different

operations using tools that IT is familiar with Live workload mobility for Master nodes Reference architecture and best practices Data Governance and Control of

52 Value of vsphere Scale-Out for Big Data and HPC Flexibility & Agility Infrastructure on demand Iterate faster Scale out more rapidly Multi-tenancy enables different multiple distros on the same set of server Operational Efficiency CapEx and OpEx Saving Cluster Consolidation Increase Server Utilization Reduced Complexity Simple operations using tools that IT is familiar with Live workload mobility for Master nodes Reference architecture and best practices Data Governance and Control of Sensitive Data Host and VM security for your customer data Security isolation Hypervisor Guests have low privileges by default Faster time to results and insights at a lower cost 54

53 Key Takeaways Gain vsphere s operational benefits with near bare-metal performance (95%) for accelerated HPC, Big Data and Machine Learning VMware's vision is seamless integration of s and other HW accelerators as native resources of VM infrastructure New vsphere Scale-Out SKU; new package with attractive price point for Big Data/HPC/ML dedicated infrastructure virtualization. VMworld 2017 Content: Not for publication Using DirectPath IO for compute doesn t require device specific certification, more here and

54 Extreme Performance Series Las Vegas SER2724BU Performance Best Practices SER2723BU Benchmarking 101 SER2343BU vsphere Compute & Memory Schedulers SER1504BU vcenter Performance Deep Dive SER2734BU Byte Addressable Non-Volatile Memory in vsphere SER2849BU Predictive DRS Performance & Best Practices SER1494BU Encrypted vmotion Architecture, Performance, & Futures STO1515BU vsan Performance Troubleshooting VIRT1445BU Fast Virtualized Hadoop and Spark on All-Flash Disks VIRT1397BU Optimize & Increase Performance Using VMware NSX VIRT2550BU Reducing Latency in Enterprise Applications with VMware NSX VIRT1052BU Monster VM Database Performance VIRT1983BU Cycle Stealing from the VDI Estate for Financial Modeling VIRT1997BU Machine Learning and Deep Learning on VMware vsphere FUT2020BU Wringing Max Perf from vsphere for Extremely Demanding Workloads FUT2761BU Sharing High Performance Interconnects across Multiple VMs CONFIDENTIAL 56

55 Extreme Performance Series Barcelona SER2724BE Performance Best Practices SER2343BE vsphere Compute & Memory Schedulers SER1504BE vcenter Performance Deep Dive SER2849BE Predictive DRS Performance & Best Practices VIRT1445BE Fast Virtualized Hadoop and Spark on All-Flash Disks VMworld 2017 VIRT1397BE Optimize & Increase Performance Using VMware NSX VIRT1052BE Monster VM Database Performance FUT2020BE Wringing Max Perf from vsphere for Extremely Demanding Workloads Content: Not for publication CONFIDENTIAL 57

56 Extreme Performance Series - Hand on Labs Don t miss these popular Extreme Performance labs: HOL SDC: vsphere 6.5 Performance Diagnostics & Benchmarking Each module dives deep into vsphere performance best practices, diagnostics, and optimizations using various interfaces and benchmarking tools. HOL CHG: vsphere Challenge Lab Each module places you in a different fictional scenario to fix common vsphere operational and performance problems. CONFIDENTIAL 58

Performance Survey The VMware Performance Engineering team is always looking for feedback about your experience with the performance of our products, our various tools, interfaces and where we

57 Performance Survey The VMware Performance Engineering team is always looking for feedback about your experience with the performance of our products, our various tools, interfaces and where we can improve. Scan this QR code to access a short survey and provide us direct feedback. Alternatively: Thank you! VMworld 2017 Content: Not for publication CONFIDENTIAL 59

Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitme

Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitme SER3052BU How VMware vsphere and NVIDIA s Accelerate Your Organization Raj Rao, NVIDIA GRID Product Management Ziv Kalmanovich, vsphere ESXi Product Management #VMworld #SER3052BU Disclaimer This presentation