MVAPICH2-GDR: Pushing the Frontier of HPC and Deep Learning

Transcription

1 MVAPICH2-GDR: Pushing the Frontier of HPC and Deep Learning Talk at Mellanox booth (SC 218) by Dhabaleswar K. (DK) Panda The Ohio State University

2 Outline Overview of the MVAPICH2 Project MVAPICH2-GPU with GPUDirect-RDMA (GDR) What s new with MVAPICH2-GDR High-Performance Deep Learning (HiDL) with MVAPICH2-GDR Conclusions SC 18 2

3 Overview of the MVAPICH2 Project High Performance open-source MPI Library for InfiniBand, Omni-Path, Ethernet/iWARP, and RDMA over Converged Ethernet (RoCE) MVAPICH (MPI-1), MVAPICH2 (MPI-2.2 and MPI-3.1), Started in 21, First version available in 22 MVAPICH2-X (MPI + PGAS), Available since 211 Support for GPGPUs (MVAPICH2-GDR) and MIC (MVAPICH2-MIC), Available since 214 Support for Virtualization (MVAPICH2-Virt), Available since 215 Support for Energy-Awareness (MVAPICH2-EA), Available since 215 Support for InfiniBand Network Analysis and Monitoring (OSU INAM) since 215 Used by more than 2,95 organizations in 86 countries More than 56, (>.5 million) downloads from the OSU site directly Empowering many TOP5 clusters (Nov 18 ranking) 3 rd ranked 1,649,64-core cluster (Sunway TaihuLight) at NSC, Wuxi, China 14 th, 556,14 cores (Oakforest-PACS) in Japan 17 th, 367,24 cores (Stampede2) at TACC 27 th, 241,18-core (Pleiades) at NASA and many others Available with software stacks of many vendors and Linux Distros (RedHat, SuSE, and OpenHPC) Empowering Top5 systems for over a decade Partner in the upcoming TACC Frontera System SC 18 3

4 Sep-4 Feb-5 Jul-5 Dec-5 May-6 Oct-6 Mar-7 Aug-7 Jan-8 Jun-8 Nov-8 Apr-9 Sep-9 Feb-1 Jul-1 Dec-1 May-11 Oct-11 Mar-12 Aug-12 Jan-13 Jun-13 Nov-13 Apr-14 Sep-14 Feb-15 Jul-15 Dec-15 May-16 Oct-16 Mar-17 Aug-17 Jan-18 Jun-18 Number of Downloads MV.9.4 MV2.9. MV2.9.8 MV2 1. MV 1. MV MV 1.1 MV2 1.4 MV2 1.5 MV2 1.6 MV2 1.7 MV2 1.8 MV2 1.9 MV2-GDR 2.3a MV2-X 2.3b MV2 Virt 2.2 OSU INAM.9.3 MV2 2.3 MV2-GDR 2.b MV2-MIC 2. 6 MVAPICH2 Release Timeline and Downloads Timeline SC 18 4

5 Architecture of MVAPICH2 Software Family High Performance Parallel Programming Models Message Passing Interface (MPI) PGAS (UPC, OpenSHMEM, CAF, UPC++) Hybrid --- MPI + X (MPI + PGAS + OpenMP/Cilk) High Performance and Scalable Communication Runtime Diverse APIs and Mechanisms Point-topoint Primitives Collectives Algorithms Job Startup Energy- Awareness Remote Memory Access I/O and File Systems Fault Tolerance Virtualization Active Messages Introspection & Analysis Support for Modern Networking Technology (InfiniBand, iwarp, RoCE, Omni-Path) Transport Protocols RC XRC UD DC UMR ODP Modern Features SR- IOV Multi Rail Support for Modern Multi-/Many-core Architectures (Intel-Xeon, OpenPower, Xeon-Phi, ARM, NVIDIA GPGPU) Shared Memory Transport Mechanisms CMA IVSHMEM XPMEM Modern Features MCDRAM * NVLink * CAPI * * Upcoming SC 18 5

6 SC 18 6 MVAPICH2 Software Family High-Performance Parallel Programming Libraries MVAPICH2 MVAPICH2-X MVAPICH2-GDR MVAPICH2-Virt MVAPICH2-EA MVAPICH2-MIC Microbenchmarks OMB Tools OSU INAM OEMT Support for InfiniBand, Omni-Path, Ethernet/iWARP, and RoCE Advanced MPI features, OSU INAM, PGAS (OpenSHMEM, UPC, UPC++, and CAF), and MPI+PGAS programming models with unified communication runtime Optimized MPI for clusters with NVIDIA GPUs High-performance and scalable MPI for hypervisor and container based HPC cloud Energy aware and High-performance MPI Optimized MPI for clusters with Intel KNC Microbenchmarks suite to evaluate MPI and PGAS (OpenSHMEM, UPC, and UPC++) libraries for CPUs and GPUs Network monitoring, profiling, and analysis for clusters with MPI and scheduler integration Utility to measure the energy consumption of MPI applications

7 PCIe GPU GPU GPU GPU SC 18 7 MVAPICH2-GDR: Optimizing MPI Data Movement on GPU Clusters Connected as PCIe devices Flexibility but Complexity Node Node 1 CPU QPI CPU CPU IB Memory buffers 1. Intra-GPU 2. Intra-Socket GPU-GPU 3. Inter-Socket GPU-GPU 4. Inter-Node GPU-GPU 5. Intra-Socket GPU-Host 6. Inter-Socket GPU-Host 7. Inter-Node GPU-Host 8. Inter-Node GPU-GPU with IB adapter on remote socket and more... For each path different schemes: Shared_mem, IPC, GPUDirect RDMA, pipeline Critical for runtimes to optimize data movement while hiding the complexity

8 GPU-Aware (CUDA-Aware) MPI Library: MVAPICH2-GPU Standard MPI interfaces used for unified data movement Takes advantage of Unified Virtual Addressing (>= CUDA 4.) Overlaps data movement from GPU with RDMA transfers At Sender: MPI_Send(s_devbuf, size, ); At Receiver: MPI_Recv(r_devbuf, size, ); inside MVAPICH2 High Performance and High Productivity SC 18 8

9 CUDA-Aware MPI: MVAPICH2-GDR Releases Support for MPI communication from NVIDIA GPU device memory High performance RDMA-based inter-node point-to-point communication (GPU-GPU, GPU-Host and Host-GPU) High performance intra-node point-to-point communication for multi-gpu adapters/node (GPU-GPU, GPU-Host and Host-GPU) Taking advantage of CUDA IPC (available since CUDA 4.1) in intra-node communication for multiple GPU adapters/node Optimized and tuned collectives for GPU device buffers MPI datatype support for point-to-point and collective communication from GPU device buffers Unified memory SC 18 9

10 SC 18 1 Using MVAPICH2-GPUDirect Version MVAPICH2-2.3 with GDR support can be downloaded from System software requirements Mellanox OFED 3.2 or later NVIDIA Driver or later NVIDIA CUDA Toolkit 7.5/8./9. or later Plugin for GPUDirect RDMA Strongly recommended GDRCOPY module from NVIDIA Contact MVAPICH help list with any questions related to the package mvapich-help@cse.ohio-state.edu

11 SC MVAPICH2-GDR 2.3 GA Released on 11/1/218 Major Features and Enhancements Based on MVAPICH2 2.3 Support for CUDA 1. Add support for Volta (V1) GPU Support for OpenPOWER with NVLink Efficient Multiple CUDA stream-based IPC communication for multi-gpu systems with and without NVLink Enhanced performance of GPU-based point-to-point communication Leverage Linux Cross Memory Attach (CMA) feature for enhanced host-based communication Enhanced performance of MPI_Allreduce for GPU-resident data InfiniBand Multicast (IB-MCAST) based designs for GPU-based broadcast and streaming applications Basic support for IB-MCAST designs with GPUDirect RDMA Advanced support for zero-copy IB-MCAST designs with GPUDirect RDMA Advanced reliability support for IB-MCAST designs Efficient broadcast and all-reduce designs for Deep Learning applications Enhanced collective tuning on Xeon, OpenPOWER, and NVIDIA DGX-1 systems

12 Bandwidth (MB/s) Latency (us) K 2K 4K 8K K 2K 4K Bandwidth (MB/s) SC K 2K 4K Optimized MVAPICH2-GDR Design GPU-GPU Inter-node Latency 1.85us 1x MV2-(NO-GDR) MV2-GDR 2.3 GPU-GPU Inter-node Bandwidth 9x GPU-GPU Inter-node Bi-Bandwidth MV2-(NO-GDR) MV2-GDR X MVAPICH2-GDR-2.3 Intel Haswell 3.1 GHz) node - 2 cores NVIDIA Volta V1 GPU Mellanox Connect-X4 EDR HCA CUDA 9. Mellanox OFED 4. with GPU-Direct-RDMA MV2-(NO-GDR) MV2-GDR-2.3

13 Average Time Steps per second (TPS) Average Time Steps per second (TPS) Application-Level Evaluation (HOOMD-blue) K Particles Platform: Wilkes (Intel Ivy Bridge + NVIDIA Tesla K2c + Mellanox Connect-IB) HoomdBlue Version 1..5 GDRCOPY enabled: MV2_USE_CUDA=1 MV2_IBA_HCA=mlx5_ MV2_IBA_EAGER_THRESHOLD=32768 MV2_VBUF_TOTAL_SIZE=32768 MV2_USE_GPUDIRECT_LOOPBACK_LIMIT=32768 MV2_USE_GPUDIRECT_GDRCOPY=1 MV2_USE_GPUDIRECT_GDRCOPY_LIMIT= X Number of Processes K Particles MV2 MV2+GDR 2X Number of Processes SC 18 13

14 Normalized Execution Time Normalized Execution Time Application-Level Evaluation (Cosmo) and Weather Forecasting in Switzerland Wilkes GPU Cluster Default Callback-based Event-based Number of GPUs CSCS GPU cluster Default Callback-based Event-based Number of GPUs 2X improvement on 32 GPUs nodes 3% improvement on 96 GPU nodes (8 GPUs/node) Cosmo model: /tasks/operational/meteoswiss/ On-going collaboration with CSCS and MeteoSwiss (Switzerland) in co-designing MV2-GDR and Cosmo Application C. Chu, K. Hamidouche, A. Venkatesh, D. Banerjee, H. Subramoni, and D. K. Panda, Exploiting Maximal Overlap for Non-Contiguous Data Movement Processing on Modern GPU-enabled Systems, IPDPS 16 SC 18 14

15 Outline Overview of the MVAPICH2 Project MVAPICH2-GPU with GPUDirect-RDMA (GDR) What s new with MVAPICH2-GDR Multi-stream Communication for IPC CMA-based Intra-node Communication Support Support for OpenPower and NVLink Maximal overlap in MPI Datatype Processing Streaming Support with IB Multicast and GDR High-Performance Deep Learning (HiDL) with MVAPICH2-GDR Conclusions SC 18 15

16 Million Bytes (MB)/second Million Bytes (MB)/second Multi-stream Communication using CUDA IPC on OpenPOWER and DGX-1 Up to 16% higher Device to Device (D2D) bandwidth on OpenPOWER + NVLink inter-connect Up to 3% higher D2D bandwidth on DGX-1 with NVLink Pt-to-pt (D-D) Bandwidth: Benefits of Multi-stream CUDA IPC Design Pt-to-pt (D-D) Bandwidth: Benefits of Multi-stream CUDA IPC Design % better 16K 32K 64K 128K 256K 512K 1M 2M 4M % better 128K 256K 512K 1M 2M 4M 1-stream 4-streams 1-stream 4-streams Available since MVAPICH2-GDR-2.3a SC 18 16

17 Latency (us) Latency (us) SC CMA-based Intra-node Communication Support Up to 3% lower Host-to-Host (H2H) latency and 3% higher H2H Bandwidth INTRA-NODE Pt-to-Pt (H2H) LATENCY 3% better INTRA-NODE Pt-to-Pt (H2H) BANDWIDTH 3% better MV2-GDR (w/out CMA) MV2-GDR (w/ CMA) MV2-GDR (w/out CMA) MV2-GDR (w/ CMA) MVAPICH2-GDR-2.3 Intel Broadwell (E GHz) node 28 cores NVIDIA Tesla K-8 GPU, and Mellanox Connect-X4 EDR HCA CUDA 8., Mellanox OFED 4. with GPU-Direct-RDMA

18 (Nodes=1, PPN=2) Latency (us) Latency (us) (Nodes=1, PPN=2) (Nodes=1, PPN=2) Latency (us) Latency (us) (Nodes=1, PPN=2) Scalable Host-based Collectives on OpenPOWER (Intra-node Reduce & AlltoAll) MVAPICH2-GDR SpectrumMPI OpenMPI K 2K 4K MVAPICH2-GDR SpectrumMPI OpenMPI X 3.2X 3.3X Reduce 5.2X Alltoall MVAPICH2-GDR SpectrumMPI OpenMPI-3.. 8K 16K 32K 64K 128K 256K 512K 1M MVAPICH2-GDR SpectrumMPI OpenMPI X 3.2X 1.3X 1.2X K 2K 4K 8K 16K 32K 64K 128K 256K 512K 1M Up to 5X and 3x performance improvement by MVAPICH2 for small and large messages respectively SC 18 18

19 Bandwidth (MB/s) Bandwidth (MB/s) Latency (us) Latency (us) Intra-node Point-to-Point Performance on OpenPower Intra-Socket Small Message Latency.3us MVAPICH2-2.3 SpectrumMPI OpenMPI K 2K Intra-Socket Bandwidth MVAPICH2-2.3 SpectrumMPI OpenMPI Intra-Socket Large Message Latency MVAPICH2-2.3 SpectrumMPI OpenMPI-3.. 4K 8K 16K 32K 64K 128K 256K 512K 1M 2M Intra-Socket Bi-directional Bandwidth MVAPICH2-2.3 SpectrumMPI OpenMPI K 32K 256K 2M K 32K 256K 2M Platform: Two nodes of OpenPOWER (Power8-ppc64le) CPU using Mellanox EDR (MT4115) HCA SC 18 19

20 K 2K 4K 8K K 4K 16K 64K 256K 1M 4M Latency (us) Latency (us) Bandwidth (GB/sec) K 2K 4K 8K K 4K 16K 64K 256K 1M 4M Latency (us) Latency (us) Bandwidth (GB/sec) MVAPICH2-GDR: Performance on OpenPOWER (NVLink + Pascal) INTRA-NODE LATENCY (SMALL) 4 2 INTRA-NODE LATENCY (LARGE) 16K 32K 64K 128K256K512K 1M 2M 4M INTRA-NODE BANDWIDTH INTRA-SOCKET(NVLINK) INTER-SOCKET INTRA-SOCKET(NVLINK) INTER-SOCKET INTRA-SOCKET(NVLINK) INTER-SOCKET Intra-node Latency: 13.8 us (without GPUDirectRDMA) Intra-node Bandwidth: 33.2 GB/sec (NVLINK) INTER-NODE LATENCY (SMALL) INTER-NODE LATENCY (LARGE) INTER-NODE BANDWIDTH Inter-node Latency: 23 us (without GPUDirectRDMA) Available since MVAPICH2-GDR 2.3a Inter-node Bandwidth: 6 GB/sec (FDR) Platform: OpenPOWER (ppc64le) nodes equipped with a dual-socket CPU, 4 Pascal P1-SXM GPUs, and 4X-FDR InfiniBand Inter-connect SC 18 2

21 (Nodes=2, PPN=2) Latency (us) Latency (us) (Nodes=1, PPN=2) Latency (us) Latency (us) Optimized All-Reduce with XPMEM on OpenPOWER 2 1 2X 16K 32K 64K 128K 256K 512K 1M 2M K 32K 64K 128K 256K 512K 1M 2M Message Size MVAPICH2-GDR-Next 4X SpectrumMPI-1.1. OpenMPI-3.. MVAPICH2-GDR-Next 3.3X SpectrumMPI-1.1. OpenMPI-3.. 3X 2 MVAPICH2-GDR-Next SpectrumMPI OpenMPI K 32K 64K 128K 256K 512K 1M 2M 4 MVAPICH2-GDR-Next 3 SpectrumMPI OpenMPI X 16K 32K 64K 128K 256K 512K 1M 2M Message Size Optimized MPI All-Reduce Design in MVAPICH2 Up to 2X performance improvement over Spectrum MPI and 4X over OpenMPI for intra-node 34% 48% 2X Optimized Runtime Parameters: MV2_CPU_BINDING_POLICY=hybrid MV2_HYBRID_BINDING_POLICY=bunch SC 18 21

22 Non-contiguous Data Exchange Halo data exchange Multi-dimensional data Row based organization Contiguous on one dimension Non-contiguous on other dimensions Halo data exchange Duplicate the boundary Exchange the boundary in each iteration SC 18 22

23 MPI Datatype support in MVAPICH2 Datatypes support in MPI Operate on customized datatypes to improve productivity Enable MPI library to optimize non-contiguous data At Sender: MPI_Type_vector (n_blocks, n_elements, stride, old_type, &new_type); MPI_Type_commit(&new_type); MPI_Send(s_buf, size, new_type, dest, tag, MPI_COMM_WORLD); Inside MVAPICH2 - Use datatype specific CUDA Kernels to pack data in chunks - Efficiently move data between nodes using RDMA - In progress - currently optimizes vector and hindexed datatypes - Transparent to the user H. Wang, S. Potluri, D. Bureddy, C. Rosales and D. K. Panda, GPU-aware MPI on RDMA-Enabled Clusters: Design, Implementation and Evaluation, IEEE Transactions on Parallel and Distributed Systems, Vol. 25, No. 1, pp , Oct 214. SC 18 23

24 SC MPI Datatype Processing (Computation Optimization ) Comprehensive support Targeted kernels for regular datatypes - vector, subarray, indexed_block Generic kernels for all other irregular datatypes Separate non-blocking stream for kernels launched by MPI library Avoids stream conflicts with application kernels Flexible set of parameters for users to tune kernels Vector MV2_CUDA_KERNEL_VECTOR_TIDBLK_SIZE MV2_CUDA_KERNEL_VECTOR_YSIZE Subarray MV2_CUDA_KERNEL_SUBARR_TIDBLK_SIZE MV2_CUDA_KERNEL_SUBARR_XDIM MV2_CUDA_KERNEL_SUBARR_YDIM MV2_CUDA_KERNEL_SUBARR_ZDIM Indexed_block MV2_CUDA_KERNEL_IDXBLK_XDIM

25 Initiate Kernel Initiate Kernel Initiate Kernel Initiate Kernel Start Send WFK Initiate Kernel Start Send WFK Start Send WFK Start Send Start Send SC Initiate Kernel Start Send MPI Datatype Processing (Communication Optimization ) Common Scenario MPI_Isend (A,.. Datatype, ) MPI_Isend (B,.. Datatype, ) MPI_Isend (C,.. Datatype, ) MPI_Isend (D,.. Datatype, ) Waste of computing resources on CPU and GPU Existing Design Isend(1) Wait For Kernel (WFK) Kernel on Stream Isend(1) Wait For Kernel (WFK) Kernel on Stream GPU Isend(1) Wait For Kernel (WFK) Kernel on Stream Wait Progress CPU MPI_Waitall ( ); Proposed Design Isend(1) Isend(2)Isend(3) Wait CPU Progress *A, B contain non-contiguous MPI Datatype Kernel on Stream Kernel on Stream Kernel on Stream GPU Expected Benefits Start Time Finish Proposed Finish Existing

26 Streaming Applications Streaming applications on HPC systems 1. Communication (MPI) Broadcast-type operations 2. Computation (CUDA) Multiple GPU nodes as workers HPC resources for real-time analytics Data Source Real-time streaming Sender Data streaming-like broadcast operations Worker CPU GPU Worker CPU GPU Worker CPU GPU Worker CPU GPU Worker CPU GPU GPU GPU GPU GPU GPU SC 18 26

27 Hardware Multicast-based Broadcast For GPU-resident data, using GPUDirect RDMA (GDR) InfiniBand Hardware Multicast (IB-MCAST) Overhead IB UD limit GDR limit A. Venkatesh, H. Subramoni, K. Hamidouche, and D. K. Panda, A High Performance Broadcast Design with Hardware Multicast and GPUDirect RDMA for Streaming Applications on InfiniBand Clusters, in HiPC 214, Dec 214. CPU GPU Source Header Data IB HCA IB Switch 1. IB Gather + GDR Read 2. IB Hardware Multicast 3. IB Scatter + GDR Write Destination 1 IB HCA IB HCA Header CPU GPU Data Destination N Header CPU GPU Data SC 18 27

28 SC Broadcast on Multi-GPU systems Proposed Intra-node Topology-Aware Broadcast CUDA InterProcess Communication (IPC) Available since MVAPICH2-GDR 2.3a C.-H. Chu, K. Hamidouche, H. Subramoni, A. Venkatesh, B. Elton, and D. K. Panda, "Designing High Performance Heterogeneous Broadcast for Streaming Applications on GPU Clusters, " SBAC-PAD'16, Oct , 216. Source CPU GPU Multicast steps IB Switch cudamemcpy (Device Device) CPU Node 1 CPU GPU Node N GPU GPU 1 GPU N

29 Latency (μs) Latency (μs) Streaming CSCS (88 GPUs) MCAST-GDR-OPT MCAST-GDR MCAST-GDR-OPT MCAST-GDR % % K 2K 4K 8K 16K 32K 64K 128K 256K 512K 1M 2M 4M IB-MCAST + GDR + Topology-aware IPC-based schemes Up to 58% and 79% reduction for small and large messages C.-H. Chu, K. Hamidouche, H. Subramoni, A. Venkatesh, B. Elton, and D. K. Panda, "Designing High Performance Heterogeneous Broadcast for Streaming Applications on GPU Clusters, " SBAC-PAD'16, Oct , 216. SC 18 29

30 Outline Overview of the MVAPICH2 Project MVAPICH2-GPU with GPUDirect-RDMA (GDR) What s new with MVAPICH2-GDR High-Performance Deep Learning (HiDL) with MVAPICH2-GDR Benefits of CUDA-Aware MPI with TensorFlow Benefits with CNTK Optimized Collectives for Deep Learning Co-designed OSU-Caffe Out-of-core DNN Training Conclusions SC 18 3

31 Deep Learning: New Challenges for MPI Runtimes Scale-up Performance Deep Learning frameworks are a different game altogether Unusually large message sizes (order of megabytes) NCCL1 NCCL2 Desired Most communication based on GPU buffers Existing State-of-the-art cudnn, cublas, NCCL --> scale-up performance cudnn MKL-DNN MPI NCCL2, CUDA-Aware MPI --> scale-out performance For small and medium message sizes only! Proposed: Can we co-design the MPI runtime (MVAPICH2- GDR) and the DL framework (Caffe) to achieve both? grpc Efficient Overlap of Computation and Communication Efficient Large-Message Communication (Reductions) What application co-designs are needed to exploit communication-runtime co-designs? Hadoop Scale-out Performance A. A. Awan, K. Hamidouche, J. M. Hashmi, and D. K. Panda, S-Caffe: Co-designing MPI Runtimes and Caffe for Scalable Deep Learning on Modern GPU Clusters. In Proceedings of the 22nd ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming (PPoPP '17) SC 18 31

32 Images/sec (higher is better) Exploiting CUDA-Aware MPI for TensorFlow (Horovod) 35 MVAPICH2-GDR offers excellent performance via advanced designs for MPI_Allreduce MVAPICH2-GDR is up to 22% faster than MVAPICH2 Up to 22% better performance on 1 Wilkes2 cluster (16 GPUs) No. of GPUs (4GPUs/node) MVAPICH2 MVAPICH2-GDR SC 18 32

33 Training Time (s) Training Time (s) Performance Benefits with CNTK Deep Learning Framework RI2 cluster, 16 GPUs, 1 GPU/node Microsoft Cognitive Toolkit (CNTK) [ AlexNet model Lower is better VGG model MV2-GDR-Knomial MV2-GDR-Ring MCAST-GDR-Opt 3 15% 24% MV2-GDR-Knomial MV2-GDR-Ring MCAST-GDR-Opt 25 6% Number of GPU nodes Number of GPU nodes Reduces up to 24% and 15% of latency for AlexNet and VGG models Higher improvement can be observed for larger system sizes C.-H. Chu, X. Lu, A. A. Awan, H. Subramoni, J. Hashmi, B. Elton, and D. K. Panda, Efficient and Scalable Multi-Source Streaming Broadcast on GPU Clusters for Deep Learning, ICPP 17. SC %

34 Latency (us) Latency (us) Latency (ms) MVAPICH2-GDR: Allreduce Comparison with Baidu and OpenMPI 16 GPUs (4 nodes) MVAPICH2-GDR vs. Baidu-Allreduce and OpenMPI ~3X better MVAPICH2 BAIDU OPENMPI ~4X better 512K 1M 2M 4M MVAPICH2 BAIDU OPENMPI 6 ~1X better OpenMPI is ~5X slower MV2 is ~2X better than Baidu than Baidu MVAPICH2 BAIDU OPENMPI *Available since MVAPICH2-GDR 2.3a SC 18 34

35 Latency (us) MVAPICH2-GDR vs. NCCL2 Reduce Operation Latency (us) Optimized designs offer better/comparable performance for most cases MPI_Reduce (MVAPICH2-GDR) vs. ncclreduce (NCCL2) on 16 GPUs 1 ~2.5X better ~5X better K 2K 4K 8K 16K 32K 64K 1 MVAPICH2-GDR NCCL2 MVAPICH2-GDR NCCL2 Platform: Intel Xeon (Broadwell) nodes equipped with a dual-socket CPU, 1 K-8 GPUs, and EDR InfiniBand Inter-connect SC 18 35

36 MVAPICH2-GDR vs. NCCL2 Allreduce Operation Latency (us) K 2K 4K 8K 16K 32K 64K Latency (us) Optimized designs in MVAPICH2-GDR 2.3 offer better/comparable performance for most cases MPI_Allreduce (MVAPICH2-GDR) vs. ncclallreduce (NCCL2) on 16 GPUs ~3X better ~1.2X better 1 1 MVAPICH2-GDR NCCL2 MVAPICH2-GDR NCCL2 Platform: Intel Xeon (Broadwell) nodes equipped with a dual-socket CPU, 1 K-8 GPUs, and EDR InfiniBand Inter-connect SC 18 36

37 Training Time (seconds) OSU-Caffe: Scalable Deep Learning Caffe : A flexible and layered Deep Learning framework. Benefits and Weaknesses Multi-GPU Training within a single node Performance degradation for GPUs across different sockets Limited Scale-out OSU-Caffe: MPI-based Parallel Training Enable Scale-up (within a node) and Scale-out (across multi-gpu nodes) Scale-out on 64 GPUs for training CIFAR-1 network on CIFAR-1 dataset Scale-out on 128 GPUs for training GoogLeNet network on ImageNet dataset OSU-Caffe publicly available from GoogLeNet (ImageNet) on 128 GPUs No. of GPUs Invalid use case Caffe OSU-Caffe (124) OSU-Caffe (248) SC 18 37

38 Out-of-Core Deep Neural Network Training with Caffe Batch Size Large DNNs cannot be trained on GPUs due to memory limitation! ResNet-5 is the state-of-the-art DNN architecture for Image Recognition but current frameworks can only go up to a small batch size of 45 Next generation models like Neural Machine Translation (NMT) are ridiculously large, consists of billions of parameters, and require even more memory Can we design Out-of-core DNN training support using new software features in CUDA 8/9 and hardware mechanisms in Pascal/Volta GPUs? General intuition is that managed allocations will be slow! The proposed framework called OC-Caffe (Out-of-Core Caffe) shows the potential of managed memory designs that can provide performance with negligible/no overhead. In addition to Out-of-core Training support, productivity can be greatly enhanced in terms of DL framework design by using the new Unified Memory features A. Awan, C. Chu, H. Subramoni, X. Lu, and D. K. Panda, OC-DNN: Exploiting Advanced Unified Memory Capabilities in CUDA 9 and Volta GPUs for Out-of-Core DNN Training, HiPC 18 SC P1 GPU Memory Limit (16 GB) AlexNet GoogLeNet VGG Out-of-core Training 496 Trainability (Memory Requirements) 124

39 Performance Trends for OC-Caffe OC-Caffe-Opt: up to 8% better than Intel-optimized CPU Caffe for ResNet-5 training on the Volta V1 GPU with CUDA9 and CUDNN7 OC-Caffe allows efficient scale-up on DGX-1 system with Pascal P1 GPUs with CUDA9 and CUDNN7 Scale-up on DGX-1 Out-of-core (over-subscription) Caffe Images Per Second (Higher is better) Images/sec (Higher is better) 2 oc-caffe-opt is 8% better than intel-caffe caffe-gpu cannot run intelcaffe-opt (N/A) X X caffe-gpu oc-caffe-naïve oc-caffe-opt caffe-cpu intel-caffe intel-caffe-opt OC-Caffe Number of GPUs (DGX-1) 8 A. Awan, C. Chu, H. Subramoni, X. Lu, and D. K. Panda, OC-DNN: Exploiting Advanced Unified Memory Capabilities in CUDA 9 and Volta GPUs for Out-of-Core DNN Training, HiPC 18 SC 18 39

40 Outline Overview of the MVAPICH2 Project MVAPICH2-GPU with GPUDirect-RDMA (GDR) What s new with MVAPICH2-GDR High-Performance Deep Learning (HiDL) with MVAPICH2-GDR Conclusions SC 18 4

41 Conclusions MVAPICH2-GDR Library provides optimized MPI communication on InfiniBand and RoCE clusters with GPUs Supports both X86 and OpenPower with NVLink Takes advantage of CUDA features like IPC and GPUDirect RDMA families Allows flexible solutions for streaming applications with GPUs Provides optimized solutions (scale-up and scale-out) for High-Performance Deep Learning SC 18 41

42 SC Join us at the OSU Booth Join the OSU Team at SC 18 at Booth #444 and grab a free T-Shirt! More details about various events available at the following link

43 SC Thank You! Network-Based Computing Laboratory The High-Performance MPI/PGAS Project The High-Performance Deep Learning Project