HPC and AI Solution Overview. Garima Kochhar HPC and AI Innovation Lab

Transcription

1 HPC and AI Solution Overview Garima Kochhar HPC and AI Innovation Lab 1

2 Dell EMC HPC and DL team charter Design, develop and integrate HPC and DL Heading systems Lorem ipsum dolor sit amet, consectetur adipiscing elit. Flexible reference architectures Systems tuned for research computing, manufacturing, life sciences, oil and gas, etc. Act as the focal point for joint R&D activities Technology collaboration with partners for joint innovation Research coordination with DSC, COEs and customers New Investment: more SMEs, huge innovation ecosystem HPC Innovation Lab Technical briefings, tours, remote access Conduct application performance Heading studies and develop best practices Lorem ipsum dolor sit amet, consectetur adipiscing elit. White papers, blogs, presentations Prototype and evaluate advanced technologies HPC+Cloud, HPC+Big Data nvmes, FPGAs, containers, DL/ML workloads, etc. 2

3 World-class infrastructure in the Innovation Lab 13K ft. 2 lab, 1,300+ servers, ~10PB storage dedicated to HPC in collaboration with the community Zenith TOP500-class system based on Intel Scalable Systems Framework (OPA, KNL, Xeon) 324 nodes with Intel Xeon Gold 6148-F processors, Omni-Path fabric and 655TF sustained performance Intel Xeon Phi (KNL) servers. 805TF combined performance. #292 on Top500, 1.4PFlop/s theoretical peak Isilon H600, F800 Rattler Research/development system with Mellanox, NVIDIA and Bright Computing 88 nodes with Intel Xeon Gold 6148 processors and EDR InfiniBand 16 nodes with Intel Xeon Gold 6148 processors and 4 V100 GPUs each 3

4 Focus areas HPC software stack Bright Cluster Manager, OpenHPC Integration of all software components Compute performance and tuning Application focus + BIOS, Memory, Interconnect Accelerators and co-processors Different workloads Interconnect performance and tuning Storage solutions Isilon, NSS, Lustre Vertical solutions Genomics research CFD/Manufacturing Proof of Concept studies Containers, FPGAs, NVMeoF, etc. 4

5 Compute - performance - tuning

6 Stream Triad GB/s (higher is better) 16 DIMM relative bandwidth when compared to 12 DIMMs. Relative memory bandwidthh [higher is better] Careful memory configuration - Skylake Impact of unbalanced 512 GB memory configuration Near-balanced configurations GB, 576 GB Processor SKU Balanced Balanced Balanced Near balanced Near balanced 12 x 16GB 12 x 32GB 24 x 32GB 12 x 16GB + 12 x 32GB 12 x 8GB + 12 x 16GB 192 GB 384 GB 768 GB 576 GB 288 GB 384 GB 12x32GB 512 GB 16x32GB Rel perf 16x32GB 6138, 2666 MT/s 6142, 2666 MT/s Unbalanced configurations are very bad for performance Balanced and near-balanced configurations are ideal for HPC 6

7 Performance Relative to E5-2697A v4 ANSYS Fluent Two Socket System Performance ANSYS Fluent v17.2 aircraft_wing_14m E v2 12c, 2.7 GHz, 130W, 1866 MT/s E v3 10c, 2.6/2.2 GHz, 105W, 2133 MT/s E v3 14c, 2.6/2.2 GHz, 145W, 2133 MT/s E5-2697A v4 16c, 2.6/2.2 GHz, 145W, 2400 MT/s E v4 18c, 2.3/2.0 GHz, 145W, 2400 MT/s c, 2.1GHz, 125W, 12x16GB 2666 MT/s c, 3.0GHz, 150W, 24x16GB 2666 MT/s c, 2.6GHz, 150W, 12x16GB 2666 MT/s c, 2.4GHz, 150W, 12x16GB 2666 MT/s c, 2.7GHz, 165W, 12x16GB 2666 MT/s c, 2.7GHz, 205W, 12x16GB 2666 MT/s IVB HSW BDW SKX Relative Performance per Core Skylake provides significantly better performance for ANSYS Fluent relative to Broadwell. Relative performance depends on the specific benchmark dataset: 1.2 to 1.4 for

8 Relative performance over 16 nodes (higher is better) Relative performance over 16 nodes (higher is better) AMD EPYC - WRF multi-node tests 30.0 WRF - conus 12km 30.0 WRF - conus 2.5 km epyc 2 epyc 4 epyc 8 epyc 16 epyc 1 epyc 2 epyc 4 epyc 8 epyc 16 epyc EPYC GHz EPYC GHz Linear scaling EPYC GHz EPYC GHz Linear scaling EPYC 7601 is ~3% better than Base frequency 10% faster, Turbo 6%. 8

9 Interconnects

10 Latency (us) EDR Latency w/ c-states SKL,BW EDR Latency (cstates enabled vs disabled) Message size (bytes) SKL cstates-en w/ switch SKL cstates-en, B2B SKL cstates-dis w/ switch SKL cstates-disabled, B2B SKL- Intel Xeon Gold 2.6GHz 16C Cstates enabled and disabled latency results are about the same. 10

11 Solver Rating (higher is better) Performance Relative to E v4 ANSYS Fluent Small Model Scaling ANSYS Fluent v17.2 ice_2m (1) 72 (2) 144 (4) 216 (6) 288 (8) 432 (12) 576 (16) C E v4 + EDR C EDR Perf relative to E v4 Number of Cores (Nodes) With a small data set, performance advantage of Skylake decreases at scale, but remains positive. Unlikely to run a model of this size with more than a few nodes. 11

12 GP GPU and accelerators

13 ns/day (higher is better) AMBER B V100 vs P % K V100 vs P % AMBER STMV PCIe SXM2 PCIe SXM2 13G C4130 P100 14G C4140 V Configuration B Configuration K V100 is significantly faster, 1x V100 faster than 4x P100 SMX2 better than PCIe 9% P100, up to 30% V100 SMX2 has slightly higher frequency than PCIe card 13

14 HPC storage solutions - NSS - IEEL - Isilon - Proof Of Concepts

15 Dell Storage for HPC with Intel EE for Lustre Solution Turn-key solution designed for high speed fast scratch storage Solution benefits & Dell differentiation Parallel scalable file system based on Intel EE for Lustre software Single file system namespace scalable to high capacities and performance Best practices developed by Dell HPC Engineering provide optimal performance on Dell hardware Tests yield peaks of roughly 15GB/s write and 17GB/s read per building block Lustre Distributed Namespace allows distribution of Lustre sub-directories across multiple MDTs to increase metadata capacity capabilities and performance Share data with other file systems utilizing optional NFS/CIFS gateway Dell Networking 10/40GbE, InfiniBand or Omni-Path Dell Po wervault MD3460 Dell Po wervault MD3420 Intel Manager for Lust re Pow eredge R630 MDS Pair Dell Po weredge R730 Ac tive/ Passive OSS Pair Dell Po weredge R730 Active/Active 12 Gbps SAS Failover Co nnect ions Dell Po wervault MD3420 (Optio nal for DNE) 12 Gbps SAS Failover Co nnect ions 15

16 16 IEEL3.0+OPA

17 Support for Isilon for Scalable NFS- Sequential performance (N-N) -- Write 17

18 Deep Learning

19 Images/sec Speedup TensorFlow+Horovod on multiple V100 nodes TensorFlow+Horovod Resnet50 on Multi-node V100 2 V100 4 V100 8 V V V100 Performance Speedup TensorFlow+Horovod scales well on multiple nodes, 12.5x speedup with 16 V100 (4 nodes) Ibverbs and MPI are used for nodes communication 4 nodes with V100-PCIe GPUs are used FP32 mode, batch size is 128 per GPU 19

20 Ready Bundle for Deep Learning - NVIDIA Open Source Frameworks TensorFlow, MxNet, CNTK, Theano, Torch, Caffe/Caffe2 Neural network Libraries MLPython, CaffeOnSpark, cudnn, cublas, NCCL, Keras, GIE Platform C4140 Processor 2 x Intel Xeon CPU 6148 Memory Drives Network GPU 384GB 2400MHz 2x200GB 1.8 SSD Mellanox ConnectX-5 VPI (EDR 100Gb/s) 4x V100-16GB SXM2 Software & Firmware [Reference] Operating System Provisioning and Management RHEL 7.4 x86_64 Bright Cluster Manager

21 Tying it together - Access to the lab - White papers and Blogs

22 Recent Publications Design Principles for HPC 14G with Skylake how much better for HPC? BIOS characterization for HPC with Intel Skylake processor Performance study of four Socket PowerEdge R940 Server with Intel Skylake processors Dell EMC HPC Systems - SKY is the limit Entering a new arena in Computing- KNL System benchmark results on KNL STREAM and HPL NAMD Performance Analysis on Skylake Architecture LAMMPS Four Node Comparative Performance Analysis on Skylake Processors De novo assembly with PowerEdge HPCG Performance study with Intel R940 Skylake processors Dell EMC HPC System for Life Skylake memory study Science v1.1 HPC Applications Performance on V100 Application Performance on P100- PCIe GPUs Containerizing HPC Applications with Singularity HPCG Performance study with Intel Performance of LS-DYNA on KNL Singularity Containers Scaling Deep Learning on Multiple V100 Nodes Deep Learning on V100 Deep Learning Inference on P40 vs P4 with Skylake Deep Learning Inference on P40 GPUs Deep Learning Performance with Intel Caffe Training, CPU model choice and Scalability Deep Learning Performance on R740 with V100 PCIe GPUs Getting Started With OpenHPC DELL EMC Isilon F800 and F600 I/O Performance DELL EMC ISILON F800 AND H600 WHOLE GENOME ANALYSIS PERFORMANCE Digital Manufacturing with 14G 22

23