An Open, Standards Based Approach to Building Supercomputers

Transcription

1 An Open, Standards Based Approach to Building Supercomputers HPC Advisory Council March 21-23, 2011 Lugano, Switzerland Reza Rooholamini

2 Development Approach Deliver ROBUST, RELIABLE, and SCALABLE solutions that solve customer problems while REMOVING COMPLEXITY and PROVIDING CHOICE Infrastructure Software Platform Software / FS Messaging Windows /Linux Operating Systems Database HPCC ESX/Xen/Hyper-V Hypervisors HA Virtualizatio n OCFS / NFS / GFS Datacenter Automation, Resource Management/ Orchestration, etc Server PowerEdge Servers (Rack / Tower / Blade) idrac, IPMI, BMC, DMC, OpenManage, etc Fabric Fibre Channel Infiniband FCoE 10GbE/1GbE SAS/SATA Storage FC SAN NAS iscsi TAPE JBOD/RBOD EqualLogic SANHQ, EMC Navisphere, etc

3 Building Block Approach for HPCC Applications Compute Nodes Middleware Parallel Benchmarks (NAS, HINT, Linpack ) and Parallel Applications MVAPICH Dell PowerEdge MPICH-GM/MX Servers MPICH (Intel and Intel AMD) MPI Protocol OS TCP Linux GM/MX InfiniBand Windows Interconnect Protocol Ethernet TCP Myrinet GM/MX InfiniBand InfiniBand File Systems Interconnect I/O Nodes Platform NFS Gigabit Ethernet `pnfs Lustre Myrinet Dell PowerEdge Servers InfiniBand Dell PowerEdge Servers (Intel and AMD) Storage Protocol SCSI and SAS Fibre channel Stornext Storage PowerVault, PS-series and Dell EMC storage

4 HPCC Program Vision: Provide the best price/performance, complete High Performance Computing solution, using standard building blocks and partnering with best of class in HPC. Proactively analyze HPC technologies, provide best practices to optimize your environments Provide integrated products including automation tools for you to design and manage your HPC infrastructure easier Evaluate new technologies and provide a window into the future to help you plan for new developments

5 HPCC Components and Enabling Technologies Lustre NFS/XFS Platform Lava (Open Source) Platform LSF (Enterprise) Resource Monitoring / Management Resource dynamic allocation Checkpoint restarting and Job redistributing Application Job Scheduler Benchmark Cluster File System Dell OMSA (Server Agent) Out-of-Band via idrac Automated BMC/BIOS setup Compilers and math library Performance tools - MPI analyzer / profiler - Debugger - Performance analyzer and optimizer Development Tools Middleware / API Node Monitoring & Management Operating Systems Interconnect Protocols Interconnects Hardware Cluster Hardware Software Monitoring & Management Dell Management Console Nagios OM integration GUI/CLI cluster admin tools Management Hardware Platform Hardware Cluster Installation

6 Top 500 Dell Presence November 2010 Rank Site System Cores RMax Rpeak 40 Lawrence Livermore National Laboratory Dell Xanadu 3 Cluster, Xeon X Ghz, QLogic InfiniBand QDR University of Colorado PowerEdge C6100 Cluster, Xeon X Ghz, Infiniband Lawrence Livermore National Laboratory Dell Xanadu 3 Cluster, Xeon X Ghz, QLogic InfiniBand QDR Maui High-Performance Computing Center Dell PowerEdge M610 Cluster, Nehalem-EP 2.8Ghz, Infiniband Lawrence Livermore National Laboratory Dell DCS Xanadu 2.5, Xeon E55xx 2.4Ghz, Infiniband DDR NCSA PowerEdge 1955, 2.33 GHz, Infiniband Texas Tech University Poweredge Cluster, Xeon X Ghz, Infiniband DDR KTH - Royal Institute of Technology PowerEdge SC1435 Dual core Opteron 2.2 GHz, Infiniband NNSA/Sandia National Laboratories PowerEdge 1850, 3.6 GHz, Infiniband Texas Advanced Computing Center/Univ. of Texas PowerEdge 1955, 2.66 GHz, Infiniband Maui High-Performance Computing Center PowerEdge 1955, 3.0 GHz, Infiniband University of Kentucky PowerEdge C6100 Cluster, Xeon X Ghz, Infiniband R-Systems Dell DCS CS23-SH, QC HT 2.8 GHz, Infiniband HHMI - Janelia Farm Research Campus Dell Poweredge R610, Xeon X Ghz, 10G Ethernet Louisiana Optical Network Initiative PowerEdge 1950, 2.33 GHz, Infiniband Harvard University - FAS Research Computing PowerEdge M600 Cluster, Xeon E Ghz, Infiniband Lawrence Livermore National Laboratory Dell DCS Xanadu 2.5, Xeon E55xx 2.4Ghz, Infiniband DDR Lawrence Livermore National Laboratory Dell DCS Xanadu 2.5, Xeon E55xx 2.4Ghz, Infiniband DDR Lawrence Livermore National Laboratory Dell Xanadu 3 Cluster, Xeon X Ghz, QLogic InfiniBand QDR University of Southern California PowerEdge 1950/SunFire X2200/iDataPlex/IBM dx340 Myrinet 10G Clemson University PowerEdge 1950/SunFire X2200/iDataPlex/IBM dx340 Myrinet 10G Saudi Aramco PowerEdge 1950, Xeon 53xx 3 GHz, Infiniband entries on the latest list

7 Modular HPC Architecture Raw Data Set Compute Fast Scratch Results Visualization Archive Compute 7 Confidential Permanent Storage Archive

8 Recommended Host: Dell PowerEdge C6100 Four 2-Socket Nodes in 2U Intel Westmere-EP Each Node: 12 DIMMs each 2 GigE (Intel) 1 Daughter Card (PCIe x8) 10GigE QDR IB One PCIe (half-length, half-height) Optional SAS controller (in-place of IB) Chassis Design: Hot Plug, Individually Serviceable System Boards / Nodes Up to 12 x 3.5 drives (3 per node) Up to 24 x 2.5 drives (6 per node) N+1 Power supplies (1100W or 1400W) NVIDIA HIC certified DDR and QDR IB PCIe card certified Dell High Performance Computing 8

9 PowerVault MD1200 Versatility and high capacity storage for mainstream applications, with optimal performance in sequential (streaming) applications. An easy way to expand server capacity and meet versatile business needs with mix drive options Performance and security for applications that focus on sequential I/O Key Features 2U 12 drive 3.5 enclosure Supports 2.5 and 3.5 drives Supports Nearline, 10K and 15K RPM HDD and SAS SSDs Support for Self-Encrypting Drives Expands to support 8 enclosures (96 drives) Utilizes PERC H800 Affinity with PowerEdge Servers Keep up with environmental demands with 80PLUS Silver Certified power supplies and improved density. Great For Mainstream and Capacity-Intensive Applications (backup to disk, , streaming media) Low cost way to expand the capacity of a single PowerEdge server DELL CONFIDENTIAL 9

10 Dell PowerEdge C410x: 3U chassis (external) - Room-and-Board for PCIe Gen-2 devices - Up to 8 hosts Sixteen (16) Gen-2 Devices - Initial Target = GPs - Support for any FH/HL or HH/HL device - Each slot Double-Wide - Individually Serviceable N+1 Power (3+1) - Gold (90%) N+1 Cooling (7+1) Dell High Performance Computing 10

11 C410x Flexibility: One Host with a Single 1:1, 2:1, 3:1, 4:1, (8:1) 1 / 2 s / Host HI C ipass cable PCI Switch C410x Host HI C ipass cable PCI Switch C410x 3 s / 4 s / Host HI C ipass cable PCI Switch C410x Host HI C ipass cable PCI Switch C410x Dell High Performance Computing 1

12 C410x Flexibility: One Host with Dual 1:1, 2:1, 3:1, 4:1, (8:1) 6 s: 3 / 8 s: 4 / Host HIC HIC ipass cable PCI Switch PCI Switch C410x Host HIC HIC ipass cable PCI Switch PCI Switch C410x Dell High Performance Computing 1

13 Performance/Watt relative to Turbo on (higher is better) -5% Perf + 14% Power - 6% Perf +20% Power - 8% Perf + 12% Power - 2% Perf + 21% Power 11G - Turbo mode 1.20 Energy efficieny gains with Turbo disabled ECLIPSE - FOURMILL FLUENT - truck_poly_14m LU - class D WRF - Conus 12km

14 11G Optimal BIOS Settings Optimal BIOS Settings for High Performance Computing with PowerEdge 11G Servers BIOS Setting Energy Max Efficiency Performance Balanced Min. Latency Power Management APC Max Perf APC Max Perf C-State Enabled Enabled Enabled Disabled Node Interleaving Disabled Disabled Disabled Disabled SMT Disabled Disabled Disabled Disabled Turbo Mode Disabled Enabled Enabled Enabled DCU/Data Reuse Disabled Enabled Enabled Enabled

15 MB/s I/O Bandwidth Comparison Ohio State University MPI Bandwidth Test QDR Infiniband (x8 PCIe gen 2) DDR Infiniband (x8 PCIe) SDR Infinband (PCI-X) Message Size (Bytes)

16 Wall clock (s) :Host Scaling : -HMMER HMMER: Scaling C410x / C6100 (1) C410x / C6100 (2) C410x / C6100 (4) Speedup 1.8X 3.6X 7.2X Length of HMM Dell High Performance Computing 16

17 Resilient Storage Dell NFS Storage Solution (NSS) Dell HPC NFS Storage Solution (NSS) Primary Storage for Small (4 64 node) cluster Provide storage for /home directories, applications and long term data storage needs for medium to large clusters Small (20TB), Medium (40TB) and Large (80TB) configurations QDR InfiniBand or 10Gb Ethernet connectivity. Dell developed best practices and tuning guidance. Based on XFS File System to provide a single namespace Built on Dell Power Edge R710 and Dell PowerVault MD years of Dell PRO Support for IT and Mission Critical 4HR 7x24 onsite pack. 17 Confidential

18 Throughput KB/s NSS Performance NSS IPoIB Sequential Writes NSS Small NSS Medium NSS Large Nodes 18 Confidential

19 An Example of Standard HPCC Bundle

20 What is an Advisor? Solution Advisors Software application that recommends the best fit Dell products and solutions based on customers specific needs Available on Dell.com Example: The HPCC Advisor asks user: OS type? Optimize for performance, power or density Desired sustained or theoretical performance Recommends a solution based on this input. Launch

21 Research Opportunities File Systems Heterogeneous operating system support for parallel file systems Content addressable storage for unstructured data Integration of HSM and de-dupe capability with HPC file systems HPC in the Cloud Features needed in Virtualization layer to support HPCC I/O bypass support Virtualization layer Peer communication capabilities Semantic/architecture of an HPCC VM Policy-based personalities Compute node, I/O node, head node as needed Unified fabric Reliability Adaptability to application classes

22 GSE Lab in Austin Texas HPCC lab with wide array of servers, heterogeneous storage (DAS, NAS, iscsi, Fibre Channel) and networking products Similar laboratory environment in Bangalore, India and Limerick, Ireland Greater than 100 engineers work on HPCC You have may personal invitation to visit us!

23 Thank You! Dell HPCC Website: Optimal BIOS Settings for High-Performance Computing With PowerEdge 11G Servers - nts~hpc_dell_11g_bios_options.pdf.aspx DellTM Terascala HPC Storage Solution - DELL PowerVault MD1200 Performance as a Network File System (NFS) Backend Storage Solution - Memory Selection Guidelines for High Performance Computing with Dell PowerEdge 11G Servers - nts~11g-memory-hpc-wp.pdf.aspx Scalable Storage for High Performance Computing: Red Hat Global File System on Dell EqualLogic - NFS Server with 10 Gigabit Ethernet Networks -

24 Questions?

25 Performance with Turbo on relative to Turbo off (higher is better) Performance gains with Turbo mode IB 1.00 IB IB GbE 0.98 IB GbE GbE 0.96 GbE ECLIPSE - FOURMILL FLUENT - truck_poly_14m LU - class D WRF - Conus 12km Turbo mode enabled helps performance by up to 8%. 16-node PowerEdge M610 cluster with Dual Intel Xeon X5570, 6 x 4GB 1333 MHz RDIMMS

26 Impact of Interconnect on Application Performance 26 Confidential

27 Impact of RAID mode and Drive Configuration on Performance Throughput (MB/s) 2000 MD H800, NearLine SAS, Seq Read R6/6 R5/6 R6/ Dr 12 Dr 24 Dr 36 Dr 48 Dr 27 Confidential

28 Wall Clock(s) :Host Scaling : LAMMPS Gay-Berne 1200 LAMMPS GB Scaling C410x / C6100 (1) C410x / C6100 (2) C410x / C6100 (4) Speedup 41X 82X 142X Number of Particles Dell High Performance Computing 28

29 Impact of RAID mode and Drive Configuration on Performance Throughput (MB/s) 1400 MD H800, NearLine SAS, Seq Write R6/6 R5/6 R6/ Dr 12 Dr 24 Dr 36 Dr 48 Dr 29 Confidential