LRZ SuperMUC One year of Operation

Transcription

1 LRZ SuperMUC One year of Operation IBM Deep Computing Klaus Gottschalk IBM HPC Architect

2 Leibniz Computing Center s new HPC System is now installed and operational 2

3 SuperMUC Technical Highlights 3 PFLOP Computer in Germany in Gauß-Center 9414 Nodes with 2 Intel Sandy Bridge EP 209 Nodes with 4 Intel Westmere EX 3 PFLOP/s Peak Performance 327 TB Memory Infiniband Interconnect Large File Space for multiple purpose 10 PByte File Space based on IBM GPFS with 200GByte/s aggregated I/O Bandwidth 2 PByte NAS Storage with 10GByte/s aggregated I/O Bandwidth No GPGPUs or other Accelerator Technology Innovative Technology for Energy Efficient Computing Hot Water Cooling Energy Aware Scheduling Most Energy and Cooling Efficient high End HPC System: PUE 1.1

4 SuperMUC Energy Efficiency Goals Stable, highly scalable, efficient Hardware based on standard x86 components Save 40% of Energy compared to air-cooled HPC systems Hot Water cooling allowing for Fee Cooling all year around Using standard components Easy serviceable Frequency controlled nodes Optimize Application Energy consumption during use Energy saving if not in use End of a LINPack Run on 240 nodes Power Aware Job Scheduling Run Application at optimal clock rate according to predefined policies Deliver an energy report after job run

5

6 idataplex dx360 M4 water cooled - with Intel Sandy Bridge CPU

7 Island Architecture Muticluster GPFS Island 2 Core SW N1 N2 GPFS Server Cluster N3 Island 2 Core SW Spine Switch 1 Spine Switch 2 Spine Switch 3 Spine Switch 124 Spine Switch 125 I/O Island Core Switch 2 N4 N5 N6 N7 N8 N65 N66 IB P2P SFA 12k Spine Switch N67 N68 N69 SFA 12k N70 Island 18 Core SW N71 N Martin W Hiegl / Uwe Tron

8 Option: Direct Water Cooling (DWC) Direct cooling in water at heat source (95%) - no media change Less noise in machine room no spinning fans Cooling of system without need for Chillers PUE Node inlet temperature between C Inlet temperature can vary with seasons based on achievable temperature Lower and more stable CPU Core temperature (max 70 C) About 10% less leak current compared to air cooled systems Similar pipework requirements as rear door heat exchangers* Clear Advantages: Less energy consumption for cooling of the system (about 40%) Less energy consumption of the CPU (10%) Enables usage of Turbo Mode with all Cores Better TCO and higher efficiency of compute power usage (*) see ASHRAE Technical Committee 9.9 Whitepaper: 2011 Thermal Guidelines for Liquid Cooled Data Processing Environments 8

9 Option: Energy Aware Scheduling (EAS) Policy based steering of node CPU clock for user batch jobs Batch scheduler estimates application run time based on clock rate Admin defined policies determine node clock rate at application execution time Currently unused nodes will be powered down EAS is part of IBM LoadLeveler and xcat and will be ported to LSF Clear Advantages: Less power consumption of application that cannot gain performance from high clock rates Reduction of power consumption of idle nodes Observation of operational limits Example SuperMUC : Default clock rate is 2.2GHz Higher rates up to 2.7 GHz (or Turbo Mode) for applications that will gain performance LINPACK measurement done with Intel Turbomode 9

10 LINPACK on May 31, :================================================================================ 0:T/V N NB P Q Time Gflops 0: :WR01C2R e+06 0:--VVV--VVV--VVV--VVV--VVV--VVV--VVV--VVV--VVV--VVV--VVV--VVV--VVV--VVV--VVV- 0:Max aggregated wall time rfact... : :+ Max aggregated wall time pfact.. : :+ Max aggregated wall time mxswp.. : :Max aggregated wall time update.. : :+ Max aggregated wall time laswp.. : :Max aggregated wall time up tr sv. : : : Ax-b _oo/(eps*( A _oo* x _oo+ b _oo)*n)= PASSED 0:============================================================================ 0: 0:Finished 1 tests with the following results: 0: 1 tests completed and passed residual checks, 0: 0 tests completed and failed residual checks, 0: 0 tests skipped because of illegal input values. 0: : 0:End of Tests. 0:============================================================================ done running linpack at :43:

11 Value of the SuperMUC System SuperMUC represents tightly integrated innovative solution, with a value proposition which reduces client s total cost of ownership and which address growth areas of x86 and green computing. Energy- and cooling efficiency characteristics of hardware and HPC Software Stack provides quantifiable cost reduction PUE 1.1 (SuperMUC incl. Cooling) Holistic view of the Supercomputer Hardware, -Software and Applications Running cost of client reduced by 40% compared to HPC standard system of similar size Scalability, functionality and quality of hardware, software and service provide a qualifiable cost advantage Fewer problems because of leveraging experience from other platforms Faster problem resolution because of integrating development and support Running cost of client reduced by less downtime Running cost of client reduced by less management effort

12 One Year of Operation Direct water cooling is relievable and stable Summer and Winter LRZ Decision: Inlet Temperature varies with outdoor temperature between C Energy Saving Goal of LRZ and IBM is achieved Hardware Failures are below the expected Range Island based architecture for Infiniband, xcat, GPFS, LoadLeveler proves its scalability for large systems Power Consumption metering based on ipdus down to outlet level Hardware and Service Monitoring based on Icinga Automated Call Home on Failure for all system parts Log file analysis based on Splunk Martin W Hiegl / Uwe Tron

13