CAS 2K13 Sept Jean-Pierre Panziera Chief Technology Director

Transcription

1 CAS 2K13 Sept Jean-Pierre Panziera Chief Technology Director 1

2 personal note 2

3 Complete solutions for Extreme Computing b ubullx ssupercomputer u p e r c o p u t e r suite s u e Production ready supercomputers Data Centers for all organizations Applications optimized for hyper-parallel supercomputers HPC for every user with public/private HPC cloud 3

4 Bull: from Supercomputers to Cloud Computing Expertise & services Software HPC Systems Architecture Applications & Performance Energy Efficiency Data Management HPC Cloud Open, scalable, reliable SW Development Environment Linux, OpenMPI, Lustre, Slurm Administration & monitoring bullx supercomputer suite Servers Full range development from ASICs to boards, blades, racks Support for accelerators Infrastructure Data Center design Mobile Data Center Water-Cooling 4

5 Leading HPC technology with Bull TERA st European PetaFlop-scale System Rank #6 CURIE st PRACE PetaFlop-scale System Rank #9 C st Intel Xeon E v2 System Direct Liquid Cooling Technology 5

6 Energy (Electricity): a significant part of HPC budget 6

7 /kwh Industrial Electricity Prices in Europe Electricity prices highly variable across Europe Avg 0.11 /kwh 0,12 0,10 0,08 Electricity Industrial Prices in UE 0,06 0,04 0,02 0,00 Allemagne Espagne France Italie Pays-Bas Pologne Royaume-Uni Norvège Electricity prices Rising steadily CAGR 12% Source Eurostat year

8 Power to the datacenter MEGAWATTS 8

9 bullx B700 series Direct Liquid Cooling (DLC) rear Max config Processors 6 PF 12 PF Direct Liquid Cooling DLC rack: Dual-pump unit (80 kw cooling capacity) Power Supply Unit + UltraCapacitor opt. Rack management (incl. Gigabit Ether.) + 5 chassis, each including: 18 dual-processor nodes Embedded 1st level InfiniBand switch Extra Embedded Gigabit Ethernet switch Silent Extra-easy maintenance Optimized PUE (< 1.1) Accelerators 20 PF 28 PF 9

10 Cooling & Power Usage Effectiveness (PUE) A / C Prototype tested not developped for chassis test Air-cooled Water-cooled door Direct-Liquid-cooling We designed a flexible cold plate to cool the two CPUs and the IOH 20 kw/rack 30 kw/rack 70 kw/rack We choose an adapted technology with manufacturer : - all copper - same fixations as for heatsink (so the tube avoids the IOH fixation) - only half tube exchange heat - inner diameter : 4 mm Easy to cool CPU with hot water : - 45 C water blade inlet => Tcase <70 C Co-generation 10

11 Direct Liquid Cooling Infrastructure With hot water cooled servers, water chillers are not required anymore hot air Water chiller Cooling Tower 42U rack bullx DLC x 90 CN bullx DLC cold air 45 C x N up to 35 C 11

12 Where do all these Watts go? DDR3 (x8) Connector to backplane HDD/SSD 2x Xeon-EP Fans 12

13 Node consumption varies with workload Wrt Linpack (max -> 100%) Memory streaming 75% Irregular memory access 55% Iddle 25% Using turbo is never energy efficient 13

14 Power Management Accounting Users billed separately for CPU, IO, and Energy Keep compute center electricity bill within budget Control power Avoid running over capacity Allow for priority jobs Adjust power consumption with electricity cost Energy consumption / cost optimization Fine & precise power monitoring Power data analysis Control all system resources power enter software Application Management Development Environment bullx DE Execution Environment bullx BM (Batch Management) bullx MPI (Message Passing Interface) Supercomputer Management Management Center bullx MC Software Manager 1 SIR Monitoring & Control Manager Infrastructure Manager Data Management Parallel File System bullx PFS (Lustre) Network File System NFS Local File System Operating System bullx Linux - Red Hat Entreprise Linux (RHEL) - SUSE Linux Entreprise Server (SLES) 14

15 Bull - TU Dresden high frequency monitoring Admin (2) Cluster Instrumented blades IB Fabric / GigE Fabric FASS Phase 1 SATA / SAS Login (2) Export (2) SMP (2) GPU Cluster 32 Nodes 48x Kepler VR CPU VR DIMM1 VR DIMM1 VR CPU VR DIMM2 VR DIMM2 VR DIMM3 VR DIMM3 CPU1 VR DIMM4 VR DIMM4 CPU2 FPGA Cluster Manager i2c - 400Kb / VR Power Measurement i2c - 400Kb Power Monitor for Node Global Power API Ethernet 100Mb BMC (Baseboard Management Controller) XBUS 50Mb FPGA Operating System (OS) NODE Freq: 500Hz - Error: 2% 15

16 Energy efficient HPC systems Green systems Interest driven by energy cost and green attitude Green systems start with Green components CPUs Power only where/when needed, throttle frequency Memory, DDR4 saves 20-30% PSUs, optimize AC/DC, DC/DC conversion Interconnect Direct Liquid Cooling save on CAPEX (chillers) & OPEX (electricity) Non-intrusive high definition power monitoring Power: another parameter in system usage optimization Energy aware batch scheduler 16

17 17