High Performance Computing at the Jülich Supercomputing Center

Transcription

1 Mitglied der Helmholtz-Gemeinschaft High Performance Computing at the Jülich Supercomputing Center Jutta Docter Institute for Advanced Simulation (IAS) Jülich Supercomputing Centre (JSC)

2 Overview Jülich Supercomputing Center at Forschungszentrum Jülich JUQUEEN Installation Administration Usage Applications Fileserver PRACE (Partnership for Advanced Computing in Europe) DeIC, , Jutta Docter, JSC 2

3 Where is Jülich? Jülich * DeIC, , Jutta Docter, JSC 3

4 Forschungszentrum Jülich (FZJ) JSC DeIC, , Jutta Docter, JSC 4

5 FZJ at a Glance (2012) Budget: 557 mio. (thereof third-party funding: 173 mio. ) Staff: 5,200 (thereof 1,650 scientists) Visiting scientists: 860 from 40 countries Trainees: 90 per year Publications: 2,200 per year Protective rights and licences: 16,892 Industry cooperations: 363 Research fields: health, energy and environment, information technology; key technologies for tomorrow DeIC, , Jutta Docter, JSC 5

6 Jülich Supercomputing Centre (JSC) DeIC, , Jutta Docter, JSC 6

7 Jülich Supercomputing Centre Supercomputer operation for: Centre FZJ Regional JARA National NIC, GCS Europe PRACE, EU projects Application support Traditional and SimLab support model Scientific visualization Peer review support and coordination R&D work Methods and algorithms, performance analysis and tools Community data management service Computer architectures, Exascale Laboratories with IBM, Intel, NVIDIA Education and Training DeIC, , Jutta Docter, JSC 7

8 Gauss Centre for Supercomputing German representative in PRACE Alliance of the three German national supercomputing centres Jülich Supercomputing Centre (JSC) Leibniz-Rechenzentrum (LRZ), Munich Höchstleistungsrechenzentrum Stuttgart (HLRS) Support of computational science through Multi-Petaflop/s supercomputers Multi-Petabyte storage Multi-Gigabit networking infrastructure Large-Scale projects DeIC, , Jutta Docter, JSC 8

9 User JSC - Overview Cross-Sectional Teams Methods& Algorithms Application Optimization Parallel Performance Simulation Laboratories Biology Climate Science Molecular Systems Plasma Physics Solid & Fluid Engeneering DeIC, , Jutta Docter, JSC 9

10 JSC Supercomputer Systems: Dual Track Approach 2004 IBM Power 4+ JUMP, 9 TFlop/s IBM Power 6 JUMP, 9 TFlop/s IBM Blue Gene/L JUBL, 45 TFlop/s 2009 JUROPA 200 TFlop/s HPC-FF 100 TFlop/s IBM Blue Gene/P JUGENE, 1 PFlop/s JUROPA++ Cluster, 1-2 PFlop/s + Booster General-Purpose File Server GPFS, Lustre Highly-Scalable IBM Blue Gene/Q JUQUEEN 5.9 PFlop/s DeIC, , Jutta Docter, JSC 11

11 Transition: JUGENE JUQUEEN JUGENE Feb 2012 Mar 2012 Apr 2012 May 2012 Jun 2012 Jul 2012 BG/P Production User Access Production Production Production Hardware Installation JUQUEEN Apr 2012 May 2012 Jun 2012 Jul 2012 Aug 2012 Sep 2012 Oct 2012 Nov 2012 Dec 2012 Jan 2013 Feb 2013 DeIC, , Jutta Docter, JSC 12

12 JUQUEEN, 8 Racks, April 2012 DeIC, , Jutta Docter, JSC 13

13 JUQUEEN, 24 Racks, November 2012 Top 500, Nov. 2012: Pos. 5 DeIC, , Jutta Docter, JSC 14

14 JUQUEEN, 28 Racks, January 2013 Top 500, Jun. 2013: Pos. 7 DeIC, , Jutta Docter, JSC 15

15 JUQUEEN - Hardware - Cables 58.9 t of computational hardware 3,584 Torus data cables (29.7 km) GE cables (18 km) 84 Ethernet cables (3.4 km) 1 DeIC, , Jutta Docter, JSC 16

16 JUQUEEN Hardware - Power 4.4 km power cables, total weight 7.9 t 112 power connections (3 phases) 336 circuit breakers 255 m steel for 7 frames (rack: 2t) 200 m cable trays DeIC, , Jutta Docter, JSC 17

17 JUQUEEN - Hardware - Water 280 m pipes (stainless steel), separate valves per row 2 pumps with a max. flow rate of 210 m³ each Special pump control system for redundancy 18 C supply temperature - demineralized water 27 C return temperature Supply rate: 28 gal/min per rack 1 DeIC, , Jutta Docter, JSC 18

18 DeIC, , Jutta Docter, JSC 19

19 Source: IBM DeIC, , Jutta Docter, JSC 24

20 JUQUEEN Configuration 28 Racks Blue Gene/Q (7 x 4 racks) Nodes: 28,672 (Cores: 458,752) Nodeboards: 896 Main memory: 448 TB (16 GB per node) Overall peak performance: 5.9 Petaflops Power consumption: kw per rack (av. 63 kw) Processor: 16 cores per node, IBM PowerPC A2 (1.6 GHz, 64 bit) 16-way SMP processor, quad floating point unit Internal network: 5D Torus (A,B,C,D,E) - 40 GBps, 2.5 μsec latency DeIC, , Jutta Docter, JSC 25

21 Blue Gene/Q node board (with water cooling) Source: IBM DeIC, , Jutta Docter, JSC 26

22 Blue Gene/Q Compute Card Heat spreader DeIC, , Jutta Docter, JSC 27

23 JUQUEEN I/O Drawer DeIC, , Jutta Docter, JSC 28

24 JUQUEEN Environment 248 I/O Nodes (8 per drawer, on top of racks): 27 x x 32 on IO rich rack ( 1 ION per nodeboard) connected to 2 CISCO Switches Nexus Front-End Nodes (Red Hat) for user login and data transfer: juqueen@fz-juelich.de 1 Service Node (Red Hat) for BG software and DB2 database 1 Backup Service Node - to be manually activated IBM p7 740, 8 cores (3.55 GHz), 128 GB memory, local storage device DS5020 (16 TB) DeIC, , Jutta Docter, JSC 29

25 JUQUEEN Environment JUQUEEN BG Control-System Backup Service Node Service Node FrontEnd FrontEnd RAID DB2 runjob Fileserver JUST Nexus Switches FrontEnd FrontEnd SSH DeIC, , Jutta Docter, JSC 30

26 JUQUEEN Cooling System R03 R13 R23 R33 R43 R53 R63 R02 R12 R22 R32 R42 R52 R62 R01 R11 R21 R31 R41 R51 R61 R00 R10 R20 R30 R40 R50 R60 Heat Exchanger Operation: 5,5 bar Down at: 0,5 bar Operation: 2,3 bar Warning: 1,5 bar No autom. shutdown DeIC, , Jutta Docter, JSC deionized water 32 P P 3 m³/min.

27 Blue Gene/Q - Coolant Monitor per Rack DeIC, , Jutta Docter, JSC 33

28 Cooling Issues Learning Curve! Cooling problems result in complete outage! External water issues, far to hot (32 C) or to cold (7 C) Pressure loss during pump switch-over reconfiguration to run both pumps in parallel Pressure loss during work in outer circle Loss of water pressure, warnings, air bulbs in system, sensor wrong alignment? replacement of empty nodeboards? no, but prefill installation of airing tools to eliminate bulbs checking of pressure variations DeIC, , Jutta Docter, JSC 34

29 Hardware Failures and Monitoring Node or nodeboard failure nodeboard in error (midplane not available) replace No refund for aborted user jobs (write checkpoints!) Correctable errors run diagnostics replace(?) Cable failure use integrated redundancy Preventative: run diagnostics as batch jobs BG/Q TEAL generates Alerts + sends messages to sysadmin Icinga Server, JSC checking scripts: send messages to sysadmin alarm people on duty (8:00-24:00, Mon-Sat) DeIC, , Jutta Docter, JSC 35

30 Hardware Replacements Regular: about 57 nodes and 7 nodeboards per month (of Nodes - MTTF >1 year) overhead: diagnostic, report, order, replace, watch, doc., Some IO drawers and midplanes (delicate pins) Service card outage on master clock rack single point of failure Additional preventive replacements: Manufacturing defects on G compute nodes Bulk Power Modules (load distribution) DeIC, , Jutta Docter, JSC 36

31 System Administration JSC Blue Gene Administration Team (3.5 PM) System administrator of the week JSC application support team IBM coverage: Mon - Fri 8:00 17:00 On site HW + SW personnel DeIC, , Jutta Docter, JSC 37

32 (New) Software Issues Blue Gene Software Updates (+ efixes) June 2012 V Jan V Aug V Mellanox optical driver firmware update (3x) GPFS, LoadLeveler, Compiler,. IO nodes aborted with Out of memory adjustment / optimization of parameters (pagepool, buffers, etc ) DeIC, , Jutta Docter, JSC 38

33 IO Performance - IO Nodes to GPFS IO optimization, GPFS ( ) ongoing at FZJ (and Argonne Nat. Lab.) Read is worse than write - bad to shared files Testing lower layers first network setup is o.k. Optimizing GPFS performance parameters Setup of special GPFS / network testing environment Regular teleconferences with IBM need to bring experts together! DeIC, , Jutta Docter, JSC 39

34 JSC LinkTest: Blue Gene torus link bandwidth tester All-to-all ping-pong test Bandwidth distribution Intra-node communication Communication via link A, B, C, D Communication via link E DeIC, , Jutta Docter, JSC 40

35 JSC llview DeIC, , Jutta Docter, JSC 41

36 LoadLeveler Batch Scheduler - Job Classes Class/Queue Max. nodes Wall Clock Limit Priority m048 / m / h on demand only m h on demand only m h m h m h m h m h n h n h n min n min serial 0 60 min juqueenx DeIC, , Jutta Docter, JSC 42

37 Blue Gene Navigator n004, n008 (ongoing replacements) n001, n002 DeIC, , Jutta Docter, JSC 43

38 LoadLeveler - New version First and biggest BG/Q with LoadLevler testing on site! Negotiator dies (and restarts automatically) Reservations bigger than one midplane are not honored Jobs run on reserved partitions Erroneously waiting for missing cables, restart helps Not scheduling jobs although resources are free (not checking all possibilities in 5D Torus) stabilizing August 2013 DeIC, , Jutta Docter, JSC 44

39 DeIC, , Jutta Docter, JSC 45

40 JUQUEEN User Groups Germany May 2013 DeIC, , Jutta Docter, JSC 48

41 Tools are ported Linktest Scalasca (application profiles and traces) SIONlib Trace analysis by Scalasca on Blue Gene/Q DeIC, , Jutta Docter, JSC 49

42 SIONlib - Scalable I/O library for parallel access to task-local files Supports writing and reading binary data to or from thousands of processors into a single or a small number of physical files General structure of a SION file: Starting positions of the blocks are aligned to the filesystem blocksize All writing and reading is done asynchronously Allows parallel access using MPI, OpenMp, or their combination, and sequential access for post-processing utilities DeIC, , Jutta Docter, JSC 50

43 High-Q Club Highest Scaling Codes on JUQUEEN Goals: Promote the idea of exascale capability computing Showcase for codes that can utilise the entire 28-racks Invest in tuning and scaling the codes Show that they are capable of using all 458,752 cores Aiming at more than 1 million concurrent threads Use a variety of programming languages and parallelisation models, demonstrating indiv. approaches to reach that goal Important milestone in application development towards future HPC (exascale) systems DeIC, , Jutta Docter, JSC 51

44 High-Q Club Members dynqcd Lattice Quantum Chromodynamics with dynamical fermions, JSC Gysela A gyrokinetic code for modelling fusion core plasmas, CEA, France PEPC A particle tree code for solving the N-body problem for Coulomb, gravitational and hydrodynamic systems, JSC PMG+PFASST A space-time parallel multilevel solver, Univ. of Wuppertal / LBNL Terra-Neo A multigrid solver for geophysics applications from the Univ. of Erlangen. walberla A widely applicable Lattice Boltzmann solver from the Univ. of Erlangen. DeIC, , Jutta Docter, JSC 52

45 JUQUEEN Porting and Tuning Workshop First workshop February 2013 Second workshop February DeIC, , Jutta Docter, JSC 53

46 Disks, Tapes, Robots DeIC, , Jutta Docter, JSC 54

47 Data Growth at JSC User Data GPFS Userdata GPFS 13000, , , , , ,00 Terabyte 7000, , , , , , ,00 0, Disk Tape DeIC, , Jutta Docter, JSC 55

48 Jülich Storage Server - JUST (~ 10 PB online) $WORK $HOME $ARCH 7.1 PB $DATA 3 x 350 TB 2 x 700 TB 1 x 350 TB 160 GB/sec Management Server JUVIS JUDGE JUQUEEN JUROPA TSM Server DeIC, , Jutta Docter, JSC 57

49 Tape Storage Tapes (44.5 PB) 3000 TSM Clients (Campus) IBM TSM Server / ACSLS Server JUQUEEN Juropa Judge SAN SAN Separate buildings Data Migration Data Archive Data Backup Priv. Tape Network ORACLE SL T10K A/B/C ORACLE SL T10K B/C Cartridge Capacity: max. 5 TB Transfer Rate: up to 240 MB/sec DeIC, , Jutta Docter, JSC 60

50 PRACE Partnership for Advanced Computing in Europe Consists of 25 European partner states, each represented by one institution DeIC, , Jutta Docter, JSC 61

51 Goals Prepares the creation of a persistent, sustainable pan-european HPC service Prepares the establishment of Tier-0 supercomputing centres at different European sites Defines and establishes a legal and organisational structure involving HPC centres, national funding agencies, and scientific user communities Develops funding and usage models and establishes a peer review process Provides training for European scientists and creates a permanent education programme DeIC, , Jutta Docter, JSC 62

52 Tier-0 Systems - today Curie, Bull Bullx cluster (France) Fermi, IBM Blue Gene/Q (Italy) Hermit, Cray XE6 (Germany) JUQUEEN, IBM Blue Gene/Q (Germany) MareNostrum, IBM System X idataplex (Spain) SuperMUC, IBM System X idataplex (Germany) DeIC, , Jutta Docter, JSC 63

53 JUQUEEN Granted PRACE Projects - May 2013 DeIC, , Jutta Docter, JSC 64

54 PRACE Calls for Proposals for Project Access Twice per year: Call opens in February: Access is provided starting September of the same year. Call opens in September: Access is provided starting March of the next year. The PRACE 8th Call for Project Access is now open until 15 October DeIC, , Jutta Docter, JSC 66

55 QUESTIONS? DeIC, , Jutta Docter, JSC 67