Vienna Scientific Cluster: Problems and Solutions

Transcription

1 Vienna Scientific Cluster: Problems and Solutions Dieter Kvasnicka Neusiedl/See February 28 th, 2012

2 Part I Past VSC History

3 Infrastructure Electric Power May 2011: 1 transformer 5kV Now: 4-5 transformer 10kV 15m cable length from transformer Air Conditioning Mobile conditioning until December 2011 Now on top of Steinmetzhalle : 3 units Humidity Cool doors + mobile humidifier Room air conditioning: 3 units, 2 with humidifier UPS (=USV) Server Network Switches VSC History Infrastructure Compute Nodes/Racks Server

4 Compute Nodes/Racks Cooldoors Visit from Knürr Cabling Well structured, (nearly) without problems Processors No problems: 2x AMD Opteron Magny Cours 6132HE, 8 cores, 2.2 GHz Memory TOP-500 tests with matrix size : runtime few hours Number 56 in TOP-500 list June 2011 (best Austrian ever) Better performance with size : Only possible after complete memory change (> DIMMs) Two Austrian computers in current TOP-500 list: VSC-1 (443) and VSC-2 (77) VSC History Infrastructure Compute Nodes/Racks Server

5 Compute Nodes/Racks (cont.) Power supply 10-20% faulty Harddisks Many faults during installation new firmware Networks Infiniband: 89 switches (36 ports) Gigabit ethernet ( Service Network ): 32 switches IPMI ( Intelligent Platform Management Interface ) via Ethernet: 33 switches Missing node cooling problem VSC History Infrastructure Compute Nodes/Racks Server

6 Shutdowns Electric power Construction work Cooling Maintenance VSC History Infrastructure Compute Nodes/Racks Server

7 Server 3 Login 8 FHGFS 6 NFS 2 Virtualization 2 Appliance VSC History Infrastructure Compute Nodes/Racks Server

8 Server Problems (solved) Connection to TU and Uni not yet final (Boku is/will be connected virtually) Problem: Uni connect allowed only packet size < 230 Byte firmware upgrade on backbone switch Login nodes: wrong SFP type (copper/fiber optic) exchange Appliances should have served \opt via glusterfs did not scale disabled VSC History Infrastructure Compute Nodes/Racks Server

9 Part II Present

10 Remaining hardware problems Memory Modules 2 modules per week no slow memory dying as in VSC-1 Processors 2 per month, mainly memory controller Motherboards 2 per month, mainly network controller Remaining hardware problems

11 Remaining hardware problems (cont.) Harddisks Nearly no problem until a week ago 6 during last week Power Supplies 1 per week Remaining hardware problems

12 Remaining hardware problems (cont. 2) Clustsafe Energy management and current distribution with network connection 15 nodes per Clustsafe / 90 Clustsafes 1 per week 5 nodes (+10 nodes) Remaining hardware problems

13 Remaining software problems Same software on all nodes (1330) Scripts for software roll out Checks at every reboot rollout on all nodes nearly no time NFS server Equal distribution of user load to servers Faulty nodes Check and categorize Performance Shared memory communication Core files (core.*) Depending on MPI version ulimit -c 0 Vergabeassistent Ongoing improvements Remaining software problems Available software Hints

14 MPI Select MPI Implementation mpi-selector / mpi-selector-menu Recommendation Intel MPI (starts faster than when using many nodes) Bug warning: over 1000 cores: export I MPI HYDRA BRANCH COUNT=130 Mvapich (with Intel compiler) OpenMPI (1.4.3, not yet 1.5.4) Recompile when changing MPI version Remaining software problems Available software Hints

15 MPI (cont.) Some parameters are different Some versions support Tight integration : no machine file and number of cores required switch off tight integration with: export PE HOSTFILE= Check job status when changing job scripts qstat -ext column CPU time qstat -f grep -B 1 $USER check load and 0/8/8 / 0/12/12 / 0/16/16 (reserved/used/total) qrsh/qlogin interactive login to node Remaining software problems Available software Hints

16 Program Libraries BLAS: Goto or MKL (Intel Math Kernel Library) Lapack / ScaLAPACK: MKL FFT: MKL (+ FFTW-Wrapper) HDF5: Parallel I/O... Remaining software problems Available software Hints

17 Hints Process pinning Supported by MPI Intel MPI: export I MPI PIN PROCESSOR LIST=0-15 Others: mpirun parameter (e.g. --bind-to-core) Sequential: taskset -c 3 myexecutable parameter (ignore warning) Compiler: Intel (several), gcc, open64 Reservation: 3 large jobs get reserved nodes Maximum run time: 3 days Longer requests remain in queue forever Graphics on VSC-1/VSC-2? Remaining software problems Available software Hints

18 Hints (cont.) Temporary files $GLOBAL /global/lvx/user FHGFS, shared $SCRATCH /scratch FHGFS, single node $TMPDIR /tmp/... tmpfs (in memory) not in qrsh/qlogin Memory requirements 2 GB/core on VSC-2 (16 cores per node) 3 GB/core on VSC-1 8-core-nodes 2 GB/core on VSC-1 12-core-nodes 4 GB/core on VSC-2, if 8 cores/node are used (-pe mpich8) 8 GB/core on VSC-2, if 4 cores/node are used (-pe mpich4) process pinning is important Remaining software problems Available software Hints

19 Hints (cont. 2) Queue length VSC cores average in use: 3960 cores average queue length: cores VSC cores (+ 64 spare) average in use: cores average queue length: 0 cores single core is 25 % slower than on VSC-1 Sequential single core jobs only on VSC-1 Nodes on VSC-2 are exclusive Remaining software problems Available software Hints

20 Support Wiki: address: Remaining software problems Available software Hints