Best Practices JUGENE. Florian Janetzko, Jülich Supercomputing Centre (JSC) Institute for Advanced Simulation, Forschungszentrum Jülich

Transcription

1 Best Practices JUGENE Florian Janetzko, Jülich Supercomputing Centre (JSC) Institute for Advanced Simulation, Forschungszentrum Jülich

2 Outline A brief Overview Best Practice JUGENE System Architecture and Configuration Production Environment Basic Porting Tuning Applications Performance Analysis Debugging 2

3 Outline A brief Overview Best Practice JUGENE System Architecture and Configuration Production Environment Basic Porting Tuning Applications Performance Analysis Debugging 3

4 Forschungszentrum Jülich 4

5 Jülich Supercomputing Centre (JSC) 5

6 Tasks of the JSC Operation of the supercomputers for local, national and European scientists. User support in programming and optimisation of applications Support of research groups from different research domains by means of simulation laboratories Research & Development work on new computer architectures, algorithms, performance analysis and tools, GRID computing, etc. Education and training of users, education of students (bachelor and master courses, PhD programmes) 6

7 Integration of JSC in Networks and Alliances Jülich Aachen Research Alliance (JARA) Initiative of RWTH Aachen and Forschungszentrum Jülich Gauss Centre for Supercomputing (GCS) Alliance of 3 German centres Höchstleistungsrechenzentrum Stuttgart (HLRS) Jülich Supercomputing Centre (JSC) Leibnitz-Rechenzentrum (LRZ) German representative in PRACE John von Neumann Institute for Computing (NIC) Joint foundation of 3 Helmholtz Centres Deutsches Elektronensynchrotron (DESY) Gesellschaft f. Schwerionenforschung (GSI) Jülich Supercomputing Centre (JSC) 7

8 2004 Supercomputer Systems: Dual Concept IBM Power 4+ JUMP, 9 TFlop/s 2005/6 IBM Blue Gene/L JUBL, 45 TFlop/s 2007/8 IBM Power 6 JUMP, 9 TFlop/s File Server GPFS IBM Blue Gene/P JUGENE, 223 TFlop/s 2009 Intel Nehalem Clusters JUROPA 200 TFlop/s HPC-FF 100 TFlop/s File Server GPFS, Lustre IBM Blue Gene/P JUGENE, 1 PFlop/s General-Purpose Highly-Scalable 8

9 User Information and Support Information about JUGENE JSC websites: PRACE JUGENE Best Practice Guide: Practice-Guides?lang=en Dispatch and User Support Applications for accounts Forschunszentrum Jülich GmbH,JSC, Dispatch, Jülich Tel: , Fax: User Support Tel:

10 Outline A brief Overview Best Practice JUGENE System Architecture and Configuration Production Environment Basic Porting Tuning Applications Performance Analysis Debugging 10

11 Blue Gene/P Design Goals Optimal performance / watt ratio Transparent high-speed reliable network Easy programming based on standard message passing interface (MPI) Extreme scalability ( > 512K cores) High reliability 11

12 Blue Gene/P Processor and Chip Microprocessor PowerPC bit microprocessor 850 MHz Double-precision floating-point multiply-add unit (double FPU) 3.4 GFLOP/s Chip Contains 4 cores Peak performance of 13.6 GFLOP/s 12

13 Blue Gene/P Compute Card (Node) Blue Gene/P compute ASIC 4 cores, 8MB cache Cu heatsink SDRAM DDR2 2 or 4 GB memory Node card connector network, power 13

14 Blue Gene/P Node (schematically) 14

15 Blue Gene/P Node Card 32 Compute cards Up to 2 I/O nodes Air-cooled 15

16 JUGENE Building Blocks Midplane 16 Node Cards Cabled 8x8x8 ~7 TF/s, 1 TB Node Card (32 chips 4x4x2) 32 compute, 0-1 IO cards 435 GF/s, 64 GB Rack 32 Node Cards Cabled 8x8x TF/s, 2 TB System 72 Racks (9x8) 1 PF/s (825.5 TF/s LINPACK), 144 TB Largest TORUS: 9x4x4 Chip 4 processors 13.6 GF/s Compute Card 1 chip, 13.6 GF/s 2 GB DDR2 16

17 Different Blue Gene/P Nodes Overview Login Nodes (3) IBM p6 550 processor (8 cores each node) 4.2 GHz 128 GB RAM SuSE Linux (SLES 10) Internet address: jugene.fz-juelich.de Compute Nodes (73,728) PowerPC bit microprocessor (4 cores each node) 850 MHz 2 GB RAM Compute Node Kernel (CNK, a light-weight kernel) Only accessible via LoadLeveler 17

18 Different Blue Gene/P Nodes Overview Service Nodes (2) IBM p6 550 processor (8 cores each node) 4.2 GHz 128 GB RAM SuSE Linux (SLES 10) Not accessible for users I/O Nodes (600) Hardware similar to the compute nodes Responsible for handling I/O No login possible for users 18

19 Blue Gene/P Networks 3-Dimentional Torus Communication Backbone for applications Bandwidth: 425 MB/s per link (5.1 GB/s bidirectional per node) Usable for jobsizes 512 nodes (1 midplane) Collective Network (Global Tree) One-to-all broadcast functionality Reduction operations functionality 850 MB/s per link Interconnects all compute and I/O nodes Other Networks Global Barrier and Interrupt network External I/O network (10 GE, external communication) Control network (1 GE, service/support purposes) 19

20 JUGENE I/O Subsystem I/O exclusively via I/O nodes Distribution of the 600 I/O nodes 71 racks equipped with 4 I/O nodes per midplane (1 per 128 CN) 1 rack equipped with 16 I/O nodes per midplane (1 per 32 CN) Connected to a GPFS (General Parallel File System, IBM) 28 IBM p6 servers Around 6000 disks 20

21 JUGENE File System Classes Home ($HOME) Accessible from FEN and CN Backup Group limits: 6 TB, 2 mio files Bandwidth: ~8.5 GB/s (using 4 midplanes) Scratch ($WORK) Accessible from FEN and CN No Backup Group limits: 20 TB, 4 mio files Bandwidth: ~33 GB/s (using 16 midplanes) Caution: Files older than 90 days on $WORK will be deleted without warning! 21

22 JUGENE File System Classes Archive ($ARCH) Accessible from FEN only Use for storage of large files (only for the lifetime of your project!) Group limits: no limits, however if you need more then 20TB please contact us Bandwidth: ~2.8 GB/s Important: Use large files only (use tar as container for many small files): Retrieving 1 file from tape takes 2 minutes + time for transfer Do NOT rename files in $ARCH 22

23 Outline A brief Overview Best Practice JUGENE System Architecture and Configuration Production Environment Basic Porting Tuning Applications Performance Analysis Debugging 23

24 Module Environment Access to software via the module command module <options> Option Description <no option> avail list load <module> help <module> Lists available options Lists all available modules Lists modules currently loaded Load <module> Lists information about <module> show <module> Module categories Information about settings done by the module <module> COMPILER, IO, MATH, MISC, SCIENTIFIC, TOOLS 24

25 Software Software for front-end and compute nodes Front-end nodes: /usr/local Compute nodes : /bgsys/local Installed software packages (selection) CPMD, GPAW, Gromacs, LAMMPS, Namd BLACS, BLAS, FFTW, LAPACK, PETSc, ScaLAPACK, SPRNG DDT, HDF5, netcdf, Scalasca, SIONlib, Tau, Totalview, Vampir 25

26 Running Simulations Batch System Execution of applications managed by LoadLeveler Command llsubmit <jobfile> llq llcancel <job ID> llstatus llclass llqx Description Sends job to the queuing system Lists all queued and running jobs Kills the specified job Displays the status of LoadLeveler Lists existing classes and their properties Shows detailed information about all jobs 26

27 Running Simulations Job command files Two blocks of instructions 1. LoadLeveler keyword block Keywords start with Determines the job configuration (number of nodes, wall time, network, etc.) 2. Applcation block Regular shell script block Can contain any shell command and call(s) to applications 27

28 Running Simulations mpirun command Launch command for parallel applications mpirun <options> Option -args prg_args -env var=value -env_all -exe <executable> -mapfile -mode -verbose Description Passes prg_args to the launched application on the CN Sets the environment variable var to value Exports all environment variables to the job on the CN Specifies the executable Specifies the process mapping (default: XYZT) Specifies the execution mode (default: SMP) Specifies the verbosity level 28

29 Running Simulations Execution Mode I SMP (Symmetrical MultiProcessing) Default mode Each CN executes 1 (MPI) process Up to 4 (OpenMP/POSIX) threads per process Maximum amount of memory per process (2 GB) Shared memory between all threads 29

30 Running Simulations Execution Mode II DUAL Each CN executes 2 (MPI) processes Up to 2 (OpenMP/POSIX) threads per process 1 GB per process available Shared memory between threads of the same (MPI) process 30

31 Running Simulations Execution Mode III VN (Virtual Node) Each CN executes 4 (MPI) processes Intended for MPI-only parallelization scheme 0.5 GB per process available Shared memory between threads of the same (MPI) process Caution: Pure MPI applications must use this mode on JUGENE 31

32 Running Simulations Job example I Pure MPI applications #!/bin/bash #@job_name = example_1 #@comment = Pure MPI application" #@error = $(job_name).$(jobid).err #@output = $(job_name).$(jobid).out #@environment = COPY_ALL #@wall_clock_limit = 24:00:00 #@notification = error #@notify_user = my_address@my.institution #@job_type = bluegene #@bg_size = 2048 #@bg_connection = TORUS #@queue mpirun -mode VN -mapfile TXYZ -verbose 2 -exe application.x 32

33 Running Simulations Job example II Hybrid MPI/OpenMP applications #!/bin/bash = example_2 = MPI/OpenMP application" = $(job_name).$(jobid).err = $(job_name).$(jobid).out = COPY_ALL = 12:00:00 = complete = #@job_type = bluegene #@bg_size = 1024 #@bg_connection = TORUS #@queue mpirun -verbose 2 env OMP_NUM_THREADS=4 -exe application.x 33

34 Interactive Access to CN Direct login to compute nodes not possible Job under interactive control with llrun Runs under LoadLeveler Up to 30 minutes in general At most 256 nodes Invocation: llrun [llrun_options] [mpirun_options] executable 34

35 Outline A brief Overview Best Practice JUGENE System Architecture and Configuration Production Environment Basic Porting Tuning Applications Performance Analysis Debugging 35

36 CompilersI Different compilers for front-end and compute nodes GNU and IBM XL family of compilers available Tip: It is recommended to use the XL suite of compilers since they produce in general better optimized code. Language XL Compiler invocation (CN code) C bgxlc, bgc89, bgc99 mpixlc C++ bgxlc++, bgxlc mpixlcxx XL MPI wrapper (thread-safe: *_r) Fortran bgxlf, bgxlf90, bgxlf95, bgxlf2003 mpixlf77, mpixlf90, mpixlf95, mpixlf

37 Compilers II Language GNU Compiler invocation (CN code) C powerpc-bgp-linux-cc mpicc C++ powerpc-bgp-linux-g++ mpicxx Compilers for front-end node GNU MPI wrapper (thread-safe: *_r) Fortran powerpc-bgp-linux-gfortran Mpif77, mpif90 Language XL Compiler Family GNU Compilers C xlc gcc C++ xlc++, xlc g++ Fortran xlf, xlf90, xlf95, xlf2003 gfortran 37

38 Compiler Flags (XL Compilers) Flags in order of increasing optimization potential Optimization Level Description -O2 qarch=450 qtune=450 Basic optimization -O3 qstrict qarch=450 qtune=450 More aggressive, not impact on acc. -O3 qhot qarch=450 qtune=450 More aggressive, may influence acc. (high-order transformations of loops) -O4 -qarch=450 qtune=450 Interprocedural optimization at compile time -O5 -qarch=450 qtune=450 Interprocedural optimization at link time, whole program analysis 38

39 ESSL Engineering and Scientific Subroutine Library (IBM) High-performance mathematical libraries Tuned for Blue Gene/P Can be called from C, C++, and Fortran Multi-threaded version (SMP) available Highly recommended if you use BLAS or LAPACK routines mpixlf90_r o prog.x prog.f L/bgsys/local/lib lesslbg mpixlg90_r o prog.x prog.f L$(LAPACK_LIB) llapack \ L/bgsys/local/lib lesslbg Caution: Pay attention to the order of libraries when linking. The last symbol found by the linker will be used. 39

40 MPI Message-passing Interface Based on MPICH2 implementation MPI 2.1 Standard, BUT Spawning of tasks is not supported Asynchronous (non-blocking) I/O (e.g. MPI_File_iwrite, MPI_File_iread) is not supported Compiling applications: MPI wrappers see compilers Running applications: a. Using mpirun in batch jobs ( LoadLeveler, mpirun) b. Using llrun for interactive jobs ( llrun) 40

41 OpenMP Open Multi-Processing XL compiler family Full OpenMP 2.5 standard is supported Compiler flag: -qsmp=omp Use thread-safe compiler version (e.g. mpif90_r, mpixlc_r, etc.) GNU compiler family Default GNU compiler (4.1.2) does not support OpenMP, use version (module load gcc/4.3.2) Full OpenMP 2.5 standard is supported Compiler flag: -fopenmp Execution Modes: SMP (4 threads per task) DUAL (2 threads per task 41

42 Outline A brief Overview Best Practice JUGENE System Architecture and Configuration Production Environment Basic Porting Tuning Applications Performance Analysis Debugging 42

43 Diagnostic Compiler Flags (XL Compilers) Diagnostic messages are given on the terminal and/or in a separate file -qreport: compilers generate a file name.lst for each source file -qlist: compiler listing including an object listing -qlistopt: options in effect during compilation included in listing Listen to the compiler -qflag=<listing-severety>:<terminal-severety> i: informal messages, w: warning messages, s: severe errors Use -qflag=i:i to get all information -qxflag=diagnostic 43

44 Example 1: Compilers Diagnostics subroutine mult(c,a,ndim) implicit none integer :: ndim,i,j double precision :: a(ndim),c(ndim,ndim)! Loop do i=1,1000 do j=1,1000 enddo enddo c(i,j) = a(i) end subroutine mult >>>>> LOOP TRANSFORMATION SECTION <<<<< 1 SUBROUTINE mult (c, a, ndim) [...] 10 IF (.FALSE.) GOTO lab_9 [...] $.CIV2 = 0 Id=1 DO $.CIV2 = $.CIV2, Id=2 DO $.LoopIV1 = $.LoopIV1, 999 [...] Loop interchanging applied to loop nest Outer loop has been unrolled 2 time(s) 44

45 Single-Core Optimization Compiler Flags -On (n=0,1,2,3,4,5) Activates different optimization options Check results! -qfloat=<suboption> Performance gain, but may result in IEEE 754 non-compliant code [no]maf combined multiple-add instructions [no]rrm rounding-to nearest mode -qipa=<suboptions> inline=<suboptions> procedure1[,procedure2[, ]] inline specified routines level=[0 1 2] level of inter procedural analysis 45

46 Single-Core Optimization Double FPU -qarch=450d Activates SIMD (Single Instruction Multiple Data) mode (vectorization) Needs -qhot=simd (only valid with -qarch=450d) Compare performance with -qarch=450 Data needs to be 16-byte aligned Check if compiler correctly applied SIMD (-qreport -qlist) *.lst file: [...] fpmadd FPMADD fp3,fp35=fp2,fp34,\ fp1,fp33,fp0,fp32,\ fcr [...] 46

47 Single-Core Optimization Double FPU Help the compiler with explicit alignment assertions (XL compilers): alignx(16,a) call alignx(16,a(1)) Avoid non-stride data access, loops with calls to functions/subroutines, many-branch or conditional instructions Use ESSL wherever possible 47

48 Example 2: Matrix Multiplication program mat_mul Integer, parameter :: n = 1024 Real*8 :: a(n,n), b(n,n), c(n,n) Integer :: i, j [...] Do i = 1,n Do j = 1,n c(i,j) = sum(a(i,:)*b(:,j)) End Do End Do [...] 1. Basic optimization -O2 -qarch=450 -qtune= ms 2. More aggressive optimization -O3 -qarch=450 -qtune= ms 3. Automatic SIMD -O3 -qarch=450d -qtune=450 -qsrtict ms 48

49 Example 2: Matrix Multiplication [...] Do j = 1,n Do k = 1,n Do i = 1,n c(i,j) = c(i,j) & & + a(i,k)*b(k,j) End Do End Do End Do [...] [...] call dgemm(transa,transa,n,n,n, alf,a,n,b,n,alf,d,n) [...] 4. Cache-aware code -O3 -qarch=450d -qtune=450 -qsrtict 3.5 ms 5. Use BLAS from ESSL -O3 -qarch=450d -qtune=450 -L/bgsys/local/lib -lesslbg 0.8 ms Performance gain: about a factor of 90! 49

50 Advanced MPI - MPI Extensions for BGP Extensions to the MPI standard to ease use of BGP hardware Only C/C++ interfaces available Include header: #include <mpix.h> Examples: int MPIX_Cart_comm_create(MPI_Comm *cart_comm) creates a four-dimensional Cartesian communicator that mimics the exact hardware on which it is run. int MPIX_Pset_same_comm_create(MPI_Comm *pset_comm) creates a communicator so that all nodes in the same communicator are served by the same I/O node. See the IBM RedBook: IBM System Blue Gene Solution: Application Development for a full list of available MPI extensions. 50

51 Advanced MPI - Environment Variables BGLMPIO_COMM=0 1 How data is exchanged on collective I/O: 0 MPI_Alltoallv (default) 1 MPI_Isend/MPI_Irecv DCMF_ALLTOALL_PREMALLOC=Y N Pre-allocation of arrays (6) for all-to-all communication DCMF_EAGER=<message size> Message size above which the rendezvous protocoll should be used (high bandwidth but initial handshake), default: 1200 bytes Short messages reduce (even down to 0 bytes) DCMF_INTERRUPT=0 1 Set 1 if your code overlaps communication and computation 51

52 Tuning the Runtime Environment Network Use the TORUS network User job sizes of 2 n *512 nodes Shape Extension of a partition in X, Y, and Z direction Optimal shape is application dependent A compact shape in general beneficial Example: Communicator 8x8x32 1x1x2 might be better than 2x1x1 [...] #@bg_shape = 1x1x2 #@bg_rotate = FALSE [...] 52

53 Tuning the Runtime Environment Mapping of processes on the TORUS Default mapping: XYZT (X varies fastest) X,Y, and Z TORUS coordinates T core coordinate within one node (T=0,1,2,3) Specify mapping with mpirun option -mapfile: mpirun -mapfile <mapping> -exe <application> <mapping> can be any permutation of X,Y,Z, and T or the name of an ASCII file Tip: It is recommended to test at least the option -mapfile TXYZ 53

54 Time (s) File I/O Task-local Files Example: Creating files in parallel in the same directory 2000 parallel create of task-local files > 33 minutes 2004, > 11 minutes 1000 > 3 minutes 676, ,7 3,2 4,0 5,5 18,2 76,2 182,3 0 1k 2k 4k 8k 16k 32k 64k 128k 256k Jugene, IBM Blue Gene/P, GPFS, filesystem /work using fopen() # of Files Caution: Creation of 256k files costs core hours! Task-local files are not an option on Blue Gene! 54

55 File I/O Guidelines Use the correct file system $WORK has the best performance use it! Pay attention to block alignment Best performance if the I/O is done in chunks with block size of the file system $WORK: 4 MB block size stat() or fstat() can be used to get block size Distribute I/O over I/O nodes Try to use as many I/O nodes as possible ( MPIX routines) Use parallel I/O (libraries) MPI I/O, ADIOS, HDF5, netcdf, SIONlib 55

56 SIONlib An Alternative to Task-Local Files SIONlib (Scalable I/O library for parallel access to tasklocal files) Collective I/O to binary shared files Logical task-local view to data Collective open/close, independent write/read Support for MPI, OpenMP, MPI+OpenMP C, C++, and Fortran-wrapper Tools for manipulation of files available (extraction of data, splitting of files, dumping meta data, etc) 56

57 SIONlib An Alternative to Task-Local Files /* Open */ sprintf(tmpfn, "%s.%06d",filename,my_nr); fileptr=fopen(tmpfn, "bw",...); [...] /* Write */ fwrite(bindata,1,nbytes,filept r); [...] /* Close */ fclose(fileptr); /* Collective Open */ nfiles=1;chunksize=nbytes; sid=sion_paropen_mpi( filename, "bw", &nfiles, &chunksize, MPI_COMM_WORLD, &lcomm, &fileptr,...); [...] /* Write */ sion_fwrite(bindata,1,nbytes,s id); [...] /* Collective Close */ sion_parclose_mpi(sid); 57

58 Outline A brief Overview Best Practice JUGENE System Architecture and Configuration Production Environment Basic Porting Tuning Applications Performance Analysis Debugging 58

59 Performance Analysis Tools Accessible via module command: module load UNITE <tool> Available tools (selection): gprof (GNU profiler) no module marmot (MPI checker) marmot/2.4.0 papi (hardware counter) papi/4.0.0 Scalasca scalasca/1.4.1 tau tau/ vampir vampir/7.5-fe Basic help, first steps and references: module help <tool> 59

60 Outline A brief Overview Best Practice JUGENE System Architecture and Configuration Production Environment Basic Porting Tuning Applications Performance Analysis Debugging 60

61 Debugging Compiler Options Compilers give hints Turn optimization off -O0 -g Switch on compiler compile-time and runtime checks (floating point exceptions, boundary checks, pointer checks, etc.) -qcheck[=<suboptions>] (C/C++) -qformat[=<suboptions>] (C/C++) -qinfo[=[<suboptions>][,<goups_list>]] (C/C++) -C (Fortran) -qsigtrap[=<trap_handler>] (Fortran) -qinit=f90ptr (Fortran) -qflttrap[=<suboptions>] 61

62 Using Debuggers Prepare executable: -O0 -g -qfullpath Available debuggers DDT Totalview Important To use tools with a GUI you need to have an X server running on you local machine to connect to JUGENE with ssh -X 62

63 Example: Totalview debugger Load the corresponding modules module load UNITE totalview Start application under Totalview control llrun -np <ntasks> -mode VN -tv application.x 63

64 Navigation Source code window Action points Process and thread view

65 Debugging Analyzing Core Dump Files Activate generation of core dump files mpirun env BG_COREDUMPDISABLED=0 [ ] Core dump files are plain ASCII files including information in hexadecimal Generate your executable with the -g flag Analyzing core dump files 1. addr2line addr2line e application.x <hexaddr> 2. Totalview 3. DDT 65

66 References and Further Information Carlos Sosa and Brant Knudson. IBM System Blue Gene Solution: Blue Gene/P Application Development. 4th edition. 11 September Copyright 2007, 2009 International Business Machines Corporation. JSC websites: PRACE JUGENE Best Practice Guide: Guides?lang=en SIONlib [ XL Compilers for Blue Gene [ index.jsp?topic=/com.ibm.bg9111.doc/bgusing/related.htm] Using the IBM XL Compilers for Blue Gene [ IBM ESSL documentation [ uster.essl.doc/esslbooks.html] 66

67 Thank You! JSC User and application support: Tel: