Mitglied der Helmholtz-Gemeinschaft JUQUEEN. Best Practices. Florian Janetzko. 29. November 2013

Transcription

1 Mitglied der Helmholtz-Gemeinschaft JUQUEEN Best Practices 29. November 2013 Florian Janetzko

2 Outline Production Environment Module Environment Job Execution Basic Porting Compilers and Wrappers Compiler/Linker Flags Tuning Applications Advanced Compiler/Linker Flags QPX 29. November 2013 Slide 2

3 JUQUEEN System Architecture IBM Blue Gene/Q JUQUEEN IBM PowerPC A2 1.6 GHz, 16 cores/node, 4-way SMT, 64-bit 4-wide (dbl) SIMD (FMA) 16 GB RAM per node Torus network 28 racks, 458,752 cores 5.9 Petaflop/s peak Connected to a Global Parallel File System (GPFS) with 10 PByte online disk and 37 PByte offline tape capacity 29. November 2013 Slide 3

4 JUQUEEN Challenges Chip 4-way SMT with1 integer + 1FPU instruction per cycle filling pipes 4-wide SIMD efficient vectorization Memory 1 GB/core and 0.5 GB/core for pure MPI codes memory consumption HW support for transactional memory efficient usage Network Torus network Mapping of tasks, communicators (communication pattern) I/O Processing large amounts of data Efficient I/O strategy and management Parallelism MPP system Scalability 29. November 2013 Slide 4

5 Outline Production Environment Module Environment Job Execution Basic Porting Compilers and Wrappers Compiler/Linker Flags Tuning Applications Advanced Compiler/Linker Flags QPX 29. November 2013 Slide 5

6 Module Environment Module concept Provides overview over available software packages Eases use of software packages Access to software packages, libraries Supply of different versions of applications Supply of application-specific information Enables dynamic modification of users environment Environment variables (PATH, LD_LIBRARY_PATH, MANPATH, ) are set appropriately Detection of conflicts between applications 29. November 2013 Slide 6

7 Module Environment $ module <options> <module> Option <no option> avail list load unload help show purge Description Lists available options of the module command Lists all available modules Lists modules currently loaded Loads a module Unloads a module Lists information about a module Information about settings done by the module Unloads all modules 29. November 2013 Slide 7

8 Module Environment Six module categories COMPILER IO Different compilers and versions of compilers I/O libraries and tools MATH MISC Mathematical libraries and software packages Software not fitting into another category SCIENTIFIC Software packages from different scientific fields TOOLS Performance analysis, debugger, etc. Software for Compute Nodes: /bgsys/local Front-end Nodes: /usr/local 29. November 2013 Slide 8

9 Module Environment $ module avail /bgsys/local/modulefiles/tools /bgsys/local/modulefiles/scientific cp2k/ cpmd/3.15.1_c(default) namd/2.8 cpmd/ lammps/30aug12 namd/2.9(default) cpmd/3.15.1_a lammps/5may12(default) cpmd/3.15.1_b libint/ /usr/local/modulefiles/compiler cmake/2.8.8(default) /usr/local/modulefiles/math /usr/local/modulefiles/scientific /usr/local/modulefiles/io /usr/local/modulefiles/tools UNITE/ /usr/local/modulefiles/misc November 2013 Slide 9

10 Module Environment Applications & Libraries Selected mathematical applications and libraries arpack (2.1) gsl (1.15) mumps (4.10.0) scalapack (2.0.2) fftw (2.1.5, 3.3.3) hypre (2.9.0) parmetis (4.0.2) sprng (2.0) gmp (5.0.5) lapack (3.4.2) petsc (3.4.2) sundials (2.5.0) Selected scientific applications CPMD (3.15.3) Gromacs (4.5.5) OpenFOAM (2.1.1) CP2K ( ) Lammps (5May12,30Aug12) QE (5.0.1) GPAW* Namd (2.8, 2.9) VASP** * In preparation ** Software not installed but makefiles are available 29. November 2013 Slide 10

11 Module Environment Applications & Libraries Selected I/O libraries and tools Darshan (2.2.4) netcdf (4.3) SIONlib (1.4.3) HDF5 (1.8.11) parallel-netcdf (1.3.1) Selected tools Cmake (2.8.11) hpctoolkit (5.3.2) Tau (2.22.3b4) Clang (3.4) PAPI (5.1.1) Totalview (8.12.0) extrae (2.4.1) Scalasca (2.1) Vampir (8.1) 29. November 2013 Slide 11

12 Outline Production Environment Module Environment Job Execution Basic Porting Compilers and Wrappers Compiler/Linker Flags Tuning Applications Advanced Compiler/Linker Flags QPX 29. November 2013 Slide 12

13 Running Simulations Batch System Execution of applications managed by LoadLeveler Users submit jobs using a job command file LoadLeveler allocates computing resources to run jobs The scheduling of jobs depends on Availability of resources Job priority (jobs with larger core counts are privileged) Jobs run in queues (job classes) Classes chosen by LoadLeveler according to core count of job For Information about LoadLeveler on JUQUEEN see November 2013 Slide 13

14 LoadLeveler - Commands Command llsubmit <jobfile> llq llq l <job ID> llq s <job ID> llq u <user> llcancel <job ID> llstatus llclass llqx Description Sends job to the queuing system Lists all queued and running jobs detailed information about the specified job detailed information about a specific queued job, e.g. expected start time lists all jobs of the specified user Kills the specified job Displays the status of LoadLeveler Lists existing classes and their properties Shows detailed information about all jobs 29. November 2013 Slide 14

15 LoadLeveler Command Examples Submitting a batch job: llsubmit $ llsubmit batch-job.js llsubmit: Processed command file through Submit Filter: "/bgdata/admin/loadl/extensions/filter". llsubmit: The job "juqueen2c1.zam.kfa-juelich.de.35395" has been submitted. Query status of submitted jobs: llq $ llq -u userid Id Owner Submitted ST PRI Class Running On juqueen2c userid 11/15 10:11 I 50 n001 1 job step(s) in query, 1 waiting, 0 pending, 0 running, 0 held, 0 preempted Cancel a submitted job: llcancel $ llcancel juqueen2c llcancel: Cancel command has been sent to the central manager. 29. November 2013 Slide 15

16 LoadLeveler Job Command File ASCII file containing two major parts 1. LoadLeveler job keywords block at the beginning of a file LoadLeveler keywords have the form #@<keyword> # can be separated by any number of blanks 2. One or more application script blocks Regular shell script Can contain any shell command 29. November 2013 Slide 16

17 LoadLeveler Standard Keywords Keyword complete error never start always file name> name for stdout> name for stderr> COPY_ALL] Description Name of the job Send notification if the job is finished if the job returned an error code 0 never upon the start of the job combination of start, end, error Mail address to send messages to Requested wall time for the job Specifies corresponding file names Environment variable to be exported to job Queue job 29. November 2013 Slide 17

18 LoadLeveler Blue Gene/Q Keywords Keyword bluegene] of nodes> Xa Xb Xc Xd Description Specifies the type of job step to process. Must be set to bluegene for parallel applications. Size of the Blue Gene job, keywords bg_size and bg_shape are mutually exclusive. Specifies the requested shape of a job (in midplanes). whether the scheduler should consider all possible rotations of the given shape Type of wiring requested for the block (can be specified for each dimension separately) 29. November 2013 Slide 18

19 LoadLeveler Job Classes Class name #Nodes Max. run time Default run time n :30:00 00:30:00 n :30:00 00:30:00 n :00:00 06:00:00 n :00:00 06:00:00 m :00:00 06:00:00 m :00:00 06:00:00 m :00:00 06:00:00 m :00:00 06:00:00 m016* :00:00 06:00:00 m032* :00:00 06:00:00 m048* :00:00 06:00:00 m056* :00:00 06:00:00 *On demand only You will be charged for the full partition (e.g. if you request 513 nodes you will be charged for 1024 nodes!) Always use full partitions! 29. November 2013 Slide 19

20 LoadLeveler Job Scheduling Backfill scheduler The biggest job has the highest priority (Top Dog) LoadLeveler fills gaps with smaller, short-running jobs while freeing the system for the Top Dog Tip: Specify the wall time for your jobs as exact as possible, because jobs requesting a shorter wall time have a better chance to be executed. Big jobs Jobs requesting 8 racks are collected and run in dedicated time slots (e.g. after a maintenance) at least once a week 29. November 2013 Slide 20

21 Running Simulations runjob Command Launch command for parallel applications runjob [options] runjob [options]: <executable> [arguments] Option --args <prg_arg> --exe <executable> --envs <ENV_Var=Value> --exp-env <ENV_Var> --np <number> --ranks-per-node <number> Description Passes "prg_arg" to the launched application on the compute node. Specifies the full path to the executable Sets the environment variable ENV_Var=Value Sets the environment variable ENV_Var Total number of (MPI) tasks Number of (MPI) tasks per compute node 29. November 2013 Slide 21

22 LoadLeveler Example Job Command File I #@job_name = MPI_code #@comment = 32 ranks per node" #@output = test_$(jobid)_$(stepid).out #@error = test_$(jobid)_$(stepid).err #@environment = COPY_ALL #@job_type = bluegene #@notification = never #@bg_size = 512 #@bg_connectivity = torus #@wall_clock_limit = 14:00:00 #@queue runjob --np ranks-per-node 32 --exe app.x Pure MPI applications need to use 32 tasks per node in order use the architecture efficiently! 29. November 2013 Slide 22

23 Running Simulations MPI/OpenMP Codes On Blue Gene/P Three modes were available 1. VN mode (4 MPI tasks, no thread per task) 2. DUAL mode (2 MPI tasks with 2 OpenMP threads each) 3. SMP mode (1 MPI task with 4 OpenMP threads) On Blue Gene/Q One node has 16 cores with 4-way SMT each Several configurations possible ntasks nthreads = 64 ntasks = 2 n, 0 n 6 Test carefully, which configuration gives the best performance for your application and setup! 29. November 2013 Slide 23

24 LoadLeveler Example Job Command File II = hybrid_code = 16x4 configuration" = test_$(jobid)_$(stepid).out = test_$(jobid)_$(stepid).err = COPY_ALL = bluegene = never = 512 = torus = 14:00:00 runjob --np ranks-per-node 16\ --env OMP_NUM_THREADS=4 : app.x i input 29. November 2013 Slide 24

25 Monitoring of Jobs LoadLeveler llq [options] Llview Client-server based application compact summary of different information (e.g. current usage of system, job prediction, expected and average waiting times, ) Customizable Developed by W. Frings (JSC) 29. November 2013 Slide 25

26 Llview 29. November 2013 Slide 26

27 Outline Production Environment Module Environment Job Execution Basic Porting Compilers and Wrappers Compiler/Linker Flags Tuning Applications Advanced Compiler/Linker Flags QPX 29. November 2013 Slide 27

28 Compilers Different compilers for front-end and compute nodes GNU and IBM XL family of compilers available Tip: It is recommended to use the XL suite of compilers for the CN since they produce in general better optimized code. Language XL compiler GNU compiler C xlc, xlc_r gcc C++ xlc++, xlc++_r, xlc, xlc_r g++ Fortran xlf, xlf90, xlf95, xlf2003 xlf_r, xlf90_r, xlf95_r, xlf2003_r gfortran 29. November 2013 Slide 28

29 Compilers for CN Language Compiler invocation MPI wrapper C powerpc64-bgq-linux-gcc mpigcc C++ powerpc64-bgq-linux-g++ mpig++ Fortran powerpc64-bgq-linux-gfortran mpigfortran Tip: Language To be on the safe side, always use the corresponding MPI wrappers with _r when compiling for the CNs. Compiler invocation MPI wrapper (thread-safe: *_r) (thread-safe: *_r) C bgxlc, bgc89, bgc99 mpixlc C++ bgxlc++, bgxlc mpixlcxx Fortran bgxlf, bgxlf90, bgxlf95, bgxlf2003 mpixlf77, mpixlf90, mpixlf95, mpixlf November 2013 Slide 29

30 Basic Compiler/Linker Options XL Compilers I Flags in order of increasing optimization potential Optimization Level Description -O2 -qarch=qp -qtune=qp Basic optimization -O3 -qstrict -qarch=qp -qtune=qp More aggressive, not impact on acc. -O3 -qhot -qarch=qp -qtune=qp More aggressive, may influence acc. (high-order transformations of loops) -O4 -qarch=qp -qtune=qp Interprocedural optimization at compile time -O5 -qarch=qp -qtune=qp Interprocedural optimization at link time, whole program analysis Some flags need to be used during compilation AND linking Check the compiler manual. If you are not sure, include flags used for compiling also in the linking step! 29. November 2013 Slide 30

31 Basic Compiler/Linker Options XL Compilers II Additional compiler flags Compiler/Linker Flag -qsmp=omp -qthreaded -qreport -qlist -qessl lessl[smp]bg Description Switch on OpenMP support Generates for each source file <name> a file <name>.lst with pseudo code and a description of the kind of code optimizations which were performed Compiler attempts to replace some intrinsic FORTRAN 90 procedures by essl routines where it is safe to do so How to link with ESSL routines, see November 2013 Slide 31

32 Outline Production Environment Module Environment Job Execution Basic Porting Compilers and Wrappers Compiler/Linker Flags Tuning Applications Advanced Compiler/Linker Flags QPX 29. November 2013 Slide 32

33 Diagnostic Compiler Flags (XL Compilers) Diagnostic messages are given on the terminal and/or in a separate file -qreport: compilers generate a file name.lst for each source file -qlist: compiler listing including an object listing -qlistopt: options in effect during compilation included in listing Listen to the compiler! -qflag=<listing-severety>:<terminal-severety> i: informal messages, w: warning messages, s: severe errors Use -qflag=i:i to get all information -qlistfmt=(xml html)=<option> 29. November 2013 Slide 33

34 Example: Compilers Diagnostics subroutine mult(c,a,ndim) implicit none integer :: ndim,i,j double precision :: a(ndim),c(ndim,ndim)! Loop do i=1,1000 do j=1,1000 c(i,j) = a(i) enddo enddo end subroutine mult >>>>> LOOP TRANSFORMATION SECTION <<<<< 1 SUBROUTINE mult (c, a, ndim) [...] Id=1 DO $$CIV2 = $$CIV2, IF (.FALSE.) GOTO lab_11 $$LoopIV1 = 0 Id=2 [...] DO $$LoopIV1 = $$LoopIV1, Loop interchanging applied to loop nest Outer loop has been unrolled 8 time(s). 29. November 2013 Slide 34

35 Single-Core Optimization Compiler/Linker Flags -qessl For Fortran codes If either lessl or -lesslsmp are also specified then ESSL routines should be used in place of some Fortran90 intrinsic procedures when there is a safe opportunity to do so. -qipa (compiling and linking step) Turns on or customizes interprocedural analysis (IPA) High potential for performance benefits May considerably increase time for compiling and linking step! 29. November 2013 Slide 35

36 Single-Core Optimization Compiler/Linker Flags -qinline[=auto:level=5, +procedure1[:procedure2[: ]], ] Attempts to inline procedures instead of generating calls to those procedures, for improved performance. Several suboptions are available, check the man page. -qsimd=auto Enables the automatic generation of vector instructions -qtm Enables support for transactional memory Use only with thread-safe compiler wrappers -qunroll=yes Instructs the compiler to search for more opportunities for loop unrolling than that performed with auto. 29. November 2013 Slide 36

37 Outline Production Environment Module Environment Job Execution Basic Porting Compilers and Wrappers Compiler/Linker Flags Tuning Applications Advanced Compiler/Linker Flags QPX 29. November 2013 Slide 37

38 Quad Floating Point Extension Unit (QPX) 4 double precision pipelines, usable as: scalar FPU 4-wide FPU SIMD (Single Instruction Multiple Data) 2-wide complex arithmetic SIMD 8 concurrent floating point ops (FMA) + load + store 29. November 2013 Slide 38

39 IBM XL Compiler Support for QPX Usage of QPX Compiler flag -qsimd=auto Check that simd vectorization is actually done! -qreport -qlist >>>> LOOP TRANSFORMATION SECTION <<<< [...] Loop with nest-level 1 and iteration count 1000 was SIMD vectorized [...] >>>> LOOP TRANSFORMATION SECTION <<<< [...] Loop was not SIMD vectorized because the loop is not the innermost loop Loop was not SIMD vectorized because it contains memory references with non- vectorizable alignment. 29. November 2013 Slide 39

40 QPX Usage Hints for the Compiler Compiler needs hints Hint compiler to likely iteration counts Instruct compiler to align fields Tell that FORTRAN assumed-shape arrays are contiguous -qassert=contig real*8 :: x(:),y(:),a!ibm* align(32, x, y)!ibm* assert(itercnt(100)) do i=m, n z(i) = x(i) + a*y(i) enddo double align(32) *x, *y; double a; #pragma disjoint(*x, *y) #pragma disjoint(*x, a) #pragma ibm iterations(100) for (int i=m;i<n;i++) z[i] = x[i] + a*y[i] void foo(double* restrict a1, double* restrict a2) { for (int i=0; i<n; i++) a1[i]=a2[i]; } 29. November 2013 Slide 40

41 IBM XL QPX Intrinsics New intrinsic variable type C/C++: vector4double FORTRAN: vector(real(8)) Wide set of elemental functions available LOAD,STORE, MULT, MULT-ADD, ROUND, CEILING, SQRT, Strengths: User may layout calculation by hand, if compiler not smart enough (e.g. where no loop) Easy to use: Leave stack, register layout, load/store scheduling to compiler 29. November 2013 Slide 41

42 QPX Example using Compiler Intrinsics typedef vector4double qv; qv dx,dy,dz,dx2,dy2,dz2 for (i=0;i<4;i++) { } xd[i] = xdipl[j]; yd[i] = ydipl[j]; zd[i] = zdipl[j]; dx2 = vec_mul(dx,dx); dy2 = vec_mul(dy,dy); dz2 = vec_mul(dz,dz); d = vec_swsqrt(dx2+dy2+dz2); Source: IBM Corporation 29. November 2013 Slide 42

43 User Information and Support Information about JUQUEEN JSC websites at IBM Blue Gene/Q Application Development Redbook Dispatch and User Support Applications for accounts (for approved projects) User Support Forschunszentrum Jülich GmbH,JSC, Dispatch, Jülich Tel: , Fax: Tel: November 2013 Slide 43

44 Workshop Announcement: Second JUQUEEN Porting and Tuning Workshop 3-5 February 2014 Forschungszentrum Jülich Jülich Supercomputing Centre Contact: Dr. Dirk Brömmel, d.broemmel(at)fz-juelich.de END 29. November 2013 Slide 44