AWP ODC QUICK START GUIDE

Transcription

1 AWP ODC QUICK START GUIDE 1. CPU 1.1. Prepare the code Obtain and compile the code, possibly at a path which does not purge. Use the appropriate makefile. For Bluewaters (/u/sciteam/poyraz/scratch/quick_start/cpu/src v1.1.2): module swap PrgEnv cray PrgEnv pgi make clean make f makefile.bluewaters 1.2. Prepare run environment Run environment consists of a run folder at a path in lustre (or other high performance file system). The run folder contains (/u/sciteam/poyraz/scratch/quick_start/cpu/run folder): run: run script to submit to the scheduler pre run: [optional] creates necessary input/output folders for use of the application pmcl3d: the executable (itself or a soft link) created by the compiling in the previous step (1.1). IN3D: the parameter input text file input/: [optional] folder that contains the inputs (see step 1.3). Run pre run:./pre run Manually stripe output_sfc: lfs setstripe c <COUNT, 1 is maximum> s <SIZE>. COUNT and SIZE depends on the size of the run. lfs setstripe c 1 s 5m output_sfc lfs getstripe output_sfc Modify run script (/u/sciteam/poyraz/scratch/quick_start/cpu/run folder/run). Make sure to set #PBS l nodes=8:ppn=32:xe #PBS l walltime=00:25:00 and have the following to run the application aprun n 256./pmcl3d > O.$PBS_JOBID For multiple serial runs you can call aprun multiple times: aprun n 256./pmcl3d > O 1.$PBS_JOBID

2 cp r output_sfc output_sfc 1 aprun n 256./pmcl3d > O 2.$PBS_JOBID cp r output_sfc output_sfc 2 aprun n 256./pmcl3d > O 3.$PBS_JOBID cp r output_sfc output_sfc 3 Total number of MPI ranks requested in aprun n must be equal to nodes*ppn. If weak scaling testing is done, modify: nodes, aprun n, walltime 1.3. Prepare inputs There are 2 input files (source and mesh), and a parameter input file (IN3D). Depending on the properties of the sources, parameters in IN3D are set accordingly. For benchmarking and this quick start guide, we use only 1 input file (source) and the parameter input file (IN3D). For Bluewaters (/u/sciteam/poyraz/scratch/quick_start/cpu/run folder/input) we use FAULTPOW. It is a point source that lasts for 91 time steps. For parameter input file IN3D, modify the following: 1.99 TMAX : Total time the simulation is run for. # timesteps = TMAX/DT NX : Total # points in X direction. This example represents a 224 cubic sub domain since 1792/8=224, 896/4= NY 896 NZ 8 NPX : Total # processors (MPI ranks) in X direction. 8 NPY 4 NPZ 10 WRITE_STEP : Number of outputs to aggregate before writing out 1792 NEDX : The last index in X direction to write the outputs for (generally equals to NX) 1792 NEDY 1 NSKPX : How many points to pass before saving velocities to write out (1 or 2 is typical) 1 NSKPY 10 NTISKP : Number of timesteps to pass before saving velocities to write out If no outputs is wanted to be written out, set: 0 IO_OPT : Disables IO If weak scaling testing is being done, modify: NX, NY, NZ, NPX, NPY, NPZ, NEDX, NEDY 1.4. Run

3 Submit your run script: qsub run Check how it is doing: qstat u <username> Check when it is going to start: showstart <JOBID> Delete job if wanted (if you realize you submitted it with wrong parameters, etc.): qdel <JOBID> 1.5. Outputs While running there are 3 text files created by the system, velocity outputs and some other information files created by the application. The velocity outputs can be found in output_sfc. At every NTISKP*WRITE_STEP (in IN3D) timesteps, application writes out velocity outputs. If you see all zeros or some NaNs, then there was an error, possibly related to parameters or inputs. See velocity outputs with od comment: od A d f output_sfc/sx System generated E.* file contains error or warning messages. Typically it should be empty however it is okay to see such warnings: Warning: ieee_inexact is signaling FORTRAN STOP System generated O.* file contains the application generated text that was directed to the standard output. You should see some information about the initializations and then time steps. Please see as /u/sciteam/poyraz/scratch/quick_start/cpu/run folder/o nid0022 1an example. At the end you can see performance statistics, such as: Final inialization time = sec get station time = sec read source time = sec read media time = sec computing time per time step = sec mpiio time per time step = sec total elapsed time is = sec Computing time per time step is the main performance statistic we are interested in. 2. GPU 2.1. Prepare the code Obtain the code and compile it. For Bluewaters see

4 /u/sciteam/poyraz/scratch/quick_start/gpu/src v2.0.3 forbenchmark. The makefile to use is Makefile sm35 woox. module swap PrgEnv cray PrgEnv gnu module load cudatoolkit make clean make f Makefile sm35 woox 2.2. Prepare run environment Create run folder: /u/sciteam/poyraz/scratch/quick_start/gpu/run folder. Have: run : run script pmcl3d : executable striper.sh : almost the same as pre run in CPU code, but additionally performs lustre striping. Depending on the size of the run, manual striping is suggested. input : the same as CPU case. Run striper.sh:./striper.sh Modify run script (/u/sciteam/poyraz/scratch/quick_start/gpu/run folder/run). Make sure to set #PBS l nodes=8:ppn=1:xk #PBS l walltime=00:15:00 and have the following to run the application aprun n 8./pmcl3d X 640 Y 320 x 4 y 2 >> O.$PBS_JOBID Notice that we have ppn=1 and request xk nodes. GPU code does not have an IN3D file so, we give parameters from command line: X : NX Y : NY Z : NZ x : NPX y : NPY <there is no z since NPZ=1 always> T : TMAX The benchmark code does not support WRITE_STEP, NTISKP, etc. For weak scaling modify X, Y, x, and y. The best sub domain size is 160x160x Prepare inputs

5 There are source and mesh inputs as was in CPU case Run Job submission and manipulations are the same Outputs Velocity outputs are the same. At the end of the O.* file, some performance statistics are printed: GPU benchmark size NX=640, NY=320, NZ=2048, ReadStep=91 GPU computing flops= GFLOPS, time = secs per timestep Please contact Efecan at epoyraz@ucsd.edu for any questions.