Parallelism. Wolfgang Kastaun. May 9, 2008

Transcription

1 Parallelism Wolfgang Kastaun May 9, 2008

2 Outline Parallel computing Frameworks MPI and the batch system Running MPI code at TAT The CACTUS framework Overview Mesh refinement Writing Cactus modules Links

3 Frameworks OpenMP Used on shared memory architectures. Needs compiler support. One has to insert special compiler statements before loops, which are then executed in parallel. Code can be compiled and tested without activating OpenMP. Message Passing Interface (MPI) Is a coding standard for Sending data to other instances of the same code, typically running on different CPUs/computers. Synchronization between the different instances. Different implementations available Just a library no special compiler support needed. Data can be send over network or shared memory. Code has to deal explicitly with the splitting of arrays and loops.

4 MPI and the batch system Command to run MPI code manually: <mpirun> <MPI opt> <exe> <par> <exe> is the actual executable, which is never started directly like normal code. <mpirun> depends on your system, usually mpirun.mpich or mpirun.lam. The code has to compiled with exactly the same MPI-variant which is used to start it. <MPI opt> are options for MPI. Most important one: -np <number of processes> <par> are the parameters passed to the code itself. When using a batch job system, it has to interact with MPI. Typically there is an extra queue for parallel code. You submit a script which calls MPI to run your code. The MPI-options have to be adjusted to the batch system.

5 Running MPI code at TAT pioneer cluster Command to submit the job: qsub -pe pioneer 1-<n> -l target=pioneer <script> where <n> is the number of CPUs and <script> looks like #!/bin/sh #$ -m e #$ -cwd mpirun.mpich -np $NSLOTS \ -machinefile $TMPDIR/machines \ -nolocal \ <your executable> <parameters>

6 Running MPI code at TAT phoenix cluster Command to submit the job: qsub -pe phoenix 1-<n> -l target=phoenix <script> where <n> is the number of CPUs and <script> looks like #!/bin/sh #$ -m e #$ -cwd mpirun.mpich -np $NSLOTS \ -machinefile $TMPDIR/machines \ -nolocal \ <your executable> <parameters>

7 Running MPI code at TAT cygnus No batch system, run your code directly. Command: mpiexec -np <n> <executable> <parameters> If you log out, execution would be aborted. To prevent this, use the screen utility (see man screen)

8 Running MPI code at TAT ZDV cluster System still very buggy, but (sometimes) very fast. Use the intel compiler. Commands to submit a parallel job: source /soft/etc/mpi.sh source /soft/etc/lsf.sh bsub -q MPI -n <n> <script> <n> where <script> looks like #!/bin/sh #$ -m e #$ -cwd /soft/bin/mpirun.zdv_intel -n $1 \ "<executable> <parameter>" if you need GSL or HDF5, add them to $LD_LIBRARY_PATH inside the script.

9 Outline Parallel computing Frameworks MPI and the batch system Running MPI code at TAT The CACTUS framework Overview Mesh refinement Writing Cactus modules Links

10 Overview Framework for large scale numerical computations which..can be solved on regular grids...have some kind of time evolution. Modular philosophy: small core ( flesh ) provides infrastructure, all the work is done by interacting modules ( thorns ) Provides transparent parallelization; your code just has to deal with regular grids which are managed and synchronized by CACTUS. Modules exist for File IO of grid variables in different formats. Evolution of spacetimes (BSSN_MOL). Relativistic hydrodynamics (PIZZA and WHISKY). Apparent and event horizon search. Much more

11 Mesh refinement CARPET adds transparent mesh refinement to CACTUS. In theory, if a CACTUS module is already working on a single grid, it might be used by CARPET without any change. Working principle: Coarser grids cover finer grids (no holes). Evolve coarser grid. Interpolate results to fill the outer boundaries of the finer grid ( prolongation ). After evolving the finer grid, copy the results to the corresponding points on the coarser grid ( restriction ). For several refinement levels, the method is applied recursively. Network intensive. Provides OpenDX import modules for nested grids stored in HDF5 files.

12 Writing Cactus modules CACTUS is not a library used by your code. Your code is a library called by CACTUS. A file schedule.ccl specifies what functions to call at which times, e.g. At startup Each timestep Before/after another function, in particular of other modules. Inside schedule groups defined in some module. A file interface.ccl specifies which grid variables you need or share with other modules. CACTUS has a parser for parameter files, which hold the runtime parameters of all modules. You just need a file param.ccl to declare which parameters your code needs or shares with other modules. From these files, CACTUS framework generates infrastructure code during compilation.

13 Links MPI implementations MPICH2: http: // MPICH: LAM: OpenMP: CACTUS documentation and download at CARPET documentation and download at