Speeding Up Reactive Transport Code Using OpenMP. OpenMP

Transcription

1 Speeding Up Reactive Transport Code Using OpenMP By Jared McLaughlin OpenMP A standard for parallelizing Fortran and C/C++ on shared memory systems Minimal changes to sequential code required Incremental parallelization OpenMP Compliant and normal compilers!$omp No message passing between processors Fine and course grained Parallelism Do Loop Sections

2 Threads A thread is forked to start the parallel region and joined at the end of the region. The thread that was forked is called the master, while the other threads are the workers. What is a thread? Parallel Region Constructor!$OMP PARALLEL clause1 clause2 parallel code is placed here!$omp END PARALLEL Optional clauses include PRIVATE (list) SHARED (list) DEFAULT (PRIVATE SHARED NONE) FIRSTPRIVATE (list) REDUCTION (operator:list) IF (scalar logical expression) NUM THREADS (scalar integer expression)

3 Work Sharing Constructs!$OMP DO clause1 clause2 DO i=1, N parallel code is placed here END DO!$OMP END DO end_clause Optional clauses include PRIVATE (list) FIRSTPRIVATE (list) LASTPRIVATE (list) REDUCTION (operator:list) SCHEDULE (type, chunk)!$omp SINGLE clause1 clause !$omp END SINGLE end_clause Optional clauses include PRIVATE (list) FIRSTPRIVATE (list)!$omp SECTIONS clause1 clause2...!$omp SECTION... parallel code is placed here!$omp SECTION... parallel code is placed here!$omp END SECTIONS end_clause Optional clauses include PRIVATE (list) FIRSTPRIVATE (list) LASTPRIVATE (list) REDUCTION (operator:list)!$omp WORKSHARE...!$OMP END WORKSHARE end_clause Clauses Shared(list) Same location of variable available to all threads; exists before and after parallel region Must check that no race conditions occur Default(NONE SHARED PRIVATE) Any unstated variables can be defaulted to shared or private None says all variables must be declared in the shared or private clauses Private(list) Each thread has its own copy of the variable considered to be local to that parallel construct Private variables have to be initialized inside the parallel region and are considered to be undefined outside of that region Do Loop Counters are always private

4 Clauses FIRSTPRIVATE (list) gives the private variable an initialized value of the original variable when entering the parallel region LASTPRIVATE (list) gives the exiting private variable the value of the last iteration or final section REDUCTION (operator:list) to ensure that a shared variable location is written to one thread at a time; each thread has a private copy of the shared variable that gets updated at the end of the parallel region; operators include +, *,,.AND.,.OR.,.EQV.,.NEQV., MAX, MIN, IAND, IOR or IEOR IF (scalar logical expression) allows the parallel region to be run sequentially if the expression is false NUM THREADS (scalar integer expression) allows the number of threads the region is fork into to be declared (still an optional command that is not needed) Clauses SCHEDULE (type, chunk) type can be static, dynamic, or guided; help determine the efficiency of the code Static divides the iterations statically in the beginning between the threads; if a chunk size is set the last thread may have a different number of iterations then the others; offers the best performance if all the iterations require the same computational time Dynamic each thread is given a small amount of work the size of chunk and when it is done it is given more; if the chunk is not specified the default is one; obviously increases overhead Guided gives a combination of the two by handing out large loads at first and then handing out smaller loads decreasing exponentially NOWAIT an end_clause causing the threads not to wait at the end of a work sharing region, but to continue on to the next work sharing region; without this clause there is an implied barrier for all the threads to catch up and synchronize with each other

5 Van der Pas, R. (2005, June 1 4). An Introduction into OpenMP. Presented at the University of Oregon. REACTION TRANSPORT MODELING

6 Performance Analysis Note: debug versus release mode Governing Equation: C t 2 C C = v + D + SS + reactions 2 x x Operator Split: C C = v t x 2 C C = D 2 t x C = SS t C = reactions t

7 100 Species in 1D Column after 40 years 100 Species Problem Specifics Parallelized in two places Advection Dispersion Equation with parallel Do Loop species iterations split between threads Reactions with parallel Do Loop node iterations split between threads the same way RT3D is done Results presented from Debug mode runs Simulation Time (yr) 40 Length (m) 2000 Velocity (m/yr) 5 x 1 t 0.1 Dispersion coefficient; Dx (m^2/yr) 50 Courant 0.5 Peclet 0.1

8 Timing (Static Scheduling) Program Run Time Program Speedup Efficiency Reaction Run Time Reaction Speedup Efficiency Dispersion RunTime Adv Disp Speedup Efficiency Time Spent in Reactions 77.87% 75.11% 71.91% 71.82% Time Spent in Adv Disp 20.45% 21.39% 23.13% 21.74% Don t focus on the Program speedup and efficiency, just the parallelized sections. Timing (Guided Scheduling) Program Run Time Program Speedup Efficiency Reaction Run Time Reaction Speedup Efficiency Dispersion RunTime Adv Disp Speedup Efficiency Time Spent in Reactions 77.87% 70.80% 68.01% 64.74% Time Spent in Adv Disp 20.45% 25.22% 27.19% 28.82%

9 Timing (Static Adv Disp & Guided Reactions) Program Run Time Program Speedup Efficiency Reaction Run Time Reaction Speedup Efficiency Dispersion RunTime Adv Disp Speedup Efficiency Time Spent in Reactions 77.87% 75.07% 71.79% 70.99% Time Spent in Adv Disp 20.45% 21.40% 23.10% 22.21% Superlinear Speedup 100 Species Runtimes

10 100 Species Speedup Vinyl Chloride after days

11 RT3D Problem Specifics A Program called MT3D solves the advection, dispersion, and source/sink equations and calls the RT3D subroutines to solve the reactions equation The specific problem solved in this example was the sequential decay of PCE, TCE, DCE, and VC. The continuous source spill concentration of PCE was 1000 mg/l at the well. The initial levels of all chemicals in the aquifer was 0.0 mg/l. The site was 510 m x 310 m x 100 m. This created a grid 51x31x10. The reactions solved were as follows: R PCE = k 1 * [PCE] R TCE = k 1 *Y TCE/PCE *[PCE] k 2 * [TCE] R DCE = k 2 *Y DCE/TCE *[TCE] k 3 * [DCE] R VC = k 3 *Y VC/DCE *[DCE] k 4 * [VC] Results presented from a Release mode version k day 1 k day 1 k day 1 k day 1 YTCE/PCE YDCE/TCE YVC/DCE Timing Loop Around Row Do Loop (Static Scheduling) Program Run Time Program Speedup Efficiency Rt3d Run Time Rt3d Speedup Efficiency Time Spent in Rt3d 58.10% 42.29% 36.67% 31.30% Don t focus on the Program speedup and efficiency, just the parallelized sections.

12 Timing Loop Around Row Do Loop (Guided Scheduling) Program Run Time Program Speedup Efficiency Rt3d Run Time Rt3d Speedup Efficiency Time Spent in Rt3d 58.10% 41.76% 32.52% 26.77% RT3D Decay Problem Runtimes

13 RT3D Decay Problem Runtimes Conclusion Clearly the capabilities of OpenMP are limited to the available computer architectures. Much more speedup is possible with hundreds of processors in a cluster system possibly using Message Passing Interface routines, but OpenMP leaves code intact sequentially, is easy to implement, and accomplishes great speedup when a limited number of processors are available in a shared memory system. Options for future research can include a Hybrid MPI/OpenMP code utilizing the benefits of both standards. OpenMP is available primarily in commercial compilers such as Intel Visual Fortran and PGI compilers. Omni compiler might be free with OpenMP I have not tried it so I don t know if it works.