AMath 483/583 Lecture 21 May 13, Notes: Notes: Jacobi iteration. Notes: Jacobi with OpenMP coarse grain

Size: px

Start display at page:

Download "AMath 483/583 Lecture 21 May 13, Notes: Notes: Jacobi iteration. Notes: Jacobi with OpenMP coarse grain"

Justin Evans
6 years ago
Views:

1 AMath 483/583 Lecture 21 May 13, 2011 Today: OpenMP and MPI versons of Jacob teraton Gauss-Sedel and SOR teratve methods Next week: More MPI Debuggng and totalvew GPU computng Read: Class notes and references $CLASSHG/codes/openmp/jacob1.f90 $CLASSHG/codes/openmp/jacob2_omp.f90 $CLASSHG/codes/mp/jacob2_mp.f90 Jacob teraton (U 1 2U + U +1 ) = x 2 f(x ) Solve for U : U = 1 2 ( U 1 + U +1 + x 2 f(x ) ). Note: Wth no heat source, f(x) = 0, the temperature at each pont s average of neghbors. Suppose U [k] s a approxmaton to soluton. Set U [k+1] = 1 ( ) U [k] U [k] +1 + x2 f(x ) Repeat for k = 0, 1, 2,... untl convergence. for = 1, 2,..., n. Can be shown to converge (eventually... very slow!) Jacob wth OpenMP coarse gran General Approach: Fork threads only once at start of program. Each thread s responsble for some porton of the arrays, from =start to =end. Each teraton, must copy u to uold, update u, check for convergence. Convergence check requres coordnaton between threads to get global dumax. Prnt out fnal result after leavng parallel block See code n the repostory or the notes: $CLASSHG/codes/openmp/jacob2_omp.f90

2 Jacob wth MPI Each process s responsble for some porton of the arrays, from =start to =end. No shared memory: each process only has part of array. Updatng formula: u() = 0.5d0*(uold(-1) + uold(+1) + dx**2*f()) Need to exchange values at boundares: Updatng at =start requres uold(start-1) Updatng at =end requres uold(start+1) Example wth n = 9 nteror ponts (plus boundares): Process 0 has start = 1, end = 5 Process 1 has start = 6, end = 9 Jacob wth MPI Other ssues: Convergence check requres coordnaton between processes to get global dumax. Use MPI_ALLREDUCE so all process check same value. Part of fnal result must be prnted by each process (nto common fle heatsoln.txt), n proper order. See code n the repostory or the notes: $CLASSHG/codes/mp/jacob2_mp.f90 Jacob wth MPI splttng up arrays real(knd = 8),dmenson(:), allocatable :: f, u, uold... ponts_per_task = (n + ntasks - 1)/ntasks call mp_comm_rank(mpi_comm_world, me, err) start = me * ponts_per_task + 1 end = mn((me + 1)*ponts_per_task, n) allocate(f(start-1:end+1), u(start-1:end+1), & uold(start-1:end+1)) Note that each process works on only a part of the array. Dstrbuted memory model, so no large shared array. Includes ghost cells to store boundary values from neghborng processes.

3 Jacob wth MPI Sendng to neghbors call mp_comm_rank(mpi_comm_world, me, err)... do ter = 1, maxter uold = u f (me > 0) then! Send left endpont value to "left" call mp_send(uold(start), 1, MPI_DOUBLE_PRECISION, & me - 1, 1, MPI_COMM_WORLD, req1, err) end f f (me < ntasks-1) then! Send rght endpont value to "rght" call mp_send(uold(end), 1, MPI_DOUBLE_PRECISION, & me + 1, 2, MPI_COMM_WORLD, req2, err) end f end do Note: Non-blockng mp_send s used, Dfferent tags (1 and 2) for left-gong, rght-gong messages. Jacob wth MPI Recevng from neghbors Note: uold(start) from me+1 goes nto uold(end+1): uold(end) from me-1 goes nto uold(start-1): do ter = 1, maxter! mp_send s from prevous slde f (me < ntasks-1) then! Receve rght endpont value call mp_recv(uold(end+1), 1, MPI_DOUBLE_PRECISION, & me + 1, 1, MPI_COMM_WORLD, mpstatus, err) end f f (me > 0) then! Receve left endpont value call mp_recv(uold(start-1), 1, MPI_DOUBLE_PRECISION, & me - 1, 2, MPI_COMM_WORLD, mpstatus, err) end f! Apply Jacob teraton on my secton of array do = start, end u() = 0.5d0*(uold(-1) + uold(+1) + dx**2*f()) dumax_task = max(dumax_task, abs(u() - uold())) end do end do Jacob wth MPI Convergence test do ter = 1, maxter! Send and receve boundary data (prevous sldes) dumax_task = 0.d0! Jacob update: do = start, end u() = 0.5d0*(uold(-1) + uold(+1) + dx**2*f()) dumax_task = max(dumax_task, abs(u() - uold())) end do! Take global maxmum of dumax values call mp_allreduce(dumax_task, dumax_global, 1, & MPI_DOUBLE_PRECISION, & MPI_MAX, MPI_COMM_WORLD, err) f (dumax_global < tol) ext

4 Jacob wth MPI Wrtng soluton n order Want to wrte table of values x(),u() n heatsoln.txt. Need them to be n proper order, so Process 0 must wrte to ths fle frst, then Process 1, etc. Approach: Each process me wats for a message from me-1 ndcatng that t has fnshed wrtng ts part. (Contents not mportant.) Each process must open the fle (wthout clobberng values already there), wrte to t, then close the fle. Assumes all processes share a fle system! On cluster or supercomputer, need to ether: send all results to sngle process for wrtng, or wrte dstrbuted fles that may need to be combned later (some vsualzaton tools handle dstrbuted data!) Heat equaton n 2 dmensons One-dmensonal equaton generalzes to u t (x, y, t) = D(u xx (x, y, t) + u yy (x, y, t)) + f(x, y, t) on some doman n the x-y plane, wth ntal and boundary condtons. We wll only consder rectangle 0 x 1, 0 y 1. Steady state problem (wth D = 1): u xx (x, y) + u yy (x, y) = f(x, y) Fnte dfference equatons n 2D 1 h 2 (U 1,j + U +1,j + U,j 1 + U,j+1 4U,j ) = f(x, y j ). On n n grd ( x = y = 1/(n + 1)) ths gves a lnear system of n 2 equatons n n 2 unknowns. The above equaton must be satsfed for = 1, 2,..., n and j = 1, 2,..., n. Matrx s n 2 n 2, e.g. on 100 by 100 grd, matrx s 10, , 000. Contans (10, 000) 2 = 100, 000, 000 elements. Matrx s sparse: each row has at most 5 nonzeros out of n 2 elements! But structure s no longer trdagonal.

5 Fnte dfference equatons n 2D Matrx has block trdagonal structure: T I A = 1 I T I h 2 I T I T = I T Jacob n 2D Updatng pont 7 for example (u 32 ): U [k+1] 32 = 1 [k] (U U [k] 42 + U [k] 21 + U [k] 41 + h2 f 32 ) Jacob n 2D usng MPI Wth two processes: Could partton unknown nto Process 0 takes grd ponts 1 8 Process 1 takes grd ponts 9 16 Each tme step: Process 0 sends top boundary (5 8) to Process 1, Process 1 sends bottom boundary (9 12) to Process 0.

6 Jacob n 2D usng MPI Wth more grd ponts and processes... Could partton several dfferent ways, e.g. wth 4 processes: The partton on the rght requres less communcaton. Wth m 2 processes on grd wth n 2 ponts, 2m 2 n boundary ponts on left, 2mn boundary ponts on rght. Jacob n 2D usng MPI For partton on left: Natural to number processes 0,1,2,3 and pass boundary data from Process k to k ± 1. For m m array of processors as on rght: How do we fgure out the neghborng process numbers? Creatng a communcator for Cartesan blocks nteger dms(2) logcal sperodc(2), reorder ndm = 2! 2d grd of processes dms(1) = 4! for 4x6 grd of processes dms(2) = 6 sperodc(1) =.false.! perodc n x? sperodc(2) =.false.! perodc n y? reorder =.true.! optmze orderng call MPI_CART_CREATE(MPI_COMM_WORLD, ndm, & dms, sperodc, reorder, comm2d, err) Can fnd neghborng processes wthn comm2d usng MPI_CART_SHIFT

7 Gauss-Sedel teraton n Fortran do ter=1,maxter dumax = 0.d0 do =1,n uold = u() u() = 0.5d0*(u(-1) + u(+1) + dx**2*f())! check for convergence: f (dumax.lt. tol) ext Note: Now u() depends on value of u(-1) that has already been updated for prevous. Good news: Ths converges about twce as fast as Jacob! But... loop carred dependence! Cannot parallelze so easly. Red-black orderng We are free to wrte equatons of lnear system n any order... reorderng rows of coeffcent matrx, rght hand sde. Can also number unknowns of lnear system n any order... reorderng elements of soluton vector. Red-black orderng: Iterate through ponts wth odd ndex frst ( = 1, 3, 5,...) and then even ndex ponts ( = 2, 4, 6,...). Then all black ponts can be updated n any order, all red ponts can then be updated n any order. Same asymptotc convergence rate as natural orderng. Red-Black Gauss-Sedel do ter=1,maxter dumax = 0.d0! UPDATE ODD INDEX POINTS:!$omp parallel do reducton(max : dumax) &!$omp prvate(uold) do =1,n,2 uold = u() u() = 0.5d0*(u(-1) + u(+1) + dx**2*f())! UPDATE EVEN INDEX POINTS:!$omp parallel do reducton(max : dumax) &!$omp prvate(uold) do =2,n,2 uold = u() u() = 0.5d0*(u(-1) + u(+1) + dx**2*f())! check for convergence: f (dumax.lt. tol) ext

8 Gauss-Sedel method n 2D If x = y = h: 1 h 2 (U 1,j + U +1,j + U,j 1 + U,j+1 4U,j ) = f(x, y j ). Solve for U,j and terate: u [k+1],j = 1 4 (u[k+1] 1,j + u[k] +1,j + u[k+1],j 1 + u[k],j+1 h2 f,j ) Agan no need for matrx A. Note: Above ndces for old and new values assumes we terate n the natural row-wse order. Gauss-Sedel n 2D Updatng pont 7 for example (u 32 ): Depends on new values at ponts 6 and 3, old values at ponts 7 and 10. U [k+1] 32 = 1 [k+1] (U 22 + U [k] U [k+1] 21 + U [k] 41 + h2 f 32 ) Red-black orderng n 2D Agan all black ponts can be updated n any order: New value depends only on red neghbors. Then all red ponts can be updated n any order: New value depends only on black neghbors.

9 SOR method Gauss-Sedel move soluton n rght drecton but not far enough n general. Iterates relax towards soluton. Successve Over-Relaxaton (SOR): Compute Gauss-Sedel approxmaton and then go further: U GS U [k+1] where 1 < ω < 2. = 1 [k+1] (U 1 + U [k] x2 f(x )) = U [k] + ω(u GS U [k] ) Optmal omega (For ths problem): ω = 2 2π x. Convergence rates 10 0 errors vs. teraton k 10 1 Jacob Gauss Sedel SOR Red-Black SOR n 1D do ter=1,maxter dumax = 0.d0! UPDATE ODD INDEX POINTS:!$omp parallel do reducton(max : dumax) &!$omp prvate(uold, ugs) do =1,n,2 uold = u() ugs = 0.5d0*(u(-1) + u(+1) + dx**2*f()) u() = uold + omega*(ugs-uold)! UPDATE EVEN INDEX POINTS:!$omp parallel do reducton(max : dumax) &!$omp prvate(uold, ugs) do =2,n,2 uold = u() ugs = 0.5d0*(u(-1) + u(+1) + dx**2*f()) u() = uold + omega*(ugs-uold)! check for convergence... Note that uold, ugs must be prvate!

AMath 483/583 Lecture 21 May 13, 2011

AMath 483/583 Lecture 21 May 13, 2011 Today: OpenMP and MPI versions of Jacobi iteration Gauss-Seidel and SOR iterative methods Next week: More MPI Debugging and totalview GPU computing Read: Class notes