Outline. Parallel Numerical Algorithms. Forward Substitution. Triangular Matrices. Solving Triangular Systems. Back Substitution. Parallel Algorithm

Transcription

1 Outine Parae Numerica Agorithms Chapter 8 Prof. Michae T. Heath Department of Computer Science University of Iinois at Urbana-Champaign CS 554 / CSE Trianguar Matrices Michae T. Heath Parae Numerica Agorithms 1 / 66 Matrix L is ower trianguar if a entries above its main diagona are zero, ij = 0 for i < j Matrix U is upper trianguar if a entries beow its main diagona are zero, u ij = 0 for i > j Trianguar matrices are important because trianguar inear systems are easiy soved by successive substitution Most direct methods for soving genera inear systems first reduce matrix to trianguar form and then sove resuting equivaent trianguar system(s) Trianguar systems are aso frequenty used as preconditioners in iterative methods for soving inear systems Back Substitution Michae T. Heath Parae Numerica Agorithms 3 / 66 For upper trianguar system Ux = b, soution can be obtained by back substitution ( x i = b i for j = n to 1 x j = b j /u jj for i = 1 to j 1 b i = b i u ij x j n u ij x j )/u ii, i = n,..., 1 j=i+1 { compute son component } { update right-hand side } Forward Substitution Michae T. Heath Parae Numerica Agorithms 2 / 66 For ower trianguar system Lx = b, soution can be obtained by forward substitution ( i 1 x i = b i ij x j )/ ii, i = 1,..., n for j = 1 to n x j = b j / jj for i = j + 1 to n j=1 Soving { compute son component } { update right-hand side } Michae T. Heath Parae Numerica Agorithms 4 / 66 Forward or back substitution requires about n 2 /2 mutipications and simiar number of additions, so mode seria time as T 1 = t c n 2 /2 where t c is cost of paired mutipication and addition (we ignore cost of n divisions) We wi consider ony ower trianguar systems, as anaogous agorithms for upper trianguar systems are simiar Michae T. Heath Parae Numerica Agorithms 5 / 66 Michae T. Heath Parae Numerica Agorithms 6 / 66 Loop Orderings for Forward Substitution Parae Agorithm for j = 1 to n x j = b j / jj for i = j + 1 to n right-ooking immediate-update data-driven fan-out for i = 1 to n for j = 1 to i 1 x i = b i / ii eft-ooking deayed-update demand-driven fan-in Partition For i = 2,..., n, j = 1,..., i 1, fine-grain task (i, j) stores ij and computes product ij x j For i = 1,..., n, fine-grain task (i, i) stores ii and b i, coects sum t i = i 1 j=1 ij x j, and computes and stores x i = (b i t i )/ ii yieding 2-D trianguar array of n (n + 1)/2 fine-grain tasks Communicate For j = 1,..., n 1, task (j, j) broadcasts x j to tasks (i, j), i = j + 1,..., n For i = 2,..., n, sum reduction of products ij x j across tasks (i, j), j = 1,..., i, with task (i, i) as root Michae T. Heath Parae Numerica Agorithms 7 / 66 Michae T. Heath Parae Numerica Agorithms 8 / 66

2 Fine-Grain Tasks and Communication Fine-Grain Parae Agorithm if i = j then t = 0 if i > 1 then recv sum reduction of t across tasks (i, k), k = 1,..., i x i = (b i t)/ ii broadcast x i to tasks (k, i), k = i + 1,..., n ese recv broadcast of x j from task (j, j) t = ij x j reduce t across tasks (i, k), k = 1,..., i Michae T. Heath Parae Numerica Agorithms 9 / 66 Michae T. Heath Parae Numerica Agorithms 10 / 66 Aggomeration If communication is suitaby pipeined, then fine-grain agorithm can achieve Θ(n) execution time, but uses Θ(n 2 ) tasks, so it is inefficient If there are mutipe right-hand-side vectors b, then successive soutions can be pipeined to increase overa efficiency Aggomerating fine-grain tasks yieds more reasonabe number of tasks and improves ratio of computation to communication Aggomerate With n n array of fine-grain tasks, natura strategies are 2-D: combine k k subarray of fine-grain tasks to form each coarse-grain task, yieding (n/k) 2 coarse-grain tasks 1-D coumn: combine n fine-grain tasks in each coumn into coarse-grain task, yieding n coarse-grain tasks 1-D row: combine n fine-grain tasks in each row into coarse-grain task, yieding n coarse-grain tasks Michae T. Heath Parae Numerica Agorithms 11 / 66 Michae T. Heath Parae Numerica Agorithms 12 / 66 2-D Aggomeration 1-D Coumn Aggomeration Michae T. Heath Parae Numerica Agorithms 13 / 66 Michae T. Heath Parae Numerica Agorithms 14 / 66 1-D Row Aggomeration Mapping Map 2-D: assign (n/k) 2 /p coarse-grain tasks to each of p processes using any desired mapping in each dimension, treating target network as 2-D mesh 1-D: assign n/p coarse-grain tasks to each of p processes using any desired mapping, treating target network as 1-D mesh Michae T. Heath Parae Numerica Agorithms 15 / 66 Michae T. Heath Parae Numerica Agorithms 16 / 66

3 2-D Aggomeration, Bock Mapping 2-D Aggomeration, Cycic Mapping Michae T. Heath Parae Numerica Agorithms 17 / 66 Michae T. Heath Parae Numerica Agorithms 18 / 66 2-D Aggomeration, Refection Mapping 2 6 For 2-D aggomeration with (n/ p ) (n/ p ) subarray of fine-grain tasks per process, both vertica broadcasts and horizonta sum reductions are required to communicate soution components and accumuate inner products, respectivey If each process hods contiguous bock of rows and coumns, we obtain bock version of origina fine-grain agorithm, with poor concurrency and efficiency Moreover, this approach yieds ony (p + p )/2 non-nu processes, wasting amost haf of 2-D mesh of processors Michae T. Heath Parae Numerica Agorithms 19 / 66 Michae T. Heath Parae Numerica Agorithms 20 / 66 Cycic assignment of rows and coumns to processes yieds p non-nu processes, so fu 2-D mesh can be utiized But obvious impementation, ooping over successive soution components and performing corresponding horizonta sum reductions and vertica broadcasts, sti has imited concurrency because computation for each component invoves ony one process row and one process coumn Better agorithm can be obtained by computing soution components in groups of p, which permits a processes to perform resuting updating concurrenty Each step of resuting agorithm has four phases 1 Computation of next p soution components by processes in ower triange using 2-D fine-grain agorithm 2 Broadcast of resuting soution components verticay from processes on diagona to processes in upper triange 3 Computation of resuting updates (partia sums in inner products) by a processes 4 Horizonta sum reduction from processes in upper triange to processes on diagona Michae T. Heath Parae Numerica Agorithms 21 / 66 Michae T. Heath Parae Numerica Agorithms 22 / 66 Tota time required is approximatey T p = t c n 2 /(2p) + (4(t s + t w ) + 5 t c ) n 1. Fine-grain agorithm 2. Broadcast To determine isoefficiency function, set t c n 2 /2 E (t c n 2 /2 + (4(t s + t w ) + 5 t c ) p n) which hods for arge p if n = Θ(p), so isoefficiency function is Θ(p 2 ), since T 1 = Θ(n 2 ) 3. Update 4. Sum reduction Michae T. Heath Parae Numerica Agorithms 23 / 66 Michae T. Heath Parae Numerica Agorithms 24 / 66

4 1-D Coumn Aggomeration, Bock Mapping 1-D Coumn Aggomeration, Cycic Mapping Michae T. Heath Parae Numerica Agorithms 25 / 66 Michae T. Heath Parae Numerica Agorithms 26 / 66 1-D Coumn Aggomeration, Refection Mapping For 1-D aggomeration with n/p coumns of fine-grain tasks per process, vertica broadcasts of components of x are unnecessary because any given matrix coumn is entirey contained in ony one process But there is aso no paraeism in computing products resuting from given component of x Horizonta communication is required for sum reductions to accumuate inner products 2 6 Michae T. Heath Parae Numerica Agorithms 27 / 66 Michae T. Heath Parae Numerica Agorithms 28 / 66 1-D Coumn Fan-in Agorithm for i = 1 to n t = 0 for j mycos, j < i, t = t + ij x j if i mycos then recv sum reduction of t x i = (b i t)/ ii ese reduce t across processes Each process remains ide unti soution component corresponding to its first coumn is computed If each process hods contiguous bock of coumns, it may remain ide through most of computation Moreover, number of products computed invoving each component of x decines with increasing coumn number Concurrency and oad baance can be improved by assigning coumns to processes in cycic manner Other mappings may aso be usefu, such as bock-cycic or refection Michae T. Heath Parae Numerica Agorithms 29 / 66 Michae T. Heath Parae Numerica Agorithms 30 / 66 If successive steps (outer oop) are overapped, then approximate execution time is T p = t c (n 2 + 2n(p 1))/(2p) + (t s + t w ) (n 1) ignoring cost of additions in sum reductions Without such overapping, term representing communication cost is mutipied by factor of p 1 for 1-D mesh 2( p 1) for 2-D mesh og p for hypercube representing path ength for sum reduction To determine isoefficiency function, set t c n 2 /2 E (t c (n 2 + 2n(p 1))/2 + (t s + t w ) p (n 1)) which hods for arge p if n = Θ(p), so isoefficiency function is Θ(p 2 ), since T 1 = Θ(n 2 ) Without overapping of successive steps, isoefficiency function becomes p 4 for 1-D mesh p 3 for 2-D mesh p 2 (og p) 2 for hypercube Michae T. Heath Parae Numerica Agorithms 31 / 66 Michae T. Heath Parae Numerica Agorithms 32 / 66

5 1-D Row Aggomeration, Bock Mapping Overap achievabe is strongy affected by network topoogy and mapping of rows to processes For exampe, cycic mapping on ring network permits amost compete overap, whereas hypercube permits much ess overap Overap of successive steps can potentiay be enhanced by compute ahead strategy Process owning coumn i coud compute most of its contribution to inner product for step i + 1 whie waiting for contributions from other processes in step i, thereby avoiding being botteneck for next step (because it wi be ast to compete step i) Michae T. Heath Parae Numerica Agorithms 33 / 66 1-D Row Aggomeration, Cycic Mapping Michae T. Heath Parae Numerica Agorithms 34 / 66 1-D Row Aggomeration, Refection Mapping Michae T. Heath Parae Numerica Agorithms 35 / 66 Michae T. Heath Parae Numerica Agorithms 36 / 66 1-D Row Fan-out Agorithm For 1-D aggomeration with n/p rows of fine-grain tasks per process, communication for horizonta sum reductions across process rows is unnecessary because any given matrix row is entirey contained in ony one process But there is aso no paraeism in computing these sums Vertica broadcasts are required to communicate components of x for j = 1 to n if j myrows then x j = b j / jj broadcast x j for i myrows, i > j, Michae T. Heath Parae Numerica Agorithms 37 / 66 Michae T. Heath Parae Numerica Agorithms 38 / 66 Each process fas ide as soon as soution component corresponding to its ast row has been computed If each process hods contiguous bock of rows, it may become ide ong before overa computation is compete Moreover, computation of inner products across rows requires successivey more work with increasing row number Concurrency and oad baance can be improved by assigning rows to processes in cycic manner Other mappings may aso be usefu, such as bock-cycic or refection If successive steps (outer oop) are overapped, then approximate execution time is T p = t c (n 2 + 2n(p 1))/(2p) + (t s + t w ) (n 1) Without such overapping, term representing communication cost is mutipied by factor of p 1 for 1-D mesh 2( p 1) for 2-D mesh og p for hypercube representing path ength for broadcast Michae T. Heath Parae Numerica Agorithms 39 / 66 Michae T. Heath Parae Numerica Agorithms 40 / 66

6 To determine isoefficiency function, set t c n 2 /2 E (t c (n 2 + 2n(p 1))/2 + (t s + t w ) p (n 1)) which hods for arge p if n = Θ(p), so isoefficiency function is Θ(p 2 ), since T 1 = Θ(n 2 ) Without overapping of successive steps, isoefficiency function becomes p 4 for 1-D mesh p 3 for 2-D mesh p 2 (og p) 2 for hypercube Overap achievabe is aso strongy affected by network topoogy and mapping of rows to processes For exampe, cycic mapping on ring network permits amost compete overap, whereas hypercube permits much ess overap Overap of successive steps can potentiay be enhanced by s ahead strategy At step j, process owning row j + 1 coud compute x j+1 and broadcast it before competing remaining updating due to x j Michae T. Heath Parae Numerica Agorithms 41 / 66 Michae T. Heath Parae Numerica Agorithms 42 / 66 Fan-out and fan-in agorithms derive their paraeism from inner oop, whose work is partitioned and distributed across processes, whie outer oop is seria Conceptuay, fan-out and fan-in agorithms work on ony one component of soution at a time, though successive steps may be pipeined to some degree Wavefront agorithms expoit paraeism in outer oop expicity by working on mutipe components of soution simutaneousy Michae T. Heath Parae Numerica Agorithms 43 / 66 To formaize wavefront coumn agorithm we introduce z : vector in which to accumuate updates to right-hand-side segment : set containing at most s consecutive components of z Michae T. Heath Parae Numerica Agorithms 45 / 66 1-D coumn fan-out agorithm seems to admit no paraeism: after process owning coumn j computes x j, resuting updating of right-hand side cannot be shared with other processes because they have no access to coumn j Instead of performing a such updates immediatey, however, process owning coumn j coud compete ony first s components of update vector and forward them to process owning coumn j + 1 before continuing with next s components of update vector, etc. Upon receiving first s components of update vector, process owning coumn j + 1 can compute x j+1, begin further updates, forward its own contributions to next process, etc. Michae T. Heath Parae Numerica Agorithms 44 / 66 for j mycos for k = 1 to # segments recv segment if k = 1 then x j = (b j z j )/ jj segment = segment {z j } for z i segment z i = z i + ij x j if segment > 0 then s segment to process owning coumn j + 1 Michae T. Heath Parae Numerica Agorithms 46 / 66 Deping on segment size, coumn mapping, communication-to-computation speed ratio, etc., it may be possibe for a processes to become busy simutaneousy, each working on different component of soution Segment size is adjustabe parameter that contros tradeoff between communication and concurrency First segment for given coumn shrinks by one eement after each component of soution is computed, disappearing after s steps, when next segment becomes first segment, etc. At of computation ony one segment remains and it contains ony one eement Communication voume decines throughout agorithm As segment ength s increases, communication start-up cost decreases but computation cost increases, and vice versa as segment ength decreases Optima choice of segment ength s can be predicted from performance mode Michae T. Heath Parae Numerica Agorithms 47 / 66 Michae T. Heath Parae Numerica Agorithms 48 / 66

7 Approximate execution time is T p = ((t s /s) + t w + t c ) (n 2 + np + s(s 1)p 2 )/(2p) where s is segment ength To determine isoefficiency function, set t c n 2 /2 E (((t s /s) + t w + t c ) (n 2 + np + s(s 1)p 2 )/2) which hods for arge p if n = Θ(p), assuming s is constant, so isoefficiency function is Θ(p 2 ), since T 1 = Θ(n 2 ) Wavefront approach can aso be appied to 1-D row fan-in agorithm Computation of ith inner product cannot be shared because ony one process has access to row i of matrix Thus, work on mutipe components must be overapped to attain any concurrency Anaogous approach is to break soution vector x into segments that are pipeined through processes Michae T. Heath Parae Numerica Agorithms 49 / 66 Michae T. Heath Parae Numerica Agorithms 50 / 66 Initiay, process owning row 1 computes and ss it to process owning row 2, which computes resuting update and then This process continues (seriay at this eary stage) unti s components of soution have been computed Henceforth, receiving processes forward any fu-size segments before they are used in updating Forwarding of currenty incompete segment is deayed unti next component of soution is computed and apped to it Michae T. Heath Parae Numerica Agorithms 51 / 66 for i myrows for k = 1 to # segments 1 recv segment s segment to process owning row i + 1 for x j segment recv segment /* ast may be empty */ for x j segment x i = b i / ii segment = segment {x i } s segment to process owning row i + 1 Michae T. Heath Parae Numerica Agorithms 53 / 66 Michae T. Heath Parae Numerica Agorithms 52 / 66 Instead of starting with fu set of segments that shrink and eventuay disappear, segments appear and grow unti there is a fu set of them It may be possibe for a processes to be busy simutaneousy, each working on different segment Segment size is adjustabe parameter that contros tradeoff between communication and concurrency, and optima vaue of segment ength s can be predicted from performance mode Performance anaysis and resuting performance mode are more compicated than for 1-D coumn wavefront agorithm, but performance and scaabiity for 1-D row wavefront agorithm are nevertheess simiar Michae T. Heath Parae Numerica Agorithms 54 / 66 In wavefront agorithms, each segment is sent up to s times and may pass through same process repeatedy, deping on mapping of rows or coumns Cycic agorithms are somewhat simiar to wavefront agorithms, but they minimize communication by expoiting cycic mapping of rows or coumns Instead of having variabe number of segments of adjustabe ength, cycic agorithms circuate singe segment of ength p 1 In cycic 1-D coumn agorithm, segment of size p 1, containing partiay accumuated components of update vector z, passes from process to process, one step for each coumn of matrix, cycing through a p 1 other processes before returning to any given process At step j, process owning coumn j receives segment from process owning coumn j 1 and uses its first eement (which has accumuated a necessary prior updates) to compute x j Task owning coumn j then modifies segment by deeting first eement, updating remaining eements, and apping new eement to begin accumuation of z j+p 1 Michae T. Heath Parae Numerica Agorithms 55 / 66 Michae T. Heath Parae Numerica Agorithms 56 / 66

8 Segment is then sent to process owning coumn j + 1, where simiar procedure is repeated After forwarding modified segment, process owning coumn j then computes remaining updates resuting from x j, which wi be needed when segment returns to this process again Updating whie segment is esewhere provides a concurrency, since computations on segment are seria for j mycos recv segment x j = (b j z j t j )/ jj segment = segment {z j } for z i segment z i = z i + t i + ij x j z j+p 1 = t j+p 1 + j+p 1,j x j segment = segment {z j+p 1 } s segment to process owning coumn j + 1 for i = j + p to n t i = t i + ij x j Michae T. Heath Parae Numerica Agorithms 57 / 66 Michae T. Heath Parae Numerica Agorithms 58 / 66 Segment must pass through a other processes before returning to any given process, so correctness deps on use of cycic mapping Maps naturay to 1-D torus (ring) network, but since ony one pair of processes communicates at any given time, aso works we with bus network Attains minimum possibe voume of interprocessor communication to sove trianguar system using coumn-oriented agorithm For n p (t p + p), where t p = (t s + t w (p 1))/t c is cost, measured in fops, of sing message of ength p 1, execution time is determined by segment cyce time, so that T p = t c (n (t p + p) p (p 1)/2 t p ) For n > p (t p + p), execution time is dominated by cost of updating, so that T p = t c ((n 2 + n p)/(2p) + p ((t p + p) 2 t p p + 1)/2 t p ) Michae T. Heath Parae Numerica Agorithms 59 / 66 Michae T. Heath Parae Numerica Agorithms 60 / 66 Two-phase behavior compicates scaabiity anaysis, but tradeoff point between phases for n as function of p grows ike p 2, so isoefficiency function is at east Θ(p 4 ) Performance of both phases can be improved Segment cyce time can be reduced by breaking segment into smaer pieces and pipeining them through processes Updating work can be reorganized, deferring excessive work unti ater cyces, to obtain more even distribution throughout computation 1-D row cycic agorithm is simiar, except processes are aggomerated by rows and segment contains p 1 components of soution x At step i, process owning row i receives segment from process owning row i 1 and uses components of x it contains to compete ith inner product, so that x i can then be computed Task then modifies segment by deeting first eement and apping new eement x i just computed Segment is then sent to process owning row i + 1, where simiar procedure is repeated Michae T. Heath Parae Numerica Agorithms 61 / 66 After forwarding modified segment, process then computes partia inner products that use components of segment, which wi be further accumuated when segment returns to this process again Latter computations, which take pace whie segment passes through other processes, provide concurrency in agorithm, because computations on segment itsef are seria Again, correctness of agorithm deps on use of cycic mapping Performance and scaabiity are simiar to those for 1-D coumn cycic agorithm, athough detais differ Michae T. Heath Parae Numerica Agorithms 63 / 66 Michae T. Heath Parae Numerica Agorithms 62 / 66 for i myrows recv segment for x j segment x i = b i / ii segment = segment {x i p } {x i } s segment to process owning row i + 1 for m myrows, m > i, for x j segment b m = b m mj x j Michae T. Heath Parae Numerica Agorithms 64 / 66

9 References References R. H. Bisseing and J. G. G. van de Vorst, Parae trianguar system soving on a mesh network of Transputers, SIAM J. Sci. Stat. Comput. 12: , 1991 S. C. Eisenstat, M. T. Heath, C. S. Henke, and C. H. Romine, Modified cycic agorithms for soving trianguar systems on distributed-memory mutiprocessors, SIAM J. Sci. Stat. Comput. 9: , 1988 M. T. Heath and C. H. Romine, Parae soution of trianguar systems on distributed-memory mutiprocessors, SIAM J. Sci. Stat. Comput. 9: , 1988 N. J. Higham, Stabiity of parae trianguar system sovers, SIAM J. Sci. Comput. 16: , 1995 G. Li and T. F. Coeman, A parae trianguar sover for a distributed-memory mutiprocessor, SIAM J. Sci. Stat. Comput. 9: , 1988 G. Li and T. F. Coeman, A new method for soving trianguar systems on distributed-memory message-passing mutiprocessors, SIAM J. Sci. Stat. Comput. 10: , 1989 C. H. Romine and J. M. Ortega, Parae soution of trianguar systems of equations, Parae Computing 6: , 1988 E. E. Santos, On designing optima parae trianguar sovers, Information and Computation 161: , 2000 Michae T. Heath Parae Numerica Agorithms 65 / 66 Michae T. Heath Parae Numerica Agorithms 66 / 66