A log n lower bound to compute any function in parallel Reduction and broadcast in O(log n) time Parallel prefix (scan) in O(log n) time

Transcription

1 CS 267 Tricks with Trees Outlie A log lower boud to compute ay fuctio i parallel Reductio ad broadcast i O(log ) time Parallel prefix (sca) i O(log ) time Addig two -bit itegers i O(log ) time Multiplyig -by- matrices i O(log ) time Ivertig -by- triagular matrices i O(log 2 ) time Ivertig -by- dese matrices i O(log 2 ) time Evaluatig arbitrary expressios i O(log ) time Evaluatig recurreces i O(log ) time James Demmel 1! 2! Outlie Outlie A log lower boud to compute ay fuctio i parallel Reductio ad broadcast i O(log ) time A log lower boud to compute ay fuctio i parallel Reductio ad broadcast i O(log ) time Parallel prefix (sca) i O(log ) time Parallel prefix (sca) i O(log ) time Addig two -bit itegers i O(log ) time Addig two -bit itegers i O(log ) time Multiplyig -by- matrices i O(log ) time Ivertig -by- triagular matrices i O(log 2 ) time Multiplyig -by- matrices i O(log ) time Ivertig -by- triagular matrices i O(log 2 ) time Ivertig -by- dese matrices i O(log 2 ) time Ivertig -by- dese matrices i O(log 2 ) time Evaluatig arbitrary expressios i O(log ) time Evaluatig arbitrary expressios i O(log ) time Evaluatig recurreces i O(log ) time Evaluatig recurreces i O(log ) time 2D parallel prefix, for image segmetatio (Brya Catazaro, Kurt Keutzer) 2D parallel prefix, for image segmetatio (Brya Catazaro, Kurt Keutzer) Sparse-Matrix-Vector-Multiply (SpMV) usig Segmeted Sca Sparse-Matrix-Vector-Multiply (SpMV) usig Segmeted Sca Parallel page layout i a browser (Leo Meyerovich, Ras Bodik) Parallel page layout i a browser (Leo Meyerovich, Ras Bodik) Solvig -by- tridiagoal matrices i O(log ) time Traversig liked lists Computig miimal spaig trees Computig covex hulls of poit sets 3! 4! 1

2 A log lower boud to compute ay fuctio of variables Broadcasts ad Reductios o Trees Assume we ca oly use biary operatios, oe per time uit After 1 time uit, a output ca oly deped o two iputs Use iductio to show that after k time uits, a output ca oly deped o 2 k iputs After log 2 time uits, output depeds o at most iputs A biary tree performs such a computatio 5! 6! Parallel Prefix, or Sca If + is a associative operator, ad x[0],,x[p-1] are iput data the parallel prefix operatio computes y[j] = x[0] + x[1] + + x[j] for j=0,1,,p-1 Notatio: j:k meas x[j]+x[j+1]+ +x[k], blue is fial value Mappig Parallel Prefix oto a Tree - Details Up-the-tree phase (from leaves to root) 1) Get values L ad R from left ad right childre 2) Save L i a local register Lsave 3) Pass sum L+R to paret By iductio, Lsave = sum of all leaves i left subtree Dow the tree phase (from root to leaves) 1) Get value S from paret (the root gets 0) 2) Sed S to the left child 3) Sed S + Lsave to the right child By iductio, S = sum of all leaves to left of vertex receivig S 7! 8! 2

3 E.g., Fiboacci via Matrix Multiply Prefix F +1 = F + F -1 F F + 1 = 1 F 0 F -1 Ca compute all F by matmul_prefix o [ 1 0 ] the select the upper left etry Addig two -bit itegers i O(log ) time Let a = a[-1]a[-2] a[0] ad b = b[-1]b[-2] b[0] be two -bit biary umbers We wat their sum s = a+b = s[]s[-1] s[0] c[-1] = 0 rightmost carry bit for i = 0 to -1 c[i] = ( (a[i] xor b[i]) ad c[i-1] ) or ( a[i] ad b[i] )... ext carry bit s[i] = ( a[i] xor b[i] ) xor c[i-1] Challege: compute all c[i] i O(log ) time via parallel prefix for all (0 <= i <= -1) p[i] = a[i] xor b[i] for all (0 <= i <= -1) g[i] = a[i] ad b[i] propagate bit geerate bit c[i] = ( p[i] ad c[i-1] ) or g[i] = p[i] g[i] * c[i-1] = C[i] * c[i-1] by-2 Boolea matrix multiplicatio (associative) = C[i] * C[i-1] * C[0] * 0 1 evaluate each P[i] = C[i] * C[i-1] * * C[0] by parallel prefix Used i all computers to implemet additio - Carry look-ahead Slide source: Ala Edelma, MIT 9! 10! Other applicatios of sca = parallel prefix There are may applicatios of scas, some more obvious tha others add multi-precisio umbers (represeted as array of umbers) evaluate recurreces, expressios solve tridiagoal systems (but umerically ustable!) implemet bucket sort ad radix sort to dyamically allocate processors to search for regular expressio (e.g., grep) may others Names: +\ (APL), cumsum (Matlab), MPI_SCAN Note: 2 operatios used whe oly -1 eeded Multiplyig -by- matrices i O(log ) time For all (1 <= i,j,k <= ) P(i,j,k) = A(i,k) * B(k,j) cost = 1 time uit, usig 3 processors For all (1 <= i,j <= ) C(i,j) = Σ P(i,j,k) cost = O(log ) time, usig 2 k =1 trees with 3 / 2 processors 11! 12! 3

4 Ivertig triagular -by- matrices i O(log 2 ) time Fact: A 0 C B -1 = -1 A B CA B Fuctio Tri_Iv(T) assume = dim(t) = 2 m for simplicity If T is 1-by-1 retur 1/T else Write T = A 0 C B I parallel do { iva = Tri_Iv(A) ivb = Tri_Iv(B) } ewc = -ivb * C * iva Retur iva 0 ewc ivb time(tri_iv()) = time(tri_iv(/2)) + O(log()) Chage variable to m = log to get time(tri_iv()) = O(log 2 ) implicitly uses a tree 13! Ivertig Dese -by- matrices i O(log ) time Lemma 1: Cayley-Hamilto Theorem expressio for A -1 via characteristic polyomial i A Lemma 2: Newto s Idetities Triagular system of equatios for coefficiets of characteristic 02/05/2015 polyomial, where matrix etries = s k Lemma 3: s k = trace(a k ) = Σ A k [i,i] Csaky s Algorithm (1976) 1) Compute the powers A 2, A 3,,A -1 by parallel prefix cost = O(log 2 ) 2) Compute the traces s k = trace(a k ) cost = O(log ) 3) Solve Newto idetities for coefficiets of characteristic polyomial cost = O(log 2 ) 4) Evaluate A -1 usig Cayley-Hamilto Theorem cost = O(log ) i=1 o Completely umerically ustable CS267 Lecture ! Evaluatig arbitrary expressios Let E be a arbitrary expressio formed from +, -, *, /, paretheses, ad variables, where each appearace of each variable is couted separately Ca thik of E as arbitrary expressio tree with leaves (the variables) ad iteral odes labeled by +, -, * ad / Theorem (Bret): E ca be evaluated i O(log ) time, if we reorgaize it usig laws of commutativity, associativity ad distributivity Sketch of (moder) proof: evaluate expressio tree E greedily by repeatedly collapsig all leaves ito their parets at each time step evaluatig all chais i E with parallel prefix Evaluatig recurreces Let x i = f i (x i-1 ), f i a ratioal fuctio, x 0 give How fast ca we compute x? Theorem (Kug): Suppose degree(f i ) = d for all i If d=1, x ca be evaluated i O(log ) usig parallel prefix If d>1, evaluatig x takes Ω() time, i.e. o speedup is possible Sketch of proof whe d=1 x i = f i (x i-1 ) = (a i * x i-1 + b i )/( c i * x i-1 + d i ) ca be writte as x i = um i / de i = (a i * um i-1 + b i * de i-1 )/(c i * um i-1 + d i * de i-1 ) or um i = a i b i * um i-1 = M i * um i-1 = M i * M i-1 * * M 1 * um 0 dem i c i d i de i-1 de i-1 de 0 Ca use parallel prefix with 2-by-2 matrix multiplicatio Sketch of proof whe d>1 degree(x i ) as a fuctio of x 0 is d i After i parallel steps, degree(aythig) 2 i Computig x i take Ω(i) steps 15! 16! 4

5 Image Segmetatio (1/4) Cotours are subjective they deped o perspective Surprise: Humas agree (somewhat) Goal: geerate cotours automatically Use them to break images ito separate segmets (subimages) J. Malik s group has leadig algorithm Eable automatic image search ad retrieval ( Fid all the pictures with Fred ) Image Segmetatio (2/4) Thik of image as matrix A(i,j) of pixels Each pixel has separate R(ed), G(ree), B(lue) itesities Bottleeck (so far) of Malik s algorithm is to compute other matrices idicatig whether pixel (i,j) likely to be o cotour Ex: C(i,j) = average R itesity of pixels i rectagle above (i,j) average R itesity of pixels i rectagle below (i,j) C(i,j) large for pixel (i,j) marked with cotour, so (i,j) likely to be o Image Huma Geerated Cotours Machie Geerated Cotours 17! Algorithm evetually computes eigevectors of sparse matrix with etries computed from matrices like C Aalogous to graph partitioig i later lecture 18! Image Segmetatio (3/4) Bottleeck: Give A(i,j), compute C(i,j) where Sa(i,j) = sum of A(i,j) for etries i k x (2k+1) rectagle above A(i,j) = Σ A(r,s) for i-k r i-1 ad j-k s j+k Sb(i,j) = similar sum of rectagle below A(i,j) C(i,j) = Sa(i,j) Sb(i,j) Approach (Brya Catazaro) Compute S(i,j) = Σ A(r,s) for r i ad s j The sum of A(i,j) over ay rectagle (I low i I high, J low j J high ) is S(I high, J high ) - S(I low -1, J high ) - S(I high, J low -1) + S(I low -1, J low -1) j J low J high Image Segmetatio (4/4) New Bottleeck: Give A(i,j), compute S(i,j) where S(i,j) = Σ A(r,s) for r i ad s j 2 dimesioal parallel prefix Do parallel prefix idepedetly o each row of A(i,j) : - S row (i,j) = Σ A(i,s) for s j Do parallel prefix idepedetly o each colum of S row - S(i,j) = Σ S row (r,j) for r i = Σ A(r,s) for s j ad r i j i S(i,j) I low I high = = =0 +1 i S(i,j) 19! 20! 5

6 Sparse-Matrix-Vector-Multiply (SpMV) y = A*x Usig Segmeted Sca (SegSca) Segsca computes prefix sums of arbitrary segmets Use CSR format of Sparse Matrix A, store x desely P = [ ] Create array S showig where segmets (rows) start Compute SegSca( P, S ) = Extract A*x = [ ] Segsca ( [3, 1, 4, 5, 6, 1, 2, 3 ], [T, F, F, T, T, F, F, T ]) = [3, 4, 8, 5, 6, 7, 9, 3] Val = [ ] A = Col_Id = [ ] Row_Ptr=[ ] Create array P of all ozero A(i,j)*x(j) = Val(k)*x(Col_Id(k)) S = [ T F F T F F T F ] [ ] 21! x= Page layout i a browser Applyig layout rules to html descriptio of a webpage is a bottleeck, sca ca help Simplest example Give widths [x 1, x 2,, x ] of items to display o page, where should each item go? Item j starts at x 1 + x x j-1 Real examples have complicated costraits Defied by geeral trees, sice i html each object to display ca be composed of other objects To get locatio of each object, eed to do preorder traversal of tree, addig up costraits of previous objects Sca ca do preorder traversal of ay tree i parallel - Not just biary trees Ras Bodik, Leo Meyerovich 22! Summary of tree algorithms Lots of problems ca be doe quickly - i theory - usig trees Some algorithms are widely used broadcasts, reductios, parallel prefix carry look ahead additio Some are of theoretical iterest oly Csaky s method for matrix iversio Solvig tridiagoal liear systems (without pivotig) Both umerically ustable Csaky eeds too may processors Embedded i various systems MPI, Split-C, Titaium, NESL, other laguages CM-5 hardware cotrol etwork 23! 6