Lecture 1: Introduction and Strassen s Algorithm

Transcription

1 5-750: Graduate Algorithms Jauary 7, 08 Lecture : Itroductio ad Strasse s Algorithm Lecturer: Gary Miller Scribe: Robert Parker Itroductio Machie models I this class, we will primarily use the Radom Access Machie (RAM) model I this model, we have a memory ad a fiite cotrol The fiite cotrol ca read or write to ay locatio i the memory i uit time I additio, it ca perform arithmetic operatios +,,, ad i uit time We will ot require a more formal defiitio Later i the semester, we will cosider the Parallel Radom Access Machie (PRAM) model ad circuit models We will ot discuss cachig models (though they may appear o a homework assigmet), memory hierarchies, or pipeliig Asymptotic complexity We will oly cosider asymptotic complexity of algorithms: The iput size (typically ) grows to ifiity We ow defie the asymptotic otatio we will use to describe time ad space used by algorithms i this settig Defiitio Let f, g : N N f() O(g()) if there exist c > 0 ad 0 N such that for all 0, f() c g() f() o(g()) if for all c > 0, there exists 0 N such that for all 0, f() c g() For ituitio, f O(g) meas that f grows o faster tha g, ad f o(g) meas that f grows more slowly tha g We ofte write f = O(g) to idicate f O(g) ad f = o(g) for f o(g) There are two alterative defiitios for f Ω(g) Defiitio f Ω(g) if g O(f) Defiitio 3 f Ω(g) if there exists c > 0 such that for all 0 N, there exists 0 such that f( ) c g() The phrase for all 0 N, there exists 0 simply meas ifiitely ofte Ituitively, f Ω(g) meas that f grows at least as quickly as g The two defiitios usually agree, but there are some cases i which they do t For istace, si Ω() uder Defiitio 3, but this is ot true uder Defiitio Such examples will ot come up i this class, so for us these defiitios are essetially equivalet Example Show that + + O( ) Proof We try c = 3 We eed to fid 0 such that whe 0 This is equivalet to 0, ad it is easy to see that this holds for We ca therefore choose c = 3 ad 0 =

2 Alteratively, we ca use L Hôpital s Rule Observe that lim = lim =, This meas that for ay ɛ > 0, there exists a 0 for which we ca choose c = + ɛ Strasse s Algorithm For a m matrix M, we will deote the (i, j) etry by M ij, the ith row by M i, ad the jth colum by M j Matrix multiplicatio Defiitio Give matrices A R k ad B R k m, AB is defied to be a matrix i R m such that k (AB) ij = A it B tj Here is a picture: t= k m A i m A i B j A k B B j = C The product of two matrices ca also be represeted as the sum of outer products of their row ad colum vectors Claim Let A R k ad B R k m The AB = k A t Bt This follows immediately from Defiitio, sice (A t B t ) ij = A it B tj We recall some basic facts about matrix multiplicatio Fact 3 (AB)C = A(BC) I geeral, AB BA t= 3 For ay scalar λ, λ λa = A λ

3 A(B + C) = AB + AC Our goal i this lecture is to give a algorithm for matrix multiplicatio that is as efficiet as possible with respect to both time ad space We will start by givig a aive algorithm that rus i time O( 3 ) ad the show how we ca do better usig Strasse s Algorithm We will oly cosider dese matrix multiplicatio, i which most of the etries of the iput matrices are ozero For sparse matrices, i which most of the etries are 0, there are algorithms for matrix multiplicatio that leverage this sparsity to get a better rutime We will ot discuss algorithms for sparse matrices i this class Naive algorithm We begi with a aive algorithm that loops through all etries of the output ad computes each oe Algorithm Naive matrix multiplicatio Iput: A, B R Output: AB for i = to do for j = to do Set C ij = t= A itb tj ed for ed for retur C This requires 3 multiplicatios ad ( ) additios, so the total rutime is O( 3 ) 3 Recursive algorithm Next, we will give a recursive algorithm that also rus i time O( 3 ) Strasse s Algorithm will use a similar recursive framework to achieve subcubic rutime We assume that = k for some iteger k For A R, we write ( ) A A A =, A A where each of the A ij s is / / 3

4 Algorithm Recursive matrix multiplicatio Iput: A, B R Output: AB fuctio M(A,B) if A is the retur a b ed if for i = to do for j = to do Set C ij = M(A i, B j ) + M(A i, B j ) ed for ed for( ) C C retur C C ed fuctio Correctess First, we eed to prove that this algorithm is correct That is, we eed to prove that M(A, B) = AB We will do this by iductio o I the base case, = ad the algorithm correctly returs a b I the iductive case, we assume that M(A, B) = AB for all matrices with < 0 From the defiitio of matrix multiplicatio, it is clear that (AB) ij = A i B j + A i B j By iductio, M(A i, B j ) = A i B j ad M(A i, B j ) = A i B j, so we see that (AB) ij = M(A i, B j ) + M(A i, B j ) = M(A, B) Rutime Defie T () to be the umber of operatios required for the algorithm to multiply two matrices The algorithm makes eight recursive calls It also adds two matrices, which requires c time for some costat c We therefore obtai the followig recurrece: Claim T () = O( 3 ) T () 8T (/) + c T () = We will prove this claim i Sectio 3 If the umber of recursive calls is smaller, the rutime will be faster Specifically, we will also prove the followig claim i Sectio 3 Claim 5 Assume The T () = O( log 7 ) T () 7T (/) + c ad T () = Strasse s Algorithm makes oly seve recursive calls, so it rus i time O( log 7 ) = O( 807 ), faster tha O( 3 )

5 Strasse s Algorithm We agai cosider multiplyig matrices broke ito / / blocks as follows: ( ) ( ) A B E F M =, N = C D G H Cosider the followig matrices: S = (B D)(G + H) S = (A + D)(E + H) S 3 = (A C)(E + F ) S = (A + B)H S 5 = A(F H) S 6 = D(G E) S 7 = (C + D)E The product MN ca be computed usig these seve matrices Claim 6 ( ) ( ) ( ) A B E F S + S = S + S 6 S + S 5 C D G H S 6 + S 7 S S 3 + S 5 S 7 Proof We will prove the claim for oly the upper right submatrix The proofs for the other three are similar I particular, we wat to show that S + S 5 = BH + AF Pluggig i defiitios, we obtai S + S 5 = (A + B)H + A(F H) = AH + BH + AF AH = BH + AF We ca ow write dow Strasse s Algorithm Algorithm 3 Strasse s Algorithm fuctio Strasse(M,N) if M is the retur M N ed if ( ) ( ) A B E F Let M = ad N = C D G H Set S = Strasse(B D, G + H) Set S = Strasse(A + D, E + H) Set S 3 = Strasse(A C, E + F ) Set S = Strasse(A + B, H) Set S 5 = Strasse(A, F H) Set S 6 = Strasse(D, G E) Set S 7 = ( Strasse(C + D, E) ) S + S retur S + S 6 S + S 5 S 6 + S 7 S S 3 + S 5 S 7 ed fuctio Correctess The correctess of this algorithm follows immediately from Claim 6 5

6 Rutime There are seve recursive calls ad the additios ad subtractios take time c for some costat c The rutime T () therefore satisfies T () 7T (/) + c T () = () By Claim 5, the rutime is O( log 7 ) As a side ote, we poit out that we are doig 8 additios of / / matrices, so we ca take the costat c to be 9/ We make two remarks about this algorithm First, eve though it is asymptotically faster tha the aive algorithm, Strasse s Algorithm does ot become faster tha the aive algorithm util the dimesio of the matrices is o the order of 0,000 Istead of recursig all the way dow to matrices, we ca just ru the aive algorithm oce the matrices become small eough to get a faster algorithm Secod, we have ot paid much attetio to the space the algorithm is usig Computig S,, S 7 ad the matrix that we retur i the last step requires O( ) space We ca therefore write a recursio that is idetical to () for space to see that the total space used is O( log 7 ) The aive algorithm uses oly O( ) space: We eed space for the iputs A ad B, allocate space for the output, ad the costruct the output by computig each A it A tj ad addig it to locatio (i, j) i the output matrix 5 Ruig Strasse s Algorithm usig oly O( ) space By beig a bit more careful about how we use space, we ca also ru Strasse s Algorithm i O( ) space Rather tha computig each of the S i s i its ow space, we ca use the same space for each of these recursive calls We start with 3 space for the two iput matrices ad the output matrix We the allocate 3(/) space for the recursive call to compute S Oce S is retured, we add it to the upper left quadrat of the output matrix I this same 3(/) space, we compute S, ad the we add it to the upper left ad lower right quadrats of the output matrix We cotiue i the same maer for the rest of the S i s Let W () be the amout of memory eeded to multiply two matrices From the above discussio, we see that W () = 3 + W (/) Claim 7 W () The picture below shows that the claim holds 6

7 The three squares are the space eeded for the top-level call, the three / / squares are the space eeded for the first level of recursio, etc We will also give a more formal proof i Sectio 3 3 Solvig recurreces I this sectio, we will show how to solve the three recurreces metioed i the last sectio usig the tree of recursive calls method Claim If it holds that the T () = O( 3 ) T () 8T (/) + c T () =, To solve this recurrece, we recursively evaluate T ad write resultig fuctio calls i a tree Problem size Work at each level c 8 childre 8c ( ) = c ( 3 ) c ( ) = c log c = O( 3 ) Total work: O( 3 ) We the calculate the total amout of work at each level ad sum over all levels of the tree Claim 5 Assume The T () = O( log 7 ) T () 7T (/) + c ad T () = To show this, we cosider a similar tree of recursive calls 7

8 Problem size Work at each level c 7 childre 7c ( ) = 7 c 7 c ( ) ( ) = 7 c ( ) 7 log c = O( log 7 ) Total work: O( log 7 ) Claim 7 Assume The W () W () = 3 + W (/) Here, there is oly oe subproblem, so the recursive tree is just a path W () = 3 + W (/) ( ) = W (/) = ( ) W (/8) ( = 3 + ) + ( ) / 3 / = 8