Space-Time Trade-offs.

Transcription

1 Space-Time Trade-offs. Chethan Kamath Motivation An important question in the study of computation is how to best use the registers in a CPU. In most cases, the amount of registers avaiabe is not sufficient to hod a the data pertaining to a computation. Even though an increase in the number of registers coud ead to a decrease in the computation time, it is costy to incude more registers in the CPU. Therefore it is interesting to study how space can be traded off for time in computations. But how does one even approach such a question? It turns out that the key is to come up with the right abstraction: for studying space-time trade-offs, one reies on pebbing games on directed acycic graphs (DAGs). Bounds on the compexity of the pebbing games transate, roughy, to bounds on space and time used in computation (for certain modes of computation). Fast Fourier Transform. In this ecture we cover the space-time trade-off for the Fast Fourier Transform (FFT) agorithm. We wi see that the savings offered by the FFT agorithm (O(n og n), compared to O(n 2 ) for the naïve discrete Fourier transform (DFT) agorithm), can be ost if sufficient amount of storage is not avaiabe. To be precise, the FFT can be computed in O(n og n) time ony if Ω(n/og n) space is used concurrenty; if ony o(n/og n) space is used then the time required grows at a rate of ω(n/og n). Overview. First, in Section 2, we discuss the pebbing game and, in particuar, focus on the reevant definitions; as a concrete exampe, we demonstrate various pebbing strategies for the perfect binary tree of depth d, denoted BT (d). Next, in Section 3, we refresh the definitions of the DFT, discuss the FFT and discuss, in detai, the properties of the 2 d -point FFT graph, denoted F (d). In Sections 4 and 5, we estabish upper and ower bounds on the pebbing compexities of F (d). We concude with some remarks in Section 6. The write-up is a summary of a ecture given as part of the course Pears of Computer Science at IDC, Herziya. Most of the materia, in particuar the figures, in the write-up is from [Sav98] and [SS78]. 1

2 2 Pebbing The pebbing game on DAGs is used to abstract computation using straight-ine programs 1, which can be viewed as computation carried out on a DAG. Roughy speaking, vertices correspond to gates (depending on the mode e.g., Booean gates AND, OR or NOT, or agebraic gates + or ) and edges correspond to wires that bring input to the gates. A pebbe paced on a vertex indicates that the vaue associated with that vertex resides in the register. The rues of the game are the foowing: Rue 1. (Initiaization) A pebbe can be paced on an input vertex (i.e., a source) at any time. Rue 2. (Computation) A pebbe can be paced on (or moved to) any non-input vertex iff its parents 2 a carry pebbes. Rue 3. (Deetion) A pebbe can be removed at any time. The goa of the pebbing game is to pebbe each output vertex (i.e., a sink) at east once. Intuitivey, Rue 1 captures reading of an input in to a register; Rue 2 captures the computation of a vaue associated with a vertex; and Rue 3 captures erasure of vaue from the register. The goa captures computation of the output. Pebbing sequences. A (pebbing) sequence for a graph G = (V, E) is an execution of rues of the pebbing game on it. It is denoted by P := P 0, P 1,..., P T, where P 0 = and P i V, 1 i T are pebbing configurations. The sequence is vaid if configuration P i+1 was derived from P i, for each 1 i < T, by appying one of the pebbing rues. The compexity measures for sequences that we are interested are: 1. Time compexity: T G (P ) = T 2. Space compexity: S G (P ) = max 0 i T P i 3. Space-time compexity: ST G (P ) = S G (P ) T G (P ) Given a compexity measure C for a sequence P, the corresponding compexity measure for the graph C(G) is the minimum over a the possibe vaid sequences; i.e., C(G) = min P (C G (P )). Since there is a rough correspondence between a straight-ine program and its underying graph, ower bounds on pebbing compexities transates to ower bounds on the resource used in the computation: e.g., a ower bounds on the spacecompexity of the graph gives a rough estimate on the number of registers that are required for the corresponding computation. Therefore, for discussing tradeoff properties of, say, FFT, the semantics of the nodes in the straight-ine program is irreevant it suffices to know the structure of the graph that underies it. 1 A straight-ine program is a restricted mode of computation in which oops and conditiona statements are not aowed c.f., [Sav98, Section 2.2] for the forma definition. It covers, for exampe, ogic circuits as we as agebraic circuits. 2 For a graph G = (V, E), the parents of a vertex v V is the set {u : u V, (u, v) E}. 2

3 2.1 Exampe: The Perfect Binary Tree The perfect binary tree of depth 3 d is denoted by BT (d). Its vertices are arranged in d + 1 ayers, with ayer i, 0 i d, comprising of 2 d i vertices (e.g., see Figure 1) therefore, the number of vertices is V = d i=0 2i = 2 d+1 1. The one vertex that resides in ayer d is caed the root, whereas the vertices at ayer 0 are the eaves. Three pebbing strategies for BT (d) aong with their compexity is discussed beow; the choice of BT (d) for the demonstration is down, mainy, to two reasons: a) it serves as a cean, non-trivia exampe; and b) the arguments for the FFT graph F (d) are buit on top of those for BT (d) (c.f., for exampe Theorems 3 and 4). Figure 1: A perfect binary tree of depth 4 (BT (4) ). The vertices are arranged in five ayers (0 through 4) with 16 (eaves), 8, 4, 2 and 1 (root) vertices respectivey. The critica point for a particuar sequence for BT 4 has aso been highighted: the eaf vertex invoved is u = 26, and the sibings of the vertices on the critica path are {15, 18, 25, 27}. ([Sav98, Figure 10.2]) 1. The naïve strategy (P 1 ) is to pebbe ayer-by-ayer starting from the eaves and ending with the root, without removing any pebbe (see Figure 2.(a)). The space-compexity of the strategy is S BT (d)(p 1 ) = V = 2 d+1 1, which aso turns out to be its time-compexity. 2. The breadth-first strategy (P 2 ) aso invoves pacing pebbes ayer-by-ayer, but pebbes are removed as soon as they are of no use (see Figure 2.(b)). To be more precise, at any point in the sequence at most two ayers carry pebbes, and pebbes on the ower ayer is used to pebbe their chidren in the higher ayer and subsequenty removed. To be precise, the steps are: (a.) Pebbe ayer 0 from eft to right (b.) Repeat for each 1 i d: for each vertex on ayer i, move pebbe from the eft parent on to the vertex and remove the pebbe on the right parent. 3 The depth of a graph is the ength of its ongest path. 3

4 The number of pebbes used is the most at the end of Step (a.), and thus the space-compexity of the sequence is S BT (d)(p 2 ) = 2 d. The number of moves is 2 d d d = 2 (2 d + 2 d d ) 2 d and therefore T BT (d)(p 2 ) = 2 (2 d+1 1) 2 d. 3. The depth-first strategy (P 3 ) resuts in the pebbe-optima sequence for BT (d) (see Figure 2.(c)). The strategy is recursive in depth: to pebbe a particuar vertex v (a.) pebbe its eft parent (recursivey) (b.) pebbe its right parent (recursivey) (c.) move the pebbe on the eft parent on to v (d.) remove the pebbe on the right parent. To pebbe BT (d), the above steps are carried out on the root. The timecompexity of the sequence is captured by the recursion T (d) = 2 T (d 1)+2, with T (0) = 1. Hence T BT (d)(p 3 ) = 2 d+1. As for the space-compexity, we show next using induction (on depth) that the number of pebbes in pay is at most (d + 1). It is cear that P 3 pebbes BT (0) using a singe pebbe (the base case); et s assume that P 3 pebbes BT (d 1) using d pebbes (the induction hypothesis). Note that BT (d) comprises of two copies of BT (d 1). For the root of BT (d), Step (a) requires ony d pebbes (by the induction hypothesis); whie the right sub-tree is pebbed in Step (b) (using at most d pebbes), ony the root of the eft sub-tree carries a pebbe and therefore the number of pebbes in pay is at most d + 1. Note that athough P 1 is naïve in terms of the number of pebbes used, it is optima in terms of the time-compexity (as at each vertex in BT (d) must carry a pebbe at east once at some point during the pebbing). Next, we show that it takes at east (d + 1) pebbes to pebbe BT (d) i.e., P 3 is pebbe-optima. The proof traces back to [PH70], and serves a warm-up to ower-bounds for the FFT graphs,. Theorem 1 ([PH70]). S(BT (d) ) = d + 1. Proof. Consider the paths in BT (d). Initiay, no paths carry pebbes; in the end a the paths end up carrying a pebbe (as the root carries one). Now, consider the atest point in time in the sequence when a path was pebbe-free, denoted the critica point (see Figure 1). It must be the case that it is one of the eaves, denoted u, that is pebbed in this move eading to the (critica) path from u to the root being cosed. Since the rest of the paths carry a pebbe at the critica point, at east d pebbes must be present on the tree, one per sibing of the vertices on the critica path (i.e., one on each ayer). Counting aso the pebbe paced on u, it foows that S(BT (d) ) d + 1; taking into account the upper bound due to depth-first pebbing competes the proof. 4

5 (a) (b) (c) Figure 2: The pebbing strategies for BT (2) : naïve pebbing (a), breadth-first pebbing (b) and depth-first pebbing (c). 5

6 3 Fast Fourier Transform: a Refresher Let n = 2 d. The n-point discrete Fourier transform (DFT) F d : R n R n maps n-tupes (a 0,..., a n 1 ) over R to n-tupes (f 0,..., f n 1 ) over R, where f i = p(ω i ) and p(x) = n 1 i=0 a ix i, ω being the n th primitive root of unity. Naïve computation of DFT through poynomia mutipication takes O(n 2 ) operations, n operations per poynomia evauation. The fast Fourier transform (FFT) expoits the foowing decomposition of p(x): p(x) = a 0 + a 1 x + + a n 1 x n 1 = (a 0 + a 2 x a n 2 x n 2 ) + x(a 1 + a 3 x a n 1 x n 2 ) = p e (x 2 ) + xp o (x 2 ). In particuar, p(ω i ) = p e (ω 2i ) + ω i p o (ω 2i ). Since ω 2 is the (n/2) th principa root of unity, we have decomposed the computation of n-point FFT to the computation of two (n/2)-point FFTs, a ring mutipication and a ring addition operation. Thus, the time-compexity of n-point FFT is captured by the recurrence T (n) = 2T (n/2) + O(n), and hence T (n) = O(n og n). The FFT graph. The straight-ine programs that compute 2-point FFT (the butterfy graph) and 16-point FFT are shown in Figure 3. In genera, the straightine program that computes n-point FFT, has n (d + 1) nodes which are arranged in d + 1 ayers (i.e., depth is d) with n nodes per ayer. Therefore, we focus on the underying graph, denoted F (d), with nodes corresponding to vertices and wires to edges. F (d) has sever interesting structura properties, of which a coupe are highighted beow. (1.) Owing to the recursive definition of the FFT agorithm, the graph F (d) is highy recursive. In particuar, it can be viewed as composed of copies of smaer FFT graphs. For exampe, in Figure 3.(b), the graph F (4) is shown decomposed into eight copies of F (1) on top of two copies of F 8 (boxed). In genera, for 0 j d, F (d) can be viewed as decomposed into (a) 2 d j copies of F (j) denoted A (j) := A (j) 0,..., A(j) above; and 2 d j 1 (b) 2 j copies of F (d j) denoted B (d j) := B (d j) 0,..., B (d j) 2 j 1 beow (see Figure 4 for a schematic, and Figure 5 for an exampe). (2.) The sub-graph rooted at an output vertex (i.e., the sub-graph consisting of a vertices from which that output vertex is reachabe) is a perfect binary tree BT (d) as highighted in red in Figure 3. We wi refer frequenty to the representation of F (d) in Item (1.) and the observation in Item (2.) in order to expain pebbing strategies for F (d). 4 Upper Bounds In this section, we show that F (d) has ST-compexity of O(n 2 ). Intuitivey, this invoves estabishing that the best pebbing strategy given S pebbes is to use, 6

7 (a) (b) Figure 3: (a). The butterfy graph (F (1) ). (b) The 16-point FFT graph F (4). It consists of 5 ayers, with the input vertices at ayer 0 (a0, a8,..., a15, obtained through the bit-reversa permutation) and the output vertices at ayer 4 (f0,..., f15 ). The edges between ayers i 1 and i, for 1 i 4, are each associ4 i ated with the vaue ω 2 ; a vertex in ayer i with parent vertices vi 1,j and vi 1,k 4 i (in that order), for 0 i, j <, represents the computation vi 1,j + ω 2 vi 1,k (a basic F (1) operation). The graph F (4) is shown decomposed into eight copies of F (1) on top of two copies of F 8 (boxed). The perfect binary tree BT (d) rooted at f0 is highighted in red. ([Sav98, Figures 11.8, 6.7]) appropriatey, a hybrid of the breadth-first pebbing P2 (which is move-efficient) and the depth-first pebbing P3 (which is pebbe-optima). Theorem 2 ([SS78, Theorem 1]). The FFT graph F (d) can be pebbed in T moves with S pebbes where ( 2n2 /2j + (j + 1)n if S d + 2j j, 1 j d 1 T (1) n (d + 1) otherwise if S = 2d + 1. Proof sketch. First, we observe that the breadth-first pebbing P2 can be easiy adapted into a breadth-first pebbing for F (d), which uses at most (n + 1) pebbes and n (d + 1) moves4 : SF (j) (P2 ) = n + 1 and TF (j) (P2 ) = n (d + 1). This estabishes the second inequaity in (1); to estabish the first inequaity, sup4 There seems to be a mistake in [SS78] with regard to this caim: it is impossibe to pebbe in n(d + 1) moves, as caimed, without not dropping pebbes (and there are ony 2d + 1 pebbes avaiabe). It takes 3nd/2 + n moves to pebbe F (d) using 2d + 1 pebbes in the manner described above. Nevertheess, that does not make any (asymptotic) difference in the anaysis. 7

8 A (j) 0 A (j) 1 A (j) 2 A (j) A (j) A (j) 3 2 d j 2 2 d j 1 (Virtua edges) B (d j) 0 B (d j) 1 B (d j) 2 j 1 Figure 4: Schematic diagram showing the decomposition of F (d) into A (j) := A (j) 0,..., A(j) 2 d j 1 and B(d j) = B (d j) 0,..., B (d j) 2 j 1. The virtua edges between A (j) and B (d j) are determined as foows: the m th input to A (j) is the th output of B m (d j). The output edges, on the other hand, are determined as foows: the th output of F (d) is the r th output of A (j) s for unique r, s 0 : i = r 2 d j +s, s < 2 d j. pose that ony S = }{{} 2 j + (d j) }{{} pebbes for P 2 pebbes for P 3 pebbes, for 1 j d 1, are avaiabe. We view the graph F (d) as decomposed into A (j) and B (d j) and, on a high eve, use a strategy that is a hybrid of the breadth-first and depth-first pebbing strategies. The hybrid pebbing uses the (d j) pebbes to pebbe, depth-first, the perfect binary tree BT (d j) rooted at each input vertex of A (j), and then uses the 2 j pebbes to pebbe A (j) breadth-first. The steps are detaied beow. Repeat for each A (j) A (j) : Step 1. Repeat for each input vertex v m A (j) : Step 1.a Pebbe, depth-first, the binary tree BT (d j) contained in and rooted at v m (using (d j) pebbes) B (d j) m Step 2. Pebbe, breadth-first, the output vertices of A (j) (using 2 j pebbes) The tota number of moves is ( ) A (j) (# input vertices in A (j) #moves in Step 1.a) + #moves in Step 2) ( ) = A (j) (# input vertices in A (j) T BT (d j) ) (P 3) + T F (j)(p 2 )) = 2 d j ((2 j (2 d j+1 1)) + (j + 1) 2 j) = 2n 2 /2 j + (j + 1) n. That competes the proof. 8

9 Figure 5: The graph F (5) viewed as composed of eight copies of F (2) FFT graphs (A := A (2) 0,..., A(2) 7 ) and four copies of the F (3) graph (B := B (3) 0,..., B(3) 3 ). Note that the vertices in the bottom ayer of A and in the top ayer of B are, in fact, the same, and hence the vertices from B to A are virtua. The reason for their introduction is aesthetic, and to simpify exposition. ([Sav98, Figure 10.8]) 5 Lower Bounds In this section, we compement the upper bound in the previous section with an (amost) tight ower bound; that is, we show that ST(F (d) ) = Ω(n 2 ). Intuitivey, we show that the hybrid pebbing discussed in the previous section is the best one can hope for. We warm up with the bound for the extreme case of S = d + 1. Theorem 3 ([SS78, Theorem 2]). At east T n(n d)/2 + n 1 moves are necessary to pebbe F (d) with S = d + 1 pebbes. Proof sketch. The proof buids on that of Theorem 1. We view F (d) as decomposed into the n output vertices and two copies of F (d 1) denoted B (d 1) 0, B (d 1) 1 (e.g., see Figure 3). Consider the first critica point of a pebbing sequence (where the notion of critica point is same as in Theorem 1). By an argument simiar to that in the proof of Theorem 1, exacty one of the (d + 1) pebbes ies on eve d 1 in either B (d 1) 0 or B (d 1) 1 without oss of generaity, et it ie in B (d 1) 0. This particuar pebbe is usefu ony in pebbing the two output vertices that it is the parent of and is, after that, useess. Therefore, before the next critica point it must be that, for B (d 1) 0, either: a. a singe pebbe is brought to eve (d 1) using 2 d 1 moves; or b. d pebbes are brought to eves 0,..., d 2 with two pebbes at eve 0 this requires 1 + d 2 j=0 (2j+1 1) = 2 d d moves. As this needs to be repeated at east 2 d 1 times, the tota number of moves is at 9

10 east (2 d 1) + 2 d 1 (2 d d) = n(n d)/2 + n 1, where (2 d 1) is the number of moves made ti the first critica point. Next, we state and prove the fu theorem. Athough it buids on Theorem 3, the proof is quite fine-grained and as a resut very technica. Theorem 4 ([SS78, Theorem 3]). The number of moves necessary to pebbe F (d) using S < n/2 + 1 pebbes satisfies where j d 1 is determined by T n (j + 2) + n(n S)/2 j+1 (2) 2 j 1 + d (j 1) < S 2 j + (d j). (3) Proof sketch. As in the proof of Theorem 3, we decompose the graph into A (j) and B (d j), and show that pebbing A (j) breadth-first (once) and pebbing B (d j) depth-first (mutipe times) is the optima strategy. Hence, the upper bound is tight up to constant factors. We estabish the ower bound for A (j) and B (d j) separatey (and this expains the two terms in (2)). A. The ower-bound for A (j) is straightforward: we use the fact that each vertex must be visited at east once. Therefore, the number of moves invoved is at east }{{} 2 d j (2 j (j + 2)) }{{} A (j) A (j) and the number of pebbes in pay are at most 2 j. This estabishes the first term of (2). First critica point A heavy critica points v 1 v i v i+1 v i t 1 t i T t i Figure 6: The timeine of a pebbing for F (d). Critica points are marked in red; heavy critica points with respect to t i are marked by the onger red bar. t 1 is the first critica point; t i is the first non-heavy critica point with respect to t i. t i+1 B. Bounding the number of moves in B (d j) is non-trivia as the sub-trees are pebbed (depth-first) mutipe times. The argument is a fine-grained version of that in Theorem 3, and is summarised beow. (i.) Let s ook at the time-ine of a pebbing sequence for F (d) (see Figure 6). Let s examine the snapshot at a critica time t i : et the path cosed be for an output vertex vi, and et the path pass through A(j) 10

11 and B (d j) m, for some 0 < 2 d j and 0 m < 2 j. Assume that at this point of time (t i ), there are a pebbes on A(j) and b pebbes on B (d j) \ {B (d j) m } (i.e., on the bottom graphs except the one invoving the critica path). Let R denote a + b. It foows that S = a + b }{{} R + (d j + 1) }{{} #pebbes on B (d j) m (3) = 2 j 1 < R < 2 j. (4) (ii.) The first step is to show that the vertex vi is heavy at time t i, where an vi is deemed heavy (at time t i ) if at east a/2 of the 2j of its paths from the input vertices in A (j) are bocked by pebbes in the pebbing configuration P t i. This makes crucia use of the inequaity in (4). (c.f., [SS78, Lemmas 3 and 4]) (iii.) Next, we show that there are at most 2 j+1 heavy output nodes after vi at the critica point t i ; et s denote the first non-heavy output node at t i by vi, i > i. This is shown by estabishing that around 2 j+1 trees in B (d j) are empty therefore, a ot of pebbes (around 2 d S) are moved on B (d j) in between the critica points for vi and v i. (c.f., [SS78, Lemma 5]) (iv.) As there must be at east 2 d /2 j+1 snapshots of the above type for the whoe sequence, (ii.) and (iii.), together, impy that at east (2 d S) 2 d (j+1) moves are made overa. This estabishes the second term of (2). 6 Epiogue A. The ower-bound technique we saw is quite genera (see [Gri76], for exampe) and can be appied to other probems that have straight-ine programs. A coupe of exampes are given beow.s 1. For mutipication of two n n matrices (S + 1) T n 3 /4; since the naïve mutipication agorithm takes T = O(n 3 ) and O(1) space, it foows that (S + 1) T = Θ(n 3 ). 2. For mutipication of two n bit integers, (S + 1) T n 2 /64; however, no matching upper bound is known the best bound is (S + 1) T = O((nog n) 2 ). B. The pebbing paradigm is quite powerfu: it aows estabishing simiar bounds for other modes of computation. For exampe, it is possibe to capture parae computation if the pebbing moves are aowed to be made in parae (i.e., Rues 2 qne 3 can be carried out in parae). It is possibe to mode reversibe computation [Ben89] by adding a constraint to Rue 3: A pebbe can be removed ony if a its parents carry pebbes. 11

12 References [Ben89] Chares H. Bennett. Time/space trade-offs for reversibe computation. SIAM Journa on Computing, 18(4): , (Cited on page 11.) [Gri76] D. Grigoriev. An appication of separabiity and independence notions for proving ower bounds of circuit compexity. Notes of the Scientific Seminar Leningrad branch of the Stekov Institute, 60:38 48, (Cited on page 11.) [PH70] Michae S. Paterson and Car E. Hewitt. Record of the project mac conference on concurrent systems and parae computation. chapter Comparative Schematoogy, pages ACM, New York, NY, USA, (Cited on page 4.) [Sav98] John E. Savage. Modes of computation - exporing the power of computing. Addison-Wesey, (Cited on pages 1, 2, 3, 7 and 9.) [SS78] J. Savage and S. Swamy. Space-time trade-offs on the fft agorithm. IEEE Transactions on Information Theory, 24(5): , September (Cited on pages 1, 7, 9, 10 and 11.) 12