CSE 4500 (228) Fall 2010 Selected Notes Set 2

Transcription

1 CSE 4500 (228) Fall 2010 Selected Notes Set 2 Alexander A. Shvartsman Computer Science and Engineering University of Connecticut Copyright c by Alexander A. Shvartsman. All rights reserved.

2 2 Basic Algorithms: Parallel Tree Traversals HERE we present basic techniques where parallel processors operate in synchrony on binary trees stored in shared memory. We include code examples for enumerating processors and load balancing. The algorithms use full binary trees for purposes of accumulation and utilization of global knowledge about the computation. 2.1 Storing and Traversing Trees It is often desirable for trees used in parallel algorithms to not require explicit construction any maintenance of their topology. This is easily accomplished by storing a tree in an array with implicit tree structure as follows (Figure 2.2). tree[0... 6] Figure 2.2: Storing a maintenance-free binary tree in an array. A complete binary tree with n leaves is represented as the linear array tree[0... 2n 2]. The root of the tree is the first element of the array, tree[0]. Any non-leaf element tree[i] has the element tree[2i + 1] as its left child and the element tree[2i + 2] as its right child 1. When the number of leaves is not a power of two, conventional padding techniques can be used. Commonly algorithms use either the bottom-up or top-down traversal of such full binary trees. The number of leaves is normally equal to, or propor- 1 Equivalently, one can use a linear array tree[1... 2n 1], in which case the root of the tree is the first element of the array, tree[1], and any non-leaf element tree[i] has the element tree[2i] as its left child and the element tree[2i + 1] as its right child. 1

3 CSE 4500 (228) Fall 2010 Set 2, Page 2 Bottom-up parallel traversal of full binary trees: Computes or updates sums of values stored at the leaves O(log N) time O(P log N) work (or cost) Examples of use: processor enumeration, progress evaluation Top-down parallel traversal of full binary trees: Divide-and-conquer according to the hierarchy specified by the tree O(log N) time O(P log N) work (or cost) Examples of use: processor allocation and load balancing log N N log N N Figure 2.3: Synchronous bottom-up and top-down tree traversals. tional to the size of the input to the algorithm, say N. The height of the tree is logarithmic in N (assuming the degree of the tree is a constant). The bottom-up traversals can be easily used to implement, for example, a parallel summation algorithm, and the top-down traversal implement a divide-and-conquer processor allocation strategy using a hierarchy, see Figure 2.3. The traversals take logarithmic time. This tree scheme is readily generalized to trees other than binary. For example, for ternary trees, a non-leaf element i has three children whose indices are 3i + 1, 3i + 2, and 3i + 3 respectively. For q-ary trees, a non-leaf element i has q children whose indices are qi + 1, qi + 2,..., qi + q. The binary trees that we illustrate below be generalized to logarithmic (base q) time traversals of full q-ary trees for some constant q. For most simple algorithms the best performance is achieved with binary trees, but there are algorithms using trees that can be parameterized so that the desired performance bounds are achieved when the trees are not binary. Now we give a few intuition-building examples. The detailed code is given later in this chapter. First we give an illustration of the use of top-down traversal to count the number of active processors. This estimate is taken to be the number of active processors who may have reached the root of the tree:

4 CSE 4500 (228) Fall 2010 Set 2, Page 3 Example Processor counting: Consider a processor counting tree with N = 4 leaves. The tree is stored in the array c[0..6], initialized to 0. There are four processors with PIDs 1, 2, 3, and 4, that start at the leaves and traverse the tree towards its root. At each step, a processor writes into the node corresponding to its location in the tree, the sum of the two child locations. c[0] : c[1, 2] : c[3, 4, 5, 6] : PID: If processors with PIDs 2, 3 and 4 participate, then the root c[0] has the value 3. Note that processor 1 did not need to participate to obtain the correct result. The following example illustrates the use of a tree, called the progress tree, that is often used in the divide-and-conquer processor allocation. We assume that each leaf is associated with a task that needs to be performed by the processors. Here we evaluate the number of completed tasks based on the information brought to the root of the tree by the processors participating in the computation: Example Progress evaluation: Consider the progress tree with four leaves (N = 4) stored in the array d[0..6]. There are 4 processors with PIDs 1, 2, 3, and 4, that begin at the leaves and traverse the tree towards the root. Each processor begins by writing into its leaf an indication (value 1) that it finished its task; if a processor did not complete its task the leaf remains untouched (value 0). The processors move towards the root of the tree one level at a time, and store in each parent the sum of its children. If processors 1 and 2 finished their tasks, but 3 and 4 did not, then the tree d will look like this after the completion of the algorithm. d[0] : d[1, 2] : d[3, 4, 5, 6] : PID: Note that the value of the root d[0] is 2. This indicates that not all tasks were completed, since d[0] = 2 < 4 = N.

5 CSE 4500 (228) Fall 2010 Set 2, Page 4 In the load balancing top-down traversal, we assume that the progress tree is an exact summation tree (i.e, the value at each internal node is the sum of the values at its two children), and the processors are allocated at each non-leaf node of the tree in proportion to the number of remaining tasks in each subtree. Example Load balancing: Consider the progress tree with four leaves (N = 4) stored in the array a[0..6]. There are two active processors with PIDs 1 and 2, that begin at the root and traverse the tree towards the leaves. PID: a[0] : a[1, 2] : a[3, 4, 5, 6] : {1, 2} In this example, starting at the root, both processors will be sent to the right subtree. This is because out of the two tasks in the left subtree, both have been completed, and none of the tasks in the right subtree are completed. At the internal node a[2], the two processors will be equally divided between the left (a[5]) and the right (a[6]) subtrees, since equal number of unfinished tasks exists in each of the two subtrees. Note that the progress-assessment traversal in Example is guaranteed to yield the exact summation tree needed by the load balancing traversal in Example above. Algorithms that do not assume that the progress tree is a summation tree must be able to deal with such situations. (We used synchronous tree traversals in the examples. Some traversals of such progress trees can also be used when processors do so without synchronizing with each other.) 2.2 Processor Enumeration We now give three algorithms designed to enumerate the participating processors. These algorithms use an approach similar to the parallel-prefix computation. Each algorithm uses auxiliary tree data structures. The algorithms differ in their assumptions about the initial values at tree nodes. The first algorithm is the simplest: it assumes clear shared memory, that is, all nodes of the tree are intialized to zeros. The second algorithm uses time-stamps to ensure that the values in the data structure are fresh. It is somewhat more complicated we present it to illustrate the time-stamping technique.

6 CSE 4500 (228) Fall 2010 Set 2, Page 5 The third algorithm clears the needed memory before using it. Each algorithm is presented as a procedure that can be called by the processors from the high-level program that is specified for all processors, e.g., forall processors pid = 1..P parbegin Enumerate(...) parend Enumeration: using clear memory The enumeration is done using a bottom-up traversal of the tree c to compute the number of active processors in c[0]. Each processor computes its enumerated number in pn. The code is in Figure 2.4. Each processor PID starts by writing a 1 in the leaf c[pid+n 2] of the tree c. If a processor is inactive then its leaf will contain 0 and this will not contribute to the overall count. procedure Enumerate1( - - Processor enumeration integer pid, - - processor id integer pn) - - enumerated id shared integer c[0..2n 2] init 0, - - processor counts: clear memory private integer j1, j2, - - sibling indices: left or right t; - - parent of j1 and j2 begin j1 := pid + (N 2); - - tree-leaf init pn := 1; - - assume this PID is no. 1 c[j1] := 1; - - count PID once - - Traverse the tree from leaf to root for 1... log(n) do t := (j1 1) div 2; - - parent of j1 and j2 if 2 t + 1 = j1 - - j1 came from... then j2 := j left subtree else j2 := j right subtree fi c[t] := c[j1] + c[j2] - - compute sum if j1 > j2 - - j1 came from right then pn := pn + c[j2] fi j1 := t - - advance up the tree rof end. Figure 2.4: Enumeration using clear memory We enumerate the active processors by each processor assuming that it is the only active one, and then adding the number of the active processors it

7 CSE 4500 (228) Fall 2010 Set 2, Page 6 sees to its left. This is a variation of the parallel prefix computation. This enumeration creates the dynamic processor number pn Enumeration: using time-stamps This is essentially the same algorithm, but it introduces the use of time-stamps. The tree cs is used to ensure that only the recent values are used when the algorithm is used multiple times. The code is in Figure 2.5. procedure Enumerate2( - - Processor enumeration integer pid, - - processor id integer step, - - time-stamp integer pn) - - enumerated id shared integer c[0..2n 2], - - processor counts cs[0..2n 2]; - - associated time-stampts step init time-stamp private integer j1, j2, - - sibling indices t; - - parent of j1 and j2 begin step := step + 1; - - new time-stamp j1 := pid + (N 2); - - tree-leaf init pn := 1; - - assume this PID is no. 1 c[j1] := 1; - - count PID once cs[j1] := step; - - update time-stamp - - Traverse the tree from leaf to root for 1... log(n) do t := (j1 1) div 2; - - parent of j1 and j2 if 2 t + 1 = j1 - - j1 came from... then j2 := j left subtree else j2 := j1 1 fi - - right subtree if cs[j1] = cs[j2] - - active sibling? then c[t] := c[j1] + c[j2] - - yes, compute sum if j1 > j2 - - j1 came from right then pn := pn + c[j2] fi else c[t] := c[j1] - - no active siblings fi cs[t] := step; - - time stamp, and j1 := t - - advance up the tree rof end. Figure 2.5: Enumeration using time-stamps We must be able to reuse our tree during several uses of the enumeration algorithm For example, if a processor had written 1 into its tree leaf and then became inactive then, the value 1 will remain there for the duration of the com-

8 CSE 4500 (228) Fall 2010 Set 2, Page 7 putation, thus preventing us from accurately computing the number of active processors. This is corrected by associating a step number with each node of the count tree c and storing it in tree cs, thus time-stamping valid data. The count step is initially zero, and during each use of the algorithm it gets incremented by each active processor. The values written previously by inactive processors will have old step numbers, and active processors detect counts that are out-of-date and treat them as zeroes Enumeration: clearing old data In this similar algorithm each active processor assumes that there are no active siblings at any tree node. This is done by clearing the sibling s value in the tree c and then writing the new value in the processor s current location in c. The code is in Figure 2.6. procedure Enumerate3( - - Processor enumeration integer pid, - - processor id integer step, - - time-stamp integer pn) - - enumerated id shared integer c[1..2n 2], - - processor counts private integer st, - - running subtree subtotal j1, j2, - - sibling indices t; - - parent of j1 and j2 begin j1 := P ID + (N 2); - - tree-leaf init pn := 1; - - assume this PID is no. 1 st := 1; - - count PID once - - Traverse the tree from leaf to root for 1... log(n) do t := (j1 1) div 2; - - parent of j1 and j2 if 2 t + 1 = j1 - - j1 came from... then j2 := j left subtree else j2 := j1 1 fi - - right subtree c[j2] := 0; - - assume sibling(s) is inactive c[j1] := st; - - self is active if j1 > j2 - - j1 came from right then pn := pn + c[j2] fi st := c[j1] + c[j2]; - - update tree subtotal, and j1 := t - - advance up the tree rof end. Figure 2.6: Enumeration and clearing old data

9 CSE 4500 (228) Fall 2010 Set 2, Page Progress Assessment Processors execute this algorithm having completed (or not completed) their individual tasks at the time the coordination needs to take place. A processor that worked on task number k makes a call to procedure Progress(k, done), where done is the task completion status. The processors proceed in synchrony from leaf to root and build the progress tree d where each subtree root contains the number of completed tasks in its subtree. Note that if the progress tree is initialized to zeros then only the processors that completed their tasks need to participate. The resulting progress tree is a summation tree where each interior node is the sum of its two children. procedure Progress( integer k, - - Current processor s task index boolean done) - - Task completion: true/false shared integer array d[1..2n 1]; - - done/progress tree private integer i1, i2 - - left/right (or left/right) sibling indices t; - - parent of i1 and i2 - - Mark the leaf done/not done i1 := k + (N 1); - - progress tree leaf index if done then d[i1] := 1; - - done else d[i1] := 0; - - not done fi - - Traverse the tree from leaf to root for 1... log(n) do - - The parent index of i1 and i2 t := (i1 1) div Compute left/right indices if 2 t + 1 = i1 - - i1 came from... then i2 := i from left, i2 is on the right else i2 := i from right, i2 is on the left fi - - Update progress tree and advance d[t] := d[i1] + d[i2]; - - update i1 := t - - advance to the parent od end. Figure 2.7: Progress estimation building a divide-and-conquer hierarchy

10 CSE 4500 (228) Fall 2010 Set 2, Page Load balancing according to a hierarchy The load balancing algorithm uses a progress tree, such as the one built by procedure Progress(, ) in Figure 2.7. The processors traverse the tree from root to leaf and allocate themselves to the left/right subtree in proportion to the tasks the remain undone in each subtree. When a processor reaches a leaf it computes the index k of the task it is going to perform. The algorithm assumes that the available processors are sequentially numbered starting with 1. Each processor uses its enumerated processor number pn instead of its permanent PID. The enumerated numbers can be computed by any of the procedures in Figures 2.4, 2.5 or 2.6. procedure Divide-and-Conquer( integer pn, - - dynamic processor number integer P, - - dynamic processor number integer k) - - computed data (work item) index shared integer array c[0..2n 2], - - processor counts (does not need to be shared) d[0..2n 2]; - - progress/done tree private integer j, j1, j2; - - current/left/right indices j := 0; - - start at the root of the progress tree size := N; - - the whole tree is visible from the root (N is const.) c[0] := P ; - - the root of the processor tree is P - - Traverse from root to leaf while size 1 do j1 := 2 j + 1; j2 := 2 j + 2; - - left/right - - Processor allocation to left/right sub-trees c[j1] := c[j] (size/2 d[j1]) div (size d[j]); c[j2] := c[j] c[j1]; - - Go left/right based on proc. no. if pn c[j1] then j := j1 else j := j2; pn := pn c[j1] fi size := size div half of leaves for next iteration od ; k := j (N 1) - - assign to task k based on j end. Figure 2.8: Load balancing according to a hierarchy

11 CSE 4500 (228) Fall 2010 Set 2, Page Broadcast Using a Tree Now we consider a simple algorithm for CREW PRAM that allows one processor to broadcast its private value to all processors. The algorithm is given in Figure 2.9. For simplicity we assume that P = 2 k 1 for some integer k > 0, and that processor with pid = 1 needs to broadcast its private value. The broadcast is done according to the tree B[1..P ] (line 02). For this algorithm the root of the tree is the first element of the array, B[1], and any non-leaf element B[i] has the element B[2i] as its left child and the element B[2i + 1] as its right child. Therefore for any node B[j] of the tree, its parent is node B[j div 2]. The algorithm starts with processor 1 writing its value to the root of the tree (line 03). Then, for each of the log P remaining levels of the tree, the processors at level h read the values from their corresponding parents at level h 1 (lines 04-07). 01 forall processors pid = 1..P parbegin 02 shared B[1..P ] - - Broadcast tree, with one node per processor 03 if pid = 1 then B[1] := value fi - - Processor 1 writes its value to the root 04 for h = 1 to log P do - - For each level h of the broadcast tree 05 if 2 h pid < 2 h+1 then - - The processors at level h 06 B[pid] := B[pid div 2] - - Read parent s value 07 od 08 parend. Figure 2.9: Pseudocode for the CREW PRAM broadcast algorithm The algorithm takes Θ(log P ) time, and its work (or cost) is Θ(P log P ). Note that the algorithm above does require concurrent reads: two siblings read the value from the their parent concurrently when they pull the broadcast value from the parent. We next slightly modify this algorithm, so that each parent pushes its value to both children, allowing the algorithm to be executed on EREW PRAM. The algorithm is gven in Figure The algorithm uses the same broadcast tree B[1..P ] (line 02). The algorithm starts with processor 1 writing its value to the root of the tree (line 03). Then, for each of the log P remaining levels of the tree, starting with the root, the processors at level h write their value to both of its children at level h + 1 (lines 04-08). The complexity of the algorithm is unchanged.

12 CSE 4500 (228) Fall 2010 Set 2, Page forall processors pid = 1..P parbegin 02 shared B[1..P ] - - Broadcast tree, with one node per processor 03 if pid = 1 then B[1] := value fi - - Processor 1 writes its value to the root 04 for h = 0 to log P 1 do - - For each level h of the broadcast tree 05 if 2 h pid < 2 h+1 then - - The processors at level h 06 B[2pid] := B[pid] - - Push value to left child 07 B[2pid + 1] := B[pid] - - Push value to right child 07 od 09 parend. Figure 2.10: Pseudocode for the EREW PRAM broadcast algorithm