CS 140 : Feb 19, 2015 Cilk Scheduling & Applications

Transcription

1 CS 140 : Feb 19, 2015 Cilk Scheduling & Applications Analyzing quicksort Optional: Master method for solving divide-and-conquer recurrences Tips on parallelism and overheads Greedy scheduling and parallel slackness Cilk runtime Thanks to Charles E. Leiserson for some of these slides 1

2 otential arallelism Because the Span Law dictates that T T, the maximum possible speedup given T 1 and T is T 1 /T = potential parallelism = the average amount of work per step along the span. 2

3 Sorting Sorting is possibly the most frequently executed operation in computing! Quicksort is the fastest sorting algorithm in practice with an average running time of O(N log N), (but O(N 2 ) worst case performance) Mergesort has worst case performance of O(N log N) for sorting N elements Both based on the recursive divide-andconquer paradigm 3

4 QUICKSORT Basic Quicksort sorting an array S works as follows: If the number of elements in S is 0 or 1, then return. ick any element v in S. Call this pivot. artition the set S-{v} into two disjoint groups: S 1 = {x ε S-{v} x v} S 2 = {x ε S-{v} x v} Return quicksort(s 1 ) followed by v followed by quicksort(s 2 ) 4

5 QUICKSORT Select ivot

6 QUICKSORT artition around ivot

7 QUICKSORT Quicksort recursively

8 arallelizing Quicksort Serial Quicksort sorts an array S as follows: If the number of elements in S is 0 or 1, then return. ick any element v in S. Call this pivot. artition the set S-{v} into two disjoint groups: S 1 = {x ε S-{v} x v} S 2 = {x ε S-{v} x v} Return quicksort(s 1 ) followed by v followed by quicksort(s 2 ) 8

9 arallel Quicksort (Basic) The second recursive to qsort does not depend on the results of the first recursive We have an opportunity to speed up the by making both s in parallel. template <typename T> void qsort(t begin, T end) { if (begin!= end) { T middle = partition(begin, end, ); } } cilk_ qsort(begin, middle); qsort(max(begin + 1, middle), end); cilk_sync; // No cilk_ on this line! 9

10 Actual erformance./qsort cilk_set_worker_count 1 >> seconds./qsort cilk_set_worker_count 16 >> seconds Speedup = T 1 /T 16 = 0.083/0.014 =

11 Actual erformance./qsort cilk_set_worker_count 1 >> seconds./qsort cilk_set_worker_count 16 >> seconds Speedup = T 1 /T 16 = 0.083/0.014 = 5.93./qsort cilk_set_worker_count 1 >> seconds./qsort cilk_set_worker_count 16 >> 1.58 seconds Speedup = T 1 /T 16 = 10.57/1.58 = 6.67 Why not better??? 11

12 Measure Work/Span Empiriy cilkview -w./qsort Work = Span = Burdened span = arallelism = Burdened parallelism = #Spawn = #Atomic instructions = 14 cilkview -w./qsort Work = Span = Burdened span = arallelism = Burdened parallelism = #Spawn = #Atomic instructions = 8 workspan ws; ws.start(); sample_qsort(a, a + n); ws.stop(); ws.report(std::cout); 12

13 Analyzing Quicksort Quicksort recursively Assume we have a great partitioner that always generates two balanced sets 13

14 Analyzing Quicksort Work: T 1 (n) = 2T 1 (n/2) + Θ(n) 2T 1 (n/2) = 4T 1 (n/4) + 2 Θ(n/2).. + n/2 T 1 (2) = n T 1 (1) + n/2 Θ(2) T 1 (n) = Θ(n lg n) Span recurrence: T (n) = T (n/2) + Θ(n) Solves to T (n) = Θ(n) 14

15 Analyzing Quicksort arallelism: T 1 (n) T (n) = Θ(lg n) Not much! Indeed, partitioning (i.e., constructing the array S 1 = {x ε S-{v} x v}) can be accomplished in parallel in time Θ(lg n) Which gives a span T (n) = Θ(lg 2 n ) And parallelism Θ(n/lg n) Way better! Basic parallel qsort can be found under $cilkpath/examples/qsort 15

16 The Master Method (Optional) The Master Method for solving recurrences applies to recurrences of the form T(n) = a T(n/b) + f(n) *, where a 1, b > 1, and f is asymptotiy positive. IDEA: Compare n log ba with f(n). * The base case is always T(n) = Θ(1) for sufficiently small n. 16

17 Master Method CASE 1 T(n) = a T(n/b) + f(n) n log b a f(n) Specifiy, f(n) = O(n log b a ε ) for some const ε > 0 Solution: T(n) = Θ(n log b a ) Strassen matrix multiplication: a =7, b=2, f(n) = n 2 è T 1 (n) = Θ(n log 2 7 ) = O(n 2.81 ) 17

18 Master Method CASE 2 T(n) = a T(n/b) + f(n) n log b a f(n) Specifiy, f(n) = Θ(n log b a lg k n) for some const k 0 Solution: T(n) = Θ(n log b a lg k+1 n)) quicksort work: a=2, b=2, f(n)=n, k=0 è T 1 (n) = Θ(n lg n) qsort span: a=1, b=2, f(n)=lg n, k=1 è T (n) = Θ(lg 2 n) 18

19 Master Method CASE 3 T(n) = a T(n/b) + f(n) n log b a f(n) Specifiy, f(n) = Ω(n log b a + ε ) for some const ε>0, and (regularity) a f(n/b) c f(n) for some const c<1 Solution: T(n) = Θ(f(n)) Eg: qsort span (bad version): a=1, b=2, f(n)=n è T (n) = Θ(n) 19

20 Master Method Summary T(n) = a T(n/b) + f(n) CASE 1: f (n) = O(n log ba ε ), constant ε > 0 T(n) = Θ(n log ba ). CASE 2: f (n) = Θ(n log ba lg k n), constant k 0 T(n) = Θ(n log ba lg k+1 n). CASE 3: f (n) = Ω(n log ba + ε ), constant ε > 0, and regularity condition T(n) = Θ(f(n)). 20

21 otential arallelism Because the Span Law dictates that T T, the maximum possible speedup given T 1 and T is T 1 /T = potential parallelism = the average amount of work per step along the span. 21

22 Three Tips on arallelism 1. Minimize span to maximize parallelism. Try to generate 10 times more parallelism than processors for near-perfect linear speedup. 2. If you have plenty of parallelism, try to trade some if it off for reduced work overheads. 3. Use divide-and-conquer recursion or parallel loops rather than ing one small thing off after another. Do this: Not this: cilk_for (int i=0; i<n; ++i) { foo(i); } for (int i=0; i<n; ++i) { cilk_ foo(i); } cilk_sync; 22

23 Three Tips on Overheads 1. Make sure that work/#s is not too small. Coarsen by using function s and inlining near the leaves of recursion rather than ing. 2. arallelize outer loops if you can, not inner loops (otherwise, you ll have high burdened parallelism, which includes runtime and scheduling overhead). If you must parallelize an inner loop, coarsen it, but not too much. 500 iterations should be plenty coarse for even the most meager loop. Fewer iterations should suffice for fatter loops. 3. Use reducers only in sufficiently fat loops. 23

24 Scheduling A strand is a sequence of instructions that doesn t contain any parallel constructs Cilk allows the programmer to express potential parallelism in an application. The Cilk scheduler maps strands onto processors dynamiy at runtime. Since on-line schedulers are complicated, we ll explore the ideas with an off-line scheduler. Memory Network I/O $ $ $ 24

25 Greedy Scheduling IDEA: Do as much as possible on every step. Definition: A strand is ready if all its predecessors have executed. 25

26 Greedy Scheduling IDEA: Do as much as possible on every step. Definition: A strand is ready if all its predecessors have executed. Complete step strands ready. Run any. = 3 26

27 Greedy Scheduling IDEA: Do as much as possible on every step. Definition: A strand is ready if all its predecessors have executed. Complete step strands ready. Run any. Incomplete step < strands ready. Run all of them. = 3 27

28 Analysis of Greedy Theorem : Any greedy scheduler achieves T T 1 / + T. roof. # complete steps T 1 /, since each complete step performs work. # incomplete steps T, since each incomplete step reduces the span of the unexecuted dag by 1. = 3 28

29 Optimality of Greedy Theorem. Any greedy scheduler achieves within a factor of 2 of optimal. roof. Let T * be the execution time produced by the optimal scheduler. Since T * max{t 1 /, T } by the Work and Span Laws, we have T T 1 / + T 2 max{t 1 /, T } 2T *. 29

30 Linear Speedup Theorem. Any greedy scheduler achieves near-perfect linear speedup whenever T 1 /T. roof. Since T 1 /T is equivalent to T T 1 /, the Greedy Scheduling Theorem gives us T T 1 / + T T 1 /. Thus, the speedup is T 1 /T. Definition. The quantity T 1 /T is ed the parallel slackness. 30

31 Cilk Runtime System Each worker (processor) maintains a work deque of ready strands, and it manipulates the bottom of the deque like a stack Call! 31

32 Cilk Runtime System Each worker (processor) maintains a work deque of ready strands, and it manipulates the bottom of the deque like a stack Spawn! 32

33 Cilk Runtime System Each worker (processor) maintains a work deque of ready strands, and it manipulates the bottom of the deque like a stack Spawn! Call! Spawn! 33

34 Cilk Runtime System Each worker (processor) maintains a work deque of ready strands, and it manipulates the bottom of the deque like a stack Return! 34

35 Cilk Runtime System Each worker (processor) maintains a work deque of ready strands, and it manipulates the bottom of the deque like a stack Return! 35

36 Cilk Runtime System Each worker (processor) maintains a work deque of ready strands, and it manipulates the bottom of the deque like a stack Steal! When a worker runs out of work, it steals from the top of a random victim s deque. 36

37 Cilk Runtime System Each worker (processor) maintains a work deque of ready strands, and it manipulates the bottom of the deque like a stack Steal! When a worker runs out of work, it steals from the top of a random victim s deque. 37

38 Cilk Runtime System Each worker (processor) maintains a work deque of ready strands, and it manipulates the bottom of the deque like a stack When a worker runs out of work, it steals from the top of a random victim s deque. 38

39 Cilk Runtime System Each worker (processor) maintains a work deque of ready strands, and it manipulates the bottom of the deque like a stack Spawn! When a worker runs out of work, it steals from the top of a random victim s deque. 39

40 Cilk Runtime System Each worker (processor) maintains a work deque of ready strands, and it manipulates the bottom of the deque like a stack When a worker runs out of work, it steals from the top of a random victim s deque. 40

41 Cilk Runtime System Each worker (processor) maintains a work deque of ready strands, and it manipulates the bottom of the deque like a stack Theorem: With sufficient parallelism, workers steal infrequently linear speed-up. 41