COMP 322: Fundamentals of Parallel Programming. Lecture 22: Parallelism in Java Streams, Parallel Prefix Sums

Transcription

1 COMP 322: Fundamentals of Parallel Programming Lecture 22: Parallelism in Java Streams, Parallel Prefix Sums Vivek Sarkar, Shams Imam Department of Computer Science, Rice University Contact COMP 322 Lecture March 2016

2 Worksheet #21 solution: Abstract Metrics with Isolated Constructs Q: Compute the WORK and CPL metrics for this program. Indicate if your answer depends on the execution order of isolated constructs. 1. finish(() -> { 2. for (int i = 0; i < 5; i++) { 3. async(() -> { 4. dowork(2); 5. isolated(() -> { dowork(1); }); 6. dowork(2); 7. }); // async 8. } // for 9. }); // finish Answer: WORK = 25, CPL = 9. These metrics do not depend on the execution order of isolated constructs. 2 COMP 322, Spring 2016 (V. Sarkar, S. Imam)

3 How Java Streams addressed pre-java-8 limitations of Java Collections 1. Iteration had to be performed explicitly using for/foreach loop, e.g., // Iterate through students (collection of Student objects) for (Student s in students) System.out.println(s); Simplified using Streams as follows students.stream().foreach(s -> System.out.println(s)); 2. Overhead of creating intermediate collections List<Student> activestudents = new ArrayList<Student>(); for (Student s in students) if (s.getstatus() == Student.ACTIVE) activestudents.add(s); for (Student a in activestudents) totalcredits += a.getcredits(); Simplified using Streams as follows totalcredits = students.stream().filter(s -> s.getstatus() == Student.ACTIVE).map(a -> a.getcredits()).sum(); 3. Complexity of parallelism simplified (for example) by replacing stream() by parallelstream() 3 COMP 322, Spring 2016 (V. Sarkar, S. Imam)

4 Java 8 Streams Cheat Sheet Source: 4 COMP 322, Spring 2016 (V. Sarkar, S. Imam)

5 Parallelism in processing Java Streams Parallelism can be introduced at a stream source e.g., library.parallelstream() or as an intermediate operation e.g., library.stream().sorted().parallel() Stateful intermediate operations should be avoided on parallel streams e.g., distinct, sorted, use-written lambda with side effects but stateless intermediate operations work just fine e.g., filter, map Parallelism is usually more efficient on unordered streams e.g., stream created from unordered source (HashSet), or from.unordered() intermediate operation and with unordered collectors e.g., ConcurrentHashMap 5 COMP 322, Spring 2016 (V. Sarkar, S. Imam)

6 Beyond Sum/Reduce Operations Prefix Sum (Scan) Problem Statement Given input array A, compute output array X as follows The above is an inclusive prefix sum since X[i] includes A[i] For an exclusive prefix sum, perform the summation for 0 <=j <i It is easy to see that inclusive prefix sums can be computed sequentially in O(n) time // Copy input array A into output array X X = new int[a.length]; System.arraycopy(A,0,X,0,A.length); // Update array X with prefix sums for (int i=1 ; i < X.length ; i++ ) X[i] += X[i-1]; and so can exclusive prefix sums 6 COMP 322, Spring 2016 (V. Sarkar, S. Imam)

7 An Inefficient Parallel Algorithm for Exclusive Prefix Sums 1. forall(0, X.length-1, (i) -> { 2. // computesum() adds A[0..i-1] 3. X[i] = computesum(a, 0, i-1); 4. } Observations: Critical path length, CPL = O(log n) Total number of operations, WORK = O(n 2 ) With P = O(n) processors, the best execution time that you can achieve is T P = max(cpl, WORK/P) = O(n), which is no better than sequential! 7 COMP 322, Spring 2016 (V. Sarkar, S. Imam)

8 How can we do better? Assume that input array A = [3, 1, 2, 0, 4, 1, 1, 3] Define scan(a) = exclusive prefix sums of A = [0, 3, 4, 6, 6, 10, 11, 12] Hint: Compute B by adding pairwise elements in A to get B = [4, 2, 5, 4] Assume that we can recursively compute scan(b) = [0, 4, 6, 11] How can we use A and scan(b) to get scan(a)? 8 COMP 322, Spring 2016 (V. Sarkar, S. Imam)

9 Another way of looking at the parallel algorithm Observation: each prefix sum can be decomposed into reusable terms of power-of-2-size e.g. Approach: Combine reduction tree idea from Parallel Array Sum with partial sum idea from Sequential Prefix Sum Use an upward sweep to perform parallel reduction, while storing partial sum terms in tree nodes Use a downward sweep to compute prefix sums while reusing partial sum terms stored in upward sweep 9 COMP 322, Spring 2016 (V. Sarkar, S. Imam)

10 Parallel Prefix Sum: Upward Sweep (while calling scan recursively) Upward sweep is just like Parallel Reduction, except that partial sums are also stored along the way 1. Receive values from left and right children 2. Compute left+right and store in box 3. Send left+right value to parent Input array, A: 10 COMP 322, Spring 2016 (V. Sarkar, S. Imam)

11 Parallel Prefix Sum: Downward Sweep (while returning from recursive calls to scan) 1. Receive value from parent (root receives 0) 2. Send parent s value to LEFT child (prefix sum for elements to left of left child s subtree) 3. Send parent s value+ left child s box value to RIGHT child (prefix sum for elements to left of right child s subtree) 4. Add A[i] to get inclusive prefix sum Exclusive prefix sums + A[i] Inclusive prefix sums COMP 322, Spring 2016 (V. Sarkar, S. Imam)

12 Summary of Parallel Prefix Sum Algorithm Critical path length, CPL = O(log n) Total number of add operations, WORK = O(n) Optimal algorithm for P = O(n/log n) processors Adding more processors does not help Parallel Prefix Sum has several applications that go beyond computing the sum of array elements Parallel Prefix Sum can be used for any operation that is associative (need not be commutative) In contrast, finish accumulators required the operator to be both associative and commutative 12 COMP 322, Spring 2016 (V. Sarkar, S. Imam)

13 Parallel Filter Operation [Credits: David Walker and Andrew W. Appel (Princeton), Dan Grossman (U. Washington)] Given an array input, produce an array output containing only elements such that f(elt) is true, i.e., output = input.parallelstream.filter(f).toarray Example: input [17, 4, 6, 8, 11, 5, 13, 19, 0, 24] f: is elt > 10 output [17, 11, 13, 19, 24] Parallelizable? Finding elements for the output is easy But getting them in the right place seems hard 13 COMP 322, Spring 2016 (V. Sarkar, S. Imam)

14 Parallel prefix to the rescue 1. Parallel map to compute a bit-vector for true elements (can use Java streams) input [17, 4, 6, 8, 11, 5, 13, 19, 0, 24] bits [1, 0, 0, 0, 1, 0, 1, 1, 0, 1] 2. Parallel-prefix sum on the bit-vector (not available in Java streams) bitsum [1, 1, 1, 1, 2, 2, 3, 4, 4, 5] 3. Parallel map to produce the output (can use Java streams) output [17, 11, 13, 19, 24] output = new array of size bitsum[n-1] FORALL(i=0; i < input.length; i++){ if(bits[i]==1) output[bitsum[i]-1] = input[i]; } 14 COMP 322, Spring 2016 (V. Sarkar, S. Imam)

15 Parallelizing Quicksort (Remember Homework 1?) Best / expected case work 1. Pick a pivot element O(1) 2. Partition all the data into: O(n) A. The elements less than the pivot B. The pivot C. The elements greater than the pivot 3. Recursively sort A and C 2T(n/2) Simple approach: Do the two recursive calls in parallel Work: unchanged at O(n log n) Span: now CPL(n) = O(n) + CPL(n/2) = O(n) So parallelism (i.e., work / span) is O(log n) Sophisticated approach: use scans for the partition step Work: unchanged at O(n log n) Span: now CPL(n) = O(log n) + CPL(n/2) = O(log 2 n) So average parallelism (i.e., work / span) is O(n / log n) 15 COMP 322, Spring 2016 (V. Sarkar, S. Imam)

16 Example Step 1: pick pivot as median of three Steps 2: implement partition step as two filter/pack operations that store result in a second array Step 3: Two recursive sorts in parallel 16 COMP 322, Spring 2016 (V. Sarkar, S. Imam)