Vorlesung Datenstrukturen und Algorithmen Letzte Vorlesung Felix Friedrich, Map/Reduce Sorting Networks Prüfung

Transcription

1 Vorlesung Datenstrukturen und Algorithmen Letzte Vorlesung 28 Felix Friedrich, Map/Reduce Sorting Networks Prüfung

2 MAP AND REDUCE AND MAP/REDUCE 2

3 Summing a Vector Accumulator Divide and conquer Q: Why is the result the same? A: associativity: (a+b) + c = a + (b+c) 3

4 Summing a Vector Divide and conquer Is this correct? Only if the operation is commutative: a + b = b + a 4

5 Reductions Simple examples: sum, max Reductions over programmer-defined operations operation properties (associativity / commutativity) define the correct executions supported in most parallel languages / frameworks powerful construct 5

6 C++ Reduction std::accumulate (requires associativity) std::reduce (requires commutativity, from C++7, can specify execution policy) std::vector<double> v;... double result = std::accumulate( v.begin(), v.end(),., [](double a, double b){return a + b;} ); 6

7 Elementwise Multiplication Map Multiply x x x x x x x x + 7

8 Scalar Product Map Multiply x x x x x x x x Reduce + Accumulate

9 C++ Scalar Product (map + reduce) // example data std::vector<double> v(24,.5); auto v2 = v; std::vector<double> result(24); // map std::transform(v.begin(), v.end(), v2.begin(), result.begin(), [](double a, double b){return a*b;}); // reduce double value = std::accumulate(result.begin(), result.end(),.); // = 256 9

10 Map & Reduce = MapReduce Combination of two parallelisation patterns result = f in f in 2 f(in 3 ) f(in 4 ) f = map = reduce (associative) Examples: numerical reduction, word count in document, (word, document) list, maximal temperature per month over 5 years (etc.)

11 Motivating Example Maximal Temperature per Month for 5 years Input: 5 * 365 Days / Temperature pairs Output: 2 Months / Max Temperature pairs Assume we (you and me) had to do this together. How would we distribute the work? What is the generic model? How would we be ideally prepared for different reductions (min, max, avg)?

12 Maximal Temperature per Month: Map data / 5 / 8 / / 2 3 / /2 5/9 5/2... 7/28 7/ /4 3/8... each map-process gets day/temperature pairs and maps them to month/temperature pairs... 2

13 Maximal Temperature per Month: Shuffle / -5 / -8 / / 2 3 / /2 5/9 5/2... 7/28 7/ /4 3/ Jan Feb Mar April Dec / -5 / -8 / / 3 2 / 4 2 / / 3 3 / 4 3 / / 23 4 / 24 4 / data gets sorted / shuffled by month 2 / 2 / -2 2/

14 Maximal Temperature per Month: Reduce Jan Feb Mar April Dec / -5 / -8 / / 3 2 / 4 2 / / 3 3 / 4 3 / / 23 4 / 24 4 / / 2 / -2 2/ 2 each reduce process gets its own... month with values and applies the reduction (here: max value) to it Jan Feb Mar April Dec

15 Map/Reduce A strategy for implementing parallel algorithms. map: A master worker takes the problem input, divides it into smaller sub-problems, and distributes the sub-problems to workers (threads). reduce: The master worker collects sub-solutions from the workers and combines them in some way to produce the overall answer. 5

16 Map/Reduce Frameworks and tools have been written to perform map/reduce. MapReduce framework by Google Hadoop framework by Yahoo! related to the ideas of Big Data and Cloud Computing also related to functional programming (and actually not that new) and available with the Streams concept in Java (>=8) Map and reduce are user-supplied plug-ins, the rest is provided by the frameworks. Jeffrey Dean and Sanjay Ghemawat. 28. MapReduce: simplified data processing on large clusters. Commun. ACM 5, (January 28), 7-3. DOI=.45/

17 MapReduce on Clusters You may have heard of Google s map/reduce or Amazon's Hadoop Idea: Perform maps/reduces on data using many machines The system takes care of distributing the data and managing fault tolerance You just write code to map one element (key-value part) and reduce elements (key-value pairs) to a combined result Separates how to do recursive divide-and-conquer from what computation to perform Old idea in higher-order functional programming transferred to large-scale distributed computing 7

18 Example Count word occurences in a very large file File = GBytes how are you today do you like the weather outside I do I do I wish you the very best for your exams. 8

19 Mappers key/value pairs (e.g. position / string) part <, "how are you today"> <5, "do you like..."> mapper huge file part 2 <35, "I do"> <39, "I do"> mapper 2 DISTRIBUTED part 3 <43, "I wish you the very best"> <7, "for your exams"> mapper 3 9

20 Mappers input output key/value pairs (e.g. position / string) key/value pairs (word, count) <, "how are you today"> <5, "do you like..."> mapper <"how",> <"are",> <"you",>... <35, "I do"> <39, "I do"> mapper 2 <"I",> <"do",> <"I",> <"do",> <43, "I wish you the very best"> <7, "for your exams"> mapper 3 <"I",> <"wish",> <"you",>... 2

21 Shuffle / Sort key/value pairs (word, count) unique key/value pairs (shuffled) (word, counts) mapper mapper 2 <"how",> <"are",> <"you",>... <"I",> <"do",> <"I",> <"do",> <"are",> <"best",>... <"do",,,> <"for",>... reducer reducer 2 DISTRIBUTED mapper 3 <"I",> <"wish",> <"you",>... 2

22 Reduce input unique key/value pairs (shuffled) (word, counts) output target file(s) <"are",> <"best",>... reducer are best you 3... <"do",,,> <"for",>... reducer 2 do 3 for I

23 SORTING NETWORKS 23

24 Lower bound on sorting Horrible algorithms: Ω(n 2 ) Bogo Sort Stooge Sort Simple algorithms: O(n 2 ) Insertion sort Selection sort Bubble Sort Shell sort Fancier algorithms: O(n log n) Heap sort Merge sort Quick sort (avg) Comparison lower bound: (n log n) 24

25 Comparator x y < min(x,y) max(x,y) x min(x,y) shorter notation: y max(x,y) 25

26 void compare(int&a, int&b, boolean dir) { if (dir==(a,b)){ std::swap(a,b); } } a b >< a b 26

27 Sorting Networks

28 Sorting Networks are Oblivious (and Redundant) Oblivious comparison tree :2 x x 2 x 3 x 4 3:4 3:4 :3 :4 2:3 2:4 2:4 2:4 2:3 2:3 :4 :4 :3 :3 2:3 4:3 2: 4: 2:4 3:4 2: 3: :3 4:3 :2 4:2 :4 3:4 :2 3:2 redundant cases 28

29 Recursive Construction : Insertion x x 2 x 3... sorting network x n x n x n+ 29

30 Recursive Construction: Selection x x 2 x sorting network... x n x n x n+ 3

31 Applied recursively.. insertion sort bubble sort with parallelism: insertion sort = bubble sort! 3

32 Question How many steps does a computer with infinite number of processors (comparators) require in order to sort using parallel bubble sort? Answer: 2n 3 Can this be improved? How many comparisons? Answer: (n-) n/2 How many comparators are required (at a time)? Answer: n/2 Reusable comparators: n- 32

33 Improving parallel Bubble Sort Odd-Even Transposition Sort:

34 void oddeventranspositionsort(std::vector<t>& v, boolean dir) { for (int i = ; i<v.size(); ++i){ for (int j = i % 2; j+<n; j+=2) compare(v[i],v[j],dir); } } 34

35 Improvement? Same number of comparators (at a time) Same number of comparisons But less parallel steps (depth): n In a massively parallel setup, bubble sort is thus not too bad. But it can go better... 35

36 Parallel sorting depth = 4 depth = 3 Prove that the two networks above sort four numbers. Easy? 36

37 Zero-one-principle Theorem: If a network with n input lines sorts all 2 n sequences of s and s into non-decreasing order, it will sort any arbitrary sequence of n numbers in nondecreasing order. 37

38 Proof Assume a monotonic function f(x) with f8 x2 3 f(y) 5 9 whenever 5 8 9x 2 3 y and Argue: a network If x is sorted Nby that a network sorts. N Let N transform (x, x 2,, x n ) into then also any monotonic function of x. (y, y 2,, y n ), then it also transforms (f(x - 4 ), f(x 99 2 ), 9, f(x - 2 n )) 4 into 9 99 (f(y ), f(y 2 ),, f(y n )). Assume y i > y i+ for some i, then consider the monotonic function Show: If x is not sorted by the network, then there is a monotonic function f that maps x to s and s and f(x) is not sorted by the network f(x) = ቊ, if x < y i, if x y i N converts x not sorted by N there is an f x, n not sorted by N (f(x ), f(x 2 ),, f(x n )) into f y, f(y 2, f y i, f y i+,, f(y n )) f sorted by N for all f, n x sorted by N for all x 38

39 Proof Assume a monotonic function f(x) with f x f(y) whenever x y and a network N that sorts. Let N transform (x, x 2,, x n ) into (y, y 2,, y n ), then it also transforms (f(x ), f(x 2 ),, f(x n )) into (f(y ), f(y 2 ),, f(y n )). Assume y i > y i+ for some i, then consider the monotonic function N converts f(x) = ቊ, if x < y i, if x y i All comparators must act in the same way for the f(x i ) as they do for the x i (f(x ), f(x 2 ),, f(x n )) into f y, f(y 2, f y i, f y i+,, f(y n )) 39

40 Bitonic Sort Bitonic (Merge) Sort is a parallel algorithm for sorting If enough processors are available, sort breaks the lower bound on sorting for comparison sort algorithm Asymptotic Runtime of O n log 2 n (sequential execution) Very good asymptotic runtime in the parallel case (as we'll see below). Worst = Average = Best case 4

41 Bitonic Sequence (x, x 2,, x n ) is, if it can be circularly shifted such that it is first monotonically increasing and then monontonically decreasing. (, 2, 3, 4, 5, 3,, ) (4, 3, 2,, 2, 4, 6, 5) 4

42 Bitonic - Sequences i j k i j k 42

43 The Half-Cleaner void halfclean(std::vector<t>& a, int lo, int n, boolean dir){ for (int i=lo; i<lo+n/2; i++) compare(a[i],a[i+n/2], dir); } clean 43

44 The Half-Cleaner void halfclean(std::vector<t>& a, int lo, int n, boolean dir){ for (int i=lo; i<lo+n/2; i++) compare(a[i],a[i+n/2], dir); } clean 44

45 Bitonic Split Example + 45

46 Lemma Input: a sequence of s and s, then for the output of the half-cleaner it holds that Upper and lower half is [One of the two halfs is clean (only s or s)] Every number in upper half every number in the lower half 46

47 Proof: All cases top bottom top bottom clean 47

48 top bottom top bottom clean 48

49 top bottom top bottom clean 49

50 top bottom top bottom clean 5

51 The four remaining cases ( ) top bottom top bottom clean top bottom top bottom clean top bottom top bottom clean top bottom top bottom clean 5

52 Construction BitonicToSorted halfclean halfclean halfclean half clean half clean half clean half clean sorted 52

53 Recursive Construction ToSorted(n) halfclean (n) ToSorted (n/2) ToSorted (n/2) 53

54 BitonicToSorted sorts a Bitonic Sequence void ToSorted (std::vector<int>& a, int lo, int n, boolean dir){ if (n>){ halfclean(a, lo, n, dir); } } ToSorted(a, lo, n/2, dir); ToSorted(a, lo+m, n-n/2, dir); halfclean (n) ToSorted (n/2) sorted ToSorted (n/2) 54

55 Bi-Merger sorted sorted sorted reverse sorted bimerge half-cleaner Bi-Merger on two sorted sequences acts like a half-cleaner on a sequence (when one of the sequences is reversed) 55

56 Bi-Merger sorted sorted void bimerge(std::vector<t>& a, int lo, int n, boolean dir){ for (int i=; i<n/2; i++) compare(a[lo+i],a[lo+n-i-], dir); } bimerge Bi-Merger on two sorted sequences acts like a half-cleaner on a sequence (when one of the sequences is reversed) 56

57 Merger Merger 57 bimerge (n) halfclean (n/2) halfclean (n/2) half clean half clean half clean half clean sorted sorted sorted

58 Merger sorted sorted bimerge (n) ToSorted (n/2) ToSorted (n/2) sorted Merger 58

59 BitonicMerge sorts a Halfsorted Sequence void Merge (std::vector<int>& a, int lo, int n, boolean dir){ if (n>){ int m=n/2; } } bimerge(a,lo,n,dir); ToSorted(a, lo, m, dir); sorted ToSorted(a, lo+m, n-m, dir); sorted bimerge ToSorted (n/2) ToSorted (n/2) sorted 59

60 Recursive Construction of a Sorter Sort(n) Sort (n/2) Sort (n/2) Merge (n) 6

61 private void Sort(std::vector&<T> a, int lo, int n, boolean dir){ if (n>){ int m=n/2; Sort(a, lo, m, ASCENDING); } } Sort(a, lo+m, n-m, DESCENDING); Merge(a, lo, n, dir); Sort (n/2) Sort (n/2) Merge (n) 6

62 Example half clean half clean bimerge half clean half clean halfclean half clean half clean half clean half clean bimerge half clean half clean bimerge halfclean half clean half clean Merge(2) Merge(4) Merge (8) 62

63 Merger (8) Example 63 Merger(4) Merger (2) bi-merger half cleaner half cleaner half cleane r half cleane r half cleane r half cleane r bimerger half cleane r half cleane r bimerger half cleane r half cleane r half cleane r half cleane r half cleane r half cleane r

64 Bitonic Merge Sort How many steps? #mergers log n i= log n log 2 i = i= i log 2 = log n (log n + ) 2 = O(log 2 n) #steps / merger 64

65 Zur Prüfung findet statt am von 9: : (2h) Inhalt: Datenstrukturen und Algorithmen, C++ Advanced, Parallel Programming, Hilfsmittel: 4 A4 Seiten, handgeschrieben oder min. Pt Fontsize. Kopieren ist erlaubt aber nicht clever. Handgeschriebenes vom Tablet drucken ist auch erlaubt, wenn dabei die Schrift nicht wesentlich verkleinert wird (in Relation zur sonst üblichen Schreibschrift). 65

66 Vorschlag: Q&A vor der Prüfung. Schulferien ab 6.Juli -- Termin sollte vorher sein. Für Sie: je später desto besser. Vorschlag: +/- 2. Juli. 66

67 Vorbereitung Übungen machen / gemacht haben. [morgen, Freitag. Juni, finden Übungen statt. Besprechung Übung 3.] Können Sie die Vorlesungsinhalte einem Kollegen (ohne Folien) erklären? Alte Prüfungen stehen auf der Webseite zur Verfügung. Die alten Prüfungen von Prof. Widmayer enthalten Material, das nicht behandelt wurde (geometrische Algorithmen, branch-and-bound) Die "neuen" Prüfungen bei mir enthalten Material, das in den Vorlesungen von Widmayer/Püschel nicht behandelt wurden (insbesondere C++ / Parallel Programming) 67

68 Ausschlüsse NICHT Prüfungsstoff sind Details der längeren Beweise (Laufzeit Algorithmus Blum, Analyse Ranomisierter Quicksort, Amortisierte Analyse von Move-To-Front, Beweis Theorem Universelles Hashing, Beweis Fibonacci Zahlen mit Erzeugendenfunktion, Beweis Amortisierte Kosten Fibonacci Heap, Atomare Register / RMW Operationen / Lock Free Programming Hardware Architekturen, Pipelining, Peterson Algorithmus 68

69 Ich bin weiterhin erreichbar unter 69