Algorithm Design for MapReduce

Transcription

1 Algorithm Design for MapReduce CprE 419X, Spring 2014 Iowa State University Srikanta Tirthapura 2/4/14 CprE 419 X, Srikanta Tirthapura 1

2 Problem 0: Sum Find the sum and average of many input integers Write the map and reduce pseudocode What is the map cost, reduce cost, per reducer cost, and communication cost of your method? 2/4/14 CprE 419 X, Srikanta Tirthapura 2

3 Problem 1: Set Difference Input: Two sets A and B Output: Set difference (A B), i.e. all those elements that are present in A but not in B Ex: Find all IP addresses that appeared in one log but not the other What is the map cost, reduce cost, per reducer cost, and communication cost of your method? 2/4/14 CprE 419 X, Srikanta Tirthapura 3

4 Set Difference MR Algorithm map(key = set_name, value = elementid) emit (key = elementid, value = set_name) reduce (key = elementid, values) if (A in values) and (B not in values): emit(key = elementid, value = 1) 2/4/14 CprE 419 X, Srikanta Tirthapura 4

5 Observation Also works if A and B are multisets (i.e. the same value appears multiple times within A and B) 2/4/14 CprE 419 X, Srikanta Tirthapura 5

6 Analysis Is the set difference algorithm good? How much does it cost? Not an easy question to answer Consider: Per Mapper Cost, Total Map Cost Per Reducer Cost, Total Reduce Cost Total Bytes of Communication 2/4/14 CprE 419 X, Srikanta Tirthapura 6

7 Set Difference Let n = input size ( A + B ), M = number of mappers, R = number of reducers Total Map Cost = Theta(n) Per Mapper Cost = Theta(n/M) Total Reduce Cost = Theta(n) Per Reducer Cost = Theta(n/R) Total Communication Cost = Theta(n) 2/4/14 CprE 419 X, Srikanta Tirthapura 7

8 Problem 2: Matrix-Vector Input: n x n matrix M Multiplication M[i,j] provided as (i,j,m[i,j]) within a HDFS file rows numbered 1.. n, columns similar Element at row i and column j denoted M[i,j] n x 1 vector A A[j] provided as (j,a[j]) within a HDFS file Output: Vector B = M A B[i] = M[i,1] * A[1] + M[i,2] * A[2] +. 2/4/14 CprE 419 X, Srikanta Tirthapura 8

9 Matrix-Vector Using Mapreduce Suppose reduce key was i, i=1 n Reduce function for key i computes B[i] Needs values M[i,1]*A[1], M[i,2]*A[2], How do we compute these? Computing M[i,j] * A[j] Use j as the key in one MapReduce round 2/4/14 CprE 419 X, Srikanta Tirthapura 9

10 Matrix-Vector MR Round 1 map1(key, value = (i,j,m[i,j])) emit (key = j, value = (i,j,m[i,j])) map1(key, value = (j,a[j])) emit (key = j, value = A[j]) reduce1(key = j, values = [A[j], M[1,j], [2,j],..]) for (i = 1 to n): emit(i, M[i,j] * A[j]) 2/4/14 CprE 419 X, Srikanta Tirthapura 10

11 Matrix-Vector MR Round 2 map2(key = i, value) emit (key = i, value) reduce2(key = i, values) B[i] = 0 for (v in values): B[i] += v emit (key = i, value = B[i]) 2/4/14 CprE 419 X, Srikanta Tirthapura 11

12 Matrix-Vector Analysis First MR Round. Total Map cost = O(n 2 +n) = O(n 2 ) Per Mapper Cost = O(n 2 /M) Total Reduce Cost = O(n 2 ) Per Reducer Cost = O(n 2 /R) Communication = O(n 2 ) Second Round is Similar Q: One Round Algorithm for Matrix-Vector using MapReduce? 2/4/14 CprE 419 X, Srikanta Tirthapura 12

13 Problem 3: Length 2 Paths in a Graph Input: A graph G presented as a list of edges Output: All paths of length 2 in the graph Solution: Similar to the problem of enumerating triangles in a graph 2/4/14 CprE 419 X, Srikanta Tirthapura 13

14 Problem 4: Finding Pairs of Nearby Bit Strings Input: Set of bitstrings, each of length k Output: All pairs of bit strings such that the two strings in the pair differ at no more than two positions, i.e. at a Hamming distance of no more than 2. 2/4/14 CprE 419 X, Srikanta Tirthapura 14

15 Example: Nearby Bit Strings Input: 10010, 00010, 11000, 11110, Output: (10010, 00010), (10010, 11000) (10010, 11110), (10010, 00000) (00010, 00000), (11000, 11110) (11000, 00000) 2/4/14 CprE 419 X, Srikanta Tirthapura 15

16 Algorithm 1 Compare all pairs of strings and see if they are within a Hamming Distance of 2 Mapper: For each string b, Send a copy of b to each reducer Reducer i: Receives entire set S Computes a subset S i Examines all pairs in S i x S 2/4/14 CprE 419 X, Srikanta Tirthapura 16

17 Algorithm 1 Analysis Let n = total number of input strings. M = # of mappers, R = # of reducers Total Map Cost = O(nkR) Per Mapper Cost = O(nkR/M) Total Reducer Cost = O(n 2 k) Per Reducer Cost = O(n 2 k/r) Total Communication = O(nkR) 2/4/14 CprE 419 X, Srikanta Tirthapura 17

18 Algorithm 2 For each bitstring b, there are only (k choose 2) strings at Hamming distance 2 (k choose 1) strings at Hamming distance 1 Search all these possibilities 2/4/14 CprE 419 X, Srikanta Tirthapura 18

19 Algorithm 2 Map(key, value = bitstring b): emit (key = b, value = b) for each b formed by flipping 1 bit of b: emit (key = b, value = b) Reduce(key = bitstring b, values): for any two strings b1, b2 in values: emit(key = (b1, b2), value = 1) 2/4/14 CprE 419 X, Srikanta Tirthapura 19

20 Analysis of Algorithm 2 Let n = total number of input strings, M = # mappers, R = # reducers Total Map Cost = O(nk 2 ) Per Mapper Cost = O(nk 2 /M) Total Reduce Cost = O(kn + output size) Per Reducer Cost = O(kn/R + output size) Total Communication = O(nk 2 ) 2/4/14 CprE 419 X, Srikanta Tirthapura 20

21 Generalization List all pairs of bitstrings that are at a Hamming distance of t or lesser. 2/4/14 CprE 419 X, Srikanta Tirthapura 21

22 Problem 5: Graph Connectivity Input: A graph, presented as a list of edges Output: Yes if graph is connected, and No if it is not connected. 2/4/14 CprE 419 X, Srikanta Tirthapura 22

23 Readings Chapter 2 of Ullman-Rajaraman, Ver 1.2, MapReduce and the New Software Stack Chapter 2 of Hadoop: The Definitive Guide 2/4/14 CprE 419 X, Srikanta Tirthapura 23