MapReduce Algorithm Design
Contents Combiner and in mapper combining Complex keys and values Secondary Sorting
Combiner and in mapper combining Purpose Carry out local aggregation before shuffle and sort phase Reduce the communication volume between map and reduce stages
Combiner example (word count) Use reducer as combiner Integer addition is both associative and commutative
MapReduce with Combiner
MapReduce with combiner MapReduce with combiner map: (k1, v1) > [(k2, v2)] combine: (k2, [v2]) > [(k2, v2)] reduce: (k2, [v2]) > [(k3, v3)] The combiner input and output key value types must match the mapper output key value type
Combiner is an optimization, not a requirement Combiner is optional A particular implementation of MapReduce framework may choose to execute the combine method many times or none Calling the combine method zero, one, or many times should produce the same output from the reducer The correctness of the MapReduce program should not rely on the assumption that the combiner is always carried out
In method combining
Use Java Map to implement AssociativeArray A Map is an object that maps keys to values A map cannot contain duplicate keys Each key can map to at most one value The Java platform contains three general purpose Map implementations HashMap, TreeMap, and LinkedHashMap Basic operations boolean containskey(object key) get(object key) put(k key, V value) keyset() Returns a Set view of the keys contained in this map
In mapper combining
Demo and Some online resource Demo on in method and in mapper combining Online resource https://vangjee.wordpress.com/2012/03/07/the in mapper combiningdesign pattern for mapreduce programming codingjunkie.net Blog by Bill Bejeck, who provides a lot of source code on MapReduce in Hadoop
Complex keys and values Both keys and values can be complex data structures Pairs Stripes Serialization and deserialization for complex structures After map stage, structures need to be serialized to be written to storage Complex keys and values need to be de serialized at the reducer side
Motivation An example to compute the mean of value
With combiner
Revised mapper
In mapper combining
A running example Build word co occurrence matrices for large corpora Build the word co occurrence matrix for all the works by Shakespeare Co occurrence within a specific context A sentence A paragraph A document A certain window of m words
Pairs For each map task For each unique pair (a, b), emit [pair (a, b), count] Use combiner or in mapper combining to reduce the volume of intermediate pairs Reducers sum up counts associated with these pairs
Pairs approach
Stripes Idea: group together pairs into an associative array (a, b) 1 (a, c) 2 (a, d) 5 (a, e) 3 (a, f) 2 a { b: 1, c: 2, d: 5, e: 3, f: 2 } Mappers emit [word, associate array] Reducers perform element wise sum of associative arrays + a { b: 1, d: 5, e: 3 } a { b: 1, c: 2, d: 2, f: 2 } a { b: 2, c: 2, d: 7, e: 3, f: 2 }
Stripes approach
Relative frequency What proportion of the time does B appear in the context A? Whenever there is a co occurrence of (A,*), for what percentage will * be B The total count of co occurrence of (A,*) is called marginal f ( B A) count( A, B) count( A,*) B' count( A, B) count( A, B') (A, *) 32 (A, B 1 ) 3 (A, B 2 ) 12 (A, B 3 ) 7 (A, B 4 ) 1 (A, B 5 ) 4 (A, B 6 ) 5 Reducer holds this value in memory (A, B 1 ) 3 / 32 (A, B 2 ) 12 / 32 (A, B 3 ) 7 / 32 (A, B 4 ) 1 / 32 (A, B 5 ) 4 / 32 (A, B 6 ) 5 / 32
Relative frequency with stripes It is easy One pass to compute (A, *) Another pass to directly compute f(b A) May have the scalability issue for really large data The final associative array holds all the neighbors and their co occurrence counts with the word A
Relative frequency with pairs Must emit extra (A, *) for every B (B can be any word) in mapper Must make sure all pairs of (A, *) and (A, B) get sent to same reducer (use partitioner) Must make sure (A, *) comes first (define sort order) Must hold state in reducer across different key value pairs This pattern is called order inversion
Secondary Sorting A motivating example The readings of m sensors are recorded over the time (t1, m1, r80521) (t1, m2, r14209) (t1, m3, r76042) (t2, m1, r21823) (t2, m2, r66508) (t2, m3, r98347) (m1, t1, r80521) (m1, t2, r21823) (m2, t1, r14209) (m2, t2, r66508) (m3, t1, r76042) (m3, t2, r98347)
First approach (t1, m1, r80521) (t1, m2, r14209) (t1, m3, r76042) (t2, m1, r21823) (t2, m2, r66508) (t2, m3, r98347) (m1, (t1, r80521)) (m2, (t1, r14209)) (m3, (t1, r76042)) (m1, (t2, r21823)) (m2, (t2, r66508)) (m3, (t2, r98347)) However, Hadoop MapReduce sorts intermediate pairs by key Values can be arbitrarily ordered E.g., (m1, [(t100, r23456), (t2, r21823),,(t234, r34870)]) Buffer values in memory, then sort Is there issue with this approach?
Second approach Value to key conversion Move part of the value into the intermediate key to form a composite key Composite key: (m, t) Intermediate pair: ((m, t), r) Let execution framework do the sorting First sort by the sensor id, i.e., m (the left element in the key) Then sort by the timestamp, i.e., t (the right element in the key) Implement the custom partitioner All pairs with the same sensor shuffled to the same reducer ((m1, t1), r80521) ((m1, t2), r21823) ((m1, t3), r149625)