MapReduce Optimizations and Algorithms 2015 Professor Sasu Tarkoma

Transcription

1 apreduce Optimizations and Algoithms 2015 Pofesso Sasu Takoma

2 Optimizations Reduce tasks cannot stat befoe the whole map phase is complete Thus single slow machine can slow down the whole pocess aste can execute many edundant map tasks and then use the esults of the fist task to complete

3 Optimizations: Combining Phase Pefomance can be inceased by unning a mini educe phase on local map output Executed on mappe nodes afte map phase Saves bandwidth befoe sending data to a full educe Reduce can be a combine if it is commutative and associative

4 Combine ap output On one mappe machine: Combine eplaces with: To educe To educe Souce: maped.pdf

5 Patitione Patitione divides the intemediate key space Assigns intemediate key-value pais to educes Thus n patitions esults in n educes Between map and educe phases: data is shuffled: paallel-soted and exchanged data is moved to the coect shad fo educing patition function accepts the key and the numbe of educes and then etuns the index of the educe Suppots load balancing

6 apreduce Summay Two key functions that need to be implemented: map (in_key, in_value) à (out_key, intemediate_value) list educe (out_key, intemediate_value list) à out_value list With two optimizations: combine (key, intemediate_value list) à intemediate_out_value list patition (key, numbe of patitions) à patition fo key

7 Patition and Shuffle!"##$%!"##$%!"##$%!"##$% &'()$%*$+'")$,- &'()$%*$+'")$,- &'()$%*$+'")$,- &'()$%*$+'")$,- 1"%)')'2($% 1"%)')'2($% 1"%)')'2($% 1"%)')'2($%,3/ 5'(! &'()$%*$+'")$,- &'()$%*$+'")$,- &'()$%*$+'")$,-.$+/0$%.$+/0$%.$+/0$%

8 Synchonization Intemediate key-value pais must be gouped by key though a distibuted sot Shuffle and sot A job with m mappes and educes involves up to m * diffeent copy opeations Each mappe may have intemediate output destined to evey educe

9 Data management as pat of apreduce Pulieus and CA Data management as pat of apreduce Need to know task types (map/educe intensive) ap intensive jobs benefit fom locality awaeness Up to 80% eduction in execution time

10 Key algoithms fo apreduce Inveted Index Statistics Soting Seaching K-eans Tansitive closue PageRank Advanced algoithms

11 Filteing Algoithms Finding files o items with specific chaacteistics Seaching fo pattens in web logs o files Filteing is mostly done in the map phase Reduce can be simply the identity

12 Seach appe key is the file name + line numbe appe value is the contents of the file Seach patten is a special paamete appe: Input: (filename, text) and patten If text matches patten output (filename, _) Reduce: Identity function Optimization: ak file only once Use a combine function to collapse (filename, _) pais into one Alleviates I/O issues

13 Aggegation Algoithms Computing the minimum, maximum, sum, aveage,... of the given values Count the numbe of Tweets pe day ap is simple o the identity, educe is doing most of the wok

14 Soting The appe output is automatically soted by keys Reduce plays a cucial ole in soting Teasot Benchmak (key, value) pais ae handled in ode by key and sent to a specific educe based on hash(key) Hash function must be chosen so that k1 < k2 => hash(k1) < hash(k2) If we have a single educe, the output is soted If we have multiple educes, we get patially soted esults: last-stage mege of the inteim esults

15 TeaSot TeaSot is a standad map/educe sot A custom patitione that uses a soted list of N 1 sampled keys that define the key ange fo each educe. In paticula, all keys such that sample[i 1] <= key < sample[i] ae sent to educe i. This guaantees that the output of educe i ae all less than the output of educe i+1. To speed up the patitioning, the patitione builds a two level tie that quickly indexes into the list of sample keys based on the fist two bytes of the key. TeaSot geneates the sample keys by sampling the input befoe the job is submitted and witing the list of keys into HDFS.

16 Iteative apreduce Algoithms

17 K-eans Clusteing Algoithm Iteative algoithm that is un until it conveges 1. K initial points (centes) ae chosen at andom. 2. K clustes ae fomed by associating evey data point (obsevation) with the neaest cente. 3. Fo each cluste, ecompute the centes (detemine centoid) 4. Repeat fom 2 until convegence. Souce: Riccado Tolone. Analytics on Big Data. Univesita Roma Te.

18 K-eans fo apreduce ap phase Each map eads the K centoids and a block fom the input dataset Each point is assigned to the closest centoid Output: <centoid, point> Reduce phase Obtain all points fo a given centoid Recompute the new centoid Output: <new centoid> Iteation: Compae the old and new set of K centoids If they ae simila then Stop Else Stat anothe iteation unless maximum of iteations has been eached.

19 apreduce K-eans k i = k centoids at iteation i P0 k i P1 k i k i+1 P2 k i k i - k i+1 < theshold? i=i+1 Client done Limitation: eads the whole point set P at each iteation Souce: HaLoop pesentation, Yyingyi Bu et al. VLDB 2010

20 Optimizing K-eans fo apreduce Combines can be used to optimize the distibuted algoithm Compute fo each centoid the local sums of the points Send to the educe: <centoid, patial sums> Use of a single educe Data to educes is vey small Single educe can tell immediately if the computation has conveged Ceation of a single output file

21 Tansitive Closue in apreduce Join Duplication elimination S i Fiend0 Fiend1 Anything new? i=i+1 Client done Souce: HaLoop pesentation, Yyingyi Bu et al. VLDB 2010

22 PageRank Algoithm Link analysis algoithm that assigns weights to each vetex in a gaph by iteatively computing the weight of each vetex based on the weight of its inbound neighbous. In elational algeba, PageRank can be expessed as: a join followed by an update with two aggegations that ae epeated until stopping condition The fist apreduce job joins the ank and linkage tables; appes emit the join column as the key and Reduces compute the join fo each unqiue souce URL and an contibution of each outbound edge. The second apreduce job computes the aggegate ank of each unique destination URL. The ap is the identity and the educes sum the ank contibutions of each incoming edge.

23 PageRank Algoithm Join & compute ank Aggegate fixpoint evaluation R i L-split0 L-split1 i=i+1 Conveged? Client Souce: HaLoop pesentation, Yyingyi Bu et al. VLDB 2010 done R = Rank table L = Linkage table

24 Limitation: iteative algoithms apreduce tasks must be witten as acyclic dataflow pogams Stateless mappes and educes Batch model Difficult to implement iteative pocessing of datasets achine leaning typically equies iteative opeation: the dataset is visited multiple times by the algoithm

25 HaLoop fo iteative apreduce apreduce cannot expess iteation o ecusion HaLooP modifies Hadoop fo suppoting fixpoint opeations, loop-awae task scheduling, and cache management ap Reduce Fixpoint model fo ecusive languages Fo example: the vecto of PageRank values of web pages is the fixed point of a linea tansfomation deived fom the link stuctue Souce: HaLoop pesentation, Yyingyi Bu et al. VLDB 2010

26 HaLoop: Inte-iteation caching Loop body appe output cache (O) Reduce output cache (RO) fo access to output of pevious iteations, fo fixpoint evaluation Reduce input cache (RI) fo loop invaiant data without map/shuffle appe input cache (I) fo access to nonlocal mappe input on late iteations Souce: HaLoop pesentation, Yyingyi Bu et al. VLDB 2010 Lagest gain by caching loop invaiant data