Databases 2 (VU) ( )

Databases 2 (VU) (707.030) Map-Reduce Denis Helic KMI, TU Graz Nov 4, 2013 Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 1 / 90

Outline 1 Motivation 2 Large Scale Computation 3 Map-Reduce 4 Environment 5 Map-Reduce Skew 6 Map-Reduce Examples Slides Slides are partially based on Mining Massive Datasets course from Stanford University by Jure Leskovec Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 2 / 90

Map-Reduce Motivation Today s data is huge Challenges How to distribute computation? Distributed/parallel programming is hard Map-reduce addresses both of these points Google s computational/data manipulation model Elegant way to work with huge data Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 3 / 90

Motivation Single node architecture CPU Memory Data fits in memory Machine learning, statistics Classical data mining Memory Disk Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 4 / 90

Motivation Motivation: Google example 20+ billion Web pages Approx. 20 KB per page Approx. 400+ TB for the whole Web Approx. 1000 hard drives to store the Web A single computer reads 30 35 MB/s from disk Approx. 4 months to read the Web with a single computer Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 5 / 90

Motivation Motivation: Google example Takes even more time to do something with the data E.g. to calculate the PageRank If m is the number of the links on the Web Average degree on the Web is approx. 10, thus m 2 10 11 To calculate PageRank we need per iteration step m multiplications We need approx. 100+ iteration steps Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 6 / 90

Motivation Motivation: Google example Today a standard architecture for such problems is emerging Cluster of commodity Linux nodes Commodity network (ethernet) to connect them 2 10 Gbps between racks 1 Gbps within racks Each rack contains 16 64 nodes Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 7 / 90

Cluster architecture Motivation Switch Switch Switch CPU CPU CPU CPU Memory Memory Memory......... Memory Memory Disk Memory Disk Memory Disk Memory Disk Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 8 / 90

Motivation Motivation: Google example 2011 estimation: Google had approx. 1 million machines http://www.datacenterknowledge.com/archives/2011/08/01/ report-google-uses-about-900000-servers/ Other examples: Facebook, Twitter, Amazon, etc. But also smaller examples: e.g. Wikipedia Single source shortest path: m + n time complexity, approx. 260 10 6 Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 9 / 90

Large Scale Computation Large scale computation Large scale computation for data mining on commodity hardware Challenges How to distribute computation? How can we make it easy to write distributed programs? How to cope with machine failures? Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 10 / 90

Large Scale Computation Large scale computation: machine failures One server may stay up 3 years (1000 days) The failure rate per day: p = 10 3 How many failures per day if we have n machines? Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 11 / 90

Large Scale Computation Large scale computation: machine failures One server may stay up 3 years (1000 days) The failure rate per day: p = 10 3 How many failures per day if we have n machines? Binomial r.v. Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 11 / 90

Large Scale Computation Large scale computation: machine failures PMF of a Binomial r.v. p(k) = ( ) n (1 p) n k p k k Expectation of a Binomial r.v. E[X ] = np Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 12 / 90

Large Scale Computation Large scale computation: machine failures n = 1000, E[X ] = 1 If we have 1000 machines we lose one per day n = 1000000, E[X ] = 1000 If we have 1 million machines (Google) we lose 1 thousand per day Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 13 / 90

Large Scale Computation Large scale computation: data copying Copying data over network takes time Bring data closer to computation I.e. process data locally at each node Replicate data to increase reliability Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 14 / 90

Solution: Map-reduce Map-Reduce Storage infrastructure Distributed file system Google: GFS Hadoop: HDFS Programming model Map-reduce Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 15 / 90

Storage infrastructure Map-Reduce Problem: if node fails how to store a file persistently Distributed file system Provides global file namespace Typical usage pattern Huge files: several 100s GB to 1 TB Data is rarely updated in place Reads and appends are common Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 16 / 90

Distributed file system Map-Reduce Chunk servers File is split into contiguous chunks Typically each chunk is 16 64 MB Each chunk is replicated (usually 2x or 3x) Try to keep replicas in different racks Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 17 / 90

Distributed file system Map-Reduce Master node Stores metadata about where files are stored Might be replicated Client library for file access Talks to master node to find chunk servers Connects directly to chunk servers to access data Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 18 / 90

Distributed file system Map-Reduce Reliable distributed file system Data kept in chunks spread across machines Each chunk replicated on different machines Reliable distributed file system Seamless recovery from node failures Bring computation directly to data Chunk servers also used as computation nodes Seamless recovery from disk or machine failure C 0 C 1 C 5 C 2 D 0 C 5 C 1 C 3 C 2 D 0 C 5 D 1 C 0 C 5 D 0 C 2 Chunk server 1 Chunk server 2 Chunk server 3 Chunk server N Bring computation directly to the data! Figure: Figure from slides by Jure Leskovec Chunk servers also serve as compute servers 1/8/2013 Denis Helic (KMI, TU Graz) Jure Leskovec, Stanford CS246: Mining Map-Reduce Massive Datasets, http://cs246.stanford.edu Nov 4, 2013 19 / 9033

Map-Reduce Programming model: Map-reduce Running example We want to count the number of occurrences for each word in a collection of documents. In this example, the input file is a repository of documents, and each document is an element. Example Example is meanwhile a standard Map-reduce example. Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 20 / 90

Map-Reduce Programming model: Map-reduce words input_file sort uniq -c Three step process 1 Split file into words, each word on a separate line 2 Group and sort all words 3 Count the occurrences Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 21 / 90

Map-Reduce Programming model: Map-reduce This captures the essence of Map-reduce Split Group Count Naturally parallelizable E.g. split and count Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 22 / 90

Map-Reduce Programming model: Map-reduce Sequentially read a lot of data Map: extract something that you care about (key, value) Group by key: sort and shuffle Reduce: Aggregate, summarize, filter or transform Write the result Outline Outline is always the same: Map and Reduce change to fit the problem Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 23 / 90

tasks, so all key-value Map-Reduce pairs with the same key wind up at the same Reduce task. Programming model: Map-reduce 3. The Reduce tasks work on one key at a time, and combine all the values associated with that key in some way. The manner of combination of values is determined by the code written by the user for the Reduce function. Figure 2.2 suggests this computation. Input chunks Key value pairs (k,v) Keys with all their values (k, [v, w,...]) Combined output Map tasks Group by keys Reduce tasks Figure 2.2: Schematic of a map-reduce computation Figure: Figure from the book: Mining massive datasets 2.2.1 The Map Tasks We view input files for a Map task as consisting of elements, which can be any type: a tuple or a document, for example. A chunk is a collection of elements, and no element is stored across two chunks. Technically, all inputs Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 24 / 90

Map-Reduce Programming model: Map-reduce Input: a set of (key, value) pairs (e.g. key is the filename, value is a single line in the file) Map(k, v) (k, v ) Takes a (k, v) pair and outputs a set of (k, v ) pairs There is one Map call for each (k, v) pair Reduce(k, (v ) ) (k, v ) All values v with same key k are reduced together and processed in v order There is one Reduce call for each unique k Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 25 / 90

Map-Reduce Programming model: Map-reduce Big Document Map Group by key Reduce Star Wars is an American epic space opera franchise centered on a film series created by George Lucas. The film series has spawned an extensive media franchise called the Expanded Universe including books, television series, computer and video games, and comic books. These supplements to the two film trilogies... (Star, 1) (Wars, 1) (is, 1) (an, 1) (American, 1) (epic, 1) (space, 1) (opera, 1) (franchise, 1) (centered, 1) (on, 1) (a, 1) (film, 1) (series, 1) (created, 1) (by, 1)...... (Star, 1) (Star, 1) (Wars, 1) (Wars, 1) (a, 1) (a, 1) (a, 1) (a, 1) (a, 1) (a, 1) (film, 1) (film, 1) (film, 1) (franchise, 1) (series, 1) (series, 1)...... (Star, 2) (Wars, 2) (a, 6) (film, 3) (franchise, 1) (series, 2)...... Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 26 / 90

Map-Reduce Programming model: Map-reduce map(key, value): // key: document name // value: a single line from a document foreach word w in value: emit(w, 1) Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 27 / 90

Map-Reduce Programming model: Map-reduce reduce(key, values): // key: a word // values: an iterator over counts result = 0 foreach count c in values: result += c emit(key, result) Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 28 / 90

Map-reduce computation Environment Map-reduce environment takes care of: 1 Partitioning the input data 2 Scheduling the program s execution across a set of machines 3 Performing the group by key step 4 Handling machine failures 5 Managing required inter-machine communication Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 29 / 90

Map-reduce computation Environment MAP: Read input and produces a set of key-value pairs Big document Group by key: Collect all pairs with same key (Hash merge, Shuffle, Sort, Partition) Reduce: Collect all values belonging to the key and output 1/8/2013 Jure Leskovec, Stanford CS246: Mining Massive Datasets, http://cs246.stanford.edu 43 Figure: Figure from the course by Jure Leskovec (Stanford University) Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 30 / 90

Map-reduce computation Environment All phases are distributed with many tasks doing the work Figure: Figure from the course by Jure Leskovec (Stanford University) 1/8/2013 Jure Leskovec, Stanford CS246: Mining Massive Datasets, http://cs246.stanford.edu 44 Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 31 / 90

Environment Data flow Input and final output are stored in distributed file system Scheduler tries to schedule map tasks close to physical storage location of input data Intermediate results are stored on local file systems of Map and Reduce workers Output is often input to another Map-reduce computation Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 32 / 90

Environment Coordination: Master Master node takes care of coordination Task status, e.g. idle, in-progress, completed Idle tasks get scheduled as workers become available When a Map task completes, it notifies the master about the size and location of its intermediate files Master pushes this info to reducers Master pings workers periodically to detect failures Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 33 / 90

Environment Map-reduce execution details 2.2. MAP-REDUCE 27 User Program fork fork Master fork assign Map assign Reduce Worker Worker Worker Worker Input Data Worker Intermediate Files Output File Figure 2.3: Overview of the execution of a map-reduce program Figure: Figure from the book: Mining massive datasets executing at a particular Worker, or completed). A Worker process reports to the Master when it finishes a task, and a new task is scheduled by the Master for that Worker process. Each Map task is assigned one or more chunks of the input file(s) and executes on it the code written by the user. The Map task creates a file for Denis Helic (KMI, TU each Graz) Reduce task on the local disk Map-Reduce of the Worker that executes the Map task. Nov 4, 2013 34 / 90

Maximizing parallelism Map-Reduce Skew If we want maximum parallelism then Use one Reduce task for each reducer (i.e. a single key and its associated value list) Execute each Reduce task at a different compute node The plan is typically not the best one Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 35 / 90

Map-Reduce Skew Maximizing parallelism There is overhead associated with each task we create We might want to keep the number of Reduce tasks lower than the number of different keys We do not want to create a task for a key with a short list There are often far more keys than there are compute nodes E.g. count words from Wikipedia or from the Web Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 36 / 90

Map-Reduce Skew Input data skew: Exercise Exercise Suppose we execute the word-count map-reduce program on a large repository such as a copy of the Web. We shall use 100 Map tasks and some number of Reduce tasks. 1 Do you expect there to be significant skew in the times taken by the various reducers to process their value list? Why or why not? 2 If we combine the reducers into a small number of Reduce tasks, say 10 tasks, at random, do you expect the skew to be significant? What if we instead combine the reducers into 10,000 Reduce tasks? Example Example is based on the example 2.2.1 from Mining Massive Datasets. Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 37 / 90

Maximizing parallelism Map-Reduce Skew There is often significant variation in the lengths of value list for different keys Different reducers take different amounts of time to finish If we make each reducer a separate Reduce task then the task execution times will exhibit significant variance Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 38 / 90

Map-Reduce Skew Input data skew Input data skew describes an uneven distribution of the number of values per key Examples include power-law graphs, e.g. the Web or Wikipedia Other data with Zipfian distribution E.g. the number of word occurrences Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 39 / 90

Map-Reduce Skew Power-law (Zipf) random variable PMF p(k) = k α ζ(α) k N, k 1, α > 1 ζ(α) is the Riemann zeta function ζ(α) = k=1 k α Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 40 / 90

Map-Reduce Skew Power-law (Zipf) random variable Probability mass function of a Zipf random variable; differing α values 0.9 α =2.0 0.8 α =3.0 0.7 0.6 probability of k 0.5 0.4 0.3 0.2 0.1 0.0 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 k Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 41 / 90

Map-Reduce Skew Power-law (Zipf) input data Key size: sum =189681,µ =1.897,σ 2 =58.853 10 5 10 4 10 3 10 2 10 1 10 0 0 2 4 6 8 10 12 Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 42 / 90

Map-Reduce Skew Tackling input data skew We need to distribute a skewed (power-law) input data into a number of reducers/reduce tasks/compute nodes The distribution of the key lengths inside of reducers/reduce tasks/compute nodes should be approximately normal The variance of these distributions should be smaller than the original variance If variance is small an efficient load balancing is possible Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 43 / 90

Map-Reduce Skew Tackling input data skew Each Reduce task receives a number of keys The total number of values to process is the sum of the number of values over all keys The average number of values that a Reduce task processes is the average of the number of values over all keys Equivalently, each compute node receives a number of Reduce tasks The sum and average for a compute node is the sum and average over all Reduce tasks for that node Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 44 / 90

Map-Reduce Skew Tackling input data skew How should we distribute keys to Reduce tasks? Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 45 / 90

Map-Reduce Skew Tackling input data skew How should we distribute keys to Reduce tasks? Uniformly at random Other possibilities? Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 45 / 90

Map-Reduce Skew Tackling input data skew How should we distribute keys to Reduce tasks? Uniformly at random Other possibilities? Calculate the capacity of a single Reduce task Add keys until capacity is reached, etc. Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 45 / 90

Map-Reduce Skew Tackling input data skew We are averaging over a skewed distribution Are there laws that describe how the averages of sufficiently large samples drawn from a probability distribution behaves? In other words, how are the averages of samples of a r.v. distributed? Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 46 / 90

Map-Reduce Skew Tackling input data skew We are averaging over a skewed distribution Are there laws that describe how the averages of sufficiently large samples drawn from a probability distribution behaves? In other words, how are the averages of samples of a r.v. distributed? Central-limit Theorem Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 46 / 90

Map-Reduce Skew Central-Limit Theorem The central-limit theorem describes the distribution of the arithmetic mean of sufficiently large samples of independent and identically distributed random variables The means are normally distributed The mean of the new distribution equals the mean of the original distribution The variance of the new distribution equals σ2 n, where σ2 is the variance of the original distribution Thus, we keep the mean and reduce the variance Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 47 / 90

Central-Limit Theorem Map-Reduce Skew Theorem Suppose X 1,..., X n are independent and identical r.v. with the expectation µ and variance σ 2. Let Y be a r.v. defined as: Y n = 1 n n i=1 X i The CDF F n (y) tends to PDF of a normal r.v. with the mean µ and variance σ 2 for n : lim F 1 y n(y) = e (x µ)2 2σ 2 n 2πσ 2 Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 48 / 90

Map-Reduce Skew Central-Limit Theorem Example Practically, it is possible to replace F n (y) with a normal distribution for n > 30 We should always average over at least 30 values Approximating uniform r.v. with a normal r.v. by sampling and averaging Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 49 / 90

Central-Limit Theorem Map-Reduce Skew 18 Averages: µ =0.5,σ 2 =0.08333 16 14 12 10 8 6 4 2 0 0.0 0.2 0.4 0.6 0.8 1.0 Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 50 / 90

Central-Limit Theorem Map-Reduce Skew 40 Averages: µ =0.499,σ 2 =0.00270 35 30 25 20 15 10 5 0 0.30 0.35 0.40 0.45 0.50 0.55 0.60 0.65 0.70 Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 51 / 90

Map-Reduce Skew Central-Limit Theorem IPython Notebook examples http: //kti.tugraz.at/staff/denis/courses/kddm1/clt.ipynb Command Line ipython notebook pylab=inline clt.ipynb Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 52 / 90

Map-Reduce Skew Input data skew We can reduce impact of the skew by using fewer Reduce tasks than there are reducers If keys are sent randomly to Reduce tasks we average over value list lengths Thus, we average over the total time for each Reduce task (Central-limit Theorem) We should make sure that the sample size is large enough (n > 30) Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 53 / 90

Map-Reduce Skew Input data skew We can further reduce the skew by using more Reduce tasks than there are compute nodes Long Reduce tasks might occupy a compute node fully Several shorter Reduce tasks are executed sequentially at a single compute node Thus, we average over the total time for each compute node (Central-limit Theorem) We should make sure that the sample size is large enough (n > 30) Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 54 / 90

Input data skew Map-Reduce Skew Key size: sum =196524,µ =1.965,σ 2 =243.245 10 5 10 4 10 3 10 2 10 1 10 0 0 2 4 6 8 10 12 Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 55 / 90

Input data skew Map-Reduce Skew 4500 Task key size: sum =196524,µ =196.524,σ 2 =25136.428 4000 3500 3000 2500 2000 1500 1000 500 0 0 200 400 600 800 1000 Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 56 / 90

Input data skew Map-Reduce Skew Task key averages: µ =1.958,σ 2 =1.886 10 2 10 1 10 0 0 1 2 3 4 5 6 Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 57 / 90

Input data skew Map-Reduce Skew 25000 Node key size: sum =196524,µ =19652.400,σ 2 =242116.267 Node key averages: µ =1.976,σ 2 =0.030 20000 15000 10000 5000 0 0 2 4 6 8 10 Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 58 / 90

Map-Reduce Skew Input data skew IPython Notebook examples http: //kti.tugraz.at/staff/denis/courses/kddm1/mrskew.ipynb Command Line ipython notebook pylab=inline mrskew.ipynb Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 59 / 90

Map-Reduce Skew Combiners Sometimes a Reduce function is associative and commutative Commutative: x y = y x Associative: (x y) z = x (y z) The values can be combined in any order, with the same result The addition in reducer of the word count example is such an operation Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 60 / 90

Map-Reduce Skew Combiners When the Reduce function is associative and commutative we can push some of the reducers work to the Map tasks E.g. instead of emitting (w, 1), (w, 1),... We can apply the Reduce function within the Map task In that way the output of the Map task is combined before grouping and sorting Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 61 / 90

Combiners Map-Reduce Skew Map Combiner Group by key Reduce (Star, 1) (Wars, 1) (is, 1)... (American, 1) (epic, 1) (space, 1)... (franchise, 1) (centered, 1) (on, 1)... (film, 1) (series, 1) (created, 1)......... (Star, 2)... (Wars, 2) (a, 6)... (a,3)... (film, 3) (franchise, 1) (series, 2)......... (Star, 2) (Wars, 2) (a, 6) (a, 3) (a, 4)... (film, 3) (franchise, 1) (series, 2)......... (Star, 2) (Wars, 2) (a, 13) (film, 3) (franchise, 1) (series, 2)...... Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 62 / 90

Map-Reduce Skew Input data skew: Exercise Exercise Suppose we execute the word-count map-reduce program on a large repository such as a copy of the Web. We shall use 100 Map tasks and some number of Reduce tasks. 1 Do you expect there to be significant skew in the times taken by the various reducers to process their value list? Why or why not? 2 If we combine the reducers into a small number of Reduce tasks, say 10 tasks, at random, do you expect the skew to be significant? What if we instead combine the reducers into 10,000 Reduce tasks? 3 Suppose we do use a combiner at the 100 Map tasks. Do you expect skew to be significant? Why or why not? Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 63 / 90

Input data skew Map-Reduce Skew Key size: sum =195279,µ =1.953,σ 2 =83.105 10 5 10 4 10 3 10 2 10 1 10 0 0 2 4 6 8 10 12 Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 64 / 90

Input data skew Map-Reduce Skew Task key averages: µ =1.793,σ 2 =10.986 10 5 10 4 10 3 10 2 10 1 10 0 0 1 2 3 4 5 6 7 Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 65 / 90

Map-Reduce Skew Input data skew IPython Notebook examples http://kti.tugraz.at/staff/denis/courses/kddm1/ combiner.ipynb Command Line ipython notebook pylab=inline combiner.ipynb Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 66 / 90

Map-Reduce Examples Map-Reduce: Applications Map-reduce computation make sense when files are large and rarely updated in place E.g. we will not see map-reduce computation when managing online sales We will not see map-reduce for handling Web request (even if we have millions of users) However, you want use map-reduce for analytic queries on the data generated by an e.g. Web application E.g. find users with similar bying patterns Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 67 / 90

Map-Reduce Examples Map-Reduce: Applications Computations such as analytic queries typically involve matrix operations The original purpose for the map-reduce implementation was to execute large matrix-vector multiplications to calculate the PageRank Matrix operations such as matrix-matrix and matrix-vector multiplications fit nicely into map-reduce programming model Another important class of operations that can use map-reduce effectively are the relational-algebra operations Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 68 / 90

Map-Reduce: Exercise Map-Reduce Examples Exercise Suppose we have an n n matrix M, whose element in row i and column j will be denoted m ij. Suppose we also have a vector v of length n, whose jth element is v j. Then the matrix-vector product is the vector x of length n, whose ith element is given by x i = n m ij v j j=1 Outline a Map-Reduce program that calculates the vector x. Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 69 / 90

Map-Reduce Examples Matrix-vector multiplication Let us first assume that the vector v is large but it still can fit into the memory The matrix M and the vector v will be each stored in a file of the DFS We assume that the row-column coordinates of a matrix element (indices) can be discovered E.g. each value is stored as a triple (i, j, m ij ) Similarly, the position of v j can be discovered in the analogous way Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 70 / 90

Map-Reduce Examples Matrix-vector multiplication The Map Function: The map function applies to one element of the matrix M The vector v is first read in its entirety and is subsequentelly available for all Map tasks at that computaion node From each matrix element m ij the map function produces the key-value pair (i, m ij v j ) All terms of the sum that make up the component x i of the matrix-vector product will get the same key i The Reduce Function: The reduce function sums all the values associated with a given key i The result is a pair (i, x i ) Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 71 / 90

Map-Reduce Examples Matrix-vector multiplication However, it is possible that the vector v can not fit into main memory It is not required that the vector v fits into the memory at a compute node, but if it does not there will be a very large number of disk accesses As an alternative we can divide the matrix M into vertical stripes of equal width and divide the vector into an equal number of horizontal stripes of the same height The goal is to use enough stripes so that the portion of the vector in one stripe can fit into main memoray at a compute node Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 72 / 90

Map-Reduce Examples disk accesses as we move pieces of the vector into main memory to multiply Matrix-vector components by multiplication elements of the matrix. Thus, as an alternative, we can divide the matrix into vertical stripes of equal width and divide the vector into an equal number of horizontal stripes, of the same height. Our goal is to use enough stripes so that the portion of the vector in one stripe can fit conveniently into main memory at a compute node. Figure 2.4 suggests what the partition looks like if the matrix and vector are each divided into five stripes. Matrix M Vector v Figure 2.4: Division of a matrix and vector into five stripes The ith stripe of the matrix multiplies only components from the ith stripe of the vector. Thus, we can divide the matrix into one file for each stripe, and do the same for the vector. Each Map task is assigned a chunk from one of the stripes of the matrix and gets the entire corresponding stripe of the vector. Denis Helic The(KMI, MapTU and Graz) Reduce tasks can then Map-Reduce act exactly as was described abovenov for4, the 2013 73 / 90

Map-Reduce Examples Matrix-vector multiplication The ith stripe of the matrix multiplies only components from the ith stripe of the vector We can divide matrix into one file for each stripe, and do the same for the vector Each Map task is assigned a chunk from one of the stripes in the matrix and it gets the entire corresponding stripe of the vector The Map and Reduce tasks can then act exactly as before We need to sum up once more the results of the stripes multiplication Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 74 / 90

Map-Reduce Examples Relational-Algebra operations There are many operation on data that can be described easily in terms of the common database-query primitives The queries themselves must not be executed within a DBMS E.g. standard operations on relations A relation is a table with column headers called attributes The set of attributes of a relation is called its schema: R(A 1, A 2,..., A n ) Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 75 / 90

Example of a relation Map-Reduce Examples From To url1 url2 url1 url3 url2 url3 url2 url4...... Table: Relation Links Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 76 / 90

Map-Reduce Examples Example of a relation A tuple is a pair or URLs such that there is at least one link from the first URL to the second The first row (url1, url2) states that the Web page at url1 points to the WEb page at url2 You would typically have a similar relation stored by a search engine With billions of tuples Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 77 / 90

Relational-Algebra Map-Reduce Examples Standard operation on relations are 1 Selection: apply a condiotion C to each tuple and output only tuples that satisfy C 2 Projection: Produce from each tuple only a subset S of attributes 3 Union, Intersection, Difference: set operations on tuples 4 Natural Join: Given two relations compare each pair of tuples and output those that agree on all common attributes 5 Grouping and Aggregation: Partition the tuples in a relation according to their values in a set of attributes. For each group perform one of the operations such as SUM, COUNT, AVG, MIN or MAX Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 78 / 90

Map-Reduce Examples Example of relational algebra Find the paths of length two in the Web using the Links relation In other words find triples of URLs (u, v, w) such that there is a link between u and v and a link between v and w We want to take natural join of Links with itself Let us describe this with two copies of Links: L1(U1, U2) and L2(U2, U3) Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 79 / 90

Map-Reduce Examples Example of relational algebra Now we compute L1 L2 For eaxh tuple t1 of L1 and each tuple t2 of L2 we see if their U2 components are same If these two components agree we produce (U1, U2, U3) as a result If we want only to check for the existence of the path of length two we might want to project onto U1 and U3 π U1,U3 (L1 L2) Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 80 / 90

Map-Reduce Examples Example of relational algebra Imagine that a social-networking site has a relation Friends(User, Friend) Suppose we want to calculate the statistics about the number of friends of each user In terms of relational algebra we would perform groupng and aggregation: γ User,COUNT (Friend) (Friends) This operation groups all tuples by the value of the first component and then counts the number of friends Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 81 / 90

Map-Reduce Examples Selection by Map-Reduce Selection Given is a relation R. We want to compute σ C (R) The Map Function: For each tuple t in R test if it satisfies C If so produce the key value pair (t, t) The Reduce Function: The reduce function is identity It simply passes each key-value pair to the output Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 82 / 90

Map-Reduce Examples Projection by Map-Reduce Selection Given is a relation R. We want to compute π S (R) The Map Function: For each tuple t in R construct a tuple t by eliminating from t those components that are not in projection S Output the key-value pair (t, t ) The Reduce Function: For each key t there will be one or more key-value pairs (t, t ) The reduce function turns (t, [t, t,..., t ]) into (t, t ) Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 83 / 90

Map-Reduce Examples Natural-Join by Map-Reduce Selection Given are relations R(A, B) and S(B, C). We want to compute R S The Map Function: For each tuple (a, b) of R produce the key-value pair (b, (R, a)) For each tuple (b, c) of S produce the key-value pair (b, (S, c)) The Reduce Function: Each key b will be associated with a list of pairs that are either of the form (R, a) or (S, c) Construct all pairs consisting of the values (a, b, c) Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 84 / 90

Map-Reduce Examples Grouping and Aggregation by Map-Reduce Selection Given is a relation R(A, B, C). We want to compute γ A,θ(B) (R) The Map Function: Map produces the grouping For each tuple (a, b, c) produce the key-value pair (a, b) The Reduce Function: The reduce function produces the aggregation Each key a represents a group Apply the aggregation operator θ to the list [b 1, b 2,..., b n ] of the values associated with a The outpur is a pair (a, x), where x is the result of θ applied to the list Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 85 / 90

Map-Reduce Examples Matrix-matrix multiplication Exercise Suppose we have an n n matrix M, whose element in row i and column j will be denoted m ij. Suppose we also have another n n matrix N with elements n ij. Then the matrix-matrix product is the matrix P of dimensions n n, whose elements are given by P ijk = n m ij n jk j=1 Outline a Map-Reduce program that calculates the matrix P. Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 86 / 90

Map-Reduce Examples Matrix-matrix multiplication We can think of a matrix as a relation with three attributes: the row number, the column number and the value in that row and column Thus, we can represent M as a relation M(I, J, V ) with tuples (i, j, m ij ) Similarly, we can represent N as a relation N(J, K, W ) with tuples (j, k, n jk ) The matrix-matrix product is almost a natural join followed by grouping and aggregation Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 87 / 90

Map-Reduce Examples Matrix-matrix multiplication The common attribute of two relations is J The natural join of M(I, J, V ) and N(J, K, W ) would produce a tuple (i, j, k, v, w) This five component relation is a pair of matrix elements (m ij, n jk ) We need (i, j, k, v w) Once when we have this relation we can perform grouping and aggregation Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 88 / 90

Map-Reduce Examples Matrix-matrix multiplication The Map Function: For each matrix element m ij produce the key-value pair (j, (M, i, m ij )) Likewise for each matrix element n jk produce the key-value pair (j, (N, k, n jk )) M and N are names indicating from which matrix the element originates The Reduce Function: For each key j examine the list of associated values For each value (M, i, m ij )) that comes from M and for each value (N, k, n jk )) that comes from N produce a key-value pair with key equal to (i, k) and value equal to m ij n jk Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 89 / 90

Map-Reduce Examples Matrix-matrix multiplication The Map Function: The Map function is identity The Reduce Function: For each key (i, k) produce the sum of the list of values associated with this key The result is a pair ((i, k), p ik ) Denis Helic (KMI, TU Graz) Map-Reduce Nov 4, 2013 90 / 90