Introduction to MapReduce

Transcription

1 Introduction to MapReduce Based on slides by Juliana Freire Some slides borrowed from Jimmy Lin, Jeff Ullman, Jerome Simeon, and Jure Leskovec

2 Required Reading! Data-Intensive Text Processing with MapReduce, by Jimmy Lin and Chris Dyer Chapters 1 and 2 Mining of Massive Datasets (2nd Edition), by Anand Rajaraman, Jure Leskovec and Jeff Ullman Chapter and 2.2 (Large-Scale File Systems and MapReduce)

3 Single-Node Architecture CPU Memory Data Analysis, Machine Learning, Statistics Disk

4 How Much Data? Google processes 20 PB a day (2008) Wayback Machine has 3 PB TB/month (03/2009) 9.6 PB recently Facebook processes 500 TB/day (08/2012) ebay has > 10 PB of user data + 50 TB/day (01/2012) CERN Data Centre has over 100 PB of physics data; 75 PB of this data comes from three years of LHC, i.e., 25 PB a year (02/2013)

5 What To Do With More Data? Machine learning algorithms really don t matter, all that matters is the amount of data you have [Banko and Brill, 2001] More data à higher accuracy More data à accuracy of different algorithms converge

6 What To Do With More Data? Answering factoid questions Pattern matching on the Web Works amazingly well Who shot Abraham Lincoln? X shot Abraham Lincoln Learning relations Start with seed instances Search for patterns on the Web Using patterns to find more instances Wolfgang Amadeus Mozart ( ) Einstein was born in 1879 Birthday-of(Mozart, 1756) Birthday-of(Einstein, 1879) PERSON (DATE DATE) PERSON was born in DATE

7 Motivation: Google Example 20+ billion web pages x 20KB = 400+ TB 1 computer reads MB/sec from disk ~4 months to read the web ~1,000 hard drives to store the web Not scalable: takes even more to do something useful with the data! A standard architecture for such problems has emerged Cluster of commodity Linux nodes Commodity network (ethernet) to connect them

8 Cluster Architecture 1 Gbps between any pair of nodes in a rack Switch 2-10 Gbps backbone between racks Switch Switch CPU CPU CPU CPU Mem Mem Mem Mem Disk Disk Disk Disk Each rack contains nodes In 2011 it was guestimated that Google had 1M machines,

9 Divide and Conquer Work Partition w 1 w 2 w 3 worker worker worker r 1 r 2 r 3 Result Combine Problems that may arise?

10 Parallelization Challenges How do we assign work units to workers? What if we have more work units than workers? What if workers need to share partial results? How do we aggregate partial results? How do we know all the workers have finished? What if workers die? What is the common theme of all of these problems?

11 Common Theme Parallelization problems arise from: Communication between workers (e.g., to exchange state) Access to shared resources (e.g., data) We need a synchronization mechanism

12 Managing Multiple Workers It is difficult because: We do not know the order in which workers run We do not know when workers interrupt each other We do not know the order in which workers access shared data Thus, we need: Semaphores Conditional variables (wait, notify, broadcast) Barriers Still, lots of problems: Deadlock, starvation, race conditions,... Dining philosophers, sleeping barbers, cigarette smokers... Moral of the story: this is a HARD problem!

13 Current Tools Programming Models Shared memory (pthreads) Message passing (MPI) Shared Memory Memory Message Passing Design Patterns P 1 P 2 P 3 P 4 P 5 P 1 P 2 P 3 P 4 P 5 Master-slaves Producer-consumer flows Shared work queues master producer consumer work queue slaves producer consumer

14 Where the rubber meets the road Concurrency is difficult to reason about At the scale of datacenters (even across datacenters) In the presence of failures In terms of multiple interacting services Not to mention debugging The reality: Lots of one-off solutions, custom code Write you own dedicated library, then program with it Burden on the programmer to explicitly manage everything

15 What s the point? It s all about the right level of abstraction We need a new model that is more appropriate for the multicore/cluster environment Hide system-level details from the developers No more race conditions, lock contention, etc. Separating the what from how Developer specifies the computation that needs to be performed Execution framework ( runtime ) handles actual execution The datacenter is the computer! MapReduce to the rescue!

16 Cloud (Utility) Computing What? Computing resources as a metered service ( pay as you go ) Ability to dynamically provision virtual machines Why? Cost: capital vs. operating expenses Scalability: infinite capacity Elasticity: scale up or down on demand Does it make sense? Benefits to cloud users Business case for cloud providers MapReduce computational model abstracts many of the complexities

17 Big Ideas: Scale Out vs. Scale Up Scale up: small number of high-end servers Symmetric multi-processing (SMP) machines, large shared memory Not cost-effective: cost of machines does not scale linearly; and no single SMP machine is big enough Not fault-tolerant Scale out: large number of commodity low-end servers is more effective for data-intensive applications core machines vs core machines Low-end server platform is about 4 times more cost efficient than a high-end shared memory platform from the same vendor. [Barroso and Hölzle, 2009]

18 Big Ideas: Failures are Common Suppose a cluster is built using machines with a mean-time between failures (MTBF) of 1,000 days For a 10,000 server cluster, there are on average 10 failures per day! MapReduce implementation copes with failures Automatic task restarts

19 Big Ideas: Move Processing to Data Supercomputers often have processing nodes and storage nodes Computationally expensive tasks High-capacity interconnect to move data around Many data-intensive applications are not very processor-demanding Data movement leads to a bottleneck in the network! New idea: move processing to where the data reside In MapReduce, processors and storage are co-located Leverage locality

20 Big Ideas: Avoid Random Access Disk seek times are determined by mechanical factors Read heads can only move so fast and platters can only spin so rapidly

21 Big Ideas: Avoid Random Access Example: 1 TB database containing byte records Random access: each update takes ~30ms (seek,read, write) Updating 1% of the records takes ~35 days Sequential access: 100MB/s throughput Reading the whole database and rewriting all the records, takes 5.6 hours MapReduce was designed for batch processing: organizing computations into long streaming operations

22 Big Ideas: Abstract System-Level Details Developing distributed software is very challenging Manage multiple threads, processes, machines Concurrency control and synchronization Race conditions, deadlocks a nightmare to debug! MapReduce isolates developers from these details Programmer defines what computations are to be performed MapReduce execution framework takes care of how the computations are carried out

23 MapReduce

24 Typical Large-Data Problem Iterate over a large number of records Map Extract something of interest from each Shuffle and sort intermediate results Aggregate intermediate results Reduce Generate final output Key idea: provide a functional abstraction for these two operations [Dean and Ghemawat, OSDI 2004]

25 Map and Reduce The idea of Map and Reduce is 40+ year old - Present in all Functional Programming Languages - See APL, Lisp and ML Alternate name for Map: Apply-All Higher Order Functions - Take function definitions as arguments, or - Return a function as output Map and Reduce are higher-order functions

26 Map Examples in Haskell ñ map (+1) [1,2,3,4,5] == [2, 3, 4, 5, 6] ñ map (tolower) == ñ map (`mod` 3) [1..10] == [1, 2, 0, 1, 2, 0, 1, 2, 0, 1]

27 Fold-Left in Haskell ñ Definition - foldl f z [] = z - foldl f z (x:xs) = foldl f (f z x) xs ñ Examples - foldl (+) 0 [1..5] = 15 - foldl (+) 10 [1..5] = 25 - foldl (div) 7 [34,56,12,4,23] = 0 (+ (+ (+ (+ (+ 0 1) 2) 3) 4) 5)

28 MapReduce Programmers specify two functions: map (k 1, v 1 ) [(k 2, v 2 )] reduce (k 2, [v 2 ]) [(k 3, v 3 )] All values with the same key are sent to the same reducer The execution framework handles everything else

29 Hello World : Word Count Map(String docid, String text): for each word w in text: Emit(w, 1); k 1 v 1 k 2 v 2 map (docid, text) [(word, 1)] Reduce(String word, Iterator<Int> values): int sum = 0; for each v in values: k sum += v; 2 [v 2 ] k 3 v 3 reduce (word, [1,, 1]) [(word, count)] Emit(word, value);

30 Word Counting with MapReduce M 1 Documents! Key! Value! Key! Value! Doc1 Doc2 Doc3 Doc4 Doc5 Financial, IMF, Eco nomics, Crisis Financial, IMF, Cris is Documents! Economics, Harry Financial, Harry, Po tter, Film Crisis, Harry, Potter" Map" Map" `" `" Financial" 1 IMF" 1" Economics" 1" `" Crisis" 1" Financial" 1" `" IMF" 1" Crisis" 1" Economics" 1 `" Harry" 1" Financial" 1" Harry" 1" `" Potter" 1" Film" 1" Crisis" 1" Harry" 1" `" Potter" 1" M 2 Kyuseok Shim (VLDB 2012 TUTORIAL)"

31 Word Counting with MapReduce Doc1 Doc2 Doc3 Doc4 Doc5 Documents! Financial, IMF, Eco nomics, Crisis Financial, IMF, Cris is Documents! Economics, Harry Financial, Harry, Po tter, Film Crisis, Harry, Potter" Map" Map" Key! Value! Key! Value! Key! Value list! Financial" Financial" 1 Crisis" 1, 1, 1 1 Financial" IMF" 1" Crisis" 1, 1" 1" Financial" Economics" 1" Crisis" 1, 1" 1" IMF" Crisis" 1" Harry" 1, 1, 1" 1" IMF" Harry" 1" Harry" 1, 1, 1" 1" Economics" Film" 1" Harry" 1" 1" Economics" Potter" 1" Film" 1, 1" 1" Potter" 1" Potter" 1" Reduce" Reduce" Key! `" `"`" Financial" 3 IMF" 2 Economics" 2 Crisis" 3 `" Harry" 3 Film" 1 Potter" 2 Value! Before reduce functions are called, for each distinct key, the list of its values is generated Kyuseok Shim (VLDB 2012 TUTORIAL)"

32 MapReduce k 1 v 1 k 2 v 2 k 3 v 3 k 4 v 4 k 5 v 5 k 6 v 6 Group by is applied to intermediate keys map map map map a 1 b 2 c 3 c 6 a 5 c 2 b 7 c 8 Shuffle and Sort: aggregate values by keys a 1 5 b 2 7 c reduce reduce reduce r 1 s 1 r 2 s 2 r 3 s 3 Output written to DFS

33 MapReduce Runtime Handles scheduling Assigns workers to map and reduce tasks Handles data distribution Moves processes to data Handles synchronization Gathers, sorts, and shuffles intermediate data Handles errors and faults Detects worker failures and restarts Everything happens on top of a distributed FS (later)

34 MapReduce Programmers specify two functions: map (k 1, v 1 ) [(k 2, v 2 )] reduce (k 2, [v 2 ]) [(k 3, v 3 )] All values with the same key are sent to the same reducer Mappers and reducers can specify arbitrary computations Be careful with access to external resources! The execution framework handles everything else Not quite often, programmers also specify: partition (k 2, number of partitions) partition for k 2 Often a simple hash of the key, e.g., hash(k 2 ) mod n Divides up key space for parallel reduce operations combine (k 2, v 2 ) (k 2, v 2 ) Mini-reducers that run in memory after the map phase Used as an optimization to reduce network traffic

35 Word Counting with MapReduce M 1 Documents! Key! Value! Key! Value! Doc1 Doc2 Financial, IMF, Eco nomics, Crisis Financial, IMF, Cris is Map" Financial" 12 IMF" 1" 2" Economics" 1" Economics" 1 Harry" 1" 3" Financial" 1" Crisis" 1" 2" Harry" Potter" 1" 2" Doc3 Doc4 Doc5 Documents! Economics, Harry Financial, Harry, Po tter, Film Crisis, Harry, Potter" Map" Financial" 1" IMF" 1" Crisis" 1" Potter" Film" 1" Film" Crisis" 1" Crisis" 1" Harry" 1" Potter" 1" M 2 Kyuseok Shim (VLDB 2012 TUTORIAL)"

36 MapReduce: The Complete Picture k 1 v 1 k 2 v 2 k 3 v 3 k 4 v 4 k 5 v 5 k 6 v 6 map map map map a 1 b 2 c 3 c 6 a 5 c 2 b 7 c 8 combine combine combine combine a 1 b 2 c 9 a 5 c 2 b 7 c 8 partition partition partition partition Shuffle and Sort: aggregate values by keys a 1 5 b 2 7 c reduce reduce reduce r 1 s 1 r 2 s 2 r 3 s 3

37 Two More Details Barrier between map and reduce phases But we can begin copying intermediate data earlier Keys arrive at each reducer in sorted order No enforced ordering across reducers

38 MapReduce can refer to The programming model The execution framework (aka runtime ) The specific implementation Usage is usually clear from context!

39 MapReduce Implementations Google has a proprietary implementation in C++ Bindings in Java, Python Hadoop is an open-source implementation in Java Development led by Yahoo, used in production Now an Apache project Rapidly expanding software ecosystem Lots of custom research implementations For GPUs, cell processors, etc.

40 Google s MapReduce vs. Hadoop MapReduce Hadoop Organization Google Yahoo/Apache Implementation C++ Java Distributed File System GFS HDFS Database Bigtable HBase Distributed Lock Manager Chubby ZooKeeper

41 User Program (1) submit Master (2) schedule map (2) schedule reduce worker split 0 split 1 split 2 split 3 split 4 (3) read worker (4) local write (5) remote read worker worker (6) write output file 0 output file 1 worker Input files Map phase Intermediate files (on local disk) Reduce phase Output files Adapted from [Dean and Ghemawat, OSDI 2004]

42 Distributed File System Do not move data to workers move workers to the data! Store data on the local disks of nodes in the cluster Start up the workers on the node that has the data local Why? Not enough RAM to hold all the data in memory Disk access is slow, but disk throughput is reasonable Moving data in the network does not scale A distributed file system is the answer GFS (Google File System) for Google s MapReduce HDFS (Hadoop Distributed File System) for Hadoop

43 GFS: Assumptions Commodity hardware over exotic hardware Scale out, not up High component failure rates Inexpensive commodity components fail all the time Modest number of huge files Multi-gigabyte files are common, if not encouraged Why many small files are not preferred? Files are write-once, mostly appended to Perhaps concurrently Large streaming reads over random access High sustained throughput over low latency

44 Files stored as chunks Fixed size (e.g., 64MB) GFS: Design Decisions Reliability through replication Each chunk replicated across 3+ chunkservers Single master to coordinate access, keep metadata Simple centralized management No data caching Little benefit due to large datasets, streaming reads Simplify the API Push some of the issues onto the client (e.g., data layout) HDFS = GFS clone (same basic ideas)

45 From GFS to HDFS Terminology differences: GFS master = Hadoop namenode GFS chunkservers = Hadoop datanodes Functional differences: No file appends in HDFS (planned feature) HDFS performance is (likely) slower

46 HDFS Architecture HDFS namenode Application HDFS Client (file name, block id) (block id, block location) File namespace /foo/bar block 3df2 instructions to datanode (block id, byte range) block data datanode state HDFS datanode Linux file system HDFS datanode Linux file system

47 Namenode Responsibilities Managing the file system namespace: Holds file/directory structure, metadata, file-to-block mapping, access permissions, etc. Coordinating file operations: Directs clients to datanodes for reads and writes No data is moved through the namenode Maintaining overall health: Periodic communication with the datanodes Block re-replication and rebalancing Garbage collection (lazy deletion)

48 Hadoop Cluster Architecture namenode job submission node namenode daemon jobtracker tasktracker datanode daemon Linux file system slave node tasktracker datanode daemon Linux file system slave node tasktracker datanode daemon Linux file system slave node

49 Fault Recovery Workers are pinged by master periodically Non-responsive workers are marked as failed All tasks in-progress or completed by failed worker become eligible for rescheduling Master could periodically checkpoint Current implementations abort on master failure: single point of failure

50 References Jeffrey Dean and Sanjay Ghemawat, MapReduce: Simplified Data Processing on Large Clusters. Usenix SDI '04, 2004: dean/dean.pdf David DeWitt, Michael Stonebraker, MapReduce: A major step backwards, craig-henderson.blogspot.com databases_are_hammers_mapreduc.php