MapReduce Simplified Data Processing on Large Clusters
|
|
|
- Bonnie Holt
- 5 years ago
- Views:
Transcription
1 MapReduce Simplified Data Processing on Large Clusters Amir H. Payberah Amirkabir University of Technology (Tehran Polytechnic) Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 1 / 50
2 What do we do when there is too much data to process? Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 2 / 50
3 Scale Up vs. Scale Out (1/2) Scale up or scale vertically: adding resources to a single node in a system. Scale out or scale horizontally: adding more nodes to a system. Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 3 / 50
4 Scale Up vs. Scale Out (2/2) Scale up: more expensive than scaling out. Scale out: more challenging for fault tolerance and software development. Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 4 / 50
5 Taxonomy of Parallel Architectures DeWitt, D. and Gray, J. Parallel database systems: the future of high performance database systems. ACM Communications, 35(6), 85-98, Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 5 / 50
6 Taxonomy of Parallel Architectures DeWitt, D. and Gray, J. Parallel database systems: the future of high performance database systems. ACM Communications, 35(6), 85-98, Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 5 / 50
7 MapReduce A shared nothing architecture for processing large data sets with a parallel/distributed algorithm on clusters of commodity hardware. Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 6 / 50
8 Challenges I How to distribute computation? I How can we make it easy to write distributed programs? I Machines failure. Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 7 / 50
9 Idea Issue: Copying data over a network takes time. Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 8 / 50
10 Idea Issue: Copying data over a network takes time. Idea: Bring computation close to the data. Store files multiple times for reliability. Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 8 / 50
11 Simplicity Don t worry about parallelization, fault tolerance, data distribution, and load balancing (MapReduce takes care of these). Hide system-level details from programmers. Simplicity! Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 9 / 50
12 MapReduce Definition A programming model: to batch process large data sets (inspired by functional programming). Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 10 / 50
13 MapReduce Definition A programming model: to batch process large data sets (inspired by functional programming). An execution framework: to run parallel algorithms on clusters of commodity hardware. Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 10 / 50
14 Programming Model Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 11 / 50
15 Warm-up Task (1/2) We have a huge text document. Count the number of times each distinct word appears in the file Application: analyze web server logs to find popular URLs. Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 12 / 50
16 Warm-up Task (2/2) File too large for memory, but all word, count pairs fit in memory. Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 13 / 50
17 Warm-up Task (2/2) File too large for memory, but all word, count pairs fit in memory. words(doc.txt) sort uniq -c where words takes a file and outputs the words in it, one per a line Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 13 / 50
18 Warm-up Task (2/2) File too large for memory, but all word, count pairs fit in memory. words(doc.txt) sort uniq -c where words takes a file and outputs the words in it, one per a line It captures the essence of MapReduce: naturally parallelizable. great thing is that it is Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 13 / 50
19 MapReduce Overview words(doc.txt) sort uniq -c Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 14 / 50
20 MapReduce Overview words(doc.txt) sort uniq -c Sequentially read a lot of data. Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 14 / 50
21 MapReduce Overview words(doc.txt) sort uniq -c Sequentially read a lot of data. Map: extract something you care about. Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 14 / 50
22 MapReduce Overview words(doc.txt) sort uniq -c Sequentially read a lot of data. Map: extract something you care about. Group by key: sort and shuffle. Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 14 / 50
23 MapReduce Overview words(doc.txt) sort uniq -c Sequentially read a lot of data. Map: extract something you care about. Group by key: sort and shuffle. Reduce: aggregate, summarize, filter or transform. Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 14 / 50
24 MapReduce Overview words(doc.txt) sort uniq -c Sequentially read a lot of data. Map: extract something you care about. Group by key: sort and shuffle. Reduce: aggregate, summarize, filter or transform. Write the result. Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 14 / 50
25 MapReduce Dataflow map function: processes data and generates a set of intermediate key/value pairs. reduce function: merges all intermediate values associated with the same intermediate key. Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 15 / 50
26 Example: Word Count Consider doing a word count of the following file using MapReduce: Hello World Bye World Hello Hadoop Goodbye Hadoop Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 16 / 50
27 Example: Word Count - map The map function reads in words one a time and outputs (word, 1) for each parsed input word. The map function output is: (Hello, 1) (World, 1) (Bye, 1) (World, 1) (Hello, 1) (Hadoop, 1) (Goodbye, 1) (Hadoop, 1) Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 17 / 50
28 Example: Word Count - shuffle The shuffle phase between map and reduce phase creates a list of values associated with each key. The reduce function input is: (Bye, (1)) (Goodbye, (1)) (Hadoop, (1, 1) (Hello, (1, 1)) (World, (1, 1)) Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 18 / 50
29 Example: Word Count - reduce The reduce function sums the numbers in the list for each key and outputs (word, count) pairs. The output of the reduce function is the output of the MapReduce job: (Bye, 1) (Goodbye, 1) (Hadoop, 2) (Hello, 2) (World, 2) Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 19 / 50
30 Combiner Function (1/2) In some cases, there is significant repetition in the intermediate keys produced by each map task, and the reduce function is commutative and associative. Machine 1: (Hello, 1) (World, 1) (Bye, 1) (World, 1) Machine 2: (Hello, 1) (Hadoop, 1) (Goodbye, 1) (Hadoop, 1) Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 20 / 50
31 Combiner Function (2/2) Users can specify an optional combiner function to merge partially data before it is sent over the network to the reduce function. Typically the same code is used to implement both the combiner and the reduce function. Machine 1: (Hello, 1) (World, 2) (Bye, 1) Machine 2: (Hello, 1) (Hadoop, 2) (Goodbye, 1) Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 21 / 50
32 Example: Word Count - map public static class MyMap extends Mapper<...> { private final static IntWritable one = new IntWritable(1); private Text word = new Text(); public void map(longwritable key, Text value, Context context) throws IOException, InterruptedException { String line = value.tostring(); StringTokenizer tokenizer = new StringTokenizer(line); while (tokenizer.hasmoretokens()) { word.set(tokenizer.nexttoken()); context.write(word, one); } } } Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 22 / 50
33 Example: Word Count - reduce public static class MyReduce extends Reducer<...> { public void reduce(text key, Iterator<...> values, Context context) throws IOException, InterruptedException { int sum = 0; while (values.hasnext()) sum += values.next().get(); } } context.write(key, new IntWritable(sum)); Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 23 / 50
34 Example: Word Count - driver public static void main(string[] args) throws Exception { Configuration conf = new Configuration(); Job job = new Job(conf, "wordcount"); job.setoutputkeyclass(text.class); job.setoutputvalueclass(intwritable.class); job.setmapperclass(mymap.class); job.setcombinerclass(myreduce.class); job.setreducerclass(myreduce.class); job.setinputformatclass(textinputformat.class); job.setoutputformatclass(textoutputformat.class); FileInputFormat.addInputPath(job, new Path(args[0])); FileOutputFormat.setOutputPath(job, new Path(args[1])); } job.waitforcompletion(true); Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 24 / 50
35 Example: Word Count - Compile and Run (1/2) # start hdfs > hadoop-daemon.sh start namenode > hadoop-daemon.sh start datanode # make the input folder in hdfs > hdfs dfs -mkdir -p input # copy input files from local filesystem into hdfs > hdfs dfs -put file0 input/file0 > hdfs dfs -put file1 input/file1 > hdfs dfs -ls input/ input/file0 input/file1 > hdfs dfs -cat input/file0 Hello World Bye World > hdfs dfs -cat input/file1 Hello Hadoop Goodbye Hadoop Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 25 / 50
36 Example: Word Count - Compile and Run (2/2) > mkdir wordcount_classes > javac -classpath $HADOOP_HOME/share/hadoop/common/hadoop-common jar: $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-client-core jar: $HADOOP_HOME/share/hadoop/common/lib/commons-cli-1.2.jar -d wordcount_classes sics/wordcount.java > jar -cvf wordcount.jar -C wordcount_classes/. > hadoop jar wordcount.jar sics.wordcount input output > hdfs dfs -ls output output/part > hdfs dfs -cat output/part Bye 1 Goodbye 1 Hadoop 2 Hello 2 World 2 Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 26 / 50
37 Execution Engine Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 27 / 50
38 MapReduce Execution (1/7) The user program divides the input files into M splits. A typical size of a split is the size of a HDFS block (64 MB). Converts them to key/value pairs. It starts up many copies of the program on a cluster of machines. J. Dean and S. Ghemawat, MapReduce: simplified data processing on large clusters, ACM Communications 51(1), Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 28 / 50
39 MapReduce Execution (2/7) One of the copies of the program is master, and the rest are workers. The master assigns works to the workers. It picks idle workers and assigns each one a map task or a reduce task. J. Dean and S. Ghemawat, MapReduce: simplified data processing on large clusters, ACM Communications 51(1), Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 29 / 50
40 MapReduce Execution (3/7) A map worker reads the contents of the corresponding input splits. It parses key/value pairs out of the input data and passes each pair to the user defined map function. The intermediate key/value pairs produced by the map function are buffered in memory. J. Dean and S. Ghemawat, MapReduce: simplified data processing on large clusters, ACM Communications 51(1), Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 30 / 50
41 MapReduce Execution (4/7) The buffered pairs are periodically written to local disk. They are partitioned into R regions (hash(key) mod R). The locations of the buffered pairs on the local disk are passed back to the master. The master forwards these locations to the reduce workers. J. Dean and S. Ghemawat, MapReduce: simplified data processing on large clusters, ACM Communications 51(1), Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 31 / 50
42 MapReduce Execution (5/7) A reduce worker reads the buffered data from the local disks of the map workers. When a reduce worker has read all intermediate data, it sorts it by the intermediate keys. J. Dean and S. Ghemawat, MapReduce: simplified data processing on large clusters, ACM Communications 51(1), Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 32 / 50
43 MapReduce Execution (6/7) The reduce worker iterates over the intermediate data. For each unique intermediate key, it passes the key and the corresponding set of intermediate values to the user defined reduce function. The output of the reduce function is appended to a final output file for this reduce partition. J. Dean and S. Ghemawat, MapReduce: simplified data processing on large clusters, ACM Communications 51(1), Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 33 / 50
44 MapReduce Execution (7/7) When all map tasks and reduce tasks have been completed, the master wakes up the user program. J. Dean and S. Ghemawat, MapReduce: simplified data processing on large clusters, ACM Communications 51(1), Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 34 / 50
45 Hadoop MapReduce and HDFS Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 35 / 50
46 Fault Tolerance - Worker Detect failure via periodic heartbeats. Re-execute in-progress map and reduce tasks. Re-execute completed map tasks: their output is stored on the local disk of the failed machine and is therefore inaccessible. Completed reduce tasks do not need to be re-executed since their output is stored in a global filesystem. Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 36 / 50
47 Fault Tolerance - Master State is periodically checkpointed: a new copy of master starts from the last checkpoint state. Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 37 / 50
48 MapReduce Weaknesses and Solving Techniques Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 38 / 50
49 W1: Access to Input Data Weakness 1 Scanning the entire input to perform the map-side processing. Initiating map tasks on all input partitions. Accessing only a subset of input data would be enough in certain cases. Lack of selective access to data. High communication cost. Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 39 / 50
50 S1: Access to Input Data Solution 1 Efficient access to data. Indexing data: Hadoop++, HAIL Intentional data placement: CoHadoop Data layout: Llama, Cheetah, RCFile, CIF Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 40 / 50
51 Weakness 2 W2: Redundant Processing and Recomputation Performing similar processing by different jobs over the same data. Jobs are processed independently: redundant processing No way to reuse the results produced by previous jobs. Future jobs may require those results: recompute everything Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 41 / 50
52 Solution 2 S2: Redundant Processing and Recomputation Batch processing of jobs: MRShare Result sharing and materialization: ReStore Incremental processing: Incoop Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 42 / 50
53 Weakness 3 W3: Lack of Early Termination Map tasks must process the entire input data before any reduce task can start processing. Some jobs may need only sampling of data. Quick retrieval of approximate results. Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 43 / 50
54 S3: Lack of Early Termination Solution 3 Sampling: EARL Sorting: RanKloud Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 44 / 50
55 W4: Lack of Iteration Weakness 4 MapReduce programmers need to write a sequence of MapReduce jobs and coordinate their execution, in order to implement an iterative processing. Data should be reloaded and reprocessed in each iteration. Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 45 / 50
56 S4: Lack of Iteration Solution 4 Looping, caching, pipelining: Stratosphere, Haloop, MapReduce online, NOVA, Twister, CBP, Pregel, PrIter Incremental processing: Stratosphere, REX, Differential dataflow Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 46 / 50
57 Weakness 5 W5: Lack of Interactive and Real-Time Processing Various overheads to guarantee fault-tolerance that negatively impact the performance. Many applications require fast response times, interactive analysis, and online analytics. Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 47 / 50
58 Solution 5 S5: Lack of Interactive and Real-Time Processing Streaming, pipelining: Dremel, Impala, Hyracks, Tenzing In-memory processing: PowerDrill, Spark/Shark, M3R Pre-computation: BlikDB Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 48 / 50
59 Summary Programming model: Map and Reduce Execution framework Batch processing Shared nothing architecture Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 49 / 50
60 Questions? Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 50 / 50
Parallel Processing - MapReduce and FlumeJava. Amir H. Payberah 14/09/2018
Parallel Processing - MapReduce and FlumeJava Amir H. Payberah payberah@kth.se 14/09/2018 The Course Web Page https://id2221kth.github.io 1 / 83 Where Are We? 2 / 83 What do we do when there is too much
An Introduction to Apache Spark
An Introduction to Apache Spark Amir H. Payberah amir@sics.se SICS Swedish ICT Amir H. Payberah (SICS) Apache Spark Feb. 2, 2016 1 / 67 Big Data small data big data Amir H. Payberah (SICS) Apache Spark
UNIT V PROCESSING YOUR DATA WITH MAPREDUCE Syllabus
UNIT V PROCESSING YOUR DATA WITH MAPREDUCE Syllabus Getting to know MapReduce MapReduce Execution Pipeline Runtime Coordination and Task Management MapReduce Application Hadoop Word Count Implementation.
Introduction to Map/Reduce. Kostas Solomos Computer Science Department University of Crete, Greece
Introduction to Map/Reduce Kostas Solomos Computer Science Department University of Crete, Greece What we will cover What is MapReduce? How does it work? A simple word count example (the Hello World! of
Large-scale Information Processing
Sommer 2013 Large-scale Information Processing Ulf Brefeld Knowledge Mining & Assessment brefeld@kma.informatik.tu-darmstadt.de Anecdotal evidence... I think there is a world market for about five computers,
Clustering Documents. Case Study 2: Document Retrieval
Case Study 2: Document Retrieval Clustering Documents Machine Learning for Big Data CSE547/STAT548, University of Washington Sham Kakade April 21 th, 2015 Sham Kakade 2016 1 Document Retrieval Goal: Retrieve
Clustering Documents. Document Retrieval. Case Study 2: Document Retrieval
Case Study 2: Document Retrieval Clustering Documents Machine Learning for Big Data CSE547/STAT548, University of Washington Sham Kakade April, 2017 Sham Kakade 2017 1 Document Retrieval n Goal: Retrieve
COMP4442. Service and Cloud Computing. Lab 12: MapReduce. Prof. George Baciu PQ838.
COMP4442 Service and Cloud Computing Lab 12: MapReduce www.comp.polyu.edu.hk/~csgeorge/comp4442 Prof. George Baciu csgeorge@comp.polyu.edu.hk PQ838 1 Contents Introduction to MapReduce A WordCount example
Parallel Data Processing with Hadoop/MapReduce. CS140 Tao Yang, 2014
Parallel Data Processing with Hadoop/MapReduce CS140 Tao Yang, 2014 Overview What is MapReduce? Example with word counting Parallel data processing with MapReduce Hadoop file system More application example
L22: SC Report, Map Reduce
L22: SC Report, Map Reduce November 23, 2010 Map Reduce What is MapReduce? Example computing environment How it works Fault Tolerance Debugging Performance Google version = Map Reduce; Hadoop = Open source
Parallel Computing: MapReduce Jin, Hai
Parallel Computing: MapReduce Jin, Hai School of Computer Science and Technology Huazhong University of Science and Technology ! MapReduce is a distributed/parallel computing framework introduced by Google
MapReduce: Simplified Data Processing on Large Clusters 유연일민철기
MapReduce: Simplified Data Processing on Large Clusters 유연일민철기 Introduction MapReduce is a programming model and an associated implementation for processing and generating large data set with parallel,
Map Reduce. MCSN - N. Tonellotto - Distributed Enabling Platforms
Map Reduce 1 MapReduce inside Google Googlers' hammer for 80% of our data crunching Large-scale web search indexing Clustering problems for Google News Produce reports for popular queries, e.g. Google
A BigData Tour HDFS, Ceph and MapReduce
A BigData Tour HDFS, Ceph and MapReduce These slides are possible thanks to these sources Jonathan Drusi - SCInet Toronto Hadoop Tutorial, Amir Payberah - Course in Data Intensive Computing SICS; Yahoo!
Java in MapReduce. Scope
Java in MapReduce Kevin Swingler Scope A specific look at the Java code you might use for performing MapReduce in Hadoop Java program recap The map method The reduce method The whole program Running on
Chapter 3. Distributed Algorithms based on MapReduce
Chapter 3 Distributed Algorithms based on MapReduce 1 Acknowledgements Hadoop: The Definitive Guide. Tome White. O Reilly. Hadoop in Action. Chuck Lam, Manning Publications. MapReduce: Simplified Data
Big Data Analytics. 4. Map Reduce I. Lars Schmidt-Thieme
Big Data Analytics 4. Map Reduce I Lars Schmidt-Thieme Information Systems and Machine Learning Lab (ISMLL) Institute of Computer Science University of Hildesheim, Germany original slides by Lucas Rego
CS 470 Spring Parallel Algorithm Development. (Foster's Methodology) Mike Lam, Professor
CS 470 Spring 2018 Mike Lam, Professor Parallel Algorithm Development (Foster's Methodology) Graphics and content taken from IPP section 2.7 and the following: http://www.mcs.anl.gov/~itf/dbpp/text/book.html
ECE5610/CSC6220 Introduction to Parallel and Distribution Computing. Lecture 6: MapReduce in Parallel Computing
ECE5610/CSC6220 Introduction to Parallel and Distribution Computing Lecture 6: MapReduce in Parallel Computing 1 MapReduce: Simplified Data Processing Motivation Large-Scale Data Processing on Large Clusters
Big Data Analysis using Hadoop. Map-Reduce An Introduction. Lecture 2
Big Data Analysis using Hadoop Map-Reduce An Introduction Lecture 2 Last Week - Recap 1 In this class Examine the Map-Reduce Framework What work each of the MR stages does Mapper Shuffle and Sort Reducer
CS 345A Data Mining. MapReduce
CS 345A Data Mining MapReduce Single-node architecture CPU Machine Learning, Statistics Memory Classical Data Mining Disk Commodity Clusters Web data sets can be very large Tens to hundreds of terabytes
Map-Reduce in Various Programming Languages
Map-Reduce in Various Programming Languages 1 Context of Map-Reduce Computing The use of LISP's map and reduce functions to solve computational problems probably dates from the 1960s -- very early in the
Clustering Documents. Document Retrieval. Case Study 2: Document Retrieval
Case Study 2: Document Retrieval Clustering Documents Machine Learning for Big Data CSE547/STAT548, University of Washington Emily Fox April 16 th, 2015 Emily Fox 2015 1 Document Retrieval n Goal: Retrieve
Map Reduce Group Meeting
Map Reduce Group Meeting Yasmine Badr 10/07/2014 A lot of material in this presenta0on has been adopted from the original MapReduce paper in OSDI 2004 What is Map Reduce? Programming paradigm/model for
CS555: Distributed Systems [Fall 2017] Dept. Of Computer Science, Colorado State University
![CS555: Distributed Systems [Fall 2017] Dept. Of Computer Science, Colorado State University](/thumbs/90/104368640.jpg "CS555: Distributed Systems [Fall 2017] Dept. Of Computer Science, Colorado State University")
CS 555: DISTRIBUTED SYSTEMS [MAPREDUCE] Shrideep Pallickara Computer Science Colorado State University Frequently asked questions from the previous class survey Bit Torrent What is the right chunk/piece
Introduction to MapReduce
732A54 Big Data Analytics Introduction to MapReduce Christoph Kessler IDA, Linköping University Towards Parallel Processing of Big-Data Big Data too large to be read+processed in reasonable time by 1 server
Clustering Lecture 8: MapReduce
Clustering Lecture 8: MapReduce Jing Gao SUNY Buffalo 1 Divide and Conquer Work Partition w 1 w 2 w 3 worker worker worker r 1 r 2 r 3 Result Combine 4 Distributed Grep Very big data Split data Split data
Guidelines For Hadoop and Spark Cluster Usage
Guidelines For Hadoop and Spark Cluster Usage Procedure to create an account in CSX. If you are taking a CS prefix course, you already have an account; to get an initial password created: 1. Login to https://cs.okstate.edu/pwreset
MapReduce. Kiril Valev LMU Kiril Valev (LMU) MapReduce / 35
MapReduce Kiril Valev LMU valevk@cip.ifi.lmu.de 23.11.2013 Kiril Valev (LMU) MapReduce 23.11.2013 1 / 35 Agenda 1 MapReduce Motivation Definition Example Why MapReduce? Distributed Environment Fault Tolerance
Parallel Programming Concepts
Parallel Programming Concepts MapReduce Frank Feinbube Source: MapReduce: Simplied Data Processing on Large Clusters; Dean et. Al. Examples for Parallel Programming Support 2 MapReduce 3 Programming model
Data-Intensive Computing with MapReduce
Data-Intensive Computing with MapReduce Session 2: Hadoop Nuts and Bolts Jimmy Lin University of Maryland Thursday, January 31, 2013 This work is licensed under a Creative Commons Attribution-Noncommercial-Share
September 2013 Alberto Abelló & Oscar Romero 1
duce-i duce-i September 2013 Alberto Abelló & Oscar Romero 1 Knowledge objectives 1. Enumerate several use cases of duce 2. Describe what the duce environment is 3. Explain 6 benefits of using duce 4.
Cloud Computing and Hadoop Distributed File System. UCSB CS170, Spring 2018
Cloud Computing and Hadoop Distributed File System UCSB CS70, Spring 08 Cluster Computing Motivations Large-scale data processing on clusters Scan 000 TB on node @ 00 MB/s = days Scan on 000-node cluster
Map-Reduce Applications: Counting, Graph Shortest Paths
Map-Reduce Applications: Counting, Graph Shortest Paths Adapted from UMD Jimmy Lin s slides, which is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States. See http://creativecommons.org/licenses/by-nc-sa/3.0/us/
Big Data Management and NoSQL Databases
NDBI040 Big Data Management and NoSQL Databases Lecture 2. MapReduce Doc. RNDr. Irena Holubova, Ph.D. holubova@ksi.mff.cuni.cz http://www.ksi.mff.cuni.cz/~holubova/ndbi040/ Framework A programming model
The MapReduce Abstraction
The MapReduce Abstraction Parallel Computing at Google Leverages multiple technologies to simplify large-scale parallel computations Proprietary computing clusters Map/Reduce software library Lots of other
MapReduce. Arend Hintze
MapReduce Arend Hintze Distributed Word Count Example Input data files cat * key-value pairs (0, This is a cat!) (14, cat is ok) (24, walk the dog) Mapper map() function key-value pairs (this, 1) (is,
Introduction to MapReduce
Basics of Cloud Computing Lecture 4 Introduction to MapReduce Satish Srirama Some material adapted from slides by Jimmy Lin, Christophe Bisciglia, Aaron Kimball, & Sierra Michels-Slettvet, Google Distributed
MapReduce Spark. Some slides are adapted from those of Jeff Dean and Matei Zaharia
MapReduce Spark Some slides are adapted from those of Jeff Dean and Matei Zaharia What have we learnt so far? Distributed storage systems consistency semantics protocols for fault tolerance Paxos, Raft,
1/30/2019 Week 2- B Sangmi Lee Pallickara
Week 2-A-0 1/30/2019 Colorado State University, Spring 2019 Week 2-A-1 CS535 BIG DATA FAQs PART A. BIG DATA TECHNOLOGY 3. DISTRIBUTED COMPUTING MODELS FOR SCALABLE BATCH COMPUTING Term project deliverable
CS MapReduce. Vitaly Shmatikov
CS 5450 MapReduce Vitaly Shmatikov BackRub (Google), 1997 slide 2 NEC Earth Simulator, 2002 slide 3 Conventional HPC Machine ucompute nodes High-end processors Lots of RAM unetwork Specialized Very high
Outline. What is Big Data? Hadoop HDFS MapReduce Twitter Analytics and Hadoop
Intro To Hadoop Bill Graham - @billgraham Data Systems Engineer, Analytics Infrastructure Info 290 - Analyzing Big Data With Twitter UC Berkeley Information School September 2012 This work is licensed
MapReduce: Recap. Juliana Freire & Cláudio Silva. Some slides borrowed from Jimmy Lin, Jeff Ullman, Jerome Simeon, and Jure Leskovec
MapReduce: Recap Some slides borrowed from Jimmy Lin, Jeff Ullman, Jerome Simeon, and Jure Leskovec MapReduce: Recap Sequentially read a lot of data Why? Map: extract something we care about map (k, v)
Cloud Computing. Up until now
Cloud Computing Lecture 9 Map Reduce 2010-2011 Introduction Up until now Definition of Cloud Computing Grid Computing Content Distribution Networks Cycle-Sharing Distributed Scheduling 1 Outline Map Reduce:
Department of Computer Science University of Cyprus EPL646 Advanced Topics in Databases. Lecture 16. Big Data Management VI (MapReduce Programming)
Department of Computer Science University of Cyprus EPL646 Advanced Topics in Databases Lecture 16 Big Data Management VI (MapReduce Programming) Credits: Pietro Michiardi (Eurecom): Scalable Algorithm
EE657 Spring 2012 HW#4 Zhou Zhao
EE657 Spring 2012 HW#4 Zhou Zhao Problem 6.3 Solution Referencing the sample application of SimpleDB in Amazon Java SDK, a simple domain which includes 5 items is prepared in the code. For instance, the
CSE Lecture 11: Map/Reduce 7 October Nate Nystrom UTA
CSE 3302 Lecture 11: Map/Reduce 7 October 2010 Nate Nystrom UTA 378,000 results in 0.17 seconds including images and video communicates with 1000s of machines web server index servers document servers
Map-Reduce. Marco Mura 2010 March, 31th
Map-Reduce Marco Mura (mura@di.unipi.it) 2010 March, 31th This paper is a note from the 2009-2010 course Strumenti di programmazione per sistemi paralleli e distribuiti and it s based by the lessons of
Batch Processing Basic architecture
Batch Processing Basic architecture in big data systems COS 518: Distributed Systems Lecture 10 Andrew Or, Mike Freedman 2 1 2 64GB RAM 32 cores 64GB RAM 32 cores 64GB RAM 32 cores 64GB RAM 32 cores 3
CS 61C: Great Ideas in Computer Architecture. MapReduce
CS 61C: Great Ideas in Computer Architecture MapReduce Guest Lecturer: Justin Hsia 3/06/2013 Spring 2013 Lecture #18 1 Review of Last Lecture Performance latency and throughput Warehouse Scale Computing
Big Data landscape Lecture #2
Big Data landscape Lecture #2 Contents 1 1 CORE Technologies 2 3 MapReduce YARN 4 SparK 5 Cassandra Contents 2 16 HBase 72 83 Accumulo memcached 94 Blur 10 5 Sqoop/Flume Contents 3 111 MongoDB 12 2 13
CS427 Multicore Architecture and Parallel Computing
CS427 Multicore Architecture and Parallel Computing Lecture 9 MapReduce Prof. Li Jiang 2014/11/19 1 What is MapReduce Origin from Google, [OSDI 04] A simple programming model Functional model For large-scale
FAQs. Topics. This Material is Built Based on, Analytics Process Model. 8/22/2018 Week 1-B Sangmi Lee Pallickara
CS435 Introduction to Big Data Week 1-B W1.B.0 CS435 Introduction to Big Data No Cell-phones in the class. W1.B.1 FAQs PA0 has been posted If you need to use a laptop, please sit in the back row. August
Informa)on Retrieval and Map- Reduce Implementa)ons. Mohammad Amir Sharif PhD Student Center for Advanced Computer Studies
Informa)on Retrieval and Map- Reduce Implementa)ons Mohammad Amir Sharif PhD Student Center for Advanced Computer Studies mas4108@louisiana.edu Map-Reduce: Why? Need to process 100TB datasets On 1 node:
Programming Systems for Big Data
Programming Systems for Big Data CS315B Lecture 17 Including material from Kunle Olukotun Prof. Aiken CS 315B Lecture 17 1 Big Data We ve focused on parallel programming for computational science There
Steps: First install hadoop (if not installed yet) by, https://sl6it.wordpress.com/2015/12/04/1-study-and-configure-hadoop-for-big-data/
SL-V BE IT EXP 7 Aim: Design and develop a distributed application to find the coolest/hottest year from the available weather data. Use weather data from the Internet and process it using MapReduce. Steps:
Big Data Analytics. Izabela Moise, Evangelos Pournaras, Dirk Helbing
Big Data Analytics Izabela Moise, Evangelos Pournaras, Dirk Helbing Izabela Moise, Evangelos Pournaras, Dirk Helbing 1 Big Data "The world is crazy. But at least it s getting regular analysis." Izabela
Where We Are. Review: Parallel DBMS. Parallel DBMS. Introduction to Data Management CSE 344
Where We Are Introduction to Data Management CSE 344 Lecture 22: MapReduce We are talking about parallel query processing There exist two main types of engines: Parallel DBMSs (last lecture + quick review)
Dept. Of Computer Science, Colorado State University
CS 455: INTRODUCTION TO DISTRIBUTED SYSTEMS [HADOOP/HDFS] Trying to have your cake and eat it too Each phase pines for tasks with locality and their numbers on a tether Alas within a phase, you get one,
W1.A.0 W2.A.0 1/22/2018 1/22/2018. CS435 Introduction to Big Data. FAQs. Readings
CS435 Introduction to Big Data 1/17/2018 W2.A.0 W1.A.0 CS435 Introduction to Big Data W2.A.1.A.1 FAQs PA0 has been posted Feb. 6, 5:00PM via Canvas Individual submission (No team submission) Accommodation
MapReduce-style data processing
MapReduce-style data processing Software Languages Team University of Koblenz-Landau Ralf Lämmel and Andrei Varanovich Related meanings of MapReduce Functional programming with map & reduce An algorithmic
Hive and Shark. Amir H. Payberah. Amirkabir University of Technology (Tehran Polytechnic)
Hive and Shark Amir H. Payberah amir@sics.se Amirkabir University of Technology (Tehran Polytechnic) Amir H. Payberah (Tehran Polytechnic) Hive and Shark 1393/8/19 1 / 45 Motivation MapReduce is hard to
CSE6331: Cloud Computing
CSE6331: Cloud Computing Leonidas Fegaras University of Texas at Arlington c 2017 by Leonidas Fegaras Map-Reduce Fundamentals Based on: J. Simeon: Introduction to MapReduce P. Michiardi: Tutorial on MapReduce
Distributed Computations MapReduce. adapted from Jeff Dean s slides
Distributed Computations MapReduce adapted from Jeff Dean s slides What we ve learnt so far Basic distributed systems concepts Consistency (sequential, eventual) Fault tolerance (recoverability, availability)
Ghislain Fourny. Big Data 6. Massive Parallel Processing (MapReduce)
Ghislain Fourny Big Data 6. Massive Parallel Processing (MapReduce) So far, we have... Storage as file system (HDFS) 13 So far, we have... Storage as tables (HBase) Storage as file system (HDFS) 14 Data
Map-Reduce Applications: Counting, Graph Shortest Paths
Map-Reduce Applications: Counting, Graph Shortest Paths Adapted from UMD Jimmy Lin s slides, which is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States. See http://creativecommons.org/licenses/by-nc-sa/3.0/us/
Batch Inherence of Map Reduce Framework
Available Online at www.ijcsmc.com International Journal of Computer Science and Mobile Computing A Monthly Journal of Computer Science and Information Technology IJCSMC, Vol. 4, Issue. 6, June 2015, pg.287
Session 1 Big Data and Hadoop - Overview. - Dr. M. R. Sanghavi
Session 1 Big Data and Hadoop - Overview - Dr. M. R. Sanghavi Acknowledgement Prof. Kainjan M. Sanghavi For preparing this prsentation This presentation is available on my blog https://maheshsanghavi.wordpress.com/expert-talk-fdp-workshop/
Data Clustering on the Parallel Hadoop MapReduce Model. Dimitrios Verraros
Data Clustering on the Parallel Hadoop MapReduce Model Dimitrios Verraros Overview The purpose of this thesis is to implement and benchmark the performance of a parallel K- means clustering algorithm on
Ghislain Fourny. Big Data Fall Massive Parallel Processing (MapReduce)
Ghislain Fourny Big Data Fall 2018 6. Massive Parallel Processing (MapReduce) Let's begin with a field experiment 2 400+ Pokemons, 10 different 3 How many of each??????????? 4 400 distributed to many volunteers
Survey Paper on Traditional Hadoop and Pipelined Map Reduce
International Journal of Computational Engineering Research Vol, 03 Issue, 12 Survey Paper on Traditional Hadoop and Pipelined Map Reduce Dhole Poonam B 1, Gunjal Baisa L 2 1 M.E.ComputerAVCOE, Sangamner,
April Final Quiz COSC MapReduce Programming a) Explain briefly the main ideas and components of the MapReduce programming model.
1. MapReduce Programming a) Explain briefly the main ideas and components of the MapReduce programming model. MapReduce is a framework for processing big data which processes data in two phases, a Map
Computer Science 572 Exam Prof. Horowitz Tuesday, April 24, 2017, 8:00am 9:00am
Computer Science 572 Exam Prof. Horowitz Tuesday, April 24, 2017, 8:00am 9:00am Name: Student Id Number: 1. This is a closed book exam. 2. Please answer all questions. 3. There are a total of 40 questions.
HDFS: Hadoop Distributed File System. CIS 612 Sunnie Chung
HDFS: Hadoop Distributed File System CIS 612 Sunnie Chung What is Big Data?? Bulk Amount Unstructured Introduction Lots of Applications which need to handle huge amount of data (in terms of 500+ TB per
Big Data: Architectures and Data Analytics
Big Data: Architectures and Data Analytics July 14, 2017 Student ID First Name Last Name The exam is open book and lasts 2 hours. Part I Answer to the following questions. There is only one right answer
PLATFORM AND SOFTWARE AS A SERVICE THE MAPREDUCE PROGRAMMING MODEL AND IMPLEMENTATIONS
PLATFORM AND SOFTWARE AS A SERVICE THE MAPREDUCE PROGRAMMING MODEL AND IMPLEMENTATIONS By HAI JIN, SHADI IBRAHIM, LI QI, HAIJUN CAO, SONG WU and XUANHUA SHI Prepared by: Dr. Faramarz Safi Islamic Azad
Introduction to Data Management CSE 344
Introduction to Data Management CSE 344 Lecture 26: Parallel Databases and MapReduce CSE 344 - Winter 2013 1 HW8 MapReduce (Hadoop) w/ declarative language (Pig) Cluster will run in Amazon s cloud (AWS)
TI2736-B Big Data Processing. Claudia Hauff
TI2736-B Big Data Processing Claudia Hauff ti2736b-ewi@tudelft.nl Intro Streams Streams Map Reduce HDFS Pig Pig Design Pattern Hadoop Mix Graphs Giraph Spark Zoo Keeper Spark But first Partitioner & Combiner
The MapReduce Framework
The MapReduce Framework In Partial fulfilment of the requirements for course CMPT 816 Presented by: Ahmed Abdel Moamen Agents Lab Overview MapReduce was firstly introduced by Google on 2004. MapReduce
Improving the MapReduce Big Data Processing Framework
Improving the MapReduce Big Data Processing Framework Gistau, Reza Akbarinia, Patrick Valduriez INRIA & LIRMM, Montpellier, France In collaboration with Divyakant Agrawal, UCSB Esther Pacitti, UM2, LIRMM
MapReduce & HyperDex. Kathleen Durant PhD Lecture 21 CS 3200 Northeastern University
MapReduce & HyperDex Kathleen Durant PhD Lecture 21 CS 3200 Northeastern University 1 Distributing Processing Mantra Scale out, not up. Assume failures are common. Move processing to the data. Process
Hadoop Map Reduce 10/17/2018 1
Hadoop Map Reduce 10/17/2018 1 MapReduce 2-in-1 A programming paradigm A query execution engine A kind of functional programming We focus on the MapReduce execution engine of Hadoop through YARN 10/17/2018
MI-PDB, MIE-PDB: Advanced Database Systems
MI-PDB, MIE-PDB: Advanced Database Systems http://www.ksi.mff.cuni.cz/~svoboda/courses/2015-2-mie-pdb/ Lecture 10: MapReduce, Hadoop 26. 4. 2016 Lecturer: Martin Svoboda svoboda@ksi.mff.cuni.cz Author:
BigData and MapReduce with Hadoop
BigData and MapReduce with Hadoop Ivan Tomašić 1, Roman Trobec 1, Aleksandra Rashkovska 1, Matjaž Depolli 1, Peter Mežnar 2, Andrej Lipej 2 1 Jožef Stefan Institute, Jamova 39, 1000 Ljubljana 2 TURBOINŠTITUT
2. MapReduce Programming Model
Introduction MapReduce was proposed by Google in a research paper: Jeffrey Dean and Sanjay Ghemawat. MapReduce: Simplified Data Processing on Large Clusters. OSDI'04: Sixth Symposium on Operating System
Experiences with a new Hadoop cluster: deployment, teaching and research. Andre Barczak February 2018
Experiences with a new Hadoop cluster: deployment, teaching and research Andre Barczak February 2018 abstract In 2017 the Machine Learning research group got funding for a new Hadoop cluster. However,
CIS 601 Graduate Seminar Presentation Introduction to MapReduce --Mechanism and Applicatoin. Presented by: Suhua Wei Yong Yu
CIS 601 Graduate Seminar Presentation Introduction to MapReduce --Mechanism and Applicatoin Presented by: Suhua Wei Yong Yu Papers: MapReduce: Simplified Data Processing on Large Clusters 1 --Jeffrey Dean
MAPREDUCE - PARTITIONER
MAPREDUCE - PARTITIONER http://www.tutorialspoint.com/map_reduce/map_reduce_partitioner.htm Copyright tutorialspoint.com A partitioner works like a condition in processing an input dataset. The partition
Data Analysis Using MapReduce in Hadoop Environment
Data Analysis Using MapReduce in Hadoop Environment Muhammad Khairul Rijal Muhammad*, Saiful Adli Ismail, Mohd Nazri Kama, Othman Mohd Yusop, Azri Azmi Advanced Informatics School (UTM AIS), Universiti
A Guide to Running Map Reduce Jobs in Java University of Stirling, Computing Science
A Guide to Running Map Reduce Jobs in Java University of Stirling, Computing Science Introduction The Hadoop cluster in Computing Science at Stirling allows users with a valid user account to submit and
Overview. Why MapReduce? What is MapReduce? The Hadoop Distributed File System Cloudera, Inc.
MapReduce and HDFS This presentation includes course content University of Washington Redistributed under the Creative Commons Attribution 3.0 license. All other contents: Overview Why MapReduce? What
Map- reduce programming paradigm
Map- reduce programming paradigm Some slides are from lecture of Matei Zaharia, and distributed computing seminar by Christophe Bisciglia, Aaron Kimball, & Sierra Michels-Slettvet. In pioneer days they
CompSci 516: Database Systems
CompSci 516 Database Systems Lecture 12 Map-Reduce and Spark Instructor: Sudeepa Roy Duke CS, Fall 2017 CompSci 516: Database Systems 1 Announcements Practice midterm posted on sakai First prepare and
Distributed Filesystem
Distributed Filesystem 1 How do we get data to the workers? NAS Compute Nodes SAN 2 Distributing Code! Don t move data to workers move workers to the data! - Store data on the local disks of nodes in the
Big Data Infrastructure CS 489/698 Big Data Infrastructure (Winter 2017)
Big Data Infrastructure CS 489/698 Big Data Infrastructure (Winter 2017) Week 2: MapReduce Algorithm Design (1/2) January 10, 2017 Jimmy Lin David R. Cheriton School of Computer Science University of Waterloo
MapReduce & YARN Hands-on Lab Exercise 1 Simple MapReduce program in Java
MapReduce & YARN Hands-on Lab Exercise 1 Simple MapReduce program in Java Contents Page 1 Copyright IBM Corporation, 2015 US Government Users Restricted Rights - Use, duplication or disclosure restricted
Announcements. Reading Material. Map Reduce. The Map-Reduce Framework 10/3/17. Big Data. CompSci 516: Database Systems
Announcements CompSci 516 Database Systems Lecture 12 - and Spark Practice midterm posted on sakai First prepare and then attempt! Midterm next Wednesday 10/11 in class Closed book/notes, no electronic
Topics covered in this lecture
9/5/2018 CS435 Introduction to Big Data - FALL 2018 W3.B.0 CS435 Introduction to Big Data 9/5/2018 CS435 Introduction to Big Data - FALL 2018 W3.B.1 FAQs How does Hadoop mapreduce run the map instance?
HADOOP FRAMEWORK FOR BIG DATA
HADOOP FRAMEWORK FOR BIG DATA Mr K. Srinivas Babu 1,Dr K. Rameshwaraiah 2 1 Research Scholar S V University, Tirupathi 2 Professor and Head NNRESGI, Hyderabad Abstract - Data has to be stored for further
MapReduce. U of Toronto, 2014
MapReduce U of Toronto, 2014 http://www.google.org/flutrends/ca/ (2012) Average Searches Per Day: 5,134,000,000 2 Motivation Process lots of data Google processed about 24 petabytes of data per day in