h p://

Transcription

1 B4M36DS2, BE4M36DS2: Database Systems 2 h p:// Prac cal Class 5 MapReduce Mar n Svoboda mar n.svoboda@fel.cvut.cz Charles University, Faculty of Mathema cs and Physics Czech Technical University in Prague, Faculty of Electrical Engineering

2 MapReduce Model Map func on Input: an input key-value pair (input record) Output: a set of intermediate key-value pairs Usually from a di erent domain Keys do not have to be unique (, ) list of (, ) Reduce func on Input: an intermediate key + a set of (all) values for this key Output: a possibly smaller set of values for this key From the same domain (, list of ) (, list of ) B4M36DS2, BE4M36DS2: Database Systems 2 Prac cal Class 5: MapReduce

3 Example: Word Frequency B4M36DS2, BE4M36DS2: Database Systems 2 Prac cal Class 5: MapReduce

4 Example: Word Frequency B4M36DS2, BE4M36DS2: Database Systems 2 Prac cal Class 5: MapReduce

5 Apache Hadoop Open-source framework Hadoop Common Hadoop Distributed File System (HDFS) Hadoop Yet Another Resource Nego ator (YARN) Hadoop MapReduce B4M36DS2, BE4M36DS2: Database Systems 2 Prac cal Class 5: MapReduce

6 Server Access Connect to our NoSQL server and on Linux PuTTY and WinSCP on Windows nosql.ms.m.cuni.cz:42222 Login and password sent by Change your ini al password (if not yet changed) B4M36DS2, BE4M36DS2: Database Systems 2 Prac cal Class 5: MapReduce

7 First Steps Get familiar with basic Hadoop commands Basic help for Hadoop commands Distributed le system commands Execu on of MapReduce jobs Browse the HDFS namespace B4M36DS2, BE4M36DS2: Database Systems 2 Prac cal Class 5: MapReduce

8 Word Count Job Create your working directory Make a copy of the sample java source le B4M36DS2, BE4M36DS2: Database Systems 2 Prac cal Class 5: MapReduce

9 Word Count Job Compile our Word Count implementa on B4M36DS2, BE4M36DS2: Database Systems 2 Prac cal Class 5: MapReduce

10 Word Count Job Create your HDFS working directories Prepare the sample input data B4M36DS2, BE4M36DS2: Database Systems 2 Prac cal Class 5: MapReduce

11 Word Count Job Run the prepared MapReduce job B4M36DS2, BE4M36DS2: Database Systems 2 Prac cal Class 5: MapReduce

12 Word Count Job Retrieve and explore the job result Clean the output HDFS directory B4M36DS2, BE4M36DS2: Database Systems 2 Prac cal Class 5: MapReduce

13 Bigger Word Count Job Run our MapReduce job on a bigger input le Create your HDFS directory Deploy a copy of the following input le Run the MapReduce job Retrieve and browse the result Clean the output HDFS directory B4M36DS2, BE4M36DS2: Database Systems 2 Prac cal Class 5: MapReduce

14 Useful Commands Addi onal MapReduce commands that might be helpful Lists iden ers of all the MapReduce jobs Prints status counters for a given MapReduce job Kills a par cular MapReduce job B4M36DS2, BE4M36DS2: Database Systems 2 Prac cal Class 5: MapReduce

15 NetBeans Project Launch NetBeans IDE and create a new project Select Java applica on as a project type Make local copies of the following Hadoop libraries Add both the libraries into the project Use Add JAR/Folder in the project context menu Replace the WordCount source le with the sample one Build the project to create a jar distribu on B4M36DS2, BE4M36DS2: Database Systems 2 Prac cal Class 5: MapReduce

16 Java Interface Mapper class Implementa on of the map func on Template parameters, types of input key-value pairs, types of intermediate key-value pairs Intermediate pairs are emi ed via B4M36DS2, BE4M36DS2: Database Systems 2 Prac cal Class 5: MapReduce

17 Java Interface Reducer class Implementa on of the reduce func on Template parameters, types of intermediate key-value pairs, types of output key-value pairs Output pairs are emi ed via B4M36DS2, BE4M36DS2: Database Systems 2 Prac cal Class 5: MapReduce

18 Inverted Index Implement an inverted index using MapReduce Use input les in Produce a list of : pairs for each word E.g.: Use to access input le names Use to process intermediate key-value pairs Use to iterate over map entries Compile, deploy and run the job B4M36DS2, BE4M36DS2: Database Systems 2 Prac cal Class 5: MapReduce

19 References HDFS: File System Shell commands h ps://hadoop.apache.org/docs/r3.1.1/ hadoop-project-dist/hadoop-common/filesystemshell.html MapReduce: tutorial h ps://hadoop.apache.org/docs/r3.1.1/ hadoop-mapreduce-client/hadoop-mapreduce-client-core/ MapReduceTutorial.html MapReduce: shell commands h ps://hadoop.apache.org/docs/r3.1.1/ hadoop-mapreduce-client/hadoop-mapreduce-client-core/ MapredCommands.html MapReduce: JavaDoc h ps://hadoop.apache.org/docs/r3.1.1/api/ B4M36DS2, BE4M36DS2: Database Systems 2 Prac cal Class 5: MapReduce