Ultimate Hadoop Developer Training

Transcription

1 First Edition Ultimate Hadoop Developer Training Lab Exercises edubrake.com

2 Hadoop Architecture 2 Following are the exercises that the student need to finish, as required for the module Hadoop Architecture

3 Lab Exercise 2.1 Your First Ever MapReduce Program Objectives What Do You Need? Your first MapReduce Job is a word count job. This job reads a specific input text file that is on your HDFS and counts every single word s word occurrences. It finally writes output to HDFS again. 1. Eclipse Project File : FirstEverMREclipseProject.zip After doing this exercise, the student will get an idea of how to run a Map Reduce job in Hadoop. The goal is not to understand how to write a Map Reduce program. The goal here is how to run a Map Reduce Job for the first time and get an experience of it. 2. DataSet : alice_in_wonderland.txt 5

4 Steps 1. First download the FirstEverMREclipseProject.zip file from your LMS. 3. Import the zip file in your eclipse. To do this, Click Import, as shown. 2. Fire-up Eclipse IDE on your linux desktop. 4. Select "Existing Projects in Workspace" option, as shown. Click Next. 6

5 6. Browse all the way to the folder where you have downloaded the zip file and select it. 5. Click "Browse" and select an archive file. 7

6 7. After your selection, the Project appears in the "Projects" section as shown. Make sure you selected it. Click "Finish". 9. You can also see the errors below in the "Problems" tab, as shown. Carefully observe the errors. Why do you think, the errors exist? This is because, the eclipse project that you have just imported, had its build path referring to the dependent jar files from a previous Hadoop version. Note that you could be running on the latest Hadoop version and there could be some jar file conflicts. 8. You have now successfully imported an eclipse project in to the Eclipse IDE. However, you may see a few errors in the project dependancies. You will notice a RED Exclamation mark on the project as shown below. 8

7 10. You can also see these errors when you go look at your Eclipse Project "Build Path" as shown. 11. Click on the "Libraries" tab and you can again see the RED exclamation marks indicating problems with those jar files. 1. Right Click on your Project. To fix these errors, you can select each jar file and replace it with the latest jar file found in the respective folders. However, that will be a very time consuming process. 2. Click on "Build Path" 3. Configure "Build Path" There is quick back door trick to fix all of these errors in oneshot. Your eclipse also has a sample "training" project that was shipped and distributed on this Virtual Machine. We can simply copy the settings from there on to this project. 9

8 12. Let me show how to do this. Open up your home directory '/home/cloudera/' on your linux VM. There is a "workspace" directory here. This is where all of the eclipse projects are stored. 14. Open up both the folders, as shown. 13. Inside this folder, you will see the 2 projects. "training" project is the one that came along with the Virtual Machine itself. "FirstEverMR" is the project you just imported. 10

9 15. To copy the "conf" folder, simply right click and hit "Copy" 16. After copying to "FirstEverMR", the directories should look like below. 18. After enabling "Show Hidden Files", you will see a few more files that has "." (dot) infront of them. Of these, you will simply take the ".classpath" file and copy it over to FirstEverMR. This is the back-door way of fixing all of the errors you have seen in the "Build Path" of your "FirstEverMR" project. So, go ahead and copy the ".classpath" file in to FirstEverMR project. 17. Now "Show Hidden Files" on your "training" folder 11

10 19. Copy & pastes in to "FirstEverMR" project. 21. Now go back to your eclipse project and simply "Refresh". 1. Right Click on your Project. 2. Hit "Refresh". 20. You will see an error mentioning that the file already exists. Click "Replace". 12

11 22. As you can see - 3. A Reducer class called "WordCountReducer.java" 1.The Red exclamation is now gone. We will not get into the details for now, but simply notice them that you are building a jar file that has your MapReduce programs in this form. Move on to the next step. 2.Also you will see no more errors. 24. Now, we will export a JAR file to run a MapReduce Job. To do this, 1. Right click on your project 23. Notice that you have 3 Java classes in your Project. 2. Hit "Export". 1. A driver class called "WordCountDriver.java" 2. A Mapper class called "WordCountMapper.java" 13

12 26. Give a name to your JAR file. In this case "FirstEverMR.jar" 25. Choose JAR file option under Java. 14

13 27.Click next here. 28. Click Finish. Export the JAR. You may see this message. Simply ignore as they are only warnings and not errors. Click OK. 29. Fire up Terminal Window. Ensure you are in your home directory. List all the files. pwd ls -lrt 15

14 30. So, this MR job requires an input file on HDFS. We will use the "alice_in_wonderland.txt" file as the input. Make sure this file exists on HDFS prior to the running your job. To check this run the following command: hadoop fs -ls /user/cloudera/datasets/ 31. Now, let us run the MapReduce job. To do this, you need to provide the following command. The command is of the format: hadoop jar {jar file} {Driver class} {Inputfile on HDFS} {Outputdir on HDFS} In our case, {jar file} ==> should be replaced with FirstEverMR.jar. {Driver class} ==> should be replaced with com.edubrake.firstmr.seeit.wordcountdriver {Inputfile on HDFS} ==> Should be replaced with a complete path to "alice_in_wonderland.txt" file on HDFS. {Outputdir on HDFS} ==> This should be where you want to write your final results on HDFS. In this case, it is "/user/cloudera/ joboutputs/firstevermroutput". So the command looks like below: hadoop jar FirstEverMR.jar com.edubrake.firstmr.seeit.wordcountdriver /user/cloudera/datasets/alice_in_wonderland.txt /user/cloudera/joboutputs/ FirstEverMROutput 16

15 32. Hit enter and your job runs for few minutes. Finally, you will see the Job Completion message as shown. 17

16 33. Now, to see the output, fire up Hue. Click on "File Browser" 35. Further navigate down in to the folders. 34.Hit on "joboutputs" folder. 36. Inside the "FirstEverMROutput", you will see 2 files. One indicates the "Success" flag for your job. And the other one has the final results in it. Click this file. 18

17 37. Here you go. Here is the final word count results. 38. Click on page#2 to see more results. Congratulations. You just finished running your First Ever MR program. This is a word count program. So, you processed an unstructured data like "Word" document, and then you took a count of every single word in the novel : "Alice In the Wonderland". For example, "Rabbit" appeared 45 times in the novel. CONGRATULATIONS. ASSIGNMENT : Upload a screenshot of this page to the Assignments folder found your Live Instructor Led Mentoring Course (Example: UHDP305). 19