CMU MSP Intro to Hadoop

Transcription

1 CMU MSP Intro to Hadoop H. Seltman, April 3 and ) Carl had created an MSP virtual machine that you can download as an appliance for VirtualBox (also used for SAS University Edition). See location and important instructions on Canvas. The machine gives you a full Hadoop setup accessed from a Linux terminal on your PC or Mac. 2) One time setup a) Start Virtual Box and choose File / Preferences on the menu, then under Input, uncheck auto capture keyboard, and click OK. b) Use File / Import appliance, then point to Hadoop2.ova, and import with default settings 3) Session startup from VirtualBox a) Start Hadoop2 by clicking it and then clicking the Start arrow. What happens next varies a bit depending on whether you are doing a first time startup or restarting from a shut down state vs. restarting from a save the machine state state. Wait many seconds until you get to the Ubuntu desktop, which provides some Linux Services without using the command prompt. While you are waiting, you can click any little x boxes on the upper right corresponding to messages about mouse integration, etc. b) It is recommended that you use full screen mode, so that it appears that your whole desktop is the Linux machine. To do this, choose View / Full screen mode from the menu. From the information box write down the information from the sentence that tells you what your VirtualBox Host Key is. You can click the box to prevent seeing this box in the future. The box suggests that In the future, to exit full screen mode, hold down the host key and press F. To see the main menu bar, hold down the host key and press the home key. It appear that this is only true if you do not uncheck auto capture keyboard as suggested in step 2a. If you unchecked the box as recommended, get to the main menu by holding your mouse at the bottom of the screen, and select View / Full Screen to exit full screen. c) On the menu, go to Devices / Shared Clipboard, and select Bidirectional so that you can copy and paste text to and from the virtual machine and your computer. In Linux, highlight the text and right-click to get to copy. Use control-y as the paste key. 4) Introduction to the Virtual Linux Machine a) If you don t use the machine for a little while you will see a window with the header Student and the word password. Enter your password in the password box. b) The Ubuntu desktop is a bare bones GUI that lets you accomplish some things without using the command prompt. You should mostly use the command prompt to get more Linux practice. The icon bar on the left has the GUI functionality. c) To start a Firefox browser, so that you can look things up while working on your virtual Linux machine, click the Firefox world icon. If it is full screen, move your mouse to the top left and click the small rectangle inside the circle (next to the x and - circles) to make it a window. Resize and move the window using the mouse in the usual way. 1

2 d) Start a terminal window by clicking the icon that shows a terminal with >_. This is where you will do most of your work. You can move and resize your terminal. You can start multiple terminals. 5) First explorations in the terminal window a) To learn about your new machine, try the following Linux commands: i) whoami ii) echo $SHELL iii) echo $HOME iv) pwd v) ls vi) ls -a vii) ls / viii) ls -l Documents ix) echo $PATH b) Write a sample shell script i) Note that /home/student/bin is at the top of your path, but is not on your machine. Create that directory. ii) Type which bash. What does that tell you? iii) Start the emacs text editor with the argument (for emacs this is the file to edit) home/student/bin/ffind, being sure to use the suffix character that starts the program in a new shell so that you will still be able to work at your command prompt. iv) In the emacs window, note that there are clickable links for help on the bottom, and that you have a check box to Never show it again. Click Dismiss this startup screen. v) Enter the following code in the editor. #!/bin/bash # Find a file by a portion of the file name in the # current folder and all folders below. if [[ $# -ne 1 ]]; then echo Usage: $0 partialfilename fi find. -type f 2>/dev/null grep "$1" vi) Note that many features of the editor are available by menu or menu bar, and that most also have hotkey versions. Click Save, then use File / Quit on the menu. vii) Go to the root directory on the computer viii) Use ffind find file names containing semaphore. 2

3 c) Setup a share folder to facilitate sharing files across machines i) Make a folder on your PC or Mac that will be used to share files across machines. Let s call in share (in some directory of your choice). ii) On the VirtualBox menu, choose Machine/Settings. Choose Shared folders on the left, then click the folder + icon ( Adds new shared folder. ). Under Folder Path navigate to the folder you want to share on your computer. Choose a Folder Name, e.g., share, and check Auto-mount and Make Permanent, but not Read-only. Click OK twice. iii) Create a share folder on your VirtualBox, e.g., /home/student/share. iv) Connect the two folders with the Linux command sudo mount -t vboxsf share /home/student/share. (If you called your Folder Name pointer to your Mac/Windows machine folder something other than share, use that instead.) v) Now any file put in either /home/student/share on the VirtualBox or the Folder Path on your PC/Mac will also appear in the other location. d) Starting Hadoop and Stopping i) Hadoop is implemented as a Java process. Check which Java processes are running with jps. You should only see Jps itself. ii) Start Hadoop and Yarn on your machine by clicking the green arrow Start Hadoop button on the left side of your Ubuntu console (ignoring the deprecated message ). It is best to use the Stop Hadoop button below the Start Hadoop button to stop the Hadoop and yarn processes before shutting down your virtual machine, because sometimes it does not restart correctly when you re-open your virtual machine. iii) Verify that all of the components are working by running the jps command again. You should see the NameNode and DataNode which are the Hadoop processes, and NodeManager and ResourceManager which are the Yarn processes. Now you know how to check if Hadoop is running on your machine. iv) To stop Hadoop, click the red Stop Hadoop button. The UNIX command jps can be used to verify that all of the Hadoop Java processes have been shut down. There are reports that you may run into difficulty if you do not stop Hadoop before closing VirtualBox. To close VirtualBox close the window or use the menu item File/Close. Either saving the machine state or powering off the machine are OK. e) Getting to know the Hadoop file system i) The Hadoop file system runs alongside the UNIX file system. We use the hdfs (hadoop distributed file system) command to manipulate the Hadoop file system and transfer files between systems. Hadoop analyses only run on the Hadoop file system. ii) Think of the hdfs command as the prefix to a set of standard commands that apply to the Hadoop file system. iii) Most of the commands of hdfs are technical things you will likely never need to use. The main command is dfs. The syntax is hdfs dfs -subcommand [subcommand options]. You can see valid the subcommands here. Many of the subcommands mimic standard UNIX file commands, e.g., ls, mkdir, etc., but not cd. 3

4 iv) On your version of the Hadoop machine, your home directory on the Hadoop side is /users/student (as opposed to /home/student on the UNIX side). Your working directory is your (unchangeable) home directory. v) Here are the most useful sub commands for the hdfs dfs command: (1) ls, e.g., hdfs dfs -ls /user/student (2) mkdir and rmdir, e.g., hdfs dfs -mkdir test (3) put <UNIX(s)> <Hadoop>, i.e., transfer one or more files from UNIX to Hadoop, e.g., echo abc > letters.txt; echo def > letters2.txt ls hdfs dfs -put letters.txt letters2.txt /user/student [or hdfs dfs -put letters.txt letters2.txt. ] hdfs dfs -ls (4) get <Hadoop(s)> UNIX, i.e., transfer one or more files from Hadoop to UNIX, e.g., hdfs dfs -get /user/student/letters*.txt ~/test (5) appendtofile <UNIX> <Hadoop>, e.g., hdfs dfs -appendtofile letters2.txt /user/student/letters.txt (6) cat, e.g., hdfs dfs -cat letters.txt (7) Additional commands (with obvious syntax) include rm, mv, cp, tail (but not head), and du (disk usages). f) Appending to ~/.bashrc to make some commands be auto-run i) The UNIX file ~/.bashrc contains important commands that are entered into your bash shells as they start up. You can add to that file (typically at the end of the file). ii) We will be running various examples stored in a jar (Java Archive) file that comes with Hadoop. First use which hadoop to find where the hadoop binary lives on your system. Then replace bin/hadoop with share/hadoop and ls at this new location. This shows the folders for various hadoop programs. Look in the mapreduce folder and note the name of the.jar file containing the examples. Make a shell variable called mrex that holds the location and name of the map-reduce examples jar. We will need this soon. iii) Use emacs ~/.bashrc & to edit the file. Add lines to set the alias dfs="hdfs dfs" and the shell variable mrex. Save the file. Close your terminal and restart it. Use echo $mrex to show that everything worked. Consider periodically adding other entries to ~/.bashrc. g) First Hadoop project: word counting i) Create UNIX folder /home/student/ex1 and cd to it. ii) Make a simple useless input dataset with ps -aux > psaux.txt iii) Create folder /user/student/ex1 on the hadoop file system using the syntax hdfs dfs - mysubcommand myoptions. Note that on the Hadoop side ex1 points to /user/student/ex1. iv) Copy the psaux.txt input text file to ex1 on the Hadoop file system using hdfs dfs - put with the appropriate source and destination. 4

5 v) Open Firefox from the Ubuntu icon, and click on the quick links bar link to Hadoop jobs. vi) Back in the terminal window, start the Hadoop job to use map-reduce to count the words in psaux.txt. We will use the hadoop command with the first argument jar to tell it that we are using the option to supply the map-reduce program we want to run in the form of compiled java byte-code in a Java ARchive. As you might expect, the parameter after hadoop jar is the name of the jar, and you can now use your shell variable $mrex rather than typing that long file path. As you also might expect, what follows hadoop jar $mrex is any options to the examples jar. In this case the parameters are first the name of one of the many examples in the jar (wordcount now), followed by any specific parameters for that example (input file and output directory). So the full command you need is hadoop jar $mrex wordcount ex1/psaux.txt ex1/out. Note that the program you run is on the normal UNIX file system, while the input data and output files are on the Hadoop file system. (The hadoop command loads the code on the Hadoop machines.) vii) Various lines of information show on the screen (hopefully no errors). Quickly go to the Firefox window and press F5 to update the information. You may see part of what is happening if you are quick enough. You can click links to get more info. Play a bit. viii) Return to the UNIX window, assure that you have a new prompt, and examine the ancillary data (in the terminal window) about your first map-reduce analysis. ix) Examine your actual Hadoop results using hdfs dfs to see the contents of the ex1/out folder. You should see a file called _SUCCESS with size 0, which indicates success, and a file called part-r with a size of about 10KB containing your results. Use the hdfs dfs -get command to get that result file into your UNIX file space. (You can use just a period for the destination to indicate here with the same file name. x) Use the plain old UNIX head and tail commands to examine the copy of your results that you pulled back into the UNIX file system. Can you write a piped command to count how many words have greater than one occurrence? xi) Use ls -lrt psaux.txt to see the size of psaux.txt. Use ps -aux >> psaux.txt to put more text into the input file. Re-check the size. Think carefully, then carry out all steps to re-run the analysis on the new input. Note that you will need to delete the old output files or use a different directory name. Rename the old output file on the UNIX side to a different name before getting the new output file. Compare the number of words between the two files, the number of non-singlicate words, and the counts for a few specific high frequency words. 5

6 h) Second Hadoop project: word counting from your own Java program i) Imagine that you knew how to write your own Java code. Create /home/student/ex2 on UNIX and /user/student/ex2 on Hadoop. Download and skim WordCount.java. (One way to get the file is wget What other ways do you know?) Here is how to byte-compile the program to (three) class files, and then combine those into a jar file called wc.jar : As one-time setup type these UNIX commands: echo $JAVA_HOME echo $PATH export PATH=${JAVA_HOME}/bin:${PATH} echo $PATH echo $HADOOP_CLASSPATH export HADOOP_CLASSPATH=${JAVA_HOME}/lib/tools.jar Compile using: hadoop com.sun.tools.javac.main WordCount.java ls -lrt Combine the classes into a Jar using: jar cf wc.jar WordCount*.class ls -lrt ii) Now think about how you could run the word count task using your jar file instead of the one from $mrex. Try it. 6

7 i) Third Hadoop project: word counting in Python i) Open the reference, in your browser. ii) Create folder ex3 and cd to it. Open mapper.py in emacs. Enter the first code block from the reference page, and the save the file (but do not close emacs). #!/usr/bin/env python import sys # input comes from STDIN (standard input) for line in sys.stdin: # remove leading and trailing whitespace line = line.strip() # split the line into words words = line.split() # increase counters for word in words: # write the results to STDOUT (standard output); # what we output here will be the input for the # Reduce step, i.e. the input for reducer.py # # tab-delimited; the trivial word count is 1 print '%s\t%s' % (word, 1) iii) Start a second file in emacs by using the menu option File / Visit new file. Enter reducer.py in the text area given. Enter the code (from the reference) and save. iv) Note that you can switch between files using the Buffers menu item or ctrl-x then b. Also note that the line at the bottom of the screen containing the current file name shows ** if the buffer has unsaved text. You can leave emacs open in case you need to fix errors. v) Try the following test code in UNIX: chmod u+x mapper.py echo "spam spam 602 spam eggs"./mapper.py chmod u+x reducer.py echo "spam spam 602 spam eggs"./mapper.py sort./reducer.py ps -aux grep -v stringprefs./mapper.py sort./reducer.py ps -aux grep -v stringprefs > psaux.txt vi) Make /user/student/ex3 in the Hadoop file system. Upload psaux.txt to Hadoop. vii) List the files in /usr/local/hadoop/share/hadoop/tools/lib restricting to those filenames containing str. This is the Java ARchive that we can use to run our Python programs. Make a shell variable called sjar that hold the full path and name of that file (and put it in.bashrc ). viii) Run the Python map/reduce program on the Hadoop system as follows: hadoop jar $sjar -file mapper.py -mapper mapper.py -file reducer.py -reducer reducer.py -input ex3/psaux.txt -output ex3/out ix) Check the results 7

8 8