HOMEWORK 8. M. Neumann. Due: THU 29 MAR PM. Getting Started SUBMISSION INSTRUCTIONS

Size: px

Start display at page:

Download "HOMEWORK 8. M. Neumann. Due: THU 29 MAR PM. Getting Started SUBMISSION INSTRUCTIONS"

Loreen May
5 years ago
Views:

1 CSE427S HOMEWORK 8 M. Neumann Due: THU 29 MAR PM Getting Started Update your SVN repository. When needed, you will find additional materials for homework x in the folder hwx. So, for the current assignment the folder is hw8. Hint: You can check your submission to the SVN repository by viewing wustl.edu/repositories/<yourwustlkey>/cse427s_sp18 in a web browser. SUBMISSION INSTRUCTIONS WRITTEN: all written work needs to be submitted electronically in pdf format 1 via GRADESCOPE provide the following information on every page of your pdf file: name student ID wustlkey (your wustlkey indicates the location (SVN repository) for your code submissions; without this information we will not be able to grade your code!) start every problem on a new page FOR GROUPS: make a group submission on GRADESCOPE and provide the following information for all group members on every page of your pdf file: names, student IDs, and location of code submissions (one student s wustlkey) 2 CODE: code needs to be submitted via SVN repository commit (detailed submission instructions are provided whenever code submissions are required) make sure to always use the required file name(s) and submission format(s) comment your code to receive maximum credit 1 Please, type your solutions or use clear hand-writing. If we cannot read your answer, we cannot give you credit nor will we be able to meet any regrade requests concerning your writing. 2 It is sufficient to commit code to one SVN repository. If you do not specify the repository for your group s code submission clearly, we will only check the first repository, sorting wustlkeys alphabetically. 1

2 Preparation 1. Complete Lab 8: Data Ingest with FLUME as you will need the data. 2. If you have not yet executed this command, run it in the command line: $ ~/training_materials/dev1/scripts/training_setup_dev1.sh Problem 1: ETL with SPARK (45%) In this problem you will parse a set of activation records in XML format to extract the account numbers and model names. Spark is commonly used for ETL (Extract/Transform/Load) operations. Sometimes data is stored in line-oriented records, like the web logs in the previous exercise, but sometimes the data is in a multi-line format that must be processed as a whole file. In this exercise you will practice working with file-based instead of line-based formats. You can do this problem using the spark shell. Include all required commands in your written answers. The data for this problem can be found in your local file system under: ~/training_materials/dev1/data/activations Review the activations data. We will be interested in the following information: accountnumber and model. Each XML file contains data for all the devices activated by customers during a specific month. Put the data into the loudacre folder in HDFS. HINT: consult the API documentation for RDD operations on the Spark API page you bookmarked in Lab 8. Review the list of available methods of the RDD class. (a) Create an RDD from the activations dataset, where the entire content of each XML file is a single RDD element. What information is stored in the first value, what in the second value of the tuples in the resulting RDD? (b) Which SPARK function can be used to map the information in the second value to separate RDD elements? Create a new RDD of separate activation records. You can use the getactivations function provided in the stub file in the hw8 folder in your SVN repository. getactivations takes an XML string, parses it, and returns a collection of XML records. (c) Extract the account number and device model for each activation record, and save the list to a text file formatted as account_number:model. Store the file in /loudacre/ account-models in HDFS. You can use the provided getaccount and getmodel functions to find the account number and device model values from the activation record. Include all required pyspark or Scala commands in your written answer in hw8.pdf. 2

3 Problem 2: Joining Datasets with SPARK (40%) In this problem you will explore the Loudacre web server log files using key-value Pair RDDs. After adding some new data containing Loudacre user account information to HDFS you will join the two RDDs. You can do this problem using the spark shell. Include all required commands in your written answers. After completing Lab 8 (FLUME) you will find the weblogs data in HDFS under: /loudacre/weblogs HINT: You will be reducing and joining large datasets, which can take a lot of time. You may wish to develop your spark commands using a smaller dataset, consisting of only a few of the web log files, rather than all of them. One way of getting a sample is to use a wildcard while loading the data as for instance textfile("/loudacre/weblogs/*6") would include only filenames ending with the digit 6. Use the full dataset to produce the results to be included in your submission! (a) Load the weblogs data into an RDD. How many records are there in the weblogs data? Filter the RDD to only contain HTML requests. How many records for HTML requests are in the weblogs data? (b) Create a Pair RDD with the user ID as the key, and the integer 1 as the value. The user ID is the third field in each line. Then, sum the values for each user ID. Your RDD data will be similar to: (userid,5) (userid,7) (userid,2)... (c) Determine how many users visited the site for each frequency. That is, how many users visited once, twice, three times and so on. How many users visited the site once, 7 times, and 12 times? Preparation: We will perform another simulation process to ingest data into HDFS similar to the one we did in the lab. Now, we will use SQOOP to get data from an RDMS and put it into HDFS. First, we have to create the database in mysql. Enter the following command in the terminal: $ mysql -u root -p when prompted enter the password cloudera. Then enter: mysql> CREATE DATABASE loudacre; mysql> quit 3

4 Put the data into this database: $ cd ~/training_materials/dev1/data $ mysql -u root -p loudacre < loudacre.sql when prompted enter the password cloudera. Now, import the accounts data from the SQL database into the loudacre\accounts folder in HDFS using SQOOP by executing the following command in the terminal: $ sqoop import \ --connect jdbc:mysql://localhost/loudacre \ --username root --password cloudera \ --table accounts \ --target-dir /loudacre/accounts \ --null-non-string '\\N' (d) Create an RDD based on the accounts data consisting of key/value-array pairs: (userid, [values...]). The results will look sth like this: (userid1,[userid1, :04:08,\N,Cheryl,West, 4905 Olive Street,San Francisco,CA,...]) (userid2,[ userid2, :05:07,\N,Elizabeth,Kerns, 4703 Eva Pearl Street,Richmond,CA,...]) (userid3,[userid3, :12:12, :42:36,Melissa, Roman,3539 James Martin Circle,Oakland,CA,...]) (e) Join this Pair RDD with the set of user-id/hit-count pairs calculated in part (a). The result will look something like this: (userid1,([userid1, :04:08,\N,Cheryl,West, 4905 Olive Street,San Francisco,CA,...],4)) (userid2,([ userid2, :05:07,\N,Elizabeth,Kerns, 4703 Eva Pearl Street,Richmond,CA,...],8)) (userid3,([userid3, :12:12, :42:36,Melissa, Roman,3539 James Martin Circle,Oakland,CA,...],1)) Display the user ID, hit count, and first name (3rd value) and last name (4th value) for the first 5 elements. The result for the above input would look like this: userid1 4 Cheryl West userid2 8 Elizabeth Kerns userid3 1 Melissa Roman HINT: you can use for-loops in the SPARK shell. Include all required pyspark or Scala commands in your written answer in hw8.pdf. 4

5 Problem 3: SPARK Job Execution (15%) (a) Describe what pipelining means in the context of a SPARK job execution. What is its benefit? (b) Give an example of two operations that can be pipelined together. (c) Give an example of two operations that cannot be pipelined together. Bonus Problem (5% up to a max. of 100%) - no group work! Write a review for this homework and store it in the file hw8_review.txt provided in your SVN repository (and commit your changes). This file should only include the review, no other information such as name, wustlkey, etc. Remember that you are not graded for the content of your review, solely it s completion. You can only earn bonus points if you write at least 50 words. Bonus points are given to the owner of the repository only (no group work!). Submit your review in the file hw8_review.txt provided in the hw8 folder in your SVN repository. To commit the file run: $ svn commit -m hw8 review submission. 5

HOMEWORK 9. M. Neumann. Due: THU 8 NOV PM. Getting Started SUBMISSION INSTRUCTIONS

HOMEWORK 9. M. Neumann. Due: THU 8 NOV PM. Getting Started SUBMISSION INSTRUCTIONS CSE427S HOMEWORK 9 M. Neumann Due: THU 8 NOV 2018 4PM Getting Started Update your SVN repository. When needed, you will find additional materials for homework x in the folder hwx. So, for the current assignment