HOMEWORK 9. M. Neumann. Due: THU 5 APR PM. Getting Started SUBMISSION INSTRUCTIONS

Transcription

1 CSE427S HOMEWORK 9 M. Neumann Due: THU 5 APR PM Getting Started Update your SVN repository. When needed, you will find additional materials for homework x in the folder hwx. So, for the current assignment the folder is hw9. Hint: You can check your submission to the SVN repository by viewing wustl.edu/repositories/<yourwustlkey>/cse427s_sp18 in a web browser. SUBMISSION INSTRUCTIONS WRITTEN: all written work needs to be submitted electronically in pdf format 1 via GRADESCOPE provide the following information on every page of your pdf file: name student ID wustlkey (your wustlkey indicates the location (SVN repository) for your code submissions; without this information we will not be able to grade your code!) start every problem on a new page FOR GROUPS: make a group submission on GRADESCOPE and provide the following information for all group members on every page of your pdf file: names, student IDs, and location of code submissions (one student s wustlkey) 2 CODE: code needs to be submitted via SVN repository commit (detailed submission instructions are provided whenever code submissions are required) make sure to always use the required file name(s) and submission format(s) comment your code to receive maximum credit 1 Please, type your solutions or use clear hand-writing. If we cannot read your answer, we cannot give you credit nor will we be able to meet any regrade requests concerning your writing. 2 It is sufficient to commit code to one SVN repository. If you do not specify the repository for your group s code submission clearly, we will only check the first repository, sorting wustlkeys alphabetically. 1

2 Problem 1: PageRank (40%) In this problem you will do some more PageRank computations for the following graph: You can use youer SPARK code from Lab 9 or do the computations by hand. In order to recieve full credit show your computations or add the (relevant) SPARK operations to your written answer. (a) Compute the PageRank of each page (a, b, c) assuming no teleportation. 3 (b) Compute the PageRank of each page (a, b, c) assuming α = (c) Which data representation is more storage efficient: raw input data which has a (source-page destination-page) format or links which has a (source-page list of destination-pages) format State the storage requirement for a general webgraph in terms of the number of pages n and the number of links m. (d) Provide both data representations for the example graph above. Problem 2: Write a SPARK Program (30%) In this problem you will write a simple SPARK program in Python or Scala that counts the number of JPG requests in the web log data. The input data location (path and/or file name) should be passed into the program as an argument. You will find the weblogs data in HDFS under: /loudacre/weblogs Note that for this problem you will not need the interactive SPARK shell. So, before running your program, be sure to exit from the SPARK shell. Text editor: you can use gedit for example, which includes language-specific support (syntax highlighting ) for Python and Scala. 3 This is essentially Exercise MMDS (page 175) 4 THis is essentially Exercise MMDS (page 176). 2

3 Stub files: The stub files for this problem are provided in the countjpg_stubs folder in your SVN repository. [Scala only] To compile your Scala program, use the following folder: ~/training_materials/dev1/exercises/spark-application/countjpgs Copy your Scala program into the src/main/scala folder in this directory. Then, run the following two commands: $ sudo service archiva start $ mvn package This invokes Apache Maven to compile your program with all required dependencies. You should now find a folder called target and within this folder you should find your.jar file. Trouble-shooting: make sure you have a file called pom.xml in your countjpg folder. (a) Implement the CountJPGs program. Your implementation should load the file passed into the program, count the number of JPG requests, and display the count. (b) Run your SPARK application on the weblogs data locally. Provide the command to submit the job and report the number of JPG requests in your written answer. Where is the Driver program executed? Where does the processing happen? Where is the result stored? (c) Run your SPARK application on the weblogs data on the cluster using client mode. Provide the command to submit the job in your answer. Where is the Driver program executed? Where does the processing happen? Where is the result stored? (d) How many stages and tasks were executed for your CountJPGs job? You can retrieve this information from the Hue Job Browser using the bookmark in firefox (or going to URL (e) We could also run the job in cluster mode. Provide the command to submit the job in your answer. Where would the Driver program be executed? Where does the processing happen? Where would the result be stored? Provide your implementation for (a) by updating and committing the stub files in your SVN repository. To commit the CountJPGs.* file to your SVN repo run: $ svn commit -m hw9 problem 2 submission. Problem 3: Word-Co-Occurrence in SPARK (30%) Word co-occurrence features are used in many text mining and natural language processing tasks such as document categorization or plagiarism detection. In a previous assignemnt you implemented a MAPREDUCE program that counts the words-pairs in shakespeare that occur right next to each other. Now, you will implement this algorithm in SPARK. 3

4 (a) Implement the word co-occurrence SPARK program using the pairs approach. There are no stubs; name you program WordCo.py or WordCo.scala. Do not perform any preprocessing such as stemming or stop-word filtering. Comment your code to receive maximum credit. Use the following test input to test your implementation: No now is no now no time The output for this test input would be: is,no 1 no,now 2 no,time 1 now,is 1 now,no 1 Once, everything works run your job on the shakespeare data. To verify that your program runs correctly, compare the results with the ones you got from your MAPRE- DUCE implementation. (b) Check the output and provide the following statistics: How many different word pairs did your program produce? How often does the following word pair occur: sweet,lovers youngest,daugther your,king HINT: the grep command will be useful for this! (c) What are the jobs and stages of your SPARK application? Provide your implementation for (a) by adding and committing the.scala or.py file to the hw9 folder in your SVN repository. To add the CountJPGs.* file to your SVN repo before committing run: $ svn add WordCo.* $ svn commit -m hw9 problem 2 submission. Bonus Problem (5% up to a max. of 100%) - no group work! Write a review for this homework and store it in the file hw9_review.txt provided in your SVN repository (and commit your changes). This file should only include the review, no other information such as name, wustlkey, etc. Remember that you are not graded for the content of your review, solely it s completion. 4

5 You can only earn bonus points if you write at least 50 words. Bonus points are given to the owner of the repository only (no group work!). Submit your review in the file hw9_review.txt provided in the hw9 folder in your SVN repository. To commit the file run: $ svn commit -m hw9 review submission. 5