Computer Science Department Picnic. FAQs. Topics. This material is developed based on, 8/27/2018 Week 2-A Sangmi Lee Pallickara

Transcription

1 Week 2- W FLL FLL 2018 W2..1 omputer Science epartment Picnic No ell-phones in the class. Welcome to the cademic year! If you need to use a laptop, please sit in the back row. I will ask you to turn off your laptop if it seems to be distracting to others. eet your faculty, department staff, and fellow students in a social setting. Food and drink will be provided. PT 1. LGE SLE T NLYTIS When: September 1st Time: 1pm 4pm Where: ity Park Shelter #7 omputer Science, olorado State University - FLL 2018 W FLL 2018 FQs pics P1 has been posted Introduction to apeduce apeduce esign Pattern I. Summarization Patterns September 18, 5:00P via anvas Individual submission (No team submission) W2..3 Team for term project 3-4 members ug. 30, 5:00P via anvas Looking for team members? Post your advertisement on Piazza! - FLL 2018 W FLL 2018 W2..5 This material is developed based on, nand ajaraman, Jure Leskovec, and Jeffrey Ullman, ining of assive atasets, ambridge University Press, hapter 2 Part 1. Large Scale ata nalytics Introduction to apeduce ownload this chapter from the S435 schedule page Hadoop: The definitive Guide, m White, eilly, 3rd Edition, 2014 apeduce esign Patterns, onald iner and dam Shook, eilly,

2 Week 2- - FLL 2018 W FLL 2018 W2..7 N data example national climate data center record Find the maximum temperature of a year (1900 ~ 1999 ) apeduce Example # USF weather station identifier # WN weather station identifier # observation date 0300 # observation time # latitude (degrees x 1000) # longitude (degrees x 1000) F # elevation (meters) V # wind direction (degrees) 1 # quality code N - FLL 2018 W2..8 The first entries for 1990 % ls raw/ 1990 head gz gz gz gz gz gz gz gz gz gz - FLL 2018 W2..9 nalyzing the data with Unix ols (1/2) program for finding the maximum recorded temperature by year from N weather records #!/ usr/ bin/ env bash for year in all/* do echo -ne ` basename $ year.gz ` \t gunzip -c $ year \awk '{ temp = substr( $ 0, 88, 5) + 0; q = substr( $ 0, 93, 1); if (temp!= 9999 && q ~ /[01459]/ && temp > max) max = temp EN { print max ' one - FLL 2018 W2..10 nalyzing the data with Unix ols (2/2) The script loops through the compressed year files Printing the year Processing each file using awk Extracts two fields ir temperature and the quality code heck if it is greater than the maximum value seen so far %./ max_temperature.sh FLL 2018 W2..11 esults? The complete run for the century took 42 minutes speed up the processing We need to run parts of the program in parallel Process different years in different processes What will be the problems? 2

3 Week 2- - FLL 2018 W2..12 hallenges ividing the work into equal-size pieces ata size per year? ombining the results from independent processes ombining results and sorting by year? You are still limited by the processing capacity of a single machine (the worst one)! - FLL 2018 W2..13 ap and educe apeduce works by breaking the processing into two phases The map phase The reduce phase Each phase has key-value pairs as input and output Programmers should specify Types of input/output key-values The map function The reduce function - FLL 2018 W2..14 Visualizing the way the apeduce works (1/3) Sample lines of input data N N N N N These lines are presented to the map function as the key-value pairs (0, N ) (106, N ) (212, N ) The keys are the line offsets within the file (optional) - FLL 2018 W2..15 Visualizing the way the apeduce works (2/3) The map function extracts the year and the air temperature and emit them as its output (1950, 0) (1950, 22) (1950, 11) (1949, 111) (1949, 78) This output key-value pairs will be sorted (by key) and grouped by key s passed to each reducer are NT sorted ur reduce function will see the following input: (1949, [111, 78]) (1950, [0, 22, 11]) - FLL 2018 W2..16 Visualizing the way the apeduce works (3/3) - FLL 2018 W2..17 omparison with other systems educe function iterates through the list and pick up the maximum reading (1949, 111) (1950, 22) This is the final output input r (0, ) (106, ) (212, ) (318, ) map (1950, 0)(1950, 22) (1950, -11) suffle (1949, [111,78]) (1950, [0,22,-11] reduce (1949, 111) (1950, 22) output 1949, , 22 PI vs. apeduce apeduce tries to collocate the data with the compute node ata access is fast ata is local! Volunteer computing vs. apeduce SETI@home Using donated PU time 3

4 Week 2- - FLL 2018 W FLL 2018 W2..19 ataset Network communication log file apeduce Example 3 Source estination There are two attributes and (IP1, IP2) : There was a communication from IP1 to IP2 - FLL 2018 W2..20 etrieve records with as a source IP heck each record and provide as output only those records containing as a source IP FLL 2018 W2..21 etrieve records with as the source ap function For each record, check the condition (source == and produce the key-value pair ( , record) How many reducers will be needed? educe function For the shuffled pairs (( , [record, record, record]) eturns ( ,,[record, record]) as the result of eliminating duplications - FLL 2018 W FLL 2018 W2..23 This material is developed based on, Part 1. Large Scale ata nalytics esign Pattern 1: Summarization Patterns nand ajaraman, Jure Leskovec, and Jeffrey Ullman, ining of assive atasets, ambridge University Press, hapter 2 ownload this chapter from the S435 schedule page Hadoop: The definitive Guide, m White, eilly, 3 rd Edition, 2014 apeduce esign Patterns, onald iner and dam Shook, eilly,

5 Week 2- - FLL 2018 W2..24 Why esign Patterns? ols for solving problems eusable and providing a general framework evelopers can spend less time figuring out how she/he is going to solve the problem - FLL 2018 W2..25 Summarization Patterns esign patterns that produce a top-level, summarized view of your data Helps you to glean insights not available from looking at a localized set of records alone Numerical summarizations ax/in, ount, verage, edian, Standard eviation Inverted index - FLL 2018 W FLL 2018 W2..27 s: Introduction esign Pattern 1: Summarization Patterns s Pattern escription General pattern for calculating aggregate statistical values over data Intent Group records together by a key field and calculate a numerical aggregate per group to get a top-level view of the larger data set pplicability Numerical summarizations should be used when both of the followings are true Numerical data or counting The data can be grouped by specific fields - FLL 2018 W2..28 s: Structure utput utput - FLL 2018 W2..29 Example 1. inimum, aximum and ount alculating the in, ax and ount of s per user USE-I ap function For each record, ap function produces the key-value pair (key, value) 1. esign your ap function educe function For the shuffled pairs (key, [a list of values]) eturns (key, value) as the result 2. esign your educe function 5

6 Week 2- - FLL 2018 W2..30 Example 1. inimum, aximum and ount alculating the in, ax and ount of per user USE-I ap function For each record, ap function produces the key-value pair (key, value) Input: <dummy_key, a list of values (a line of string)> utput: < USE-I, a data structure contains (in, ax, ount)> Functionality: kenizing the string, fill out the output data structure, emit an <key,value> pair educe function Input: (userid, [a list of data structures]) eturns (userid, [ax, in, ount]) as the result Functionality: Iterates over the list and calculate in, ax and ount. - FLL 2018 W2..31 apper ode public static class inaxountapper extends apper < bject, Text, Text, inaxounttuple > { // ur output key and value Writables private Text outuserid = new Text(); private inaxounttuple outtuple = new inaxounttuple(); public void map(bject key, Text value, ontext context) throws IException, InterruptedException { ap < String, String > parsed = transformxmlap(value.tostring()); // ssume we have developed a class, parse. // Grab the " field since it is what we are finding // the min and max value of String value = parsed.get( value ); // Grab the UserI since it is what we are grouping by String userid = parsed.get("userid"); // Set the minimum and maximum date values to the value outtuple.setin(value); outtuple.setax(value); // Set the comment count to 1 outtuple.setount(1); // Set our user I as the output key outuserid.set(userid); // Write out the hour and the average comment length context.write(outuserid, outtuple); - FLL 2018 W2..32 educer ode public static class inaxounteducer extends educer < Text, inaxounttuple, Text, inaxounttuple > { // ur output value Writable private inaxounttuple result = new inaxounttuple(); public void reduce(text key, Iterable < inaxounttuple > values, ontext context) throws IException, InterruptedException { // Initialize our result result.setin(null); result.setax(null); result.setount(0); int sum = 0; - FLL 2018 W2..33 educer ode --continued // Iterate through all input values for this key for (inaxounttuple val : values) { // If the value's min is less than the result's min // Set the result's min to value's if (result.getin()==null val.getin().compare(result.getin())<0) { result.setin(val.getin()); // If the value's max is more than the result's max // Set the result's max to value's if (result.getax()==null val.getax(). compare(result.getax())>0) { result.setax(val.getax()); // dd to our sum the count for value sum += val.getount(); //Set our count to the number of input values result.setount(sum); context.write(key, result); - FLL 2018 W2..34 Example 2. verage alculating an average of s per user USE-I ap function For each record, ap function produces the key-value pair (key, value) 1. esign your ap function 2. esign your educe function educe function For the shuffled pairs (key, [a list of values]) eturns (key, value) as the result - FLL 2018 W2..35 Example 2. verage alculating an verage of s per user USE-I ap function emit <useri, > educe function For the shuffled pairs (useri, [a list of values]) eturns (useri, average-value) as the result 6

7 Week 2- - FLL 2018 W FLL 2018 W2..37 Hadoop ombiner inimizes the data transferred between map and reduce tasks esign Pattern 1: Summarization Patterns ombiner Functions Users can specify a combiner function be run on the map output replace the map output with the combiner output Hadoop does NT guarantee how many times it will call combiner for a particular map output record - FLL 2018 W2..38 Example: Find the maximum temperature First map produces (1950, 0) (1950, 20) (1950, 10) Second map produces (1950, 25) (1950, 15) Input to the reduce function (1950, [0, 20, 10, 25, 15]) utput (1950, 25) - FLL 2018 W2..39 If a combiner finds the maximum temperature for each map output: First map produces (1950, 0) (1950, 20) (1950, 10) à (1950, 20) Second map produces (1950, 25) (1950, 15) à (1950, 25) Input to the reduce function (1950, [0, 20, 10, 25, 15]) à(1950, [20, 25]) utput (1950, 25)à(1950, 25) - FLL 2018 W2..40 an we use a combiner function for finding average value? - FLL 2018 W2..41 Example 2. verage alculating the verage of s per user USE-I ap function For each record, ap function produces the key-value pair (key, value) 1. esign your ap function 2. esign your ombiner function 3. esign your educe function educe function For the shuffled pairs (key, [a list of values]) eturns (key, value) as the result 7

8 Week 2- - FLL 2018 W2..42 Example 2. verage alculating the verage of s per user USE-I ap function emit <useri, (, count)> ombiner function alculate local average eturns (useri, (local-average-value, local-count)) as the result educe function For the shuffled pairs (useri, [a list of (local-average-values, local-count)]) eturns (useri, global-average-value) as the result - FLL 2018 W2..43 esign Pattern 1: Summarization Patterns Inverted Index - FLL 2018 W2..44 Inverted Index Generate Index from a data dataset to map from contents, such as words or numbers educes the amount of time to find related items Keyword based search, Web search, and document search e.g. dding Stackverflow links to each Wikipedia page that is reference in a Stackverflow comment - FLL 2018 W2..45 Structure * Example of an inverted index for document 1 and 2 (1, and 2) 1= olorado State University {olorado, State, University 2= University of olorado {University, of, olorado Inverted Index olorado {1, 2 State - {1 University {1, 2 of {2 - FLL 2018 W2..46 Inverted index of Stackverflow links to Wikipedia dding Stackverflow links to each Wikipedia page that is reference in a Stackverflow comment - FLL 2018 W2..47 Inverted index of Stackverflow links to Wikipedia ommenti E F G H I J omment omment-1 omment-2 omment-3 omment-4 omment-5 omment-6 omment-7 omment-8 omment-9 omment-10 ap function For each record, ap function produces the key-value pair (key, value) 1. esign your ap function 2. esign your educe function educe function For the shuffled pairs (key, [a list of values]) eturns (key, value) as the result 8

9 Week 2- - FLL 2018 W2..48 Inverted index of Stackverflow links to Wikipedia ommenti E F G H I J omment omment-1 omment-2 omment-3 omment-4 omment-5 omment-6 omment-7 omment-8 omment-9 omment-10 ap function If there is any link included in comment emit <link, commenti> educe function ll the commentis that referred same wikipedia page will be grouped together - FLL 2018 W2..49 apper ode public static class WikipediaExtractor extends apper < bject, Text, Text, Text > { private Text link = new Text(); private Text outkey = new Text(); public void map(bject key, Text value, ontext context) throws IException, InterruptedException { ap <String, String> parsed = PUtils.transformXmlap(value.toString()); // Grab the necessary XL attributes String txt = parsed.get("ody"); String posttype = parsed.get("posttypeid"); String row_id = parsed.get("id"); // if the body is null, or the post is a question (1), skip if (txt == null (posttype!= null && posttype.equals("1"))) { return; // Unescape the HTL because the S data is escaped. txt = StringEscapeUtils.unescapeHtml(txt.toLowerase()); link.set(getwikipediaul(txt)); outkey.set(row_id); context.write(link, outkey); - FLL 2018 W2..50 Questions? 9