Dr. Chuck Cartledge. 4 Mar. 2015

Transcription

1 CS-495/595 Pig (part 2) Lecture #8 Dr. Chuck Cartledge 4 Mar /23

2 Table of contents I 1 Miscellanea 2 The Book 3 Chapter 11 4 Assignment #3. 5 Project 6 Conclusion 7 References 2/23

3 Corrections and additions since last lecture. Completed grading Exam #1 3/23

4 Corrections and additions since last lecture. Mechanics of grading paper Graded the same question on all tests at the same time. Required parts: Undergraduates Top part of each question Graduates Top and bottom of 10 questions Extra credit: Undergraduates Any bottom part fo any question Graduates Questions marked as Extra Credit, or lowest scoring questions Details to follow. 4/23

5 Corrections and additions since last lecture. Points per question Some people read everything #2 Everyone likes languages #12 People like images #15 Repetition helps #3, #14 Blow-up to follow. 5/23

6 Corrections and additions since last lecture. Points per question (same image) Given data, you can make correlations. 6/23

7 Corrections and additions since last lecture. Histogram of the required points This is a histogram of the required points. These are the points from which the base grade will be establihsed. (If you are in the left tail then you might be in trouble.) 7/23

8 Corrections and additions since last lecture. Histogram of the extra credit points Interesting distribution. Not sure what it is showing, except that everyone got some extra credit. 8/23

9 Corrections and additions since last lecture. Histogram of the combined required and extra credit points Combining the required and the extra credit points creates problems. There are a number of people with over 100 points. Grading scale tops out at 100. Extra points will be cached and applied somehow at the end of the course. 9/23

10 Hadoop, The Definitive Guide Version 3 is specified in the syllabus [2] Version 4 came out in November 2015 We ll use Version 3 as much as possible 10/23

11 A cook book The correct Pig version. Some people have been using Apache Pig version I have tested the hadoop host with Apache Pig version I was running bash I executed: 1 export PIG HOME= /ClassWork/CS- 495/Pig/pig export PATH=$PATH:$PIG HOME/bin 3 export JAVA HOME= which java The latest information is available from: 11/23

12 A cook book Pig Latin Pig has two parts: A data flow language (Pig Latin) An execution environment Pig Latin Logical Plan Physical Plan MapReduce Plan Execution There are diagnostic tools to show each transformation. Image from [1]. Latest information is available from: ref2.html 12/23

13 A cook book A repeat of the image. Image from [1]. 13/23

14 Details Undergraduate portion The professor has a limited understanding of averages, and realizes that the number of values is important. She is interested in the costs in Virginia. Sample output: 3 Cardiology Meaning there are 3 records for a the Cardiology practitioner and the average cost is 120. This is sample data, not the final results. In reality, there will be more than just one practitioner. 14/23

15 Details Graduate portion The professor could move anywhere in the US. So she is also interested in the average cost in different states and US possessions that are not states. She wants to see the data displayed as a heat map of costs. Need to find average cost per reporting area, regardless of practitioner Need to color code the ordered values from cold to hot Need to plot area in the correct color Norfolk, VA is latitude, longitude , {Latitude, Longitude} {Y,X}, most plotting packages want {X,Y} 15/23

16 Details Heat maps A heat map is a graphical representation of data where the individual values contained in a matrix are represented as colors. [3] With only a small abuse of the definition of a heat map; instead of values in a matrix, we will use values in polygons. The polygons happen to be, more or less contiguous. Order the values from lowest to highest Divide the range into 10 parts (even distribution is OK) Color each polygon based which range bin its value falls into The choice of color range should make it obvious what is cold and what is hot. 16/23

17 Details Heat maps (continued) Heat maps libraries and routines are available in lots of different languages. Excel use of conditional formatting Java use of 2D Graphics plotting, JHeatChart, others Python heatmap.py Ruby heatmap R builtin Creating heatmaps that aren t strictly rectangles is a little more of a challenge. Using a predefined package probably won t get you the presentation that you need. 17/23

18 Details Undergraduate deliverables Only for the state of Virginia. r xy = A Venn diagram An X-Y plot of payments vs. billings A numerical correlation between the two data sets Correlation (r) will be 1.0 r 1.0 n i=1 (x i x)(y i y) n i=1 (x i x) 2 n i=1 (y i y) 2 Sample data shown. 18/23

19 Details Graduate deliverables For all the practitioners that receive Medicare payments. The same deliverables as the Undergraduates, except... For the entire intersection of the two data sets. Correlation (r) will be 1.0 r 1.0 Warning: we are assuming that there is a linear correlation between the two data sets. We will know if this is a valid assumption when we see the data. 19/23

20 Details Beware!! Just because data correlates mathematically, doesn t mean there is any causation. Math is pretty dumb. Garbage In, Garbage Out (GIGO) The site: has lots of silly examples. 20/23

21 Details The same image 21/23

22 What have we covered? Lots of time reviewing the exam and its results Details about installing and running Pig on CS s hadoop cluster High level Pig Latin commands Details about Assignment #3 Details about the project Next lecture: Hadoop book, Chapter 12. Enjoy your break. 22/23

23 References I [1] Mortar staff, Pig cheat sheet, [2] Tom White, Hadoop: The definitive guide, 3rd edition, O Reilly Media, Inc., [3] Wikipedia, Heat map wikipedia, the free encyclopedia, /23