Dr. Chuck Cartledge. 18 Feb. 2015

Size: px

Start display at page:

Download "Dr. Chuck Cartledge. 18 Feb. 2015"

Hope Anderson
5 years ago
Views:

1 CS-495/595 Pig Lecture #6 Dr. Chuck Cartledge 18 Feb /18

2 Table of contents I 1 Miscellanea 2 The Book 3 Chapter 11 4 Conclusion 5 References 2/18

3 Corrections and additions since last lecture. Completed grading Assignment #1. 3/18

4 Hadoop, The Definitive Guide Version 3 is specified in the syllabus [2] Version 4 came out in November 2015 We ll use Version 3 as much as possible 4/18

5 The essence of Pig. Pig provides a level of abstraction for dealing with large data sets. There are two major parts to the Pig ecosystem: The language (Pig Latin), The execution environment In a previous lecture, we touched on how a JOIN operation could be performed using MapReduce technology. Pig hides all that complexity. 5/18

6 Pig is not installed on the Hadoop cluster You will have to download it and install it. The tar.gz file is about 120 MB You ll need to download it, untar it, and test your installation There are some gotchas: 1 Pig is looking for the environment variable JAVA HOME to be set 2 Hadoop cluster runs tcsh vice BASH by default 3 Have to set JAVA HOME before Pig will run Some things are left as an exercise for the student. Section Installing and Running Pig on page 366 gives you information on where to download it from, how to install it, and how to test it. 6/18

Pig runs on top of Hadoop Pig can run in three different modes: Script: a file contains Pig Latin commands Grunt: an interactive mode Embedded: run from Java using the

7 Pig runs on top of Hadoop Pig can run in three different modes: Script: a file contains Pig Latin commands Grunt: an interactive mode Embedded: run from Java using the PigServer class The eclipse and NetBeans IDEs are supposed to have a Pig plug-ins. Initially you will tickle your Pig installation via grunt, later we will use scripts. 7/18

8 Language differences between Pig and RDBMS. Pig Latin is a data flow programming language... dataflow programming emphasizes the movement of data and models programs as a series of connections. Explicitly defined inputs and outputs connect operations, which function like black boxes. [3] SQL is a declarative programming language... declarative programming is a programming paradigm, a style of building the structure and elements of computer programs, that expresses the logic of a computation without describing its control flow. [4] 8/18

9 Schema differences between Pig and RDBMS. Pig allows optional schema definition at run time RDBMS store data in tables, and schemas are well known in advance Pig defaults to tab delimited fields, csv files processed via UDF. Pig reads data at program start (roughly) vs. data already in tables at start. 9/18

10 Data differences between Pig and RDBMS. Pig allows complex, nested data structures RDBMS tables are much flatter Pig Latin is generally more customizable than most SQL dialects. 10/18

11 Access time differences between Pig and RDBMS. Pig does not support random reads and writes to the data (WORM) RDBMS supports random access (indices, views, etc.) RDBMS are good for interactive, or low latency activities. Pig uses Hadoop and HDFS as its underpinnings and inherits all those strengths and weaknesses. 11/18

12 A simple example LOAD establishes where the data will be coming from AS defines the schema FILTER, GROUP similar to a SQL FOREACH processes each tuple MAX one of many functions DUMP output the data Nothing happens until a dataflow is defined and a trigger event occurs. 12/18

13 What are trigger events? Pig Latin is a data flow language, something has to start the data flow. Different commands act as triggers. DUMP a diagnostic statement STORE depends on when the statement is encountered Image from [1]. Pig Latin Logical Plan Physical Plan MapReduce Plan Execution 13/18

14 Image from [1]. 14/18

15 Image from [1]. 15/18

16 Image from [1]. 16/18

17 What have we covered? Covered the essence of Pig Pig runs on top of the Hadoop ecosystem and has all the strengths and limitations thereof Compared Pig to traditional RDBMS Pig is a dataflow programming language Next lecture: Hadoop book, Chapter 12 and return exam 17/18

18 References I [1] Prashanth Babu, Introduction to pig, [2] Tom White, Hadoop: The definitive guide, 3rd edition, O Reilly Media, Inc., [3] Wikipedia, Dataflow programming wikipedia, the free encyclopedia, [4], Declarative programming wikipedia, the free encyclopedia, http: //en.wikipedia.org/wiki/declarative_programming, /18

Dr. Chuck Cartledge. 4 Mar. 2015

Dr. Chuck Cartledge. 4 Mar. 2015 CS-495/595 Pig (part 2) Lecture #8 Dr. Chuck Cartledge 4 Mar. 2015 1/23 Table of contents I 1 Miscellanea 2 The Book 3 Chapter 11 4 Assignment #3. 5 Project 6 Conclusion 7 References 2/23 Corrections and