Apache Pig. Craig Douglas and Mookwon Seo University of Wyoming

Size: px

Start display at page:

Download "Apache Pig. Craig Douglas and Mookwon Seo University of Wyoming"

Florence Cummings
5 years ago
Views:

1 Apache Pig Craig Douglas and Mookwon Seo University of Wyoming

2 Why were they invented? Apache Pig Latin and Sandia OINK are scripting languages that interface to HADOOP and MR- MPI, respectively. Both allow users to quickly prototype Map- Reduce applications and avoid Java, C, C++, Fortran, and other legacy programming languages. OINK approximates the aroma of Pig. 2

3 What is the benefit of Pig Latin Pig Latin is simple to understand data flow language for analysts familiar with scripting languages. Fast and iterative language with MapReduce compilation engine. Rich, multivalued, nested operations performed on large data sets. Pig scripts are automatically converted into mapreduce jobs by the pig interpreter. 3

4 Pig Latin complex data formats Tuple: enclosed by (), items separated by ", Nonempty tuple: (item1,item2, ) Empty tuple is valid: () Bag: enclosed by {}, tuples separated by "," Nonempty bag: {code}{(tuple1),(tuple2), }{code} Empty bag is valid: {} Map: enclosed by [], items separated by ",", key and value separated by "# Nonempty map: [key1#value1,key2#value2, ] Empty map is valid: [] 4

5 Simple Pig Latin A LOAD statement to read data A series of transformation statements A DUMP or STORE statement to see or save output: A = LOAD students.txt USING PigStorage() {name:chararray, year:int, gpa:float}; B = FOREACH A GENERATE name; DUMP A; DUMP B; 5

6 Pig Latin DUMP Example DUMP A; (Fooey,2010,2.6F) (Bar,2011,3.7F) (Foo,2011,4.0F) DUMP B; (Fooey) (Bar) (Foo) 6

7 Simple Pig Latin FILTER to work with tuples or rows of data. FOREACH to work with columns of data. GROUP operator to group data in a single relation COGROUP to group 2 or more relations. inner JOIN or outer JOIN to join 2 or more relations. 7

8 Simple Pig Latin UNION operator to merge the contents of 2 or more relations. SPLIT operator to partition a relation into 2 or more relations. Debugging commands DESCRIBE operator displays relations. EXPLAIN operator to view logical, physical, or map- reduce operators. ILLUSTRATE operator to single step statements. 8

9 Pig Latin data types int long float double chararray bytearray 9

10 Pig dynamic invokers DEFINE Can be used to invoke a built in static Java function subject to the following: Accepts no arguments, or Accepts combination of strings, ints, longs, doubles, floats, or arrays with these same types. Returns a string, an int, a long, a double, or a float. 10

11 Pig Latin DEFINE example DEFINE UrlDecode InvokeForString('java.net.URLDecoder.decode', 'String String'); encoded_strings = LOAD 'encoded_strings.txt' as (encoded:chararray); decoded_strings = FOREACH encoded_strings GENERATE UrlDecode(encoded, 'UTF- 8'); 11

12 Pig Latin eval functions - AVG AVG(expression) computes the average of the numeric values in a single- column bag. - A = LOAD 'student.txt' AS (name:chararray, term:chararray, gpa:float); - B = GROUP A BY name; - C = FOREACH B GENERATE A.name, AVG(A.gpa); 12

13 Pig Latin AVG example - DUMP B; (John,{(John,fl,3.9F),(John,wt,3.7F),(John,sp,4.0F),(John,sm,3.8F)}) (Mary,{(Mary,fl,3.8F),(Mary,wt,3.9F),(Mary,sp,4.0F),(Mary,sm,4.0 F)}) - DUMP C; ({(John),(John),(John),(John)}, ) ({(Mary),(Mary),(Mary),(Mary)}, ) 13

14 Pig Latin eval functions - CONCAT CONCAT (expression, expression) concatenates two expressions of identical type. - A = LOAD 'data' as (f1:chararray, f2:chararray, f3:chararray); - X = FOREACH A GENERATE CONCAT(f2,f3); 14

15 Pig Latin CONCAT example - DUMP A; (apache,open,source) (hadoop,map,reduce) (pig,pig,latin) - DUMP X; (opensource) (mapreduce) (piglatin) 15

16 Pig Latin eval functions - COUNT COUNT(expression) computes the number of elements in a bag. This requires GROUP statement. - A = LOAD data.txt' AS (f1:int, f2:int, f3:int); - B = GROUP A BY f1; - C = FOREACH B GENERATE COUNT(A); 16

17 Pig Latin COUNT example - DUMP A; (4, 2, 1) (8, 3, 4) (4, 3, 3) - DUMP B; (4,{(4,2,1),(4,3,3)}) (8,{(8,3,4)}) - DUMP C; (2L) (1L) 17

18 Pig Latin eval functions COUNT_STAR(expression) computes the number of elements in a bag. This requires GROUP statement. COUNT_STAR includes NULL values in the count computation. - X = FOREACH B GENERATE COUNT_STAR(A); 18

19 Pig Latin eval functions - DIFF DIFF (expression, expression) compares two fields in a tuple. - A = LOAD 'bag_data' AS (B1:bag{T1:tuple(t1:int,t2:int)},B2:bag{T2:tuple(f1:int,f2 :int)}); - X = FOREACH A DIFF(B1,B2); 19

20 Pig Latin DIFF example - DUMP A; ({(8,9),(0,1)},{(8,9),(1,1)}) ({(2,3),(4,5)},{(2,3),(4,5)}) ({(6,7),(3,7)},{(2,2),(3,7)}) - DISCRIBE A; a: {B1: {T1: (t1: int,t2: int)},b2: {T2: (f1: int,f2: int)}} - DUMP X; ({(0,1),(1,1)}) ({}) ({(6,7),(2,2)}) 20

21 Pig Latin eval functions IsEmpty(expression) checks if a bag or map is empty. MAX(expression) or MIN(expression) compute the maximum or minimum of the numeric values or chararrays in a single- column bag. Both require a preceding GROUP statement. 21

22 Pig Latin eval functions - SIZE SIZE(expression) computes the number of elements based on any Pig data type. - A = LOAD 'data' as (f1:chararray, f2:chararray, f3:chararray); - B = FOREACH A GENERATE SIZE(f1); 22

23 Pig Latin SIZE example - DUMP A; (apache,open,source) (hadoop,map,reduce) (pig,pig,latin) - DUMP B; (6L) (6L) (3L) 23

24 Pig Latin eval functions - SUM SUM(expression) computes the sum of the numeric values in a single- column bag. It requires a preceding GROUP statement. - A = LOAD 'data' AS (owner:chararray, pet_type:chararray, pet_num:int); - B = GROUP A BY owner; - X = FOREACH B GENERATE group,sum(a.pet_num); 24

25 Pig Latin SUM example - DUMP A; (Alice,turtle,1) (Alice,goldfish,5) (Alice,cat,2) (Bob,dog,2) (Bob,cat,2) - DUMP X; (Alice,8L) (Bob,4L) 25

26 Pig Latin eval functions - TOKENIZE TOKENIZE(expression [, 'field_delimiter']) splits a string and outputs a bag of words. - A = LOAD 'data' AS (f1:chararray); - X = FOREACH A GENERATE TOKENIZE(f1); 26

27 Pig Latin TOKENIZE example - DUMP A; (Here is the first string.) (Here is the second string.) (Here is the third string.) - DUMP X; ({(Here),(is),(the),(first),(string.)}) ({(Here),(is),(the),(second),(string.)}) ({(Here),(is),(the),(third),(string.)}) 27

28 Pig Latin I/O functions LOAD/STORE support gzip and bzip2 file compression. A = load students.txt.gz ; Store A into sorted.txt.bz2 ; BinStorage() loads and stores data in machine- readable format. JsonLoader( [ schema ] ) and JsonStorage( ) load and store JSON data. 28

29 Pig Latin I/O functions PigDump() stores data in human readable tuples using UTF- 8 format. PigStorage( [field_delimiter], ['options'] ) loads and stores data as structured text files. TextLoader() loads unstructured data in UTF- 8 format. 29

30 Pig Latin math functions Simple numeric ABS, EXP, LOG, LOG10, SQRT, CBRT Rounding CEIL, FLOOR, ROUND Trigonometry ACOS, ASIN, ATAN, COS, COSH, SIN, SINH, TAN, TANH Random numbers RAND 30

31 Pig Latin string functions Find in a string INDEXOF, LAST_INDEX_OF, REPLACE REGEX_EXTRACT, REGEX_EXTRACT_ALL Substrings SUBSTRING, STRSPLIT, TRIM Conversion LCFIRST, LOWER, UCFIRST, UPPER 31

32 Pig Latin convert to functions TOTUPLE(expression [, expression...]) converts one or more expressions to type tuple. TOBAG(expression [, expression...]) converts one or more expressions to type bag. TOMAP(key- expression, value- expression [, key- expression, value- expression...]) converts key/value expression pairs into a map. TOP(topN,column,relation) returns the top- n tuples from a bag of tuples. 32

33 Pig Latin user defined functions Written in Java L Use REGISTER operator: REGISTER myudfs.jar; Example usage: A = LOAD 'student_data' AS (name: chararray, age: int); B = FOREACH A GENERATE myudfs.upper(name); DUMP B; 33

34 Pig Latin user defined eval function package myudfs; import java.io.ioexception; import org.apache.pig.evalfunc; import org.apache.pig.data.tuple; public class UPPER extends EvalFunc<String> { public String exec(tuple input) throws IOException { if (input == null input.size() == 0) return null; try { String str = (String)input.get(0); return str.touppercase(); } catch(exception e){ throw new IOException("Caught exception processing input row ", e); } } } 34

35 Pig Latin user defined aggregate functions Aggregate functions are another common type of eval function and are applied to grouped data: A = LOAD 'student_data' AS (name: chararray, age: int); B = GROUP A BY name; C = FOREACH B GENERATE group, COUNT(A); DUMP C; COUNT extends EvalFunc<Long> using the Algebraic Interface. 35

36 Pig Latin function interfaces An aggregate function is an eval function that takes a bag and returns a scalar value. Interfaces include public interface Algebraic { public String getinitial(); public String getintermed(); public String getfinal(); } Accumulator similar 36

37 Pig Latin function interfaces Filter functions are eval functions that return a boolean value. 0 false Anything else true IsEmpty is an example that takes a tuple and returns either 0 or 1. Throws an exception if the data is not a Tuple. Much more complicated functions can be constructed using complex interfaces and simulation. 37

38 OINK Small collection of commands: set mr input, output, include clear, echo, print, variable shell, log if, jump, label, next mypgm map reduce etc. 38

39 OINK Like Pig Latin, MR- MPI s scripting language is based on the functions in the MR- MPI API. If you know how to program MR- MPI in C++, learning OINK is easy and obvious. If you only know HADOOP, then there is a steep learning curve. For highly parallel computational science that uses MapReduce, learn MR- MPI and OINK. 39

40 Final thoughts Pig Latin and OINK are incompatible with each other and developed independently. L OINK encourages new, user contributed commands that can be added to the scripting language. HADOOP has an extensive and large user base. MR- MPI is much faster for computational science applications. 40

Index. Symbols A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

Index. Symbols A B C D E F G H I J K L M N O P Q R S T U V W X Y Z Symbols A B C D E F G H I J K L M N O P Q R S T U V W X Y Z Symbols + addition operator?: bincond operator /* */ comments - multi-line -- comments - single-line # deference operator (map). deference operator