Pig Latin. Dominique Fonteyn Wim Leers. Universiteit Hasselt

Size: px
Start display at page:

Download "Pig Latin. Dominique Fonteyn Wim Leers. Universiteit Hasselt"

Transcription

1 Pig Latin Dominique Fonteyn Wim Leers Universiteit Hasselt

2 Pig Latin is an English word game in which we place the rst letter of a word at the end and add the sux -ay. Pig Latin becomes igpay atinlay banana becomes anana-bay What does this have to do with computer sciences?

3 Will the real Pig Latin please stand up? Pig Latin is a language developed by Yahoo! designed for ad-hoc data analysis. Combination of high-level declarative querying (SQL style) low-level procedural programming (map-reduce)

4 First example Find the average pagerank of high-pagerank URLs for each suciently large category in a table urls (url, category, pagerank). SQL: SELECT category, AVG(pagerank) FROM urls WHERE pagerank > 0.2 GROUP BY category HAVING COUNT(*) > 10 6

5 First example (2) Find the average pagerank of high-pagerank URLs for each suciently large category in a table urls (url, category, pagerank). PIG LATIN: good_urls = FILTER urls BY pagerank > 0.2; groups = GROUP good_urls BY category; big_groups = FILTER groups BY COUNT(good_urls) >10 6 ; output = FOREACH big_groups GENERATE category, AVG(good_urls.pagerank);

6 First example (3) Pig Latin programs are sequences of steps Each step carries out a single data transformation Transformations are fairly high-level e.g. ltering, grouping, aggregation low-level manipulations are unnecessary Writing Pig Latin programs is similar to specifying a query execution plan and thus easier for programmers to understand and control how their data is being processed.

7 Presentation Overview 1 Features and Motivation 2 Pig Latin, the Language 3 Implementation 4 Practical Notes 5 Copresentation

8 Presentation Overview 1 Features and Motivation 2 Pig Latin, the Language 3 Implementation 4 Practical Notes 5 Copresentation

9 Dataow Language Pig Latin is a high-level data ow language. The user species a sequence of steps. Each step performs only a single, high-level data transfomation. It is not necessary that the operations be executed in the order of that sequence. Usage of high-level relational algebra-style primitives like group and filter allows traditional database optimizations.

10 Dataow Language (2) Find the URLs of all pages that are classied as spam, but have a high pagerank. spam_urls = FILTER urls BY isspam(url); culprit_urls = FILTER spam_urls BY pagerank > 0.8; isspam() is a user-dened function and may be expensive not the most ecient method

11 Dataow Language (3) More ecient would be, Find the URLs of all pages that are classied as spam, but have a high pagerank. culprit_urls = FILTER urls BY pagerank > 0.8; spam_urls = FILTER spam_urls BY isspam(url); 1 get all high pagerank pages rst 2 invoke isspam() only on these high pagerank pages This optimization can be done automatically by the system.

12 Quick Start and Interoperability Pig Latin is designed to support ad-hoc data analysis. queries can be run directly over data les the user must provide a function to parse the content into tuples Similar for output.

13 Quick Start and Interoperability (2) Stored schemas are strictly optional. Schema information can be provided on the y, or even not at all. Because...

14 Quick Start and Interoperability (2) Stored schemas are strictly optional. Schema information can be provided on the y, or even not at all. Because... PIGS EAT ANYTHING!

15 Nested Data Model Programmers often think in terms of nested data structures. Example: Capture information of each pig in a collection of pig farms. Map<pigFarmId, Set<pig>>

16 Nested Data Model (2) Databases allow only at tables, i.e., columns are atomic elds. pig_farms: (pigfarmid, pigfarmname,...) pigs: (pigid, pigname,...) pig_info: (pigfarmid, pigid)

17 Nested Data Model (3) Pig Latin oers a exible, fully nested data model and allows complex, non-atomic data types as eld or table. Some reasons for having a nested data model: closer to how programmers think and thus much more natural to them than normalization allows programmers to easily write a rich set of user-dened functions

18 UDFs as First-Class Citizens Custom processing is a signicant part of analysing data. Pig Latin has extensive support for user-dened functions (UDFs). All aspects of Pig Latin processing can be customized through the use of UDFs. Input and output of UDFS in Pig Latin follow the nested data model. A UDF can take non-atomic parameters as input, and also output non-atomic values.

19 UDFs as First-Class Citizens (2) Example: Find the top 10 URLs according to pagerank for each category. groups = GROUP urls BY category; output = FOREACH groups GENERATE category, top10(urls); Here, top10() is a UDF that accepts a set of URLs, and outputs a set containing the top 10 URLs by pagerank for that group. The nal output contains non-atomic elds: there is a tuple for each category, and one of the elds is the set of top 10 URLs.

20 UDFs as First-Class Citizens (3) Practical notes UDFs are written in Java. Yahoo! is building support for other languages, including C/C++, Perl (Erlpay) and Python (Ythonpay).

21 Parallellism Required Processing web-scale data requires parallelism. Pig Latin includes a small set of carefully chosen primitives that can be easily parallelized. Other primitives that do not lend themselves to ecient parallel evaluation have been deliberately excluded. They can still be carried out by UDFs. The user is then responsible for how ecient his programs are and whether they will be parallelized.

22 Debugging Environment Getting a data processing program right usually takes many iterations. With web-scale data, a single iteration can take many minutes or hours. The usual run-debug-run cycle can be very slow and inecient. Pig comes with a novel interactive debugging environment that generates concise example data tables illustrating the output of each step of the user's program.

23 Debugging Environment (2)

24 Presentation Overview 1 Features and Motivation 2 Pig Latin, the Language 3 Implementation 4 Practical Notes 5 Copresentation

25 Data Model Pig uses a rich, yet simple data model consisting of 4 types: Atom Tuple Bag Map

26 Data Model (3)

27 Specifying Input Data The rst step is to specify what the input data les are, and how the le contents are to be deserialized. We use the LOAD command. We assume the input le is a bag, i.e., it contains a sequence of tuples.

28 Specifying Input Data (2) queries = LOAD 'query_log.txt' USING myload() AS (userid, querystring, timestamp); input le is query_log.txt input is converted into tuples by using a custom myload deserializer loaded tuples have 3 elds named userid, querystring and timestamp

29 Specifying Input Data (3) queries = LOAD 'query_log.txt' USING myload() AS (userid, querystring, timestamp); Both the USING and AS clause are optional. If no deserializer is specied, Pig uses a default one that expects a plain text, tab-delimited le. If no schema is used, elds must be referred to by position instead of by name. For readability it is desirable to include schemas.

30 Per-tuple Processing The FOREACH command applies some processing to each tuple of a data set. expanded_queries = FOREACH queries GENERATE userid, expandquery(querystring); Each tuple of the bag queries should be processed independently to produce an output tuple. The rst eld is the userid eld of the input tuple. The second eld is the result of applying the UDF expandquery() to the querystring eld of the input tuple.

31 Per-tuple Processing (2) The GENERATE clause can be followed by a list of expressions. A common expression type is attening. The FLATTEN keyword eliminates nesting by extracting the elds of the tuples in the bag, and making them elds of the tuple being output by GENERATE. This removes one level of nesting. expanded_queries = FOREACH queries GENERATE userid, FLATTEN(expandQuery(queryString));

32 Per-tuple Processing (3)

33 Per-tuple Processing (4)

34 Discarding Unwanted Data The FILTER command discards all data that is not of interest. Example: Get rid of bot trac. real_queries = FILTER queries BY userid neq 'bot'; comparison operators: ==,!=, <, >,... (numbers) eq, neq (strings) logical operators: AND, OR, NOT

35 Discarding Unwanted Data (2) We can use UDFs as well. Example: Get rid of bot trac. real_queries = FILTER queries BY NOT isbot(userid);

36 Getting Related Data Together It is often necessary to group together related tuples from one or more data sets. This is done with the COGROUP command. Example: we have 2 data sets specied results: (querystring, url, position) revenue: (querystring, adslot, amount)

37 Getting Related Data Together (2) Example: group together all search result data and revenue data for the same query string grouped_data = COGROUP results BY querystring, revenue BY querystring; Output: grouped_data: (group, results, revenue) rst eld is the group identier, the value of querystring each next eld is a bag, one for each input being cogrouped and is named the same as the alias of that input

38 Getting Related Data Together (3) Example: join all search result data and revenue data for the same query string join_result = JOIN results BY querystring, revenue BY querystring; What is the dierence with COGROUP?

39 Getting Related Data Together (4)

40 Getting Related Data Together (5) When there is only one data set, we use GROUP. grouped_revenue = GROUP results BY querystring;

41 Getting Related Data Together - Summarized When there is one data set GROUP When there are two or more data sets JOIN COGROUP JOIN equals a COGROUP followed by FLATTEN

42 Map-Reduce in Pig Latin The GROUP and FOREACH statements allow us to express a map-reduce program. map_result = FOREACH input GENERATE FLATTEN(map(*)); key_groups = GROUP map_result BY $0; output = FOREACH key_groups GENERATE reduce(*);

43 Other Commands Other commands are, UNION CROSS ORDER DISTINCT

44 Nested Operations Each command operates over one or more bags or tuples as input. When we have nested bags within tuples, we can nest some commands within a FOREACH command. grouped_revenue = group revenue BY querystring; query_revenue = FOREACH grouped_revenue { top_slot = FILTER revenue BY adslot eq 'top'; GENERATE querystring, SUM(top_slot.amount), SUM(revenue.amount); };

45 Asking for Output Write results to le with STORE STORE query_revenues INTO 'myoutput' USING mystore();

46 Presentation Overview 1 Features and Motivation 2 Pig Latin, the Language 3 Implementation 4 Practical Notes 5 Copresentation

47 Implementation Pig Latin is implemented by the Pig sytem. Programs are compiled into map-reduce jobs and executed by Hadoop. It is an open source project in the Apache incubator.

48 Building a Logical Plan The Pig interpreter rst parses the Pig Latin commands and veries that the referred input les and bags are valid. e.g. when entering c = COGROUP a BY..., b BY..., Pig veries that a and b are already dened It builds a logical plan for each dened bag.

49 Building a Logical Plan (2) When dening a new bag, the logical plan is constructed by combining the logical plans for the input bags, and the current command. e.g. when entering c = COGROUP a BY..., b BY..., The logical plan for c consists of a cogroup command with the plans for a and b as input.

50 Building a Logical Plan (3) When the logical plans are constructed, no processing is carried out. Processing is only triggered when invoking a STORE command. Then the logical plan is compiled into a physical plan and executed. This lazy style of execution permits in-memory pipelining and other optimizations.

51 Map-Reduce Plan Compilation Map-reduce provides the ability to do a large-scale group by the map tasks assign keys for grouping the reduce tasks process a group at a time

52 Presentation Overview 1 Features and Motivation 2 Pig Latin, the Language 3 Implementation 4 Practical Notes 5 Copresentation

53 Practical Notes More information can be found at pig.apache.org. Pig is a project under active development. New features are to be added: safe optimizer user interfaces external functions unied environment

54 Presentation Overview 1 Features and Motivation 2 Pig Latin, the Language 3 Implementation 4 Practical Notes 5 Copresentation

Introduction to Database Systems CSE 444. Lecture 22: Pig Latin

Introduction to Database Systems CSE 444. Lecture 22: Pig Latin Introduction to Database Systems CSE 444 Lecture 22: Pig Latin Outline Based entirely on Pig Latin: A not-so-foreign language for data processing, by Olston, Reed, Srivastava, Kumar, and Tomkins, 2008

More information

Lecture 23: Supplementary slides for Pig Latin. Friday, May 28, 2010

Lecture 23: Supplementary slides for Pig Latin. Friday, May 28, 2010 Lecture 23: Supplementary slides for Pig Latin Friday, May 28, 2010 1 Outline Based entirely on Pig Latin: A not-so-foreign language for data processing, by Olston, Reed, Srivastava, Kumar, and Tomkins,

More information

Section 8. Pig Latin

Section 8. Pig Latin Section 8 Pig Latin Outline Based on Pig Latin: A not-so-foreign language for data processing, by Olston, Reed, Srivastava, Kumar, and Tomkins, 2008 2 Pig Engine Overview Data model = loosely typed nested

More information

"Big Data" Open Source Systems. CS347: Map-Reduce & Pig. Motivation for Map-Reduce. Building Text Index - Part II. Building Text Index - Part I

Big Data Open Source Systems. CS347: Map-Reduce & Pig. Motivation for Map-Reduce. Building Text Index - Part II. Building Text Index - Part I "Big Data" Open Source Systems CS347: Map-Reduce & Pig Hector Garcia-Molina Stanford University Infrastructure for distributed data computations Map-Reduce, S4, Hyracks, Pregel [Storm, Mupet] Components

More information

Pig Latin: A Not-So-Foreign Language for Data Processing

Pig Latin: A Not-So-Foreign Language for Data Processing Pig Latin: A Not-So-Foreign Language for Data Processing Christopher Olston, Benjamin Reed, Utkarsh Srivastava, Ravi Kumar, Andrew Tomkins (Yahoo! Research) Presented by Aaron Moss (University of Waterloo)

More information

Introduction to Data Management CSE 344

Introduction to Data Management CSE 344 Introduction to Data Management CSE 344 Lecture 27: Map Reduce and Pig Latin CSE 344 - Fall 214 1 Announcements HW8 out now, due last Thursday of the qtr You should have received AWS credit code via email.

More information

1.2 Why Not Use SQL or Plain MapReduce?

1.2 Why Not Use SQL or Plain MapReduce? 1. Introduction The Pig system and the Pig Latin programming language were first proposed in 2008 in a top-tier database research conference: Christopher Olston, Benjamin Reed, Utkarsh Srivastava, Ravi

More information

Data-intensive computing systems

Data-intensive computing systems Data-intensive computing systems High-Level Languages University of Verona Computer Science Department Damiano Carra Acknowledgements! Credits Part of the course material is based on slides provided by

More information

Data Management in the Cloud PIG LATIN AND HIVE. The Google Stack. Sawzall. Map/Reduce. Bigtable GFS

Data Management in the Cloud PIG LATIN AND HIVE. The Google Stack. Sawzall. Map/Reduce. Bigtable GFS Data Management in the Cloud PIG LATIN AND HIVE 191 The Google Stack Sawzall Map/Reduce Bigtable GFS 192 The Hadoop Stack SQUEEQL! ZZZZZQL! EXCUSEZ- MOI?!? Pig/Pig Latin Hive REALLY!?! Hadoop HDFS At your

More information

Pig Latin: A Not-So-Foreign Language for Data Processing

Pig Latin: A Not-So-Foreign Language for Data Processing Pig Latin: A Not-So-Foreign Language for Data Processing Christopher Olston Yahoo! Research Ravi Kumar Yahoo! Research Benjamin Reed Yahoo! Research Andrew Tomkins Yahoo! Research Utkarsh Srivastava Yahoo!

More information

Motivation: Building a Text Index. CS 347 Distributed Databases and Transaction Processing Distributed Data Processing Using MapReduce

Motivation: Building a Text Index. CS 347 Distributed Databases and Transaction Processing Distributed Data Processing Using MapReduce Motivation: Building a Text Index CS 347 Distributed Databases and Transaction Processing Distributed Data Processing Using MapReduce Hector Garcia-Molina Zoltan Gyongyi Web page stream 1 rat dog 2 dog

More information

Outline. MapReduce Data Model. MapReduce. Step 2: the REDUCE Phase. Step 1: the MAP Phase 11/29/11. Introduction to Data Management CSE 344

Outline. MapReduce Data Model. MapReduce. Step 2: the REDUCE Phase. Step 1: the MAP Phase 11/29/11. Introduction to Data Management CSE 344 Outline Introduction to Data Management CSE 344 Review of MapReduce Introduction to Pig System Pig Latin tutorial Lecture 23: Pig Latin Some slides are courtesy of Alan Gates, Yahoo!Research 1 2 MapReduce

More information

Going beyond MapReduce

Going beyond MapReduce Going beyond MapReduce MapReduce provides a simple abstraction to write distributed programs running on large-scale systems on large amounts of data MapReduce is not suitable for everyone MapReduce abstraction

More information

The Pig Experience. A. Gates et al., VLDB 2009

The Pig Experience. A. Gates et al., VLDB 2009 The Pig Experience A. Gates et al., VLDB 2009 Why not Map-Reduce? Does not directly support complex N-Step dataflows All operations have to be expressed using MR primitives Lacks explicit support for processing

More information

The Hadoop Stack, Part 1 Introduction to Pig Latin. CSE Cloud Computing Fall 2018 Prof. Douglas Thain University of Notre Dame

The Hadoop Stack, Part 1 Introduction to Pig Latin. CSE Cloud Computing Fall 2018 Prof. Douglas Thain University of Notre Dame The Hadoop Stack, Part 1 Introduction to Pig Latin CSE 40822 Cloud Computing Fall 2018 Prof. Douglas Thain University of Notre Dame Three Case Studies Workflow: Pig Latin A dataflow language and execution

More information

Pig A language for data processing in Hadoop

Pig A language for data processing in Hadoop Pig A language for data processing in Hadoop Antonino Virgillito THE CONTRACTOR IS ACTING UNDER A FRAMEWORK CONTRACT CONCLUDED WITH THE COMMISSION Apache Pig: Introduction Tool for querying data on Hadoop

More information

Systems Infrastructure for Data Science. Web Science Group Uni Freiburg WS 2014/15

Systems Infrastructure for Data Science. Web Science Group Uni Freiburg WS 2014/15 Systems Infrastructure for Data Science Web Science Group Uni Freiburg WS 2014/15 Hadoop Evolution and Ecosystem Hadoop Map/Reduce has been an incredible success, but not everybody is happy with it 3 DB

More information

Hadoop is supplemented by an ecosystem of open source projects IBM Corporation. How to Analyze Large Data Sets in Hadoop

Hadoop is supplemented by an ecosystem of open source projects IBM Corporation. How to Analyze Large Data Sets in Hadoop Hadoop Open Source Projects Hadoop is supplemented by an ecosystem of open source projects Oozie 25 How to Analyze Large Data Sets in Hadoop Although the Hadoop framework is implemented in Java, MapReduce

More information

MAP-REDUCE ABSTRACTIONS

MAP-REDUCE ABSTRACTIONS MAP-REDUCE ABSTRACTIONS 1 Abstractions On Top Of Hadoop We ve decomposed some algorithms into a map- reduce work9low (series of map- reduce steps) naive Bayes training naïve Bayes testing phrase scoring

More information

Click Stream Data Analysis Using Hadoop

Click Stream Data Analysis Using Hadoop Governors State University OPUS Open Portal to University Scholarship All Capstone Projects Student Capstone Projects Spring 2015 Click Stream Data Analysis Using Hadoop Krishna Chand Reddy Gaddam Governors

More information

Scaling Up Pig. Duen Horng (Polo) Chau Assistant Professor Associate Director, MS Analytics Georgia Tech. CSE6242 / CX4242: Data & Visual Analytics

Scaling Up Pig. Duen Horng (Polo) Chau Assistant Professor Associate Director, MS Analytics Georgia Tech. CSE6242 / CX4242: Data & Visual Analytics http://poloclub.gatech.edu/cse6242 CSE6242 / CX4242: Data & Visual Analytics Scaling Up Pig Duen Horng (Polo) Chau Assistant Professor Associate Director, MS Analytics Georgia Tech Partly based on materials

More information

Innovatus Technologies

Innovatus Technologies HADOOP 2.X BIGDATA ANALYTICS 1. Java Overview of Java Classes and Objects Garbage Collection and Modifiers Inheritance, Aggregation, Polymorphism Command line argument Abstract class and Interfaces String

More information

Pig Latin Reference Manual 1

Pig Latin Reference Manual 1 Table of contents 1 Overview.2 2 Pig Latin Statements. 2 3 Multi-Query Execution 5 4 Specialized Joins..10 5 Optimization Rules. 13 6 Memory Management15 7 Zebra Integration..15 1. Overview Use this manual

More information

this is so cumbersome!

this is so cumbersome! Pig Arend Hintze this is so cumbersome! Instead of programming everything in java MapReduce or streaming: wouldn t it we wonderful to have a simpler interface? Problem: break down complex MapReduce tasks

More information

Overview. : Cloudera Data Analyst Training. Course Outline :: Cloudera Data Analyst Training::

Overview. : Cloudera Data Analyst Training. Course Outline :: Cloudera Data Analyst Training:: Module Title Duration : Cloudera Data Analyst Training : 4 days Overview Take your knowledge to the next level Cloudera University s four-day data analyst training course will teach you to apply traditional

More information

Hadoop Development Introduction

Hadoop Development Introduction Hadoop Development Introduction What is Bigdata? Evolution of Bigdata Types of Data and their Significance Need for Bigdata Analytics Why Bigdata with Hadoop? History of Hadoop Why Hadoop is in demand

More information

CC PROCESAMIENTO MASIVO DE DATOS OTOÑO 2018

CC PROCESAMIENTO MASIVO DE DATOS OTOÑO 2018 CC5212-1 PROCESAMIENTO MASIVO DE DATOS OTOÑO 2018 Lecture 4: Apache Pig Aidan Hogan aidhog@gmail.com HADOOP: WRAPPING UP 0. Reading/Writing to HDFS Creates a file system for default configuration Check

More information

Andrew Pavlo, Erik Paulson, Alexander Rasin, Daniel Abadi, David DeWitt, Samuel Madden, and Michael Stonebraker SIGMOD'09. Presented by: Daniel Isaacs

Andrew Pavlo, Erik Paulson, Alexander Rasin, Daniel Abadi, David DeWitt, Samuel Madden, and Michael Stonebraker SIGMOD'09. Presented by: Daniel Isaacs Andrew Pavlo, Erik Paulson, Alexander Rasin, Daniel Abadi, David DeWitt, Samuel Madden, and Michael Stonebraker SIGMOD'09 Presented by: Daniel Isaacs It all starts with cluster computing. MapReduce Why

More information

Pig on Spark project proposes to add Spark as an execution engine option for Pig, similar to current options of MapReduce and Tez.

Pig on Spark project proposes to add Spark as an execution engine option for Pig, similar to current options of MapReduce and Tez. Pig on Spark Mohit Sabharwal and Xuefu Zhang, 06/30/2015 Objective The initial patch of Pig on Spark feature was delivered by Sigmoid Analytics in September 2014. Since then, there has been effort by a

More information

CSE 444: Database Internals. Lecture 23 Spark

CSE 444: Database Internals. Lecture 23 Spark CSE 444: Database Internals Lecture 23 Spark References Spark is an open source system from Berkeley Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing. Matei

More information

Index. Symbols A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

Index. Symbols A B C D E F G H I J K L M N O P Q R S T U V W X Y Z Symbols A B C D E F G H I J K L M N O P Q R S T U V W X Y Z Symbols + addition operator?: bincond operator /* */ comments - multi-line -- comments - single-line # deference operator (map). deference operator

More information

Beyond Hive Pig and Python

Beyond Hive Pig and Python Beyond Hive Pig and Python What is Pig? Pig performs a series of transformations to data relations based on Pig Latin statements Relations are loaded using schema on read semantics to project table structure

More information

Dr. Chuck Cartledge. 18 Feb. 2015

Dr. Chuck Cartledge. 18 Feb. 2015 CS-495/595 Pig Lecture #6 Dr. Chuck Cartledge 18 Feb. 2015 1/18 Table of contents I 1 Miscellanea 2 The Book 3 Chapter 11 4 Conclusion 5 References 2/18 Corrections and additions since last lecture. Completed

More information

Big Data for Oracle DBAs. Arup Nanda

Big Data for Oracle DBAs. Arup Nanda Big Data for Oracle DBAs Arup Nanda fcrawler.looksmart.com - - [26/Apr/2000:00:00:12-0400] "GET /contacts.html HTTP/1.0" 200 4595 "-" "FAST-WebCrawler/2.1-pre2 (ashen@looksmart.net)" fcrawle fcrawler.looksmart.com

More information

Blended Learning Outline: Cloudera Data Analyst Training (171219a)

Blended Learning Outline: Cloudera Data Analyst Training (171219a) Blended Learning Outline: Cloudera Data Analyst Training (171219a) Cloudera Univeristy s data analyst training course will teach you to apply traditional data analytics and business intelligence skills

More information

Declarative MapReduce 10/29/2018 1

Declarative MapReduce 10/29/2018 1 Declarative Reduce 10/29/2018 1 Reduce Examples Filter Aggregate Grouped aggregated Reduce Reduce Equi-join Reduce Non-equi-join Reduce 10/29/2018 2 Declarative Languages Describe what you want to do not

More information

Introduction to Apache Pig ja Hive

Introduction to Apache Pig ja Hive Introduction to Apache Pig ja Hive Pelle Jakovits 30 September, 2014, Tartu Outline Why Pig or Hive instead of MapReduce Apache Pig Pig Latin language Examples Architecture Hive Hive Query Language Examples

More information

Generating Continuation Passing Style Code for the Co-op Language

Generating Continuation Passing Style Code for the Co-op Language Generating Continuation Passing Style Code for the Co-op Language Mark Laarakkers University of Twente Faculty: Computer Science Chair: Software engineering Graduation committee: dr.ing. C.M. Bockisch

More information

Scaling Up Pig. Duen Horng (Polo) Chau Assistant Professor Associate Director, MS Analytics Georgia Tech. CSE6242 / CX4242: Data & Visual Analytics

Scaling Up Pig. Duen Horng (Polo) Chau Assistant Professor Associate Director, MS Analytics Georgia Tech. CSE6242 / CX4242: Data & Visual Analytics http://poloclub.gatech.edu/cse6242 CSE6242 / CX4242: Data & Visual Analytics Scaling Up Pig Duen Horng (Polo) Chau Assistant Professor Associate Director, MS Analytics Georgia Tech Partly based on materials

More information

TEXT MINING INTRO TO PYTHON

TEXT MINING INTRO TO PYTHON TEXT MINING INTRO TO PYTHON Johan Falkenjack (based on slides by Mattias Villani) NLPLAB Dept. of Computer and Information Science Linköping University JOHAN FALKENJACK (NLPLAB, LIU) TEXT MINING 1 / 23

More information

Research challenges in data-intensive computing The Stratosphere Project Apache Flink

Research challenges in data-intensive computing The Stratosphere Project Apache Flink Research challenges in data-intensive computing The Stratosphere Project Apache Flink Seif Haridi KTH/SICS haridi@kth.se e2e-clouds.org Presented by: Seif Haridi May 2014 Research Areas Data-intensive

More information

Big Data Hadoop Stack

Big Data Hadoop Stack Big Data Hadoop Stack Lecture #1 Hadoop Beginnings What is Hadoop? Apache Hadoop is an open source software framework for storage and large scale processing of data-sets on clusters of commodity hardware

More information

Hadoop ecosystem. Nikos Parlavantzas

Hadoop ecosystem. Nikos Parlavantzas 1 Hadoop ecosystem Nikos Parlavantzas Lecture overview 2 Objective Provide an overview of a selection of technologies in the Hadoop ecosystem Hadoop ecosystem 3 Hadoop ecosystem 4 Outline 5 HBase Hive

More information

Introduction to Python - Part I CNV Lab

Introduction to Python - Part I CNV Lab Introduction to Python - Part I CNV Lab Paolo Besana 22-26 January 2007 This quick overview of Python is a reduced and altered version of the online tutorial written by Guido Van Rossum (the creator of

More information

Apache Pig. Craig Douglas and Mookwon Seo University of Wyoming

Apache Pig. Craig Douglas and Mookwon Seo University of Wyoming Apache Pig Craig Douglas and Mookwon Seo University of Wyoming Why were they invented? Apache Pig Latin and Sandia OINK are scripting languages that interface to HADOOP and MR- MPI, respectively. http://pig.apache.org

More information

SQL: Queries, Programming, Triggers. Basic SQL Query. Conceptual Evaluation Strategy. Example of Conceptual Evaluation. A Note on Range Variables

SQL: Queries, Programming, Triggers. Basic SQL Query. Conceptual Evaluation Strategy. Example of Conceptual Evaluation. A Note on Range Variables SQL: Queries, Programming, Triggers Chapter 5 Database Management Systems, R. Ramakrishnan and J. Gehrke 1 R1 Example Instances We will use these instances of the Sailors and Reserves relations in our

More information

Getting Started. Table of contents. 1 Pig Setup Running Pig Pig Latin Statements Pig Properties Pig Tutorial...

Getting Started. Table of contents. 1 Pig Setup Running Pig Pig Latin Statements Pig Properties Pig Tutorial... Table of contents 1 Pig Setup... 2 2 Running Pig... 3 3 Pig Latin Statements... 6 4 Pig Properties... 8 5 Pig Tutorial... 9 1. Pig Setup 1.1. Requirements Mandatory Unix and Windows users need the following:

More information

CSE 344 Final Review. August 16 th

CSE 344 Final Review. August 16 th CSE 344 Final Review August 16 th Final In class on Friday One sheet of notes, front and back cost formulas also provided Practice exam on web site Good luck! Primary Topics Parallel DBs parallel join

More information

Data Models and Query Languages for Data Streams

Data Models and Query Languages for Data Streams Data Models and Query Languages for Data Streams Master's Thesis Jes Søndergaard Department of Computer Science Aalborg University Denmark June, 2005 Faculty of Science and Engineering University of Aalborg

More information

The Evolution of Big Data Platforms and Data Science

The Evolution of Big Data Platforms and Data Science IBM Analytics The Evolution of Big Data Platforms and Data Science ECC Conference 2016 Brandon MacKenzie June 13, 2016 2016 IBM Corporation Hello, I m Brandon MacKenzie. I work at IBM. Data Science - Offering

More information

Getting Started. Table of contents. 1 Pig Setup Running Pig Pig Latin Statements Pig Properties Pig Tutorial...

Getting Started. Table of contents. 1 Pig Setup Running Pig Pig Latin Statements Pig Properties Pig Tutorial... Table of contents 1 Pig Setup... 2 2 Running Pig... 3 3 Pig Latin Statements... 6 4 Pig Properties... 8 5 Pig Tutorial... 9 1 Pig Setup 1.1 Requirements Mandatory Unix and Windows users need the following:

More information

Jaql. Kevin Beyer, Vuk Ercegovac, Eugene Shekita, Jun Rao, Ning Li, Sandeep Tata. IBM Almaden Research Center

Jaql. Kevin Beyer, Vuk Ercegovac, Eugene Shekita, Jun Rao, Ning Li, Sandeep Tata. IBM Almaden Research Center Jaql Running Pipes in the Clouds Kevin Beyer, Vuk Ercegovac, Eugene Shekita, Jun Rao, Ning Li, Sandeep Tata IBM Almaden Research Center http://code.google.com/p/jaql/ 2009 IBM Corporation Motivating Scenarios

More information

Scaling Up 1 CSE 6242 / CX Duen Horng (Polo) Chau Georgia Tech. Hadoop, Pig

Scaling Up 1 CSE 6242 / CX Duen Horng (Polo) Chau Georgia Tech. Hadoop, Pig CSE 6242 / CX 4242 Scaling Up 1 Hadoop, Pig Duen Horng (Polo) Chau Georgia Tech Some lectures are partly based on materials by Professors Guy Lebanon, Jeffrey Heer, John Stasko, Christos Faloutsos, Le

More information

SQL. CS 564- Fall ACKs: Dan Suciu, Jignesh Patel, AnHai Doan

SQL. CS 564- Fall ACKs: Dan Suciu, Jignesh Patel, AnHai Doan SQL CS 564- Fall 2015 ACKs: Dan Suciu, Jignesh Patel, AnHai Doan MOTIVATION The most widely used database language Used to query and manipulate data SQL stands for Structured Query Language many SQL standards:

More information

Apache Spark is a fast and general-purpose engine for large-scale data processing Spark aims at achieving the following goals in the Big data context

Apache Spark is a fast and general-purpose engine for large-scale data processing Spark aims at achieving the following goals in the Big data context 1 Apache Spark is a fast and general-purpose engine for large-scale data processing Spark aims at achieving the following goals in the Big data context Generality: diverse workloads, operators, job sizes

More information

SQL STRUCTURED QUERY LANGUAGE

SQL STRUCTURED QUERY LANGUAGE STRUCTURED QUERY LANGUAGE SQL Structured Query Language 4.1 Introduction Originally, SQL was called SEQUEL (for Structured English QUery Language) and implemented at IBM Research as the interface for an

More information

SQL. The Basics Advanced Manipulation Constraints Authorization 1. 1

SQL. The Basics Advanced Manipulation Constraints Authorization 1. 1 SQL The Basics Advanced Manipulation Constraints Authorization 1. 1 Table of Contents SQL 0 Table of Contents 0/1 Parke Godfrey 0/2 Acknowledgments 0/3 SQL: a standard language for accessing databases

More information

Templates for Supporting Sequenced Temporal Semantics in Pig Latin

Templates for Supporting Sequenced Temporal Semantics in Pig Latin Utah State University DigitalCommons@USU All Graduate Plan B and other Reports Graduate Studies 5-2011 Templates for Supporting Sequenced Temporal Semantics in Pig Latin Dhaval Deshpande Utah State University

More information

Introduction to Data Management. Lecture #14 (Relational Languages IV)

Introduction to Data Management. Lecture #14 (Relational Languages IV) Introduction to Data Management Lecture #14 (Relational Languages IV) Instructor: Mike Carey mjcarey@ics.uci.edu Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 1 It s time again for...

More information

Apache DataFu (incubating)

Apache DataFu (incubating) Apache DataFu (incubating) William Vaughan Staff Software Engineer, LinkedIn www.linkedin.com/in/williamgvaughan Apache DataFu Apache DataFu is a collection of libraries for working with large-scale data

More information

data parallelism Chris Olston Yahoo! Research

data parallelism Chris Olston Yahoo! Research data parallelism Chris Olston Yahoo! Research set-oriented computation data management operations tend to be set-oriented, e.g.: apply f() to each member of a set compute intersection of two sets easy

More information

Distributed Data Management Summer Semester 2013 TU Kaiserslautern

Distributed Data Management Summer Semester 2013 TU Kaiserslautern Distributed Data Management Summer Semester 2013 TU Kaiserslautern Dr.- Ing. Sebas4an Michel smichel@mmci.uni- saarland.de Distributed Data Management, SoSe 2013, S. Michel 1 Lecture 4 PIG/HIVE Distributed

More information

USER SPECIFICATIONS. 2 Physical Constraints. Optimized Algebraic Representation. Optimization

USER SPECIFICATIONS. 2 Physical Constraints. Optimized Algebraic Representation. Optimization Bulk Loading Techniques for Object Databases and an Application to Relational Data Sihem Amer-Yahia, Sophie Cluet and Claude Delobel contact author: Sihem Amer-Yahia, INRIA, BP 105, 78153 Le Chesnay, France

More information

> Semantic Web Use Cases and Case Studies

> Semantic Web Use Cases and Case Studies > Semantic Web Use Cases and Case Studies Case Study: Improving Web Search using Metadata Peter Mika, Yahoo! Research, Spain November 2008 Presenting compelling search results depends critically on understanding

More information

Practical Big Data Processing An Overview of Apache Flink

Practical Big Data Processing An Overview of Apache Flink Practical Big Data Processing An Overview of Apache Flink Tilmann Rabl Berlin Big Data Center www.dima.tu-berlin.de bbdc.berlin rabl@tu-berlin.de With slides from Volker Markl and data artisans 1 2013

More information

CSE 544 Principles of Database Management Systems. Alvin Cheung Fall 2015 Lecture 10 Parallel Programming Models: Map Reduce and Spark

CSE 544 Principles of Database Management Systems. Alvin Cheung Fall 2015 Lecture 10 Parallel Programming Models: Map Reduce and Spark CSE 544 Principles of Database Management Systems Alvin Cheung Fall 2015 Lecture 10 Parallel Programming Models: Map Reduce and Spark Announcements HW2 due this Thursday AWS accounts Any success? Feel

More information

SQL: Queries, Constraints, Triggers

SQL: Queries, Constraints, Triggers SQL: Queries, Constraints, Triggers [R&G] Chapter 5 CS4320 1 Example Instances We will use these instances of the Sailors and Reserves relations in our examples. If the key for the Reserves relation contained

More information

International Journal of Advance Research in Engineering, Science & Technology

International Journal of Advance Research in Engineering, Science & Technology Impact Factor (SJIF): 3.632 International Journal of Advance Research in Engineering, Science & Technology e-issn: 2393-9877, p-issn: 2394-2444 Volume 3, Issue 2, February-2016 A SURVEY ON HADOOP PIG SYSTEM

More information

Copyright 2016 Ramez Elmasri and Shamkant B. Navathe

Copyright 2016 Ramez Elmasri and Shamkant B. Navathe CHAPTER 19 Query Optimization Introduction Query optimization Conducted by a query optimizer in a DBMS Goal: select best available strategy for executing query Based on information available Most RDBMSs

More information

Apache Pig Releases. Table of contents

Apache Pig Releases. Table of contents Table of contents 1 Download...3 2 News... 3 2.1 19 June, 2017: release 0.17.0 available...3 2.2 8 June, 2016: release 0.16.0 available...3 2.3 6 June, 2015: release 0.15.0 available...3 2.4 20 November,

More information

CSE 344 MAY 7 TH EXAM REVIEW

CSE 344 MAY 7 TH EXAM REVIEW CSE 344 MAY 7 TH EXAM REVIEW EXAMINATION STATIONS Exam Wednesday 9:30-10:20 One sheet of notes, front and back Practice solutions out after class Good luck! EXAM LENGTH Production v. Verification Practice

More information

2.3 Algorithms Using Map-Reduce

2.3 Algorithms Using Map-Reduce 28 CHAPTER 2. MAP-REDUCE AND THE NEW SOFTWARE STACK one becomes available. The Master must also inform each Reduce task that the location of its input from that Map task has changed. Dealing with a failure

More information

CIS 612 Advanced Topics in Database Big Data Project Lawrence Ni, Priya Patil, James Tench

CIS 612 Advanced Topics in Database Big Data Project Lawrence Ni, Priya Patil, James Tench CIS 612 Advanced Topics in Database Big Data Project Lawrence Ni, Priya Patil, James Tench Abstract Implementing a Hadoop-based system for processing big data and doing analytics is a topic which has been

More information

Web Services for Relational Data Access

Web Services for Relational Data Access Web Services for Relational Data Access Sal Valente CS 6750 Fall 2010 Abstract I describe services which make it easy for users of a grid system to share data from an RDBMS. The producer runs a web services

More information

Data Base Management System LAB LECTURES

Data Base Management System LAB LECTURES Data Base Management System LAB LECTURES Taif University faculty of Computers and Information Technology First Semester 34-1435 H A. Arwa Bokhari & A. Khlood Alharthi & A. Aamal Alghamdi OBJECTIVE u Stored

More information

IN ACTION. Chuck Lam SAMPLE CHAPTER MANNING

IN ACTION. Chuck Lam SAMPLE CHAPTER MANNING IN ACTION Chuck Lam SAMPLE CHAPTER MANNING Hadoop in Action by Chuck Lam Chapter 10 Copyright 2010 Manning Publications brief contents PART I HADOOP A DISTRIBUTED PROGRAMMING FRAMEWORK... 1 1 Introducing

More information

Midterm Review. March 27, 2017

Midterm Review. March 27, 2017 Midterm Review March 27, 2017 1 Overview Relational Algebra & Query Evaluation Relational Algebra Rewrites Index Design / Selection Physical Layouts 2 Relational Algebra & Query Evaluation 3 Relational

More information

20461: Querying Microsoft SQL Server 2014 Databases

20461: Querying Microsoft SQL Server 2014 Databases Course Outline 20461: Querying Microsoft SQL Server 2014 Databases Module 1: Introduction to Microsoft SQL Server 2014 This module introduces the SQL Server platform and major tools. It discusses editions,

More information

Querying Data with Transact SQL

Querying Data with Transact SQL Course 20761A: Querying Data with Transact SQL Course details Course Outline Module 1: Introduction to Microsoft SQL Server 2016 This module introduces SQL Server, the versions of SQL Server, including

More information

Lesson 13 Transcript: User-Defined Functions

Lesson 13 Transcript: User-Defined Functions Lesson 13 Transcript: User-Defined Functions Slide 1: Cover Welcome to Lesson 13 of DB2 ON CAMPUS LECTURE SERIES. Today, we are going to talk about User-defined Functions. My name is Raul Chong, and I'm

More information

2/26/2017. Originally developed at the University of California - Berkeley's AMPLab

2/26/2017. Originally developed at the University of California - Berkeley's AMPLab Apache is a fast and general engine for large-scale data processing aims at achieving the following goals in the Big data context Generality: diverse workloads, operators, job sizes Low latency: sub-second

More information

Principles of Data Management. Lecture #9 (Query Processing Overview)

Principles of Data Management. Lecture #9 (Query Processing Overview) Principles of Data Management Lecture #9 (Query Processing Overview) Instructor: Mike Carey mjcarey@ics.uci.edu Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke 1 Today s Notable News v Midterm

More information

MapReduce and Friends

MapReduce and Friends MapReduce and Friends Craig C. Douglas University of Wyoming with thanks to Mookwon Seo Why was it invented? MapReduce is a mergesort for large distributed memory computers. It was the basis for a web

More information

SQL OVERVIEW. CS121: Relational Databases Fall 2017 Lecture 4

SQL OVERVIEW. CS121: Relational Databases Fall 2017 Lecture 4 SQL OVERVIEW CS121: Relational Databases Fall 2017 Lecture 4 SQL 2 SQL = Structured Query Language Original language was SEQUEL IBM s System R project (early 1970 s) Structured English Query Language Caught

More information

Notes. Some of these slides are based on a slide set provided by Ulf Leser. CS 640 Query Processing Winter / 30. Notes

Notes. Some of these slides are based on a slide set provided by Ulf Leser. CS 640 Query Processing Winter / 30. Notes uery Processing Olaf Hartig David R. Cheriton School of Computer Science University of Waterloo CS 640 Principles of Database Management and Use Winter 2013 Some of these slides are based on a slide set

More information

Relational Algebra. Study Chapter Comp 521 Files and Databases Fall

Relational Algebra. Study Chapter Comp 521 Files and Databases Fall Relational Algebra Study Chapter 4.1-4.2 Comp 521 Files and Databases Fall 2010 1 Relational Query Languages Query languages: Allow manipulation and retrieval of data from a database. Relational model

More information

SQL. Chapter 5 FROM WHERE

SQL. Chapter 5 FROM WHERE SQL Chapter 5 Instructor: Vladimir Zadorozhny vladimir@sis.pitt.edu Information Science Program School of Information Sciences, University of Pittsburgh 1 Basic SQL Query SELECT FROM WHERE [DISTINCT] target-list

More information

An SQL query is parsed into a collection of query blocks optimize one block at a time. Nested blocks are usually treated as calls to a subroutine

An SQL query is parsed into a collection of query blocks optimize one block at a time. Nested blocks are usually treated as calls to a subroutine QUERY OPTIMIZATION 1 QUERY OPTIMIZATION QUERY SUB-SYSTEM 2 ROADMAP 3. 12 QUERY BLOCKS: UNITS OF OPTIMIZATION An SQL query is parsed into a collection of query blocks optimize one block at a time. Nested

More information

Hadoop. Course Duration: 25 days (60 hours duration). Bigdata Fundamentals. Day1: (2hours)

Hadoop. Course Duration: 25 days (60 hours duration). Bigdata Fundamentals. Day1: (2hours) Bigdata Fundamentals Day1: (2hours) 1. Understanding BigData. a. What is Big Data? b. Big-Data characteristics. c. Challenges with the traditional Data Base Systems and Distributed Systems. 2. Distributions:

More information

Introduction to Hadoop. High Availability Scaling Advantages and Challenges. Introduction to Big Data

Introduction to Hadoop. High Availability Scaling Advantages and Challenges. Introduction to Big Data Introduction to Hadoop High Availability Scaling Advantages and Challenges Introduction to Big Data What is Big data Big Data opportunities Big Data Challenges Characteristics of Big data Introduction

More information

Module 4. Implementation of XQuery. Part 0: Background on relational query processing

Module 4. Implementation of XQuery. Part 0: Background on relational query processing Module 4 Implementation of XQuery Part 0: Background on relational query processing The Data Management Universe Lecture Part I Lecture Part 2 2 What does a Database System do? Input: SQL statement Output:

More information

This is the Pre-Published Version

This is the Pre-Published Version This is the Pre-Published Version Path Dictionary: A New Approach to Query Processing in Object-Oriented Databases Wang-chien Lee Dept of Computer and Information Science The Ohio State University Columbus,

More information

Distributed Systems. 21. Graph Computing Frameworks. Paul Krzyzanowski. Rutgers University. Fall 2016

Distributed Systems. 21. Graph Computing Frameworks. Paul Krzyzanowski. Rutgers University. Fall 2016 Distributed Systems 21. Graph Computing Frameworks Paul Krzyzanowski Rutgers University Fall 2016 November 21, 2016 2014-2016 Paul Krzyzanowski 1 Can we make MapReduce easier? November 21, 2016 2014-2016

More information

Forward declaration of enumerations

Forward declaration of enumerations Doc. no.: N2499=08-0009 Date: 2008-01-09 Project: Programming Language C++ Reply to: Alberto Ganesh Barbati Forward declaration of enumerations 1 Introduction In C++03 every declaration

More information

The members of the Committee approve the thesis of Baosheng Cai defended on March David B. Whalley Professor Directing Thesis Xin Yuan Commit

The members of the Committee approve the thesis of Baosheng Cai defended on March David B. Whalley Professor Directing Thesis Xin Yuan Commit THE FLORIDA STATE UNIVERSITY COLLEGE OF ARTS AND SCIENCES COMPILER MODIFICATIONS TO SUPPORT INTERACTIVE COMPILATION By BAOSHENG CAI A Thesis submitted to the Department of Computer Science in partial fulllment

More information

Incremental Flow Analysis. Andreas Krall and Thomas Berger. Institut fur Computersprachen. Technische Universitat Wien. Argentinierstrae 8

Incremental Flow Analysis. Andreas Krall and Thomas Berger. Institut fur Computersprachen. Technische Universitat Wien. Argentinierstrae 8 Incremental Flow Analysis Andreas Krall and Thomas Berger Institut fur Computersprachen Technische Universitat Wien Argentinierstrae 8 A-1040 Wien fandi,tbg@mips.complang.tuwien.ac.at Abstract Abstract

More information

CIS 330: Applied Database Systems

CIS 330: Applied Database Systems 1 CIS 330: Applied Database Systems Lecture 7: SQL Johannes Gehrke johannes@cs.cornell.edu http://www.cs.cornell.edu/johannes Logistics Office hours role call: Mondays, 3-4pm Tuesdays, 4:30-5:30 Wednesdays,

More information

C2: How to work with a petabyte

C2: How to work with a petabyte GREAT 2011 Summer School C2: How to work with a petabyte Matthew J. Graham (Caltech, VAO) Overview Strategy MapReduce Hadoop family GPUs 2/17 Divide-and-conquer strategy Most problems in astronomy are

More information

Safe Harbor Statement

Safe Harbor Statement Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment

More information

Part I: Structured Data

Part I: Structured Data Inf1-DA 2011 2012 I: 92 / 117 Part I Structured Data Data Representation: I.1 The entity-relationship (ER) data model I.2 The relational model Data Manipulation: I.3 Relational algebra I.4 Tuple-relational

More information