Aims. Background. This exercise aims to get you to:
|
|
- Stuart Ferguson
- 6 years ago
- Views:
Transcription
1 Aims This exercise aims to get you to: Import data into HBase using bulk load Read MapReduce input from HBase and write MapReduce output to HBase Manage data using Hive Manage data using Pig Background In HBase-speak, bulk loading is the process of preparing and loading HFiles (HBase s own file format) directly into the RegionServers. Bulk load steps: 1. Extract the data from a source, typically text files or another database. 2. Transform the data into HFiles. This step requires a MapReduce job and for most input types you will have to write the Mapper yourself. The job will need to emit the row key as the Key, and either a KeyValue, a Put, or a Delete as the Value. The Reducer is handled by HBase; you configure it using HFileOutputFormat2.configureIncrementalLoad(). 3. Load the files into HBase by telling the RegionServers where to find them. It requires using LoadIncrementalHFiles (more commonly known as the completebulkload tool), and by passing it a URL that locates the files in HDFS, it will load each file into the relevant region via the RegionServer that serves it. Here s an illustration of this process. The data flow goes from the original source to HDFS, where the RegionServers will simply move the files to their regions directories. See more details at:
2 Because HBase is not installed in the VM image in the lab computers, you need to install HBase again following the instructions in Lab 5. Create a project Lab6 and create a package comp9313.lab6 in this project. Put all your java codes in this package and keep a copy. Right click the project -> Properties -> Java Build Path -> Libraries -> Add Externals JARs -> go to the folder comp9313/base-1.2.2/lib, and add all the jar files to the project. Data Set Download the two files Votes and Posts from the course homepage. The data set contains many questions asked on and the corresponding answers. The two file used in this week s lab are obtained at: part of datascience.stackexchange.com.7z. The format of the data set is shown at: The data format of Votes is (the field BountyAmount is ignored): - **votes**.xml - Id - PostId - VoteTypeId - ` 1`: AcceptedByOriginator - ` 2`: UpMod - ` 3`: DownMod - ` 4`: Offensive - ` 5`: Favorite - if VoteTypeId = 5 UserId will be populated - ` 6`: Close - ` 7`: Reopen - ` 8`: BountyStart - ` 9`: BountyClose - `10`: Deletion - `11`: Undeletion - `12`: Spam - `13`: InformModerator - CreationDate - UserId (only for VoteTypeId 5) - BountyAmount (only for VoteTypeId 9) For example: The data format of Comments is:
3 - **comments**.xml - Id - PostId - Score - Text, e.g.: "@Stu Thompson: Seems possible to me - why not try it?" - CreationDate, e.g.:" t08:07:10.730" - UserId For example: HBase Data Bulk Load Import Votes as a table in HBase. 1. HBase will use a staging folder to store temporary data, and we need to configure this directory for HBase. Create a folder /tmp/hbase-staging in HDFS, and change its mode to 711 (i.e., rwx x x). $ hdfs dfs mkdir /tmp/hbase-staging $ hdfs dfs chmod 711 /tmp/hbase-staging Add the following lines to $HBASE_HOME/conf/hbase-site.xml (in between <configuration> and </configuration>: <property> <name>hbase.bulkload.staging.dir</name> <value>/tmp/hbase-staging</value> </property> <property> <name>hbase.coprocessor.region.classes</name> <value>org.apache.hadoop.hbase.security.token.tokenprovider,org.apach e.hadoop.hbase.security.access.accesscontroller,org.apache.hadoop.hba se.security.access.securebulkloadendpoint</value> </property> In your MapReduce code, you need to configure the two properties: hbase.fs.tmp.dir and hbase.bulkload.staging.dir. After creating a Configuration object, you need to: Configuration conf = HBaseConfiguration.create(); conf.set( hbase.fs.tmp.dir, /tmp/hbase-staging ); conf.set( hbase.bulkload.staging.dir, /tmp/hbase-staging );
4 2. The code for bulk loading Votes into HBase is available at the course homepage, i.e., Vote.java and HBaseBulkLoadExample.java. Below lists some explanations of the code: Only the mapper is required in bulk load, because the Reducer is handled by HBase and you configure it using HFileOutputFormat2.configureIncrementalLoad(). The map output key data type must be ImmutableBytesWritable, and the map output value data type can only be a KeyValue/Put/Delete object. In this example, you create a Put object, which will be used to insert the data into the HBase table. The table can either be created using HBase shell or HBase Java API. In the give code, the table is created using Java API. In the example code, the class HBaseBulkLoadExample implements the interface Tool, and the job is configured and started in the run() function. Then, ToolRunner.run() is used to invoke HBaseBulkLoadExample.run(). You can also configure and start the job in the main function, as you did in the previous labs on MapReduce. Before starting the job, you need to use HFileOutputFormat2.configureIncrementalLoad() to configure the bulk load. After the job is completed, that is, the mapper generate the Put objects for all input data, you use LoadIncrementalHFiles to do the bulk load. It is the tool to load the output of HFileOutputFormat2 into an existing table. 3. After Votes is loaded into the table votes, open the HBase shell to check the table and its contents. Your Task: Import Comments as a table in HBase. Create a class HBaseBulkLoadComments.java and a class Comment.java in package comp9313.lab6 to finish this task. Use Id as the rowkey, and create three column families, postinfo (containing PostId), commentinfo (containing Score, Text, and CreationDate), and userinfo (containing UserId ). Read MapReduce Input from HBase Problem 1. Read input data from table votes in HBase, and count for each post the number of each type of vote for this post. The output data is of format: (PostID, {<VoteTypeId, count>}).
5 For example, if post with ID 1 has two votes, one is of type 1 and another is of type 2, then you should output (1, {<1, 1>, <2, 1>}). Please refer to for the examples of HBase MapReduce read. Hints: 1. Your mapper should be extended from TableMapper<K, V>. The input key data type is ImmutableBytesWritable, and value data type is Result. Each map() function will read one row from the HBase table, and you can use Result.getValue(CF, COLUMN) to get the value in a cell. Your mapper code will be like: public static class AggregateMapper extends TableMapper<Text, Text>{ public void map(immutablebyteswritable row, Result value, Context context) throws IOException, InterruptedException { //do your job } } 2. The reducer is just like a normal MapReduce reducer 3. In the main function, you will need to use the function TableMapReduceUtil.initTableMapperJob() to configure the mapper. 4. Because the data is read from HBase, you do not need to configure the data input path. You only need to specify the output path in Eclipse. The code ReadHBaseExample.java is available at the course webpage. Try to write the mapper by yourself, and learn how to configure the HBase read job from that file. Problem 2: Read input data from table comments in HBase, and calculate the number of comments per UserID. Refer to the code ReadHBaseExample.java and write your code in ReadHBaseComment.java in package comp9313.lab6. Write MapReduce Output to HBase Problem 1. Read input data from Votes, and count the number of votes per user. The result will be written to an HBase table votestats, rather than storing in files generated by reducers.
6 Please refer to for the examples of HBase MapReduce Write. Hints: 1. The mapper is just like a normal MapReduce mapper. 2. Your reducer should be extended from TableReducer<K, V>. The output key is ignored, and the value data type is ImmutableBytesWritable. The reduce() function will aggregate the number of comments for a user. You need to create a Put object to store the information, and HBase will use this object to insert the information into table votestats. Your reducer code will be like: public static class UserVotesReducer extends TableReducer<Text, IntWritable, ImmutableBytesWritable> { public void reduce(text key, Iterable<IntWritable> values, Context context)throws IOException, InterruptedException { //do your job } } 3. In the main function, you will need to use the function TableMapReduceUtil.initTableReducerJob() to configure the reducer. 4. You can create the table in the main function, or using the HBase shell. 5. Because the data is written to HBase, you do not need to configure the data output path. You only need to specify the input path in Eclipse. The code WriteHBaseExample.java is available at the course webpage. Try to write the reducer by yourself, and learn how to configure the HBase write job from that file. Problem 2: Read input data from Comments, and calculate the average score of comments for each question. The result will be written to an HBase table post_comment_score, with only one column family stats. Refer to the code WriteHBaseExample.java and write your code in WriteHBaseComment.java in package comp9313.lab6. Manage Data Using Hive Hive Installation and Configuration 1. Download Hive 2.1.0
7 $ wget bin.tar.gz Then unpack the package: $ tar xvf apache-hive bin.tar.gz 2. Define environment variables for Hive We need to configure the working directory of Hive, i.e., HIVE_HOME. Open the file ~/.bashrc and add the following lines at the end of this file: export HIVE_HOME = ~/apache-hive bin export PATH = $HIVE_HOME/bin:$PATH Save the file, and then run the following command to take these configurations into effect: $ source ~/.bashrc 3. Create /tmp and /user/hive/warehouse and set them chmod g+w for more than one user usage $ hdfs dfs -mkdir /tmp $ hdfs dfs -mkdir p /user/hive/warehouse $ hdfs dfs -chmod g+w /tmp $ hdfs dfs -chmod g+w /user/hive/warehouse 4. Run the schematool command to initialize Hive $ schematool -dbtype derby -initschema Now you have already done the basic configuration of Hive, and it is ready to use. Start Hive shell by the following command (start HDFS and YARN first!): $ hive
8 Practice Hive 1. Download the test file employees.txt from the course webpage. The file contains only 7 records. Put the file at the home folder. 2. Create a database $ hive> create database employee_data; $ hive> use employee_data; 3. All databases are created under /user/hive/warehouse directory. $ hdfs dfs ls /user/hive/warehouse 4. Create the employee table $ hive> CREATE TABLE employees ( name STRING, salary FLOAT, subordinates ARRAY<STRING>, deductions MAP<STRING, FLOAT>, address STRUCT<street:STRING, city:string, state:string, zip:int> ) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\001' COLLECTION ITEMS TERMINATED BY '\002' MAP KEYS TERMINATED BY '\003' LINES TERMINATED BY '\n' STORED AS TEXTFILE; Because '\001', '\002', '\003', and '\n' are by default, and thus you can ignore ROW FORMAT DELIMITED. STORED AS TEXTFILE is also by default, and can be ignored as well. 5. Show all tables in the current database $ hive> show tables; 6. Load data from local file system into table $ hive> LOAD DATA LOCAL INPATH '/home/comp9313/employees.txt' OVERWRITE INTO TABLE employees;
9 After loading the data into the table, you can check in HDFS what happened: $ hdfs dfs ls /user/hive/warehouse/employee_data.db/employees The file employees.txt is copied into this folder corresponding to the table. 7. Check the data in the table $ select * from employees; 8. You can do various queries based on the employees table, just as in an RDBMS. For example: Question 1: show the number of employees and their average salary Hint: use count() and avg() Question 2: find the employee who has the highest salary Hint: use max(), IN clause, and subquery in where clause 9. Usage of explode(). Find all employees who are the subordinate of another person. explode() takes in an array (or a map) as an input and outputs the elements of the array (map) as separate rows. $ hive> SELECT explode(subordinates) FROM employees; 10. Hive partitions. When defining employees, it is not partitioned, and thus you cannot add a partition to it. You can only add a new partition to a table has already been partitioned! Create a table employees2, and load the same file into it. $ hive> CREATE TABLE employees2 ( name STRING, salary FLOAT, subordinates ARRAY<STRING>, deductions MAP<STRING, FLOAT>,
10 address STRUCT<street:STRING, city:string, state:string, zip:int> )PARTITIONED BY (join_year STRING); $ hive> LOAD DATA LOCAL INPATH '/home/comp9313/employees.txt' OVERWRITE INTO TABLE employees2 PARTITION (join_year= 2015 ); Now check HDFS again to see what happened: $ hdfs dfs ls /user/hive/warehouse/employ_data.db/employees2 You will see a folder join_year=2015 created in this folder, corresponding to the partition join_year= Add a new partition join_year= 2016 to the table. $ hive> ALTER TABLE employees2 ADD PARTITION (join_year= 2016 ) LOCATION /user/hive/warehouse/employee_data.db/employees2/join_year=2016 ; Check in HDFS, and you will see a new folder created for this partition. 11. Insert a record to partition join_year= Because Hive does not support literals for complex types (array, map, struct, union), so it is not possible to use them in INSERT INTO...VALUES clauses. You need to create a file to store the new record, and then load it into the partition. $ cp employees.txt employees2016.txt Then use vim or gedit to edit employees2016.txt to add some records, and then load the file into the partition. 12. Query on a partition. Question: find all employees joined in the year 2016 whose salary is more than (optional) Do word count in Hive, using the file employees.txt. Manage Data Using Pig Pig Installation and Configuration 1. Download Pig $ wget tar.gz Then unpack the package: $ tar xvf pig tar.gz
11 2. Define environment variables for Pig We need to configure the working directory of Hive, i.e., PIG_HOME. Open the file ~/.bashrc and add the following lines at the end of this file: export PIG_HOME = ~/pig export PATH = $PIG_HOME/bin:$PATH Save the file, and then run the following command to take these configurations into effect: $ source ~/.bashrc 3. Now you have already done the basic configuration of Pig, and it is ready to use. Start Pig Grunt shell by the following command (start HDFS and YARN first!): $ pig Practice Pig 1. Download the test file NYSE_dividends.txt from the course webpage. The file contains 670 records. Put the file to HDFS. $ hdfs dfs put NYSE_dividends.txt Start the Hadoop job history server. $ mr-jobhistory-daemon.sh start historyserver 2. Load Data using load command into Schema exchange, symbol, date, dividend. $ grunt> dividends = load 'NYSE_dividends.txt' as (exchange:chararray, symbol:chararray, date:chararray, dividend:float); $ grunt> dump dividends;
12 You should see results like: 3. Group rows by symbol. $ grunt> grouped = group dividends by symbol; 4. Compute the average dividends for each symbol. Dividend value is obtained using expression dividends.dividend (or dividends.$3). Store this result in a variable avg. $ grunt> avg = foreach grouped generate group, AVG(dividends.$3); Use dump to check the contents of avg, you should see: 5. Store result avg into HDFS using store command $ grunt> store avg into 'average_dividend'; 6. Store result avg into HDFS using store command $ grunt> fs -cat /user/comp9313/average_dividend/* 7. (optional) Do word count in Pig, using the file employees.txt. More Practices More practices of Hive and Pig are put into the second assignment.
HBase Installation and Configuration
Aims This exercise aims to get you to: Install and configure HBase Manage data using HBase Shell Install and configure Hive Manage data using Hive HBase Installation and Configuration 1. Download HBase
More informationHBase Installation and Configuration
Aims This exercise aims to get you to: Install and configure HBase Manage data using HBase Shell Manage data using HBase Java API HBase Installation and Configuration 1. Download HBase 1.2.2 $ wget http://apache.uberglobalmirror.com/hbase/1.2.2/hbase-1.2.2-
More informationIntroduction to Hive. Feng Li School of Statistics and Mathematics Central University of Finance and Economics
Introduction to Hive Feng Li feng.li@cufe.edu.cn School of Statistics and Mathematics Central University of Finance and Economics Revised on December 14, 2017 Today we are going to learn... 1 Introduction
More informationApache Hive. CMSC 491 Hadoop-Based Distributed Compu<ng Spring 2016 Adam Shook
Apache Hive CMSC 491 Hadoop-Based Distributed Compu
More informationCOSC 6397 Big Data Analytics. Data Formats (III) HBase: Java API, HBase in MapReduce and HBase Bulk Loading. Edgar Gabriel Spring 2014.
COSC 6397 Big Data Analytics Data Formats (III) HBase: Java API, HBase in MapReduce and HBase Bulk Loading Edgar Gabriel Spring 2014 Recap on HBase Column-Oriented data store NoSQL DB Data is stored in
More informationCompile and Run WordCount via Command Line
Aims This exercise aims to get you to: Compile, run, and debug MapReduce tasks via Command Line Compile, run, and debug MapReduce tasks via Eclipse One Tip on Hadoop File System Shell Following are the
More informationHadoop is essentially an operating system for distributed processing. Its primary subsystems are HDFS and MapReduce (and Yarn).
1 Hadoop Primer Hadoop is essentially an operating system for distributed processing. Its primary subsystems are HDFS and MapReduce (and Yarn). 2 Passwordless SSH Before setting up Hadoop, setup passwordless
More informationThe detailed Spark programming guide is available at:
Aims This exercise aims to get you to: Analyze data using Spark shell Monitor Spark tasks using Web UI Write self-contained Spark applications using Scala in Eclipse Background Spark is already installed
More informationLab: Hive Management
Managing & Using Hive/HiveQL 2018 ABYRES Enterprise Technologies 1 of 30 1. Table of Contents 1. Table of Contents...2 2. Accessing Hive With Beeline...3 3. Accessing Hive With Squirrel SQL...4 4. Accessing
More informationAccessing Hadoop Data Using Hive
An IBM Proof of Technology Accessing Hadoop Data Using Hive Unit 3: Hive DML in action An IBM Proof of Technology Catalog Number Copyright IBM Corporation, 2015 US Government Users Restricted Rights -
More informationHadoop is supplemented by an ecosystem of open source projects IBM Corporation. How to Analyze Large Data Sets in Hadoop
Hadoop Open Source Projects Hadoop is supplemented by an ecosystem of open source projects Oozie 25 How to Analyze Large Data Sets in Hadoop Although the Hadoop framework is implemented in Java, MapReduce
More informationBig Data Hadoop Developer Course Content. Big Data Hadoop Developer - The Complete Course Course Duration: 45 Hours
Big Data Hadoop Developer Course Content Who is the target audience? Big Data Hadoop Developer - The Complete Course Course Duration: 45 Hours Complete beginners who want to learn Big Data Hadoop Professionals
More informationMapReduce & YARN Hands-on Lab Exercise 1 Simple MapReduce program in Java
MapReduce & YARN Hands-on Lab Exercise 1 Simple MapReduce program in Java Contents Page 1 Copyright IBM Corporation, 2015 US Government Users Restricted Rights - Use, duplication or disclosure restricted
More informationIntroduction to BigData, Hadoop:-
Introduction to BigData, Hadoop:- Big Data Introduction: Hadoop Introduction What is Hadoop? Why Hadoop? Hadoop History. Different types of Components in Hadoop? HDFS, MapReduce, PIG, Hive, SQOOP, HBASE,
More informationPart 1: Installing MongoDB
Samantha Orogvany-Charpentier CSU ID: 2570586 Installing NoSQL Systems Part 1: Installing MongoDB For my lab, I installed MongoDB version 3.2.12 on Ubuntu 16.04. I followed the instructions detailed at
More informationHadoop Development Introduction
Hadoop Development Introduction What is Bigdata? Evolution of Bigdata Types of Data and their Significance Need for Bigdata Analytics Why Bigdata with Hadoop? History of Hadoop Why Hadoop is in demand
More informationHadoop Lab 3 Creating your first Map-Reduce Process
Programming for Big Data Hadoop Lab 3 Creating your first Map-Reduce Process Lab work Take the map-reduce code from these notes and get it running on your Hadoop VM Driver Code Mapper Code Reducer Code
More informationHadoop. Course Duration: 25 days (60 hours duration). Bigdata Fundamentals. Day1: (2hours)
Bigdata Fundamentals Day1: (2hours) 1. Understanding BigData. a. What is Big Data? b. Big-Data characteristics. c. Challenges with the traditional Data Base Systems and Distributed Systems. 2. Distributions:
More informationHadoop Lab 2 Exploring the Hadoop Environment
Programming for Big Data Hadoop Lab 2 Exploring the Hadoop Environment Video A short video guide for some of what is covered in this lab. Link for this video is on my module webpage 1 Open a Terminal window
More informationMore Access to Data Managed by Hadoop
How to Train Your Elephant 2 More Access to Data Managed by Hadoop Clif Kranish 1 Topics Introduction What is Hadoop Apache Hive Arrays, Maps and Structs Apache Drill Apache HBase Apache Phoenix Adapter
More informationInnovatus Technologies
HADOOP 2.X BIGDATA ANALYTICS 1. Java Overview of Java Classes and Objects Garbage Collection and Modifiers Inheritance, Aggregation, Polymorphism Command line argument Abstract class and Interfaces String
More informationIntroduction to Hadoop. High Availability Scaling Advantages and Challenges. Introduction to Big Data
Introduction to Hadoop High Availability Scaling Advantages and Challenges Introduction to Big Data What is Big data Big Data opportunities Big Data Challenges Characteristics of Big data Introduction
More informationGetting Started. Table of contents. 1 Pig Setup Running Pig Pig Latin Statements Pig Properties Pig Tutorial...
Table of contents 1 Pig Setup... 2 2 Running Pig... 3 3 Pig Latin Statements... 6 4 Pig Properties... 8 5 Pig Tutorial... 9 1. Pig Setup 1.1. Requirements Mandatory Unix and Windows users need the following:
More informationApache Hadoop Installation and Single Node Cluster Configuration on Ubuntu A guide to install and setup Single-Node Apache Hadoop 2.
SDJ INFOSOFT PVT. LTD Apache Hadoop 2.6.0 Installation and Single Node Cluster Configuration on Ubuntu A guide to install and setup Single-Node Apache Hadoop 2.x Table of Contents Topic Software Requirements
More informationCOMP4442. Service and Cloud Computing. Lab 12: MapReduce. Prof. George Baciu PQ838.
COMP4442 Service and Cloud Computing Lab 12: MapReduce www.comp.polyu.edu.hk/~csgeorge/comp4442 Prof. George Baciu csgeorge@comp.polyu.edu.hk PQ838 1 Contents Introduction to MapReduce A WordCount example
More informationApache Hive. Big Data - 16/04/2018
Apache Hive Big Data - 16/04/2018 Hive Configuration Translates HiveQL statements into a set of MapReduce jobs which are then executed on a Hadoop Cluster Execute on Hadoop Cluster HiveQL Hive Monitor/Report
More informationExamTorrent. Best exam torrent, excellent test torrent, valid exam dumps are here waiting for you
ExamTorrent http://www.examtorrent.com Best exam torrent, excellent test torrent, valid exam dumps are here waiting for you Exam : Apache-Hadoop-Developer Title : Hadoop 2.0 Certification exam for Pig
More informationIntroduction to Hive Cloudera, Inc.
Introduction to Hive Outline Motivation Overview Data Model Working with Hive Wrap up & Conclusions Background Started at Facebook Data was collected by nightly cron jobs into Oracle DB ETL via hand-coded
More informationIntroduction to HDFS and MapReduce
Introduction to HDFS and MapReduce Who Am I - Ryan Tabora - Data Developer at Think Big Analytics - Big Data Consulting - Experience working with Hadoop, HBase, Hive, Solr, Cassandra, etc. 2 Who Am I -
More informationLab 3 Pig, Hive, and JAQL
Lab 3 Pig, Hive, and JAQL Lab objectives In this lab you will practice what you have learned in this lesson, specifically you will practice with Pig, Hive, and Jaql languages. Lab instructions This lab
More informationHadoop Online Training
Hadoop Online Training IQ training facility offers Hadoop Online Training. Our Hadoop trainers come with vast work experience and teaching skills. Our Hadoop training online is regarded as the one of the
More informationBlended Learning Outline: Cloudera Data Analyst Training (171219a)
Blended Learning Outline: Cloudera Data Analyst Training (171219a) Cloudera Univeristy s data analyst training course will teach you to apply traditional data analytics and business intelligence skills
More informationUsing Hive for Data Warehousing
An IBM Proof of Technology Using Hive for Data Warehousing Unit 3: Hive DML in action An IBM Proof of Technology Catalog Number Copyright IBM Corporation, 2013 US Government Users Restricted Rights - Use,
More informationCOSC 6339 Big Data Analytics. NoSQL (III) HBase in Hadoop MapReduce 3 rd homework assignment. Edgar Gabriel Spring 2017.
COSC 6339 Big Data Analytics NoSQL (III) HBase in Hadoop MapReduce 3 rd homework assignment Edgar Gabriel Spring 2017 Recap on HBase Column-Oriented data store NoSQL DB Data is stored in Tables Tables
More informationUltimate Hadoop Developer Training
First Edition Ultimate Hadoop Developer Training Lab Exercises edubrake.com Hadoop Architecture 2 Following are the exercises that the student need to finish, as required for the module Hadoop Architecture
More informationAbout 1. Chapter 1: Getting started with oozie 2. Remarks 2. Versions 2. Examples 2. Installation or Setup 2. Chapter 2: Oozie
oozie #oozie Table of Contents About 1 Chapter 1: Getting started with oozie 2 Remarks 2 Versions 2 Examples 2 Installation or Setup 2 Chapter 2: Oozie 101 7 Examples 7 Oozie Architecture 7 Oozie Application
More informationSE256 : Scalable Systems for Data Science
SE256 : Scalable Systems for Data Science Lab Session: 2 Maven setup: Run the following commands to download and extract maven. wget http://www.eu.apache.org/dist/maven/maven 3/3.3.9/binaries/apache maven
More informationParallel Data Processing with Hadoop/MapReduce. CS140 Tao Yang, 2014
Parallel Data Processing with Hadoop/MapReduce CS140 Tao Yang, 2014 Overview What is MapReduce? Example with word counting Parallel data processing with MapReduce Hadoop file system More application example
More informationGetting Started. Table of contents. 1 Pig Setup Running Pig Pig Latin Statements Pig Properties Pig Tutorial...
Table of contents 1 Pig Setup... 2 2 Running Pig... 3 3 Pig Latin Statements... 6 4 Pig Properties... 8 5 Pig Tutorial... 9 1 Pig Setup 1.1 Requirements Mandatory Unix and Windows users need the following:
More informationHive SQL over Hadoop
Hive SQL over Hadoop Antonino Virgillito THE CONTRACTOR IS ACTING UNDER A FRAMEWORK CONTRACT CONCLUDED WITH THE COMMISSION Introduction Apache Hive is a high-level abstraction on top of MapReduce Uses
More informationBig Data Syllabus. Understanding big data and Hadoop. Limitations and Solutions of existing Data Analytics Architecture
Big Data Syllabus Hadoop YARN Setup Programming in YARN framework j Understanding big data and Hadoop Big Data Limitations and Solutions of existing Data Analytics Architecture Hadoop Features Hadoop Ecosystem
More informationOutline Introduction Big Data Sources of Big Data Tools HDFS Installation Configuration Starting & Stopping Map Reduc.
D. Praveen Kumar Junior Research Fellow Department of Computer Science & Engineering Indian Institute of Technology (Indian School of Mines) Dhanbad, Jharkhand, India Head of IT & ITES, Skill Subsist Impels
More informationThis is a brief tutorial that explains how to make use of Sqoop in Hadoop ecosystem.
About the Tutorial Sqoop is a tool designed to transfer data between Hadoop and relational database servers. It is used to import data from relational databases such as MySQL, Oracle to Hadoop HDFS, and
More informationJava in MapReduce. Scope
Java in MapReduce Kevin Swingler Scope A specific look at the Java code you might use for performing MapReduce in Hadoop Java program recap The map method The reduce method The whole program Running on
More informationHadoop. Introduction to BIGDATA and HADOOP
Hadoop Introduction to BIGDATA and HADOOP What is Big Data? What is Hadoop? Relation between Big Data and Hadoop What is the need of going ahead with Hadoop? Scenarios to apt Hadoop Technology in REAL
More informationHadoop & Big Data Analytics Complete Practical & Real-time Training
An ISO Certified Training Institute A Unit of Sequelgate Innovative Technologies Pvt. Ltd. www.sqlschool.com Hadoop & Big Data Analytics Complete Practical & Real-time Training Mode : Instructor Led LIVE
More informationA Guide to Running Map Reduce Jobs in Java University of Stirling, Computing Science
A Guide to Running Map Reduce Jobs in Java University of Stirling, Computing Science Introduction The Hadoop cluster in Computing Science at Stirling allows users with a valid user account to submit and
More informationDelving Deep into Hadoop Course Contents Introduction to Hadoop and Architecture
Delving Deep into Hadoop Course Contents Introduction to Hadoop and Architecture Hadoop 1.0 Architecture Introduction to Hadoop & Big Data Hadoop Evolution Hadoop Architecture Networking Concepts Use cases
More informationC2: How to work with a petabyte
GREAT 2011 Summer School C2: How to work with a petabyte Matthew J. Graham (Caltech, VAO) Overview Strategy MapReduce Hadoop family GPUs 2/17 Divide-and-conquer strategy Most problems in astronomy are
More informationWe are ready to serve Latest Testing Trends, Are you ready to learn.?? New Batches Info
We are ready to serve Latest Testing Trends, Are you ready to learn.?? New Batches Info START DATE : TIMINGS : DURATION : TYPE OF BATCH : FEE : FACULTY NAME : LAB TIMINGS : About Quality Thought We are
More informationDepartment of Computer Science University of Cyprus EPL646 Advanced Topics in Databases. Lecture 16. Big Data Management VI (MapReduce Programming)
Department of Computer Science University of Cyprus EPL646 Advanced Topics in Databases Lecture 16 Big Data Management VI (MapReduce Programming) Credits: Pietro Michiardi (Eurecom): Scalable Algorithm
More informationUsing Hive for Data Warehousing
An IBM Proof of Technology Using Hive for Data Warehousing Unit 5: Hive Storage Formats An IBM Proof of Technology Catalog Number Copyright IBM Corporation, 2015 US Government Users Restricted Rights -
More informationBig Data Hadoop Course Content
Big Data Hadoop Course Content Topics covered in the training Introduction to Linux and Big Data Virtual Machine ( VM) Introduction/ Installation of VirtualBox and the Big Data VM Introduction to Linux
More informationHIVE INTERVIEW QUESTIONS
HIVE INTERVIEW QUESTIONS http://www.tutorialspoint.com/hive/hive_interview_questions.htm Copyright tutorialspoint.com Dear readers, these Hive Interview Questions have been designed specially to get you
More informationExpert Lecture plan proposal Hadoop& itsapplication
Expert Lecture plan proposal Hadoop& itsapplication STARTING UP WITH BIG Introduction to BIG Data Use cases of Big Data The Big data core components Knowing the requirements, knowledge on Analyst job profile
More informationCOSC 6339 Big Data Analytics. Hadoop MapReduce Infrastructure: Pig, Hive, and Mahout. Edgar Gabriel Fall Pig
COSC 6339 Big Data Analytics Hadoop MapReduce Infrastructure: Pig, Hive, and Mahout Edgar Gabriel Fall 2018 Pig Pig is a platform for analyzing large data sets abstraction on top of Hadoop Provides high
More informationHadoop. copyright 2011 Trainologic LTD
Hadoop Hadoop is a framework for processing large amounts of data in a distributed manner. It can scale up to thousands of machines. It provides high-availability. Provides map-reduce functionality. Hides
More informationExercise #1: ANALYZING SOCIAL MEDIA AND CUSTOMER SENTIMENT WITH APACHE NIFI AND HDP SEARCH INTRODUCTION CONFIGURE AND START SOLR
Exercise #1: ANALYZING SOCIAL MEDIA AND CUSTOMER SENTIMENT WITH APACHE NIFI AND HDP SEARCH INTRODUCTION We will use Solr and the LucidWorks HDP Search to view our streamed data in real time to gather insights
More informationHadoop ecosystem. Nikos Parlavantzas
1 Hadoop ecosystem Nikos Parlavantzas Lecture overview 2 Objective Provide an overview of a selection of technologies in the Hadoop ecosystem Hadoop ecosystem 3 Hadoop ecosystem 4 Outline 5 HBase Hive
More informationHIVE MOCK TEST HIVE MOCK TEST III
http://www.tutorialspoint.com HIVE MOCK TEST Copyright tutorialspoint.com This section presents you various set of Mock Tests related to Hive. You can download these sample mock tests at your local machine
More informationAbout the Tutorial. Audience. Prerequisites. Copyright & Disclaimer. HCatalog
About the Tutorial HCatalog is a table storage management tool for Hadoop that exposes the tabular data of Hive metastore to other Hadoop applications. It enables users with different data processing tools
More informationWe are ready to serve Latest Testing Trends, Are you ready to learn?? New Batches Info
We are ready to serve Latest Testing Trends, Are you ready to learn?? New Batches Info START DATE : TIMINGS : DURATION : TYPE OF BATCH : FEE : FACULTY NAME : LAB TIMINGS : PH NO: 9963799240, 040-40025423
More information50 Must Read Hadoop Interview Questions & Answers
50 Must Read Hadoop Interview Questions & Answers Whizlabs Dec 29th, 2017 Big Data Are you planning to land a job with big data and data analytics? Are you worried about cracking the Hadoop job interview?
More informationProcessing big data with modern applications: Hadoop as DWH backend at Pro7. Dr. Kathrin Spreyer Big data engineer
Processing big data with modern applications: Hadoop as DWH backend at Pro7 Dr. Kathrin Spreyer Big data engineer GridKa School Karlsruhe, 02.09.2014 Outline 1. Relational DWH 2. Data integration with
More informationHortonworks Technical Preview for Stinger Phase 3 Released: 12/17/2013
Architecting the Future of Big Data Hortonworks Technical Preview for Stinger Phase 3 Released: 12/17/2013 Document Version 1.0 2013 Hortonworks Inc. All Rights Reserved. Architecting the Future of Big
More informationApache TM Hadoop TM - based Services for Windows Azure How- To and FAQ Guide
Apache TM Hadoop TM - based Services for Windows Azure How- To and FAQ Guide Welcome to Hadoop for Azure CTP How- To Guide 1. Setup your Hadoop on Azure cluster 2. How to run a job on Hadoop on Azure 3.
More informationImporting and Exporting Data Between Hadoop and MySQL
Importing and Exporting Data Between Hadoop and MySQL + 1 About me Sarah Sproehnle Former MySQL instructor Joined Cloudera in March 2010 sarah@cloudera.com 2 What is Hadoop? An open-source framework for
More informationPig A language for data processing in Hadoop
Pig A language for data processing in Hadoop Antonino Virgillito THE CONTRACTOR IS ACTING UNDER A FRAMEWORK CONTRACT CONCLUDED WITH THE COMMISSION Apache Pig: Introduction Tool for querying data on Hadoop
More informationBig Data Hadoop Stack
Big Data Hadoop Stack Lecture #1 Hadoop Beginnings What is Hadoop? Apache Hadoop is an open source software framework for storage and large scale processing of data-sets on clusters of commodity hardware
More informationArchitecture of Enterprise Applications 22 HBase & Hive
Architecture of Enterprise Applications 22 HBase & Hive Haopeng Chen REliable, INtelligent and Scalable Systems Group (REINS) Shanghai Jiao Tong University Shanghai, China http://reins.se.sjtu.edu.cn/~chenhp
More informationHortonworks Certified Developer (HDPCD Exam) Training Program
Hortonworks Certified Developer (HDPCD Exam) Training Program Having this badge on your resume can be your chance of standing out from the crowd. The HDP Certified Developer (HDPCD) exam is designed for
More informationAPACHE HIVE CIS 612 SUNNIE CHUNG
APACHE HIVE CIS 612 SUNNIE CHUNG APACHE HIVE IS Data warehouse infrastructure built on top of Hadoop enabling data summarization and ad-hoc queries. Initially developed by Facebook. Hive stores data in
More informationData Access 3. Migrating data. Date of Publish:
3 Migrating data Date of Publish: 2018-07-12 http://docs.hortonworks.com Contents Data migration to Apache Hive... 3 Moving data from databases to Apache Hive...3 Create a Sqoop import command...4 Import
More informationDATA MIGRATION METHODOLOGY FROM SQL TO COLUMN ORIENTED DATABASES (HBase)
1 DATA MIGRATION METHODOLOGY FROM SQL TO COLUMN ORIENTED DATABASES (HBase) Nosql No predefined schema/ less rigid schema Unstructured and constantly evolving data No Declarative Query Language Weaker Transactional
More informationTopics covered in this lecture
9/5/2018 CS435 Introduction to Big Data - FALL 2018 W3.B.0 CS435 Introduction to Big Data 9/5/2018 CS435 Introduction to Big Data - FALL 2018 W3.B.1 FAQs How does Hadoop mapreduce run the map instance?
More informationGuidelines For Hadoop and Spark Cluster Usage
Guidelines For Hadoop and Spark Cluster Usage Procedure to create an account in CSX. If you are taking a CS prefix course, you already have an account; to get an initial password created: 1. Login to https://cs.okstate.edu/pwreset
More informationOverview. : Cloudera Data Analyst Training. Course Outline :: Cloudera Data Analyst Training::
Module Title Duration : Cloudera Data Analyst Training : 4 days Overview Take your knowledge to the next level Cloudera University s four-day data analyst training course will teach you to apply traditional
More informationBlended Learning Outline: Developer Training for Apache Spark and Hadoop (180404a)
Blended Learning Outline: Developer Training for Apache Spark and Hadoop (180404a) Cloudera s Developer Training for Apache Spark and Hadoop delivers the key concepts and expertise need to develop high-performance
More informationLogging on to the Hadoop Cluster Nodes. To login to the Hadoop cluster in ROGER, a user needs to login to ROGER first, for example:
Hadoop User Guide Logging on to the Hadoop Cluster Nodes To login to the Hadoop cluster in ROGER, a user needs to login to ROGER first, for example: ssh username@roger-login.ncsa. illinois.edu after entering
More informationBig Data Analytics using Apache Hadoop and Spark with Scala
Big Data Analytics using Apache Hadoop and Spark with Scala Training Highlights : 80% of the training is with Practical Demo (On Custom Cloudera and Ubuntu Machines) 20% Theory Portion will be important
More informationHadoop An Overview. - Socrates CCDH
Hadoop An Overview - Socrates CCDH What is Big Data? Volume Not Gigabyte. Terabyte, Petabyte, Exabyte, Zettabyte - Due to handheld gadgets,and HD format images and videos - In total data, 90% of them collected
More informationCERTIFICATE IN SOFTWARE DEVELOPMENT LIFE CYCLE IN BIG DATA AND BUSINESS INTELLIGENCE (SDLC-BD & BI)
CERTIFICATE IN SOFTWARE DEVELOPMENT LIFE CYCLE IN BIG DATA AND BUSINESS INTELLIGENCE (SDLC-BD & BI) The Certificate in Software Development Life Cycle in BIGDATA, Business Intelligence and Tableau program
More informationsqoop Automatic database import Aaron Kimball Cloudera Inc. June 18, 2009
sqoop Automatic database import Aaron Kimball Cloudera Inc. June 18, 2009 The problem Structured data already captured in databases should be used with unstructured data in Hadoop Tedious glue code necessary
More informationRails on HBase. Zachary Pinter and Tony Hillerson RailsConf 2011
Rails on HBase Zachary Pinter and Tony Hillerson RailsConf 2011 What we will cover What is it? What are the tradeoffs that HBase makes? Why HBase is probably the wrong choice for your app Why HBase might
More informationHortonworks Data Platform
Hortonworks Data Platform Teradata Connector User Guide (April 3, 2017) docs.hortonworks.com Hortonworks Data Platform: Teradata Connector User Guide Copyright 2012-2017 Hortonworks, Inc. Some rights reserved.
More informationHortonworks Data Platform
Hortonworks Data Platform Workflow Management (August 31, 2017) docs.hortonworks.com Hortonworks Data Platform: Workflow Management Copyright 2012-2017 Hortonworks, Inc. Some rights reserved. The Hortonworks
More informationUNIT V BIG DATA FRAMEWORKS
UNIT V BIG DATA FRAMEWORKS Introduction to NoSQL Aggregate Data Models Hbase: Data Model and Implementations Hbase Clients Examples.Cassandra: Data Model Examples Cassandra Clients Hadoop Integration.
More informationHADOOP COURSE CONTENT (HADOOP-1.X, 2.X & 3.X) (Development, Administration & REAL TIME Projects Implementation)
HADOOP COURSE CONTENT (HADOOP-1.X, 2.X & 3.X) (Development, Administration & REAL TIME Projects Implementation) Introduction to BIGDATA and HADOOP What is Big Data? What is Hadoop? Relation between Big
More informationGetting Started with Hadoop/YARN
Getting Started with Hadoop/YARN Michael Völske 1 April 28, 2016 1 michael.voelske@uni-weimar.de Michael Völske Getting Started with Hadoop/YARN April 28, 2016 1 / 66 Outline Part One: Hadoop, HDFS, and
More informationHadoop: The Definitive Guide
THIRD EDITION Hadoop: The Definitive Guide Tom White Q'REILLY Beijing Cambridge Farnham Köln Sebastopol Tokyo labte of Contents Foreword Preface xv xvii 1. Meet Hadoop 1 Daw! 1 Data Storage and Analysis
More informationIntroduction to Apache Pig ja Hive
Introduction to Apache Pig ja Hive Pelle Jakovits 30 September, 2014, Tartu Outline Why Pig or Hive instead of MapReduce Apache Pig Pig Latin language Examples Architecture Hive Hive Query Language Examples
More informationHortonworks HDPCD. Hortonworks Data Platform Certified Developer. Download Full Version :
Hortonworks HDPCD Hortonworks Data Platform Certified Developer Download Full Version : https://killexams.com/pass4sure/exam-detail/hdpcd QUESTION: 97 You write MapReduce job to process 100 files in HDFS.
More informationA Glimpse of the Hadoop Echosystem
A Glimpse of the Hadoop Echosystem 1 Hadoop Echosystem A cluster is shared among several users in an organization Different services HDFS and MapReduce provide the lower layers of the infrastructures Other
More informationMapReduce Simplified Data Processing on Large Clusters
MapReduce Simplified Data Processing on Large Clusters Amir H. Payberah amir@sics.se Amirkabir University of Technology (Tehran Polytechnic) Amir H. Payberah (Tehran Polytechnic) MapReduce 1393/8/5 1 /
More informationBig Data Analysis using Hadoop. Lecture 4. Hadoop EcoSystem
Big Data Analysis using Hadoop Lecture 4 Hadoop EcoSystem Hadoop Ecosytems 1 Overview Hive HBase Sqoop Pig Mahoot / Spark / Flink / Storm Hive 2 Hive Data Warehousing Solution built on top of Hadoop Provides
More informationMap Reduce & Hadoop Recommended Text:
Map Reduce & Hadoop Recommended Text: Hadoop: The Definitive Guide Tom White O Reilly 2010 VMware Inc. All rights reserved Big Data! Large datasets are becoming more common The New York Stock Exchange
More informationPresented by Sunnie S Chung CIS 612
By Yasin N. Silva, Arizona State University Presented by Sunnie S Chung CIS 612 This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. See http://creativecommons.org/licenses/by-nc-sa/4.0/
More informationInstalling Hadoop. You need a *nix system (Linux, Mac OS X, ) with a working installation of Java 1.7, either OpenJDK or the Oracle JDK. See, e.g.
Big Data Computing Instructor: Prof. Irene Finocchi Master's Degree in Computer Science Academic Year 2013-2014, spring semester Installing Hadoop Emanuele Fusco (fusco@di.uniroma1.it) Prerequisites You
More informationBig Data and Hadoop. Course Curriculum: Your 10 Module Learning Plan. About Edureka
Course Curriculum: Your 10 Module Learning Plan Big Data and Hadoop About Edureka Edureka is a leading e-learning platform providing live instructor-led interactive online training. We cater to professionals
More informationThird Party Software: How can I import data from MySQL to Hadoop with Apache Sqoop? How can I import data from MySQL to Hadoop with Apache Sqoop?
How can I import data from MySQL to This is best explained with an example. Apache Sqoop is a data transfer tool used to move data between Hadoop and structured datastores. We will show how to "ingest"
More information