Accessing Hadoop Data Using Hive

Size: px

Start display at page:

Download "Accessing Hadoop Data Using Hive"

Veronica Bennett
6 years ago
Views:

1 An IBM Proof of Technology Accessing Hadoop Data Using Hive Unit 3: Hive DML in action

2 An IBM Proof of Technology Catalog Number Copyright IBM Corporation, 2015 US Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

3 Contents LAB 3 HIVE DML IN ACTION SETTING HDFS USER PERMISSIONS ACCESSING THE HIVE CLI SAMPLE DATA DESCRIPTIONS LOADING DATA HDFS DATA FROM THE LINUX CONSOLE LOADING DATA INTO THE MANAGED NON-PARTITIONED TABLES LOADING DATA INTO THE MANAGED PARTITIONED TABLE RUNNING QUERIES SELECTING DATA TAKING ADVANTAGE OF PARTITIONED DATA JOINS VIEWS EXPORTING DATA EXPLAIN SUMMARY Contents Page 3

4 Lab 3 Hive DML in action Now that we have learned how to create databases, tables, and partitions in Hive, we are ready to work with Hive s Data Manipulation Language and get some work done! Let s load some data and start writing queries. After completing this hands-on lab, you will be able to: Load data into Hive tables and partitions. Run a variety of HiveQL queries against your data. Take advantage of partitioning to speed up queries. Use Views to reduce the complexity of your queries. Export data out of Hive. Use Explain to learn more about your queries. Allow 45 minutes to 1 hour to complete this section of lab. This version of the lab was designed using the IBM BigInsights 4.1 Quick Start Edition. Throughout this lab it is assumed that you will be using the following account login information: Username Password VM image setup screen root password Ambari admin admin If you are continuing this series of hands on labs immediately after completing Accessing Hadoop Data Using Hive Unit 2: Working with Hive DDL, you may move on to section 1.1 of this lab. Otherwise please complete the prior labs to get started. (All Hadoop components should be running and you should have already created the databases, tables and partitions that will be used in this unit.) Page 4 Hive DML

5 1.1 Setting HDFS User Permissions Before we continue, we will make an ownership change to one of the directories on HDFS that Hive uses. This will allow us to continue the rest of the lab without having to worry about the details of Hadoop permissions. 1. We will execute a few commands: $ su - hdfs $ hadoop fs chmod R 777 / Once you run those commands enter the exit command to return back to the virtuser id. $ exit Now you can continue to the next step of the lab. Possible error if this section is ignored (if we don t specify permissions). Hands-on-Lab Page 5

Connect to Hive. beeline>!connect jdbc:hive2://rvm.svl.ibm.com:10000 virtuser password org.apache.hive.jdbc.hivedriver 5.

6 1.2 Accessing the Hive CLI In this section we will navigate to the Hive Beeline CLI and start an interactive CLI session. 2. In a Linux terminal change to the Hive bin directory $ cd /usr/iop/ /hive/bin 3. Start an interactive Hive shell session. $./beeline 4. Connect to Hive. beeline>!connect jdbc:hive2://rvm.svl.ibm.com:10000 virtuser password org.apache.hive.jdbc.hivedriver 5. Tell Hive to use the computersalesdb (we will use this database for the rest of this interactive session). hive> USE computersalesdb; Note: Please run these commands whenever you exit and re-enter Hive. Page 6 Hive DML

1.2.1 Sample Data Descriptions Before we begin working in Hive, review the data we will be using. It may be useful to print these pages so you can refer to the data when running your queries.

7 1.2.1 Sample Data Descriptions Before we begin working in Hive, review the data we will be using. It may be useful to print these pages so you can refer to the data when running your queries. Our sample data is from a fictitious computer retailer. The company sells computer parts and generally serves a single State in the country. Customer.csv: Purpose: Hold customer records. Columns: FNAME LNAME STATUS TELNO CUSTOMER_ID CITY ZIP Customer s First Name Customer s Last Name Active or Inactive status Telephone # Customer s unique ID City and Zip code separated by the character. Example of contents: Hands-on-Lab Page 7

8 Product.csv: Purpose: Hold product records. Columns: PROD_ NAME DESCRIPTION CATEGORY QTY_ON_HAND PROD_ NUM PACKAGED_ WITH Name of product Description of computer product Category product belongs to Quantity of product in warehouse Unique product number Colon separated list of things that come in package with product. Example of contents: Page 8 Hive DML

9 Sales.csv: Purpose: Holds all historical sales records. Company updates once a month. Columns: CUST_ID PROD_NUM QTY DATE SALES_ID ID of customer who made purchase ID of product that was purchased QTY purchased Date of sale Unique sale ID Example of contents: 1.3 Loading Data In our previous lab, we created 4 new tables. These tables need to have data in them to be useful to us. The table names are customer, products, sales_staging, and sales. Remember that the customer table is an External table and we already loaded data into it. Hands-on-Lab Page 9

1.3.1 HDFS Data from the Linux Console In this lab, we use the Linux console along with HDFS commands to view our data in HDFS. 1. Let s see what the /apps/hive/warehouse/computersalesdb.

10 1.3.1 HDFS Data from the Linux Console In this lab, we use the Linux console along with HDFS commands to view our data in HDFS. 1. Let s see what the /apps/hive/warehouse/computersalesdb.db directory currently contains on HDFS. This is the path of the directory that all of our Hive managed tables will be stored in. $ hadoop fs -ls /apps/hive/warehouse/computersalesdb.db 2. Now we will copy the rest of our data files to a location on HDFS. First let s make a new directory to put the data files in. Let s create this /tmp/tempdata directory on HDFS. We will then load our tables with this data in the next section of the lab. $ hadoop fs -mkdir /tmp/tempdata; Next to make things easy, let s just copy our whole local WithoutHeaders directory right into the /tmp/tempdata directory we just created on HDFS. $ hadoop fs -put /mnt/hgfs/labfiles/computer_business/withoutheaders/ /tmp/tempdata; Then list out the contents of the new directory on HDFS $ hadoop fs -ls /tmp/tempdata/withoutheaders Page 10 Hive DML

Now go into HDFS and run step 1.1 Setting HDFS User Permissions so we can write into the folder. 1.3.

11 Now go into HDFS and run step 1.1 Setting HDFS User Permissions so we can write into the folder Loading Data into the Managed Non-Partitioned Tables The first table we created in the previous lab was the products table. This table is fully managed by Hive and does not contain any partitions. We will now load the products table with the data that is stored in the local /tmp/tempdata/withoutheaders/product.csv file on HDFS. 1. In the CLI, create the new Products table in Hive. hive> LOAD DATA INPATH '/tmp/tempdata/withoutheaders/product.csv' OVERWRITE INTO TABLE products; Hive copies the data from the file. Our products table now has data in it. Let s check it out! Hands-on-Lab Page 11

2. Quit out of Hive and list out the contents of the /apps/hive/warehouse/computersalesdb.db/products directory. You will see the Product.csv file is there within that directory.

12 2. Quit out of Hive and list out the contents of the /apps/hive/warehouse/computersalesdb.db/products directory. You will see the Product.csv file is there within that directory. $ hadoop fs -ls /apps/hive/warehouse/computersalesdb.db/products We can easily view the contents of the Product.csv file by running the Hadoop fs -cat command. $ hadoop fs -cat /apps/hive/warehouse/computersalesdb.db/products/product.csv Look at the data displayed - our products table is now loaded into Hive! 3. Next we will load sales records into the sales_staging table. hive> LOAD DATA INPATH '/tmp/tempdata/withoutheaders/sales.csv' INTO TABLE sales_staging; Page 12 Hive DML

Notice we left off the OVERWRITE keyword in this statement. This is because we normally would want to add the monthly sales data to our historical list of records already in the sales_staging table.

13 Notice we left off the OVERWRITE keyword in this statement. This is because we normally would want to add the monthly sales data to our historical list of records already in the sales_staging table. We would not want to overwrite all the records in that table with just this month s data. 4. Again, let s verify that our data is now within the Hive warehouse on HDFS, by checking out the file in the Linux console using the hdfs fs cat command. $ hadoop fs -cat /apps/hive/warehouse/computersalesdb.db/sales_staging/sales.csv Hands-on-Lab Page 13

14 Looks like a success! Loading Data into the Managed Partitioned Table Now that our sales_staging table has data we can work with, let s write some queries that will allow us to load our partitioned sales table (partitioned on sales_date) with data coming from sales_staging. 1. In the Beeline CLI, load sales data from into a partition of the sales table. hive> INSERT OVERWRITE TABLE sales PARTITION (sales_date = ' ') SELECT cust_id, prod_num, qty, sales_id FROM sales_staging ss WHERE ss.sale_date = ' '; Page 14 Hive DML

15 Notice it took a little while for this to complete. This is because you ve invoked a MapReduce job (thanks to the WHERE clause)! Also make sure that you have HDFS, MapReduce2, and Hive all running on Ambari, or this query will NOT work. 2. In a Linux Console, do a hadoop fs ls command and take a look in the sales folder now. You will see a new subdirectory named sales_date= Within that directory is a data file named _0. If you cat that file you will see that it only contains the data for our sales. $ hadoop fs ls /apps/hive/warehouse/computersalesdb.db/sales $ hadoop fs ls /apps/hive/warehouse/computersalesdb.db/sales/sales_date= $ hadoop fs cat /apps/hive/warehouse/computersalesdb.db/sales/sales_date= /000000_0 3. Following the same procedures, load sales data from into a new partition of the sales table. Hands-on-Lab Page 15

hive> INSERT OVERWRITE TABLE sales PARTITION (sales_date = '2012-01-24') SELECT cust_id, prod_num, qty, sales_id FROM sales_staging ss WHERE ss.

You will see a new subdirectory named sales_date=2012-01-24. Within that directory is a data file named 000000_0.

16 hive> INSERT OVERWRITE TABLE sales PARTITION (sales_date = ' ') SELECT cust_id, prod_num, qty, sales_id FROM sales_staging ss WHERE ss.sale_date = ' '; 4. In a Linux Console, do a hadoop fs -ls command and take a look in the sales folder now. You will see a new subdirectory named sales_date= Within that directory is a data file named _0. If you cat that file you will see that it only contains the data for our sales. $ hadoop fs ls /apps/hive/warehouse/computersalesdb.db/sales $ hadoop fs ls /apps/hive/warehouse/computersalesdb.db/sales/sales_date= Page 16 Hive DML

17 $ hadoop fs cat /apps/hive/warehouse/computersalesdb.db/sales/sales_date= /000000_0 1.4 Running Queries Selecting Data Now that our tables have data in them, let s start running queries against that data. 1. In the CLI, select all the data in the products table where the product category is Video. Order the results by prod_num. hive> SELECT * FROM products WHERE category= 'Video' ORDER BY prod_num; Hands-on-Lab Page 17

18 Now open up Ambari in the web browser. Click MapReduce2 in the left hand menu. Then select JobHistory UI from the Quick Links drop down menu. Page 18 Hive DML

Note: If opening the job history doesn t work this way, try opening a new tab and typing rvm.svl.ibm.com:19888/jobhistory Inside the job history table, you will notice the job that Hive just ran.

You can click on the Job ID for more details. Hive printed the query results in the CLI. The 3 records that have the category= Video are in fact returned back in the CLI! 2.

19 Note: If opening the job history doesn t work this way, try opening a new tab and typing rvm.svl.ibm.com:19888/jobhistory Inside the job history table, you will notice the job that Hive just ran. Our Hive query (first row) resulted in one MapReduce job being run. The MapReduce job required one Mapper and one Reducer. It took about 47 seconds to run the query. You can click on the Job ID for more details. Hive printed the query results in the CLI. The 3 records that have the category= Video are in fact returned back in the CLI! 2. Now select all the products where category= Video AND the first element of the PACKAGED_WITH array contains dvd. hive> SELECT * FROM products WHERE category= 'Video' AND PACKAGED_WITH[0]= 'dvd'; 3. Find out how many products we have for each category of item by using the GROUP BY clause. hive> SELECT category, count(*) FROM products GROUP BY category; Hands-on-Lab Page 19

hive> FROM( SELECT category, count(*) as count FROM products GROUP BY category) cats SELECT * WHERE cats.

20 We can see our results and they look good. 4. Use a nested select to show the product categories that contain more than 3 products. hive> FROM( SELECT category, count(*) as count FROM products GROUP BY category) cats SELECT * WHERE cats.count > 3; Your output should like similar to the above screen capture. You will notice that in addition to using a subquery we also used two column aliases ( count and cats ) Page 20 Hive DML

1.4.2 Taking Advantage of Partitioned Data The sales table is partitioned on sales_date. In a previous exercise we loaded data into two partitions of this table (2012-01-09 and 2012-01-24 partitions).

21 1.4.2 Taking Advantage of Partitioned Data The sales table is partitioned on sales_date. In a previous exercise we loaded data into two partitions of this table ( and partitions). Let s take advantage of this partitioning to improve latency. 1. In the CLI, run a SELECT query that finds only the sales that occurred on Order the results by the sale_id. hive> SELECT * FROM sales WHERE sales_date = ' ' ORDER BY sales_id; Hands-on-Lab Page 21

Just the 2012-01-24 sales records were returned. Let s take a look at the job that was run on Hadoop to fulfill our Hive query. We can do this easily back on the Hadoop JobHistory page.

22 Just the sales records were returned. Let s take a look at the job that was run on Hadoop to fulfill our Hive query. We can do this easily back on the Hadoop JobHistory page. You will see our job, named after our query, highlighted in red below: One MapReduce Job was required to complete this query. This job contained one Map task and one Reduce task. Only the sales_id= partition file had to be read in to complete this query, saving us some wait time theoretically a lot of wait time if we had a large number of historical sales data Joins 1. Let s show all Optical category sales that occurred on We will need to do an equijoin between the sales and products tables to gather this information. hive> SELECT s.cust_id, s.prod_num, s.qty, s.sales_id, p.prod_name, p.category FROM sales s JOIN products p ON s.prod_num = p.prod_num WHERE s.sales_date = ' ' AND p.category = 'Optical'; Page 22 Hive DML

The join was successful. We got the 3 records we were looking for. 1.4.4 Views 1.

hive> CREATE VIEW optical_sales AS SELECT s.cust_id, s.prod_num, s.qty, s.sales_id, p.prod_name, p.category FROM sales s JOIN products p ON s.prod_num = p.

23 The join was successful. We got the 3 records we were looking for Views 1. Now let s create a View that stores a query which returns all the sales records (joined with products table) where the product category is Optical. hive> CREATE VIEW optical_sales AS SELECT s.cust_id, s.prod_num, s.qty, s.sales_id, p.prod_name, p.category FROM sales s JOIN products p ON s.prod_num = p.prod_num WHERE p.category = 'Optical'; The View was successfully created. 2. Now that we have the optical_sales view, we can use it within other queries, just like it were a table. Notice how short this query is. hive> SELECT * FROM optical_sales WHERE qty > 1; Hands-on-Lab Page 23

1.5 Exporting Data Imagine our management team wants us to extract all sales of Optical devices and give it to them in a format outside of the Hive CLI. We can do this by exporting the data from Hive.

24 1.5 Exporting Data Imagine our management team wants us to extract all sales of Optical devices and give it to them in a format outside of the Hive CLI. We can do this by exporting the data from Hive. 1. Exit out of Hive. We are going to create a new directory on HDFS called reports. Create this directory within the /tmp directory on HDFS. $ hadoop fs -mkdir /tmp/reports; 2. Make this new folder writable by all users. hadoop fs chmod R 777 /tmp/reports 3. Back in the Hive CLI, we will utilize our previous query and write the data to the /tmp/reports directory on the HDFS file system. hive> INSERT OVERWRITE DIRECTORY '/tmp/reports' SELECT * FROM optical_sales WHERE qty > 1; Page 24 Hive DML

4. Exit out of Hive. Enter the following command: $ hadoop fs ls '/tmp/reports' Notice we have one new file called 000000_0! This is the data that Hive wrote out.

25 4. Exit out of Hive. Enter the following command: $ hadoop fs ls '/tmp/reports' Notice we have one new file called _0! This is the data that Hive wrote out. Now let s cat this file and see what the contents look like: $ hadoop fs cat '/tmp/reports/000000_0' You could copy the new report out of HDFS and work with the results in the tools of your choice. 1.6 Explain Let s take a brief look at using EXPLAIN in a Hive query. 1. We will have Hive explain the execution plan for a simple query that selects all customer records. Hands-on-Lab Page 25

26 hive> EXPLAIN SELECT * FROM customer; Page 26 Hive DML

27 Can you tell how many MapReduce jobs would be required to run this Hive query? If you guessed none, you would be correct. Hive is able to read the records and dump the output to the console without using MapReduce. Hive is using local mode to do this. 1.7 Summary Congratulations! You now know how to load data into Hive. You are able to run a variety of queries using familiar clauses such as SELECT, WHERE, GROUP BY, ORDER BY, JOIN, and more. You can export data out of Hive and even run the Explain tool to get an execution plan for your query. You may move on to the next Unit. Hands-on-Lab Page 27

28 NOTES

29 NOTES

Copyright IBM Corporation 2015. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any kind, express or implied.

30 Copyright IBM Corporation The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, these materials. Nothing contained in these materials is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software. References in these materials to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. This information is based on current IBM product plans and strategy, which are subject to change by IBM without notice. Product release dates and/or capabilities referenced in these materials may change at any time at IBM s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. IBM, the IBM logo and ibm.com are trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at Copyright and trademark information at

Using Hive for Data Warehousing

An IBM Proof of Technology Using Hive for Data Warehousing Unit 3: Hive DML in action An IBM Proof of Technology Catalog Number Copyright IBM Corporation, 2013 US Government Users Restricted Rights - Use,