Using Hive for Data Warehousing

Size: px

Start display at page:

Download "Using Hive for Data Warehousing"

Dwight Campbell
5 years ago
Views:

1 An IBM Proof of Technology Using Hive for Data Warehousing Unit 3: Hive DML in action

2 An IBM Proof of Technology Catalog Number Copyright IBM Corporation, 2013 US Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

3 Contents LAB 1 HIVE DML IN ACTION ACCESSING THE HIVE CLI SAMPLE DATA DESCRIPTIONS LOADING DATA VIEWING DATA FROM BIGINSIGHTS WEB CONSOLE LOADING DATA INTO THE MANAGED NON-PARTITIONED TABLES LOADING DATA INTO THE MANAGED PARTITIONED TABLE RUNNING QUERIES SELECTING DATA TAKING ADVANTAGE OF PARTITIONED DATA JOINS VIEWS EXPORTING DATA EXPLAIN SUMMARY Contents Page 3

4 Lab 1 Hive DML in action Now that we have learned how to create databases, tables, and partitions in Hive, we are ready to work with Hive s Data Manipulation Language and get some work done! Let s load some data and start writing queries. After completing this hands-on lab, you will be able to: Load data into Hive tables and partitions. Run a variety of HiveQL queries against your data. Take advantage of partitioning to speed up queries. Use Views to reduce the complexity of your queries. Export data out of Hive. Use Explain to learn more about your queries. Allow 45 minutes to 1 hour to complete this section of lab. This version of the lab was designed using the InfoSphere BigInsights 2.1 Quick Start Edition. Throughout this lab you will be using the following account login information: Username Password VM image setup screen root password Linux biadmin biadmin If you are continuing this series of hands on labs immediately after completing Using Hive for Data Warehousing Unit 2: Working with Hive DDL, you may move on to section 1.1 of this lab. Otherwise please complete the prior labs to get started. (All Hadoop components should be running and you should have already created the databases, tables and partitions that will be used in this unit.) Page 4 Using Hive

5 1.1 Accessing the Hive CLI In this section we will navigate to the Hive CLI and start an interactive Hive CLI session. 1. Open the Linux terminal by double clicking the Terminal icon from within the BigInsights Shell directory on the desktop. 2. In the Linux terminal change to the Hive home bin directory $ cd $HIVE_HOME/bin 3. Start an interactive Hive shell session. $./hive 4. Tell Hive to Use the computersalesdb (we will use this database for the rest of this interactive session). hive> USE computersalesdb; Sample Data Descriptions Before we begin working in Hive, review the data we will be using. It may be useful to print these pages so you can refer to the data when running your queries. Our sample data is from a fictitious computer retailer. The company sells computer parts and generally serves a single State in the country. Hands-on-Lab Page 5

6 Customer.csv: Purpose: Hold customer records. Columns: FNAME LNAME STATUS TELNO CUSTOMER_ID CITY ZIP Customer s First Name Customer s Last Name Active or Inactive status Telephone # Customer s unique ID City and Zip code separated by the character. Example of contents: Page 6 Using Hive

7 Product.csv: Purpose: Hold product records. Columns: PROD_ NAME DESCRIPTION CATEGORY QTY_ON_HAND PROD_ NUM PACKAGED_ WITH Name of product Description of computer product Category product belongs to Quantity of product in warehouse Unique product number Colon separated list of things that come in package with product. Example of contents: Hands-on-Lab Page 7

8 Sales.csv: Purpose: Holds all historical sales records. Company updates once a month. Columns: CUST_ID PROD_NUM QTY DATE SALES_ID ID of customer who made purchase ID of product that was purchased QTY purchased Date of sale Unique sale ID Example of contents: Page 8 Using Hive

1.2 Loading Data In our previous lab, we created 4 new tables. These tables need to have data in them to be useful to us. The table names are customer, products, sales_staging, and sales.

9 1.2 Loading Data In our previous lab, we created 4 new tables. These tables need to have data in them to be useful to us. The table names are customer, products, sales_staging, and sales. Remember that the customer table is an External table and we already loaded data into it Viewing Data from BigInsights Web Console In this lab, we will take advantage of the BigInsights Web Console to view our data in HDFS. 1. Open the BigInsights Web Console by clicking the desktop shortcut. Login if required. 2. Click the Files tab at the top. In the left navigational menu, navigate to hdfs:/bivm:9000/- >biginsights->hive->warehouse->computersalesdb.db This is the path that all of our Hive managed tables will be down. Keep this browser open as you work with the Hive CLI. It will allow us to quickly check HDFS and see changes to the files and directories as we work with the Hive DML. Hands-on-Lab Page 9

10 1.2.2 Loading Data into the Managed Non-Partitioned Tables The first table we created in the previous lab was the products table. This table is fully managed by Hive and does not contain any partitions. We will now load the products table with the data that is stored in the local /home/biadmin/sampledata/computer_business/withoutheaders/product.csv file. 1. In the CLI, create the new Products table in Hive. hive> LOAD DATA LOCAL INPATH /home/biadmin/sampledata/computer_business/withoutheaders/product.csv' OVERWRITE INTO TABLE products; Hive copies the data from the file. Our products table now has data in it. Let s check it out! 2. Using the BigInsights Web Console, navigate to hdfs:/bivm:9000/->biginsights->hive- >warehouse->computersalesdb.db->products directory. Click on the Products.csv file within that directory. Page 10 Using Hive

We can easily view the contents of the Products.csv file here. Look at the data displayed in the right hand side of the browser. Our products table is now loaded! 3.

11 We can easily view the contents of the Products.csv file here. Look at the data displayed in the right hand side of the browser. Our products table is now loaded! 3. Next we will load sales records into the sales_staging table. hive> LOAD DATA LOCAL INPATH /home/biadmin/sampledata/computer_business/withoutheaders/sales.csv' INTO TABLE sales_staging; Notice we left off the OVERWRITE keyword in this statement. This is because we normally would want to add the monthly sales data to our historical list of records already in the sales_staging table. We would not want to overwrite all the records in that table with just this month s data. 4. Again, let s verify that our data is now in HDFS, by checking out the file in the BigInsights Web Console. First click the refresh button to update the navigational tree. Hands-on-Lab Page 11

Now open the sales_staging folder, click on the Sales.csv file and browse the data on the right hand side. Looks like a success. 1.2.

12 Now open the sales_staging folder, click on the Sales.csv file and browse the data on the right hand side. Looks like a success Loading Data into the Managed Partitioned Table Now that our sales_staging table has data we can work with, let s write some queries that will allow us to load our partitioned sales table (partitioned on sales_date) with data coming from sales_staging. 1. In the CLI, load sales data from into a partition of the sales table. hive> INSERT OVERWRITE TABLE sales PARTITION (sales_date = ) SELECT cust_id, prod_num, qty, sales_id FROM sales_staging ss WHERE ss.sale_date = ; Page 12 Using Hive

13 There is a lot of output there. It is because this is the first statement we ve run that has actually invoked a MapReduce job (thanks to the WHERE clause)! Carefully investigate the output. 2. In the BigInsights Web Console, refresh the file tree and take a look in the sales folder now. You will see a new subdirectory named sales_date= Within that directory is a data file named _0. If you select that file you will see in the right hand preview that it only contains the data for our sales. Hands-on-Lab Page 13

3. Following the same procedures, load sales data from 2012-01-24 into a new partition of the sales table.

14 3. Following the same procedures, load sales data from into a new partition of the sales table. hive> INSERT OVERWRITE TABLE sales PARTITION (sales_date = ) SELECT cust_id, prod_num, qty, sales_id FROM sales_staging ss WHERE ss.sale_date = ; Page 14 Using Hive

4. In the BigInsights Web Console, refresh the file tree and take a look in the sales folder now. You will see a new subdirectory named sales_date=2012-01-24.

15 4. In the BigInsights Web Console, refresh the file tree and take a look in the sales folder now. You will see a new subdirectory named sales_date= Within that directory is a data file named _0. If you select that file you will see in the right hand preview that it only contains the data for our sales. Hands-on-Lab Page 15

16 1.3 Running Queries Selecting Data Now that our tables have data in them, let s start running queries against that data. 1. In the CLI, select all the data in the products table where the product category is Video. hive> SELECT * FROM products WHERE category= Video ; This query resulted in one MapReduce job being run. It took 17 seconds to run this simple query. The results were also printed to the CLI. The 3 records that have the category= Video are in fact returned! 2. Now select all the products where category= Video AND the first element of the PACKAGED_WITH array contains dvd. hive> SELECT * FROM products WHERE category= Video AND PACKAGED_WITH[0]= dvd ; Page 16 Using Hive

17 3. Find out how many products we have for each category of item by using the GROUP BY clause. hive> SELECT category, count(*) FROM products GROUP BY category; Note the screen capture only shows the last part of Hive s output. We can see our results and they look good. 4. Use a nested select to show the product categories that contain more than 4 products. hive> FROM( SELECT category, count(*) as count FROM products GROUP BY category) cats SELECT * WHERE cats.count > 3; Hands-on-Lab Page 17

18 The end of your output should like similar to the above screen capture. You will notice that in addition to using a subquery we also used two column aliases ( count and cats ) Taking Advantage of Partitioned Data The sales table is partitioned on sales_date. In a previous exercise we loaded data into two partitions of this table ( and partitions). Let s take advantage of this partitioning to improve latency. 1. In the CLI, run a SELECT query that finds only the sales that occurred on Order the results by the sale_id. hive> SELECT * FROM sales WHERE sales_date = ORDER BY sales_id; Just the sales records were returned. One MapReduce Job was required to complete this query. Only the sales_id= partition file had to be read in to complete this query, saving us some wait time theoretically a lot of wait time if we had a large number of historical sales data. Page 18 Using Hive

19 2. Let s take a look at the job that was run on Hadoop to fulfill our Hive query. We can do this easily back in the BigInsights Web Console. In the web console, click the Application Status tab, then click on the Jobs link (the proper places to click are highlighted in red in the screen capture below). You will see our job, named after our query click on that and investigate the job Joins 1. Let s show all Optical category sales that occurred on We will need to do an equijoin between the sales and products tables to gather this information. hive> SELECT s.cust_id, s.prod_num, s.qty, s.sales_id, p.prod_name, p.category FROM sales s JOIN products p ON s.prod_num = p.prod_num WHERE s.sales_date = AND p.category = Optical ; Hands-on-Lab Page 19

20 The join was successful. We got the 3 records we were looking for. This Hive query created 1 MapReduce job that had 2 Mappers and 1 Reducer Views 1. Now let s create a View that stores a query which returns all the sales records (joined with products table) where the product category is Optical. hive> CREATE VIEW optical_sales AS SELECT s.cust_id, s.prod_num, s.qty, s.sales_id, p.prod_name, p.category FROM sales s JOIN products p ON s.prod_num = p.prod_num WHERE p.category = Optical ; The View was successfully created. 2. Now that we have the optical_sales view, we can use it within other queries, just like it were a table. Notice how short this query is. hive> SELECT * FROM optical_sales WHERE qty > 1; Page 20 Using Hive

1.4 Exporting Data Imagine our management team wants us to extract all sales of Optical devices and give it to them in a format outside of the Hive CLI.

21 1.4 Exporting Data Imagine our management team wants us to extract all sales of Optical devices and give it to them in a format outside of the Hive CLI. We can do this by exporting the data to our local file system. 1. Open a new Linux terminal by double clicking the Terminal icon from within the BigInsights Shell directory on the desktop. 2. Create a new directory called hive_output down the /home/biadmin path. Then cd into this new directory. $ mkdir /home/biadmin/hive_output $ cd /home/biadmin/hive_output 3. Back in the Hive CLI, we will utilize our previous query and write the data to the /home/biadmin/hive_output directory on our local Linux file system. hive> INSERT OVERWRITE LOCAL DIRECTORY /home/biadmin/hive_output SELECT * FROM optical_sales WHERE qty > 1; Hive tells us that our results have been copied to the /home/biadmin/hive_output directory and that the job was a success. Hands-on-Lab Page 21

22 4. Click the biadmin s Home shortcut on the Linux desktop. Then open the hive_output folder. Notice we have one new file called _0! This is the data that Hive wrote out. Let s open this file in the text editor and verify the data is there. If you were to open the file up in a text editor like nano, you d see something a bit different. Page 22 Using Hive

Notice the ^A that is used by hive as a delimiter to separate the column data. You may want to massage the data before you send the report to the management team! 1.

23 Notice the ^A that is used by hive as a delimiter to separate the column data. You may want to massage the data before you send the report to the management team! 1.5 Explain Let s take a brief look at using EXPLAIN in a Hive query. 1. We will have Hive explain the execution plan for a simple query that selects all customer records where the customer status is active. hive> EXPLAIN SELECT * FROM customer; Can you tell how many MapReduce jobs would be required to run this Hive query? If you guessed none, you would be correct. Hive is able to read the records and dump the output to the console without using MapReduce. Hive is using local mode to do this. 1.6 Summary Congratulations! You now know how to load data into Hive. You are able to run a variety of queries using familiar clauses such as SELECT, WHERE, GROUP BY, ORDER BY, JOIN, and more. You can export data out of Hive and even run the Explain tool to get an execution plan for your query. You may move on to the next Unit. Hands-on-Lab Page 23

24 NOTES

25 NOTES

Copyright IBM Corporation 2013. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any kind, express or implied.

26 Copyright IBM Corporation The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, these materials. Nothing contained in these materials is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software. References in these materials to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. This information is based on current IBM product plans and strategy, which are subject to change by IBM without notice. Product release dates and/or capabilities referenced in these materials may change at any time at IBM s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. IBM, the IBM logo and ibm.com are trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at Copyright and trademark information at

Accessing Hadoop Data Using Hive

An IBM Proof of Technology Accessing Hadoop Data Using Hive Unit 3: Hive DML in action An IBM Proof of Technology Catalog Number Copyright IBM Corporation, 2015 US Government Users Restricted Rights -