Using Hive for Data Warehousing

Size: px

Start display at page:

Download "Using Hive for Data Warehousing"

Shawn Shepherd
6 years ago
Views:

1 An IBM Proof of Technology Using Hive for Data Warehousing Unit 5: Hive Storage Formats

2 An IBM Proof of Technology Catalog Number Copyright IBM Corporation, 2013 US Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

3 Contents LAB 5 HIVE STORAGE FORMATS ACCESSING THE HIVE CLI WORKING WITH DIFFERENT FILE AND RECORD FORMATS SEQUENCEFILE RCFILE ORC FILE COMPRESSION SUMMARY Contents Page 3

4 Lab 5 Hive Storage Formats We can use Hive to work with a variety of different file and record formats. Understanding how to work with formats beyond the simple TextFile will empower you to make better decisions when working with Hive and Hadoop. We can also use compression with Hive to reduce the footprint of our data potentially buying us performance increases and/or storage improvement. After completing this hands-on lab, you will be able to: Create SequenceFiles, RCFiles and ORC Files. Explicitly state Input/Output formats and SerDe in a CREATE TABLE statement. Turn output compression on in Hive. Allow 45 minutes to complete this section of lab. This version of the lab was designed using the InfoSphere BigInsights 3.0 Quick Start Edition. Throughout this lab you will be using the following account login information: Username Password VM image setup screen root password Linux biadmin biadmin If you are continuing this series of hands on labs immediately after completing Accessing Hadoop Data Using Hive Unit 4: Hive Operators and Functions, you may move on to section 1.1 of this lab. Otherwise please complete the prior labs to get started. (All Hadoop components should be running and you should have already created the databases, tables and partitions that will be used in this unit.) Page 4 Hive Storage Formats

1.1 Accessing the Hive CLI In this section we will navigate to the Hive Beeline CLI, just as we have done in previous labs, and start an interactive Hive CLI session. 1.

5 1.1 Accessing the Hive CLI In this section we will navigate to the Hive Beeline CLI, just as we have done in previous labs, and start an interactive Hive CLI session. 1. Open the Linux terminal by double clicking the Terminal icon from within the BigInsights Shell directory on the desktop. 2. In the Linux terminal change to the Hive home bin directory ~> cd $HIVE_HOME/bin 3. Start an interactive Hive shell session. ~>./beeline u jdbc:hive2://bivm.ibm.com:10000 n biadmin p biadmin 4. Tell Hive to Use the computersalesdb (we will use this database for the rest of this interactive session). hive> USE computersalesdb; 1.2 Working with Different File and Record Formats In this section we will experiment with different file and record formats in Hive. We already created multiple tables on our system that are in TEXTFILE format. Let s take advantage of this fact and create some new tables of different formats then insert data into the new tables by running SELECT queries on the originals. Hands-on-Lab Page 5

6 1.2.1 SequenceFile In a previous lab you created the products table and described the table with the STORED AS TEXTFILE clause. Now we will create a new table called products_sequenceformat that will be stored as a SEQUENCEFILE. We will have Hive do the conversion from TEXTFILE to SEQUENCEFILE for us by simple running an INSERT statement. 1. Create a new table in Hive called products_sequenceformat. We will use the exact same DDL we used to create the original products table EXCEPT we will change the STORED AS clause to be SEQUENCEFILE instead of TEXTFILE. hive> CREATE TABLE products_sequenceformat ( prod_name STRING, description STRING, category STRING, qty_on_hand INT, prod_num STRING, packaged_with ARRAY<STRING> ) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' COLLECTION ITEMS TERMINATED BY ':' STORED AS SEQUENCEFILE; 2. Run the SHOW TABLES command to verify the table has been created. hive> SHOW TABLES; Page 6 Hive Storage Formats

7 3. Set Hive s outputformat to vertical, then ask Hive to give us the EXTENDED details of our new products_sequenceformat table. hive>!set outputformat vertical hive> DESCRIBE EXTENDED products_sequenceformat; You ll notice some interesting things listed here: InputFormat is: org.apache.hadoop.mapred.sequencefileinputformat OutputFormat is: org.apache.hadoop.hive.ql.io.hivesequencefileoutputformat SerDe serializationlib is: org.apache.hadoop.hive.serde2.lazy.lazysimpleserde The HDFS location of the data for this table is: /biginsights/hive/warehouse/computersalesdb.db/products_sequenceformat It s important to recognize that using the STORED AS SEQUENCEFILE clause in our create table statement set the Input and Output formats behind the scenes appropriately for us. We didn t have to explicitly specify those values when creating the table (though we could have). 4. Time to load our products_sequenceformat table with data. We will do this by running an INSERT query. hive> FROM products INSERT OVERWRITE TABLE products_sequenceformat SELECT *; Hands-on-Lab Page 7

Behind the scenes a temp space is being used to store the intermediate data. 5. Let s examine the products_sequenceformat table on HDFS and see what Hive did for us.

8 Behind the scenes a temp space is being used to store the intermediate data. 5. Let s examine the products_sequenceformat table on HDFS and see what Hive did for us. Open the BigInsights Web Console and navigate to the /biginsights/hive/warehouse/computersalesdb.db/products_sequenceformat directory. You will find a _0 file. If you click on this file, the data will be displayed in the right hand viewing pane. That doesn t look like a regular TEXTFILE does it? Some of the characters are displayed oddly. That s a good thing we are looking at a SEQUENCEFILE here! 6. Let s try something different. Open a new Linux terminal. We can use the text option in the HDFS fs command to see the data in a human readable format. ~> hadoop fs text /biginsights/hive/warehouse/computersalesdb.db/products_sequenceformat/ _0; Page 8 Hive Storage Formats

9 That output looks much better! RCFile 1. Create a new table in Hive called products_rcfileformat. We will use similar DDL that we used to create the original products table EXCEPT in this example we will explicitly tell Hive which SerDe and Input/Output formats to use. hive> CREATE TABLE products_rcfileformat ( prod_name STRING, description STRING, category STRING, qty_on_hand INT, prod_num STRING, packaged_with ARRAY<STRING> ) ROW FORMAT SERDE org.apache.hadoop.hive.serde2.columnar.columnarserde STORED AS INPUTFORMAT org.apache.hadoop.hive.ql.io.rcfileinputformat OUTPUTFORMAT org.apache.hadoop.hive.ql.io.rcfileoutputformat ; Hands-on-Lab Page 9

10 2. Ask Hive to give us the EXTENDED details of our new products_rcfileformat table. hive> DESCRIBE EXTENDED products_rcfileformat; InputFormat is: org.apache.hadoop.hive.ql.io.rcfileinputformat OutputFormat is: org.apache.hadoop.hive.ql.io.rcfileoutputformat SerDe serializationlib is: org.apache.hadoop.hive.serde2.columnar.columnarserde The HDFS location of the data for this table is: /biginsights/hive/warehouse/computersalesdb.db/products_rcfileformat Seeing how we explicitly specified the Input/Output formats and the SerDe, these values come as no surprise. Page 10 Hive Storage Formats

11 3. Now let s load our products_rcfileformat table with data. We will do this by running an INSERT query. hive> FROM products INSERT OVERWRITE TABLE products_rcfileformat SELECT *; 4. Set the Beeline outputformat back to table. Then SELECT * from the products_rcfileformat table to make sure it was loaded with the product data. hive>!set outputformat table hive> SELECT * FROM products_rcfileformat; All the data should be there. 5. We can assume that Hive created a _0 file in the products_rcfileformat directory on HDFS. Let s examine this file on HDFS and see exactly what Hive did for us. Open a Linux terminal and run the hadoop fs -cat command. ~> hadoop fs -cat /biginsights/hive/warehouse/computersalesdb.db/products_rcfileformat/ _0; Hands-on-Lab Page 11

The hadoop fs text tool also performs the same, so we will need to use a different tool. 6.

12 The file in RCFile format looks a bit wacky to the human eye. We need to view it using a tool that displays the contents in more human friendly format. The hadoop fs text tool also performs the same, so we will need to use a different tool. 6. In the Linux terminal change to the Hive home bin directory. From the bin directory we will pass a statement to hive that will allow us to view the RCFile. ~> cd $HIVE_HOME/bin./hive -service rcfilecat /biginsights/hive/warehouse/computersalesdb.db/products_rcfileformat/ _0 Page 12 Hive Storage Formats

13 That output looks quite a bit better! ORC File 1. Create a new table in Hive called products_orcformat. We will use the exact same DDL we used to create the original products table EXCEPT we will change the STORED AS clause to be ORC instead of TEXTFILE. hive> CREATE TABLE products_orcformat ( prod_name STRING, description STRING, category STRING, qty_on_hand INT, prod_num STRING, packaged_with ARRAY<STRING> ) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' COLLECTION ITEMS TERMINATED BY ':' STORED AS ORC; 2. Load data into the new ORC table. hive> FROM products INSERT OVERWRITE TABLE products_orcformat SELECT *; 3. SELECT * from the products_orcformat table to make sure it was loaded with the product data. hive> SELECT * FROM products_orcformat; Hands-on-Lab Page 13

4. Let s examine the products_orcformat table on HDFS and see what Hive did for us. Open the BigInsights Web Console and navigate to the /biginsights/hive/warehouse/computersalesdb.

14 4. Let s examine the products_orcformat table on HDFS and see what Hive did for us. Open the BigInsights Web Console and navigate to the /biginsights/hive/warehouse/computersalesdb.db/products_orcformat directory. You will find a _0 file. If you click on this file, the data will be displayed in the right hand viewing pane. Analyze the details of the file, including file size. Page 14 Hive Storage Formats

1.3 Compression In this section we will take a peak at using compression with Hive. 1. From within the Hive CLI, enter the following set commands. These changes will only last this one session.

15 1.3 Compression In this section we will take a peak at using compression with Hive. 1. From within the Hive CLI, enter the following set commands. These changes will only last this one session. hive> set hive.exec.compress.output=true; hive> set mapred.output.compression.code=org.apache.hadoop.io.compress.gzipcodec; We are telling Hive to compress the output (not intermediate compression) into Gzip format. 2. Open the BigInsights Web Console and navigate to the /biginsights/hive/warehouse/computersalesdb.db/products_rcfileformat/ _0 file On the right you can see the file is 1.7KB. 3. Now let s reload our products_rcfileformat table with data from the products table. hive> FROM products INSERT OVERWRITE TABLE products_rcfileformat SELECT *; Hands-on-Lab Page 15

4. Back in the BigInsights Web Console refresh the file view and navigate to the /biginsights/hive/warehouse/computersalesdb.db/products_rcfileformat/00 0000_0 file again.

16 4. Back in the BigInsights Web Console refresh the file view and navigate to the /biginsights/hive/warehouse/computersalesdb.db/products_rcfileformat/ _0 file again. You can see the file has now been compressed and is certainly smaller than before! 5. For good measure, run a SELECT query from the products_rcfileformat table to make sure you are still able to query this compressed data. hive> SELECT prod_name FROM products_rcfileformat LIMIT 5; 6. From within the Hive CLI, enter the following set command to turn compression off for this session. hive> set hive.exec.compress.output=false; Page 16 Hive Storage Formats

17 1.4 Summary Congratulations! You now know how to work with different file formats include SequenceFiles, RCFiles and ORC Files. You explicitly stated Input and Output formats and SerDe in a CREATE TABLE statement. You ve also turned output compression on in Hive and saw a size reduction with your own eyes. You may move on to the next Unit. Hands-on-Lab Page 17

18 NOTES

19 NOTES

Copyright IBM Corporation 2013. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any kind, express or implied.

20 Copyright IBM Corporation The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, these materials. Nothing contained in these materials is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software. References in these materials to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. This information is based on current IBM product plans and strategy, which are subject to change by IBM without notice. Product release dates and/or capabilities referenced in these materials may change at any time at IBM s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. IBM, the IBM logo and ibm.com are trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at Copyright and trademark information at

Using Hive for Data Warehousing

An IBM Proof of Technology Using Hive for Data Warehousing Unit 5: Hive Storage Formats An IBM Proof of Technology Catalog Number Copyright IBM Corporation, 2015 US Government Users Restricted Rights -