Using Hive for Data Warehousing

Size: px

Start display at page:

Download "Using Hive for Data Warehousing"

Kristin Warren
5 years ago
Views:

1 An IBM Proof of Technology Using Hive for Data Warehousing Unit 5: Hive Storage Formats

2 An IBM Proof of Technology Catalog Number Copyright IBM Corporation, 2015 US Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

3 Contents LAB 5 HIVE STORAGE FORMATS SETTING HDFS USER PERMISSIONS ACCESSING THE HIVE CLI WORKING WITH DIFFERENT FILE AND RECORD FORMATS SEQUENCEFILE RCFILE ORC FILE COMPRESSION SUMMARY Contents Page 3

4 Lab 5 Hive Storage Formats We can use Hive to work with a variety of different file and record formats. Understanding how to work with formats beyond the simple TextFile will empower you to make better decisions when working with Hive and Hadoop. We can also use compression with Hive to reduce the footprint of our data potentially buying us performance increases and/or storage improvement. After completing this hands-on lab, you will be able to: Create SequenceFiles, RCFiles and ORC Files. Explicitly state Input/Output formats and SerDe in a CREATE TABLE statement. Turn output compression on in Hive. Allow 45 minutes to complete this section of lab. This version of the lab was designed using the IBM BigInsights 4.1 Quick Start Edition. Throughout this lab it is assumed that you will be using the following account login information: Username Password VM image setup screen root password Ambari admin admin If you are continuing this series of hands on labs immediately after completing Accessing Hadoop Data Using Hive Unit 4: Hive Operators and Functions, you may move on to section 1.1 of this lab. Otherwise please complete the prior labs to get started. (All Hadoop components should be running and you should have already created the databases, tables and partitions that will be used in this unit.) Page 4 Hive Storage Formats

5 1.1 Setting HDFS User Permissions Before we continue, we will make an ownership change to one of the directories on HDFS that Hive uses. This will allow us to continue the rest of the lab without having to worry about the details of Hadoop permissions. 1. We will execute a few commands: $ su - hdfs $ hadoop fs chmod R 777 / Once you run those commands enter the exit command to return back to the virtuser id. $ exit Now you can continue to the next step of the lab. Possible error if this section is ignored (if we don t specify permissions). Hands-on-Lab Page 5

1.2 Accessing the Hive CLI In this section we will navigate to the Hive Beeline CLI, just as we have done in previous labs, and start an interactive CLI session. 2.

beeline>!connect jdbc:hive2://rvm.svl.ibm.com:10000 virtuser password org.apache.hive.jdbc.hivedriver 5.

6 1.2 Accessing the Hive CLI In this section we will navigate to the Hive Beeline CLI, just as we have done in previous labs, and start an interactive CLI session. 2. In a Linux terminal change to the Hive bin directory $ cd /usr/iop/ /hive/bin 3. Start an interactive Hive shell session. $./beeline 4. Connect to Hive. beeline>!connect jdbc:hive2://rvm.svl.ibm.com:10000 virtuser password org.apache.hive.jdbc.hivedriver 5. Tell Hive to use the computersalesdb (we will use this database for the rest of this interactive session). hive> USE computersalesdb; Note: Please run these commands whenever you exit and re-enter Hive. Page 6 Hive Storage Formats

7 1.3 Working with Different File and Record Formats In this section we will experiment with different file and record formats in Hive. We already created multiple tables on our system that are in TEXTFILE format. Let s take advantage of this fact and create some new tables of different formats then insert data into the new tables by running SELECT queries on the originals SequenceFile In a previous lab you created the products table and described the table with the STORED AS TEXTFILE clause. Now we will create a new table called products_sequenceformat that will be stored as a SEQUENCEFILE. We will have Hive do the conversion from TEXTFILE to SEQUENCEFILE for us by simple running an INSERT statement. 1. Create a new table in Hive called products_sequenceformat. We will use the exact same DDL we used to create the original products table EXCEPT we will change the STORED AS clause to be SEQUENCEFILE instead of TEXTFILE. hive> CREATE TABLE products_sequenceformat ( prod_name STRING, description STRING, category STRING, qty_on_hand INT, prod_num STRING, packaged_with ARRAY<STRING> ) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' COLLECTION ITEMS TERMINATED BY ':' STORED AS SEQUENCEFILE; 2. Run the SHOW TABLES command to verify the table has been created. hive> SHOW TABLES; Hands-on-Lab Page 7

8 3. Set Hive s outputformat to vertical, then ask Hive to give us the EXTENDED details of our new products_sequenceformat table. hive>!set outputformat vertical hive> DESCRIBE EXTENDED products_sequenceformat; Page 8 Hive Storage Formats

You ll notice some interesting things listed here: InputFormat is: org.apache.hadoop.mapred.sequencefileinputformat OutputFormat is: org.apache.hadoop.hive.ql.io.

9 You ll notice some interesting things listed here: InputFormat is: org.apache.hadoop.mapred.sequencefileinputformat OutputFormat is: org.apache.hadoop.hive.ql.io.hivesequencefileoutputformat SerDe serializationlib is: org.apache.hadoop.hive.serde2.lazy.lazysimpleserde The HDFS location of the data for this table is: /apps/hive/warehouse/computersalesdb.db/products_sequenceformat It s important to recognize that using the STORED AS SEQUENCEFILE clause in our create table statement set the Input and Output formats behind the scenes appropriately for us. We didn t have to explicitly specify those values when creating the table (though we could have). 4. Time to load our products_sequenceformat table with data. We will do this by running an INSERT query. hive> FROM products INSERT OVERWRITE TABLE products_sequenceformat SELECT *; Behind the scenes a temp space is being used to store the intermediate data. Hands-on-Lab Page 9

Let s examine the products_sequenceformat table on HDFS and see what Hive did for us. Exit out of Hive and list out the contents of the following directory: /apps/hive/warehouse/computersalesdb.

$ hadoop fs -cat /apps/hive/warehouse/computersalesdb.db/products_sequenceformat/000000_ 0 That doesn t look like a regular TEXTFILE does it? Some of the characters are displayed oddly.

10 Let s examine the products_sequenceformat table on HDFS and see what Hive did for us. Exit out of Hive and list out the contents of the following directory: /apps/hive/warehouse/computersalesdb.db/products_sequenceformat directory. $ hadoop fs -ls /apps/hive/warehouse/computersalesdb.db/products_sequenceformat You will find a _0 file. Explore the contents of the file. $ hadoop fs -cat /apps/hive/warehouse/computersalesdb.db/products_sequenceformat/000000_ 0 That doesn t look like a regular TEXTFILE does it? Some of the characters are displayed oddly. That s a good thing we are looking at a SEQUENCEFILE here! 5. Let s try something different. Exit out of Hive. We can use the text option in the HDFS fs command to see the data in a human readable format. $ hadoop fs -text /apps/hive/warehouse/computersalesdb.db/products_sequenceformat/000000_ 0 Page 10 Hive Storage Formats

That output looks much better! 1.3.2 RCFile 1. Create a new table in Hive called products_rcfileformat.

11 That output looks much better! RCFile 1. Create a new table in Hive called products_rcfileformat. We will use similar DDL that we used to create the original products table EXCEPT in this example we will explicitly tell Hive which SerDe and Input/Output formats to use. hive> CREATE TABLE products_rcfileformat ( prod_name STRING, description STRING, category STRING, qty_on_hand INT, prod_num STRING, packaged_with ARRAY<STRING> ) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.columnar.columnarserde' STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.rcfileinputformat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.rcfileoutputformat'; Hands-on-Lab Page 11

12 2. Ask Hive to give us the EXTENDED details of our new products_rcfileformat table. hive> DESCRIBE EXTENDED products_rcfileformat; Page 12 Hive Storage Formats

InputFormat is: org.apache.hadoop.hive.ql.io.rcfileinputformat OutputFormat is: org.apache.hadoop.hive.ql.io.rcfileoutputformat SerDe serializationlib is: org.apache.hadoop.hive.serde2.columnar.

13 InputFormat is: org.apache.hadoop.hive.ql.io.rcfileinputformat OutputFormat is: org.apache.hadoop.hive.ql.io.rcfileoutputformat SerDe serializationlib is: org.apache.hadoop.hive.serde2.columnar.columnarserde The HDFS location of the data for this table is: /apps/hive/warehouse/computersalesdb.db/products_rcfileformat Seeing how we explicitly specified the Input/Output formats and the SerDe, these values come as no surprise. 3. Now let s load our products_rcfileformat table with data. We will do this by running an INSERT query. hive> FROM products INSERT OVERWRITE TABLE products_rcfileformat SELECT *; 4. Set the Beeline outputformat back to table. Then SELECT * from the products_rcfileformat table to make sure it was loaded with the product data. hive>!set outputformat table hive> SELECT * FROM products_rcfileformat; Hands-on-Lab Page 13

14 Page 14 Hive Storage Formats

All the data should be there. 5. We can assume that Hive created a 000000_0 file in the products_rcfileformat directory on HDFS. Let s examine this file on HDFS and see exactly what Hive did for us.

15 All the data should be there. 5. We can assume that Hive created a _0 file in the products_rcfileformat directory on HDFS. Let s examine this file on HDFS and see exactly what Hive did for us. Exit out of Hive and run the hadoop fs -cat command. $ hadoop fs -cat /apps/hive/warehouse/computersalesdb.db/products_rcfileformat/000000_0; The file in RCFile format looks a bit wacky to the human eye. We need to view it using a tool that displays the contents in more human friendly format. The hadoop fs text tool also performs the same, so we will need to use a different tool. 6. In the Linux terminal ensure you are in Hive s bin directory. From the bin directory we will pass a statement to hive that will allow us to view the RCFile. $ cd /usr/iop/ /hive/bin./hive -service rcfilecat /apps/hive/warehouse/computersalesdb.db/products_rcfileformat/000000_0 Hands-on-Lab Page 15

16 That output looks quite a bit better! Page 16 Hive Storage Formats

17 1.3.3 ORC File 1. Create a new table in Hive called products_orcformat. We will use the exact same DDL we used to create the original products table EXCEPT we will change the STORED AS clause to be ORC instead of TEXTFILE. hive> CREATE TABLE products_orcformat ( prod_name STRING, description STRING, category STRING, qty_on_hand INT, prod_num STRING, packaged_with ARRAY<STRING> ) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' COLLECTION ITEMS TERMINATED BY ':' STORED AS ORC; 2. Load data into the new ORC table. hive> FROM products INSERT OVERWRITE TABLE products_orcformat SELECT *; Hands-on-Lab Page 17

18 3. SELECT * from the products_orcformat table to make sure it was loaded with the product data. hive> SELECT * FROM products_orcformat; Page 18 Hive Storage Formats

19 Hands-on-Lab Page 19

4. Let s examine the products_orcformat table on HDFS and see what Hive did for us. Exit out of Hive and check out the /apps/hive/warehouse/computersalesdb.db/products_orcformat directory.

20 4. Let s examine the products_orcformat table on HDFS and see what Hive did for us. Exit out of Hive and check out the /apps/hive/warehouse/computersalesdb.db/products_orcformat directory. You will find a _0 file. Check out the data inside the file using the hadoop fs cat command. $ hadoop fs -ls /apps/hive/warehouse/computersalesdb.db/products_orcformat; Analyze the details of the file, including file size. $ hadoop fs -cat /apps/hive/warehouse/computersalesdb.db/products_orcformat/000000_0; Page 20 Hive Storage Formats

21 5. To be able to read this data file, we will need to use orcfiledump -d rather than cat. Note that the d parameter specifies that you want to dump the data rather than the metadata. So run $./hive --orcfiledump d /apps/hive/warehouse/computersalesdb.db/products_orcformat/000000_0; Hands-on-Lab Page 21

22 1.4 Compression In this section we will take a peak at using compression with Hive. 1. In a Linux console list the contents of the following HDFS directory: $ hadoop fs ls /apps/hive/warehouse/computersalesdb.db/products_rcfileformat You can see the _0 file is about 1.7KB. 2. From within the Hive CLI, enter the following set commands. These changes will only last this one session. hive> set hive.exec.compress.output=true; hive> set mapred.output.compression.code=org.apache.hadoop.io.compress.gzipcodec; We are telling Hive to compress the output (not intermediate compression) into Gzip format. 3. Now let s reload our products_rcfileformat table with data from the products table. hive> FROM products INSERT OVERWRITE TABLE products_rcfileformat SELECT *; Page 22 Hive Storage Formats

4. Back in the Linux Console check the size of the /apps/hive/warehouse/computersalesdb.db/products_rcfileformat/000000_0 file again. $ hadoop fs ls /apps/hive/warehouse/computersalesdb.

23 4. Back in the Linux Console check the size of the /apps/hive/warehouse/computersalesdb.db/products_rcfileformat/000000_0 file again. $ hadoop fs ls /apps/hive/warehouse/computersalesdb.db/products_rcfileformat You can see the file has now been compressed and is certainly smaller than before! 5. For good measure, run a SELECT query from the products_rcfileformat table to make sure you are still able to query this compressed data. hive> SELECT prod_name FROM products_rcfileformat LIMIT 5; Hands-on-Lab Page 23

24 6. From within the Hive CLI, enter the following set command to turn compression off for this session. hive> set hive.exec.compress.output=false; 1.5 Summary Congratulations! You now know how to work with different file formats include SequenceFiles, RCFiles and ORC Files. You explicitly stated Input and Output formats and SerDe in a CREATE TABLE statement. You ve also turned output compression on in Hive and saw a size reduction with your own eyes. You may move on to the next Unit. Page 24 Hive Storage Formats

25 NOTES

26 NOTES

Copyright IBM Corporation 2015. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any kind, express or implied.

28 Copyright IBM Corporation The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, these materials. Nothing contained in these materials is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software. References in these materials to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. This information is based on current IBM product plans and strategy, which are subject to change by IBM without notice. Product release dates and/or capabilities referenced in these materials may change at any time at IBM s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. IBM, the IBM logo and ibm.com are trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at Copyright and trademark information at

Using Hive for Data Warehousing

An IBM Proof of Technology Using Hive for Data Warehousing Unit 5: Hive Storage Formats An IBM Proof of Technology Catalog Number Copyright IBM Corporation, 2013 US Government Users Restricted Rights -