Dr. Chuck Cartledge. 18 Mar. 2015

Transcription

1 CS-495/595 Hive Lecture #9 Dr. Chuck Cartledge 18 Mar /25

2 Table of contents I 1 Miscellanea 2 Assignment #3 3 The Book 4 Chapter 12 6 Project 7 Conclusion 8 References 5 Break 2/25

3 Corrections and additions since last lecture. Assignment #3 due in a few hours. 3/25

4 Pay attention to the assignment details. Things like: Getting the average of the correct procedure based on the numeric code Grouping the practitioners by type Getting the average for the state based on the numeric code Addressing those geographic areas that aren t in your cartographic file If appropriate a heat scale 4/25

5 Hadoop, The Definitive Guide Version 3 is specified in the syllabus [5] Version 4 came out in November 2015 We ll use Version 3 as much as possible 5/25

6 Overview Where to get it.... was created to make it possible for analysts with SQL skills... to run queries on huge data... Installable much like Pig 6/25

7 Overview How to get it running. Assuming that you have hadoop up and running There are logical conflicts between hadoop and Hive 7/25

8 Overview Different interfaces Like so many in the hadoop ecosystem. A command line interface (CLI, our old friend) Web interface (not sure if this works on our cluster) JDBC and ODBC All feed a compiler, and optimizer, and executor. 8/25

9 Overview Simple things like creating tables Tables can be created from external data files MapReduce underlying everything, so rows and fields are user definable Tables can be partitioned for optimization 9/25

10 Overview Blow up the same image. 10/25

11 Overview More information about architecture Focusing on where data is stored. Actual data is stored either locally, or in the HDFS Metadata is stored in the metastore used by hive metastore is a Derby database (data about the hive is stored in a RDBMS) Hive tables are stored on the HDFS under /user/hive/warehouse Image from [5]. 11/25

12 Overview Comparison with Traditional Databases. Schema enforcement Traditional enforced at load time (called Write) Hive enforced at query time (called Read) Updates, transactions, and indexes Traditional these are mainstays Hive HDFS doesn t support these actions Image from [1]. 12/25

13 Overview Same image 13/25

14 Overview Hive has its roots in MySQL Relatively small differences in syntax between HiveQL and MySQL Limitations on VIEWs, Indexes, and updates (workaround is to create new tables) HiveQL supports partitioning the table (column or row) to better fit the HDFS, and MapReduce paradigm HiveQL does not support updating existing records Image from [2]. 14/25

15 Overview Same image 15/25

16 Break time. Take about 10 minutes. 16/25

17 Additional details Techniques and tools to solve the problem Database heavy lifting MapReduce Pig Latin Hive Display and analysis Custom code (language de jure) Excel Some cartographic package 17/25

18 Additional details Class membership Undergrads 1 CAMPBELL, CHRISTOPHER G. 2 CRUZ, JOSHUA T. 3 DAVIS, RANDALL A. 4 JIANG, MING H. 5 PHELPS, NATHAN A. 6 ZHANG, HEMIAO Graduates 1 ALFURAYJ, HAIFA S. 2 ARAB, MARYAM 3 BETHU, ANVESH 4 DASARI, VICTOR PRABHU 5 GARNER, KEVIN M. 6 HAVANUR, SRINIVAS J. 7 LAMBI, ROHIT D. 8 PATEL, PRIYANK A. 9 POTINENI, BHAVYATEJA 10 SADANA, PRANEET 11 SAJJAN, PRASANNA KUMAR BASAVARAJ So many people. 18/25

19 Additional details Team membership Undergrads 1 Campbell and Cruz 2 Davis and Phelps 3 Jiang and Zhang Graduates 1 Alfurayj and Arab 2 Bethu and Dasari 3 Garner, Havanur and Sajjan 4 Lambi and Sadana 5 Patel and Potineni 19/25

20 Additional details Scatter graphs We don t have a priori knowledge of any relationship between Medicare billings and pharmaceutical payments. So create a scatter graph of the data. Just plot one value versus the other and look Looking if there is any apparent relationship Relationship may not exist, may not be linear, may not be monotonic 20/25

21 Additional details Examples of scatter graphs Different types of relationships: None (shotgun) Positive (strong positive) Negative (strong negative) Independent (or low) Independent and non-monotonic (or low) Spurious Image from [4]. The scatter graph will be our guide for computations. 21/25

22 Additional details Presentation Short (on the order of 5 minutes) Address the 6 questions Power point, or other format Still need the standard PDF submission 22/25

23 What have we covered? Gave some final hints/directions for assignment #3 Talked about Hive, HiveQL, origins, strengths, and weaknesses Talked about the project Next lecture: Discussion of current real-world applications of Big Data 23/25

24 References I [1] Amr A. Awadallah, Schema-on-read vs schema-on-write, [2] Marc Holmes, Hive for sql users, blog/hive-cheat-sheet-for-sql-users/, [3] Jasper Pei Lee, Hive a sql-like wrapper over hadoop, hive-a-sql-like-wrapper-over-hadoop/, [4] Bioscience Staff, Numbers numerical methods for bioscience students, [5] Tom White, Hadoop: The definitive guide, 3rd edition, O Reilly Media, Inc., /25

25 References II 25/25