Big Data Analysis using Hadoop. Lecture 4. Hadoop EcoSystem

Transcription

1 Big Data Analysis using Hadoop Lecture 4 Hadoop EcoSystem Hadoop Ecosytems 1

2 Overview Hive HBase Sqoop Pig Mahoot / Spark / Flink / Storm Hive 2

3 Hive Data Warehousing Solution built on top of Hadoop Provides SQL-like query language named HiveQL Minimal learning curve for people with SQL expertise Data analysts are target audience Ability to bring structure to various data formats Simple interface for ad hoc querying, analyzing and summarizing large amountsof data Access to files on various data stores such as HDFS, etc Website : Download : Documentation : Hive Hive does NOT provide low latency or real-time queries Even querying small amounts of data may take minutes Designed for scalability and ease-of-use rather than low latency responses Translates HiveQL statements into a set of MapReduce Jobs which are then executed on a Hadoop Cluster 3

4 Hive Hive Concepts Re-used from Relational Databases Database: Set of Tables, used for name conflicts resolution Table: Set of Rows that have the same schema (same columns) Row: A single record; a set of columns Column: provides value and type for a single value Can can be dived up based on Partitions Buckets Hive Let s work through a simple example 1. Create a Table 2. Load Data into a Table 3. Query Data 4. Drop a Table 4

5 Hive 1. Create a table hive>!cat data/user-posts.txt; user1,funny Story, user2,cool Deal, user4,interesting Post, user5,yet Another Blog, hive> Values are separate by, and each row represents a record; first value is user name, second is post content and third is timestamp hive> CREATE TABLE posts (user STRING, post STRING, time BIGINT) > ROW FORMAT DELIMITED > FIELDS TERMINATED BY ',' > STORED AS TEXTFILE; OK Time taken: seconds 1st line: creates a table with 3 columns 2nd and 3rd line: how the underlying file should be parsed 4th line: how to store data hive> show tables; OK posts Time taken: seconds hive> describe posts; OK user string post string time bigint Time taken: seconds Hive - 2. Load Data into a Table hive> LOAD DATA LOCAL INPATH 'data/user-posts.txt' > OVERWRITE INTO TABLE posts; Copying data from file:/home/hadoop/training/play_area/data/user-posts.txt Copying file: file:/home/hadoop/training/play_area/data/user-posts.txt Loading data to table default.posts Deleted /user/hive/warehouse/posts OK Time taken: seconds hive> Existing records the table posts are deleted; data in user-posts.txt is loaded into Hive s posts table $ hdfs dfs -cat /user/hive/warehouse/posts/user-posts.txt user1,funny Story, user2,cool Deal, user4,interesting Post, user5,yet Another Blog,

6 Hive 3. Query Data hive> select count (1) from posts; Total MapReduce jobs = 1 Launching Job 1 out of 1... Starting Job = job_ _0004, Tracking URL = Kill Command = hadoop job -Dmapred.job.tracker=localhost: kill job_ _0004 Hadoop job information for Stage-1: number of mappers: 1; number of reducers: :37:24,962 Stage-1 map = 0%, reduce = 0% :37:30,497 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 0.87 sec :37:31,577 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 0.87 sec :37:32,664 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 2.64 sec MapReduce Total cumulative CPU time: 2 seconds 640 msec Ended Job = job_ _0004 MapReduce Jobs Launched: Job 0: Map: 1 Reduce: 1 Accumulative CPU: 2.64 sec HDFS Read: 0 HDFS Write: 0 SUCESS Total MapReduce CPU Time Spent: 2 seconds 640 msec OK 4 Time taken: seconds Count number of records in posts table Transformed HiveQL into 1 MapReduce Job Result is 4 records How long did it take to run? Hive 3. Query Data hive> select * from posts where user="user2"; OK user2 Cool Deal Time taken: seconds Select records for "user2" hive> select * from posts where time<= limit 2; OK user1 Funny Story user2 Cool Deal Time taken: seconds hive> Usually there are too many results to display, then one could utilize limit command to bound the display Select records whose timestamp is less or equals to the provided value 6

7 Hive - 4. Drop a Table hive> DROP TABLE posts; OK Time taken: seconds hive> exit; $ hdfs dfs -ls /user/hive/warehouse/ If hive was managing underlying file then it will be removed Hive 7

8 Partitions and Buckets Partitions divide data by grouping similar type of data together based on a column or partition key. Each Table can have one or more partition keys to identify a particular partition. This allows us to have a faster query on slices of the data. A bucket divide each partition or the unpartitioned table into N Buckets based on the hash function of a column(s) in the table Partitions and Buckets CREATE TABLE page_view( viewtime INT, userid BIGINT, page_url STRING, referrer_url STRING, ip STRING COMMENT 'IP Address of the User') COMMENT 'This is the page view table' PARTITIONED BY(dt STRING, country STRING) STORED AS SEQUENCEFILE; 8

9 Partitions and Buckets CREATE TABLE page_view( viewtime INT, userid BIGINT, page_url STRING, referrer_url STRING, ip STRING COMMENT 'IP Address of the User') COMMENT 'This is the page view table' PARTITIONED BY(dt STRING, country STRING) CLUSTERED BY(userid) SORTED BY(viewTime) INTO 32 BUCKETS STORED AS SEQUENCEFILE; The table is clustered by a hash function of userid into 32 buckets. Within each bucket the data is sorted in increasing order of viewtime. Old vs New Hive Old Hive = Map-Reduce New Hive = Map-Reduce, Spark, Tez Tez = an application framework which allows for a complex directedacyclic-graph of tasks for processing data 9

10 Hive Word Count CREATE TABLE docs (line STRING); LOAD_ DATA INPATH docs OVERWRITE INTO TABLE docs; CREATE TABLE word_counts AS SELECT word, count(1) AS count FROM (SELECT explode(split(line, \s )) AS word FROM docs) w GROUP BY word ORDER BY word; HBase 10

11 HBase HBase is a distributed column-oriented database built on top of the Hadoop file system. HBase is a data model that is similar to Google s big table designed to provide quick random access to huge amounts of unstructured data. Columnar oriented database The components of HBase data model consist of tables, rows, column families, columns, cells and versions. Tables are like logical collection of rows stored in separate partitions. Data in a row are grouped together as Column Families. Each Column Family has one or more Columns and these Columns in a family are stored together. Website : Download : Documentation : 11

12 HBase Non-Acid compliant database Hbase shell with limited range of commands Libraries for Java and many other languages MR supported HBase? 12

13 HBase HBase Good at Single random selects and range scans Querying one or a small subset of columns Data compaction as nulls are ignored Not so good at Transactions Joins, Group bys, Where Only one index for a table 13

14 Pig Pig is a platform for analyzing large data sets that consists of a high-level language for expressing data analysis programs, coupled with infrastructure for evaluating these programs. Pig is an abstraction on top of Hadoop Provides high level programming language designed for data processing Converted into MapReduce and executed on Hadoop Clusters MapReduce requires programmers Must think in terms of map and reduce functions More than likely will require Java programmers Pig provides high-level language that can be used by Analysts, Data Scientists, Statisticians, Etc... Different type of user compared to those who write MR functions Website : Download : Documentation : 14

15 Pig Pig's infrastructure layer consists of a compiler that produces sequences of Map-Reduce programs, Pig's language layer currently consists of a textual language called Pig Latin, which has the following key properties: Ease of programming. It is trivial to achieve parallel execution of simple, "embarrassingly parallel" data analysis tasks. Complex tasks comprised of multiple interrelated data transformations are explicitly encoded as data flow sequences, making them easy to write, understand, and maintain. Optimization opportunities. The way in which tasks are encoded permits the system to optimize their execution automatically, allowing the user to focus on semantics rather than efficiency. Extensibility. Users can create their own functions to do special-purpose processing. Pig Latin Command based language Data flow language rather than procedural or declarative Designed specifically for data transformation and flow expression Pig compiler converts Pig Latin to MapReduce Compiler strives to optimize execution You automatically get optimization improvements with Pig updates Provides common operations like join, group, filter, sort. Pig Examples - Aggregation Let s count the number of times each user appears in the excite data set. log = LOAD excite-small.log AS (user, timestamp, query); Grpd = GROUP log BY user; Cntd = FOREACH grpd GENERATE group, COUNT(log) AS cnt; STORE cntd INTO output ; Results: 002BB5A52580A8ED BD9CD3AC6BB

16 Pig Examples - Filtering Let s apply a filter to the groups so that we only get the high frequency users. log = LOAD excite-small.log AS (user, timestamp, query); grpd = GROUP log BY user; cntd = FOREACH grpd GENERATE group, COUNT(log) AS cnt; fltrd = FILTER cntd BY cnt > 50; STORE cntd INTO output ; Results: 0B294E3062F036C CE647F6 78 7D286B5592D83BBE 59.. Pig Examples - Ordering Let s ort the high frequency users by frequency. log = LOAD excite-small.log AS (user, timestamp, query); grpd = GROUP log BY user; cntd = FOREACH grpd GENERATE group, COUNT(log) AS cnt; fltrd = FILTER cntd BY cnt > 50; srtd = ORDER fltrd by cnt; STORE cntd INTO output ; Results: 7D286B5592D83BBE 59 0B294E3062F036C CE647F

17 Pig Examples Joining Data Join the 2 data sets Join the data sets based on words that are common in both bard = LOAD shakespeare_freq AS (freq, word); Kjv = LOAD bible_freq AS (freq, word); Inboth = JOIN bard BY word, kjv BY word; STORE inboth INTO output ; Results: 2 Abide 1 Abide 2 Abraham 111 Abraham 3... Pig Summary of Commands Summary of Pig Commands DESCRIBE Returns the schema of a relation. DUMP Dumps or displays results to screen. EXPLAIN Displays execution plans. ILLUSTRATE Displays a step-by-step execution of a sequence of statements. 17

18 Pig Word Count We know how much coding is needed to perform the WordCount using MR in Java In Pig is is a bit simpler input = load data.txt as (line); words = foreach input generate flatten(tokenize(line)) as word; grpd = group words by word; cntd = foreach grpd generate group, COUNT(words); dump cntd; That was much easier or was it? Pig Summary of Pig Commands Data Types Relational Operators User Defined Functions (UDF) Mathematical Functions Evaluation Functions String Functions Date & Time Functions Bag & Tuple Functions Load & Store Functions SQL -> PIG mapping Good examples in Cheat Sheet doc Pig Cheat Sheet : 18

19 Sqoop Sqoop Apache Sqoop allows the transfer of data between Hadoop and RDBMS Can import one or any number of tables or just portions of tables Uses map-reduce to do the import or export Works with any JDBC compatible database Website : Download : Documentation : 19

20 Sqoop The data set being transferred is sliced up into different partitions A map only job is launched with individual mappers responsible for transferring a slice of the data set Sqoop using the database meta data to infer data types Sqoop Basic set of commands Most common are Import Export 20

21 Sqoop Import Data Need the correct JDBC driver for the database you are using E.g. Oracle JDBC driver ojdbc6.jar Copy this to your./lib directory Import from Oracle : only certain attributes from a table Data saved to hdfs $ bin/sqoop import --connect "jdbc:oracle:thin:@localhost:1521:xe" \ --password "calgary10" --username "SYSTEM" --table "HR.DEPARTMENTS \ --columns "DEPARTMENT_ID,DEPARTMENT_NAME,MANAGER_ID,LOCATION_ID Sqoop Import Data Import in Hive sqoop takes care of populating the Hive metastore with the appropriate metadata for the table Invokes the necessary commands to load the table or partition as the case may be. All of this is done by simply specifying the option hive-import with the import command. $ sqoop import --connect "jdbc:oracle:thin:@localhost:1521:xe" \ --table ORDERS --password "calgary10" --username "SYSTEM \ --hive-import Sqoop converts the data from the native datatypes within the external datastore into the corresponding types within Hive. Sqoop automatically chooses the native delimiter set used by Hive 21

22 Sqoop Export Data Exports the data in the form of SQL INSERT statements Table in the destination DB needs to exists before your export the data Destination table must be empty Data types must correspond Can export from HDFS or Hive $ sqoop export --connect "jdbc:oracle:thin:@localhost:1521:xe" \ --table ORDERS --password "calgary10" --username "SYSTEM \ --export-dir /user/hive/warehouse/test_db.db/my_objects export-dir <directory path>: This is the directory from which data will be exported. Export to DB table with existing data --update-key <primary key> $ sqoop export --connect "jdbc:oracle:thin:@localhost:1521:xe" \ --table ORDERS --password "calgary10" --username "SYSTEM \ --export-dir /user/hive/warehouse/test_db.db/my_objects \ --update-key id Mahout / Flink / Storm / Spark 22

23 mahout Mahout started back in 2008 as a subproject of Apache Lucence, which provides an open source search engine. Lucence provides advanced implementation of search, text mining and information retrieval Related topics with machine learning, and committers migrated onto Mahout Mahout supports four main data science use cases: Collaborative filtering mines user behavior and makes product recommendations (e.g. Amazon recommendations) Clustering takes items in a particular class (such as web pages or newspaper articles) and organizes them into naturally occurring groups, such that items belonging to the same group are similar to each other Classification learns from existing categorizations and then assigns unclassified items to the best category Frequent itemset mining analyzes items in a group (e.g. items in a shopping cart or terms in a query session) and then identifies which items typically appear together Website : Download : Documentation : mahout Originally developed to use MapReduce Java based set of libraries Mahout can now use Spark and Flink for the backend Many still using MapReduce version Original ML language for Hadoop But limited functionality Newer projects have a lot more functionality 23

24 Similarity Measure example Recommender using Nearest Neighbor Storm Storm makes it easy to reliably process unbounded streams of data, doing for realtime processing what Hadoop did for batch processing Use Cases: real-time analytics, online machine learning, continuous computation, distributed RPC, ETL Website : Download : Documentation : 24

25 1/11/18 Storm Storm - WordCount 25

26 Flink Designed for data that is continuously produced => streaming data activity logs, web logs, machines, sensors, and database transactions! Flink can process streaming and batch data Batch is a special case of stream processing Storage-agnostic Good integration with Hadoop ecosystem Flink integrates with a wide variety of open source systems for data input and output (e.g., HDFS, Kafka, Elasticsearch, HBase, and others), deployment (e.g., YARN), as well as acting as an execution engine for other frameworks (e.g., Cascading, Apache Beam incubating aka Google Cloud Dataflow). The Flink project itself comes bundled with a Hadoop MapReduce compatibility layer, a Storm compatibility layer, as well as libraries for machine learning and graph processing Website : Download : Documentation : Flink Flink is used in conjunction with data storage or brokering systems. A typical architectural pattern is to use Flink in conjunction with Apache Kafka to: Ingest data into other systems such as HDFS, databases, or search indices and create continuous ETL pipelines. Perform analytics directly on the moving data to create alerts, dashboards, or power operational applications obviating the need for ingestion and ETL. Perform machine learning on streams by continuously building models of the events as they arrive and using the model to serve online recommendations. Since Flink is a full-fledged system for batch processing as well, it can also be used for applications on top of static data. 26

27 Flink Examples of typical functions in Flink Map FlatMap Filter KeyBy Reduce Fold Aggregations Window WindowAll Window Apply Window Reduce Window Fold Aggregations on windows Union Window Join Window CoGroup Connect CoMap, CoFlatMap Split Select Iterate Extract Timestamps Project (for data streams of Tuples) Several APIs in Java/Scala/Python DataSet API Batch Processing DataStream API Real-time streaming analytics Table API Relational Queries Domain Speciific Libraries FlinkML Machine learningl library for Flink Gelly Graph library for Flink Some benchmarks show Flink is faster at processing streaming data than Spark. But Spark is faster at processing Batch data. At the moment Spark seems to be better as an all round solution. Flink 27

28 Spark MapReduce problems: Many problems aren t easily described as map-reduce Persistence to disk typically slower than in-memory work Apache Spark a general-purpose processing engine that can be used instead of MapReduce Supports Java, Scala and Python Main idea: use the memory resources of the cluster for better performance Supported by many commercial companies, Cloudera, IBM, Oracle, Microsoft Website : Download : Documentation : Spark Many libraries Spark SQL Spark Streaming stream processing of live datastreams MLlib - machine learning GraphX graph manipulation extends Spark RDD with Graph abstraction: a directed multigraph with properties attached to each vertex and edge. 28

29 Spark Key construct: Resilient Distributed Dataset (RDD) Resilient if the data is lost, it can be recreated for previous steps Distributed appears as a single chunk of code, but is actually distributed across nodes Dataset initial data can come from file or created programmatically an RDD is a read-only, partitioned collection of records. RDDs can be created through deterministic operations on either data on stable storage or other RDDs. supports in-memory processing computation RDDs represent data or transformations on data Transformations define a new RDD based on the current one(s) Actions can be applied to RDDs; actions force calculations and return values Lazy evaluation: Nothing computed until an action requires it Transformations set things up Actions cause calculations to actually be performed RDDs are best suited for applications that apply the same operation to all elements of a dataset Spark 29

30 Spark Next topic for this module Check out databricks Most of the developers of Spark work here Lots of resources Developer resources : Videos : Certification : 30

31 Hadoop & Data Architectures From Small Data to Big Data : Working together Working together to get the job done 31

32 SQL : The one language to rule all your data Store External View External View Access Analyze Conceptual Schema Protect Physical Schema SQL : The one language to rule all your data Store External View External View Access Analyze Conceptual Schema Protect Physical Schema Spatial & Graph Oracle NoSQ L 32

33 1/11/18 Current trends for Hadoop Current trends for Hadoop 33

34 Typical Application Story: Before Gluent Products Preferences A complex business application running on a RDBMS Customers SALES Promotions Prices Years of application development & improvement Upstream & downstream dependencies Terabytes of historical data (usually years of history) Big queries run for too long or never complete (or never tried) Does not scale with modern demand Way too expensive RDBMS + SAN Application rewrite very costly & risky or virtually impossible Gluent Data Virtualization (90/10) 10% RDBMS + SAN Customers Products Preferences Promotions Columnar compression: 6-20x data size reduction Application still sees all data: App code & architecture unchanged! SALES (10%) Virtual (90%) Prices Gluent SALES (90%) 90% Hadoop Reduce cost, offload data, increase performance RDBMS + SAN Automatic data flow, No ETL development! 34

35 1/11/18 Current trends for Hadoop 35

36 Current trends for Hadoop Current trends for Hadoop 36

37 Hadoop Ecosystem The ever changing World What will it look like in 12 months from now? 37

38 Technology keeps on changing Hybrid It s all about the DATA 38