Cmprssd Intrduction To

Transcription

1 Cmprssd Intrduction To Hadoop, SQL-on-Hadoop, NoSQL Singapore University of Technology & Design

2 Thank You For Inviting! My special kind regards to: Professor Meihui Zhang Associate Director Hou Liang Seah Industry Outreach Manager Robin Soo

3 What am I supposed to do?.. Please raise hand if you got enough kopi / teh / red bull for next 1 hour?.. want to learn about modern data analytics?.. are OK if I use words like Java or Command Line or Port?.. have hands-on experience with Hadoop, Spark, Hive?..

4 Shameless Self-Intro

5 Hi, My Name Is Arseny, And I m 5

6 Hadoop In A Nutshell

7 It All Started At Google

8 Hadoop is Google s Tech in Open Source

9 However, in 2016 big data & Hadoop don t need a hyperscale datacenter 9 Hadoop Originates From Hyperscale Approach

10 Closer Look, i.e. Hortonworks Data Platform (HDP) Hortonworks Data Platform 2.3 GOVERNANCE & INTEGRATION DATA ACCESS SECURITY OPERATIONS Data Lifecycle & Governance Falcon Atlas Data Workflow Batch MapReduce Script Pig SQL Hive NoSQL HBase Accumulo Phoenix Stream Storm Tez Tez Tez Slider Slider YARN : Data Operating System Search Solr In-memory Spark Others ISV Engines Administration Authentication Authorization Auditing Data Protection Ranger Knox Atlas HDFS Encryption Provisioning, Managing, & Monitoring Ambari Cloudbreak Zookeeper Sqoop Flume Kafka NFS WebHDFS 1 HDFS Hadoop Distributed File System N Scheduling Oozie DATA MANAGEMENT We will compress all these topics during next 1 hour Page 10 Hortonworks Inc All Rights Reserved

11 Quick demo

12 HDFS In A Nutshell Hadoop Distributed File System

13 Reading Data From HDFS 1: open 2: Request file block locations HDFS Client 6: close Client JVM Client Node 3: read Distributed FileSystem FSData InputStream 5: read from block NameNode JVM namenode 4: read from block DataNode JVM datanode DataNode JVM datanode DataNode JVM datanode 2015 Pivotal Software, Inc. All rights reserved. 13

14 Writing Data to HDFS 1: create HDFS Client 6: close Client JVM Client Node 3a: write 4a: write packet Distributed FileSystem FSDataOutputStr eam DataStreamer 5c: ack packet 2: create 7: complete 3b: Request allocation (as new blocks required) 3c: Three data-node, data-block pairs returned NameNode JVM namenode Diagram shows 3x replication 4b: write packet 4c: write packet DataNode DataNode DataNode JVM datanode 5b: ack packet JVM datanode 5a: ack packet JVM datanode 2015 Pivotal Software, Inc. All rights reserved. 14

15 Quick demo

16 YARN In A Nutshell Yet Another Resource Negotiator

17 Legacy SQL Is All Structured Traditional SQL databases: structured Schema-on-Write 1 create schema on file or block storage 2 load data 3 query data select ROW KEY, COLOR from where row row... row keys color shape timestamp first red square HH:MM:SS second blue round HH:MM:SS... row Can t add data before the schema is created. To change schema, drop and re-loaded entire table. A drop of TB-size table with Foreign Keys could last days. 17

18 MapReduce In Color Unstructured Schema-on-Read Query 1 load data straight from HDFS 2 query data - map - shuffle - reduce file.csv & other.txt 18

19 MapReduce In Process Diagram 19

20 Starting Job MapReduce v2.0 1: initiate job 2: request new application MapReduce program Client JVM Client Node Job 4: submit job 5c: initialize job 5a: start container ResourceManager JVM Jobtracker Node 7a: allocate task resources 3: copy job jars, config Node Manager JVM 5b: launch Node Manager Node MRApp Master 7b: start container 8: launch Node manager 10: run JVM Shared File-System (e.g. HDFS) 6: determine input splits 9: retrieve job jars, data YARN child Mapper or Reducer Child JVM Node Manager Node 2015 Pivotal Software, Inc. All rights reserved. 20

21 Quick demo

22 Hive In A Nutshell SQL interface to MapReduce Jobs

23 Relational DB Relational DB and SQL conceived to Remove repeated data, replace with tabular structure & relationships Provide efficient & robust structure for data storage Exploit regular structure with declarative query language Structured Query Language DRY Don t Repeat Yourself 23

24 What Hive Is A SQL-like processing capability based on Hadoop Enables easy data summarisation, ad-hoc reporting and querying, and analysis of large volumes of data Built on HQL, a SQL-like query language Statements run as mapreduce jobs Also allows mapreduce programmers to plugin custom mappers and reducers Works with Plain text, Hbase, ORC, Parquet and others formats Metadata is stored in MySQL 24

25 Hive Schemas Hive is schema-on-read Schema is only enforced when the data is read (at query time) Allows greater flexibility: same data can be read using multiple schemas Contrast with RDBMSes, which are schema-on-write Schema is enforced when the data is loaded Speeds up queries at the expense of load times 25

26 Hive Architecture 26 Hive Metastore + MySQL

27 What Hive Is Not Hive, like Hadoop, is designed for batch processing of large datasets Not a real-time system, not fully SQL-92 compliant Sibling solutions like Tez, Impala and HAWQ offer more compliance Latency and throughput are both high compared to a traditional RDBMS Even when dealing with relatively small data (<100 MB) 27

28 Quick demo

29 HBASE In A Nutshell SQL interface to MapReduce Jobs

30 ACID is Business Requirement for RDBMs Traditional DB-s have excellent support for ACID transactions Atomic: All write operations succeed, or nothing is written Consistent: Integrity rules guaranteed at commit Isolation: It appears to the user as if only one process executes at a time. (Two concurrent transactions will not see on another s transaction while in flight.) Durable: The updates made to the database in a committed transaction will be visible to future transactions. (Effects of a process do not get lost if the system crashes.) 30

31 Scale RDBMS?.. RDBMS is bad fit for huge scale, online applications Sharding?.. Scaling up?.. No Joins?.. Master-Slave?.. Big Data describes problem, Not only SQLdefines the general approach to solution: Emphasis on scale, distributed processing, use of commodity hardware 31

32 Business Needs for Not Only SQL Not Only SQL DBs evolved from web-scale use-cases Google, Amazon, Facebook, Twitter, Yahoo, Google Cache = Entire page saved in to a cell of a BigTable database Columnar layout preferred filters to reduce the disk lookups for non-existent rows or columns increases the performance of a database query operation. Requirement for massive scale, relational fits badly Queries relatively simple Direct interaction with online customers Cost-effective, dynamic horizontal scaling required Many nodes based on inexpensive (commodity) hardware Must manage frequent node failures & addition of nodes at any time 32

33 But how to build such DB?..

34 Reminder: The CAP Theorem (2 not 3) Once a writer has written, all readers will see that write Single Version of Truth? Consistency Availability Partition tolerance System is Available to serve 100% of requests and complete them successfully. No SPOF?.. A system can continue to operate in the presence of a network Partitions Replicas?.. 34

35 Eventually Consistent vs. ACID An artificial acronym you may see is BASE Basically Available System seems to work all the time Soft State Not wholly consistent all the time, but Eventual Consistency After a period with no updates, a given dataset will be consistent Resulting systems characterized as eventually consistent Overbooking an airline or hotel and passing risk to customer 35

36 Non-relational distributed database HBase is a database: has a schema, but it s non-relational 1.) Create column families 2.) Load data, multiples of rows form region files on HDFS 3.) Query data row row row keys column family color first red : #F00 blue : #00F yellow : #F0F second column family shape square : round : size : XXL hbase>get first, color : yellow COLUMN CELL yellow timestamp= , value= #F0F hbase>get second, shape : size COLUMN CELL size timestamp= , value= XXL 36

37 Column Oriented Storage 37

38 HBase Architecture, Read & Write Zookeeper Sequential HDFS Write & L2 Read Hbase API Client Hbase Client (1) Put/Delete Adaptive Pre Fetch & L2 Reads Sequential Writes Memstore = Eventual Consistency SQL ODBC Client SQL JDBC Client Pivotal HAWQ PXF Apache Phoenix Hbase Client Hbase Client Region Server (2.0) Write to WAL Write-Ahead Log (WAL) (2.1) Write to MemStore Memstore (3) Flush to HDFS HFile 38 (4) Get/Scan Read Request Client RAM Pre-Fetch

39 HBase namespace layout 39

40 Compression (HBase and others) From Hbase Definitive Guide 40

41 @arsenyspb Q&A?..

42