Getting Started with Hadoop and BigInsights

Transcription

1 Getting Started with Hadoop and BigInsights Alan Fischer e Silva Hadoop Sales Engineer Nov 2015

2 Agenda! Intro! Q&A! Break! Hands on Lab 2

3 Hadoop Timeline 3

4 In a Big Data World. The Technology exists now for us to:! Store everything, for as long as we want! Efficiently analyze everything, without sub-setting! Connect tiny nugget of valuable information buried in piles of worthless bytes 4

5 5 5

6 Apache Hadoop Modules! Hadoop Common: Common Utilities that supports all other modules.! Hadoop Distributed File System (HDFS ): - File system that spans all the nodes in a Hadoop cluster for data storage. - Links together the file systems on many local nodes to make them into one big file system.! Hadoop MapReduce - Software framework for easily writing applications which process vast amounts of data (multi-terabyte data-sets) in-parallel on large clusters (thousands of nodes) of commodity hardware in a reliable, faulttolerant manner.! YARN (Yet Another Resource Negotiator) - Scheduling & Resource Management! Large Hadoop Ecosystem: Open-source Apache project Hive Pig HBase Oozie Sqoop Flume 6 Zookeep er Mahout Spark Avro Spark

7 HDFS Architecture! Master / Slave architecture! Master: NameNode - Manages the file system namespace and metadata FsImage EditLog - Regulates access by files by clients NameNode File1 a b c d! Slave: DataNode - Many DataNodes per cluster - Manages storage attached to the nodes - Periodically reports status to NameNode - Data is stored across multiple nodes - Nodes and components will fail, so for reliability data is replicated across multiple nodes a b d b a c a d c c b d DataNodes 7

8 HDFS Blocks! HDFS is designed to support very large files! Each file is split into blocks - Hadoop default: 64MB - BigInsights default: 128MB! Behind the scenes, 1 HDFS block is supported by multiple operating system blocks 64 MB HDFS blocks OS blocks! If a file or a chunk of the file is smaller than the block size, only needed space is used. E.g.: a 210MB file is split as follows: 64 MB 64 MB 64 MB 18 MB 8

9 Replication of Data and Rack Awareness Data Node 1 A C Data Node 2 C Data Node 3 Data Node 5 B A Data Node 6 A Data Node 7 Data Node 9 C B Data Node 10 B Data Node 11 Name Node Rack aware: R1: 1,2,3,4 R2: 5,6,7,8 R3: 9,10,11 Data Node 4 Rack 1 Data Node 8 Rack 2 Data Node 12 Rack 3 Metadata file.txt= A: 1, 5, 6 B: 5, 9, 10 C: 9, 1, 2 Blocks of data are replicated to multiple nodes Behavior is controlled by replication factor, configurable per file - Default is 3 replicas Replication is rack-aware to reduce inter-rack network hops/latency: 1 copy in first rack 2 nd and 3 rd copy together in a separate rack. 9

10 MapReduce Processing Summary! Map Reduce computation model - Data stored in a distributed file system spanning many inexpensive computers - Bring function to the data - Distribute application to the compute resources where the data is stored! Scalable to thousands of nodes and petabytes of data public static class TokenizerMapper extends Mapper<Object,Text,Text,IntWritable> { private final static IntWritable one = new IntWritable(1); private Text word = new Text(); Hadoop Data Nodes } public void map(object key, Text val, Context StringTokenizer itr = new StringTokenizer(val.toString()); } while (itr.hasmoretokens()) { word.set(itr.nexttoken()); context.write(word, one); } public static class IntSumReducer extends Reducer<Text,IntWritable,Text,IntWrita private IntWritable result = new IntWritable(); public void reduce(text key, Iterable<IntWritable> val, Context context){ int sum = 0;... for (IntWritable v : val) { sum += v.get(); MapReduce Application Distribute map tasks to cluster Shuffle 1. Map Phase (break job into small parts) 2. Shuffle (transfer interim output for final processing) 3. Reduce Phase (boil all output down to a single result set) 10 Result Set Return a single result set

11 Map Task Client How many times does Hello appear in Wordcount.txt MR Application Master Name Node Count Hello in Block C Map Task Map Task Map Task Data Node 1 Data Node 5 Data Node 9 Count=8 A Count=3 B Count=10 C 11

12 Reduce Task Client MR Application Master Results.txt Count=21 HDFS Reduce Task Data Node 3 Sum of Hello from Map tasks Count=8 Count=3 Count=10 Map Task Map Task Map Task Data Node 1 Data Node 5 Data Node 9 A B C 12

13 Hadoop Open Source Ecosystem Diagram Workflow Management Resource Management Data Access Processing Framework MR v2 YARN Coordination Zookeeper Data Storage IBM Corporation 1

14 What is Apache Spark?! Spark is a fast and general engine for large-scale data processing! Speed - In memory, Spark can run programs up to 100X faster than MapReduce - On disk, Spark can run programs up to 10X faster than MapReduce! Application Development - Java - Scala - Python 14

15 What is Apache Spark?! High Level Capabilities - SQL - Streaming - Complex Analytics (ML, R, etc)! Run on multiple platforms - Hadoop, Mesos, standalone, Cloud! Connectivity - HDFS, Cassandra, Hbase, S3, JDBC, ODBC 15

16 Yet Another Resource Scheduler (YARN) Generic scheduling and resource management Support more than just MapReduce Support for more workloads (Hadoop 1.x Map Reduce was primarily batch processing) 16

17 What is HBase?! An open source Apache Top Level Project - An industry leading implementation of Google s BigTable Design - Considered as the Hadoop database - HBase powers some of the leading sites on the Web! A NoSQL data store - NoSQL stands for Not Only SQL - Flexible data model to accommodate semi-structured data - Cost Effective to handle Peta-bytes of data - Traditional RDBMS sharding (partitioning) lacks flexibility.! Why HBASE? - Key, Value store Column Oriented - Highly Scalable: Automatic partitioning, scales linearly and automatically with new nodes. - Low Latency: Support random read/write, small range scan - Highly Available - Strong Consistency - Flexible Data Model, very good for sparse data (no fixed columns) 17

18 How to Analyze Large Data Sets in Hadoop! Although the Hadoop framework is implemented in Java, MapReduce applications do not need to be written in Java! To abstract complexities of Hadoop programming model, a few application development languages have emerged that build on top of Hadoop: - Pig - Hive 18

19 What is Hive? SQL Map Reduce Output! Developed by Facebook in 2007! Provides a Hive-QL or HQL, SQL interface: - DDL and DML are similar to SQL - HQL Queries are translated into Map Reduce jobs - Schema-on-read capability: Projects a table structure onto existing data.! Not a true RDBMS: - Suited for batch mode processing, not real-time, latency. - No transaction support, no single row INSERT, no UPDATE or DELETE. - Limited SQL Support 19

20 Pig: Data Transformation! Pig vs MapReduce 20

21 --split-by tbl_primarykey Data Ingestion - Structured! Sqoop Efficient transferring bulk data between Hadoop clusters and RDBMS $sqoop import --connect jdbc:db2://db2.my.com:50000/sample --username db2user --password db2pwd --table ORDERS -split-by tbl_primarykey target-dir sqoopimports $sqoop export --connect jdbc:db2://db2.my.com:50000/sample --username db2user --password db2pwd --table ORDERS --exportdir /sqoop/datafile.csv 21

22 Data Ingestion - Unstructured! Flume Distributed data collection service Aggregates data from one or many sources to a centralized place Great for logs, twitter feeds, unstructured data in general 22

23 Other Hadoop Related Projects! Data serialization: AVRO - Uses JSON schemas for defining data types, data serialization in compact format! Machine learning: MAHOUT - Library of scalable machine-learning algorithms! Distributed coordination: ZOOKEEPER - Distributed configuration, synchronization, naming registry! Jobs management: OOZIE - Simplifies workflow and coordination of MapReduce jobs 23

24 Introduction to BigInsights & Open Data Platform Initiative (ODPi) 24

25 Goal of the Apache Software Foundation: Let 1000 Flowers Bloom! 249 Top Level Projects, 40 Incubating 2 Million+ Code Commits IBM co-founded the ASF in 1999 and is a Gold Sponsor The Apache Way is about fostering open innovation Not a standards organization 25

26 Harmonize on Open Data Platform to Accelerate Big Data Solutions! Over 30 members from the leading companies in the world! Provides a solid foundation of standard core Apache Hadoop components Goal: Achieve standardization and interoperability of software from ODP members 26

27 Goal of the ODP: Enable Innovation to Flourish on a Common Platform Complements the Apache Software Foundation s governance model ODP efforts focus on integration, testing, and certifying a standard core of Apache Hadoop ecosystem projects Fixes for issues found in ODP testing will be contributed to the ASF projects in line with ASF processes The ODP will not override or replace any aspect of ASF governance 27

28 IBM Open Data Platform as of V4.0 Open Data Platform (ODP) benefits and IBM open source project currency commitment Component IBM Open Platform V4.0 (ODPi) Hortonworks HDP 2.2 (ODPi) Cloudera CDH 5.3 Ambari N/A Flume Hadoop/YARN Hbase Hive Knox N/A Oozie Pig Slider N/A Solr Spark Sqoop Zookeeper

29 IBM Open Data Platform as of V4.1 Open Data Platform (ODP) benefits and IBM open source project currency commitment Component IBM Open Platform V4.1 (ODPi) Hortonworks HDP 2.3 (ODPi) Cloudera CDH Ambari N/A Flume Hadoop / YARN Hbase Hive Kafka Knox N/A Oozie Pig Slider N/A Solr Spark Sqoop Zookeeper

30 Enabling Personas with Capabilities IBM Value Need Persona Business Analyst Data Scientist Administrator Discover data for analysis Visualize data for action Reduce learning curve by leveraging existing skills (SQL, spreadsheets) Complete and Fast Big SQL runs 100% of Hadoop-DS queries and 3.6x times faster query time over Impala (Audited Hadoop-DS benchmark) Identify patterns, trends, insights with machine learning algorithms Apply statistical models to large scale data Customer Insight Large financial services company analyzed 4 billion tweets and identified 110 million client profiles that matched with at least 90 percent precision Manage workloads and schedule jobs to ensure performance Secure environment to reduce risk Performance 4x improvement in running MapReduce jobs over (STAC report) 30

31 Packaging Structure IBM BigInsights for Apache Hadoop IBM BigInsights Analyst Big SQL BigSheets IBM BigInsights Data Scientist Text Analytics Machine Learning on Big R Big R Big SQL BigSheets IBM BigInsights Enterprise Management POSIX Distributed Filesystem Multi-workload, Multi-tenant scheduling IBM Open Platform with Apache Hadoop 31

32 Text Analytics

33 BigInsights and Text Analytics Distills structured info from unstructured text - Sentiment analysis - Consumer behavior - Illegal or suspicious activities - Parses text and detects meaning with annotators Understands the context in which the text is analyzed Unstructured text (document, , etc) Football World Cup 2010, one team distinguished themselves well, losing to the eventual champions 1-0 in the Final. Early in the second half, Netherlands striker, Arjen Robben, had a breakaway, but the keeper for Spain, Iker Casillas made the save. Winger Andres Iniesta scored for Spain for the win. Features pre-built extractors for names, addresses, phone numbers, etc Classification and Insight Multiple Languages 33

34 Text Analytics Tooling Web-based tool to define rules to extract data and derive information from unstructured text Graphical interface to describe structure of various textual formats from log file data to natural language 34

35 BigSheets

36 BigSheets Browser based analytics tool for BigData! Explore, visualize, transform unstructured and structured data! Visual data cleansing and analysis! Filter and enrich content! Visualize Data! Export data into common formats No programming knowledge needed! 36

37 Geospatial Analytics! New for Version 4! Lattitude/Longitude Inputs in WKT format! Over 35 Geo Spatial functions such as: - Area - Distance - Contains - Crosses - Difference - Union - many more! 37

38 Big R and Scalable Machine Learning

39 Challenges with Running Large-Scale Analytics TRADITIONAL APPROACH BIG DATA APPROACH All available information Analyzed information All available information analyzed Analyze small subsets of information Analyze all information 39

40 User Experience for Big R Connect to BI cluster Data frame proxy to large data file Data transformation step Run scalable linear regression on cluster 40

41 NEW: Underneath Big R s ML Algorithms High-level declarative language with R-like syntax Cost-based Optimizer Automatic parallelization Execution plan based on data characteristics and Hadoop configuration 5+ Years Development in IBM Research Key Benefits: " Automatic performance tuning " Protects data science investment as platform progresses 41

42 Big R Machine Learning -- Scalability and Performance 28X Speedup Performance (data fit in memory) R out-of-memory Scalability (data larger than aggr. memory) bigr.lm 28x Scales beyond cluster memory 42

43 3 Key Capabilities of Big R 1 Use of familiar R language on Hadoop - Running native R functions - Existing R assets (code & CRAN) 2 NEW: Run scalable machine learning algorithms beyond R in Hadoop - Wide class of algorithms and growing - R-like syntax for new algorithms & customize existing algorithms 3 NEW: Leverage scale of Hadoop for faster insights - Only IBM can use the entire cluster memory - Only IBM can spill to disk - Only IBM can run thousands of models in parallel 43

44 Big SQL

45 SQL is the most prevalent analytical skill available in most data driven organizations SQL ON HADOOP = BROAD ACCESS TO ANALYTICS ON HADOOP 45

46 SQL on Hadoop Matters for Big Data Analytics For BI Tools like Cognos 46 Visualizations from Cognos

47 Hive is Really 3 Things Storage Model, Metastore, and Execution Engine Applications Hive (Open Source) SQL Execution Engine Hive Metastore (open source) MapReduce CSV Tab Delim. Parquet RC Others Hive Storage Model (open source) 47 47

48 Big SQL preserves open source foundation Co-exists with Hive by using metastore and storage formats. No Lock-in. Applications IBM BigSQL (IBM) Hive (Open Source) SQL Execution Engines Hive Metastore (open source) C/C++ MPP Engine MapReduce CSV Tab Delim. Parquet RC Others Hive Storage Model (open source) 48 48

49 Ok. But.. WHY WOULD I WANT TO DO THAT? 49

50 IBM First to Produce Audited Benchmark Hadoop-DS (based on TPC-DS)! Letters of attestation are available for both Hadoop- DS benchmarks at 10TB and 30TB scale! InfoSizing, Transaction Processing Performance Council Certified Auditors verified both IBM results as well as results on Cloudera Impala and HortonWorks HIVE.! These results are for a non- TPC benchmark. A subset of the TPC-DS Benchmark standard requirements was implemented 50

51 IBM Big SQL Runs 100% of the queries Other environments require significant effort at scale Key points! With Impala and Hive, many queries needed to be re-written, some significantly! Owing to various restrictions, some queries could not be rewritten or failed at run-time! Re-writing queries in a benchmark scenario where results are known is one thing doing this against real databases in production is another 51 Results for 10TB scale shown here

52 Hadoop-DS benchmark Single user 10TB Big SQL is 3.6x faster than Impala and 5.4x faster than Hive 0.13 for single query stream using 46 common queries Based on IBM internal tests comparing BigInsights Big SQL, Cloudera Impala and Hortonworks Hive (current versions available as of 9/01/2014) running on identical hardware. The test workload was based on the latest revision of the TPC-DS benchmark specification at 10TB data size. Successful executions measure the ability to execute queries a) directly from the specification without modification, b) after simple modifications, c) after extensive query rewrites. All minor modifications are either permitted by the TPC-DS benchmark specification or are of a similar nature. All queries were reviewed and attested by a TPC certified auditor. Development effort measured time required by a skilled SQL developer familiar with each system to modify queries so they will execute correctly. Performance test measured scaled query throughput per hour of 4 concurrent users executing a common subset of 46 queries across all 3 systems at 10TB data size. Results may not be typical and will vary based 52 on actual workload, configuration, applications, queries and other variables in a production environment. Cloudera, the Cloudera logo, Cloudera Impala are trademarks of Cloudera. Hortonworks, the Hortonworks logo and other Hortonworks trademarks are trademarks of Hortonworks Inc. in the United States and other countries.

53 But, benchmarking against Hive ISN T ALL THAT INTERESTING ANYMORE 53

54 Hadoop-DS Performance Test Update Spark SQL vs. Big SQL November 14, 2015

55 Performance Test Hadoop-DS (based on TPC-DS) 20 (Physical Node) Cluster! TPC-DS stands for Transaction Processing Council Decision Support (workload) which is an industry standard benchmark for SQL Spark IBM Open Platform V Nodes Big SQL V4.1 IBM Open Platform V Nodes 55 IBM Open Platform V4.1 shipped with Spark but upgraded to *Not an official TPC-DS Benchmark.

56 But first,.is raw performance EVERYTHING? 56

57 Big SQL Security Best In Class Role Based Access Control Row Level Security BRANCH_B BRANCH_A FINANCE Colum Level Security Separation of Duties / Audit 57

58 Big SQL runs more SQL out-of-box Big SQL 4.1 Spark SQL Big SQL is the only engine that can execute all 99 queries with minimal porting effort 58 Porting Effort: 1 hour 3-4 weeks

59 Big SQL 4.1 vs. Spark 1.5.0, Single 1TB 59

60 Big SQL 4.1 vs. Spark 1.5.0, Single 1TB 60

61 Big SQL 4.1 vs. Spark 1.5.0, Single 1TB 61

62 Big SQL 4.1 vs. Spark 1.5.0, Single 1TB 62

63 Conclusions: Big SQL vs. Spark 1TB TPC-DS! Single Stream Results: - Big SQL was faster than Spark SQL 76 / 99 Queries When Big SQL was slower, it was only slower by 1.6X on average - Query vs. Query, Big SQL was on average 5.5X faster - Removing Top 5 / Bottom 5, Big SQL was 2.5X faster 63

64 But that was just a single stream/user. What happens when you SCALE IT.? 64

65 But, what happens when you scale it? More Users Scale Single Stream 4 Concurrent Streams More Data 1 TB Big SQL was faster on 76 / 99 Queries Big SQL averaged 5.5X faster Removing Top / Bottom 5, Big SQL averaged 2.5X faster 10 TB Big SQL was faster on 87/99 Queries Spark SQL FAILED on 7 queries Big SQL averaged 6.2X faster* Removing Top / Bottom 5, Big SQL averaged 4.6X faster Spark SQL FAILED on 3 queries Big SQL was 4.4X faster* Big SQL elapsed time for workload was better than linear Spark SQL could not complete the workload (numerous issues). Partial results possible with only 2 concurrent streams. 65 *Compares only queries that both Big SQL and Spark SQL could complete (benefits Spark SQL)

66 Recommendation: Right Tool for the Right Job Big SQL Spark SQL Migrating existing workloads to Hadoop Security Many Concurrent Users Best in-class Performance Machine Learning Simpler SQL Good Performance 66 Ideal tool for BI Data Analysts and production workloads Ideal tool for Data Scientists and discovery Not Mutually Exclusive. Big SQL & Spark SQL can co-exist in the cluster

67 QUESTIONS 67