Cloudera Improvements in Apache Spark

Transcription

1 Cloudera Improvements in Apache Spark Brian Baillod Sales Engineer 1

2 Agenda Spark One PlaCorm Spark Overview and Improvements Spark Proof of Concept Kudu and Record Service 2

3 Cloudera company snapshot Founded 2008, by former employees of Employees Today 900+ worldwide World Class Support More than 75 24x7 global staff Cloudera University Over 40,000 trained We help code Hadoop Cloudera employees are leading developers & contributors to the complete Apache Hadoop ecosystem of projects We help fix Hadoop Cloudera fixed 60% of all Hadoop JIRA bugs 3

4 Hadoop Categories of Hadoop adop/on Free/Developer Over 2.5 million downloads Subscrip/on Training 60% of Fortune 100 a`ended Cloudera training, over 40,000 trained since 2009 Business Need Training Free/ Developer Services & Support Big Data Maturity Service & Support 9/10 for support ability to solve technical issues #1 recommenda/on Over 2x revenue of nearest 90% renewal rate 4

5 What is Spark Fast general purpose processing engine for large data Provides API s in Java, Scala and Python Includes an advanced DAG execu@on engine that supports in- memory compu@ng Includes high level tools like SparkSQL, Mllib, GraphX, and Spark Streaming Can run in a cluster, standalone, or local Latest version is Spark.apache.org LOTS of momentum 5

6 Cloudera One PlaCorm Cloudera is doubling down on Spark Outlining a vision for the future Kudu, Record Service, Auto- tuning, Security, Kaia integra@on Challenging other vendors to par@cipate in Spark Development 6

7 Cloudera s Engineering Commitment to Spark Spark CommiPers by Hadoop Distribu/on* Spark Patches by Hadoop Distribu/on Hortonworks 17% Intel 17% Cloudera 67% Cloudera, 370 Hortonworks, 4 IBM, 12 MapR, 1 Intel, 400 * IBM and MapR have 0 commipers 7

8 Spark will replace MapReduce To become the standard engine for Hadoop 8

9 The Future of Data Processing on Hadoop Spark complemented by specialized fit- for- purpose engines General Data Processing w/ Spark Fast Batch Processing, Machine Learning, and Stream Processing Full- Text Search w/solr Querying textual data Analy/c Database w/ Impala Low- Latency Massively Concurrent Queries On- Disk Processing w/mapreduce Jobs at extreme scale and extremely disk IO intensive Shared: Data Storage Metadata Resource Management Security Governance 9

10 Why is Cloudera leading this Cloudera was the first Hadoop vendor to ship and support Spark Spark is a fully integrated part of Cloudera s placorm Shared data, metadata, resource management, administra@on, security, and governance Cloudera is the first Hadoop vendor to offer Spark training Trained more customers than any other vendor Cloudera has more Spark customers in produc@on than all other companies combined 10

11 Spark Overview and Improvements 11

12 Apache Spark Flexible, in- memory data processing for Hadoop Easy Development Rich APIs for Scala, Java, and Python shell Flexible Extensible API APIs for different types of workloads: Batch Streaming Machine Learning Graph Fast Batch & Stream Processing In- Memory processing and caching 12

13 Easy Development High Language Support Python lines = sc.textfile(...) lines.filter(lambda s: ERROR in s).count() Scala val lines = sc.textfile(...) lines.filter(s => s.contains( ERROR )).count() Java JavaRDD<String> lines = sc.textfile(...); lines.filter(new Function<String, Boolean>() { Boolean call(string s) { return s.contains( error ); } }).count(); Na@ve support for mul@ple languages with iden@cal APIs Scala, Java, Python Use of closures, itera@ons, and other common language constructs to minimize code 2-5x less code 13

14 Python Or Scala? Use Python for prototyping Spark Python API is slower than Scala Use Scala for development Steep learning curve for programming 14

15 Easy Development Use percolateur:spark srowen$./bin/spark-shell --master local[*]... Welcome to / / / / \ \/ _ \/ _ `/ / '_/ / /. /\_,_/_/ /_/\_\ version SNAPSHOT /_/ Using Scala version (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_51) Type in expressions to have them evaluated. Type :help for more information.... scala> val words = sc.textfile("file:/usr/share/dict/words")... words: org.apache.spark.rdd.rdd[string] = MapPartitionsRDD[1] at textfile at <console>:21 Interac@ve explora@on of data for data scien@sts No need to develop applica@ons Developers can prototype applica@on on live system scala> words.count... res0: Long = scala> 15

16 Easy Development Expressive API map filter groupby sort union join leftouterjoin rightouterjoin reduce count fold reducebykey groupbykey cogroup cross zip sample take first partitionby mapwith pipe save 16

17 Memory Management for Greater Performance Memory can be enabler for high performance big data applica/ons GB RAM 50 GB per second Trends: ½ price every 18 months 2x bandwidth every 3 years 16 cores 17

18 Spark Concepts RDD Resilient Distributed Dataset Caching DataFrames Spark Streaming SparkSQL Pluggable Spark 18

19 Resilient Distributed Dataset (RDD) Read- only of records Created through: of data in storage of RDDs Contains lineage to compute from storage Lazy Users control persistence and 19

20 RDD Transforma/ons create new RDD from an one Ac/ons run on RDD and return a value Transforma@ons are lazy Ac@ons materialize RDDs by compu@ng transforma@ons RDDs can be cached to avoid re- compu@ng 20

21 Example Transforma/ons Map Filter Sample Join Ac/ons Reduce Count First, Take SaveAs 21

22 Fault- Tolerance RDDs contain lineage Lineage: Source and list of Lost can be re- computed from source data msgs = textfile.filter(lambda s: s.startswith( ERROR )).map(lambda s: s.split( \t )[2]) HDFS File Filtered RDD Mapped RDD filter (func = startswith( )) map (func = split(...)) 22

23 Caching Storage Levels Different provide tradeoffs between memory usage and CPU efficiency. Cache when using algorithms. MEMORY_ONLY most CPU efficient, data has to fit in memory MEMORY_ONLY_SER More space efficient but reasonably fast MEMORY_AND_DISK MEMORY_AND_DISK_SER DISK_ONLY MEMORY_ONLY_2, MEMORY_AND_DISK_2 23

24 Data Frames Distributed of rows organized into named columns Spark SQL s Data Source API can read and write Data Frames using a variety of formats Hive, JSON, Parquet, HDFS Calling the DataFrame API can let you Select the columns you want Join data sources Aggregate and Filter Spark 1.5 lets you access the Hive Metastore to read/write schemas directly. 24

25 Spark Streaming What is it? Run con,nuous processing of data using Spark s core API Extends Spark concepts to fault- tolerant, transformable streams Adds rolling window opera@ons Example: Compute rolling averages or counts for data over last five minutes Benefits: Same programming paradigm for streaming and batch Excellent throughput Scale easily to support large volumes of data ingest Common Use Cases: On- the- fly ETL as data is ingested into Hadoop/HDFS Detect anomalous behavior and trigger alerts Con@nuous repor@ng of summary metrics for incoming data 25

26 Spark Streaming Architectures Spark Stream Processing Data Sources Integra/on Layer Data Prep / Scoring Ingest Flume Kaia Transformed Results HBase Real- Time Result Serving HDFS Spark Long- Term Analy/cs/ Model Building 26

27 SparkSQL Machine Learning Goal: Spark/Java Developers and Data can inline SQL into Spark apps Designed for: Ease of development for Spark developers Handful of concurrent Spark jobs Strengths: Ease of embedding SQL into Java or Scala SQL for common in developer flow (eg. filters, samples) 27

28 Impala Remains Tool of Choice for SQL Time (in seconds) Single User vs 10 User Response Time/Impala Times Faster (Lower bars = be`er) Single User, 5 10 Users, 11 Single User, x 10 Users, x Impala Spark SQL Presto Hive- on- Tez Single User, x 10 Users, x Single User, x 10 Users, x 28

29 Pluggable Spark replace MapReduce Cloudera is leading community development to port components to Spark: Stage 1 Stage 2 Stage 3 Crunch on Spark Search on Spark Hive on Spark (beta) Spark on HBase (beta) Pig on Spark (alpha) Sqoop on Spark Spark on Kudu 29

30 Spark Customer Use Cases Core Spark Financial Services Health PorColio Risk Analysis ETL Pipeline Speed- Up 20+ years of stock data disease- causing genes in the full human genome Calculate Jaccard scores on health care data sets Spark Streaming Financial Services Health Online Fraud Incident for Sepsis ERP Character and Bill Retail Online Systems Real- Time Inventory Management 1010 Data Services Trend analysis Document (LDA) Fraud Ad Tech Real- Time Ad Performance Analysis 30

31 Doing the Math Executors and Cores Determine the resource for the Spark job 4 Core 4 Core C 1 Core for Applica@on Master Cores for Executors 1 Executor 4 Cores x 15 Cores 3 Executors with 4 Cores Each (Leaves 3 Cores un- u@lized) Other Ra@os may lead to be`er resource u@liza@on 4 Core 1 Executor 2 Cores x 15 Cores 7 Executors with 2 Cores Each (Leaves 1 Core un- u@lized) 4 Core Don t exceed= 5 Cores per Executor h`p://blog.cloudera.com/blog/2015/03/how- to- tune- your- apache- spark- jobs- part- 2/ 16 Total Cores in Cluster Core Alloca@on Allocate Executors 31

32 Kudu 32

33 Kudu Design Goals High throughput for big scans (columnar storage and Goal: Within 2x of Parquet Low- latency for short accesses (primary key indexes and quorum design) Goal: 1ms read/write on SSD Database- like single- row ACID) Rela/onal data model SQL query NoSQL style scan/insert/update (Java client) 33

34 Kudu Storage for Fast on Fast Data DATA ENGINEERING DATA DISCOVERY & ANALYTICS DATA APPS BATCH STREAM SQL SEARCH MODEL ONLINE SPARK, HIVE, PIG SPARK IMPALA SOLR SPARK HBASE UNIFIED DATA SERVICES RESOURCE MANAGEMENT YARN SECURITY SENTRY New column store for Hadoop Simplifies the architecture for building on changing data Designed for fast performance integrated with Hadoop FILESYSTEM HDFS DATA INTEGRATION & STORAGE RELATIONAL KUDU NoSQL HBASE Apache- licensed open source (intent to donate to ASF) INGEST SQOOP, FLUME, KAFKA Beta now available 34

35 Kudu Trade- Offs Random updates will be slower HBase model allows random updates without incurring a disk seek Kudu requires a key lookup before update, Bloom lookup before insert Single- row reads may be slower Columnar design is op@mized for scans Future: may introduce column groups for applica@ons where single- row access is more important 35

36 Resources Join the community h`p://getkudu.io Download the Beta cloudera.com/downloads Read the Whitepaper getkudu.io/kudu.pdf 36

37 RecordService 37

38 Hadoop started out with zero security Didn t need it for the Silicon Valley applica@ons Does need it for Corporate applica@ons Cloudera is working on providing full featured Spark Security 38

39 Comprehensive, Compliance- Ready Security Audit, and Compliance Perimeter Access Visibility Data Guarding access to the cluster itself Defining what users and can do with data on where data came from and how it s being used Protec@ng data in the cluster from unauthorized visibility Technical Concepts: Authen@ca@on Network isola@on Technical Concepts: Permissions Authoriza@on Technical Concepts: Audi@ng Lineage Technical Concepts: Encryp@on, Tokeniza@on, Data masking Cloudera Manager Apache Sentry & RecordService Cloudera Navigator Navigator Encrypt & Key Trustee Partners 39

40 Directory and Kerberos Directory Manages Users, Groups, and Services Provides username / password authen@ca@on Group membership determines Service access Kerberos Trusted and standard third- party Authen@cated users receive Tickets Tickets gain access to Services User [ssmith] Password[***** ] User authen@cates to AD Authen@cated user gets Kerberos Ticket Ticket grants access to Services e.g. Impala 40

41 Fine- Grained Access Control in HDFS Across All Hadoop Paths Columns: column visibility varies by role (Ex. credit card numbers) Managers: Call Center: XXXX XXXX XXXX 5678 Analysts: XXXX XXXX XXXX XXXX Others: No access to credit card column Rows: Different user groups need access to different records European privacy laws Government security clearance Financial 41

42 RecordService Unified Access Control Enforcement DATA ENGINEERING DATA DISCOVERY & ANALYTICS DATA APPS BATCH STREAM SQL SEARCH MODEL ONLINE SPARK, HIVE, PIG SPARK IMPALA SOLR SPARK HBASE UNIFIED DATA SERVICES RESOURCE MANAGEMENT YARN SECURITY SENTRY, RECORDSERVICE New high performance security layer that centrally enforces access control policies across Hadoop Complements Apache Sentry s unified policy defini@on Row- and column- based security Dynamic data masking DATA INTEGRATION & STORAGE FILESYSTEM HDFS INGEST SQOOP, FLUME, KAFKA NoSQL HBASE Apache- licensed open source Beta now available 42

43 Fine- Grained HDFS Access without RecordService Split the original file Use HDFS permissions to limit access Date//me Accnt # SSN Asset Trade Country 09:33: :33: :12: :22: :55: :22: :45: :03: :55: AAPL Sell US TBT Buy EU IBM Sell UK INTC Buy US F Buy US UA Buy UK AMZN Sell EU TMV Buy US MA Buy UK Date//me Accnt # SSN Asset Trade Country 09:33: :22: :55: :03: Date//me Accnt # SSN Asset Trade Country 11:33: :45: Date//me Accnt # SSN Asset Trade Country 14:12: :22: :55: AAPL Sell US INTC Buy US F Buy US TMV Buy US TBT Buy EU AMZN Sell EU IBM Sell UK UA Buy UK MA Buy UK 43

44 Fine- Grained HDFS Access Control with RecordService Apply controls to the master data file Row, column, and sub- column (masking) controls Enforce these across all access paths Column- Level Controls What U.S. Brokers See Column- Level Controls Date//me Accnt # SSN Asset Trade Country Date//me Accnt # SSN Asset Trade Country 09:33: AAPL Sell US 09:33: XXX- XX AAPL Sell US 11:33: :12: :22: :55: TBT Buy EU IBM Sell EU INTC Buy US F Buy US Row- Level Controls 11:33: :12: :22: :55: TBT Buy group IBM Sell group XXX- XX INTC Buy US XXX- XX F Buy US Row- Level Controls 10:22: UA Buy EU 10:22: UA Buy group3 13:45: AMZN Sell EU 13:45: AMZN Sell group2 44

45 Spark Resources Learn Spark Spark Cookbook by Rishi Yadav O Reilly Advanced Analy@cs with Spark ebook (wri`en by Clouderans) Cloudera Developer Blog cloudera.com/spark Get Trained Cloudera Spark Training 45