Spark, Shark and Spark Streaming Introduction

Transcription

1 Spark, Shark and Spark Streaming Introduction Tushar Kale June 2015

2 This Talk Introduction to Shark, Spark and Spark Streaming Architecture Deployment Methodology Performance References

3 Project History 2002 Google 2004 MapReduce paper 2006 Yahoo 2008 Hadoop Summit 2010 Spark paper 2014 Apache Spark top-level release in December release in March Spark is HOT!!! Most active project in Hadoop ecosystem One of top 3 most active Apache projects Databricks founded by the creators of Spark from UC Berkeley s AMPLab Activity for 6 months in 2014 (from Matei Zaharia 2014 Spark Summit)

4 What is Spark? Apache Spark is a fast, general purpose, easy-to-use cluster computing system for large-scale data processing - Fast - Leverages aggressively cached in-memory distributed computing and JVM threads - Faster than MapReduce - Generality - Covers a wide range of workloads - Provides SQL, streaming and complex analytics - Ease of use for programmers - Spark is written in Scala, an object oriented, functional programming language - Scala, Python and Java APIs - Scala and Python interactive shells - Runs on YARN, Mesos, standalone or cloud Logistic regression in Hadoop and Spark from

5 What is Big Data used For? Reports, e.g., - Track business processes, transactions Diagnosis, e.g., - Why is user engagement dropping? - Why is the system slow? - Detect spam, worms, viruses, DDoS attacks Decisions, e.g., - Personalized medical treatment - Decide what feature to add to a product - Decide what ads to show Data is only as useful as the decisions it enables

6 Data Processing Goals Low latency (interactive) queries on historical data: enable faster decisions - E.g., identify why a site is slow and fix it Low latency queries on live data (streaming): enable decisions on real-time data - E.g., detect & block worms in real-time (a worm may infect 1mil hosts in 1.3sec) Sophisticated data processing: enable better decisions - E.g., anomaly detection, trend analysis

7 Our Goal Batch Single Stack! Interactive Streaming Support batch, streaming, and interactive computations and make it easy to compose them Easy to develop sophisticated algorithms (e.g., graph, ML algos)

8 The Need for Unification (1/2) Today s state-of-art analytics stack Logs Demux Streaming stack (e.g., Storm) Batch stack (e.g., Hadoop) Real-Time Analytics Ad-Hoc queries on historical data Interactive queries (e.g., HBase, Impala, SQL) Challenges:» Need to maintain three separate stacks Expensive and complex Hard to compute consistent metrics across stacks» Hard and slow to share data across stacks Interactive queries on historical data

9 The Need for Unification (2/2) Make real-time decisions - Detect DDoS, fraud, etc E.g.,: what s needed to detect a DDoS attack? 1. Detect attack pattern in real time streaming 2. Is traffic surge expected? interactive queries 3. Making queries fast pre-computation (batch) And need to implement complex algos (e.g., ML)!

10 Key Idea Resilient Distributed Datasets (RDDs) Spark s basic unit of data Immutable, fault tolerant collection of elements that can be operated on in parallel across a cluster Two types of operations - Transformations - Data lineage DAG (Directed Acyclic Graph) - Single run with many stages, versus multiple jobs with MR - Lazy evaluations - Actions - Performs transformations and action - Returns a value Fault tolerance - If data in memory is lost it will be recreated from lineage Caching, persistence (memory, spilling, disk) and check-pointing

11 Motivation Many important applications must process large streams of live data and provide results in near-real-time - Social network trends - Website statistics - Ad impressions Distributed stream processing framework is required to - Scale to large clusters (100s of machines) - Achieve low latency (few seconds)

12 Integration with Batch Processing Many environments require processing same data in live streaming as well as batch post processing Existing framework cannot do both - Either do stream processing of 100s of MB/s with low latency - Or do batch processing of TBs / PBs of data with high latency Extremely painful to maintain two different stacks - Different programming models - Double the implementation effort - Double the number of bugs

13 Stateful Stream Processing Traditional streaming systems have a record-at-a-time processing model - Each node has mutable state - For each record, update state and send new records State is lost if node dies! input records input records mutable state node 1 node 2 node 3 Making stateful stream processing be faulttolerant is challenging

14 Existing Streaming Systems Storm - Replays record if not processed by a node - Processes each record at least once - May update mutable state twice! - Mutable state can be lost due to failure! Trident Use transactions to update state - Processes each record exactly once - Per state transaction to external database is slow

15 Spark Streaming

16 DStream Input Sources Scalable, high-throughput, fault-tolerant stream processing of live data streams

17 Discretized Stream Processing Run a streaming computation as a series of very small, deterministic batch jobs Chop up the live stream into batches of X seconds Spark treats each batch of data as Resilient Distributed Datasets RDDs and processes them using RDD operations Finally, the processed results of the RDD operations are returned in batches live data stream batches of X seconds processed results Spark Streaming Spark

18 Discretized Stream Processing Run a streaming computation as a series of very small, deterministic batch jobs Batch sizes as low as ½ second, latency of about 1 second Potential for combining batch processing and streaming processing in the same system live data stream batches of X seconds processed results Spark Streaming Spark

19 Example Get hashtags from Twitter val tweets = ssc.twitterstream() DStream: a sequence of RDDs representing a stream of data Twitter Streaming API t t+1 t+2 tweets DStream stored in memory as an RDD (immutable, distributed)

20 Example Get hashtags from Twitter val tweets = ssc.twitterstream() val hashtags = tweets.flatmap (status => gettags(status)) new DStream transformation: modify data in one DStream to create another DStream t t+1 t+2 tweets DStream flatmap flatmap flatmap hashtags Dstream [#cat, #dog, ] new RDDs created for every batch

21 Example Get hashtags from Twitter val tweets = ssc.twitterstream() val hashtags = tweets.flatmap (status => gettags(status)) hashtags.saveashadoopfiles("hdfs://...") output operation: to push data to external storage tweets DStream hashtags DStream t t+1 t+2 flatmap flatmap flatmap save save save every batch saved to HDFS

22 Example Get hashtags from Twitter val tweets = ssc.twitterstream() val hashtags = tweets.flatmap (status => gettags(status)) hashtags.foreach(hashtagrdd => {... }) foreach: do whatever you want with the processed data tweets DStream hashtags DStream t t+1 t+2 flatmap flatmap flatmap foreach foreach foreach Write to database, update analytics UI, do whatever you want

23 Java Example Scala val tweets = ssc.twitterstream() val hashtags = tweets.flatmap (status => gettags(status)) hashtags.saveashadoopfiles("hdfs://...") Java JavaDStream<Status> tweets = ssc.twitterstream() JavaDstream<String> hashtags = tweets.flatmap(new Function<...> { }) hashtags.saveashadoopfiles("hdfs://...") Function object

24 Window-based Transformations val tweets = ssc.twitterstream() val hashtags = tweets.flatmap (status => gettags(status)) val tagcounts = hashtags.window(minutes(1), Seconds(5)).countByValue() sliding window operation window length sliding interval window length DStream of data sliding interval

25 Arbitrary Stateful Computations Specify function to generate new state based on previous state and new data - Example: Maintain per-user mood as state, and update it with their tweets updatemood(newtweets, lastmood) => newmood moods = tweets.updatestatebykey(updatemood _)

26 Arbitrary Combinations of Batch and Streaming Computations Inter-mix RDD and DStream operations! - Example: Join incoming tweets with a spam HDFS file to filter out bad tweets tweets.transform(tweetsrdd => { }) tweetsrdd.join(spamhdfsfile).filter(...)

27 Fault-tolerance: Worker RDDs remember the operations that created them Batches of input data are replicated in memory for fault-tolerance tweets RDD flatmap input data replicated in memory Data lost due to worker failure, can be recomputed from replicated input data hashtags RDD All transformed data is fault-tolerant, and exactly-once transformations lost partitions recomputed on other workers

28 Fault-tolerance: Master Master saves the state of the DStreams to a checkpoint file - Checkpoint file saved to HDFS periodically If master fails, it can be restarted using the checkpoint file More information in the Spark Streaming guide - Link later in the presentation Automated master fault recovery coming soon

29 Unifying Batch and Stream Processing Models Spark program on Twitter log file using RDDs val tweets = sc.hadoopfile("hdfs://...") val hashtags = tweets.flatmap (status => gettags(status)) hashtags.saveashadoopfile("hdfs://...") Spark Streaming program on Twitter stream using DStreams val tweets = ssc.twitterstream() val hashtags = tweets.flatmap (status => gettags(status)) hashtags.saveashadoopfiles("hdfs://...")

30 Vision - one stack to rule them all Explore data interactively using Spark Shell to identify problems Use same code in Spark standalone programs to identify problems in production logs Use similar code in Spark Streaming to identify problems in live log streams $./spark-shell scala> val file = sc.hadoopfile( smalllogs )... scala> val filtered = file.filter(_.contains( ERROR ))... scala> val mapped = filtered.map(...).. ȯbject ProcessProductionData { def main(args: Array[String]) { val sc = new SparkContext(...) val file = sc.hadoopfile( productionlogs ) val filtered = file.filter(_.contains( ERROR )) val mapped = filtered.map(...)... } } object ProcessLiveStream { def main(args: Array[String]) { val sc = new StreamingContext(...) val stream = sc.kafkastream(...) val filtered = file.filter(_.contains( ERROR )) val mapped = filtered.map(...)... } }

31 Vision - one stack to rule them all Stream Processing Spark + Shark + Spark Streaming Ad-hoc Queries Batch Processing

32 Part2 continued THANK YOU