Data at the Speed of your Users

Transcription

1 Data at the Speed of your Users Apache Cassandra and Spark for simple, distributed, near real-time stream processing. GOTO Copenhagen 2014

2 Rustam Aliyev Solution Architect

3 Big Data? Photo: Flickr / Watches En Masse

4 " Volume # Variety $ Velocity

5 Velocity = Near Real Time

6 Near Real Time?

7 Near Real Time 0.5 sec 60 sec

8 Use Cases Photo: Flickr / Swiss Army / Jim Pennucci

9 Web Analytics Dynamic Pricing Recommendation Fraud Detection

10 Architecture Photo: Ilkin Kangarli / Baku Haydar Aliyev Center

11 Architecture Goals Low Latency High Availability Horizontal Scalability Simplicity

12 Stream Processing %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Collection Processing Storing Delivery

13 Stream Processing Collection %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Spark Cassandra Delivery

14 Cassandra Distributed Database Photo: Flickr / Hypostyle Hall / Jorge Láscar

15 Data Model

16 Partition Partition Key Cell 1 Cell 2 Cell 3

17 Partition Nexus 5 os: Android storage: 32GB version: 4.4 weight: 130g sort order on disk

18 Table Nexus 5 os: Android storage: 32GB version: 4.4 weight: 130g iphone 6 os: ios storage: 64GB version: 8.0 weight: 129g

19 Distribution

20 0000 Nexus 3D97 5 E C A

21 0000 iphone 9C4F 6 E C000 3D A

22 Replication

23 1 replica 0000 E C000 3D C4F A

24 2 replicas 0000 E C4F C000 3D C4F A000 3D

25 Spark Distributed Data Processing Engine Photo: Flickr / Sparklers / Alexandra Compo / CreativeCommons

26 Fast In-memory

27 Logistic Regression 4000 Running Time (s) Spark Hadoop Number of Iterations

28 Easy

29 map reduce

30 map filter groupby sort union join leftouterjoin rightouterjoin reduce count fold reducebykey groupbykey cogroup cross zip sample take first partitionby mapwith pipe save...

31 RDD Resilient Distributed Datasets Node 2 Node 1 Node 3 Node 2 Node 1 Node 3

32 Operator DAG groupby join map filter Disk RDD Memory RDD

33 Spark Streaming Micro-batching

34 RDD Data Stream DStream

35 Spark + Cassandra DataStax Spark Cassandra Connector

36

37 M Cassandra Spark Worker M M Spark Master & Worker

38 Demo Twitter Analytics

39 Cassandra Data Model

40 #hashtag ALL: : : : 129 sort order

41 CREATE TABLE hashtags ( hashtag text, interval text, mentions counter, PRIMARY KEY((hashtag), interval) ) WITH CLUSTERING ORDER BY (interval DESC);

42 Processing Data Stream

43 import com.datastax.spark.connector.streaming._ val sc = new SparkConf().setMaster("spark:// :7077").setAppName("Twitter- Demo").setJars("demo- assembly- 1.0.jar")).set("spark.cassandra.connection.host", " ") val ssc = new StreamingContext(sc, Seconds(2)) val stream = TwitterUtils. createstream(ssc, None, Nil, storagelevel = StorageLevel.MEMORY_ONLY_SER_2) val hashtags = stream.flatmap(tweet => tweet.gettext.tolowercase.split(" "). filter(tags.contains(seq("#iphone", "#android")))) val tagcounts = hashtags.map((_, 1)).reduceByKey(_ + _) val tagcountsall = tagcounts.map{ case (tag, mentions) => (tag, mentions, "ALL") }

44 import com.datastax.spark.connector.streaming._ val sc = new SparkConf().setMaster("spark:// :7077").setAppName("Twitter- Demo").setJars("demo- assembly- 1.0.jar")).set("spark.cassandra.connection.host", " ") val ssc = new StreamingContext(sc, Seconds(2)) val stream = TwitterUtils. createstream(ssc, None, Nil, storagelevel = StorageLevel.MEMORY_ONLY_SER_2) val hashtags = stream.flatmap(tweet => tweet.gettext.tolowercase.split(" "). filter(tags.contains(seq("#iphone", "#android")))) val tagcounts = hashtags.map((_, 1)).reduceByKey(_ + _) val tagcountsall = tagcounts.map{ case (tag, mentions) => (tag, mentions, "ALL") }

45 import com.datastax.spark.connector.streaming._ val sc = new SparkConf().setMaster("spark:// :7077").setAppName("Twitter- Demo").setJars("demo- assembly- 1.0.jar")).set("spark.cassandra.connection.host", " ") val ssc = new StreamingContext(sc, Seconds(2)) val stream = TwitterUtils. createstream(ssc, None, Nil, storagelevel = StorageLevel.MEMORY_ONLY_SER_2) val hashtags = stream.flatmap(tweet => tweet.gettext.tolowercase.split(" "). filter(tags.contains(seq("#iphone", "#android")))) val tagcounts = hashtags.map((_, 1)).reduceByKey(_ + _) val tagcountsall = tagcounts.map{ case (tag, mentions) => (tag, mentions, "ALL") }

46 val ssc = new StreamingContext(sc, Seconds(2)) val stream = TwitterUtils. createstream(ssc, None, Nil, storagelevel = StorageLevel.MEMORY_ONLY_SER_2) val hashtags = stream.flatmap(tweet => tweet.gettext.tolowercase.split(" "). filter(tags.contains(seq("#iphone", "#android")))) val tagcounts = hashtags.map((_, 1)).reduceByKey(_ + _) val tagcountsall = tagcounts.map{ case (tag, mentions) => (tag, mentions, "ALL") } tagcountsall.savetocassandra( "demo_ks", "hashtags", Seq("hashtag", "mentions", "interval")) ssc.start() ssc.awaittermination()

47 val ssc = new StreamingContext(sc, Seconds(2)) val stream = TwitterUtils. createstream(ssc, None, Nil, storagelevel = StorageLevel.MEMORY_ONLY_SER_2) val hashtags = stream.flatmap(tweet => tweet.gettext.tolowercase.split(" "). filter(tags.contains(seq("#iphone", "#android")))) val tagcounts = hashtags.map((_, 1)).reduceByKey(_ + _) val tagcountsbyday = tagcounts.map{ case (tag, mentions) => (tag, mentions, DateTime.now.toString("yyyyMMdd")) } tagcountsbyday.savetocassandra( "demo_ks", "hashtags", Seq("hashtag", "mentions", "interval")) ssc.start() ssc.awaittermination()

48 val ssc = new StreamingContext(sc, Seconds(2)) val stream = TwitterUtils. createstream(ssc, None, Nil, storagelevel = StorageLevel.MEMORY_ONLY_SER_2) val hashtags = stream.flatmap(tweet => tweet.gettext.tolowercase.split(" "). filter(tags.contains(seq("#iphone", "#android")))) val tagcounts = hashtags.map((_, 1)).reduceByKey(_ + _) val tagcountsall = tagcounts.map{ case (tag, mentions) => (tag, mentions, "ALL") } tagcountsall.savetocassandra( "demo_ks", "hashtags", Seq("hashtag", "mentions", "interval")) ssc.start() ssc.awaittermination()

49 Questions?