Massively scalable NoSQL with Apache Cassandra! Jonathan Ellis Project Chair, Apache Cassandra CTO,

Transcription

1 Massively scalable NoSQL with Apache Cassandra! Jonathan Ellis Project Chair, Apache Cassandra CTO,

2 Big data Analytics (Hadoop)? Realtime ( NoSQL )

3 Some Casandra users

4 ebay Application/Use Case Social Signals: like/want/own features for ebay product and item pages Hunch taste graph for ebay users and items Many time series use cases Why Cassandra? Multi-datacenter Scalable Write performance Distributed counters Hadoop support ACE

5 Time series data

6 Multi-datacenter support

7 Distributed counters

8 Hadoop support

9 Disney Application/Use Case Meet the data management needs of user facing applications across The Walt Disney Company with a single platform Why Cassandra? DataStax Enterprise can tackle real-time and search functions in the same cluster Scalability 24x7 uptime NDI

10 Multitenancy

11 Multitenancy

12 Enterprise search

13 SimpleReach Application/Use Case SimpleReach tracks social actions for content creators, from Twitter and Facebook to Pinterest and Reddit, to deliver detailed insights and clear metrics around social behavior. Why Cassandra? Very high velocity data ingest rate and large data volumes Workload separation between realtime and batch applications NDE

14 SourceNinja Application/Use Case SourceNinja notifies you to performance, security, and bug fixes for the software you depend on Why Cassandra? Previous database system could not handle load; HBase has too many points of failure and was too slow Fast real time capabilities, batch analytics on that data, and enterprise search RDE

15 Netflix Application/Use Case General purpose backend for large scale highly available cloud based web services supporting Netflix Streaming Why Cassandra? Highly available, highly robust and no schema change downtime Highly scalable, optimized for SSD Much lower cost than previous Oracle and SimpleDB implementations Flexible data model Ability to directly influence/implement OSS feature set Supports local and wide area distributed operations, spanning US and Europe RCE

16 Optimized for SSD

17 Open source

18 Use case patterns Massively scalable High performance Reliable/Available

19

20 reads/s writes/s Cassandra 0.6 Cassandra 1.0 0

21

22 Classic partitioning with SPOF partition 1 partition 2 partition 3 partition 4 router client

23 Availability High availability implies that a single fault will not bring down your system. Not we ll recover quickly. -- Ben Coverston: DataStax The biggest problem with failover is that you're almost never using it until it really hurts. It's like backups that you never test. -- Rick Branson: Instagram

24 Fully distributed, no SPOF client p3 p1 p6 p1 p1

25 Partitioning jim carol age: 36 car: camaro gender: M age: 37 car: subaru gender: F johnny suzy age:12 age:10 gender: M gender: F

26 Partitioning Primary key determines placement* jim carol age: 36 car: camaro gender: M age: 37 car: subaru gender: F johnny suzy age:12 age:10 gender: M gender: F

27 PK MD5 Hash jim carol johnny suzy 5e a9a f4eb27cea b421309e... MD5* hash operation yields a 128-bit number for keys of any size.

28 The token ring Node A Node B Node D Node C

29 Start End A B C D 0xc x x x x x x xc jim carol johnny suzy 5e a9a f4eb27cea b421309e...

30 Start End A B C D 0xc x x x x x x xc jim carol johnny suzy 5e a9a f4eb27cea b421309e...

31 Start End A B C D 0xc x x x x x x xc jim carol johnny suzy 5e a9a f4eb27cea b421309e...

32 Start End A B C D 0xc x x x x x x xc jim carol johnny suzy 5e a9a f4eb27cea b421309e...

33 Start End A B C D 0xc x x x x x x xc jim carol johnny suzy 5e a9a f4eb27cea b421309e...

34 Replication Node A Node B Node D Node C carol a9a

35 Node A Node B Node D Node C carol a9a

36 Node A Node B Node D Node C carol a9a

37 Highlights Adding capacity is application-transparent and requires no downtime No SPOF, not even temporarily No primary replica Configurable synchronous/asynchronous Tolerates node failure; never have to restart replication from scratch Smart replication avoids correlated failures

38 CQL: You got SQL in my NoSQL! CREATE TABLE users ( id uuid PRIMARY KEY, name text, state text, birth_date int ); CREATE INDEX ON users(state); SELECT * FROM users WHERE state= Texas AND birth_date > 1950;

39 Strictly realtime focused No joins No subqueries No aggregation functions* or GROUP BY ORDER BY?

40 SELECT * FROM tweets WHERE user_id IN (SELECT follower FROM followers WHERE user_id = driftx ) followers? tweets

41 Clustering in Cassandra CREATE TABLE timeline ( user_id uuid, tweet_id timeuuid, tweet_author uuid, tweet_body text, PRIMARY KEY (user_id, tweet_id) ); user_id tweet_id _author _body jbellis 3290f9da.. rbranson lorem jbellis a.. tjake ipsum driftx 3290f9da.. rbranson lorem driftx 71b46a84.. yzhang dolor yukim 3290f9da.. rbranson lorem yukim e451dd42.. tjake amet

42 Clustering in Cassandra CREATE TABLE timeline ( user_id uuid, tweet_id timeuuid, tweet_author uuid, tweet_body text, PRIMARY KEY (user_id, tweet_id) ); SELECT * FROM timeline WHERE user_id = driftx ; user_id tweet_id _author _body jbellis 3290f9da.. rbranson lorem jbellis a.. tjake ipsum driftx 3290f9da.. rbranson lorem driftx 71b46a84.. yzhang dolor yukim 3290f9da.. rbranson lorem yukim e451dd42.. tjake amet

43 Collections CREATE TABLE users ( id uuid PRIMARY KEY, name text, state text, birth_date int ); CREATE TABLE users_addresses ( user_id uuid REFERENCES users, text ); SELECT * FROM users NATURAL JOIN users_addresses;

44 Collections CREATE TABLE users ( id uuid PRIMARY KEY, name text, state text, birth_date int ); X CREATE TABLE users_addresses ( user_id uuid REFERENCES users, text ); SELECT * FROM users NATURAL JOIN users_addresses;

45 Collections CREATE TABLE users ( id uuid PRIMARY KEY, name text, state text, birth_date int, _addresses set<text> ); UPDATE users SET _addresses = _addresses + { jbellis@gmail.com, jbellis@datastax.com };

46 Big data Analytics (Hadoop)? Realtime ( NoSQL )

47 The evolution of Analytics Analytics + Realtime

48 The evolution of Analytics replication Analytics Realtime

49 The evolution of Analytics ETL

50 Big data Analytics (Hadoop) Datastax Enterprise Realtime (Cassandra)

51

52 Better Hadoop than Hadoop Vanilla Hadoop 8+ services to setup, monitor, backup, and recover (NameNode, SecondaryNameNode, DataNode, JobTracker, TaskTracker, Zookeeper, Region Server,...) Single points of failure Can't separate online and offline processing DataStax Enterprise Single, simplified component Self-organizes based on workload Peer to peer JobTracker failover

53 Enterprise search with Solr SELECT title FROM solr WHERE solr_query='title:natio*'; title Bolivia national football team 2002 List of French born footballers who have played for other national teams Lithuania national basketball team at Eurobasket 2009 Bolivia national football team 2000 Kenya national under-20 football team Bolivia national football team 1999 Israel men's national inline hockey team Bolivia national football team 2001

54 Managing & Monitoring Big Data

55

56 Questions?