Massively scalable NoSQL with Apache Cassandra! Jonathan Ellis Project Chair, Apache Cassandra CTO,

Transcription

1 Massively scalable NoSQL with Apache Cassandra! Jonathan Ellis Project Chair, Apache Cassandra CTO,

2 Cassandra Job Trends

3 Big Data trend

4 Why Big Data Matters

5 Big data Analytics (Hadoop)? Realtime ( NoSQL )

6 Some Casandra users

7 Industries & use cases Social Media Financial Advertising Entertainment Energy E-tail Health care Government Time series data Messaging Ad tracking Data mining User activity streams User sessions Anything requiring: Scalable, performant, and highly available

8 Why Cassandra? Fully distributed, no SPOF Multi-master, multi-dc Linearly scalable Larger-than-memory datasets Best-in-class performance (not just writes!) Fully durable Integrated caching Tuneable consistency

9 Availability There is no such thing as standby infrastructure: there is stuff you always use and stuff that won t work when you need it. -- Ben Black: co-founder, Boundary The biggest problem with failover is that you're almost never using it until it really hurts. It's like backups that you never test. -- Rick Branson: Instagram

10 Classic partitioning with SPOF partition 1 partition 2 partition 3 partition 4 router client

11 Fully distributed, no SPOF client p3 p1 p6 p1 p1

12

13 Partitioning jim carol age: 36 car: camaro gender: M age: 37 car: subaru gender: F johnny suzy age:12 age:10 gender: M gender: F

14 Partitioning Primary key determines placement* jim carol age: 36 car: camaro gender: M age: 37 car: subaru gender: F johnny suzy age:12 age:10 gender: M gender: F

15 PK MD5 Hash jim carol johnny suzy 5e a9a f4eb27cea b421309e... MD5 hash operation yields a 128-bit number for keys of any size.

16 The token ring Node A Node B Node D Node C

17 Start End A B C D 0xc x x x x x x xc jim carol johnny suzy 5e a9a f4eb27cea b421309e...

18 Start End A B C D 0xc x x x x x x xc jim carol johnny suzy 5e a9a f4eb27cea b421309e...

19 Start End A B C D 0xc x x x x x x xc jim carol johnny suzy 5e a9a f4eb27cea b421309e...

20 Start End A B C D 0xc x x x x x x xc jim carol johnny suzy 5e a9a f4eb27cea b421309e...

21 Start End A B C D 0xc x x x x x x xc jim carol johnny suzy 5e a9a f4eb27cea b421309e...

22 Replication Node A Node B Node D Node C carol a9a

23 Node A Node B Node D Node C carol a9a

24 Node A Node B Node D Node C carol a9a

25 Highlights Adding capacity is application-transparent and requires no downtime No SPOF, not even temporarily No primary replica Configurable synchronous/asynchronous Tolerates node failure; never have to restart replication from scratch Smart replication avoids correlated failures

26 What about performance? Log-structured storage engine avoids random i/o no write amplification Excellent performance on both reads and writes Row-level isolation via concurrent algorithms no locking Built-in compression improves cache hotness Row cache can replace memcached

27 reads/s writes/s Cassandra 0.6 Cassandra 1.0 0

28

29 CQL: You got SQL in my NoSQL! CREATE TABLE USERS ( id uuid PRIMARY KEY, name text, state text, birth_date int );

30 CREATE TABLE timeline ( user_id uuid, tweet_id uuid, author varchar, body varchar, PRIMARY KEY (user_id, tweet_id) );

31 INSERT INTO users (id, name, state, birth_date) VALUES (8321b1a8-d0e8-11e1-bcfa-34159e154f4c, jbellis, Texas, 1976); UPDATE users SET name= jbellis, state= Texas, birth_date=1976 WHERE id = 8321b1a8-d0e8-11e1-bcfa-34159e154f4c;

32 SELECT * FROM users WHERE id= jbellis ;

33 CREATE INDEX ON users(state); SELECT * FROM users WHERE state= Texas AND birth_date > 1950;

34 from pycassa.pool import ConnectionPool from pycassa.columnfamily import ColumnFamily pool = ConnectionPool('Demo', ['localhost:9160']) cf = ColumnFamily(pool, 'users') cf.insert(uuid.uuid4(), {'name': 'jbellis', 'state': 'Texas', 'birth_date': 1976}) from pycassa.types import * from pycassa.columnfamilymap import ColumnFamilyMap class User(object): key = UUIDType() name = Utf8Type() state = Utf8Type() birth_date = IntegerType() cfmap = ColumnFamilyMap(User, pool, 'UserInfo') user = cfmap.get('jbellis')

35 Strictly realtime focused No joins No subqueries No aggregation functions* or GROUP BY

36 Big data Analytics (Hadoop)? Realtime ( NoSQL )

37 The evolution of Analytics Analytics + Realtime

38 The evolution of Analytics replication Analytics Realtime

39 The evolution of Analytics ETL

40 Big data Analytics (Hadoop) Datastax Enterprise Realtime (Cassandra)

41 Reunification of realtime + analytics

42

43 Portfolio Demo dataflow Portfolios Historical Prices Portfolios Live Prices for today Intermediate Results Largest loss Largest loss

44 Better Hadoop than Hadoop Vanilla Hadoop 8+ services to setup, monitor, backup, and recover (NameNode, SecondaryNameNode, DataNode, JobTracker, TaskTracker, Zookeeper, Region Server,...) Single points of failure Can't separate online and offline processing DataStax Enterprise Single, simplified component Self-organizes based on workload Peer to peer JobTracker failover

45 Enterprise search with Solr SELECT title FROM solr WHERE solr_query='title:natio*'; title Bolivia national football team 2002 List of French born footballers who have played for other national teams Lithuania national basketball team at Eurobasket 2009 Bolivia national football team 2000 Kenya national under-20 football team Bolivia national football team 1999 Israel men's national inline hockey team Bolivia national football team 2001

46 Managing & Monitoring Big Data DataStax OpsCenter manages and monitors all Cassandra and Hadoop operations

47 Questions?