Getting Started with Cassandra

Transcription

1 Getting Started with Cassandra A Tutorial Ben

2 Introduction Scaling Relational Databases is Easy! It s a solved problem! Just ask EBay in

3 \ 3

4 Wow! Let s do that again! 4

5 Data Requirements Volume Velocity Variety Variability What is the word for Availability that starts with a V? 5

6 Whatever...! I need my data I need it quickly Always available Simple to operate Simple to maintain 6

7 Why Do I Need This? Data is Too Big Moves Too Fast Doesn t Fit 7

8 Who Else Uses Cassandra? 8

9 Cassandra Design Scale Linearly Continuous Availability High Performance 9

10 Apache Cassandra Google Big Table Amazon Dynamo Facebook Cassandra 10

11 Google Big Table Scalable Data Model Flexible Schema 11

12 Amazon Dynamo Scale-Out Architecture Key Distribution Automatic Partitioning Peer to Peer Architecture 12

13 Installing Cassandra 13

14 Virtual Machine: USB Drive Cassandra is there Virtual Machine Image username: notroot password: notroot Install Cassandra on the Virtual Machine (tarball in the /home/notroot directory) 14

15 Starting the VM Create a new VM in VirtualBox Name the VM Linux/64Bit 2048 RAM (could be less) Select your VM Copied from the USB Drive 15

16 Installing Cassandra 16

17 Installing Cassandra (Cont.) 17

18 Install Cassandra (Cont) Unzip the tarball to your home directory $ tar -xzvf apache-cassandra bin.tar.gz $ rm *.tar.gz 18

19 Install Cassandra (Cont) Rename The Directory $ mv apache-cassandra cassandra 19

20 Install Cassandra (Cont) Create The Data Directory $ cd cassandra $ mkdir cassandra-data 20

21 Install Cassandra (Cont) Create saved_caches, data, and commitlog directories $ cd cassandra-data $ mkdir data $ mkdir saved_caches $ mkdir commitlog 21

22 Install Cassandra (Cont) Create saved_caches, data, and commitlog directories $ cd ~/cassandra/conf Set the following values in the cassandra.yaml initial_token: 0 data_file_directories: - ~/cassandra/cassandra-data/data commitlog_directory: ~/cassandra/cassandra-data/commitlog saved_caches_directory: ~/cassandra/cassandra-data/saved_caches 22

23 Install Cassandra (Cont) Change the default logging directory in the log4j- server.properties log4j.appender.r.file= ~/cassandra/cassandra-data/system.log 23

24 Install Cassandra (Cont) Start Cassandra in the foreground $ cd ~/cassandra $ bin/cassandra -f 24

25 If You Couldn t Keep Up Or you just entered the room... You can just run as root username: notroot password: notroot 25

26 Data Fundamentals 26

27 Approaches Data Modeling in Cassandra Logical DataModels are the SAME Create Logical Models for Understanding Physical data models have different goals Physical Models for Storage and Retrieval Mostly for Retrieval 27

28 Storage Concerns Disk space is not scarce IO is scarce 28

29 Data Modeling Goals Reduce IO Fewer Round Trips 29

30 The More You Know Cassandra has a DSL called CQL It s similar to SQL It s not SQL... It s not SQL 30

31 CQL Samples Create Schema $ cqlsh cqlsh> CREATE KEYSPACE datastax... WITH strategy_class = 'SimpleStrategy'... AND strategy_options:replication_factor = 1; use datastax; cqlsh:datastax> CREATE COLUMNFAMILY users... (id text PRIMARY KEY,... fname text,... lname text,... age int); 31

32 CQL Sample: Insert Data cqlsh:datastax> INSERT INTO users... (id, fname, lname, age)... VALUES ( 'bob', 'Robert', 'Done', 33 ); cqlsh:datastax> INSERT INTO users... (id, fname, lname, age)... VALUES ( 'alice', 'Allison', 'Smith', 24 ); cqlsh:datastax> INSERT INTO users... (id, fname, lname, age)... VALUES ( 'chuck', 'Charles', 'Smith', 22 ); 32

33 Simple Range Queries cqlsh:datastax> SELECT *... FROM users; id age fname lname chuck 22 Charles Smith bob 33 Robert Done alice 24 Allison Smith 33

34 Inserts are Upserts cqlsh:datastax> INSERT INTO users... (id, fname, lname, age)... VALUES ( 'chuck', 'Charlie', 'Smithers', 50 ); cqlsh:datastax> select * from users; id age fname lname chuck 50 Charlie Smithers bob 33 Robert Done alice 24 Allison Smith 34

35 Primary Key Violation? Would require a read before a write. UUIDs are helpful v1 (TimeUUID) v3 (Faster) 35

36 Schema is loose cqlsh:datastax> INSERT INTO users... (id, address)... VALUES ( 'charlie', 123 Apple Street ); 36

37 Indexing Primary Key Secondary Index Custom Index 37

38 Primary Key We find row data by key Cassandra is a Key Value Store Rows have columns 38

39 Primary Key Query cqlsh:datastax> SELECT *... FROM users... WHERE ID = 'chuck'; id age fname lname chuck 50 Charlie Smithers 39

40 Querying on a Column cqlsh:datastax> SELECT *... FROM users... where fname = 'Charlie'; Bad Request: No indexed columns present in by-columns clause with Equal operator 40

41 Secondary Indexes cqlsh:datastax> CREATE INDEX ON users (fname); cqlsh:datastax> SELECT *... FROM users... WHERE fname = 'Charlie'; id age fname lname chuck 50 Charlie Smithers 41

42 Secondary Indexes (cont.) They are nice. But they have a cost. Require a read before a write. Must use the equals clause. Results should be low cardinality. 42

43 Foreign Keys No Concept of Foreign Keys in Cassandra All relationships are managed by you No Triggers (yet) You can have Foreign Keys It s up to you to keep the data and relationship current You may need to write to many places on an update. 43

44 Data Modeling 44

45 Data Modeling Your relationships are in your data An RDBMS can model some of those well Cassandra can model others 45

46 Schema Evolution in Cassandra Structure Follows Closely Bigtable Grouping is related to columns Early Versions Schemaless 46

47 Why Have Schema? Share Information Describe the Data Validation 47

48 Schema in Cassandra Introduced in 0.7 Consists of DataTypes You can ignore it But you shouldn t cqlsh:datastax> CREATE COLUMNFAMILY users... (id text PRIMARY KEY,... fname text,... lname text); ALTER TABLE users ADD age INT; 48

49 Looks like my RDBMS (again) No pre-allocation for rows that may be added No wasted space No limit on what can be added in the future No forced Data Migration (UPDATE TABLE) Can have thousands of columns 49

50 Composite Primary Keys Consider the CassandraFS blocks -- May be on different partitions subblocks -- kept on contiguous physical blocks Enforced by the data model CREATE TABLE sblocks ( block_id uuid, subblock_id uuid, data blob, PRIMARY KEY (block_id, subblock_id) ) Partition Key WITH COMPACT STORAGE; Column Name (Clustered) 50

51 Clustering Accessing a Row requires Seeking Bad, but unavoidable But getting a range of contiguous columns is fast After the seek Goal: We want to minimize seeking If I can pack the same information into a single row Rather than multiple rows Then I can save on seeks Wide rows are good! 51

52 Scenario: Real Time Feed Capture the data Non real-time feed here (if we need it): ( usagov_bitly_data ) 52

53 Data Definition? { "a": USER_AGENT, "c": COUNTRY_CODE, # 2-character iso code "nk": KNOWN_USER, # 1 or 0. "g": GLOBAL_BITLY_HASH, "h": ENCODING_USER_BITLY_HASH, "l": ENCODING_USER_LOGIN, "hh": SHORT_URL_CNAME, "r": REFERRING_URL, "u": LONG_URL, "t": TIMESTAMP, "gr": GEO_REGION, "ll": [LATITUDE, LONGITUDE], "cy": GEO_CITY_NAME, "tz": TIMEZONE # "hc": TIMESTAMP OF TIME HASH WAS CREATED, "al": ACCEPT_LANGUAGE } 53

54 Data { "a": "Mozilla\/5.0 (Windows NT 5.1)...", "c": "HK", "nk": 0, "tz": "Asia\/Hong_Kong", "gr": "00", "g": "NEQ8H9", "h": "P2GwTT", "l": "nasatwitter", "al": "zh-tw,zh;q=0.8,en-us;q=0.6,en;q=0.4", "hh": "go.nasa.gov", "r": " "u": " "t": , "hc": , "cy": "Central District", "ll": [ , ] } 54

55 What do we want to do? Step 1: Capture the data Step 2: Store It? Step 3:??? Step 4: Profit! 55

56 Step 3 Daily Aggregates for each URL? Daily Rankings? Just a few examples... 56

57 Daily URL Aggregation (Model) $ cqlsh -3 Version 3! cqlsh> USE datastax; cqlsh:datastax> CREATE TABLE clicks_for_hash ( global_hash text, "timestamp" double, country_code text, long_url text, PRIMARY KEY (global_hash, "timestamp") ); 57

58 Test the Data Model cqlsh:datastax> insert into clicks_for_hash... (global_hash, "timestamp", country_code, long_url)... VALUES ( 'badfood', 00001, 'US', ' badfood'); cqlsh:datastax> insert into clicks_for_hash... (global_hash, "timestamp", country_code, long_url)... VALUES ( 'foobar', 00002, 'US', ' foobar'); cqlsh:datastax> insert into clicks_for_hash... (global_hash, "timestamp", country_code, long_url)... VALUES ( 'cafebabe', 00001, 'US', ' cafebabe'); 58

59 Get Counts cqlsh:datastax> select COUNT(*)... from clicks_for_hash... where global_hash = 'cafebabe'; count cqlsh:datastax> select COUNT(*)... from clicks_for_hash... where global_hash = 'badfood'; count

60 Data Questions When were the link hashes created? How many were created today (range)? Store this data in a new column family. Sort columns by timestamp. Query By CQL 60

61 Data Model (Created?) cqlsh:datastax> CREATE COLUMNFAMILY sort_created... (key text,... created_time int,... link_hash text,... url text,... PRIMARY KEY (key, created_time)); 61

62 Simple Code $ ipython In [1]: import urllib In [2]: import cql In [3]: import json In [4]: connection = cql.connect('localhost', 9160, 'datastax', cql_version='3.0.0') In [5]: clicks = urllib.urlopen(' In [6]: clicks.readline() In [7]: cursor = connection.cursor() In [8]: while 1: obj = json.loads(clicks.readline()) if obj.has_key('h'): cursor.execute(''' INSERT INTO sort_created ("key", created_time, link_hash, url) VALUES ('all', %s, '%s', '%s')''' % (obj['hc'], obj['h'], obj['u'])) 62

63 All The Links Sorted By Created Time Could create more rows.. By Region... By Short Url CName Etc.. All sorted the same way 63

64 Some Queries cqlsh:datastax> SELECT *... FROM sort_created... WHERE created_time > 0... LIMIT 10; cqlsh:datastax> SELECT *... FROM sort_created... WHERE key = 'all'... AND created_time > LIMIT 10; cqlsh:datastax> SELECT *... FROM sort_created... WHERE key = 'all'... AND created_time > ORDER BY created_time DESC... LIMIT 10; 64

65 Overview This is just a taste Modeling OLTP Use Cases DataStax Enterprise provides integration with: Apache Hadoop (Batch Analytics) Solr (Real Time Search) Mahout (Machine Learning) 65

66 Want more Info? Drop by our Booth! 66