Developing in NoSQL with Couchbase

Size: px

Start display at page:

Download "Developing in NoSQL with Couchbase"

Gwen Goodman
6 years ago
Views:

1 Developing in NoSQL with Couchbase Raghavan Rags Srinivas Developer Advocate Simple. Fast. Elastic.

2 Speaker Introduction Architect and Evangelist working with developers Speaker at JavaOne, RSA conferences, Sun Tech Days, JUGs and other developer conferences Taught undergrad and grad courses Technology Evangelist at Sun Microsystems for 10+ years Still trying to understand and work more effectively on Java and distributed systems Couchbase Developer Advocate working with Java and Ruby developers Philosophy: Better to have an unanswered question than a unquestioned answer

3 Agenda > Introduction and Getting Started with the APIs > Demonstration: Topology changes and xdcr > Hadoop Sqoop connector and use cases > Secondary Indexing with Query and Views > Roadmap and conclusions

4 A BIT OF NOSQL Intro. Demos. Hadoop Connector Views Summary

5 NoSQL (from a Couchbase perspective) Short for Not Only SQL Not single class of database (i.e., NoSQL is not analogous to the RDMBS classification) Generally refers to databases that neither expose a SQL interface nor use the relational data model > Perhaps better thought of as non-relational databases Typically don t offer common RDBMS features for performance gains > Often schema-less > No key constraints > No multi-step transactions > No concept of a join Well suited for cloud deployments

CAP Theorem/Brewer s Conjecture The CAP theorem states that it is impossible for a distributed computer system to simultaneously provide all three of the following guarantees: Consistency means that

6 CAP Theorem/Brewer s Conjecture The CAP theorem states that it is impossible for a distributed computer system to simultaneously provide all three of the following guarantees: Consistency means that each node/client always has the same view of the data, Availability means that all clients can always read and write, Partition tolerance means that the system works well across physical network partitions.

7 CAP Systems CA: Consistency and Availability > Consider a system that uses a two phase commit for distributed transactions > The commit is impeded if a network segment is unavailable CP: Consistency and Partition Tolerance > Consider a system that uses pessimistic locking for concurrency control > Node failures hinder availability AP: Availability and Partition Tolerance > Consider a system that always responds (e.g., DNS) > Reads might be dirty as data awaits propagation

8 NoSQL Taxonomies NoSQL Databases Come in Many Flavors* > XML (myxmldb, Tamino, Sedna) > Tabular (Hbase, Big Table) > Key/Value (Couchbase, Redis, Riak, Cassandra) > Object (db4o, JADE) > Graph (Trinity, neo4j, InfoGrid) > Document store (Couchbase Server 2.0, CouchDB, MongoDB) * These are loose taxonomies

Web Application Architecture and Performance www.

com/animalparty Application Scales Out Just add

Balancer RelaConal Database Database Scales Up

9 Web Application Architecture and Performance Application Scales Out Just add more commodity web servers Web Servers Load Balancer RelaConal Database Database Scales Up Get a bigger, more complex server - Expensive and disrupcve sharding - Doesn t perform at Web Scale

Couchbase data layer scales like application logic tier

commodity web servers Web Servers Load Balancer

commodity data servers Horizontally scalable, schema-

10 Couchbase data layer scales like application logic tier Data layer now scales with linear cost and constant performance. Application Scales Out Just add more commodity web servers Web Servers Load Balancer Couchbase Servers Database Scales Out Just add more commodity data servers Horizontally scalable, schema- less, auto- sharding, high- performance at Web Scale Scaling out flattens the cost and performance curves.

11 WHAT IS COUCHBASE? Intro. Demos. Hadoop Connector Views Summary

12 Couchbase Server Overview Open source, Apache 2.0 licensed Database Commercial support provided by Couchbase Free community edition for developers Enterprise edition free for 2 nodes, support license required for larger clusters Administered via web console or RESTful API Available for Windows, Linux and Mac (development only) Written in C/C++ with some components in Erlang Both a distributed key/value store and a document oriented database > Implements Memcached API (version 1.8) > Implements most of CouchDB RESTful view API (version 2.0)

13 memcached Couchbase Membase CouchDB

14 Rapidly Growing Community 1,000s of community deployments More than 150 paying customers Over 3,500 nodes deployed in production by customers

15 Customers with Production Deployments

16 OMGPOP s DrawSomething Went Viral Pictionary Like Game #1 Paid & Free App in 84 Countries 35m Downloads

Couchbase Server 1.8 Architecture 11211 Memcapable 1.0 11210 Memcapable 2.

management API/Web UI Heartbeat Process monitor ConfiguraCon manager Global

and replicacon manager Cluster Manager Persistence Layer hop on each node one

17 Couchbase Server 1.8 Architecture Memcapable Memcapable 2.0 Moxi Memcached Couchbase EP Engine Data Manager storage interface REST management API/Web UI Heartbeat Process monitor ConfiguraCon manager Global singleton supervisor Rebalance orchestrator Node health monitor vbucket state and replicacon manager Cluster Manager Persistence Layer hop on each node one per cluster Erlang/OTP HTTP 8091 Erlang port mapper 4369 Distributed Erlang

18 Component Architecture Memcapable Memcapable 2.0 Moxi Memcached Couchbase EP Engine storage interface REST Management / WebUI Heartbeat Process monitor Configuration Manager Global singleton supervisor Rebalance orchestrator Node health monitor vbucket state and replicacon manager Persistence Layer htt p on each node one per cluster Erlang/OTP HTTP 8091 Erlang port mapper 4369 Distributed Erlang

19 A Distributed Hash Table Document Store Application set(key, json) get(key) returns json Database Cluster {! DData id": brewery_legacy,! type : brewery,! "name" : "Legacy Brewing,! address": "525 Canal Street Reading", "updated": " :00:20", }!

20 Couchbase Distributed Key/Value Store Features All documents/items are stored and retrieved using standard hashtable operations (i.e, get, add, etc.) Keys are distributed evenly across the nodes of a cluster by way of vbuckets > A vbucket is a level of indirection between a key and value, and the node on which it is stored Constant time access to items via keys Single dimension hashtable with no secondary keys

21 Couchbase Document Store Features Items can be stored as JSON documents Non-JSON items are stored as binary attachments to CouchDB JSON documents Secondary, B-tree indexes are created by way of views > Views are accessed via RESTful API > Indexes are incrementally updated after access not insert > Caller specifies whether to accept stale data Views may be queried by key or range Indexes are stored across the cluster and accumulated from all nodes when requested

22 Workflow (including replication) User action results in the need to change the VALUE of KEY 1 4 SET request sent over network to master server 2 3 Application updates key s VALUE, performs SET operation Couchbase driver hashes KEY, identifies KEY s master server 5 Couchbase replicates KEY-VALUE pair, caches it in memory and stores it to disk

23 GETTING STARTED Intro. Demos. Hadoop Connector Views Summary

24 Installation of the Server for development Operating systems supported RedHat RPM Ubuntu dpkg Windows Mac OS GUI Installer Unattended installation Quickstart once installed (invokes the Admin Console)

25 COUCHBASE CLIENT INTERFACE Intro. Demos. Hadoop Connector Views Summary

26 Client Setup: Getting Cluster Configuration Couchbase Client {. Cluster Configuration over REST "bucketcapabilities": [ "touch", "sync", "couchapi" ], "bucketcapabilitiesver": "sync-1.0", "buckettype": couchbase", "name": "default", "nodelocator": "vbucket", "nodes": [ Server Node Server Node Server Node Server Node

27 Client at Runtime: Adding a node Couchbase Client New node Cluster Topology Update coming online Server Node Server Node Server Node Server Node Server Node

28 Opening a Connection (Java) Connect to the Cluster URI of any cluster node List<URI> uris = new LinkedList<URI>(); uris.add(uri.create(" try { client = new CouchbaseClient(uris, "default", ""); } catch (Exception e) { System.err.println("Error connecting to Couchbase: " + e.getmessage()); System.exit(0); }

29 PROTOCOL OVERVIEW Intro. Demos Hadoop Connector Views Summary

30 Store Operations Operation Description add() Adds new value if key does not exist or returns error. replace() Replaces existing value if specified key already exists. set() Sets new value for specified key. TTL allow for expiry time specified in seconds: Expiry Value Values < 30*24*60*60 Values > 30*24*60*60 Description Time in seconds to expiry Absolute time from epoch for expiry

31 Retrieve Operations Operation Description get() Get a value. getandtouch() Get a value and update the expiry time. getbulk() Get multiple values simultaneously, more efficient. gets() Get a value and the CAS value.

32 ASYNCHRONOUS OPERATIONS Intro. Advanced opertations Doc. Design Views Summary

33 Couchbase Java Client Asynchronous Functions Synchronous commands Force wait on application until return from server Asynchronous commands Allow background operation until response available Operations return Future object

34 It s Asynchronous Set Operation is asynchronous // Do an asynchronous set! OperationFuture<Boolean> setop =! client.set(key, EXP_TIME, VALUE);!! // Do something not related to set!! // Check to see if our set succeeded! // We block when we call the get()! if (setop.get().booleanvalue()) {! System.out.println("Set Succeeded");! } else {! System.err.println("Set failed: " +! setop.getstatus().getmessage());! }!

35 Distributed System Design: Concurrency Controls Compare and Swap Operations > Often referred to as CAS > Optimistic concurrency control > Available with many mutation operations, depending on client. Success Get with Lock > Often referred to as GETL > Pessimistic concurrency control > Locks have a short TTL > Locks released with CAS operations > Useful when working with object graphs Actor 1 Actor 2 Couchbase Server A CAS mismatch B C D F E

36 CAS Operation CAS Example! CASValue<Object> casv = client.gets(key);! Thread.sleep(random.nextInt(1000)); // a random workload!! // Wake up and do a set based on the previous CAS value! Future<CASResponse> setop =! client.asynccas(key, casv.getcas(), random.nextlong());! // Wait for the CAS Response! try {! if (setop.get().equals(casresponse.ok)) {! System.out.println("Set Succeeded");! } else {! System.err.println("Set failed: ");! }! }!!

37 CONCURRENCY Intro. Demos Hadoop Connector Views Summary

38 KEYS AND RELATIONSHIPS Intro. Demos Hadoop Connector Views Summary

39 Utilizing the strengths of the Platform Couchbase > Distributed Key, Value pairs > De-normalized Data > Fast access Session store Hadoop > Unstructured/semi-structured data > Analytics and frequent data analysis > Leverage the Hadoop Ecosystem (Pig, Hive, Hbase, etc.)

40 Use cases Ad Targeting > Decisions have to be made in milliseconds based on historical trends, location, user preferences, etc. Virtual Reality > Sub-second response time based on data available from number of different sources Gaming applications > Responsive applications that integrate with data in the back end

41 SQOOP

42 Introducing Sqoop Easily Import/Export Data into Hadoop Generate Datatypes for use in MapReduce Applications Integrate with Pig, Hive and Hbase Easily export Data from Hadoop HDFS Hive HBase Sqoop Pig Hadoop

43 COUCHBASE PLUGIN

44 Couchbase Plugin Based on the Couchbase Tap Interface Allows importing and exporting of entire database key mutations Couchbase 1. Data imported via Tap mechanism 3. Data exported back to Couchbase HDFS 2. Hadoop Processing

45 Couchbase Import and Export $ sqoop import -connect --table DUMP $ sqoop import -connect --table BACKFILL_5 $ sqoop export --connect --table DUMP export-dir DUMP For Imports, table must be: > DUMP: All keys currently in Couchbase > BACKFILL_n: All key mutations for n minutes Specified username maps to bucket > By default set to default bucket

46 Ad Targeting Application Couchbase > System of Record > Provides fast access to users and being able to record events Hadoop > Analytics > Generates a count of the # of clicks Couchbase 1. User/Event Data is imported to HDFS HDFS 2. Event data is consolidated with Map/Reduce 3. Consolidated data Is used to target ads.

47 Importing to Hadoop

48 Counting the clicks

49 Analyze using Pig

50 Counting the ads

51 Analyze and Consolidate

52 THE VIEW AND QUERY API Intro. Demos Hadoop Connector Views Summary

53 Couchbase Server 2.0 Identical feature set to Couchbase Server 1.8 > Cache-layer > High-performance > Cluster > Elastic > Core interface Adds > Document based storage (JSON) > Views with materialized indexes > Querying

54 Why use JSON? JSON (JavaScript Object Notation) > Lightweight Data-interchange format > Easy for humans to read and manipulate > Easy for machines to parse and generate (many parsers are available) > JSON is language independent (although it uses similar constructs)

55 What Are Views Views create perspectives on a collection of documents Views can cover a few different use cases Simple secondary indexes (the most common) Aggregation functions Example: count the number of North American Ales Organizing related data Use Map/Reduce Map defines the relationship between fields in documents and output table Reduce provides method for collating/summarizing Views materialized indexes Views are not create until accessed Data writes are fast (no index) Index updates all changes since last update

56 View Processing Extract fields from JSON documents and produce an index of the selected information

57 View Creation using Incremental Map/ Reduce map function creates a mapping between input data and output reduce function provides a summary (such as count, sum, etc.)

58 Map Func:ons Map outputs one or more rows of data Map outputs: > Document ID > View Key (user configurable) > View Value View Key Controls > SorCng > Querying Key + Value stored in the Index Maps provide query base

59 Map/Reduce Map Function A map function to get breweries by province function (doc) { if (doc.type == "brewery" && doc.province) { emit(doc.province, doc.name); } }

60 Map/Reduce Map Results The conceptual output of the map function is [ {"key":"connecticut", "value":"cottrell Brewing"}, {"key":"connecticut","value":"thomas Hooker Brewing Co."}, {"key":"massachusetts","value":"harpoon Brewing Company"}, {"key":"new York","value":"Brewery Ommegang"}, {"key":"vermont","value":"long Trail Brewing Company"} ]

61 Map/Reduce Reduce Function A reduce function to count breweries by province > Or simply use the build in _count function function (keys, values) { } return values.length; The conceptual input to the reduce function is: [ {"keys":["connecticut", "Connecticut"], "values": ["Cottrell Brewing", "Thomas Hooker Brewing Co."]}, {"keys":["massachusetts"],"values":["harpoon Brewing Co."]}, {"keys":["new York"],"values":["Brewery Ommegang"]}, {"keys":["vermont"],"values":["long Trail Brewing Company ]} ]

62 Map/Reduce Reduce Results The conceptual result of the reduce is {"key":"connecticut","value":2}, {"key":"massachusetts","value":1}, {"key":"new York","value":1}, {"key":"vermont","value":1}

63 THE QUERY API Intro. Demos Hadoop Connector Views Summary

64 Query APIs // map function! function (doc) {! if (doc.type == "beer") {! emit(doc._id, null);! }! }!! // Java code! Query query = new Query();!! query.setreduce(false);! query.setincludedocs(true);! query.setstale(stale.false);!!

65 THE VIEW API Intro. Demos Hadoop Connector Views Summary

66 View APIs with Java // map function! function (doc) {! if (doc.type == "beer") {! emit(doc._id, null);! }! }!! // Java code! View view = client.getview("beers", "beers");!! ViewResponse result = client.query(view, query);!! Iterator<ViewRow> itr = result.iterator();!! while (itr.hasnext()) {! row = itr.next();! doc = (String) row.getdocument();! // do something! }!!!

67 USING THE QUERY API FOR CALCULATION Intro. Advanced opertations Doc. Design Views Summary

68 Querying with Java Custom Reduce // map function! function(doc) {! if (doc.type == "beer") {! if(doc.abv) {! emit(doc.abv, 1);! }}}!! //reduce function! _count!! // Java code! View view = client.getview("beers", "beers_count_abv");! query.setgroup(true);!! while (itr.hasnext()) {! row = itr.next();! System.out.println(String.format("%s: %s,! row.getkey(), row.getvalue()));! }!

69 View Calculation Result 10: 2! 9.6: 5! 9.1: 1! 9: 3! 8.7: 2! 8.5: 1! 8: 2! 7.5: 4! 7: 4! 6.7: 1! 6.6: 1! 6.2: 1! 6: 2! 5.9: 1! 5.6: 1! 5.2: 1! 5: 1! 4.8: 2! 4.5: 1! 4: 2!

70 RESOURCES AND SUMMARY Intro. Demos Hadoop Connector Views Summary

71 Java Client Library Roadmap Server 1.8 Compatible > 1.0.1/2.8.0 > 1.0.2/2.8.0 (Memcache Node imbalance) > 1.0.x/2.8.0 (Replica Read) Server 2.0 Compatible > 1.1-dp/2.8.1 (Views) > 1.1-dp2/2.8.x (Observe) > 1.1/2.8.x (Final) Other Features > Spring Integration > Your feedback?!

72 Resources and Call For Action Couchbase Server Downloads > > Views > Developing with Client libraries > Couchbase Java Client Library wiki tips and tricks > +Library Open Beer Database > Couchbase Internals >

73 THANKS - Q&A Raghavan Rags Srinivas, Couchbase Inc. 73

Developing in NoSQL with Couchbase and Java. Raghavan N. Srinivas Couchbase 123

Developing in NoSQL with Couchbase and Java Raghavan N. Srinivas Couchbase 123 Objective A technical overview To getting started on programming with Couchbase using Java To learn about the basic operations,