The NoSQL Landscape. Frank Weigel VP, Field Technical Opera;ons

Transcription

1

2 The NoSQL Landscape Frank Weigel VP, Field Technical Opera;ons

3 What we ll talk about Why RDBMS are not enough? What are the different NoSQL taxonomies? Which NoSQL is right for me?

4 Macro Trends Driving NoSQL Technology More Data More Users Interac;ve Apps + + NoSQL

5 Draw Something Goes Viral 3 Weeks ATer Launch Draw Something by OMGPOP Daily Ac)ve Users (millions) Feb 2012 March 2012

6 Does it work with RDMBS backend? Application Scales Out Just add more commodity web servers Database Scales Up Get a bigger, more complex server Note Rela3onal database technology is great for what it is great for, but it is not great for this.

7 Some alterna;ves to scale out your RDBMS Scale out your RDBMS Run many SQL Servers Data are sharded (most of the *me using client code) Memcached for faster response ;me

8 Scale out with RDBMS Is this a good approach to scale? Lot of components to deploy Scale by Hand Caching Sharding/Replica3on Learn From Others This Scenario Costs Time and Money. Scaling SQL is poten3ally disastrous when going Viral: Very risky 3me for major code changes and migra3ons... You have no Time when skyrocke3ng up!

9 Lacking Solu;ons, Users Forced to Invent Bigtable November 2006 Dynamo October 2007 Cassandra August 2008 Voldemort February 2009 Very few organiza;ons can build and maintain database sotware technology. But every organiza;on building interac;ve web applica;ons needs this technology.

10 NoSQL database matches applica;on logic ;er architecture Data layer now scales with linear cost and constant performance Application Scales Out Just add more commodity web servers NoSQL Database Servers Database Scales Out Just add more commodity data servers Scaling out flattens the cost and performance curves.

11 What Is Biggest Data Management Problem Driving Use of NoSQL in Coming Year? 49 % 35 % 29 % 16 % 12 % 11 % Lack of flexibility/ rigid schemas Inability to scale out data Performance challenges Cost All of these Other Source: Couchbase Survey, December 2011, n = 1351.

12 NoSQL Taxonomy

13 The Key- Value Store the founda;on of NoSQL Key Opaque Binary Value

14 Memcached the NoSQL precursor Key Opaque Binary Value Memcached In- memory only Limited set of opera3ons Blob Storage: Set, Add, Replace, CAS Retrieval: Get Structured Data: Append, Increment Simple and fast. Challenges: cold cache, disrup3ve elas3city

15 NoSQL catalog Key- Value Cache (memory only) Memcached Database (memory/disk)

16 Redis More Structured Data commands Key Data Structures Blob List Set Hash Redis Disk Persistence (eventual consistency on the disk) Vast set of opera3ons Blob Storage: Set, Add, Replace, CAS Retrieval: Get, Pub- Sub Structured Data: Strings, Hashes, Lists, Sets, Sorted lists Example opera3ons for a Set Add, count, subtract sets, intersec3on, is member, atomic move from one set to another

17 NoSQL catalog Key- Value Data Structure Cache (memory only) Memcached Redis Database (memory/disk)

18 Membase From key- value cache to database Key Opaque Binary Value Membase Disk- based with built- in memcached cache Cache refill on restart Memcached compa3ble (drop in replacement) Highly- available (data replica3on) Add or remove capacity to live cluster Simple, fast, elas3c.

19 NoSQL catalog Key- Value Data Structure Cache (memory only) Memcached Redis Database (memory/disk) Membase

20 Couchbase document- oriented database Key { } string : string, string : value, string : { string JSON : string, string : value }, OBJECT string : [ array ] ( DOCUMENT ) Couchbase Auto- sharding Disk- based with built- in memcached cache Cache refill on restart Memcached compa3ble (drop in replace) Highly- available (data replica3on) Add or remove capacity to live cluster When values are JSON objects ( documents ): Create indices, views and query against the views

21 NoSQL catalog Key- Value Data Structure Document Cache (memory only) Memcached Redis Database (memory/disk) Membase Couchbase

22 MongoDB Document- oriented database Key { } string : string, string : value, string : { string BSON : string, string OBJECT : value }, string : [ array ] ( DOCUMENT ) MongoDB Disk- based with in- memory caching BSON ( binary JSON ) format and wire protocol Master- slave replica3on Auto- sharding Values are BSON objects Supports ad hoc queries best when indexed

23 NoSQL catalog Key- Value Data Structure Document Cache (memory only) Memcached Redis Database (memory/disk) Membase Couchbase MongoDB

24 Cassandra Column overlays Column 1 Column 2 Column 3 (not present) Key Opaque Binary Value Cassandra Disk- based system Clustered External caching required for low- latency reads Columns are overlaid on the data Not all rows must have all columns Supports efficient queries on columns Restart required when adding columns

25 NoSQL catalog Key- Value Data Structure Document Column Cache (memory only) Memcached Redis Database (memory/disk) Membase Couchbase Cassandra MongoDB

26 Neo4j Graph database Key Opaque Binary Value Neo4j Key Opaque Binary Value Key Opaque Binary Value Disk- based system External caching required for low- latency reads Nodes, rela3onships and paths Proper3es on nodes Delete, Insert, Traverse, etc. Key Opaque Binary Value Key Opaque Binary Value

27 NoSQL catalog Key- Value Data Structure Document Column Graph Cache (memory only) Memcached Redis Database (memory/disk) Membase Couchbase Cassandra Neo4j MongoDB

28 NoSQL catalog Key- Value Data Structure Document Column Graph Cache (memory only) Memcached Redis Coherence Database (memory/disk) Membase Couchbase Cassandra Neo4j MongoDB HBase InfiniteGraph

29 Speed and Scale

30 What about Hadoop?

31 Hadoop: Big Data Swiss Army Knife Oozie: Workflow, coordina;on Sqoop : Data connector to import/export data Hive : SQL- Like interface Pig : High level programming language Mahout : Machine learning library Whirr : Hadoop management tools for cloud services Flume : Aggregator Map Reduce : Framework to process large volume of data HBase : Key Value data store Zookeeper : Centralized configura;on management HDFS : Distributed file system

32 So what? Hadoop & Couchbase 40 milliseconds to respond with the decision. 3 profiles, real 3me campaign sta3s3cs 1 click stream events 2 profiles, campaigns

33 Which one is right for me?

34 Survey: Schema inflexibility #1 adop;on driver What is the biggest data management problem driving your use of NoSQL in the coming year? Lack of flexibility/rigid schemas 49% Inability to scale out data 35% High latency/low performance 29% Costs 16% All of these 12% Other 11% Source: Couchbase NoSQL Survey, December 2011, n=1351

35 Lack of Flexibility / Rigid Schema Aggregate Data Models (Mar*n Fowler) Flexible Data Structure Op3mized Access Easy to distribute data o::1001 hqp://mar3nfowler.com/bliki/aggregateorienteddatabase.html { uid: ji22jd, customer: Ann, line_items: [ { sku: , quan: 3, unit_price: 48.0 }, { sku: , quan: 1, unit_price: 39.0 }, { sku: , quan: 1, unit_price: 51.0 } ], payment: { type: Amex, expiry: 04/2001, last5: } }

36 Use Cases Key Value Document Columns Graph Session Management User Profile/Preferences Shopping Cart Event Logging Content Management Web Analy;cs E- Commerce Applica;on Event Logging Content Management Counters Connected Data / Social Networks Rou;ng, Dispatch Recommenda;ons based on Social Graph

37 Scale out your data Modify cluster topology should be simple Add, Remove, Configure Nodes on a running system What is the impact of topology changes? Sharding, Caching of the data Availability of the service during cluster changes More hardware = More failures Availability, reliability of the system: failover support

38 Produc;on Environment EMEA DC US DATA CENTER APAC DC

39 Add Nodes to Cluster APP SERVER 1 COUCHBASE Client Library CLUSTER MAP APP SERVER 2 COUCHBASE Client Library CLUSTER MAP READ/WRITE/UPDATE READ/WRITE/UPDATE SERVER 1 ACTIVE SERVER 2 ACTIVE SERVER 3 SERVER 4 SERVER 5 ACTIVE ACTIVE ACTIVE Two servers added One- click opera;on Doc 5 Doc Doc 2 Doc Doc 9 Doc REPLICA Doc 4 Doc Doc 1 Doc Doc 4 Doc Doc 7 Doc Doc 8 Doc REPLICA Doc 6 Doc Doc 3 Doc Doc 1 Doc Doc 2 Doc Doc 6 Doc REPLICA Doc 7 Doc Doc 9 Doc REPLICA REPLICA Docs automa;cally rebalanced across cluster Even distribu3on of docs Minimum doc movement Cluster map updated App database calls now distributed over larger number of servers Doc 8 Doc Doc 2 Doc Doc 5 Doc COUCHBASE SERVER CLUSTER User Configured Replica Count = 1

40 Fail Over Node APP SERVER 1 COUCHBASE Client Library CLUSTER MAP APP SERVER 2 COUCHBASE Client Library CLUSTER MAP SERVER 1 SERVER 2 ACTIVE ACTIVE Doc 5 Doc 2 Doc Doc Doc 4 Doc 7 Doc Doc Doc 1 Doc 3 REPLICA REPLICA Doc 4 Doc 1 Doc Doc Doc 6 Doc 3 Doc Doc SERVER 3 SERVER 4 SERVER 5 ACTIVE ACTIVE ACTIVE Doc 1 Doc 2 Doc Doc Doc 9 Doc 8 Doc Doc Doc 6 Doc Doc REPLICA REPLICA REPLICA Doc 7 Doc Doc 5 Doc Doc 8 Doc Doc 9 Doc Doc 2 Doc App servers accessing docs Requests to Server 3 fail Cluster detects server failed Promotes replicas of docs to ac3ve Updates cluster map Requests for docs now go to appropriate server Typically rebalance would follow COUCHBASE SERVER CLUSTER User Configured Replica Count = 1

41 Performance What is my working set? How cache is working? Put your data in RAM How to design my data model? Aggregate Model Easy to change

42 18 Read performance comparison - NoSQL databases Read latencies against throughput Cassandra 95th Percen;le Latency (ms) MongoDB cannot handle throughput above ~ 8000 ops / sec MongoDB Couchbase handles ~3X throughput with significantly lower latency Couchbase hqps://github.com/altoros/ycsb Opera;ons per Second

43 Write performance comparison - NoSQL databases 30 Insert/update latencies against throughput 95th Percen;le Latency (ms) MongoDB MongoDB latency shoots up beyond 6000 ops / sec Cassandra Couchbase latency stays consistently low even at ops / sec Couchbase Opera;ons per Second

44 Management and Monitoring Do not forget about Opera;ons! Service Reliability Engineering Team will thank you! Manage your cluster easily: Command Line, Administra3on Console to change cluster toplogy Monitor your NoSQL Analyze the overall status of your cluster View and fix boqlenecks

45

46 Conclusion One Size Does Not Fit All Overview of the the NoSQL types Choose the right solu;on Developer Produc;vity Large Scale Data

47 Q&A

48 Thanks!

49