The NoSQL Ecosystem. Adam Marcus MIT CSAIL

Transcription

1 The NoSQL Ecosystem Adam Marcus MIT CSAIL

2 About Me Social Computing + Database Systems Easily Distracted: Wrote The NoSQL Ecosystem in The Architecture of Open Source Applications 1 1

3 History, Compressed Late 1990s Today

4 History, Compressed Buy RAM Late 1990s Today

5 History, Compressed Buy RAM Wrap RDBMSs Late 1990s Today

6 History, Compressed Buy RAM Wrap RDBMSs Late 1990s NoSQL Today

7 History, Compressed Not Only SQL Buy RAM Wrap RDBMSs Late 1990s NoSQL Today

8 History, Compressed Buy RAM Wrap RDBMSs Late 1990s NoSQL Commercialize Today

9 History, Compressed Buy RAM Wrap RDBMSs Late 1990s NoSQL Commercialize Today

10 The List, So You Don't Yell at Me HBase Riak Cassandra BerkeleyDB Voldemort Redis VertexDB InfoGrid HyperTable MongoDB AllegroGraph HyperGraphDB Tokyo Cabinet Neo4j CouchDB Sones DEX MemcacheDB FlockDB Oracle NoSQL

11 The List, So You Don't Yell at Me Marcus' Law of Databases: CouchDB HBase Riak Cassandra Neo4j InfoGrid The number of persistence HyperTable BerkeleyDB Voldemort options doubles every 1.5 years Redis MongoDB AllegroGraph HyperGraphDB VertexDB Tokyo Cabinet Sones DEX MemcacheDB FlockDB Oracle NoSQL

12 The List, So You Don't Yell at Me Marcus' Law of Databases: CouchDB HBase Cassandra Riak Neo4j InfoGrid The number of persistence HyperTable BerkeleyDB Voldemort options doubles every 1.5 years Redis MongoDB Sones Corollary: AllegroGraph HyperGraphDB I'm sorry I misseddex your FlockDB VertexDB persistence optionoracle NoSQL Tokyo Cabinet MemcacheDB

13 interesting properties real-world usage takeaways questions

14 interesting properties real-world usage takeaways questions

15 Conventional Wisdom on NoSQL Key-based data model Sloppy schema Single-key transactions In-app joins Eventually consistent

16 Exceptions are More Interesting Data Model Query Model Transactions Consistency

17 Data Model Usually key-based... Column-families Documents Data structures

18 Data Model Usually key-based... Column-families Documents Data structures...but not always Graph stores

19 Query Model Redis: data structure-specific operations CouchDB, Riak: MapReduce Cassandra, MongoDB: SQL-like languages, no joins or transactions

20 Query Model Redis: data structure-specific operations CouchDB, Riak: MapReduce Cassandra, MongoDB: SQL-like languages, no joins or transactions Third-party High-level: PigLatin, HiveQL Library: Cascading, Crunch Streaming: Flume, Kafka, S4, Scribe

21 Transactions Full ACID for single key Redis: multi-key single-node transactions

22 Consistency Strong: Appears that all replicas see all writes Eventual: Replicas may have different, divergent versions

23 Consistency Strong: Appears that all replicas see all writes Eventual: Replicas may have different, divergent versions Dynamo: strong or eventual (quorum size)

24 Consistency Strong: Appears that all replicas see all writes Eventual: Replicas may have different, divergent versions Dynamo: strong or eventual (quorum size) PNUTs: Timeline consistency...many consistency models! See:

25 interesting properties real-world usage takeaways questions

26 NoSQL Use-cases Cassandra HBase MongoDB

27 Cassandra BigTable data model: key column family Dynamo sharding model: consistent hashing Eventual or strong consistency

28 Cassandra at Netflix Transitioned from Oracle Store customer profiles, customer:movie watch log, and detailed usage logging Note: no multi-record locking (e.g., bank transfer) In-datacenter: 3 replicas, per-app consistency read/write quorum = 1, ~1ms latency read/write quorum = 2, ~3ms latency Replicating Datacenter Oracle with Global Apache Cassandra on AWS by Adrian Cockcroft

29 Cassandra at Netflix Transitioned from Oracle Store customer profiles, customer:movie watch log, and detailed usage logging Note: no multi-record locking (e.g., bank transfer) In-datacenter: 3 replicas, per-app consistency read/write quorum = 1, ~1ms latency read/write quorum = 2, ~3ms latency Replicating Datacenter Oracle with Global Apache Cassandra on AWS by Adrian Cockcroft

30 Cassandra at Netflix Transitioned from Oracle Store customer profiles, customer:movie watch log, and detailed usage logging Note: no multi-record locking (e.g., bank transfer) In-datacenter: 3 replicas, per-app consistency Replicating Datacenter Oracle with Global Apache Cassandra on AWS by Adrian Cockcroft

31 Cassandra at Netflix (cont'd) Benefit: async inter-datacenter replication Benefit: no downtime for schema changes Benefit: hooks for live backups

32 HBase Data model: key column family Sharding model: range partitioning Strong consistency Applications Logging events/crawls, storing analytics Twitter: replicate data from MySQL, Hadoop analytics Facebook Messages

33 HBase Data model: key column family Sharding model: range partitioning Strong consistency Applications Logging events/crawls, storing analytics Twitter: replicate data from MySQL, Hadoop analytics Facebook Messages

34 HBase for Facebook Messages Cassandra/Dynamo eventual consistency was difficult to program against Benefit: simple consistency model Benefit: flexible data model Benefit: simple sharding, load balancing, replication

35 HBase for Facebook Messages Cassandra/Dynamo eventual consistency was difficult to program against Benefit: simple consistency model Benefit: flexible data model Benefit: simple sharding, load balancing, replication

36 MongoDB Document-based data model Range-based partitioning Consistency depends on how you use it

37 MongoDB: Two use-cases Archiving at Craigslist 2.2B historical posts, semi-structured Relatively large blobs: avg 2KB, max > 4 MB

38 MongoDB: Two use-cases Archiving at Craigslist 2.2B historical posts, semi-structured Relatively large blobs: avg 2KB, max > 4 MB Checkins at Foursquare Geospatial indexing Small location-based updates, sharded on user

39 interesting properties real-world usage takeaways questions

40 Takeaways Developer accessibility Ecosystem of reuse Soft spot

41 Developer Accessibility

42 Developer Accessibility

43 Developer Accessibility self-taught dev

44 A Different Five-Minute Rule What does a first-time user of your system experience in their first five minutes? (idea credit: Justin Sheehy, Basho)

45 Redis

46 PostgreSQL

47 Cassandra

48 Accessibility is Forever Beyond five minutes: changing schemas scaling up modifying topology

49 Accessibility is Forever Beyond five minutes: changing schemas scaling up modifying topology An accessible data store will ease first-time users in, and support experienced users as they expand

50 Ecosystem of Reuse

51 Spectrum of Reusability Monolithic System System Kernel

52 Spectrum of Reusability Monolithic System MongoDB Redis Use as provided System Kernel

53 Spectrum of Reusability Monolithic System MongoDB Redis Cassandra Voldemort Use as provided Pick between components System Kernel

54 Spectrum of Reusability Monolithic System MongoDB Redis Use as provided Cassandra Voldemort System Kernel ZooKeeper LevelDB Pick between Reusable components components

55 Spectrum of Reusability Monolithic System MongoDB Redis Use as provided Cassandra Voldemort System Kernel ZooKeeper LevelDB Pick between Reusable components components riak_core Build your own system

56 And finally, a soft spot

57 polyglot persistence = right tool for right task

58 Polyglot persistence: Where's the data? HBase at Mendeley, Dan Harvey

59 Polyglot persistence: Where's the data? HBase at Mendeley, Dan Harvey

60 Polyglot persistence: Where's the data? Aid developers in reasoning about data consistency across multiple storage engines HBase at Mendeley, Dan Harvey

61 interesting properties real-world usage takeaways questions

62 Systems: Polyglot persistence + data consistency?

63 Systems: Polyglot persistence + data consistency? Operations: Availability/consistency/latency tradeoffs in datacenters?

64 Systems: Polyglot persistence + data consistency? Operations: Availability/consistency/latency tradeoffs in datacenters? Accessibility: RDBMS five-minute usability?

65 Systems: Polyglot persistence + data consistency? Operations: Availability/consistency/latency tradeoffs in datacenters? Accessibility: RDBMS five-minute usability? Accessibility: Scale-up wizard for storage systems?

66 Systems: Polyglot persistence + data consistency? Operations: Availability/consistency/latency tradeoffs in datacenters? Accessibility: RDBMS five-minute usability? Accessibility: Scale-up wizard for storage systems? Comparisons: Design or implementation?

67 Systems: Polyglot persistence + data consistency? Operations: Availability/consistency/latency tradeoffs in datacenters? Accessibility: RDBMS five-minute usability? Accessibility: Scale-up wizard for storage systems? Comparisons: Design or implementation? Future: Next-generation NoSQL stores?

68 Thank You! Systems: Polyglot persistence + data consistency? Operations: Availability/consistency/latency tradeoffs in datacenters? Accessibility: RDBMS five-minute usability? Accessibility: Scale-up wizard for storage systems? Comparisons: Design or implementation? Future: Next-generation NoSQL stores? Adam Marcus marcua@csail.mit.edu

69 Photo Credits / /sizes/m/in/photostream/ ne-heart/