NOSQL EGCO321 DATABASE SYSTEMS KANAT POOLSAWASD DEPARTMENT OF COMPUTER ENGINEERING MAHIDOL UNIVERSITY

Transcription

1 NOSQL EGCO321 DATABASE SYSTEMS KANAT POOLSAWASD DEPARTMENT OF COMPUTER ENGINEERING MAHIDOL UNIVERSITY

2 WHAT IS NOSQL? Stands for No-SQL or Not Only SQL. Class of non-relational data storage systems E.g. MongoDB, Neo4j, Usually do not require a fixed table schema nor do they use the concept of joins Distributed data storage systems All NoSQL offerings relax one or more of the ACID properties.

3 DATABASE SCALING Relational Database is "scaled up" by adding hardware processing power NoSQL is "scaled out" by spreading the load

4 R E L AT I O N A L D ATA B A S E S C A L I N G At certain point relational database won't scale

5 NOSQL DATABASE SCALING Scaling horizontally is possible with NoSQL Scaling up / down is easy Supports rapid production-ready prototyping Better handling of traffic spikes

6 WHERE NOSQL IS USED? Google (BigTable, LevelDB) LinkedIn (Voldemort) Facebook (Cassandra) Twitter (Hadoop/Hbase, FlockDB, Cassandra) Netflix (SimpleDB, Hadoop/HBase, Cassandra) CERN (CouchDB)

7 CAP THEOREM (1) It is impossible for a distributed computer system to simultaneously provide all three of the following guarantees: Consistency (all nodes see the same data at the same time) Availability (a guarantee that every request receives a response about whether it was successful or failed) Partition tolerance (the system continues to operate despite arbitrary message loss or failure of part of the system)

8 CAP THEOREM (2) In other words, CAP can be expressed as "If the network is broken, your database won t work In Relational DBMS we do not have P (network partitions) Consistency and Availability are achieved In NoSQL we want to have P Need to select either C or A Drop A -> Accept waiting until data is consistent Drop C -> Accept getting inconsistent data sometimes

9 NOSQL SYSTEMS AND CAP

10 ACID VS BASE Scalability and better performance of NoSQL is achieved by sacrificing ACID compatibility. Atomic, Consistent, Isolated, Durable NoSQL is having BASE compatibility instead. Basically Available, Soft state, Eventual consistency

11 BASE BASICALLY AVAILABLE Use replication and sharding to reduce the likelihood of data unavailability and use sharding, or partitioning the data among many different storage servers, to make any remaining failures partial. The result is a system that is always available, even if subsets of the data become unavailable for short periods of time.

12 BASE AND AVAILABILITY The availability of BASE is achieved through supporting partial failures without total system failure. Example. If users are partitioned across five database servers, BASE design encourages crafting operations in such a way that a user database failure impacts only the 20 percent of the users on that particular host. This leads to higher perceived availability of the system. Even though a single node is failing, the interface is still operational.

13 BASE EVENTUALLY CONSISTENT Although applications must deal with instantaneous consistency, NoSQL systems ensure that at some future point in time the data assumes a consistent state. In contrast to ACID systems that enforce consistency at transaction commit, NoSQL guarantees consistency only at some undefined future time. Where ACID is pessimistic and forces consistency at the end of every operation, BASE is optimistic and accepts that the database consistency will be in a state of flux.

14 BASE AND CONSISTENCY As DB nodes are added while scaling up, need for synchronisation arises If absolute consistency is required, nodes need to communicate when read/write operations are performed on a node Consistency over availability -> bottleneck As a trade-off, "eventual consistency" is used Consistency is maintained later Numerous approaches for keeping up "distributed consistency" are available Amazon Dynamo - consistent hashing CouchDB - asynchronous master-master replication MongoDB - auto-sharding+replication cluster with a master server

15 BASE SOFT STATE While ACID systems assume that data consistency is a hard requirement, NoSQL systems allow data to be inconsistent and relegate designing around such inconsistencies to application developers. In other words, soft state indicates that the state of the system may change over time, even without input. This is because of the eventual consistency model (the acronym is a bit contrived).

16 SOME BREEDS OF NOSQL SOLUTIONS Key-Value Stores Column Family Stores Document Databases Graph Databases In addition: Object and RDF databases as well as Tuple stores

17 KEY-VALUE STORES Key-Value is based on a hash table where there is a unique key and a pointer to a particular item of data. Mappings are usually accompanied by cache mechanisms to maximise performance. API is typically simple implementation is often complex. Example: Dynamo, Voldemort, Rhino DHT, etc.

18 COLUMN FAMILY STORES Store and process very large amounts of data distributed over many machines. "Petabytes of data across thousands of servers" Keys point to multiple columns Example: BigTable, Cassandra, HBase, Hadoop

19 DOCUMENT DATABASES (STORES) Documents are addressed in the database via a unique key that represents that document. Semi-structured documents can be XML or JSON formatted, for instance. In addition to the key, documents can be retrieved with queries. Redis is sometimes referred to as data structure server since keys can contain strings, hashes, lists, sets and sorted sets. Example: CouchDB, MongoDB, Lotus Notes, Redis, etc.

20 GRAPH DATABASES Graph Databases are built with nodes, relationships between nodes (edges) and the properties of nodes. Nodes represent entities (e.g. "Bob" or "Alice"). Similar in nature to the objects as in object-oriented programming. Properties are pertinent information related to nodes (eg. age: 18). Edges connect nodes to nodes or nodes to properties. Represent the relationship between the two. Scaling graph DBs is problematic Neo4J: cache sharding, sharding strategy heuristics Example: Neo4J, FlockDB, GraphBase, InfoGrip, etc.

21 SOME NOSQL CHALLENGES Lack of maturity numerous solutions still in their beta stages Lack of commercial support for enterprise users Lack of support for data analysis Maintenance efforts and skills are required experts are hard to find

22 WHAT IS MONGODB? Developed by 10gen, founded in 2007 A document oriented, NoSQL database Hash - based, schema - less database No Data Definition Language In practice, this means you can store hashes with any keys and values that you choose Keys are a basic data type but in reality stored as strings Document Identifiers (_id) will be created for each document, field name reserved by system Application tracks the schema and mapping Uses BSON format Based on JSON B stands for Binary Supports APIs (drivers) in many computer languages JavaScript, Python, Ruby, Perl, Java, Java Scala, C#, C++, Haskell, Erlang

23 FUNCTIONALITY OF MONGODB Dynamic Schema (No DDL) Document - based database Secondary indexes Query language via an API Atomic writes and fully - consistent reads If system configured that way Master - slave replication with automated failover (replica sets) Built-in horizontal scaling via automated range - based partitioning of data (sharding) No joins nor transactions

24 WHY USE MONGODB? Simple queries Functionality provided applicable to most web applications Easy and fast integration of data (No ERD diagram) Not well suited for heavy and complex transactions systems

25 MONGODB: CAP APPROACH Focus on Consistency and Partition Tolerance.

26 MONGODB: HIERARCHICAL OBJECT 0 or more Databases 0 or more Collections 0 or moredocuments 0 or more Fields

27 RELATIONAL DB CONCEPTS TO NOSQL Relational DB Database Table, View Row Column Index Join Foreign Key Partition MongoDB Database Collection Document (BSON) Field Index Embedded Reference Shard

28 MONGODB PROCESSES AND CONFIGURATION Mongod Database instance Mongos - Sharding processes Analogous to a database router. Processes all requests Decides how many and which mongod s should receive the query Mongos collates the results, and sends it back to the client. Mongo an interactive shell ( a client) Fully functional JavaScript environment for use with a MongoDB You can have one mongos for the whole system no matter how many mongods you have OR you can have one local mongos for every client if you wanted to minimise network latency.

29 CHOICES MADE FOR DESIGN OF MONGODB Scale horizontally over commodity hardware Lots of relatively inexpensive servers Keep the functionality that works well in RDBMSs Ad hoc queries Fully featured indexes Secondary indexes What doesn t distribute well in RDB? Long running multi - row transactions Joins Both artifacts of the relational data model (row x column)

30 BSON FORMAT Binary-encoded serialisation of JSON - like documents Zero or more key/value pairs are stored as a single entity Each entry consists of a field name, a data type, and a value Large elements in a BSON document are prefixed with a length field to facilitate scanning

31 SCHEMA FREE MongoDB does not need any pre - defined data schema Every document in a collection could have different data Addresses NULL data fields

32 JSON FORMAT Data is in name/value pairs Data is separated by commas Curly braces hold objects Square brackets hold arrays Example: {"employees":[ { "firstname":"john", "lastname":"doe" }, { "firstname":"anna", "lastname":"smith" }, { "firstname":"peter", "lastname":"jones" } ]}

33 JSON VS. XML JSON Example: {"employees":[ { "firstname":"john", "lastname":"doe" }, { "firstname":"anna", "lastname":"smith" }, { "firstname":"peter", "lastname":"jones" } ]} XML Example <employees> <employee> <firstname>john</firstname> <lastname>doe</lastname> </employee> <employee> <firstname>anna</firstname> <lastname>smith</lastname> </employee> <employee> <firstname>peter</firstname> <lastname>jones</lastname> </employee> </employees>

34 CRUD OPERATIONS Create db.collection.insert(<document>) db.collection.update(<query>,<update>,{update:true}) Read db.collection.find(<query>,<projection>) Update db.collection.update(<query>,<update>,<options>) Delete db.collection.remove(<query>,<justone>)

35 CRUD EXAMPLE Create db.employee.insert( firstname":"john","lastname":"doe ") db.employee.insert( firstname : John, lastname : Doe : Age":40) Read db.employee.find() db.employee.find({firstname: John }) Update db.employee.update({firstname: John },{$set: {firstname: Jane },{multi:true}) Delete db.employee.remove({firstname: John },{justone:true})

36 EMBEDDED DATA MODELS

37 NORMALISED DATA MODELS