NoSQL Databases Analysis

Transcription

1 NoSQL Databases Analysis Jeffrey Young Intro I chose to investigate Redis, MongoDB, and Neo4j. I chose Redis because I always read about Redis use and its extreme popularity yet I know little about it. I chose MongoDB because everyone uses it, I would like to know when not to use it. I chose Neo4j because I have no experience with graph databases. Neo4j is currently the most popular graph database so I figured it would be a good place to start. Redis History Redis was started in 2009 by an Italian developer named Salvatore Sanfilippo. Redis was written to improve the performance a personal real time web analytics startup of Salvatore. Soon Redis was stable enough to be replace the MySQL installation of Salvatore s startup. From there Redis gained much popularity and gained a large community. In 2010 VMWare hired Salvatore to work full time on Redis. With monetary funding from VMWare, and a large community, Redis has continued to grow into what it is today. Data Model Read: Redis is a key value database. Keys are stored as strings, Values can be stored as Lists, Strings, Sets, Hashes. A value can be retrieved with the command GET <KEY>. Create: Each value object type can be set using the command SET <KEY> <VALUE>. If the key is has a value before the SET command, the value will be overwritten. Update/Remove: Values can be updated using the SET command or additionally there are commands specific to each data type. The command APPEND <KEY> <VALUE> can be used to append to string values. For lists Command like PUSH, and POP exist. Sets can be added to with the command SADD <key> <member>. SREM can be called to remove members from a set. Use the command HGET <key> <field> to get the value of a hash field and HSET <key> <field> <value> to set the string value of a hash field.

2 These commands are executed through a terminal running a Redis, or additionally can be executed programmatically using a Redis Client in a language of your choice. Physical Storage Redis is an in memory data store. So all data is stored in memory. For persistance, Redis saves snap shots of the data to disk in a binary file called dump.rdb. You can configure how often Redis creates this dump of data, or do it manually before powering off your Redis application. When Redis restarted, it reads the dataset from disc, back into memory. Transactions With Redis transactions exist in the form of an ordered list of commands. All the commands in a transaction are serialized and executed sequentially. Since Redis is single threaded, ti can never happen that a request issued by another client is served in the middle of the execution of a Redis transaction. Either all the command or none are processed so a Redis transaction is also atomic. To create a transaction, use the MULTI command. Following commands will then be queued, and the EXEC command will execute the queued commands in order as a transaction. If there is an error inside a transaction before calling EXEC, the command will not be queued. A command may fail to be queued if the command is syntactically wrong, or there may be some critical condition like an out of memory condition. If there is an error after calling EXEC, all the commands queued commands will be executed in order except for any erroring commands. Redis does not support roll backs Redis does not support roll backs because errors should be very rare in redis. Most errors are syntactical errors and should be detected during development, not in production. Redis is gives faster performance by not supporting roll backs Since Redis is stored in memory, the DB always has perfect integrity. It is important to decide how often to create snapshots of the DB, to ensure that no data is lost in the case of a pull the plug from the wall type error. Scalability Since Redis is data is stored in memory, it is extremely fast. If the data you are storing will never exceed RAM, then Redis is very scalable and can handle lots of I/O. If you know that you may have

3 data sets that are larger than can be contained in RAM, Redis probably isn t the best option and a NoSQL database stores and retrieves data from disk should be used. MongoDB History MongoDB was first developed by a software company called 10gen in 2007 to be a database to handle lots of dynamic data. It was planned on being a component of a planned platform as a service product similar to Google App Engine. After a year of work, the MongoDB was ripped out of the app engine and open sourced. Immediately MongoDB began to gain users. Today MongoDB has thousands of users and a rich community of users. Data Model MongoDB is a document database. In MongoDB a record is a document, which is a data structure composed of field value pairs. Fields are always strings, values may include other documents, arrays, or arrays of other documents. A Document is similar to a JSON object. Each document has a unique _id field that acts as the primary key of the document. MongoDB stores documents in collections. A Collection is the analogous to a table in a relation database. Unlike a table, a collection does not require its documents to have the same schema. To summarize, MongoDB contains a set of collections. Each collection has documents. Documents are essentially JSON objects that need not conform to any schema. CRUD operations via a MongDB shell: CREATE: The command to insert is db.<collection_name>.insert(<json object string> ). If the colleciton does not exist it will be created. READ: To read all documents of a collection enter the command db.<collection_name>.find() To find documents with a certain field, value one enter the command db.<collection>.find({ <field>: <value> }) UPDATE: To update enter the command db.<collection>.update( arg1, arg2)

4 where arg1 contains the field you are selecting by, and arg2 is the field that you would like to update with the given value db.<collection>.update( { <field> : <value> ), { $set: { <field>: <value> } } ) To update multiple documents pass a third parameter to the update function, {multi: true } DELETE: To remove enter the command db.<collection>.remove( { <field>: <value> ) this command will remove all documents with the given field and value To drop a collection enter the command db.<collection>.drop() In addition to being executed in a MongoDB shell, all these commands can be executed programmatically using a client in the language of your chosing. Physical Storage A given mongo database is broken up into a series of BSON files of size up to 2 GB. BSON is a unique file format built specifically for MongoDB. BSON is short for Binary JSON, and was built to be lightweight, traversable, and have efficient encoding and decoding. By default that data is stored in the directory /data/db. On the disk records are via memory mapped files where files are mapped to a region of virtual memories. By using memory mapped files, MongoDB can treat data files as if they were in memory. When a data set grows too large for a single machine, Sharding can be used. Sharding divides the dataset, and distributes the dataset over multiple servers or shards. When a client requests a document, config servers help to determine in which shard the document resides. All free RAM on a machine is dynamically used as MongDB s cache. MongoDB s document representation is similar to its representation in RAM. Transactions MongoDB does not support multi document transactions. Atomic operations may be only done a singular document. Changes to a single document, will be consistent across the database.

5 Scalability Sharding provides much scalability. Data can scale horizontally through shards. MongoDB may run into scaling issues when too much critical data is located on a single shard. If all queries end up retrieving a document from a single shard, that shard s ability to do I/O will be the limiting factor of the application's scalability. It is important to evenly distribute critical data across different shards in order to increase mongo scalability. Neo4j History In 2000 Neo4j s founders were running into performance issues using Relational Databases to model graph data. In 2002 they developed the first version of Neo4j and used it for personal projects. In the next 8 years, the founders of Neo4j had dropped their other projects Neo4j version 1.0 was released in February, In 2011 Neo4j raised a round of funding and moved to Silicon Valley to focus on Neo4j development. Neo4j has multiple Data Model Neo4j is a graph database, so all data is modeled as a Node (vertex), Relationship (edge) or Property (attribute). Nodes and Relationships contain properties. Relationships connect nodes and are directional, thus each relationship has a start node and a end node. Properties are key value pairs. Crud with the Cypher query language CREATE NODE: CREATE (n:user { firstname: Jeffrey, lastname: Young } ) User is the label. (I think of Labels as the interface that a Node implements, so this is a Node that implements the User interface) n is the variable for the new node Properties are listed within the brackets CREATE EDGE: CREATE (n:user { firstname: Jeffrey, lastname: Young } ) CREATE (n1:user {fistname: Terrell, lastname: Young } ) CREATE (n) [r:is_son_of {since: 10_24_1990}] >(n1)

6 n is variable for son n1 is variable for father r is variable for relation IS_SON_OF is label for relation > indicates direction READ: MATCH (n:user) WHERE n.lastname = Young RETURN n User is the label n is the variable for the node WHERE restricts the result to a critera RETURN the properties on the node UPDATE: MATCH (u:user { name: Jeffrey }) SET u = {name: Updated Jeffrey } MATCH selects user where name is Jeffrey u is variable representing selected node set updates attributes on user DELETE: MATCH (user:user) WHERE user.lastname = Young DELETE user User is the label restricts search to the nodes under User label DELETE clause deletes a node from graph Physical Storage Neo4j stores graph data in several store files, each store file contains the data for a specific part of the graph neostore.nodestore.db neostore.relationshipstore.db neostore.propertystore.db neostore.propertystore.db.index neostore.propertystore.db.strings

7 neostore.propertystore.db.arrays In the node store each Node is stored as 9 bytes: 1st byte: in use flag next 4 bytes: ID of first relationship Last 4 bytes: ID of first property Fixed size records enable fast lookups Neo4j utilizes two caches. The file buffer cache stores data in the same format as it is stored in the store files. The object cache, caches the nodes, relationships and properties in a format that is optimized for fast graph traversal. Neo4j will use as much of the JVM heap as possible to cahche data. Transactions Neo4j supports all acid properties (atomicity, consistency, isolation, durability). To achieve ACIDity Neo4j implements the following transaction cycle: 1. Begin a transaction 2. Perform database operations 3. Mark the transaction as successful or not 4. Finish the transaction Finishing the transaction unlocks the nodes and relationships so that they can be updated by other transactions. In the case of an unsuccessful transaction, all changes are rolled back and an error is thrown. It is then up to the programmer to decide how to handle the error. Since locks are used during transactions it is possible that dead locks occur. Dead locks are automatically detected and Neo4j throws an error in such an instance. To handle a deadlock it is suggested to use a retry loop with an incremental back off. Scalability Neo4j is very scalable. From the neo4j website: Even on very modest hardware, Neo4j can handle millions of traversals per second between nodes in a graph on a single machine, and many thousands of transactional writes per second. This extreme speed is the result of an architecture that is natively engineered to store and process graph data. Neo4j can be configured to have a master database and several slave databases where several slave databases can be configured to be exact replicas of a single master database. This

8 enables a system to handle more read load than a single Neo4j database instance can. Sadly, Neo4j does not scale horizontally. All inserts and updates must go to a singular machine. Differences Each database has a very different use case and for that reason the databases are quite different. Each database should be used to handle different data sets: MongoDB excels when you need to store all sorts of dynamic data and want to have dynamic queries on a large data set. It also has great, user friendly horizontal scalability. MongoDB s weakness is handling relationships. Neo4j, being a graph database, is meant to handle datasets with complex relationships, and data that is highly connected. Such as a social network. Neo4j s weakness is its lack of horizontal scalability. If horizontal scalability is a large issue, out of these 3 options MongoDB would be a better option Redis is a great choice if you have a dataset that you KNOW will always fit in RAM. Because redis is in memory, it s super fast. If your data set is too large to fit in RAM, Redis works great as a caching layer. In the context of the CAP Theorem MongoDB retains Consistency, Availability Strongly consistent by default. This is because MongoDB is a single master system, all reads go through a singular master, thus data will never go out of sync on different shards MongoDB can be configured to optionally do reading from secondary nodes and thus will be eventually consistent MongoDB has high availability using automatic failover. If one node is down, a secondary will become primary and data can still be retrieved Redis is generally used in a single server deployment model thus Redis doesn t fit very well into the context of the CAP theorem. Redis maintains Consistency, and Partition Tolerance, however it sacrifices Availability if a node goes down Conclusion Each database has very different use cases. However I am most interested in Neo4j. I think Neo4j, being a graph database, gives it the greatest ability to handle interesting datasets and gain interesting insights from those data sets. I was also interested on how the Neo4j team worked to create a data model that is highly performant, such that 1 Neo4j instance can handle 1,000,000 s of I/O.