MongoDB - a No SQL Database What you need to know as an Oracle DBA

Transcription

1 MongoDB - a No SQL Database What you need to know as an Oracle DBA David Burnham

2 Aims of this Presentation To introduce NoSQL database technology specifically using MongoDB as an example To enable the audience to be able to participate in discussions about its appropriate usage. Provide a critical evaluation of the technology Highlight some of the trials and tribulations that have affected our project which relies on MongoDB for its data storage. Talk about the appropriate use and some of the benefits that could be realised by taking a different approach to data storage. Provide an opportunity to discuss NoSQL technology with fellow Oracle centric folk. 2

3 Why you need to know about MongoDB? Third standard within Enterprise Database Technologies Very easy for developers to get started with Cost of ownership is relatively low (even with commercial support) Success stories for big mission critical apps Lots of media attention for NoSQL solutions and MongoDB in particular 3

4 Why the Interest in NoSQL/MongoDB? Based around the idea that relation database technology is not always the best way of storing data for every application Unstructured or not very structured data Marketing data where constraints/relations are hard to enforce Relational databases incur an overhead in ensuring data manipulations and transactions have ACID properties If we really don t need ACID features we can realise significant performance benefits Relational model is often difficult to map to objects and application programmers can struggle with is Relational data design can be challenging and inflexible and can be seen as a limiter on change in agile projects SQL is a difficult language to program in effectively requiring specialised programmers 4

5 Examining MongoDB as an example of NoSQL Technology 5

6 MongoDB has many common features with RDBMS s MongoDB stores its data in table like structures called collections Related collections are grouped together as a database A database is like an Oracle schema in this respect Collections with contain zero or more documents Documents are JSON documents and are analogous to rows Each document does have a unique identifier on it Collections can and do have B-tree indexes on them Uses Javascript as a query language Javascript functions provide select/create/update/delete Journaling of writes and replication provides (some) resilience Network access from clients a central database or cluster 6

7 MongoDB is quite different from RDBMS s The data stored in collection in MongoDB is essentially schema-less Documents in a collection do not need to have any common attributes Attributes are defined within each document Any relationships between documents within a collection or within collections are only maintained by the application Atomicity is at the document level and there is no concept of transactions or linked operations again you have to manage that in the application Resilience is provided by journaling or replication and is configurable at the request level 7

8 Documents in a collection { "_id" : ObjectId("4fe8be4bbd dbb78"), "userid" : BinData(3,"H0WDAWVzbI66QuNIRb6Xkg=="), "classname" : "com.tvchannel.scrapbook.service.models.bookmark", "title" : "PerfTest/Bookmark/50/278", "uri" : " "imageuri" : " "description" : "Setup05_MongoDb_CreateFirstYearDataVolumes/449/1", "categories" : [ "4fed bc1b37990cb2e", "4fed bc1b37990cb2f", "4fed bc1b37990cb30" ], "programid" : "", "seriesid" : "", "episodeid" : "", "hidden" : false, "hiddentimestamp" : ISODate(" T23:59:59.999Z"), "createdtimestamp" : ISODate(" T12:24:13.131Z"), "displaytimestamp" : ISODate(" T19:38:51Z") } { "_id" : ObjectId("4fe8be4bbd db75a"), "userid" : BinData(3,"AUbUcdy0w/DaAnBXEgIDlw=="), "classname" : "com.tvchannel.scrapbook.service.models.bookmark", "title" : "PerfTest/Bookmark/50/349", "uri" : " "imageuri" : " "description" : "Setup05_MongoDb_CreateFirstYearDataVolumes/11/1", "categories" : [ "4fed bc1b37990cb2e", "4fed bc1b37990cb2f", "4fed bc1b37990cb30", "4fed bc1b37990cb32", "4fed bc1b37990cb33" ], "programid" : "", "seriesid" : "", "episodeid" : "", "hidden" : false, "hiddentimestamp" : ISODate(" T23:59:59.999Z"), "createdtimestamp" : ISODate(" T12:24:13.131Z"), "displaytimestamp" : ISODate(" T19:38:51Z") } { "_id" : ObjectId("4fe8be4bbd dbbaa"), "userid" : BinData(3,"HUJ/+61+X9u58Paf0KIYtw=="), "classname" : "com.tvchannel.scrapbook.service.models.bookmark", "title" : "PerfTest/Bookmark/51/28", "uri" : " "imageuri" : " "description" : "Setup05_MongoDb_CreateFirstYearDataVolumes/271/1", "categories" : [ "4fed bc1b37990cb30", "4fed bc1b37990cb34" ], "programid" : "", "seriesid" : "", "episodeid" : "", "hidden" : false, "hiddentimestamp" : ISODate(" T23:59:59.999Z"), "createdtimestamp" : ISODate(" T12:24:13.131Z"), "displaytimestamp" : ISODate(" T19:38:51Z") } 8

9 Scaling out MongoDB Performance is surprisingly good from a single MongoDB server about 3000 ops/s on 4 vcpu/16gb Amazon EC2 node (m1.large) MongoDB allows for horizontal scaling with sharding Management of sharding is done more or less automatically You select the collections you want to shard and choose a shard key and MongoDB should do the sharding for you Shard Key is an attribute shared by all documents in a collection Choosing a good shard key is a non trivial exercise Should provide a good split of data to distribute load Should allow for further splitting for future growth Could use _id but often this does not make a good shard key 9

10 Resilience and MongoDB Resilience in MongoDB is achieved by having replica copies of the contents of a data server on one or more separate servers All servers and replicas are called a replica set and a replica set has one primary server and some replicas, which read from the primary The server selected to be primary in a replica set is dynamic and if it fails the remaining servers will elect a new primary A majority of servers in a replica set is needed in order to elect a new primary and continue processing write requests, so odd numbers of servers in a replica set are a good idea Each data operation can be configured to have a different level of resilience, set by specifying how many nodes in the replica set have to get the change before it succeeds Different operations can be configured with varying resilience 10

11 Processes to support replication and scale-out through sharding clusters Mongos Intelligent database proxy that learns which node in a replica set is the primary and directs updates and consistent reads to it Learns from the config. databases where data chunks reside if a collection is split up into shards and directs updates and searches between shards accordingly Runs the balancer thread that will rebalance chunks of data between the shards to ensure even data distribution Configuration Databases Usually 3 config databases in a cluster that are all kept synchronised Keeps all the data about which node is primary in a replica sets and data locations in sharded collections Actually just another mongodb (mongod) proceess same as the data nodes 11

12 Example MongoDB Cluster 12

13 Good Things about MongoDB Programming model for a cluster is just the same as for a single node quite similar to Oracle RAC in this respect Auto shard management does (more or less) take care of itself, its easy to make sure the balancer only runs in the quiet periods of the day Very easy to create a data model that maps closely to how the data is used in the application As a programmer you can decide which operations you really care about and which you can afford to lose 10gen the developers and maintainers of Mongodb (the code is open source and on Github) provide excellent support and their commercial support is quite reasonably priced Good utilities are provided to backup and dump data etc 13

14 Worrying things about MongoDB (i) Schema-less nature of collections Each document attribute is really just a name value pair and the names are repeated in every document in the collection taking up a considerable amount of space After an application has evolved for a few years one may end up with a collection of documents in slightly different formats. The application may be the only piece of software that can make sense of this data, effectively preventing its re-purposing. Sharding has not demonstrated linear scaling for us Choosing a suitable shard key requires a lot of thought important to get one that will distribute the data evenly and yet allow the addition of more shards In our performance testing we never managed to get good balancing between the 3 shards we configured 14

15 Worrying things about MongoDB (ii) Replication is used as a get-out-jail card for resilience and durability Having many copies of my data, but not knowing which one is the correct one is not much use Oracle RAC one database = single point of truth is a powerful concept Lack of any serious aggregation functions can limit the product s usefulness Really only has the count() function and we found bugs with that! Can use external map/reduce but not really for an online DB app Still quite buggy Although 10gen are responsive and serious issues do seem to get their attention I get the feeling they ll get there soon but they have a way to go yet 15

16 Things I worried about before testing, but didn t really affect us Lack of a sophisticated block caching mechanism MongoDB simply mmap() s the data files and allows the operating system to page fault and load data that s not in memory With large memory servers (>10GB) this approach seems to work well. Guidance is that all of the indexes on your collections should fit into memory after that you need to add more shards. Locking granularity and concurrency MongoDB locks at the mongod process level (as of V2.0.3) If you have a mongod process having lots of databases all with lots of collections a write in any one of them will lock out writes to any of the others In practice we found that, because writes take place in memory, they happen very quickly and locking does not seem an issue. Again the guidance for reducing locking problems is to shard more 16

17 Bugs and issues that did catch us out Authentication appears to have been a rather recent bolt on to the data server and it shows Granularity is read-only or read/write at the database level and this is just about sufficient Suffered performance problems due to connection leakage when we turned on authentication (now fixed in latest release) Aggregation functions seem buggy Needed to have counts and ordering in the display of our data, but bugs in the MongoDB server forced us to process this in the application Mongos stops working properly if one of the config servers fails If a config server is lost mongos (the proxy) keeps trying to contact the server and the timeouts waiting, affects its ability to process requests The mongos process will not start unless all 3 of the config servers are present The config servers (all of them!) are a single point of failure 17

18 Good Use Cases for MongoDB A sharded MongoDB cluster makes a good sink for capturing large volumes of (non-critical) data. A great application for it is as a repository for log data from farms of servers. The log aggregation platform Greylog2 make use of MongoDB for this MongoDB makes a good repository for web content. JSON data storage maps well onto application data and a large amount of communication between browsers and web applications takes place using JSON. Anti Pattern for MongoDB MongoDB is a poor choice for applications where the data has high value or has stringent security requirements Highly structured data that is likely to be re-used is better stored in a traditional RDBMS Data that needs to be audited is best stored in an RDBMS 18

19 Final thoughts on MongoDB Really interesting to get to look a features of database technology that don t often get considered by DBA s and Engineers Balance and trade-offs between resilience/durability and performance Bit disappointed that we didn t ever see breath-taking performance. We could probably configure a sharded RDBMS to achieve similar performance (but we were quite conservative with the risk of data loss we were happy with) Question as the need for complex cache subsystems (like Oracle has) on modern high memory servers It s been a very positive experience for me looking at MongoDB and although I think it has a way to go yet, I think it could develop into a useful product, with wider application 19

20 Questions Presentation Name June 2012