INTRODUCTION TO MONGODB. MONGODB INTRODUCTION AND PRACTICE Raja CHIKY ISEP Paris

Transcription

1 INTRODUCTION TO MONGODB 1 MONGODB INTRODUCTION AND PRACTICE Raja CHIKY ISEP Paris raja.chiky@isep.fr

2 DOCUMENT MODEL Collection of «documents» «Key/Value» Model, te value is a Hierarchical semistructured document of type JSON or XML No schema for documents Tree structure: A list of fields, a field has a value that can be a list of fields, CRUD Best known: CouchDB, MongoDB, Terrastore, 2 INTRODUCTION TO MONGODB

3 STRENGTHS AND WEAKNESSES Strengths : Simple but powerful data model Good scaling up Strong expressiveness for queries (over nested structures) Faiblesses : Unsuitable for interconnected data Can be slow for queries with MapReduce 3 INTRODUCTION TO MONGODB

4 USE CASES To use for: l Content management systems (user profiles, reviews and comments, etc.) l Event management l E-commerce applications (catalog storage, flexible schema for products and orders) 4 INTRODUCTION TO MONGODB

5 COUCHDB ~2005 No schema, document-oriented databasedocuments stored in JSON format (XML in older versions) Indexed storage following B-tree No locking No join, no primary key or secondary key Implemented in Erlang 1st version in C ++ then in Erlang Replication Versioning 5 INTRODUCTION TO MONGODB

6 COUCHDB JSON FORMAT Similar to XML Easier to read and write { "Type" : "Auteur", "Prenom": "Julien", "Nom": "Dupont", "Sexe": "Femme", "date_naissance": " ", "date_inscription": " " } 6 INTRODUCTION TO MONGODB

7 JSON {"menu": { "id": "file", "value": "File", "popup": { "menuitem": [ {"value": "New", "onclick": "CreateNewDoc()"}, {"value": "Open", "onclick": "OpenDoc()"}, {"value": "Close", "onclick": "CloseDoc()"} ] } }} XML <menu id="file" value="file"> <popup> <menuitem value="new" onclick="createnewdoc()" /> <menuitem value="open" onclick="opendoc()" /> <menuitem value="close" onclick="closedoc()" /> </popup> </menu> 7 INTRODUCTION TO MONGODB

8 MONGODB DBMS open source, document-oriented. It is scalable, without a schema, and not relational Very rich query template System for indexing documents, Geo-spatial queries 8 INTRODUCTION TO MONGODB

9 DATA MODEL Collection Document Field Field Database Collection Document Field Collection Document 9 INTRODUCTION TO MONGODB

10 MONGODB- MAPREDUCE MongoDB implements the Map / Reduce framework Example: (in JavaScript) To get the list of age averages by city: >db.users.mapreduce(map, reduce, {out: { inline : 1}}); (The third argument makes it possible not to store the results in a new collection and therefore to do everything in memory). 10 INTRODUCTION TO MONGODB

11 MONGODB: MODÈLE DE DONNÉES 11 INTRODUCTION TO MONGODB

12 MONGODB - REQUÊTAGE 12 INTRODUCTION TO MONGODB

13 13 INTRODUCTION TO MONGODB

14 MONGODB: TOP 1 OF NOSQL DB 14 INTRODUCTION TO MONGODB

15 MONGODB: TOP 1 OF NOSQL DBS Many features {! name: John Smith,! pfxs: [ Dr., Mr. ],! address: 10 3 rd St.,! phones: [ { number: , type: land },! { number: , type: mobile }! ]! }! Documented oriented model Open- Source 15 INTRODUCTION TO MONGODB

16 DOCUMENT? {! }! _id: 123,! title: "MongoDB: The Definitive Guide",! authors: [! { _id: "kchodorow", name: "Kristina Chodorow },! { _id: "mdirold", name: Mike Dirolf }! ],! published_date: ISODate( ),! pages: 216,! language: "English",! thumbnail: BinData(0,"AREhMQ=="),! publisher: {! name: "O Reilly Media",! founded: 1980,! locations: ["CA, NY ]! }! 16 INTRODUCTION TO MONGODB

17 17 INTRODUCTION TO MONGODB

18 DOCUMENT DB SGBDR Database Table, View Row Column Index Join Partition MongoDB Database Collection Document (JSON, BSON) Field Index Embedded Document Shard Documents may have a different number of keys even if they are part of the same collection {name: Chiky, city: Paris, hobby:[ Tennis, Movies ]} {name: Chiky, city: Paris, profession: Associate professor, tel: } 18 INTRODUCTION TO MONGODB

19 DOCUMENT DB SGBDR Database Table, View Row Column Index Join Partition 19 INTRODUCTION TO MONGODB MongoDB > db.user.findone({age:39}) Database { Collection "_id" : ObjectId("5114e0bd42 "), "first" : "John", Document (JSON, "last" : "Doe", BSON) "age" : 39, Field "interests" : [ "Reading", Index "Mountain Biking ] Embedded "favorites": Document { Shard "color": "Blue", "sport": "Soccer"} }

20 RELATIONAL VS MONGODB Relational Customer ID First Name Last Name City 0 John Doe New York 1 Mark Smith San Francisco 2 Jay White Dallas 3 Meagan White London 4 Edward Daniels Boston Phone Number Type DNC Customer ID home T home T cell F home T cell (null) home F 2 20 INTRODUCTION TO MONGODB MongoDB { customer_id : 1,! name : {! }! f :"Mark,! l :"Smith },! city : "San Francisco",! phones: [!{!!! number : ,!! dnc : true,!!! type : home!!},!!{!!! number : ,!!! type : cell!!}]!

21 CRUD EXAMPLE > db.user.insert({ first: "John", last : "Doe", age: 39 }) > db.user.find () { "_id" : ObjectId("51 "), "first" : "John", "last" : "Doe", "age" : 39 } 21 > db.user.update( {"_id" : ObjectId("51 ")}, { $set: { age: 40, salary: 7000} } ) INTRODUCTION TO MONGODB > db.user.remove({ "first": /^J/ })

22 MONGODB- DATA STORAGE Storgage in BSON format ( binary JSON ) Supported types: String, Integer, Double, Date, Byte Array, Booléen, Null, BSON object et BSON Array Example in the "users» collection : { "_id": "c9167a fb439c7078 ", "username": "rchiky", "firstname": "Raja", "lastname": "Chiky" } Existing drivers in most programming C, C++, JavaScript, Python, Perl, C# /.NET, Java, PHP,... Flexibility: Optional Authentication Automatic collection generation Access to advanced features 22 INTRODUCTION TO MONGODB

23 LAUNCHING./bin/mongod 23 INTRODUCTION TO MONGODB

24 CLIENT LAUNCHING bin/mongo To familiarize yourself with the environment try these few commands >help >db.help() 24 INTRODUCTION TO MONGODB

25 «HELLO WORLD» EXAMPLE 25 INTRODUCTION TO MONGODB

26 QUERYING Display data db.location.find() ObjectID: Unique identifier of each document. Declared explicitly by the developer or implicitly by mongodb Format: BSON (binary JSON): binary serialization of "JSON-Like" documents with an extension for other types (date, binary data, etc.) Specification of BSON: 26 INTRODUCTION TO MONGODB

27 QUERYING Add records to the collection «location» Find record with zip code :75005 (Idem for name : Raja Chiky ) 27 INTRODUCTION TO MONGODB

28 LAB! >use library >document = ( { Type : "Book", Title : "Definitive Guide to MongoDB", ISBN : " ", Publisher : "Apress", Author: ["Membrey, Peter", "Plugge, Eelco", "Hawkins, Tim" ] } ) >db.media.insert(document) >db.media.insert( { Type : "CD",Artist : "Nirvana",Title : "Nevermind", Tracklist : [ { Track : "1 ", Title : "Smells like teen spirit", Length : "5:02 "}, { Track : "2 ", Title : "In Bloom", Length : "4:15 " } ]} ) 28 INTRODUCTION TO MONGODB

29 QUERIES: WHAT DO THESE COMMANDS DO? >db.media.find() >db.media.find ( { Artist : "Nirvana" } ) >db.media.find ( {Artist : "Nirvana"}, {Title: 1} ) >db.media.find ( {Artist : "Nirvana"}, {Title: 0} ) >db.media.find( { "Tracklist.Title" : "In Bloom" } ) >db.media.findone() Ajoutez la fonction pretty() pour l indentation >db.media.find().pretty() 29 INTRODUCTION TO MONGODB

30 FUNCTIONS: SORT, LIMIT AND SKIP >db.media.find().sort( { Title: 1 }) >db.media.find().sort( { Title: -1 }) >db.media.find().limit( 10 ) >db.media.find().skip( 20 ) >db.media.find().sort ( { Title : -1 } ).limit ( 10 ).skip ( 20 ) 30 INTRODUCTION TO MONGODB

31 AGGREGATION >db.media.count() >db.media.find( { Publisher : "Apress", Type: "Book" } ).count() >db.media.find( { Publisher: "Apress", Type: "Book" }).skip(2).count(true) 31 INTRODUCTION TO MONGODB

32 DISTINCT() Add a new record >document = ( { Type : "Book",Title : "Definitive Guide to MongoDB", ISBN: " ", Publisher : "Apress", Author : ["Membrey, Peter","Plugge, Eelco","Hawkins, Tim"] } ) >db.media.insert (document) >db.media.distinct( "Title") >db.media.distinct ("ISBN") >db.media.distinct ("Tracklist.Title") 32 INTRODUCTION TO MONGODB

33 AGGREGATION >db.media.group ( { key: {Title : true}, initial: {Total : 0}, reduce : function (items,prev) { prev.total += 1 } } ) Key: grouping parameter Initial: initial value (0 by default) Reduce: takes 2 arguments, the document (items) and the counter (prev) and performs aggregation Cond: condition that the attributes of the document must respect 33 INTRODUCTION TO MONGODB

34 ADD MORE RECORDS >dvd = ( { Type : "DVD", Title : "Matrix, The", Released : 1999, Cast: ["Keanu Reeves","Carry-Anne Moss","Laurence Fishburne","Hugo Weaving","Gloria Foster","Joe Pantoliano"] } ) >db.media.insert(dvd) >dvd = ( { "Type" : "DVD", "Title" : "Toy Story 3", "Released" : 2010 } ) >db.media.insert(dvd) Insert with JavaScript >function insertmedia( type, title, released ){ db.media.insert({ "Type":type, "Title":title, "Released":released }); } 36 >insertmedia("dvd", "Blade Runner", 1982 )

35 COMPARISON OPERATORS $gt, $lt, $gte, $lte, $ne, $in, $nin (resp. >,<,>=,<=,!=,IN, NOT IN) What do these queries do? >db.media.find ( { Released : {$gt : 2000} }, { "Cast" : 0 } ) >db.media.find( {Released : {$gte: 1990, $lt : 2010}}, { "Cast" : 0 }) >db.media.find( { Type : "Book", Author: {$ne : "Plugge, Eelco"}}) >db.media.find( {Released : {$in : ["1999","2008","2009"] } }, { "Cast" : 0 } ) >db.media.find( {Released : {$nin : ["1999","2008","2009"] },Type : "DVD" }, { "Cast" : 0 } ) $or >db.media.find({ $or : [ { "Title" : "Toy Story 3" }, { "ISBN" : " " } ] } ) >db.media.find({ "Type" : "DVD", $or : [ { "Title" : "Toy Story 3" }, { "ISBN" : " " } ] }) 37 INTRODUCTION TO MONGODB

36 $SLICE $slice: combines limit() and skip() $slice: [20, 10] // skip 20, limit 10 $slice: 5 // The first 5 $slice:-5 //The last 5 >db.media.find({"title" : "Matrix, The"}, {"Cast" : {$slice: 3}}) >db.media.find({"title" : "Matrix, The"}, {"Cast" : {$slice: -3}}) 38 INTRODUCTION TO MONGODB

37 $SIZE AND $EXISTS >db.media.find ( { Tracklist : {$size : 2} } ) >db.media.find ( { Author : {$exists : true } } ) >db.media.find ( { Author : {$exists : false } } ) 39 INTRODUCTION TO MONGODB

38 INDEX CREATION Ascending index >db.media.ensureindex( { Title :1 } ) Descending index >db.media.ensureindex( { Title :-1 } ) Index for enbed objects >db.media.ensureindex( { "Tracklist.Title" : 1 } ) Force the index usage: hint() >db.media.find( { ISBN: " "} ). hint ( { ISBN: -1 } ) error: { "$err" : "bad hint", "code" : } >db.media.ensureindex({isbn: 1}) >db.media.find( { ISBN: " "} ). hint ( { ISBN: 1 } >db.media.getindexes() 40 INTRODUCTION TO MONGODB

39 DATA UPDATE update(condition,newobject,upsert,multi) upsert=true //create the object if does not exist Multi Specifies whether the change is made on a single object (default) or on all objects that meet the condition >db.media.update( { "Title" : "Matrix, the"}, {"Type" : "DVD", "Title" : "Matrix, the", "Released" : "1999", "Genre" : "Action"}, true) Add/delete an attribute >db.media.update ( { "Title" : "Matrix, the" }, {$set : { Genre : "Sci- Fi" } } ) >db.media.update ( {"Title": "Matrix, the"}, {$unset : { "Genre" : 1 } } ) Delete Documents meeting a condition >db.media.remove( { "Title" : "Different Title" } ) All documents >db.media.remove({}) All the collection >db.media.drop() 41 INTRODUCTION TO MONGODB

40 GEOGRAPHIC DATABASE Import data >./bin/mongoimport --type json -d geodb -c earthquakes --file earthquakes.json Some basic queries: Count the number of documents Show first 5 View the 6th document How many separate statuses exist in the DB? 42 INTRODUCTION TO MONGODB

41 MODIFICATION OF DATA How many documents contain the property felt (propriety.felt! = Null) Delete this field for documents for which it is null Add an iso_date column whose value is the conversion of the timestamp contained in properties.time > db.earthquakes.find().foreach( } ); function(eq){ eq.properties.iso_date = new Date(eq.properties.time); db.earthquakes.save(eq); 43 INTRODUCTION TO MONGODB

42 DATA CLEANING Convert the string from the properties.types field to an array and put it in a field types_as_array Use the function ch.split (",") to separate a string ch into several words according to the separator ', db.earthquakes.find().foreach( function(eq){ var str = new String(eq.properties.types); eq.properties.types_as_array = str.split(","); db.earthquakes.save(eq); } ); Check by showing the 1st document 44 INTRODUCTION TO MONGODB

43 DATA CLEANING Clean the empty elements ("") from the array properties.types_as_array >db.earthquakes.update( ) {}, { $pullall: { "properties.types_as_array" : [""] } }, { multi: true } Check by showing the 1st document INTRODUCTION TO MONGODB 45

44 QUERIES Give the number of documents whose type list (properties.type_as_array) contains "geoserve" and "tectonicsummary» Give the number of documents whose type list (properties.type_as_array) contains "geoserve" or "tectonicsummary" 46 INTRODUCTION TO MONGODB

45 GEOGRAPHIC INDEXING We will now modify the data in order to adapt the geographical coordinates to the format that will allow us to build a 2dsphere index. Normalize the data by removing the last element from the 'geometry.coordinates' table and copy it to a 'depth' field. Example: 47 INTRODUCTION TO MONGODB

46 INDEX CREATION Create a 2dsphere type index on «geometry» attributes queries l Execute a query that looks for earthquakes near -74, (within a radius of 1000m) (use $geowithin and center) Documentation : INTRODUCTION TO MONGODB 49

47 QUERY Find the earthquakes that are around the square '8km NW of Cobb, California' with a maximum distance of 500km 51 INTRODUCTION TO MONGODB

48 GO FURTHER REPLICATION PARTITIONING (SHARDING) ADMIN / SECURITY 52 INTRODUCTION TO MONGODB

49 REPLICATION: 1 MASTER AND 1 OR MORE SLAVES Simple case: 1 master and 1 slave >mkdir /data/master >./bin/mongod --master --dbpath /data/master >mkdir /data/slave >./bin/mongod --slave --source localhost: dbpath /data/slave -port INTRODUCTION TO MONGODB

50 TESTING 1. Master connection >./bin/mongo >show dbs >use local >show collections >db.slaves.find() 3. Test >use testdb >db.testcollection.insert({name:'tim',su rname:'hawkins'}) > db.testcollection.find() 2. Slave connection >mongo --port >show dbs >use local >show collections 4. Test collection (slave side) >show dbs >use testdb >show collections > db.testcollection.find() 5. Add more slaves: >mkdir p /data/slave2 >mongod --slave --source localhost: dbpath / data/slave2 port INTRODUCTION TO MONGODB 54

51 REPLICATION: MANY MASTERS AND 1 SLAVE >mkdir /data/master1 >./bin/mongod --master --dbpath /data/master1 port Open a terminal >mongo localhost:27021 >use foo >db.foocollection.insert({foodata: our first foo document }) >quit() >mkdir /data/master2 >./bin/mongod --master --dbpath /data/master1 port >mongo localhost:27022 >use bar >db.barcollection.insert({bardata: our first bar document }) >quit() >mongod --slave --dbpath /data/slave port INTRODUCTION TO MONGODB

52 REPLICATION: MANY MASTERS AND 1 SLAVE Slave side >mongod --slave --dbpath /data/slave port >mongo localhost:27023 >use local >db.sources.insert({host:'localhost:27021'}) >db.sources.insert({host:'localhost:27022'}) >db.sources.find() >show dbs Exercice: Check the contents of the bar and foo collections You can decide to replicate only one collection by the following commands: >db.sources.insert({ host: localhost:27021, only: foo }) >db.sources.insert({ host: localhost:27022, only: bar }) 56 INTRODUCTION TO MONGODB

53 TESTING 1. Master connection >./bin/mongo >show dbs >use local >show collections >db.slaves.find() 3. Test >use testdb >db.testcollection.insert({name:'tim',su rname:'hawkins'}) > db.testcollection.find() 2. Slave connection >mongo --port >show dbs >use local >show collections 4. Test collection (slave side) >show dbs >use testdb >show collections > db.testcollection.find() 5. Add more slaves >mkdir p /data/slave2 >mongod --slave --source localhost: dbpath / data/slave2 port INTRODUCTION TO MONGODB 57

54 SHARDING We will construct the following architecture: Service Port dbpath Config server /data/config Shard /data/shard0 Shard /data/shard1 Create the required directories >mongod --port dbpath /data/config configsvr Then run the controller >mongos --configdb localhost: port chunksize 1 >mongod --port dbpath /data/shard0 shardsvr >mongod --port dbpath /data/shard1 --shardsvr 58 INTRODUCTION TO MONGODB

55 SHARDING Connect to the controller >mongo localhost:27021 >use admin >db.runcommand( { addshard : "localhost:27023", allowlocal : true } ) >db.runcommand( { addshard : "localhost:27024", allowlocal : true } ) Add a DB >phones=db.getsisterdb( phones ) >db.runcommand({enablesharding: phones }) >db.runcommand({ shardcollection : phones.testcollection", key : {_id : 1}}) >use phones 59 INTRODUCTION TO MONGODB

56 EXERCISE : FILL THE SHARDS Use the file populate_phones.js Analyze the code, in your opinion what does it do? Execute the code with the following parameters: >populatephones(800, , ) Display the 2 first documents With each new collection, MongoDB automatically creates an index following the _id identifier. These indexes can be found in system.indexes >db.system.indexes.find() Test the following three lines and explain the results >db.phones.find({display: " "}).explain() >db.phones.ensureindex( { display : 1 }, { unique : true, dropdups : true } ) >db.phones.find({ display: " " }).explain() 60 INTRODUCTION TO MONGODB

57 POPULATE_PHONE.JS populatephones = function(area,start,stop) { for(var i=start; i < stop; i++) { var country = 1 + ((Math.random() * 8) << 0); var num = (country * 1e10) + (area * 1e7) + i; db.testcollection.insert({ _id: num, components: { country: country, area: area, prefix: (i * 1e-4) << 0, number: i, testtext:"because of the nature of MongoDB, many of the more traditional functions that a DB Administrator would perform are not required. Creating new databases collections and new fields on the server are no longer necessary, as MongoDB will create these elements on-the-fly as you access them. Therefore, for the vast majority of cases managing databases and schemas is not required." }, display: "+" + country + " " + area + "-" + i }); } } 61 INTRODUCTION TO MONGODB

58 SHARDING - TESTING >mongo localhost:27021 >use phones >db.testcollection.count() >mongo localhost:27023 >use phones >db.testcollection.count() >mongo localhost:27024 >use phones >db.testcollection.count() 62 INTRODUCTION TO MONGODB

59 ADMIN Backup >mkdir testmongobackup >cd testmongobackup >../mongodb/bin/mongodump --help >../mongo/bin/mongodump >../mongodump --db library --collection media è./dump/[databasename]/[collectionname].bson Restore >cd testmongobackup >../mongo/bin/mongorestore help l Tout restaurer >../mongo/bin/mongorestore drop l Restaurer une seule collection >../mongo/bin/mongorestore d library -c media drop INTRODUCTION TO MONGODB 63

60 SÉCURITY Authentication l Client side > use admin >db.adduser( admin, adminpassword ) l Server side > use admin >db.adduser( admin, adminpassword ) l Shell (Restart the server) >sudo service mongodb restart Or >db.shutdownserver() l Authenticate >use admin >db.auth("admin","adminpassword") 64 >use library >db.adduser( raja, rajapassword ) >db.adduser( sylvain, sylvainpassword,true) //read only >db.removeuser( raja )

61 SÉCURITY: USER, PRIVILEGE, RESOURCE db.createuser( { user: "reportsuser", pwd: " ", roles: [ { role: "read", db: "reporting" }, { role: "read", db: "products" }, { role: "read", db: "sales" }, { role: "readwrite", db: "accounts" } ] } ) db.dropuser("reportsuser") db.dropallusers( ) 65 INTRODUCTION TO MONGODB

62 CREATE ROLE Db.createRole({ role: "<name>", privileges: [ { resource: { <resource> }, actions: [ "<action>",... ] },... ], roles: [ { role: "<role>", db: "<database>" } "<role>",... ] }) db.updaterole("< rolename >", ) db.droprole("< rolename >") db.grantprivilegestorole( "< rolename >", [ { resource: { <resource> }, actions: [ "<action>",... ] },... ], ) db.grantrolestorole( "<rolename>", [ <roles> ],) 66 INTRODUCTION TO MONGODB

63 FOR MORE INFORMATION Resource Case Studies Presentations Free Online Training Webinars and Events Documentation MongoDB Downloads Additional Info Location mongodb.com/customers mongodb.com/presentations university.mongodb.com mongodb.com/events docs.mongodb.org mongodb.com/download 67 INTRODUCTION TO MONGODB