Document Databases: MongoDB

Transcription

1 NDBI040: Big Data Management and NoSQL Databases hp:// Lecture 9 Document Databases: MongoDB Marn Svoboda svoboda@ksi.mff.cuni.cz Charles University in Prague, Faculty of Mathemacs and Physics Czech Technical University in Prague, Faculty of Electrical Engineering

2 Lecture Outline Document databases Introducon MongoDB Data model CRUD operaons Insert Update Remove Find: projecon, selecon, modifiers NDBI040: Big Data Management and NoSQL Databases Lecture 9: Document Databases: MongoDB

3 Document Stores Data model Documents Self-describing Hierarchical tree structures (JSON, XML, ) Scalar values, maps, lists, sets, nested documents, Idenfied by a unique idenfier (key, ) Documents are organized into collecons Query paerns Create, update or remove a document Retrieve documents according to complex query condions Observaon Extended key-value stores where the value part is examinable NDBI040: Big Data Management and NoSQL Databases Lecture 9: Document Databases: MongoDB

4 MongoDB Document Database

5 MongoDB JSON document database hps:// Features Open source, high availability, eventual consistency, automac sharding, master-slave replicaon, automac failover, secondary indices, Developed by MongoDB Implemented in C++, C, and JavaScript Operang systems: Windows, Linux, Mac OS X, Inial release in 2009 NDBI040: Big Data Management and NoSQL Databases Lecture 9: Document Databases: MongoDB

6 Query Example Collecon of movies Query statement Titles of movies filmed in 2005 and later, sorted by these tles in descending order Query result NDBI040: Big Data Management and NoSQL Databases Lecture 9: Document Databases: MongoDB

7 Data Model Database system structure Instance databases collecons documents Database Collecon Collecon of documents, usually of a similar structure Document MongoDB document = one JSON object I.e. even a complex JSON object with other recursively nested objects, arrays or values Each document has a unique idenfier (primary key) Technically realized using a top-level field NDBI040: Big Data Management and NoSQL Databases Lecture 9: Document Databases: MongoDB

8 Data Model MongoDB document Internally stored in BSON format (Binary JSON) Maximal allowed size 16 MB GridFS can be used to split larger files into smaller chunks Restricons on field names Top-level is reserved for a primary key Field names cannot start with $ Reserved for query operators Field names cannot contain. Used when accessing nested fields NDBI040: Big Data Management and NoSQL Databases Lecture 9: Document Databases: MongoDB

9 Primary Keys Features of idenfiers Unique within a collecon Immutable (cannot be changed once assigned) Can be of any type other than a JSON array Key management Natural idenfier Auto-incremenng number not recommended UUID (Universally Unique Idenfier) ObjectId special 12-byte BSON type (the default opon) Small, likely unique, fast to generate, ordered, based on a mestamp, machine id, process id, and a process-local counter NDBI040: Big Data Management and NoSQL Databases Lecture 9: Document Databases: MongoDB

10 Design Quesons Data modeling (in terms of collecons and documents) No explicit schema is provided, nor expected or enforced However documents within a collecon are similar in pracce implicit schema is required nevertheless Challenge Balancing applicaon requirements, performance aspects, data structure, mutual relaonships, query paerns, Two main concepts References Embedded documents NDBI040: Big Data Management and NoSQL Databases Lecture 9: Document Databases: MongoDB

11 Denormalized Data Models Embedded documents Related data in a single document with embedded JSON objects, so called subdocuments Pros: data manipulaon (fewer queries need to be issued) Cons: possible data redundancies Suitable for one-to-one or one-to-many relaonships NDBI040: Big Data Management and NoSQL Databases Lecture 9: Document Databases: MongoDB

12 Normalized Data Models References Related data in separate documents These are interconnected via directed links (references) Technically expressed using ordinary values with idenfiers of target documents (i.e. no special construct is provided) Features: higher flexibility, follow up queries might be needed Suitable for many-to-many relaonships NDBI040: Big Data Management and NoSQL Databases Lecture 9: Document Databases: MongoDB

13 Sample Data Collecon of movies Collecon of actors NDBI040: Big Data Management and NoSQL Databases Lecture 9: Document Databases: MongoDB

14 Applicaon Interfaces shell Interacve interface to MongoDB Drivers Java, C, C++, C#, Perl, PHP, Python, Ruby, Scala,... NDBI040: Big Data Management and NoSQL Databases Lecture 9: Document Databases: MongoDB

15 Query Language MongoDB query language is based on JavaScript Single command / enre script Read queries return a cursor Allows us to iterate over all the selected documents Each command is always evaluated over a single collecon Query paerns Basic CRUD operaons Accessing documents via idenfiers or condions on fields Aggregaons: MapReduce, pipelines, grouping NDBI040: Big Data Management and NoSQL Databases Lecture 9: Document Databases: MongoDB

16 CRUD Operaons Overview Inserts a new document into a collecon Modifies an exisng document / documents or inserts a new one Deletes an exisng document / documents Finds documents based on filtering condions Projecon and / or sorng may be applied too NDBI040: Big Data Management and NoSQL Databases Lecture 9: Document Databases: MongoDB

17 Insert Operaon

18 Insert Operaon Inserts a new document / documents into a given collecon db. collection. insert ( document [ document ], options ), Parameters Document: one or more documents to be inserted Provided document idenfiers ( fields) must be unique When missing, they are generated automacally (ObjectId) Opons Collecons are created automacally when not yet exist NDBI040: Big Data Management and NoSQL Databases Lecture 9: Document Databases: MongoDB

19 Insert Operaon: Examples Insert a new actor document Insert two new movies NDBI040: Big Data Management and NoSQL Databases Lecture 9: Document Databases: MongoDB

20 Update Operaon

21 Update Operaon Modifies / replaces an exisng document / documents db. collection. update ( query, update, options ) Parameters Query: descripon of documents to be updated The same behavior as in find operaons Update: modificaon acons to be applied Opons At most one document is updated by default Unless opon is specified NDBI040: Big Data Management and NoSQL Databases Lecture 9: Document Databases: MongoDB

22 Update Operaon: Examples Replace the whole document of at most one specified actor Update all movies filmed in 2015 or later NDBI040: Big Data Management and NoSQL Databases Lecture 9: Document Databases: MongoDB

23 Update Operaon Update / replace modes document { update operator }, Replace when the parameter contains no update operators The whole document is replaced ( is preserved) Update when the parameter contains only update operators Current document is updated using these operators,,,, Each operator can be used at most once NDBI040: Big Data Management and NoSQL Databases Lecture 9: Document Databases: MongoDB

24 Update Operators Field operators sets the value of a given field / fields $set : { field : value / array / document } removes a given field / fields, $unset : { field : " " } renames a given field / fields, $rename : { field : new field name }, NDBI040: Big Data Management and NoSQL Databases Lecture 9: Document Databases: MongoDB

25 Update Operators Field operators increments the value of a given field / fields $inc : { field : increment } mulplies the value of a given field / fields, $mul : { field : multiplier }, stores the current date me / mestamp to a given field / fields $currentdate : { field : { $type : "date" } } { $type : "timestamp" }, NDBI040: Big Data Management and NoSQL Databases Lecture 9: Document Databases: MongoDB

26 Update Operators Array operators adds one item / all items to the end of an array $push : { array field : value / array / document } { $each : array } adds one item / all items to the end of an array, but duplicate values are ignored, $addtoset : { array field : value / array / document } { $each : array }, NDBI040: Big Data Management and NoSQL Databases Lecture 9: Document Databases: MongoDB

27 Update Operators Array operators removes the first / last item of an array $pop : { array field : 1 } -1, removes all array items that match a specified query $pull : { array field : value / array / document } query, NDBI040: Big Data Management and NoSQL Databases Lecture 9: Document Databases: MongoDB

28 Upsert Mode Upsert behavior of update operaon When opon is specified, and, at the same me, no document was updated new document is inserted What this document will contain? In case of the replace mode All the fields (i.e. value fields) from the parameter In case of the update mode All the value fields from the parameter, and the outcome of all the update operators from the parameter field is preserved, or newly generated if necessary NDBI040: Big Data Management and NoSQL Databases Lecture 9: Document Databases: MongoDB

29 Upsert Mode: Example Unsuccessful update of a movie resulng to an inseron NDBI040: Big Data Management and NoSQL Databases Lecture 9: Document Databases: MongoDB

30 Remove Operaon

31 Remove Operaon Removes a document / documents from a given collecon db. collection. remove ( query, options ) Parameters Query: descripon of documents to be removed The same behavior as in find operaons Opons All the matching documents are removed unless opon is provided NDBI040: Big Data Management and NoSQL Databases Lecture 9: Document Databases: MongoDB

32 Find Operaon

33 Find Operaon Selects documents from a given collecon db. collection. find ( query, projection ) Parameters Query: descripon of documents to be selected Projecon: fields to be included / excluded in the result Matching documents are returned via an iterable cursor This allows us to chain further, or operaons NDBI040: Big Data Management and NoSQL Databases Lecture 9: Document Databases: MongoDB

34 Find Operaon: Examples Select all movies from our collecon Select a parcular movie based on its document idenfier Select movies filmed in 2000 with a rang greater than 1 Select movies filmed between 2005 and 2015 NDBI040: Big Data Management and NoSQL Databases Lecture 9: Document Databases: MongoDB

35 Selecon Query parameter describes the documents we are interested in { and / or logical operator } { field : value / array / document } { query operator },, Condions on fields (each field can be used at most once) Value equality The actual field value must be idencal to the specified value (including, e.g., the order of nested fields or array items) Query operators (each operator can be used at most once) The actual field value must sasfy all the provided operators NDBI040: Big Data Management and NoSQL Databases Lecture 9: Document Databases: MongoDB

36 Value Equality Condions: Examples Select movies having a specific director Select movies having specific actors Queries in both the pairs are not equivalent! NDBI040: Big Data Management and NoSQL Databases Lecture 9: Document Databases: MongoDB

37 Dot Notaon The dot notaon for field names field. field position Accessing fields of embedded documents E.g.: Accessing items of arrays E.g.: Posions start at 0 NDBI040: Big Data Management and NoSQL Databases Lecture 9: Document Databases: MongoDB

38 Value Equality Condions Example (revisited) Select movies having a specific director NDBI040: Big Data Management and NoSQL Databases Lecture 9: Document Databases: MongoDB

39 Query Operators Comparison operators $eq : value / array / document $ne $lt : value $lte $gte $gt $in $nin : [ value / array / document, ] Comparisons take parcular BSON data types into account Certain numeric conversions are automacally applied NDBI040: Big Data Management and NoSQL Databases Lecture 9: Document Databases: MongoDB

40 Query Operators Comparison operators, Tests the actual field value for equality / inequality The same behavior as in case of value equality condions,,, Tests whether the actual field value is less than / less than or equal / greater than or equal / greater than the provided value Tests whether the actual field value is equal to at least one of the provided values Negaon of NDBI040: Big Data Management and NoSQL Databases Lecture 9: Document Databases: MongoDB

41 Query Operators Element operators tests whether a given field exists / not exists $exists : true Evaluaon operators false tests whether a given field value matches a specified regular expression (PCRE) performs text search (text index must exists) NDBI040: Big Data Management and NoSQL Databases Lecture 9: Document Databases: MongoDB

42 Query Operators Array operators tests whether a given array contains all the specified items (in any order) $all : [ value / array / document ] Example (revisited) Select movies having specific actors, NDBI040: Big Data Management and NoSQL Databases Lecture 9: Document Databases: MongoDB

43 Query Operators Array operators tests the size of a given array against a fixed number (and not, e.g., a range, unfortunately) $size : size tests whether a given array contains at least one item that sasfies all the involved query operaons $elemmatch : query NDBI040: Big Data Management and NoSQL Databases Lecture 9: Document Databases: MongoDB

44 Query Operators Logical operators, $and : [ query, query, query ] $or Logical connecves for conjuncon / disjuncon At least 2 involved query expressions must be provided Only allowed at the top level of a query $not : { query operator } Logical negaon of exactly one involved query operator I.e. cannot be used at the top level of a query NDBI040: Big Data Management and NoSQL Databases Lecture 9: Document Databases: MongoDB

45 Querying Arrays Condion based on value equality is sasfied when... the given field as a whole is idencal to the provided value, or at least one item of the array is idencal to the provided value NDBI040: Big Data Management and NoSQL Databases Lecture 9: Document Databases: MongoDB

46 Querying Arrays Condion based on query operators is sasfied when... the given field as a whole sasfies all the involved operators, or each of the involved operators is sasfied by at least one item of the given array note, however, that this item might not be the same for all the individual operators Use when just one array item should be found for all the operators NDBI040: Big Data Management and NoSQL Databases Lecture 9: Document Databases: MongoDB

47 Projecon Projecon allows us to determine the fields returned in the result { field : 1 } true 0 false array field : { projection operator }, or for fields to be included or for fields to be excluded Posive and negave enumeraons cannot be combined! The only excepon is which is included by default Projecon operators allow to select parcular array items NDBI040: Big Data Management and NoSQL Databases Lecture 9: Document Databases: MongoDB

48 Projecon Operators Array operators selects the first matching item of an array This item must sasfy all the operators included in When there is no such item, the field is not returned at all $elemmatch : query selects the first items of an array (when is posive) / the last items (when negave) Certain number of items can also be skipped $slice : count [ skip, count ] NDBI040: Big Data Management and NoSQL Databases Lecture 9: Document Databases: MongoDB

49 Projecon: Examples Find a parcular movie, select its idenfier, tle and actors Find movies from 2000, select their tles and the last two actors NDBI040: Big Data Management and NoSQL Databases Lecture 9: Document Databases: MongoDB

50 Modifiers Modifiers change the order and number of returned documents orders the documents in the result returns at most a certain number of documents limit ( count ) skips a certain number of documents from the beginning skip ( offset ) All the modifiers are oponal, can be chained in any order, but must all be specified before any documents are retrieved via a given cursor NDBI040: Big Data Management and NoSQL Databases Lecture 9: Document Databases: MongoDB

51 Modifiers Sort modifier orders the documents in the result sort ( { field : 1 } ) -1, for ascending, for descending order The order of documents is undefined unless explicitly sorted Sorng of larger datasets should be supported by indices Sorng happens before the projecon phase I.e. not included fields can be used for sorng purposes as well NDBI040: Big Data Management and NoSQL Databases Lecture 9: Document Databases: MongoDB

52

53 Lecture Conclusion MongoDB Document database for JSON documents Sharding with master-slave replicaon architecture Query funconality CRUD operaons Insert, find, update, remove Complex filtering condions MapReduce Index structures NDBI040: Big Data Management and NoSQL Databases Lecture 9: Document Databases: MongoDB