Import, Export, Index MongoDB

Transcription

1 Import, Export, Index MongoDB Kevin Swingler Single Line Entry So far, we have seen data entered into MongoDB like this: db.collection.insert({ name : value }) The data inside the () is in JSON format But we don t want to type this for every entry! 1

2 Data Import mongoimport - collection col -file filename.json Reads data from the file called filename.json into the collection called col You can also specify the database with --db dbname Read CSV files and use --ignoreblanks if the CSV contains sparse data (i.e. blanks) A separate utility, not a MongoDB command! Data Import If no collection name is given, the collection is named from the input file name without the extension Can choose to insert new data or --upsert, which inserts if no matching record is in the collection already, or updates if it is Specify fields to match with --upsertfields Collection needs to be indexed on fields used in upsert 2

3 Other Options Data Import --fields <field1[,field2... ]> Specify the names of fields if they are missing from the file to import --headerline File has a header row of field names --fieldfile <filename> Name of file containing field names instead of giving a list. One field per line. --type <json csv tsv> Specify the data format Data Import program/mongoimport/ 3

4 Data Export mongoexport --collection col --out coln.json Exports all documents from the collection named col into a file called coln.json You can select part of a collection by supplying a query: -- query {qry} Where the qry part is any MongoDB query Export from a non-local database: mongoexport --host db1.example.net --port username user --password pass MongoExport --fields <field1[,field2... ]> Specify the names of fields to export if not all fields are needed --sort Sort the output By default, output is JSON, but CSV causes output in CSV Use --skip and --limit to control how many documents are exported 4

5 JSON is not BSON MongoDB stores data in BSON format (Binary JSON), which includes extra functionality such as data types When exporting to JSON, this information is lost (same goes for CSV) docs.mongodb.org/manual/reference/bson-types/ docs.mongodb.org/manual/reference/mongodb-extended-json/ Indexes Indexes support the efficient execution of queries in MongoDB. Without indexes, MongoDB must scan every document in a collection to select those documents that match the query statement MongoDB defines indexes at the collection level Supported on any field or sub-field of the documents in a MongoDB collection. 5

6 Indexes Indexes are special data structures that store a small portion of the collection s data set in an easy to traverse form. The index stores the value of a specific field or set of fields, ordered by the value of the field. Each entry in the index points to the record in the DB Just like an index in a book Index in a Book A Ant Armadillo Apple B Bee Ant Armadillo

7 Index in MongoDB Index Structure Uses B-Trees, which are common DB index structures Search, insert and delete are all possible in O(log n) time 7

8 Why Index Searching every record is O(n) every field must be searched Searching B-Tree index is O(log n) much faster Other benefits: Results already sorted without the need for an additional sort operation after a query Covered results don t need DB access at all, just return the data from the index (e.g. Count the scores over 30) if the score field has an index Range search easy Index Types Default _id The default ID given to each field by MongoDB is indexed by default Single field User defined indexes on single fields, e.g username Compound Index Sorts first by one index, then by the next etc. Useful if values are duplicated or coverage needed across more than one field 8

9 Index array fields Multikey Index MongoDB creates separate index entries for every element of the array Other Special Indexes Geospatial Indexes Text indexes ignore words like a, the etc. See Hashed Indexes Indexes the hash of a field to support hash bases sharding 9

10 Index Properties Unique Indexes reject duplicate values for the index field Sparse Indexes only index entries that have the index field, so only contain a subset of the whole collection Full (non-sparse) indexes contain every document in the collection, using a null for those that do not contain the indexed field Ascending or Descending You can define the order of an index as ascending or descending Not generally important, but there might be scenarios where it is useful Think about the order you might want to sort results in when you query on an indexed field 10

11 Syntax db.col.ensureindex({field:order},{options}) Order {field:1} means acending, {field:-1} means descending, Useful options: {background:true} builds the index in the background to allow access to the DB during the build. Default=false {unique:true} Create a unique index. Default=false {name:newname} Give the index a name {sparse:true} Create a sparse index. Default=false docs.mongodb.org/manual/reference/method/db.collection.ensureindex/#db.collection.ensureindex Examples db.collection.ensureindex({orderdate: 1}) Index on orderdate, ascending db.collection.ensureindex({orderdate: 1, zipcode: -1}) Index on orderdate ascending, then zipcode descending db.reviews.ensureindex( { comments: "text" } ) Order on comments, with a text index 11

12 Other Index Operations db.collection.getindexes() db.collection.dropindex(name) Name is the one you gave it or, by default, fieldname_dir E.g. orderdate_1 Choosing Indexes Think about common queries Compound indexes speed queries across multiple fields Indexes are stored in RAM for speed Don t build so many that RAM is filled Also, Indexes can slow down writes as the index needs to be altered too, so only index what you need 12

13 Index Hints If you are searching across more than one field And more than one field has an index, And there is no compound index Which index should be used? Hints tell the DB which index to search on db.users.find().hint( { age: 1 } ) db.users.find().hint( "age_1" ) Index Hints Can also speed up search if many fields do not have a given field, if the index is sparse Causes it to only search the fields in the index (i.e. Those with the given field). 13