Index everything One query type Low latency High concurrency. Index nothing Queries as programs High latency Low concurrency

Transcription

1

2 SCHEMA ON READ

3 Index everything One query type Low latency High concurrency Index nothing Queries as programs High latency Low concurrency

4 Index everything One query type Low latency High concurrency Index nothing Queries as programs High latency Low concurrency

5 IT S POPULAR, BUT WHY?

6

7 Diverse operational workloads are common Top 5 Marketing Firm Government Agency Top 5 Investment Bank Data Key / Value 10+ fields, arrays, nested documents 20+ fields, arrays, nested documents Queries Key based docs / query 80/20 read/write Compound queries Range queries MapReduce 20/80 read/write Compound queries Range queries 50/50 read/write Servers ~250 ~50 4 Ops / Sec 1,200, ,000 30,000 7

8 Some deployments are large Cluster Scale Performance Scale Data Scale Entertainment Company 1,400 servers 250 Million Ticks / Sec Petabytes Asian Internet Company 1,000+ servers 300k Ops / Sec 10s of billions of objects 250+ servers Federal Agency 500k Ops / Sec 13 billion documents 8

9 Multiple indicators suggest adoption is strong RANK DBMS MODEL SCORE GROWTH (20 MO) 1. Oracle Relational DBMS 1,442-5% 2. MySQL Relational DBMS 1,294 2% 3. Microsoft SQL Server Relational DBMS 1,131-10% 4. MongoDB Document Store % 5. PostgreSQL Relational DBMS % 6. DB2 Relational DBMS % 7. Microsoft Access Relational DBMS % 8. Cassandra Wide Column % 9. SQLite Relational DBMS % Source: 9 DB-engines database popularity rankings; May 2015

10 Source: Stack Overflow via Stackoverkill.com

11 Source: Stack Overflow via Stackoverkill.com

12 TO ME, THREE THINGS DRIVE THIS ADOPTION

13 We asked users why, here s what they told us { CODE } XML CONFIG DB SCHEMA APPLICATION OBJECT RELATIONAL MAPPING RELATIONAL DATABASE 13

14 We asked users why, here s what they told us { CODE } XML CONFIG DB SCHEMA APPLICATION OBJECT RELATIONAL MAPPING RELATIONAL DATABASE 14

15 #1 The data model RDBMS Database Table Index Row Join MongoDB Database Collection Index Document Embedding & Linking 15

16 Documents are rich data structures { Fields } first_name: Paul, surname: Miller, cell: , city: London, location: [45.123,47.232], Profession: [ banking, finance, trader ], cars: [ ] { model: Bentley, year: 1973, value: }, { model: Rolls Royce, year: 1965, value: } String Number Geo-Location Fields can contain an array of sub-documents Typed field values Fields can contain arrays 16

17 Documents are self-describing Documents in the same product catalog collection in MongoDB { { } product_name: Acme Paint, color: [ Red, Green ], size_oz: [8, 32], finish: [ satin, eggshell ] { } product_name: T-shirt, size: [ S, M, L, XL ], color: [ Heather Gray ], material: 100% cotton, wash: cold, dry: tumble dry low product_name: Mountain Bike, brake_style: mechanical disc, color: grey, frame_material: aluminum, no_speeds: 21, package_height: 7.5x32.9x55, weight_lbs: 44.05, suspension_type: dual, wheel_size_in: 26 } 17

18 #2 Idiomatic drivers & frameworks MEAN Stack Morphia 18

19 Documents map to language constructs // Java: maps DBObject query = new BasicDBObject( publisher.founded, 1980)); Map m = collection.findone(query); Date pubdate = (Date)m.get( published_date ); // Javascript: objects m = collection.findone({ publisher.founded : 1980}); pubdate = m.published_date; // ISODate year = pubdate.getutcfullyear(); # Python: dictionaries m = coll.find_one({ publisher.founded : 1980 }); pubdate = m[ pubdate ].year # datetime.datetime

20 #3 It s easy and fun Easy to acquire AGPL license Easy to install and configure up and running in <5 min Easy to get high performance no black magic for millisecond latency, scale out architecture Easy to deliver always on replication and automatic failover built in Easy to add, query data no complex modeling, no DDL 20

21 #3 It s easy and fun Easy to acquire AGPL license Easy to install and configure up and running in <5 min Easy to get high performance no black magic for millisecond latency, scale out architecture Easy to deliver always on replication and automatic failover built in Easy to add, query data no complex modeling, no DDL BUT WHAT ABOUT Data governance? Referential integrity? Analytics? 21

22 DOCUMENT VALIDATION

23 Data governance: document validation Implement data governance without sacrificing the agility that comes from schema on read 23

24 Document validation gives you flexible control Use familiar MongoDB Query Language Automatically tests each insert/update; delivers warning or error if a rule is broken You choose what keys to validate and how db.runcommand({ collmod: "contacts", validator: { $and: [ {year_of_birth: {$lte: 1994}}, {$or: [ {phone: { $type: string"}}, { { $type: string"}} ]}] }}) 24

25 Example validation failure db.contacts.insert( name: "Fred", year_of_birth: 2012 }) Document failed validation WriteResult({ "ninserted": 0, "writeerror": { "code": 121, "errmsg": "Document failed validation }}) 25

26 Many ways to validate, no foreign keys yet Can check most things that work with a find expression Existence Non-existence Data type of values <, <=, >, >=, ==,!= AND, OR Regular expressions Some geospatial operators (e.g. $geowithin & $geointersects) Validate existing data by wrapping expression in $not 26

27 Where MongoDB validation excels (vs. RDBMS) Simple Use familiar search expressions (MQL) No need for stored procedures Flexible Only enforced on mandatory parts of the schema Can start adding new data at any point and then add validation later if needed Practical to deploy Simple to role out new rules across thousands of production servers Light weight Negligible impact to performance 27

28 Controlling validation validationlevel off moderate strict validationaction warn error No checks No checks Warn on validation failure for inserts & updates to existing valid documents. Updates to existing invalid docs OK. Reject invalid inserts & updates to existing valid documents. Updates to existing invalid docs OK. Warn on any validation failure for any insert or update. Reject any violation of validation rules for any insert or update. DEFAULT 28

29 Versioning of validators (optional) Application can lazily update documents with an older version or with no version set at all 29 db.runcommand({ collmod: "contacts", validator: {$or: [{version: {"$exists": false}}, {version: 1, {Name: {"$exists": true}} }, {version: 2, {Name: {"$type": string"}} } ] } })

30 SCHEMA DISCOVERY

31

32 FUTURE DECISIONS

33 Still lots of hard problems to solve Schema evolution Specialized storage engines WORM Blockchain Proprietary hardware Integrated data warehouse Complex transactions 33

34 One surface fits all Content Repo IoT Sensor Backend Ad Service Customer Analytics Archive Security MongoDB Query Language (MQL) + Native Drivers MongoDB Document Data Model Management BTree LSM In-memory WORM Archive 34

35