Introducing: MongoDB. David J. C. Beach

Transcription

1 Introducing: MongoDB David J. C. Beach

2 David Beach Software Consultant (past 6 years) Python since v1.4 (late 90 s) Design, Algorithms, Data Structures Sometimes Database stuff not a frameworks guy Organizer: Front Range Pythoneers

3 Outline Part I: Trends in Databases Part II: Mongo Basic Usage Part III: Advanced Features

4 Part I: Trends in Databases

5 Database Trends Past: Relational (RDBMS) WARNING: extreme oversimplification Data stored in Tables, Rows, Columns Relationships designated by Primary, Foreign keys Data is controlled & queried via SQL

6 Trends: Criticisms of RDBMS Rigid data model Hard to scale / distribute Slow (transactions, disk seeks) Lots of disagreement over this There are points & counterpoints from both sides The debate is not over Not here to deliver a verdict POINT: This is why we see an explosion of new databases. SQL not well standardized Awkward for modern/dynamic languages

7 As with so many things in technology, we re seeing... FRAGMENTATION! Trends: Fragmentation some examples of DB categories Relational with ORM (Hibernate, SQLAlchemy) ODBMS / ORDBMS (push OO-concepts into database) Key-Value Stores (MemcacheDB, Redis, Cassandra) Graph (neo4j) Document Oriented (Mongo, Couch, etc...) categories are incomplete some don t fit neatly into categories

8 Where Mongo Fits Mongo s Tagline (taken from website) The Best Features of Document Databases, Key-Value Stores, and RDBMSes.

9 What is Mongo Document-Oriented Database Produced by 10gen / Implemented in C++ Source Code Available Runs on Linux, Mac, Windows, Solaris Database: GNU AGPL v3.0 License Drivers: Apache License v2.0

10 Mongo many of these taken straight from home page Advantages json-style documents (dynamic schemas) flexible indexing (B-Tree) replication and highavailability (HA) automatic sharding support (v1.6)* fast queries (auto-tuning planner) fast insert & deletes (sometimes trade-offs) sharding support available as of v1.6 (late July 2010) easy-to-use API

11 Mongo Language Bindings C, C++, Java Python, Ruby, Perl PHP, JavaScript (many more community supported ones)

12 Mongo Disadvantages No Relational Model / SQL No Explicit Transactions / ACID Limited Query API Can mimic with foreign IDs, but referential integrity not enforced. Operations can only be atomic within single collection. (Generally) You can do a lot more with MapReduce and JavaScript!

13 When to use Mongo My personal take on this... Rich semistructured records (Documents) Transaction isolation not essential Humongous amounts of data Need for extreme speed You hate schema migrations Caveat: I ve never used Mongo in Production!

14 Part II: Mongo Basic Usage BRIEFLY cover: - Download, Install, Configure - connection, creating DB, creating Collection - CRUD operations (Insert, Query, Update, Delete)

15 Installing Mongo Use a 64-bit OS (Linux, Mac, Windows) Get Binaries: Run mongod process 32-bit available; not for production PyMongo uses memory-mapped files. 32-bits limits database to 2 GB!

16 Installing PyMongo Download: Build with setuptools (includes C extension for speed) # python setup.py install # python setup.py --no-ext install (to compile without extension)

17 Mongo Anatomy Mongo Server Database Collection Document

18 Getting a Connection Connection required for using Mongo >>> import pymongo >>> connection = pymongo.connection( localhost )

19 Finding a Database Databases = logically separate stores Navigation using properties Will create DB if not found >>> db = connection.mydatabase

20 Using a Collection Collection is analogous to Table Contains documents Will create collection if not found >>> blog = db.blog

21 Inserting collection.insert(document) => document_id >>> entry1 = { title : Mongo Tutorial, body : Here s a document to insert. } >>> blog.insert(entry1) ObjectId('4c3a12eb1d41c ') document

22 Inserting (contd.) Documents must have _id field Automatically generated unless assigned 12-byte unique binary value You can also assign your own _id, can be any unique value. >>> entry1 {'_id': ObjectId('4c3a12eb1d41c '), 'body': "Here's a document to insert.", 'title': 'Mongo Tutorial'} Mongo s IDs are designed to be unique... ID generated by driver. No waiting on DB....even if hundreds of thousands of documents are generated per second, on numerous clustered machines.

23 Inserting (contd.) Documents may have different properties Properties may be atomic, lists, dictionaries >>> entry2 = {"title": "Another Post", "body": "Mongo is powerful", "author": "David", "tags": ["Mongo", "Power"]} >>> blog.insert(entry2) ObjectId('4c3a1a501d41c ') another document

24 Indexing May create index on any field If field is list => index associates all values index by single value >>> blog.ensure_index( author ) >>> blog.ensure_index( tags ) by multiple values

25 Bulk Insert Let s produce 100,000 fake posts bulk_entries = [ ] for i in range(100000): entry = { "title": "Bulk Entry #%i" % (i+1), "body": "What Content!", "author": random.choice(["david", "Robot"]), "tags": ["bulk", random.choice(["red", "Blue", "Green"])] } bulk_entries.append(entry)

26 Bulk Insert (contd.) collection.insert(list_of_documents) Inserts 100,000 entries into blog Returns in 2.11 seconds >>> blog.insert(bulk_entries) [ObjectId(...), ObjectId(...),...]

27 Bulk Insert (contd.) returns in 7.90 seconds (vs seconds) driver returns early; DB is still working...unless you specify safe=true >>> blog.remove() # clear everything >>> blog.insert(bulk_entries, safe=true)

28 Querying collection.find_one(spec) => document spec = document of query parameters >>> blog.find_one({ title : Bulk Entry #12253 }) {u'_id': ObjectId('4c3a1e411d41c a89'), u'author': u'robot', u'body': u'what Content!', u'tags': [u'bulk', u'green'], u'title': u'bulk Entry #99999'} returned in 0.04s - extremely fast No index created for title! presumably, need more entries to effectively test index performance...

29 Querying (Specs) Multiple conditions on document => AND Value for tags is an ANY match >>> blog.find_one({ title : Bulk Entry #12253, tags : Green }) {u'_id': ObjectId('4c3a1e411d41c a89'), u'author': u'robot', u'body': u'what Content!', u'tags': [u'bulk', u'green'], u'title': u'bulk Entry #99999'} presumably, need more entries to effectively test index performance...

30 Querying (Multiple) collection.find(spec) => cursor new items are fetched in bulk (behind the scenes) >>> green_items = [ ] >>> for item in blog.find({ tags : Green }): green_items.append(item) - or - >>> green_items = list(blog.find({ tags : Green }))

31 Querying (Counting) Use the find() method + count() Returns number of matches found >>> blog.find({"tags": "Green"}).count() presumably, need more entries to effectively test index performance...

32 Updating collection.update(spec, document) updates single document matching spec multi=true => updates all matching docs >>> item = blog.find_one({ title : Bulk Entry #12253 }) >>> item.tags.append( New ) >>> blog.update({ _id : item[ _id ]}, item)

33 Deleting use remove(...) it works like find(...) >>> blog.remove({"author":"robot"}, safe=true) Example removed approximately 50% of records. Took 2.48 seconds

34 Part III: Advanced Features

35 Advanced Querying Regular Expressions { tag : re.compile(r ^Green Blue$ )} Nested Values { foo.bar.x : 3} $where Clause (JavaScript)

36 Advanced Querying $lt, $gt, $lte, $gte, $ne $in, $nin, $mod, $all, $size, $exists, $type $or, $not $elemmatch >>> blog.find({ $or : [{ tags : Green }, { tags : Blue }]})

37 Advanced Querying collection.find(...) sort( name ) - sorting limit(...) & skip(...) [like LIMIT & OFFSET] distinct(...) [like SQL s DISTINCT] collection.group(...) - like SQL s GROUP BY won t be showing detailed examples of all these... there are good tutorials online for all of this >>> blog.find().limit(50) # find 50 articles >>> blog.find().sort( title ).limit(30) # 30 titles >>> blog.find().distinct( author ) # unique author more interesting names let s move on to something even

38 Map/Reduce Most powerful querying mechanism collection.map_reduce(mapper, reducer) ultimate in querying power distribute across multiple nodes

39 Map/Reduce Visualized Figure 2-1. MapReduce logical data flow also see: Map/Reduce : A Visual Explanation Diagram Credit: Hadoop: The Definitive Guide by Tom White; O Reilly Books Chapter 2, page 20

40 19OPQ A*2=*LR SELECT Dim1, Dim2, SUM(Measure1) AS MSum, COUNT(*) AS RecordCount, AVG(Measure2) AS MAvg, MIN(Measure1) AS MMin MAX(CASE WHEN Measure2 < 100 THEN Measure2 END) AS MMax FROM DenormAggTable WHERE (Filter1 IN ( A, B )) AND (Filter2 = C ) AND (Filter3 > 123) GROUP BY Dim1, Dim2 HAVING (MMin > 0) ORDER BY RecordCount DESC LIMIT 4, 8! " # $ % ' & ()*+,-./ *2/4*5+123/6)-/,+55-./ *+7/63/8-93/02/7:-/16,/;+2470*2</ )-.+402=/7:-/30>-/*;/7:-/?*)802=/3-7@ A-63+)-3/1+37/B-/ /6==)-=67-.@ C==)-=67-3/.-,-2.02=/*2/)-4*)./4*+273/ 1+37/?607/+2705/;02650>670*2@ A-63+)-3/462/+3-/,)*4-.+)65/5*=04@ D057-)3/:6E-/62/FGAHC470E-G-4*).I 5**802=/3795-@ C==)-=67-/;057-)02=/1+37/B-/6,,50-./7*/ 7:-/)-3+57/3-7</2*7/02/7:-/16,H)-.+4-@ C =J/!K/L =J/I!! " # $ %! & ' db.runcommand({ mapreduce: "DenormAggCollection", query: { filter1: { '$in': [ 'A', 'B' ] }, filter2: 'C', filter3: { '$gt': 123 } }, map: function() { emit( { d1: this.dim1, d2: this.dim2 }, { msum: this.measure1, recs: 1, mmin: this.measure1, mmax: this.measure2 < 100? this.measure2 : 0 } );}, reduce: function(key, vals) { var ret = { msum: 0, recs: 0, mmin: 0, mmax: 0 }; for(var i = 0; i < vals.length; i++) { ret.msum += vals[i].msum; ret.recs += vals[i].recs; if(vals[i].mmin < ret.mmin) ret.mmin = vals[i].mmin; if((vals[i].mmax < 100) && (vals[i].mmax > ret.mmax)) ret.mmax = vals[i].mmax; } return ret; }, finalize: function(key, val) { val.mavg = val.msum / val.recs; return val; }, out: 'result1', verbose: true }); db.result1. find({ mmin: { '$gt': 0 } }). sort({ recs: -1 }). skip(4). limit(8); G-E030*2/$</M)-67-./"N!NIN#IN' G048/F3B*)2-</)048*3B*)2-@*)=

41 Map/Reduce Examples This is me, playing with Map/Reduce

42 Health Clinic Example Person registers with the Clinic Weighs in on the scale 1 year => comes in 100 times

43 Health Clinic Example person = { name : Bob,! weighings : [!! { date : date(2009, 1, 15), weight : 165.0},!! { date : date(2009, 2, 12), weight : 163.2},!!... ] }

44 Map/Reduce Insert Script for i in range(n): person = { 'name': 'person%04i' % i } weighings = person['weighings'] = [ ] std_weight = random.uniform(100, 200) for w in range(100): date = (datetime.datetime(2009, 1, 1) + datetime.timedelta( days=random.randint(0, 365)) weight = random.normalvariate(std_weight, 5.0) weighings.append({ 'date': date, 'weight': weight }) weighings.sort(key=lambda x: x['date']) all_people.append(person)

45 Insert Data LOG-LOG scale Linear scaling Performance Insert s s s 1k 10k 100k

46 Map/Reduce Total Weight by Day map_fn = Code("""function () { this.weighings.foreach(function(z) { emit(z.date, z.weight); }); }""") reduce_fn = Code("""function (key, values) { var total = 0; for (var i = 0; i < values.length; i++) { total += values[i]; } return total; }""") result = people.map_reduce(map_fn, reduce_fn)

47 Map/Reduce Total Weight by Day >>> for doc in result.find(): print doc {u'_id': datetime.datetime(2009, 1, 1, 0, 0), u'value': } {u'_id': datetime.datetime(2009, 1, 2, 0, 0), u'value': } {u'_id': datetime.datetime(2009, 1, 3, 0, 0), u'value': }... lots more...

48 Total Weight by Day Performance MapReduce s s s 1 1k 10k 100k

49 Map/Reduce Weight on Day map_fn = Code("""function () { var target_date = new Date(2009, 9, 5); var pos = bsearch(this.weighings, "date", target_date); var recent = this.weighings[pos]; emit(this._id, { name: this.name, date: recent.date, weight: recent.weight }); };""") reduce_fn = Code("""function (key, values) { return values[0]; };""") result = people.map_reduce(map_fn, reduce_fn, scope={"bsearch": bsearch})

50 Map/Reduce bsearch() function bsearch = Code("""function(array, prop, value) { var min, max, mid, midval; for(min = 0, max = array.length - 1; min <= max; ) { mid = min + Math.floor((max - min) / 2); midval = array[mid][prop]; if(value === midval) { break; } else if(value > midval) { min = mid + 1; } else { max = mid - 1; } } return (midval > value)? mid - 1 : mid; };""")

51 Weight on Day Performance 1000 MapReduce s 10 10s s 1k 10k 100k

52 Weight on Day (Python Version) target_date = datetime.datetime(2009, 10, 5) for person in people.find(): dates = [ w['date'] for w in person['weighings'] ] pos = bisect.bisect_right(dates, target_date) val = person['weighings'][pos]

53 Map/Reduce Performance 1000 MapReduce Python s 108s 26s s 2.2s s 1k 10k 100k

54 Summary

55 Resources PyMongo api.mongodb.org/python MongoDB The Definitive Guide O Reilly

56 END OF SLIDES

57 Chalkboard is not Comic Sans This is Chalkboard, not Comic Sans. This isn t Chalkboard, it s Comic Sans. does it matter, anyway?