Breaking Barriers: MongoDB Design Patterns. Nikolaos Vyzas & Christos Soulios

Transcription

1 Breaking Barriers: MongoDB Design Patterns Nikolaos Vyzas & Christos Soulios

2 a bit about us and this talk Who we are what we do

3 Christos Soulios Christos is a principal architect at Pythian Delivers Big Data platforms for some of the world's top tech organizations Expert in Big Data, Hadoop, NoSQL etc Working with MongoDB since v1.7 (back in 2011) 3

4 Nik Vyzas Nik is a Sr. TechOps Architect at Percona 10+ years experience in production support and enterprise software development for large scale distributed environments Expert in a variety of open-source technologies especially RHEL, Debian, Percona Server, XtraDB Cluster, MongoDB, Ansible, Java and Python. Over the years he has also mastered the dark art of turning caffeine into new software and bug fixes. 4

5 What is this talk about? Proven MongoDB design patterns Data modelling and indexing principles Common MongoDB pitfalls and how to avoid them Balancing performance and data consistency Best practices for scaling out / sharding How to generally press the go-faster button 5

6 Session Overview Indexing Strategy Data Modeling: To reference or to embed? Ranking / Fast Accounting in MongoDB Atomic Updates and the Optimistic Locking Pattern Keyword Search Pattern Defensive Programming Read / Write Concern Sharding Considerations 6

7 Indexing Strategy Optimizing Query Performance

8 Sometimes queries are a bit slow 8

9 Think about your indexing 9

10 Index Types Basic index types: Single field indexes Compound indexes Other noteworthy indexes: Text indexes Geospatial indexes Hashed indexes (* mainly for sharding) 10

11 Index Properties Unique - acts as index and constraint Sparse - only when field exists Partial - based on specified criteria TTL - * Keep in mind the TTLDeleter thread runs every 60 seconds 11

12 Indexing Tips Ensure your indexes fit in memory. Try to be minimal Don't index everything Indexes are costly When indexing timestamps, always index coarsely. NEVER index milliseconds Don t index fields with low cardinality Careful with text indexes 12

13 More Indexing Tips Prefer compound indexes that will improve multiple queries Create indexes that cover the queries - all data is retrieved from the index When developing code you can start mongod with the notablescan option Over time schemas and query patterns evolve, always review your indexes 13

14 The explain() plan 14

15 The explain() plan Returns the query execution plan for a specific query Provides execution statistics e.g. rows scanned, indexes used etc. For sharded collections information regarding shards accessed is included Use the explain plan to identify required indexes for filtering and sorting documents 15

16 The explain() plan No Index db.movies.find({'year': '2001'}).explain(true) "queryplanner" : { "winningplan" : { "stage" : "COLLSCAN", "filter" : { "year" : {"$eq" : "2001" } }, "direction" : "forward" } }

17 The explain() plan No Index "executionstats" : { "totaldocsexamined" : 250, "executionstages" : { "nreturned" : 7, "advanced" : 7, "direction" : "forward", "docsexamined" : 250 }, }, 17

18 The explain() plan - Indexed "queryplanner" : { "winningplan" : { "stage" : "IXSCAN", "keypattern" : { "year" : 1 }, "indexname" : "year_1", "ismultikey" : false, "isunique" : false, "issparse" : false, "ispartial" : false, "indexversion" : 1, "direction" : "forward", "indexbounds" : { "year" : ["[\"2001\", \"2001\"]" ] } 18

19 The explain() plan - Indexed "executionstats" : { "nreturned" : 7, "executiontimemillis" : 0, "totalkeysexamined" : 7, "totaldocsexamined" : 7, } 19

20 The Database Profiler Collects data about operations, cursors and db commands Configurable per database or per instance Allows setting slowopthresholdms to capture only slow queries or all Crucial for identifying bottlenecks and understanding workload 20

21 Data Modeling: To reference or to embed? Embedded vs. referenced pattern implementations

22 Data modeling We define data relationships between collections How do I join data? Effective data modeling Application side joins Two basic models: Embedded or Referenced ALWAYS ask yourself to reference or to embed? 22

23 Pattern: Embedded One-to-One Relationship {'_id' : ObjectId( ), } 'title' : 'Shawshank Redemption', 'director': { 'name' : 'Frank Darabont', }, 23

24 Pattern: Embedded One-to-Many Relationship } {'_id' : ObjectId( ), 'title' : 'Shawshank Redemption', 'writers' : [{'name':'stephen King', }, {'name':'frank Darabont', }], 24

25 The Embedded Model Faster reads / writes whole BSON is retrieved in 1x database call Updates at the document level enforce atomicity Duplication can lead to data inconsistencies Avoid embedding data with unbound growth Never embed documents that grow after creation (MMAPv1 storage engine) 25

26 Pattern: Referenced One-to-Many Relationship Reference on movie_id = 'xyz' { } movies '_id' : 'xyz', 'title' : 'The Wall',... { } { } reviews '_id' : ObjectId( ), 'movie_id' : 'xyz', 'rating' : 8,... '_id' : ObjectId( ), 'movie_id' : 'xyz', 'rating' : 2,... 26

27 The Referenced Model Enforces data consistency Allows for Parent & Child Tree References Each relationship requires an additional call Ensure that your referenced fields are indexed This becomes costly: Makes reading slower Makes writing slower Requires more indexes 27

28 Atomic Updates & the Optimistic Lock Pattern Atomic Updates & Collection Versioning

29 Atomic Update Operations >>> db.movies.update_one( {'rating': {'$gt': 9 }}, {'$set' : {'favorite' : True }}) >>> old_doc = db.movies.find_one_and_update( {'rating': {'$gt': 9 }}, {'$set' : {'favorite' : True }}) 29

30 Atomic Update Operations >>> db.movies.update_one( {'rating': {'$gt': 9 }}, {'$set' : {'favorite' : True }}) >>> old_doc = db.movies.find_one_and_update( {'rating': {'$gt': 9 }}, {'$set' : {'favorite' : True }}) The update and return occurs within a single atomic operation 30

31 Update Operators The list of valid update operators: $inc $set $unset $addtoset $push / $pushall : Add value into array $pop / $popall : Remove first / last value(s) of array : Increment counter : Set a new value : Set value = NULL : Add value into array (duplicates not inserted) $pull / $pullall : Remove instance(s) of value from array $rename : Update key name(s) 31

32 Pattern: Optimistic Locking For complex changes use the Optimistic Locking Design Pattern: Include a version field in all documents {'_id': ObjectId( ), 'title':'zootopia', 'v':1 } Retrieve a document and remember its version Make a series of complex transformations to the document or create a new one Do not forget to increment the version of the new document Update the document only if the version has not changed 32

33 Pattern: Optimistic Locking Update only if the document version has not changed m = db.movies.find_one({'title' : 'Zootopia'}) v = m['v'] # Remember the old version m = complex_transformations(m) m['v'] = v + 1 # Increment the version r = db.movies.replace_one({'_id' : m['_id'], 'v' : v}, m) if r.modified_count == 0: compensate() 33

34 Ranking / Fast Accounting in MongoDB High performance accounting to avoid aggregation

35 Pattern: Fast Accounting Use case: Count daily and monthly reviews posted for each movie. Display a histogram on the movie page Naive solution: Run counts on the reviews collection when histograms must be rendered Slow and resource consuming to aggregate millions of documents Calculating on every page view is too often Indexing may help but it will not solve the problem Fetching old data destroys page cache 35

36 Pattern: Fast Accounting Solution: Fast Accounting Design Pattern Create a separate collection to store aggregate counters Update counters when a new review is submitted If there are more than one counters, multiple updates will be performed This is a pattern taken from Complex Event Processing (CEP) 36

37 Pattern: Fast Accounting - Schema Create a separate collection named 'review_counts': { '_id': {'movie_id: ObjectId( ), 'day' : ' '}, 'count' : }, { '_id': {'movie_id: ObjectId( ), 'month' : ' '}, 'count' : } Query Dimensions 37

38 Pattern: Fast Accounting Increment counts Update daily counts: >>> db.review_counts.update_one({'_id': {'movie_id': ObjectId( ), 'day' : ' '}}, {'$inc' : {'count' : 1}}, upsert=true) Update monthly counts: >>> db.review_counts.update_one({'_id': {'movie_id': ObjectId( ), 'month' : ' '}}, {'$inc' : {'count' : 1}}, upsert=true) 38

39 Pattern: Fast Accounting Retrieve counts Retrieve daily count for a single day: >>> db.review_counts.find_one({'_id': {'movie_id': ObjectId( ), 'day' : ' '}})['count']

40 Pattern: Fast Accounting Documents for the latest dates and months are in memory Retrievals are very fast because they search indexed data Updates are very fast They happen in memory Use the _id index to ensure uniqueness and save space 40

41 Pattern: Fast Accounting Updates are atomic They can scale to thousands of concurrent updates Always use upsert=true to create new counters More dimensions can be added in the counter don't overdo it This pattern can be adopted for aggregating any timeseries data 41

42 Keyword Search Pattern Modelling data for retrieval based on specific keyword or tag

43 Living in the #hashtag world Use case: Retrieve a document based on a specific hashtag or keyword Naive solution: Add all tags delimited to a tags field and create a text index e.g. db.movies.createindex({"tags":"text"}) Text indexes require more space Take very long to build Significantly reduce insertion time More intensive retrieval processing 43

44 Keyword Search Pattern Solution: Keyword Search Pattern Create separate index entries per tags Groups documents based on tags Leverages multi-key indexes using an array (automatically created) Results in smaller and faster indexes compared to text 44

45 Keyword Search Pattern: Schema A movies collection with search keywords / tags: { '_id': {'movie_id: ObjectId( ), 'title' : 'World War Z'}, 'tags' : ['thriller','2016','zombies'] }, db.movies.createindex({tags: 1}) ### Separate index entries have now been created for: ### - thriller ### ### - zombies 45

46 Gotchas and Pitfalls Be careful: Insertion degrades on high cardinality (i.e. thousands) If indexes get too large asynchronous indexing may be required Allowing for free-text entry can lead to high cardinality - try to maintain a list if possible Be weary of case-sensitivity - consider forcing UPPER / LOWER case Do not use this pattern for full-text-search, rather prefer text indexing 46

47 Defensive programming Best practices for reading and writing data with a schemaless database

48 Structure in a schemaless world MongoDB does not enforce schema Key considerations for coding: Is the data I m writing valid? Is the data I m reading valid? 48

49 Structure in a schemaless world Methods for ensuring data is valid: Using BSON document types Document validation capability (3.2+) 49

50 BSON Document Types BSON provides support for common variable types, most importantly: bool int long double string array timestamp date objectid object 50

51 BSON Document Types Python types supported by Pymongo Pymongo converts Python types in a JSON document to BSON types BSON types also supported by the Java driver Generally - language specific drivers support BSON Custom Types can also be defined using a class Document types can also be defined using an ORM such as MongoEngine 51

52 BSON Document Types For example - insert a document in Pymongo enforcing datetime: >>> doc = { } "date": datetime(2003, 11, 26), "title_id":"tt ", "user_location":"texas", "title_name":"the Shawshank Redemption", "summary":"best movie ever!! >>> db.bsontest.insert(doc) 52

53 BSON Document Types Then retrieve the value in Mongo cli: > db.bsontest.find() { "_id" : ObjectId("570aae6d0059a38a781fed60"), "title_id" : "tt ", "user_location" : "Texas", "summary" : "Tied for the best movie I have ever seen", "date" : ISODate(" T00:00:00Z"), "title_name" : "The Shawshank Redemption" } 53

54 Document Validation Document validation is supported in Mongo 3.2+ Validation can be set during collection creation or on an existing collection Two modes of operation: Strict - Applied to all document inserts / updates Moderate - Applied to inserts / updates on documents that conform Setting validationaction: warn for testing (logs errors) error for enforcing (throws an error) 54

55 New Document Validation Create a validation on the dvtest collection: db.createcollection( dvtest", { validator : { $and: [ {"title_id" : { $type: "string" }}, {"user_location" : { $exists: true }}, {"title_name" : { $type: "string" }} ] } } 55

56 New Document Validation Insert an invalid document into the dvtest collection: db.dvtest.insert({ foo": "bar"}) WriteResult({ "ninserted" : 0, "writeerror" : { "code" : 121, "errmsg" : "Document failed validation" } }) 56

57 Existing Document Validation Add a new validation to an existing dvtest collection with moderate validation: db.runcommand( { collmod: "dvtest", validator: {$and:[{title_id: {$exists:true}}]}, validationlevel: "moderate" } ) 57

58 Read / Write Concern Read / Write Concern levels for CRUD operations

59 What is Write Concern Write concern determines the level of acknowledgement for data written by mongod processes: w = 0: No acknowledgement at all. It fails only if connectivity errors occur at the client application w = 1 (default): Require acknowledgement by the Primary replica w > 1: Acknowledgment by the number of replicas equal to w w = majority 59

60 Write Concern wtimeout: Time (ms) for an acknowledgement to return The j=true option requires an acknowledgement that data was written to the database journal 60

61 Write Concern Tips Never do unsafe writes (w=0) Except if you don t care about your data w=1 is not safe at all. A write can be overwritten by an outdated replica after a fail over w='majority' is safe. But it s slow w>1 is your best bet Always use wtimeout when w>1. If write concern cannot be achieved, the write will block forever 61

62 Read Preference read_preference specifies the replica instance that read operations are directed at: Possible values: PRIMARY [default] PRIMARY_PREFERRED SECONDARY SECONDARY_PREFERRED NEAREST 62

63 Read Concern Read concern specifies the isolation level for read operations ReadConcern( local ) returns local data stored on the replica queried [default] ReadConcern( majority ) returns data replicated to the majority of replicas i.e. already replicated majority is only supported by the WiredTiger storage engine, not by the MMAPv1 63

64 Read Concern Tips Read from secondaries when possible to scale reads All read preference modes except PRIMARY may return stale data because of replication lag majority read concern is slow majority read concern does not guarantee the latest data, but the latest data replicated to the majority of replicas 64

65 Sharding Considerations Hash vs. Timestamp Distribution

66 Sharding in MongoDB Sharding: Horizontal partitioning of data across multiple nodes / replicasets Sharded replicasets are recommended for HA Collections are sharded across replicasets based on a shard key High cardinality of the shard-key ensures even distribution across replicasets Collections which are not sharded remain on the primary shard 66

67 Sharding in MongoDB What are my sharding options? Hash based Range based Tag based 67

68 Hash Based Sharding Use hash indexes for the ranges Evenly distributed reads / writes Random operations due to random sharding algorithm Retrieving multiple documents can lead to scatter - gather Key use cases: Scaling: Load balancing reads & writes (example to follow) Disaster recovery: Parallel shard recovery 68

69 Hash Based Sharding Example: Shard key hash(datetime) - good write distribution WRITES :00: :00: :00:00 Shard1 (Primary) Shard2 Shard3 69

70 Hash Based Sharding Example: Shard key hash(datetime) - scatter gather reads Find datetime values between 17th & 21st of April READS :00: :00: :00:00 Shard1 (Primary) Shard2 Shard3 70

71 Hash Based Sharding Example: Shard key hash(userid) - good read distribution Find user with id = ed4f7269 READS ed4f7269 Shard1 (Primary) Shard2 Shard3 71

72 Range Based Sharding Ranges are defined on the defined data e.g. number / date-time Data is divided across range of documents E.G. 4x shards with int >> Shard1 with values etc. Can lead to hotspot shards on date-based ranges As ranges change chunk migration may cause overhead Key use cases: Scaling: Load balancing reads & writes Disaster recovery: Parallel shard recovery 72

73 Range Based Sharding Example: Shard key datetime value - bad write distribution On 20th April all writes go to Shard3 WRITES :00: :00: :00:00 Shard1 (Primary) Shard2 Shard3 73

74 Range Based Sharding Example: Shard key datetime value - bad write distribution Similar scenario with reads READS :00: :00: :00:00 Shard1 (Primary) Shard2 Shard3 74

75 Tag Based Sharding Allows for custom data distribution Data is divided across predefined tags E.G. Americas on Shard1.. EU on Shard2.. APAC on Shard3 Can lead to hotspots depending on use-case Key use cases: Geo-locality: Force data into suitable geographically dispersed shards HW Optimization: Force hot data onto faster hardware 75

76 Tag Based Sharding Example: Shard tags on location - faster response times Writes occur in a local DC WRITES USA GREECE AUSTRALIA Shard1 (Primary) Tag: AM Shard2 Tag: EU Shard3 Tag: APAC 76

77 Tag Based Sharding Example: Shard tags on location - faster response times Reads occur in a local DC READS WRITES USA GREECE AUSTRALIA Shard1 (Primary) Tag: AM Shard2 Tag: EU Shard3 Tag: APAC 77

78 Tag Based Sharding Example: Shard tags on year ranges - automatic archiving New data is written to high speed node WRITES <NEW DATA> <FEWER WRITES> <NO WRITE ACTIVITY> Shard1 (Primary) Tag: 2016 Shard2 Tag: Shard3 Tag: < x Cores - 128GB RAM - SSD 16x Cores - 64GB RAM - SSD 4x Cores - 32GB RAM - Rotational 78

79 Tag Based Sharding Example: Shard tags on year ranges - automatic archiving New data is written to high speed node READS WRITES <NEW DATA> <FEWER READS> <ONLY REPORTING> Shard1 (Primary) Tag: 2016 Shard2 Tag: Shard3 Tag: < x Cores - 256GB RAM - SSD 16x Cores - 64GB RAM - SSD 4x Cores - 32GB RAM - Rotational 79

80 the end Q&A