Scaling MongoDB: Avoiding Common Pitfalls. Jon Tobin Senior Systems

Transcription

1 Scaling MongoDB: Avoiding Common Pitfalls Jon Tobin Senior Systems

2 Agenda Document Design Data Management Replica3on & Failover Sharding

3 Document Design We re Not in SQL Anymore Embed vs Reference The Gray Area

4 Document Design Most common source of confusion & substandard performance Dynamic schemas are meant for agility (flexibility) No subs3tute for careful design You have to live with your decisions Consider design carefully NoSQL for a reason Denormaliza3on is common Flexibility means that there are oien no right or wrong answers Only the correct design for your use case MongoDB is not always the correct solu3on Consider mixing databases to maximize strengths

5 QnD Doc Design Pointers Embed Query performance priority Fields are fairly sta3c Size of doc can be reasonably determined Eventual consistency acceptable { } _id : ObjectId("53d98f1 ") firstname" : Jonathan, lastname" : Tobin, year" : 3, classes" : [ { class : Calc 101, credits : 3, }, { -etc- Reference Insert performance priority Updates are common Immediate consistency necessary Field size can t be determined { } _id : ObjectId("53d98f1 ") firstname" : Jonathan, lastname" : Tobin, year" : 3, classes" : [ ObjectID(<of_class_1>), ObjectID(<of_class_2>), ObjectID(<of_class_3>), ]

6 Finding Middle Ground Embed fields that are oien fetched If they don t grow boundlessly Limit growing keys to 1/per doc Move to last key Reference fields that are vola3le Or are occasionally queried Atomicity can be single doc level Take care in design Try TokuMX (ACID & MVCC) Index judiciously Re- evaluate oien Store relevant data Archive old data (when possible) Or delete

7 Doc Design Useful Resources MongoDB: The Defini3ve Guide - On Amazon A li^le outdated, but s3ll useful MongoDB Doc Design MongoDB Use Cases - Design

8 Data Management Why & How Details & Uses Other Options

9 Data Management Why? Ephemeral data Logs Sensor readings Space management Performance?? wiredtiger Op3ons TTL Capped Collec3on Par33oned Collec3ons (TokuMX) Ghe^o Par33oning

10 Capped Collec?ons Good Easy to use Db.createCollection( test, { capped : true, size : <size_in_bytes>, max : <opt_max_no_of_docs> } ) Automa3cally drop documents Based on inser3on order Easy to tail Bad No sharding Limited ability to update docs No deletes Unpredictable scale Especially queries

11 TTL Internals Addi3onal Index Single key only (date()) Uses a background thread on every RS member Runs EVERY 60 seconds Query full collec3on by TTL Deletes every doc > expireafterseconds older than current server 3me Single doc deletes Docs are only deleted from primary Thread s3ll runs on every secondary

12 Time to Live (TTL) Index Good Easy to use Db.coll.createIndex( { moddate : 1}, {expireafterseconds : 600} ) Automa3c expira3on of docs Bad Query & single doc deletes No guarantees Math 1000 inserts/sec * 600 (expireafterseconds)= 600,000 single doc deletes!!

13 Other Op?ons Par33oned Collec3ons TokuMX v1.5 and later Splits collec3on into separate files based on key Like par33oned tables in MySQL Deletes are batched and lightning fast Ghe^o Par33oning Separate your data into collec3ons based on 3me Difficult to s3tch collec3ons together Need to keep track of collec3ons in applica3on Dropping a collec3on is very efficient (rela3vely speaking)

14 TTL Tips When adding TTL to exis3ng collec3on Be careful of first run!!! Monitor disk u3liza3on closely 1. Increase resource pool 2. Shard Keep an eye on fragmenta3on Consider using an expireat field instead Will set the clock 3me expira3on instead Can use low workload period for dele3ons Check out preso by Kim Wilkins from ObjectRocket In Useful Resources slide (hidden)

15 TTL - Useful Resources Kimberly Wilkins (ObjectRocket) Math is Hard: TTL Configura3on & Considera3ons MongoDB Indexing Whitepaper Open TTL Jira Ticket MongoDB TTL Tutorial MongoDB Google Group Discussion- Par33oning TokuMX Par33oned Collec3ons

16 Durability & High Availability Life of a Write Durability Replication s Effects Considerations Along Your Journey

17 Life of a Write 1. Send by applica3on 2. Receipt by mongod 3. Applica3on in memory 4. Applica3on to journal (if enabled) 5. Write to data file 6. Applica3on to replica3on journal (oplog) 7. writeconcern determines strength of guarantee (more to come) Default acknowledged (in memory) 8. Acknowledge of write sent to host (depends on writeconcern)

18 Replica?on Every shard consists of a replica set Each replica set has a Primary & (n) Secondaries or Arbiters (no data) Secondaries read the Primary s replication log (oplog) and apply it asynchronously

19 Replica?on Lag Secondaries may lag behind the primary Rs.status() will get the current replica set config and replica3on informa3on Will affect: Durability Elec3ons Write throughput (depending on writeconcern) Why? Primary has concurrency. Replica3on is single threaded High resource u3liza3on on secondary Likely I/O bound Network latency

20 Concerns, Tips & Tricks Lag affects everything we ve talked about 1000 inserts/sec + 5 sec lag + elec3on 5000 inserts rolled back Determine it s source and a^empt to eliminate Most common cause is single threaded replica3on Double check network latency between hosts Solu3on: micro- sharding One server mul3ple mongods Parallelize replica3on TokuMX Read Free Replica3on Use Fractal Tree to speed up replay

21 Replica?on Useful Resources K Chodrow Elec3on Internals Blogs MongoDB Replica3on Whitepaper MongoDB Troubleshoo3ng Replica Sets

22 Sharding & Balancing Setup Tips Shard Keys A Look Into the Balancer

23 Setup Tips & Gotchas Make use mul3ple config servers Three is a safe bet Back them up regularly One mongos per app server When conver3ng from single RS Point all app servers to mongos Prevent app servers from connec3ng directly to RS Chunk size determines efficiency of balancing Also effects frequency of splixng overweight chunks Make sure config servers stay up One server down prevents shard reconfig ops Balancing Splixng Use DNS CNAMEs to refer to config servers Otherwise restart every mongod & mongos on rename

24 Shard Keys For insert speed: High- entropy shard key (mostly random). Balances load across all shards. Avoid migra3ons, too expensive in MongoDB. Sca^er is good. For query speed: Low- entropy shard key (mostly sequen3al). Range queries should only hit 1 shard. Sca^er is bad.

25 Balancing Main problem: random workload Everything relies on shard (secondary) key Documents aren t stored in chunk (shard key) order Three biggest opera3ons are high ac3vity 1. Range query on shard key for chunk 2. Dona3on to recipient shard 3. Range delete on shard key for dona3ng shard Take away: You have to balance Don t put it off for long periods Schedule balancing during low impact periods Manually balance

26 Sharding Resources MongoDB Balancing Process MongoDB Balancing Thresholds MongoDB: The Defini3ve Guide - On Amazon A li^le outdated, but s3ll useful TokuMX Balancing Improvements & Benchmarks Managing the Balancer

27 Use Discount Code WebinarPLAM for an additional 40 Euros off standard registration rates!