MongoDB Click Replication to edit Master and Sharding title style David Murphy MongoDB Practice Manager, Percona
Who is this Person and What Does He Know? Former MongoDB Master Former Lead DBA for ObjectRocket, original high-performance DBaaS for Mongo RDBMS background: 15+ years in MySQL NoSQL/MySQL Architect @ Electronic Arts Contributor to Mongo Core and tool builder 2
MongoDB Replication and Sharding Mongo Architecture OpLogs and Journals How replication works Special Replicaset settings What s in a configsvr? Talk through key collections What exactly is a chunk? What is it, and what happens in its life? Picking a shard key Bearing Fruit Understand what happened recently 3 Keeping thing balanced
4 Mongo Architecture - Overview
Mongo Architecture - Node Single Mongod process A single server could have many processes Holds the full data and index data Uses journalling for recovery Similar to MySQL node with no replication being used 5
Mongo Architecture - Replica Set Group of Nodes Forms the basis of HA via elections All members are equal by default and replicate to each other Oplog used in place of MySQL stype binlogs, its is a Capped collection, which is timestamp based and idempotent Will auto-heal/ rebuild bad secondary nodes 6
Mongo Architecture - Shard If sharding enabled Replica Sets are groups or shards in a clustered setup Additional thread started on primary call chunk manager Filters queries for only data the chunk should own Do the leg work of balancing while another starts the process 7
Mongo Architecture - Config Server In 3.2+ this is a special replica set, before that 1 or 3 stand alone machines for meta data Database Collections Chunks Changelog Lock Manager Mongos list 8
Mongo Architecture - Mongos If config servers are the memory Mongos is the brain Sharding Commands run on this Responsible for splitting chunks to keep cluster balanced chunk Responsible for tell 1 shard to drain and another to receive a Groups and sort multi-shard queries (scatter-gather) 9 Connect to shards in HA way for fault tolerance
Oplogs and Journals Oplog is like a MySQL binary log but bound by Timestamp (ntpd needed) is a capped collection / circular buffer uses statement based not binary is idempotent Unable to change an oplog size online (may change in 3.4) One statement updating 1M docs is 1M oplog entries Journaling Mongo has journaling like iblogs in MySQL but Different engines may or may not need them 10
How replication works Replication connections are normal connections, and may hit max cons No Separate SQL/IO threads Relay logs kept on node for buffering Uses tailable cursor model to detect when more data is available No indexes on timestamp Hard coded batch sizes High latency is a killer! Rollbacks are pure EVIL due to lack of re-integration 11
Special Replicaset settings 12 Delayed Secondary Used to have an offset node ( can be used for reporting) incase something bad happens. Many people will run ½ the backup offset, so at worst they roll forward 50% of the data in a recovery Hidden Used with Tags or delayed to prevent normal queries from using a node, sometimes you will backup with a hidden node. Tags Able to let queries tags specific nodes, maybe with special index nodes Chained Replication Secondary can replicate from Secondaries
What s in a configsvr? Key Collections Databases/Collections Chunks Shards Changelog Mongos Locks Types of Changes: Splits movechunks 13
What exactly is a chunk? What is it Simply said is a logical range of documents Is not physical Splits do not cause data movement by themselves Ranges live in config.chunks and tell mongos where to find data Lifecycle Splits - When a chunk grows the range is broken into part to keep balance. MoveChunks has stages of start, to, from, commit movechunk.from stage 4 is copying to new location Orphans - uncleaned up docs on source 14
Picking a shard key Must be immutable Should not be non-unique ( low cardinality prevents splits) No arrays, geo, sparse/partial indexes, or nulls No incremental fields on the left side of the shard key Never use _id as a stand alone key unless it is hashed Avoid using dates and sequential user id's If they are need put them on the right side Text fields are big try and avoid them Unique fields must be on the left side 15
Understand what happened recently Changelog - Best source of truth for all sharding actions Balancing and Splits New/Drop DB s and Collections New Sharding Locks Good to help understand if something is still running or done Logs Need to look at Mongos and Mongod logs ( donor and target) 16
Keeping things balanced Balancer can get stuck on a problematic collection Can manually run sh.movechunk to balance other collections Using changelog find bad chunk and move it by hand If it s a jumbo should be able to run sh.split on it Draining can be a pain Lookup sh.movechunk, sometimes doing it yourself will be easier than removeshard Avoid restarting mongos to often, they track when a split should happen based on a in memory counter, restarts lose this. 17
Bonus movechunk.from Stages Parse and Prepare Check config servers, and activate a migration lock Find documents on the donor shard and get a sort ordered list Copy chunk data to the destination shard (24 hour timeout), keep oplog Take a global lock Log event to changelog Update config.chunks and config version Cleanup* ( delete from source) and unlock for next migration 18
Thanks and Enjoy the Conference! Follow me on Twitter : @dmurphy_data Percona is hiring, are you ready to be challenged?