MongoDB: Replica Sets and Sharded Cluster. Monday, November 5, :30 AM - 5:00 PM - Bull

Transcription

1 MongoDB: Replica Sets and Sharded Cluster Monday, November 5, :30 AM - 5:00 PM - Bull

2 About me Adamo Tonete Senior Support Engineer São Paulo /

3 Replicaset and Shards This is a tutorial and you can follow the steps locally Suggested hardware: 8GB RAM Dual Core processor MongoDB 3.6.x/4.0.x Linux/MacOS or Windows

4 Preparing the Environment cd ~ mkdir percona cd percona git clone chmod +x PL18/*.sh

5 Agenda - Single instance - Replica-sets and most of its configurations - Shards and most of its configurations

6 Choosing MongoDB The idea is to demonstrate all common configurations that can be applied in an environment according to its size requirements.

7 Beginning

8 What to consider Choosing MongoDB instead of a relational database; What are the advantages and disadvantages of MongoDB Documents style works for me? Dynamic schema (or schemaless) Speed Simple queries instead of complex joins

9 Starting MongoDB First contact with MongoDB Installing MongoDB./prepare_environment.sh

10 Starting MongoDB First contact with MongoDB mkdir single_instance mongod --dbpath single_instance --logpath single_instance/log.log --bind_ip wiredtigercachesizegb fork mongo

11 Starting MongoDB Does it mean my database is ready for production? Well, no, but we will not cover OS optimization in this talk - Check The minimum expected right now is to create an user and password to guarantee access control.

12 User Management

13 User management MongoDB doesn't come with an user such as root/sa in a relational database we need to enable the authentication in order to use this feature. Not having user was a problem in earlier versions where MongoDB listened all the network adapters by default.

14 User management This is what we should expect when we leave authentication disabled.

15 User management Creating an user and enabling authentication killall mongod mongod --dbpath single_instance --logpath single_instance/log.log --bind_ip , wiredtigercachesizegb fork --auth mongo > use admin > db.createuser({user : 'admin', pwd : '123', roles : ["root"]}) Successfully added user: { "user" : "admin", "roles" : [ "root" ] }

16 User management Creating read only user mongo > use admin > db.createuser({user : 'intern', pwd : '123', roles : ["readanydatabase"]}) Successfully added user: { "user" : "'intern'", "roles" : [ "readanydatabase" ] }

17 User management Testing credentials mongo -uadmin -p --authenticationdatabase admin use test db.foo.insert({today : new Date()}) mongo -uintern -p --authenticationdatabase admin use test db.foo.insert({today : new Date()})

18 Default Roles read readwrite dbadmin dbowner useradmin clusteradmin clustermonitor clustermanager hostmanager backup restore readanydatabase readwriteanydatabase useradminanydatabase dbadminanydatabase root system

19 Single Instance

20 Single Instance Mongo is running, time to start working in the application.

21 Single Instance issues If the instance goes down the application is down (single point of failure); Backups/Reports may affect application performance; No HA in place reads and writes are limited to the box limitations. Scaling up becomes very expensive (vertical scaling).

22 Replicasets

23 Replica-set Concept The replica-set is a group of mongodb instances that keeps the SAME data in it's nodes (like standby nodes in traditional RDBMS). Does offer high availability by default where only the primary receives writes. In case of a failure the remaining instances will vote in election to promote a SECONDARY to PRIMARY

24 A standard Replica-set Source:

25 Replica-set Concept Initialization Oplog Heartbeat Elections - Priorities and Votes Delayed Secondaries Hidden Secondaries Tags Removing members

26 Initializing a replica-set Let's change the single instance to a replica-set. Currently mongodb is not configured to use replica-set and we can check this in the config file or running the following command: db.admincommand( { getcmdlineopts: 1 } ) Use this command to check if the config file was loaded correctly

27 Config file MongoDB uses YAML file format since version 2.6 and here is how it looks like. storage: dbpath: replicaset1/instance1 journal: enabled: true wiredtiger: engineconfig: cachesizegb: 0.3 systemlog: destination: file logappend: true path: replicaset1/instance1/logs.log Also refer

28 Initializing a replica-set Using the config file available in the repository and Name your replica-set with replset variable in all nodes and start the 3 instances with: mongod -f <config cfg[1-3]>

29 Initializing The initialization process will let the database know it will be a replica-set and start recording all the operations after the rs.initiate() command was run In the run: mongo -u admin -p --authenticationdatabase admin rs.initiate()

30 Adding new Members

31 Adding new Members A replica-set is made of one or more instances. In order to have some HA we need at least 3 machines. rs.add(' ':27011') rs.status() rs.config() Note: rs.config() is an alias of rs.conf()

32 Adding a new Member Primary N: I'm new here P: Sure, I see we have the same replicaset name. N: May you send your data? P: Yes, but what is the password? N: I don't know I don't have data. P: I'm sorry then no data for you. New instance

33 Keyfile / Certificate As the primary does have an user and a password there is no way to the new member to sync and for this reason we need to use a shared key or certificates signed by the same CA to make the communication trustable. This we call as INTERNAL AUTHENTICATION.

34 Keyfile / Certificate Creating a sharded key: openssl rand -base > mykey.key chmod 600 mykey.key

35 Adding a new Member Primary New instance N: I'm new here P: Sure, I see we have the same replica-set name. N: May you send your data? P: Yes, but what is the password? N: I have a secret here that is supposed to be the same as yours. May you confirm? P: Yes, same as me, sending data.

36 Adding new Members

37 The Oplog

38 Oplog Oplog is a capped collection that holds the database commands in order to replicate the operations to other instances. It works in a similar way to the redo log (MySQL, Oracle, SQL Server)

39 Oplog Default size is 5% of free disk space - upto 50GB by default Can be set to higher or smaller values through replication.oplogsizemb parameter

40 Oplog Let's check what is inside of the oplog collection Connect to the primary and run: Use percona_test2 db.mycollection.insert({today : new Date()}) use local db.oplog.rs.find({}).sort({_id : '$natural'})

41 Oplog Window Oplog Collection

42 Oplog Window Oplog Collection Operations being recorded...

43 Oplog Window Oplog Collection Operations being recorded Almost full

44 Oplog Window Oplog Collection Operations being recorded Replacing old documents Oplog window - Seconds

45 Oplog If a secondary go out of the replication window a new initial sync will be performed. Replication lag (100s)

46 Oplog If a secondary go out of the replication window a new initial sync will be performed. Replication lag (30s)

47 Oplog resize Size should be set when starting a replica-set for the first time: Resize : => 3.6 { replsetresizeoplog: <boolean>, size: <num MB> } = <3.4 Need shutdown and manual intervention.

48 HA in Replicaset

49 Heartbeat The replicaset needs to check if the other instance is available and in order to do so every 10 seconds, the database will send a message to the others asking if they are alive

50 Heartbeat If the heartbeat fails it is possible an election will be triggered by eligible voting members of the replica set.

51 Replicaset Elections When the primary is not in a health state the remaining instances needs to elect a new primary. Election can happen due to: Instance crash Network issues (timeout) Adding new instances to the Replica-set Manual switchover

52 Replicaset Elections Priority means how likely would a instance will become a primary. Votes is the number of votes each instance has. The limit is to 7 voting members.

53 Replicaset Votes and Priorities Priority 1 Votes: 1 PRIMARY Priority 1 Votes: 1 SECONDARY Priority 1 Votes: 1 SECONDARY

54 Replicaset Votes and Priorities Priority 1 Votes: 1 PRIMARY Priority 1 Votes: 1 SECONDARY Priority 1 Votes: 1 SECONDARY

55 Replicaset Votes and Priorities Primary is Dead, how up-to-date are you? Priority 1 Votes: 1 SECONDARY Priority 1 Votes: 1 SECONDARY

56 Replicaset Votes and Priorities My last oplog is from 12: Primary is Dead, how up-to-date are you? Mine is 12:50.230, here is my vote Priority 1 Votes: 1 SECONDARY Priority 1 Votes: 1 SECONDARY

57 Replicaset Votes and Priorities My last oplog is from 12: Primary is Dead, how up-to-date are you? Mine is 12:50.230, here is my vote 2 of 3 votes "majority" Priority 1 Votes: 1 Primary Priority 1 Votes: 1 SECONDARY

58 Replicaset Votes and Priorities Priority 1 Votes: 1 Secondary Priority 1 Votes: 1 Primary Priority 1 Votes: 1 SECONDARY

59 Replicaset Elections Forcing an election with rs.stepdown() will make the replica-set pick another primary. We can specify if we want a member to don't become a primary with rs.freeze(num) where the number is in seconds

60 Delayed Secondaries Delayed instance is an replica-set member with no priority or votes that has an snapshot of the data from the desired time. We can have a delayed secondary to be able to check how the data was 1 hour ago. It is not a backup

61 Delayed secondaries No Replication Lag Secondary delay 100 seconds

62 Hidden instances Drivers will check the hidden flag to DO NOT connect to hidden instances. The goal of having hidden instances is to be able to work on those instances without affecting the application/driver performance.

63 Arbiters What if the replicaset has a even number of votes? Arbiters can be used to untie a voting and they are a replicaset member. The only difference is that arbiter doesn t hold data.

64 Arbiters Priority 1 Votes: 1 Secondary Priority 1 Votes: 1 Primary arbiteronly: true Votes: 1 (NO DATA)

65 Removing Instances rs.remove( myip:27002 ) If instance is not online the rs.remove() may fail. We can manually edit the replica-set configuration manually if needed with rs.config() and rs.reconfig()

66 Demo

67 Read and Write Preferences

68 Read Preference By default, reads and writes go to the primary. We can change the read Preference to read from secondaries only, for the nearest or only primaries.

69 Read Preference PRIMARY PRIMARY_PREFERRED SECONDARY SECONDARY_PREFERRED NEAREST TAGS

70 Example rs1:primary> db.getmongo().setreadprefmode('secondary') rs1:primary> db.getmongo().getreadprefmode() secondary rs1:primary> db.serverstatus().host testserver:27010 rs1:primary> db.foo.find() T19:27: I NETWORK [js] Successfully connected to :27012 (1 connections now open to :27012 with a 0 second timeout) rs1:primary>

71 WriteConcern WriteConcern is the ability to write to one or more instances before sending an acknowledge to the client. By default, MongoDB will only write to one instance. However for a better consistence the writeconcern parameter can be changed according to the business requirements.

72 Write Concern Primary Secondary Secondary

73 Modifying WriteConcern db.item.insert( { item: "car"}, { writeconcern: { w: 2, wtimeout: 5000 } } ) cfg = rs.conf() cfg.settings.getlasterrordefaults = { w: "majority", wtimeout: 5000 } rs.reconfig(cfg)

74 Tags Additionally to the default read preferences it is possible to add TAGs to a replicaset. Drivers will be able to only read from a specific instance when using tags

75 TAGS Primary Secondary Secondary TAG= { disk : 'fast'}

76 Tags from pymongo.read_preferences import Secondary db = client.get_database(... 'test', read_preference=secondary([{'disk': 'fast'}]))

77 Backup

78 Mongodump Mongodump generates a logical backup and it is one of the most used tool to backup mongodb. When running a replicaset it is possible to have a point in time backup

79 Disk Snapshot LVM Snapshots are a good option to take a point in time backup from a instance. Possible issue is the database can slow down while the process is running

80 SCP/RSYNC SCP or RSYNC can be done with the instance stopped or when the database is not receiving any writes. It was a very common way to sync new members with MMAP

81 Hot Backup Only available on Percona Server for MongoDB the hot backup is a lightweight process that generates a binary copy of the data path in a different folder. This is a point in time backup and can be used to start new members in a replicaset (includes oplog)

82 Common Issues

83 Replication Delay Replication process runs asynchronous and this leads to possible replication delay. Remote instances with slow connection and/or slow disks may slow down the replication threads and create replication delay.

84 Clock strew Having different machines in different datacenter also increase the probability of having clock strew. It is highly recommended to use NTP service in all the boxes to avoid possible issues within mongodb. New versions does have an internal clock but still a good practice.

85 Break time 15 minutes

86 Sharded Cluster

87 Sharded Cluster Concept A shard is way to distribute data in multiple machines in a transparent fashion to the application/final user. A shard is made of MONGOS, MONGOD(Config) and MONGOD instances holding data.

88 Sharded Cluster Concept

89 Sharded Cluster Concept The data is here

90 Sharded Cluster Concept And here as well (but not the same as shard1)

91 Sharded Cluster Concept It is a Replica-set

92 Sharded Cluster Concept Config servers does have Metadata information in the other words this instance is responsible to telling the mongos where the data is

93 Sharded Cluster Concept For the application. It is a big mongod (Mongos is a proxy)

94 What is a Shard In a very simple way, a shard is a replicaset holding part of the data. And those replicasets together hold the entire database. It is a way to scale out writes, as the writes may go to multiple shards instead of just one

95 Adding new Shards Before adding new shards we need to have the following processes configured and running: mongo config mongos

96 Adding new Shards In order to add new shards we need to use: sh.add()

97 Default shard Every time a database is created it is assigned to a default shard. Databases are not sharded automatically when created. It is necessary to enable sharding and also to shard each collection individually.

98 Deploying the Cluster

99 Data Management

100 Shard Key Is the key the database will use to distribute the data in chunks/shards. Use a good cardinality key Avoid monotonically increased key Not possible to change online

101 Ideal World

102 Monolithic inserts

103 Hash Key

104 Hash Key

105 Chunks By Default chunk is 64MB or documents but this can be changed in the shard config. Ideally chunks are evenly distributed among shards (if the collection is sharded) Indivisible chunks are marked as JUMBO chunk and some manual intervention may be necessary

106 Chunks Chunk is a subset of a sharded data. We can consider a box as a chunk in this shelf chunk SHARD01 SHARD02

107 Chunk migrations Chunks may migrate automatically, or manually. There is a process called Balancer running on the config servers to evaluate how distributed the data is in the shards. Use config db.settings.save( { _id:"chunksize", value: <sizeinmb> } )

108 Migrations Manual migration using movechunk() db.admincommand( { movechunk : 'percona_shard.smalcollection', bounds : [{ "_id" : NumberLong(" ") }, { "_id" : NumberLong(" ")}], to : 'rs1' } )

109 Config Servers Keeps the metadata from the entire cluster. User management Balancer processes. Before 3.2 config servers were mirrored and nowadays they become a replicaset.

110 Config server internals

111 Config server internals changelog chunks collections databases lockpings locks migrations mongos shards tags version

112 Config server internals Changelog will log all the cluster operations. As exemple it records when a collection has been sharded. This collection is very useful to troubleshooting.

113 Config server internals Chunks collections will show all individual chunks and where they live.

114 Config server internals All the sharded collections are present in the collections collection with its shard key.

115 Config server internals All the databases are in the database collection. Even though the database is not sharded.

116 Config server internals Lockping records last time it received an ping reply from the hosts in the shard:

117 Config server internals Locks - Some operations need to run individually and the idea of having the locks collection is to control actions. It is not possible to enable sharding in one collection from 2 different mongos at the same time.

118 Config server internals Migrations will save a log for each migration that happened in the cluster

119 Config server internals Mongos collection keep one line for each mongos in the shard. Even not available mongos will have one line here.

120 Config server internals Settings holds all the shard configuration. Some values are hidden if default.

121 Config server internals Shard collections will have one line for each shard, and each shard is a replicaset.

122 Config server internals Finally tags does have tags configured for each shard and the version collection is used for internal purposes.

123 Shard Zones

124 Geo Distributed Shard It is possible to take advantage of the shard key to distribute data according to the application/business need. We can use a specific shard to only hold data from a range or application

125 Geo Distributed Shard Config mongos Shard1 NA Shard2 EU

126 Geo Distributed Shard Add tag to a shard sh.addshardtag("rs1", "NA") sh.addshardtag("rs2", "EU") Configure collection accordingly to the needs: sh.addtagrange( "percona.events", {"location" : "NA","_id" : MinKey }, { "location" : "NA", "_id" : MaxKey }, "NA" ) sh.addtagrange( "percona.events", { "location" : "EU", "_id" : MinKey }, { "location" : "EU", "_id" : MaxKey }, "EU" )

127 Geo Distributed Shard Same idea can be used to split by application/hardware resources or any other important parameter for the company. The shard key must have the key used to distribute data. If the value in the key doesn't match any parameter the write will be to the default shard.

128 MongoS

129 MongoS Router process that makes the application talk to the shards in a transparent way. It reads the metadata from the mongoconfig and merge results to act like a single instance.

130 MongoS Target queries are queries that includes the shard key and the mongos knows where to query. Scatter gater queries are queries that needs to run in all the shards and then return to the user.

131 Querying mongos

132 Backup

133 Backup Shards are a bit more complicated to backup than a replicaset. The shards are not linked each other and we have no guarantee the backups will end exactly at the same time. A point in time backup includes the config database

134 Consistent Backup

135 Extra tools

136 mongotop Mongotop will demonstrate the avg operations time in the database per instance. This is important to mention we need to investigate the

137 mongostat

138 mongoreplay

139 PMM

140 Wrapping UP

141 Wrapping up Don't use a single instance in production Do not write to the shard directly

142 Questions