Become a MongoDB Replica Set Expert in Under 5 Minutes: USING PERCONA SERVER FOR MONGODB IN A FAILOVER ARCHITECTURE This solution brief outlines a way to run a MongoDB replica set for read scaling in production. It also demonstrates advanced features that empower you to use a best practice architecture from day one. Best fit for: Production environments that need automated high availability and recovery Environments that have not yet reached the need for full sharding solutions
Summary With many classic databases, you need to use additional services, third-party software and scripts to facilitate data replication. MongoDB provides the replication (in a form of a replica set) out of the box. This document describes a proven, MongoDB replica set architecture that is built upon Percona Server for MongoDB. It provides an elastic, highlyavailable, fast-to-recover and resilient database layer out of the box. We also describe the best practices underlying the architecture of a replicaset, including the biggest mistake people make. To ensure you understand how each option is used, we have a slightly complicated diagram. However, it shows you where these features fit. Use Case Almost all web and application projects need to somehow persistently store data. Storing data on one single server does not provide redundancy and scalability. Modern web applications are usually read-heavy, so scaling reads are essential for the application performance. MongoDB provides data replication and read scalability out of the box. The Replica Set in MongoDB is a quick an easy way to implement: Data redundancy Automatic failover (little to no application downtime) Read scalability (via the ReadConcern option) Ability to perform rolling upgrades and maintenance with zero downtime PRO Quicker failover means higher application uptime with little to no application downtime, with consistent data across nodes. Failover is transparent to applications and doesn t affect application performance. Allows you to tailor specific MongoDB nodes to specific database functions in order to distribute working load across environment. Scales out to 50 read replicas. CON Limit of seven voting nodes ON-PREMISES OR CLOUD ARCHITECTURE SOLUTION PROPERTIES AVAILABILITY DATA PROTECTION OPERATIONAL COST SIMPLICITY READ SCALABILITY WRITE SCALABILITY BACKUP RECOVERY TIME Chaining replicas can cause lags in data between nodes, which can mean stale reads. You need to set how safe reads should be, which means that reads between nodes in a replica set might not be consistent. Secondary nodes may return stale data due to asynchronous replication on write. No scalability for writes MongoDB always writes to a Primary node. 2
Architecture We will look at two replica sets in action: a Best Practice Basic Replica Set and an Every Feature Configuration Replica Set. The second is very complex, but labeled so you can see how someone might use the setup when running two data centers that need, for example, US and APAC local reads for performance, local backup nodes to improve recovery, a delayed slave in the US and even a preferred pair of nodes that can become Primary. This is explained in more detail in Ultra Low Latency Reads Per Region with MongoDB Replica Set. ARCHITECTURE DIAGRAM In the diagram above: All nodes are data bearing with the same hardware Each node has the same priority, any can be Primary as needed None are Arbiters If any two nodes are up, we maintain read and write capabilities. With two nodes down, an election can t happen so the surviving node cannot be made primary. In this case, we can read data from the surviving node but cannot write any data to the replica set. An Arbiter node can be helpful, but there are some caveats regarding their use. An arbiter node is to decides elections for a new primary in case of a tie from the remaining nodes. If you have an odd number of nodes, an arbiter is not needed as there is always be a sufficient number of surviving nodes to decide an election. As a cost-saving method, some organizations decide to have one of the nodes in a three-node replica set as an Arbiter (to save on disk space and to decrease costs as well as arbiters don t store data and can run in any machine), and this is where problems can arise. If you do not have at least three nodes in the full data set, any work such as building indexes, nodes or backups can have a measurable impact on your system or even worse, can risk your high availability coverage. Since the Arbiter does not have the full data set, it cannot function as a database node in case of a failure of one of the other nodes. You are left with one node that handles all of the work, and imperils backups and high availability. 3
ARCHITECTURE DIAGRAM Components MongoDB Community, MongoDB Enterprise, or Percona Server for MongoDB will all work here with no need for any third-party tooling. The table explains the functions of the nodes in the diagram: NODE P B SETUP All writes are sent to this node in the replica set. S1-S5 could also become Primary nodes, but since S3-S5 have priority:1, and P, S1 and S2 have priority:2, the system prefers to keep the Primary on the west coast (only moving to the east coast in dire need). This non-voting node is dedicated for backups (shown by the gray heartbeat). It doesn t vote or answer queries and is not in the West tag-set. S1/2 Both of these nodes are normal Secondaries that do vote, and are also members of the West tag-set. This means any applications in the West tag-set query them instead of S3-S5. In the event of an election, since this is the primary data center they have a priority of 2. P, S1 and S2 are preferred in order to keep the Primary in the west. S3/4/5 DS A Failover Like S1/S2, these nodes are Secondaries and only become primary if the West data center is down. Additionally, they are in the East tag-set. Reads for East Coast applications go here and not to the West Coast. Writes must still go to the Primary. This is a special node is hidden so it won t take reads, and delayed so that it applies changes a full two hours behind the rest of the cluster. This allows the final data center to be a proper disaster recovery site. This node can never become Primary and it is unable to vote. Finally, we get the Arbiter. With three voting members in the East and West each, we have an even six votes. The Arbiter can break any ties, and does not hold data to ensure that either the West of East always has a Primary if the opposing data center fails. Unlike other databases, MongoDB includes automatic failover and recover. This makes replicas inherently valuable for even beginner users who might not know how to set up manual failover. For comparison, MySQL users have to build and implement failover systems themselves, or turn to third-party tools. MongoDB replication allows for relatively easy set up by having the most difficult aspects already done and automatically managed on your behalf. The connection string must be properly configured, otherwise the application will fail due to no Primary node available. 4
Backups Backups are a complex topic in any database. Like other databases, MongoDB is capable of binary, logical, and snapshot backups. This flexibility allows you to choose the right tool for the job. When it comes to taking a logical backup, you can make it consistent by using the --oplog flag to record any changes occurring while the backup is taking place. When you perform a restore, MongoDB applies those operations and your database is consistent to the point the backup stopped. MongoDB automatically tries to use a read Primary node for a backup when it notices it is part of a replica set for cluster-wide (multiple replica sets) backed-up at the same time. Sharding adds complications but there are still safe methods. Please email us for more information and solutions. Monitoring For query analytics and time-based database performance information, we strongly recommend our Percona Monitoring and Management open source tool. This should be installed on a third host, either by using the Docker container images, Virtual Appliance (OVA) or as is available through the Amazon Marketplace. Dealing with Growth You can expand the entry level environment by adding more nodes for reads, or increasing their resources for writes. Once the write workload grows beyond a single server, we recommend sharding or workload separation (sharding is far easier to implement). Replica sets provide extensive read scaling at a low operational cost. It is simple to deploy an additional node with the same configuration file, and then run rs.add( $newhost:$newport ). MongoDB do the lifting for you. Percona Can Help Managing your organization s database operations, on-premises or in the cloud, requires in-depth knowledge of potential issues plus diligent, dedicated practice. Being aware of the issues above will help protect your organization s data-based applications when setting up a high availability database environment. It will also significantly enhance both performance and scalability to deliver a better user experience. Percona Support services are accessible 24x7x365 online or by phone to ensure that your database installation is running optimally. We can also provide onsite or remote Percona Consulting for current or planned projects, or in emergency situations. Every engagement is unique and we will work with you to create the most effective solution for your business. Percona Managed Services can support your existing database infrastructure whether it is hosted onpremise or at a colocation facility or if you purchase services from a cloud provider or database-as-aservice provider. To learn about how Percona can help you, and for pricing information, please contact us at +1-888-316-9775 (USA), +44 203 608 6727 (Europe), or email us at sales@percona.com. 5