The Definitive Guide to MongoDB Backup and Recovery

Transcription

1 The Definitive Guide to MongoDB Backup and Recovery

2 EBOOK The Definitive Guide to MongoDB Backup and Recovery Table of Contents Executive Summary 1. MongoDB Technology Overview 2. The Need for Backup & Recovery for MongoDB 3. Backup & Recovery Requirements for MongoDB 4. Existing Practices for Backup & Recovery 5. Rubrik Datos IO RecoverX Overview 6. Compare and Contrast MongoDB Backup & Recovery Solutions 7. Maxwell Health Customer Success Story Executive Summary NoSQL databases, led by MongoDB, have disrupted the traditional database market and are quickly becoming the database platform of choice for enterprises everywhere deploying new, modern cloud applications (such as analytics, IoT, SaaS, and others). However, the rapid proliferation of modern applications built on NoSQL databases has created a critical data protection gap. This has exposed organizations to data loss, ransomware attacks, and has made them unable to address compliance requirements, especially with the upcoming GDPR regulations. This ebook provides insight to why NoSQL databases have become widely adopted, the requirement for backup and recovery for NoSQL databases, and recommendations on how best to address this data protection challenge. Why NoSQL Databases Traditional relational databases are widely deployed in enterprises running many business-critical applications. But why the rapid proliferation of NoSQL? The simple answer is the cloud and its predecessor, the internet. The genesis of NoSQL was early internet companies including EBOOK 2

3 Facebook, Google, and Amazon who did not operate in the traditional datacenter mindset, but instead needed to write and read data anywhere in the world, and were building a new generation of applications with massive data sets that scaled to millions of users. This required a fundamentally different database architecture one that is rooted in high availability, built for application scale, built for geo-distributed architectures, and most importantly one that is developer friendly. In simple terms, NoSQL databases are different than relational databases in several key ways: Schema vs. Schema-less: Relational databases require a schema (data layout) to be defined before you can add any data. What you may gain in performance, you lose in flexibility. In contrast, NoSQL databases don t require you to define the schema first providing a high level of flexibility as data requirements change over time. Structured vs. Unstructured Data: Most data being created today is unstructured, creating a challenge for relational databases which were architected for data that was mostly structured and clearly defined by their relationships. NoSQL databases on the other hand, are designed with unstructured data in mind. Examples of unstructured data includes social media, videos, , IoT, and documents. Scale-Up vs. Scale-Out: NoSQL databases are designed to scale performance and capacity very cost effectively over inexpensive, commodity servers, while relational databases are typically hosted in a single server. When you need more capacity or performance, you need to buy an expensive, bigger, faster server. Cost of Deployment: Because they are mostly open source, NoSQL databases provide a low barrier to entry enabling organizations to start projects with only a nominal investment. This is unlike relational databases which are a) proprietary, and b) require expensive licensing fees. Part I: MongoDB Technology Overview Given the significant cost and architectural benefits of NoSQL databases, it is easy to understand why so many organizations are adopting them, and MongoDB is the most downloaded NoSQL database (reference: Let s look at what makes MongoDB unique. MongoDB is an open-source, non-relational NoSQL database. It is a cross-platform, document oriented database that provides high performance and availability, and easy scalability. MongoDB uses a concept of collection tables and documents. A collection is a group of MongoDB documents that exists within a single database but does not enforce a single schema. A document is a set of key-value pairs with a dynamic schema that does not need to have the same set of fields or structure, and common fields in a collection s documents may hold different types of data. Since MongoDB is schema-less, its document databases can have any number of fields, content, and size that differ from one document to another. MongoDB provides easy scaling through replica sets and auto-sharding, tunable consistency for reads, and built-in replication. EBOOK 3

4 In summary, MongoDB is ideal at addressing the following development requirements: High-Volume Data with Little Structure Because MongoDB is flexible and document-based, you can store large data sets of different data types in one place without having to define data types in advance. Additionally, sharding and replication let you scale read and write traffic across servers to optimize performance. Hybrid Cloud Deployment MongoDB s built-in sharding makes it easy to spread data across multiple servers spanning data-centers or public clouds. Agile Development Because MongoDB offers dynamic schemas, organizations can test and iterate quickly. Cost-Effective Scalability Like most NoSQL databases, MongoDB is architected to scale performance and capacity across low-cost hardware on-site or in the cloud. Based upon these core attributes, popular enterprise use-cases for MongoDB include: Single View Real-time views of your business with aggregate data from multiple different data sources. Internet of Things (IoT) Organizations leverage MongoDB to store and analyze the massive quantities of data generated by connected machines and sensors. Personalization These applications focus on creating customized online experiences for customers based upon real-time analysis of demographic and behavioral data. Catalogs MongoDB s schema-less architecture is ideal for catalog applications which typically contain large quantities of unstructured data and change frequently. Content Management Content management applications are a bedrock of ecommerce, storing and providing content to support everything from websites, to online publications, to archives. EBOOK 4

5 Part II: The Need for Backup & Recovery for MongoDB As organizations increasingly rely upon NoSQL databases for their business-critical applications, and as these same organizations adopt a cloud-first strategy, the need for a comprehensive backup and recovery strategy has never been more critical. In this section, we will explore the need for a complete data protection strategy. One of the advantages of MongoDB is that it includes native replication. Put simply, replication creates an alternative copy of an original and changes with the original in real time. So, while native replication is good for protecting against media and network failures by replicating data across nodes, it can be a serious detriment in the case of data corruption or loss whereby the corrupted data set gets replicated, exacerbating an already bad situation. Furthermore, the most common cause for data corruption and loss today is human error including operator error, fat finger mistakes, and accidental deletions. Recently, more malicious cyber threats have propagated including malware and ransomware. To protect against these threats, a comprehensive backup and recovery strategy is required. Unlike replication, backup is a copy of data taken at a certain point in time that does not change with time. A backup copy enables an administrator to restore data at a certain time in the past, typically a time before the data loss or ransomware attack occurred. In summary, backup plays a critical role in an organization s overall data protection strategy. In the next section, we ll discuss in more detail the technical requirements for backup and recovery of MongoDB databases. Part III: Backup & Recovery Requirements for MongoDB This section will introduce the key requirements for protecting data that resides on MongoDB, deployed either onpremise, or on private cloud with as-a-service model, or in public cloud with Amazon Web Services (AWS), Google Cloud Platform (GCP), Pivotal Cloud Foundry (PCF), or others. Requirement #1: Online Application-Consistent Backups One of the key requirements of next-generation cloud applications that are deployed on MongoDB database is the always-on nature. This means that quiescing the database for taking backups is not feasible and moreover, the backup operation should not impact the performance of the application. As the application scales, the underlying MongoDB also needs to scale-out to multiple shards and replica sets. In this case, a backup solution must provide a consistent backup copy across shards and replicas sets without disrupting database and application performance during backup operations. EBOOK 5

6 Requirement #2: Granular Backups (Collection Level) Depending on the application, data may have different change rate and patterns. For example, in a product catalog, certain items may be refreshed everyday (fast selling goods), while others may have longer shelf life (premium items). Based on the application requirements, some collections may need to be backed up every hour versus the others that may be backed up daily. Providing this flexibility to schedule backups at any interval and at collection level granularity is another requirement that we have heard from customers who are using MongoDB. More importantly, these backups should always be stored on the secondary storage in native formats to avoid vendor lock-in. Requirement #3: Orchestrated Recovery to Alternate Topologies During its lifecycle, data resides in multiple stages such as development, test, pre-production and production, and may also reside in multiple clouds (private cloud and public cloud). The topology of MongoDB clusters at each stage is different. For production, the application could be deployed on a sharded MongoDB cluster on-premise but the test team might only have access to unsharded MongoDB clusters in Amazon AWS (public cloud). Hence, the backup solution should allow multiple restore operations such as sharded to sharded (such as from 5 3 cluster to 2 3 sharded cluster) or sharded to unsharded (such as 5 3 cluster to 1 3 unsharded) across such cloud configuration. Likewise, the backup solution should allow restore operations of geographically distributed clusters. Requirement #4: Handling Node Failures Failures are a norm in the distributed database world. However, the backup solution should be resilient to database process failures, node failures, network failure and even logical corruption of data during backup and recovery operations. Finally, the backup solution should be able to handle failures of MongoDB config servers that store metadata for sharded clusters. EBOOK 6

7 Part IV: Existing Practices for Backup & Recovery of MongoDB 1. Traditional Backup Products Don t Work The implications for traditional backup and recovery products are profound. With an average architecture age of 20 years, legacy backup and recovery products from vendors including Veritas, Dell EMC, and CommVault were all designed before the advent of the cloud and modern applications, and are not architected to support the new, modern IT stack. They were architected for traditional IT applications deployed on traditional scale-up and static resources of compute and storage. They were designed for scale-up application environments with uncompressed workloads built to be operated within walls of on-premises infrastructure. Media server based legacy backup architecture has no place in the cloud. Likewise, legacy media server based architectures have no place protecting next generation scale-out data. Legacy backup solutions from long established backup vendors (e.g., Veritas, EMC/Legato, CommVault), are all based on the same legacy architecture: single vendor, end-to-end architectures which centralize the control of the backup process (the control plane ) and moving/storing the backed-up data (the data plane ). In these legacy backup solutions media servers act as the consolidated control plane and data plane. Each backup vendor s unique client agent software, installed on each host, used database specific APIs to invoke and process backups of structured data (e.g. Oracle RMAN). Likewise, client agents installed on each file server processed backing up unstructured files on each file server or used APIs on each NAS (an appliance based dedicated file server). For backing up data in these scenarios the agent software identified the data to be backed-up, under control of backup administration issued from a media server (control plane). And then moved the backed-up data (data plane) to an assigned media server, which stored the backup in proprietary format on some kind of dedicated storage (Tape, Disk, Purpose Built Backup Appliance storage). Specific media servers backed up specific hosts or clients. Adding more database hosts or file servers eventually required adding more media servers. The inevitable result of this architecture was silos of groups of hosts and a specific media server protecting them. 2. Manual Scripted Solutions Manual solutions leverage native MongoDB utilities and scripts to transfer data to secondary storage. The scripts (via mongodump) are customized for each MongoDB cluster and require significant operational effort to scale or adapt to any topology changes (such as addition or removal of nodes to your MongoDB database). Further, these scripts are not resilient to failure scenarios e.g., failure of a node (primary or secondary) or intermittent network issues. Finally, recovery is a manual process, hence, time consuming and results in very high application downtime and contains data loss risk due to any bugs in the scripts. Overall, these solutions work when the MongoDB environment is small and some data loss may be permitted in the application. EBOOK 7

8 Top Issues of Scripted Solutions: 1. Lack of enterprise backup solution for sharded configurations 2. Database needs to be offline when snapshots are taken 3. Both backup and recovery fail under node failure and other infrastructure failures 4. Recovery process is manual and requires verifications, which increases recovery time 5. Recovery at collection-level requires manual recovery that is time consuming 6. Recovery to unlike topologies (sharded to unsharded) for test/dev refresh is not available 3. MongoDB Backup and Recovery Options (MongoDB Cloud Manager and MongoDB Ops Manager) Good news all around is that MongoDB understands that backups are critical for enterprises and itself provides a couple of native methods to backup MongoDB databases. One choice is a managed backup offering, MongoDB Cloud Manager, that runs in a public cloud. The other, for licensed MongoDB Enterprise customers, is to deploy MongoDB Ops Manager as a dedicated solution locally to the MongoDB database clusters. In addition to being exorbitantly costly, the managed backup service stores customers data in a public cloud. While this may be a suitable option for companies who deploy cloud native applications on public cloud such as AWS, however given that the backups have to be stored on EBS storage. Backup data transfer over Wide Area Network (WAN) may not work for customers who deploy MongoDB based applications on-premise and for customers who need to keep their sensitive data in-house. Further, there are significant data limitations per shard to use this service. MongoDB Ops Manager is complex to deploy and operationalize. For Ops Manager, enterprises need to deploy multiple servers, an additional replica of each database protected and about ~6-9x storage capacity of the databases protected. Either of the 2 solutions brings significant complexity and require CAPEX and OPEX investments: 1. Complexity of deployment and operation 2. Cost of additional infrastructure to deploy and scale (servers and primary storage) 3. Orchestrated recovery from sharded to unsharded environments, and vice versa 4. Significant Monthly cost, per GB protected, for Cloud-based solution 5. Siloed backup infrastructure for only MongoDB database EBOOK 8

9 Realizing backup and recovery requirements of enterprise customers, the emerging era of NoSQL distributed databases, and the limitations of the solutions described above, Rubrik Datos IO has built the industry-first, scale-out data protection software product for next-generation applications deployed on distributed and cloud databases such as MongoDB and Apache Cassandra (DataStax). The Rubrik Datos IO solution is built from the ground-up for next-generation applications, caters to the needs of application owners and DevOps, and takes away the operational hassles of deploying and managing protection infrastructure. Most importantly, it is a reliable and scalable solution to use even in scenarios of node failures which leads to optimal performance through minimized recovery time (RTO). Part V: Rubrik Datos IO RecoverX Overview Rubrik Datos IO RecoverX is application-centric, data management solution providing backup and recovery for NoSQL, Non-Relational enterprise database systems. RecoverX delivers scalable versioning, single-click recovery, and industry-first semantic de-duplication, reducing redundant data and lowering storage costs. RecoverX includes any point-in-time backup and orchestrated recovery to safeguard against logical and human errors, malicious data corruption, application schema corruption, and other soft errors. Organizations can protect their data at any granularity and any point in time (flexible RPOs) to reduce application downtime with recovery in minutes (low RTOs), save up to 80% on secondary storage costs, and increase productivity of applications and DevOps teams. RecoverX enables continuous integration and development by fully automating the refresh of test and development environments using production data. EBOOK 9

10 Advanced Features Advanced Recovery: The amount of data organizations need to protect is rapidly increasing but customers are constantly demanding faster RTOs to meet their business agility requirements and to address mandatory IT requirements such as those required by GDPR. These same organizations are typically building customer-centric applications that span hundreds of tables, with hundreds of TB s of data (aka Big Data ), with change rates approaching 300%, yet they expect recovery times of just a few hours. To address this new class of recovery requirements, RecoverX 2.5 now delivers advanced features including queryable recovery enabling fast, sub-table level recovery, as well as incremental and streaming recovery, both of which reduce recovery times down to minutes and minimize storage requirements for large scale restore operations. Backup Anywhere, Recover Anywhere: Cyber-attacks like ransomware present a real threat and organizations need to protect their data across data sources, applications, and any geographical boundary. Increasingly, customers have database tables spanning multiple data centers and multiple continents, but they want their data to be backed up and restored at the granularity of a single data center location. RecoverX 2.5 now delivers on this requirement with support for local backup of geo-distributed applications enabling local data center recovery for faster and more flexible RTO. Enterprise-Grade Security: Rubrik Datos IO is experiencing massive adoption in large enterprise customers across retail, ecommerce, financial services and healthcare organizations all of which have significant security requirements. RecoverX 2.5 introduces support for TLS/SSL encryption, support for X.509 certificates, LDAP authorization and Kerberos authentication to deliver optimized security between RecoverX and data sources and integration with enterprise security management tools. Benefits RecoverX provides organizations with the following benefits: Minimize application downtime through application consistent point-in-time backups by removing database repairs post recovery operations Reduce secondary storage costs by ~80% using industry-first global semantic de-duplication Failure resiliency and elastic performance via highly-available, software-only product Operational efficiency through fully orchestrated recovery Lightweight deployment in public cloud or private datacenter EBOOK 10

11 Part VI: Compare and Contrast: How RecoverX Stacks up for MongoDB Backup & Recovery How Rubrik Datos IO Stacks up for MongoDB Backup and Recovery Native Tool Mongodump Native Managed Backup - Ops Manager Rubrik Datos IO RecoverX Customer Value & Benefits Backup Consistency Not Consistent Consistent Consistent versioning at any interval and granularity Enterprise-grade backup and recovery Infrastructure Cost and Complexity Implementation Dependent Very High: Multiple Servers + Dedicated Storage + Protection Storage Single Rubrik Datos IO Software Server 8X lower TCO - savings of 60% to 70% Recovery Options Manual and Risk Prone Limited Options - shard level restore requires original topology, overwrites existing data Enterprise-grade with all combinations such as sharded to unsharded Flexible options to enable test/dev and workload migration uses cases Failure Handling None Limited Yes, resilient to database and node failures (including primary) Resiliency and reduced enterprise risk Secondary Storage Savings None Variable, depending on choice of backup snapshot storage Yes, semantic de-duplication for all backup version storage choices Replica data is always deduplicated Support for Multiple Databases Platforms None None Yes, Apache Cassandra, DataStax Enterprise and MongoDB Multi-platform enterprise-grade data protection software Data Source Scaling No options to scale Difficult to scale - requires adding multiple resources - backup servers, local storage, backup snapshot storage Adapts to changes in cluster topology and cluster size (support for source topology reconfiguration) Granular scaling. No dedicated backup storage on RecoverX server Increased operational efficiency, lower cost to scale EBOOK 11

12 Part VII: Maxwell Health Case Study The Customer Maxwell Health is an H/R benefits management service that simplifies the process of creating and managing employee health plans, benefits and payroll for small and medium sized businesses. They deliver a cloudbased SaaS platform for their customers that includes a marketplace to make selecting benefits intuitive for employees. The cloud-based SaaS platform runs on Amazon AWS public cloud and uses MongoDB databases for critical data requirements. Maxwell Health Achieved SLA of 4 Hours The Need Maxwell Health s challenges are summarized in two broad categories. The first challenge is backup and recovery of MongoDB in AWS public cloud. Maxwell Health built their online business application using a microservicesbased cloud-native application architecture. The microservices are deployed natively in AWS, and use underlying MongoDB databases to populate information for different customer-facing platforms. Their core online product is based on this web-scale cloud-based application and any data loss is detrimental to their business. Maxwell Health requires the ability to backup multiple MongoDB databases and recover from any data loss with an SLA of 2 hours. Using RecoverX, Maxwell Health was able to replace a homegrown solution of snapshots and scripts with a fully automated backup and recovery solution for multiple MongoDB databases that achieved an SLA of less than 4 hours along with massive operational cost savings, including: 90% reduction in AWS S3 backup storage costs 30% reduction in recovery time versus native MongoDB backup solution 300% ROI in the first 12 months The second challenge is test cluster refresh using production data. Maxwell Health has multiple production and test clusters, and they implement continuous integration and continuous development (CI/CD) methodologies. Initially, they used snapshots and home-grown scripts to address these challenges but they were constantly challenged with high S3 storage costs and the process for refreshing their test clusters daily was manual, cumbersome, and time consuming. They needed a data management solution that could achieve enterprise-level data protection with an SLA of <4 hours while also automating the refresh of test clusters using production data at regular intervals to clusters of different topologies. EBOOK 12

13 The Solution Maxwell deployed Rubrik Datos IO RecoverX to meet their cloud backup and recovery requirements. RecoverX was deployed within 15 minutes using the AMI (Amazon Machine Image) in the cloud as a complete solution. The Results With Rubrik Datos IO, Maxwell Health achieved 90% reduction in AWS S3 backup storage costs, 30% reduction in recovery time versus a native MongoDB backup solution, 300% ROI in the first 12 months, and a guaranteed 4-hour backup SLA, 2-hour recovery SLA. Because RecoverX offers fully orchestrated recovery back to any MongoDB database cluster with any topology, they are also able to restore their production data into test clusters in a fully automated manner. Key Challenges Cloud-native backups of MongoDB deployed on AWS public cloud Reduce spiraling backup storage costs Improve backup and recovery SLAs and minimize application downtime Automate daily backup and refresh/ restore to QA database clusters Why Rubrik Datos IO: The only truly cloud-native backup and recovery solution for MongoDB Fully automated, rapid, repair free recovery in minutes Ability to recover to same or different topology cluster High-performance to guarantee 4-hour SLA API-based architecture and fully featured policy engine allows for ease of day-to-day management About Rubrik Datos IO Rubrik Datos IO is the application-centric data management company for the multi-cloud world. Our flagship Rubrik Datos IO RecoverX delivers a radically novel approach to data management helping organizations embrace the cloud with confidence by delivering solutions that protect, mobilize, and monetize their data at scale. Rubrik Datos IO was recently awarded Product of the Year by Storage Magazine, and was recognized by Gartner in the 2016 and 2017 Hype Cycle for Storage Technologies. Rubrik Datos IO is headquartered in Palo Alto, California. EBOOK 13

14 The Definitive Guide to MongoDB Backup and Recovery Page Mill Rd, Bldg 2, Palo Alto, CA 94304