The Backup and Recovery Guide for Cassandra

Transcription

1 The Backup and Recovery Guide for Cassandra

2 EBOOK The Backup and Recovery Guide for Cassandra Introduction In today s era of big data and cloud-native environments, enterprise applications ingest and process vast amounts of information from various sources, often in real time, to deliver critical transmissions, contextual information or business insights. This data has weight. It can influence whether fleets of airplanes are grounded at gates all across the country. It can restrict customers from accessing funds from online banking. It can dictate whether millions of online retail dollars go unspent. The reliance on accurate data, and its grip on lives and business, is growing at an exponential rate like everwidening rings in a lake. Learning Objectives This ebook introduces you to backup and recovery for Cassandra, what such a solution looks like, what a journey of companies that have adopted a backup and recovery solution looks like, and how you can apply cloud data backup to improve the quality and performance of your software, and, in doing so, drive business outcomes that deliver a positive impact to the bottom line and help your company stay ahead of the competition. To support all of this data, new applications are developed and deployed on scale-out, non-relational databases, such as Apache Cassandra, MongoDB, Amazon DynamoDB or ported from traditional relational databases, such as Oracle, Microsoft SQL Server, and others. But, corruptions and operational errors (such as dropped table) by developers or database administrators (DBAs) occur in any and all enterprise IT environments. Enterprises know this, and billions have been invested in traditional data protection products to protect against such err moments. And as enterprises continue to adopt cloud applications and migrate to multi-cloud application environments, they must also be able to manage and recover at scale without data loss or corruption and with minimal application downtime. However, the irony is that data protection products, like those that exist for traditional relational databases) do not exist for today s modern, scale-out and non-relational databases supplying the new types of data that increasingly control our lives. In other words, enterprises today face a significant gap in addressing their data protection requirements. Their most valuable data, often needed 24/7 and without fail, runs absent any point-in-time backup options for that Houston, we have a problem moment. Most studies suggest that more than half of data downtime and loss is triggered by human error, corruption or virus attacks. And yes, while native database replication provides protection from hardware failures or even natural disasters, it is insufficient to meet nextgen enterprise data protection demands. Case in point: if a minor schema corruption impacts the primary data copy, it causes a multitudinous ripple effect that impacts all replicas (across the nodes of the database) down the line. Yet, even without protection, industries continue to adopt and deploy such scale-out database systems at an accelerated pace. EBOOK 2

3 Top 10 Reasons to Use Cassandra The Apache Cassandra database is the right choice of database if you are looking for scalability and high availability without compromising performance for your mission-critical applications. Additionally, Cassandra s support for replicating across multiple datacenters is best-in-class, providing lower latency for users and the peace of mind of knowing that you can survive regional outages. To that end, here s a list of the Top 10 Reasons why enterprises onboarding and deploying missioncritical applications should use Cassandra (Apache Cassandra, DataStax Enterprise). 2. Can Handle Massive Datasets If there were any questions about whether or not Cassandra is capable of handling large data sets, there is no need to look any further than the companies using it. They operate at massive scale. Netflix, Hulu, Instagram, ebay, Apple, and Spotify all have Cassandra working in interesting ways as part of their offerings. The other way you know Cassandra is up to the challenge is in use case examples. Many organizations use Cassandra for applications where data grows in an unbounded way very quickly. These include: Twitter clones, a web log analytics data warehouse, and telemetry or sensor data. 1. Easily Deals with Data Velocity, Data Variety, and Data Complexity Issues 3. Homogeneous Environment Many of the challenges associated with next-generation cloud applications center around data volume and data velocity. Is Cassandra able to handle the speed of data coming into the system? The answer is yes based on the amount of data and cluster size. Not only does Cassandra come with this ability out of the box, but there are systems of data pipeline architectures being built around ingestion speed. And to top it off, Cassandra scales linearly, making it easy to determine the right amount of capacity based on data flow. But, there are also two often overlooked components that Cassandra provides: 1) Data variety and 2) Data complexity. Data variety is an alternate way of saying that data coming into one database can come in different forms. An example of this would be sensor input from a heart monitor and sensor input from an IV both for the same patient. The second component, data complexity, extends the previous example. The heart monitor might report 100 metrics twice per second under normal operating circumstances and up to 125 metrics once per second while the wearer is sleeping. This means the write patterns, locations, and frequencies can vary. Cassandra handles these situations gracefully. Unlike some of the legacy distributed systems, Cassandra does not require outside support for synchronization. All of the required components for basic operation are built directly into Cassandra. Since Cassandra also operates in a peer-to-peer fashion, this means that there is no master-slave or sharding setup and that all nodes in the ring are equal. Additionally, there is only one machine type that an administrator needs to worry about. 4. Highly Fault Tolerant Cassandra employs many mechanisms for fault tolerance. Since Cassandra is masterless, there is no single point of failure. There is also the potential for zero downtime rolling upgrades. This is because Cassandra can support the temporary loss of multiple nodes (depending on cluster size) with negligible impact to the overall performance of the cluster. The safety net Cassandra offers extends outside of your datacenter as well. Cassandra allows you to replicate your data to other data centers and keep multiple copies in multiple locations. This helps satisfy many regulatory requirements in addition to being a part of a strong disaster recovery and business continuity plan. EBOOK 3

4 5. Proven Success Across Enterprise Applications and in Many Use Cases Already There are already many examples where Cassandra is being used effectively. Banks and other financial institutions are storing large quantities of financial data in Cassandra. Analytics companies are using Cassandra to store web analytics data. Medical companies are using Cassandra to store sensor data and other time series inputs. There are also many companies making use of Cassandra for storing IoT data. 6. Ease of Administration Cassandra is a straightforward system to administer. With Cassandra being a masterless system, all nodes in the ring are the same; a homogenous system. It s fault tolerant and can support the temporary loss of nodes with minimal impact to production performance. This means that nodes are easy to replace and the requirement to replace downed nodes immediately isn t as strict. 7. Custom Tuning There are a lot of knobs and levers that can be turned to get Cassandra to perform optimally for your workload and environment. You can setup Cassandra to operate in a way that is consistent with your workload. For example, if you write lots of log data and read infrequently, then there are configuration tweaks to be made for write heavy systems. If you write heavily to one data center and then do all your reading from another data center, then you can adjust the settings on a data center by data center basis. This idea of tuning isn t just available at the Cassandra application level, you can also tune the JVM and Java settings. This includes things like GC and logging levels. Changes can even be made by the drivers at connection time to aid in the performance of your system. 8. Core Applications A lot of work has been done on data manipulation and parsing systems to integrate with Cassandra. For instance, the full text search engine Apache Solr has been packaged to work with Cassandra to provide full featured search capabilities to an existing Cassandra database. Apache Spark, a big data analytics engine, also has been plugged in to work on an existing Cassandra database. There are entire suites of tools that can be integrated or bolted on to Cassandra to increase its capabilities. These include things like Apache Mahout, Kafka, and Zipkin just to name a few. This is important because the more tools you have available to you, the more powerful your data becomes. You also have the ability to gain more insight about your data without having to build and maintain the application systems that were previously required. 9. Excellent Monitoring Options Included in the system of tools referenced in #8 are monitoring packages. If you are a user of automated monitoring platforms like Datadog or Netuitive, you ll find examples of prepackaged agents to monitor the important parts of Cassandra. You can then tack on your own additions of other metrics that are important to you. This is made possible by Cassandra taking advantage of Java MBeans and exposing them to the client. You can use these to get at much of the internal information Cassandra uses to make its own decisions and decide on it s own health. Datastax also offers their own monitoring and control application called Opscenter. 10. Amazing Community One of the best things about any piece of software is having a great community of developers and experts available to you for help or guidance. There is a huge yearly database summit put on by Datastax, the primary backer and largest contributor of code to Cassandra. They also sponsor community events all over the world so you can meet and interact with other Cassandra developers around you. EBOOK 4

5 There are many reasons that Cassandra could be the right tool for your application. Knowing your system s requirements, workloads, and future will help you make the right choice 11 Pitfalls of Native Cassandra Backup Tools Cassandra (Apache OSS version, DataStax Enterprise Edition / DSE) is a NoSQL non-relational database that is becoming increasingly popular with the emergence of enterprise use cases such as customer 360, IoT, and personalization. One of the things to keep in mind is that you need an enterprise-grade backup and recovery solution that can effectively and efficiently protect data at scale. Although native backup tools exist, here are 11 pitfalls to avoid. 1. Level Snapshots Are Not Equal to Database Backup The method of backup in native solutions is node-bynode snapshots which are essentially local snapshot backups (i.e. hand coded scripts). This is the most rudimentary solution which means it is not scalable and is error prone. Above all, this solution does not provide a point-in-time backup so you cannot recover in the event of catastrophic data loss. 2. Increased Backup Storage Costs In a native backup solution, all replica copies are kept in backup storage. These backups are stored in the Cassandra nodes themselves, as well as in optional secondary storage. In addition, there is no special handling of compacted SSTables which means that newly generated compacted SSTtables are backed up, in addition to the original SSTables where the new stables originated from. All of this translates into increased backup storage costs. EBOOK 5

6 3. Longer Recovery Times with Repairs Upon Recovery Scripted solutions require a huge amount of manual effort to restore the data, and node-by-node restore from backup storage increases recovery time and network traffic. In addition, you need to restore all replicas during recovery which increases the time it takes for restore. But, it isn t just about the length of time. It is also the hidden cost of restore. After restoring the data, DBAs need to run cluster-wide repairs to bring the cluster to a consistent state. result in data loss or a large amount of inconsistency in a backed up data set. This is a significant limitation in any large scale production environment where nodes may fail often. In fact, you need your backup solution to perform most when failures occur. 7. Lack of Support of Time-to-Live Handling There is no ability to adjust TTL during restores. Hence, if TTL is already expired during recovery, restored data is automatically expired by Cassandra. 4. Lack of Any Point-in-Time Recovery (APIT) 8. Limited Support for Backup Storage Targets Enterprises frequently need to refresh their test and development clusters with the latest production data to enable continuous integration and continuous development. However, these clusters have different topologies (number of nodes) than production database clusters. It takes hours, if not days to refresh each cluster using native solutions leading to loss of developer productivity. Native tools are limited to choosing local file system or Amazon S3 as backup storage targets. There is no option to store backups to other S3 compatible object storage providers. Furthermore, there is no option to store backups to Google Cloud Storage or object storage targets for on-premise deployments. 9. Limited Protection Granularity Level 5. No Data Masking Option During Recovery Native tools do not give you the option to mask out certain columns during recovery of confidential data, such as personally identifiable information (PII) data. This has large implications for enterprises that handle sensitive data, including name, address, phone number, and social security number. In native solutions, only keyspace-level backup is available. There is no flexibility to back up using columnfamily level. This means all column families in a keyspace will be backed up using the same policy (backup frequency and retention). Additionally, column families that are not needed, but in the same keyspace will be backed up as well. 10. Lower Performance 6. Lack of Failure Handling Support During Backup / Recovery Operations Native solutions use sequential data streams for both backup and recovery as opposed to using parallel and distributed data movement. In a native solution, if a source node fails during backup operations, the backups for that node stop. This may EBOOK 6

7 11. Limited Data Management Use Cases Native solutions do not give you the ability to restore to a different cluster with a different topology. What this means is that you cannot use the same cluster for restore to QA/Dev/Test clusters of different topology/capacity. DSE OpsCenter 6.0 Backup Service Although native backup tools exist, here are key points to keep in mind as you look at those backup methods: Key Shortcomings: DSE OpsCenter 6.0 is only compatible with DSE clusters REST API or GUI (OpsCenter UI) Full data protection, but Point in Time restores are only supported if cluster topology is unchanged from when the backup was taken. Attempts to restore to different topology will result in failure. Commit log backups are also setup separately from snapshot backups. This feature is only available on DSE 4.6 or newer. Backup snapshots (SSTable files) are stored locally on each node, can specify additional locations such as Amazon S3 and custom location on local file system. There must be enough free disk space on each node to accommodate backups. It isn t wise to put backups on local disk. It would be best if each node use some sort of network storage mounted locally (like NFS) to avoid lost backup data in case of lost/failed node. If you do this on AWS, this means you have to setup an NFS server using EBS storage. You will pay for EBS + S3 storage. All SSTables for a particular node and table will only be stored once in S3 to optimize storage space. They didn t mention the word dedupe, however. Can enable/disable S3 server-side encryption from UI. Backup granularity per keyspace, selected multiple keyspaces, or all keyspaces in cluster. No mention of individual/selected column family backups, but admin can restore keyspace(s) and selected column families. Stores backups in native SSTable format. no mention of semantic dedupe. Therefore, storagen requirements for on-premise and cloud deployments will be greater compared to ours. Backups run automatically according to schedule or adhoc backup (create new backup policy, job runs only once). Detailed backup and recovery reports. They didn t mention specific out of the can backup reports that are included, but at least it s there. Backup retention can be setup for minutes, hours, days, and weeks. There s also Retain All to keep backups indefinitely. Native backup data encryption at rest. Can throttle backup data transfers. Can compress backup data before transmission. Compression is done before storing data. Enabling compression will impact performance on nodes since this will require CPU resources. Alerts and notifications. Restores are done via SSTableloader utility. Restore status UI is intuitive. Lists name of keyspace/ column family, completion graph, size and speed of restore. Restore throttling is also supported. No mention whether compaction and/or repair are needed steps after restore completes. EBOOK 7

8 Case Study Our Customer Achieved: ~ 67% cost reduction in secondary storage while using low cost S3 object storage With the cloud storage savings and operational ease that Datos IO RecoverX provides us, our DevOps teams are able to focus on developing IoT platform software that better serves our customers. Adrian Caceres, co-founder and CTO Ayla Networks Customer Background Key Challenges Scalability Failure Resiliency Storage Costs Our customer, Ayla Networks, is the leading Internet-of-Things (IoT) platform company that provides a number of products that enable customers to quickly connect any device, to the cloud and applications, and make meaningful decisions from their data. Application Environment Ayla Networks uses Apache Cassandra database (DataStax Enterprise) for their IoT software-as-a-service application. Consumer appliance manufacturers connect to their IoT SaaS application that is deployed in AWS Cloud over the internet and store appliance specific time-series data. Why Datos IO: Fast and reliable recovery (low RTO) Easy operational adoption Cloud-scale software only Secondary storage efficiency The Challenge Ayla Networks initially implemented a scripted snapshot-based backup and recovery solution but soon realized the cons: 1) Lack of scalability, 2) Lack of failure resiliency, and 3) Storage inefficiency, that prompted them to look elsewhere. Using a scripted solution, our customer found that the solution did not work effectively when they experienced node failures in the Cassandra database environment. Moreover, they had to spend significant manual effort and IT resources to maintain and evolve the scripted solution as their Cassandra cluster scaled according to the application needs. EBOOK 8

9 Given the critical nature of our customer-facing data, a scalable, next-generation, cloud-native backup and recovery solution is a Adrian Caceres, co-founder and CTO Ayla Networks requirement. How Did Datos IO Help? Ayla Networks deployed the 3-node RecoverX cluster across multiple Availability Zones (AZ) in Amazon AWS cloud for highavailability. The native failure handling capabilities of RecoverX provided them with the ability to backup their database under the scenario where one of the database nodes could fail. From a secondary storage perspective, this customer wanted to use cloud-native AWS S3 storage and minimize the storage costs. RecoverX allowed them to reduce their secondary storage cost by ~67% while using low-cost S3 object storage. Deployment diagram: Cassandra Cluster 2 Cassandra Cluster S3 Bucket Storage Parallel Data Transfer Parallel Data Transfer Consistency & Deduplication Control Plane Control Plane Environment details: Cassandra Cluster 1 Cassandra Cluster 2 Database Version DSE 4.8 DSE 4.8 Deployment Amazon AWS Amazon AWS Cluster Size 6-node, single DC 12-node, single DC Database Size 200GB 3TB Number of Tables ~20 ~30 Secondary Storage Type Amazon AWS S3 Amazon AWS S3 Workload Insert-heavy Insert-heavy EBOOK 9

10 Case Study Our Customer Achieved: The Datos IO team listens. Over the last 9 months, we have given them a number of product features, and the team delivers at a record pace. The Customer 1 Hour Using RecoverX, our customer was able to backup multiple Cassandra databases and recover from data loss with an SLA of 1 hour. In addition, our customer increased their compute and memory resources during peak seasons for seasonality, and was able to get the required backup performance. Our customer is a Fortune 100 home improvement retailer with thousands of brick and mortar stores in the U.S., Canada and Mexico. In addition, they have an ecommerce application that is central to their digital transformation journey. Their hyper-scale application is built with a cloud-first methodology that is deployed natively on Google Cloud Platform (GCP). The Problem Our customer s challenges can be summarized in three broad categories. The first is Backup and Recovery for Cassandra in Google Cloud. The company is in the midst of their digital transformation journey and as their online business grew, they experienced challenges in scaling their existing relational databases. They re-architected their online business application using a microservices based cloud native application architecture. This new application is deployed in Google Cloud Platform, and uses underlying Cassandra (DataStax) databases to populate information for different customer-facing platforms (cart order, inventory management, etc.). Given that their core online business is based on this web-scale ecommerce application, any data loss is detrimental for their business. Our customer requires the ability to backup multiple Cassandra databases and recover from any data loss with an SLA of 1 hour. The second category is Backup Software Elasticity. Being in the e-commerce industry, our customer experiences huge spikes in data volume during the holidays, especially Thanksgiving and Christmas. They could not rely on a static infrastructure footprint, and instead, wanted to scale resources only during peak seasons, and reduce the resources back after the season ended. This would ultimately allow them to optimize the cost of operating a data protection solution, without sacrificing highavailability. The third category is Test Cluster Refresh Using Production Data. Our customer has multiple production and test clusters, and they implement continuous integration and continuous development (CI/CD) methodologies. Initially, they had to spend a considerable amount of time refreshing their test clusters because they are of different topologies. They needed a data management solution that would automate the refresh of test clusters using production data at regular intervals. EBOOK 10

11 Key Challenges Cloud-Native Backups of Cassandra (DataStax Enterprise) Data protection SLA of 1 hour and minimize application downtime Automate refresh of test clusters using production data at regular intervals Why Datos IO: The only cloud-native data protection solution for Apache Cassandra and DataStax Enterprise in Google Cloud Cluster consistent backups that don t require repairs on recovery, leading to low RTO 80% reduction in secondary storage costs High-performance to guarantee 1-hour SLA using elastic compute/storage resources The Solution Initially, our customer s DevOps team used database native tools, but could not achieve the enterprise level data protection features that they needed. To quantify the impact of this any outage, it is estimated to be hundreds of thousands of dollars in lost business. Datos IO RecoverX was deployed by customer natively in Google Cloud environment. RecoverX software was deployed in a clustered configuration to achieve high availability. A single RecoverX cluster was deployed to protect as many as 6 Cassandra clusters with a 1-hour backup interval. The data was stored on Google Cloud Storage for cost effectiveness. The Result Using RecoverX industry-first features, the customer was able to achieve operational and capital cost savings. At the same time, they were able to meet the organizational goal of 1-hour SLA. Semantic deduplication resulted in 82% storage cost savings. Further, customer optimized its infrastructure costs by allocating appropriate compute resources depending on their application requirements. About Datos IO Datos IO is the application-centric data management company for the multi-cloud world. Our flagship Datos IO RecoverX delivers a radically novel approach to data management helping organizations embrace the cloud with confidence by delivering solutions that protect, mobilize, and monetize their data at scale. Datos IO was recently awarded Product of the Year by Storage Magazine, and was recognized by Gartner in the 2016 Hype Cycle for Storage Technologies. Backed by Lightspeed Venture Partners and True Ventures, Datos IO is headquartered in San Jose, California. EBOOK 11