Continuous Data Protection

Transcription

1 Continuous Data Protection

2 Comprehensive protection of data is a critical responsibility of a data warehouse. This includes both protection against unauthorized access and protection against data loss or corruption. Traditional data warehouses provide only a baseline level of protection. For example, data warehouses often provide basic access control mechanisms to restrict data access and an internal data replication factor to maintain additional copies of data within the cluster. However, additional protection beyond this baseline level is left to the user the user needs to evaluate, select, implement, and monitor firewalls, database backup, external data replication, network security, datacenter security, and more. The cost and complexity of these additional measures adds a significant burden to already-stretched IT organizations. Continuous data protection is built into the Snowflake data warehousing service so that every customer can realize enterprise-class data protection by default. DATA SECURITY Snowflake uses a broad set of security features and operational practices to provide end-to-end protection against unauthorized access to customer data, whether by users within a customer account or attackers outside a customer account. Our multilayered approach ensures comprehensive protection that is based on industryrecognized frameworks and validated by third-party SOC 2 certification. Robust authentication. Snowflake supports twofactor authentication for access to the Snowflake service as well as robust password requirements and lockouts after multiple failed login attempts. End-to-end encryption. Snowflake encrypts all communication over the Internet as well as all customer data stored in Snowflake. Encryption in Snowflake is always enabled for all customer data. Role-based access control. Snowflake provides sophisticated role-based access control to ensure that customer data can only be accessed by authorized users in the customer account. SNOWFLAKE S TOTAL DATA PROTECTION: Encryption: data always encrypted, automatically Data protection: data replication to provide 11 9 s of durability Instant recovery: self-service data recovery via SQL Monitoring and alerting. Snowflake uses security applications such as file integrity monitoring tools (FIM) to ensure that critical system files (e.g. system and application executable files, libraries, and configuration files) have not been tampered with. In the event of data modification or deletion by a malicious user, the data protection and recovery capabilities discussed in the next section of this paper provide an easy way to recover data. For more information about security in Snowflake, refer to our Security whitepaper. 3

3 DATA PROTECTION AND RECOVERY Snowflake was designed from the ground up to be a continously available cloud service that is resilient to failures without customer disruption or data loss. Our Continuous Data Protection capabilities protect against and provide easy self-service recovery from accidental errors, system failures, or malicious acts. Recovery from Accidental Errors The most common cause of data loss or corruption in a database is accidental errors made by a system administrator, privileged user, or an automated process. Snowflake provides a feature called Time Travel that provides easy recovery from such errors. Time Travel is a unique Snowflake feature that makes it possible to instantly restore or query any previous version of a table or database from an arbitrary past point in time within a retention period. ACCESS PAST VERSIONS OF DATA How Time Travel Works Time Travel is made possible by Snowflake s implementation of data modification (DML) operations. Snowflake is a fully updatable relational database with a complete set of SQL DML operators that support updates or deletion of rows of data. When any data is modified, Snowflake internally writes those changes to a new storage object and automatically retains the previous storage object for a period of time (the retention period) so that both versions are preserved. When data is deleted or database objects are dropped, Snowflake updates its metadata to reflect that change but keeps the data for the retention period. During the retention period, all data and data objects are fully recoverable by customers. Users granted administrative privileges can undo a DROP of a database, table or schema with a simple SQL command. Past versions of a data object from any point in time within the retention period can also be accessed via SQL, both for direct access by a SELECT statement as well as for cloning in order to create a copy of a past version of the data object. After the retention period has passed, Snowflake waits an additional seven days (the failsafe period) to provide a sufficient length of time during which Snowflake can, at a customer s request, recover any data which was maliciously or inadvertently deleted by human or software error. At the end of that period, an automated process physically deletes the data. Because of this design, it is impossible for the Snowflake service, any Snowflake personnel, or malicious intruders to physically delete data. Continuous Data Protection and Time Travel are standard features built into the Snowflake service. The length of the default retention period is determined by the customer s service agreement. Customers can specify extended retention periods at the time that a new database, table or schema is created via SQL DDL commands. Extended retention periods incur additional storage costs for the time that Snowflake retains the data during the retention and failsafe periods. Time Travel Examples These features provide customers with unprecedented ability to recover from accidental errors during the retention period. For example, if an errant data loading script corrupted a database it is possible to create a logical duplicate of the database (a clone) from the point-in-time just prior to the execution of a specific statement. To illustrate, here are some examples of statements using the Time Travel feature: Recovering dropped objects: The UNDROP command can be used to recover any dropped object: UNDROP TABLE T; UNDROP DATABASE DB; F s 4

4 Recovering previous versions: Cloning a past version of a table at a specific point in time: CREATE TABLE orders_clone CLONE orders AT(TIMESTAMP => TO_TIMESTAMP_TZ( 04/05/ :02:03, mm/dd/yyyy hh24:mi:ss )); Cloning a past version of a database just before a query (identified here by a query ID) was processed: CREATE DATABASE db_clone CLONE marketing_db BEFORE(STATEMENT => 8e5d0ca9-005e-44e6- b858-a8f5b37c5726 ); Selecting data from past versions: Selecting data AT or BEFORE an arbitrary time during the retention period: without data loss, in the face of such failures. Snowflake, which runs on Amazon Web Services (AWS) infrastructure, uses a fully distributed and resilient architecture combined with the resiliency capabilities available in AWS to protect against a wide array of possible failures. As illustrated below, the Snowflake architecture consists of three layers, each of which is resilient to failures. Data storage layer. Stores all customer data in cloud storage services. Compute layer. Consists of one or more virtual warehouses, each of which is a multi-node compute cluster that processes queries. Virtual warehouses cache data from the data storage layer in encrypted form, but do not store persistent data. SERVICES AND METADATA SELECT max(t1.c1), max(t2.c2) FROM t1 AT(TIMESTAMP => :34:56 ::TIMESTAMP), t2 WHERE t1.c1 = t2.c1; COMPUTE Combining the current state of table t2 with a historical state of table t1 as it existed before a previous query (identified by the query ID): DATA SELECT OLD.ID FROM T BEFORE(STATEMENT => ea33c3a1-4ca0-41ed-b31e-c8f19d ) AS OLD LEFT OUTER JOIN T AS CUR ON OLD.ID = CUR.ID WHERE CUR.ID IS NULL; SNOWFLAKE S ARCHITECTURE PROTECTS BOTH DATA AND THE ABILITY TO ACCESS THAT DATA. Further details about Time Travel and commands are described in the Snowflake documentation. Protection Against System Failures The second most common type of data loss is caused by some form of system failure both software failures and infrastructure failures such as the loss of a disk, a disk array, a server, or most significantly, a data center. The Snowflake architecture is designed for resilience, Cloud services layer. The brain of the system, this layer manages infrastructure, queries, security, and metadata. The services running in this layer are implemented as a set of stateless processes. Each layer in the Snowflake architecture is distributed across at least three AWS availability zones. Because availability zones are geographically separated datacenters with independent access to power and networking, operations continue even if one or two AWS datacenters become unavailable. 5

5 zone by a customer. Protection of Data The database storage layer stores all customer data in encrypted form. It uses Amazon s S3 storage service to provide highly-durable, cost effective storage. When a transaction is committed in Snowflake, the data is securely stored in S3. S3 provides the base level of data protection for Snowflake highly durable data storage that allows data survival in the event of the loss of one or more disks, servers, or even datacenters. S3 synchronously and redundantly stores data across multiple devices in multiple facilities. It is designed for 11 9 s ( %) of data durability. Protection of Access to Data The compute and cloud services layers in the Snowflake architecture are also resilient to failures. Virtual warehouses are built to transparently recover from individual node failures without user intervention. In the case of an availability zone outage, virtual warehouses can be provisioned in another availability The cloud services layer is similarly resilient the stateless processes in this layer are redundant and are supported by a metadata store that is synchronously replicated to multiple nodes across all three availability zones. This ensures continuous operation even in the event of one or more metadata node failures or the loss of up to two datacenters. Additionally, the metadata store is regularly backed up to minimize the amount of data loss in the event of some form of catastrophic failure. For further protection, those backups are encrypted and are stored in a different region as well as sent to an offline storage vault. The web user interface in Snowflake is fully stateless and supported by the services in the cloud services layer such that it does not require backup and remains available even in the case of infrastructure and system failures. For more information about Snowflake s approach to CONCLUSION Data protection is a requirement for any data warehouse. The Snowflake data warehouse service is designed to provide robust data protection against a wide array of scenarios. All Snowflake customers benefit from enterprise-class data protection out of the box with no added complexity or overhead. GET STARTED To find out how Snowflake can help you get more value from your data, go to and try Snowflake for free today. 6

6 Snowflake Computing, the cloud data warehousing company, has reinvented the data warehouse for the cloud and today s data. The Snowflake Elastic Data Warehouse is built from the cloud up with a patent-pending new architecture that delivers the power of data warehousing, the flexibility of big data platforms and the elasticity of the cloud at a fraction of the cost of traditional solutions. Snowflake is headquartered in Silicon Valley and can be found online at snowflake.net.