Amazon Aurora Relational databases reimagined.

Size: px

Start display at page:

Download "Amazon Aurora Relational databases reimagined."

Coleen Walsh
6 years ago
Views:

2 Current DB Architectures are Monolithic SQL Transactions Caching Multiple layers of functionality all on a single box Logging

3 Current DB Architectures are Monolithic SQL Transactions Caching Logging Application SQL Transactions Caching Logging Even when you scale it out, you re still replicating the same stack

4 Current DB Architectures are Monolithic SQL Transactions Caching Logging Application SQL Transactions Caching Logging Even when you scale it out, you re still replicating the same stack

5 Current DB Architectures are Monolithic SQL Transactions Caching Logging Application SQL Transactions Caching Logging Even when you scale it out, you re still replicating the same stack Storage

6 This is a problem. For cost. For flexibility. And for availability.

7 Re-imagining the Relational Database What if we were inventing the database today? You wouldn t design it the way we did in At least not entirely You d build something scale-out, self-healing, that leverage existing AWS services

8 Relational databases reimagined for the cloud. speed and availability of high-end commercial databases simplicity and cost-effectiveness of open source databases drop-in compatibility with MySQL simple pay as you go pricing Delivered as a managed service.

Amazon Aurora applying a Service-oriented architecture to the database Moved

database-optimized storage service Integrated with other AWS Services like

SQL Transactions Caching Logging + Storage Control Plane DynamoDB Amazon SWF

9 Amazon Aurora applying a Service-oriented architecture to the database Moved the logging and storage layer into a multi-tenant, scale-out database-optimized storage service Integrated with other AWS Services like EC2, VPC, DynamoDB, SWF, Route 53 for control plane operations Data Plane SQL Transactions Caching Logging + Storage Control Plane DynamoDB Amazon SWF Integrated with S3 for continuous backup and % durability Amazon S3 Amazon Route 53

10 Aurora Works with Your Existing Apps

11 An Established Ecosystem It is great to see Amazon Aurora remains MySQL compatible; we have found our connectors work with Aurora seamlessly. Today, customers can take our drivers and connect to Aurora, MariaDB or MySQL without worrying about compatibility. We look forward to working with the Aurora team in the future to further accelerate innovation within the MySQL ecosystem. Rasmus Johansson, VP Engineering Business Intelligence Data Integration Query & Monitoring SI & Consulting

12 Amazon Aurora is Easy to Use

13 Aurora Makes it Easy to Run Your Databases Create a database in minutes Automatic patching Push-button scaling Failure detection and failover. Amazon RDS Read Replica s are available as failover targets, with no data loss

14 Aurora simplifies storage management Instant creation of user-snapshots Continuous backups to S3 Automatic storage scaling up to 64 TB - no performance or availability impact Amazon RDS Automatic restriping, mirror repair, hot spot management, encryption

Aurora simplifies Data Security Encryption

KMS Customer VPC AZ 1 AZ 2 AZ 3 MySQL App

Instance No direct access to nodes Supports

15 Aurora simplifies Data Security Encryption to secure data at rest AES-256; hardware accelerated All blocks on disk and in Amazon S3 encrypted Key management via AWS KMS Customer VPC AZ 1 AZ 2 AZ 3 MySQL App SSL to secure data in transit Network isolation via Amazon VPC by default Internal VPC Primary Instance Replica Instance No direct access to nodes Supports industry standard security and data protection certifications Amazon S3

16 Amazon Aurora is Highly Available

17 Aurora is Highly Available Highly available by default 6-way replication across 3 AZs 4 of 6 write quorum Automatic fallback to 3 of 4 if an AZ is unavailable 3 of 6 read quorum SSD, scale-out, multi-tenant storage AZ 1 AZ 2 AZ 3 SQL Transaction s Caching Seamless storage scalability Up to 64TB database size Only pay for what you use Log-structured storage Many small segments, each with their own redo logs Amazon S3 Log pages used to generate data pages Eliminates chatter between database and storage

18 Aurora Performs Consistent, Low Latency Writes MySQL Multi-AZ with Standby AZ 1 AZ 2 Primary Instance 2 phase commit Standby Instance AZ 1 AZ 2 AZ 3 Primary Instance Amazon Aurora Replica Instance async 4/6 quorum PiTR EBS EBS EBS mirror Sequential write EBS mirror Sequential write Distributed writes Amazon S3 Amazon S3 Improvements Consistency - tolerance to outliers Latency - 2 phase commit vs. asynchronous replication Type of writes Log records Binlog Data Doublewrite buffer Significantly more efficient use of network IO FRM files, metadata

write Distributed writes Amazon S3 Amazon S3 Improvements Consistency - tolerance to outliers Latency - 2 phase commit vs.

19 Aurora Performs Consistent, Low Latency Writes MySQL with Standby AZ 1 AZ 2 Primary Instance 2 phase commit Standby Instance AZ 1 AZ 2 AZ 3 Primary Instance Amazon Aurora Replica Instance async 4/6 quorum PiTR EBS EBS EBS mirror Sequential write EBS mirror Sequential write Distributed writes Amazon S3 Amazon S3 Improvements Consistency - tolerance to outliers Latency - 2 phase commit vs. asynchronous replication Type of writes Log records Binlog Data Doublewrite buffer Significantly more efficient use of network IO FRM files, metadata

20 Self-healing and fault-tolerant Lose 2 copies or an AZ failure without read or write availability impact Lose 3 copies without read availability impact Automatic detection, replication and repair AZ 1 AZ 2 AZ 3 SQL Transaction Caching AZ 1 AZ 2 AZ 3 SQL Transaction Caching Read Availability Read & Write Availability

21 Aurora Has Instant Crash Recovery Traditional Databases Have to replay logs since the last checkpoint Single threaded in MySQL; requires a large number of disk accesses Crash at T 0 requires a re-application of the SQL in the redo log since last checkpoint Amazon Aurora Underlying storage replays redo records on demand as part of a disk read Parallel, distributed, asynchronous Crash at T 0 will result in redo logs being applied to each segment on demand, in parallel, asynchronously Checkpointed Data Redo Log T 0 T 0

22 Aurora s Cache Survives a DB Restart We moved the cache out of the database process Cache remains warm in the event of a database restart Caching Process is Outside the DB process and remains warm across a database restart SQL Transactions SQL Transactions SQL Transactions Lets you resume fully loaded operations much faster Caching Caching Caching Instant crash recovery + survivable cache = quick and easy recovery from DB failures

23 Multiple failover targets, without data loss. MySQL Master 70% Write 30% Read Single threaded binlog apply MySQL Replica 70% Write 30% New Reads Aurora Master 70% Write 30% Read Page cache invalidation Aurora Replica 100% New Reads Data Volume Data Volume Shared Multi-AZ Storage MySQL Read Scaling Replicas must replay logs Replicas place additional load on master Replica lag can grow indefinitely Failover results in data loss

24 You Can Simulate Failures Using SQL To cause the failure of a component at the database node: ALTER SYSTEM CRASH [{INSTANCE DISPATCHER NODE}] To simulate the failure of disks: ALTER SYSTEM SIMULATE percent_failure DISK failure_type IN [DISK index NODE index] FOR INTERVAL interval To simulate the failure of networking: ALTER SYSTEM SIMULATE percent_failure NETWORK failure_type [TO {ALL read_replica availability_zone}] FOR INTERVAL interval

25 Intermission

26 Intercom

27 Intercom s technology - Largely monolithic Ruby on Rails application. Strong culture around Continuous Deployment and general DevOps best practices. Some adoption of SOA/Microservices. - Built using MySQL and Ruby on Rails ActiveRecord ORM for rapid development. Unstructured customer data stored in MongoDB, however all messages in MySQL. - Heavy use of AWS services and other SaaS services (New Relic, Code Climate, CodeShip, LogEntries). - Custom infrastructure/code orchestration & deployment system.

29 Recent MySQL woes - Highly sensitive to MySQL performance and started experiencing regular inexplicable performance degradation. Could not vertically scale our way out of the problem. Engaged RDS Support & MySQL consultants. - Adjusted parameters e.g. lock wait timeouts, transaction read isolation levels and txn_flush_at_commit, etc. Instrumented and reduced number of long-running transactions in an attempt to reduce lock contention. - Greatly increased the number of MySQL metrics being collected, built application level fingerprinting and automated data collection during outages. Got our hands dirty with MySQL s performance schema. Reducing read throughput stabilised.

30 Why we re interested in Aurora - While we bought ourselves time with improved caching, Aurora gives us more options and a lot more vertical scaling opportunities. - Operational experiences of using read-replicas with RDS/MySQL means we don t trust them for customer facing queries. The eventual consistency guarantees of Aurora look good enough for our application to use for practically all read queries.

31 intercom.io blog.intercom.io

32 Enterprise grade features and performance at open source prices

33 Aurora Pricing vcpu Mem Hourly Price db.r3.large $0.29 db.r3.xlarge $0.58 db.r3.2xlarge 8 61 $1.16 db.r3.4xlarge $2.32 db.r3.8xlarge $4.64 Storage consumed, up to 64TB, is $0.10/GB/month IOs consumed are billed at $0.20 per million IO Prices are for Virginia Simple pricing No licenses No lock-in Pay only for what you use Discounts 44% with a 1 year RI 63% with a 3 year RI

34 Aurora Enterprise Grade. Open Source Prices Expanding to unlimited preview Adding preview support for US West (Oregon) and EU (Ireland) Signup for preview access at: Full service launch in the coming months

35 LONDON

BERLIN. 2015, Amazon Web Services, Inc. or its affiliates. All rights reserved

BERLIN. 2015, Amazon Web Services, Inc. or its affiliates. All rights reserved BERLIN 2015, Amazon Web Services, Inc. or its affiliates. All rights reserved Amazon Aurora: Amazon s New Relational Database Engine Carlos Conde Technology Evangelist @caarlco 2015, Amazon Web Services,