Amazon Aurora Deep Dive

Size: px

Start display at page:

Download "Amazon Aurora Deep Dive"

Agatha Cook
6 years ago
Views:

1 Amazon Aurora Deep Dive Anurag Gupta VP, Big Data Amazon Web Services April, 2016 Up Buffer Quorum 100K to Less Proactive 1/10 15 caches Custom, Shared 6-way Peer than read writes/second Automated Pay the Less Automatic Up data 30s replicas cost MySQL for replication that than storage peer continuous scale-out the failovers tolerate block hotspot 64TB of survive fault IOPS repair 10ms commercial storage gossip volume that compatible & across corruption databases injection 500K drive provisioning management act of SSD a backups replication bad crash you growth reads/second replicas storage failover databases 3 disks use lag detection AZs recovery failures restarts S3 targets 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

2 What is Amazon Aurora? MySQL-compatible relational database Performance and availability of commercial databases Simplicity and cost-effectiveness of open source databases Delivered as a managed service

Re-imagined for the cloud Data Plane Control Plane

moved the 1 logging and storage layer into a multitenant,

Caching Amazon DynamoDB 2 Leverages existing AWS services:

Amazon S3 Logging + Storage Amazon SWF 3 Maintain

3 Re-imagined for the cloud Data Plane Control Plane Architected for the cloud e.g. moved the 1 logging and storage layer into a multitenant, scale-out database-optimized SQL Transactions storage service Caching Amazon DynamoDB 2 Leverages existing AWS services: Amazon EC2, Amazon VPC, Amazon DynamoDB, Amazon SWF, and Amazon S3 Logging + Storage Amazon SWF 3 Maintain compatibility with MySQL customers can migrate their MySQL applications as-is, use all MySQL tools. Amazon S3 Amazon Route 53

4 You ve probably heard about our benchmark numbers

Reproducing benchmark results 1 Create an Amazon VPC (or use an existing one). 2 Create four EC2 R3.8XL client instances to run the SysBench client. All four should be in the same AZ.

5 Reproducing benchmark results 1 Create an Amazon VPC (or use an existing one). 2 Create four EC2 R3.8XL client instances to run the SysBench client. All four should be in the same AZ. 3 Enable enhanced networking on your clients R3.8XLARGE R3.8XLARGE AMAZON AURORA R3.8XLARGE 4 Tune your Linux settings (see whitepaper) 5 Install Sysbench version 0.5 R3.8XLARGE 6 Launch a r3.8xlarge Amazon Aurora DB Instance in the same VPC and AZ as your clients 7 Start your benchmark! R3.8XLARGE https: //d 0. a w s s t a t i c. c o m / p r o d u c t - m a r k e t i n g / A u r o r a / R D S _ A u r o r a _ P e r f o r m a n c e _ A s s e s s m e n t _ B e n c h m a r k i n g _ v 1-2. pdf

6 5X faster than RDS MySQL 5.6 & 5.7 WRITE PERFORMANCE READ PERFORMANCE MySQL SysBench results R3.8XL: 32 cores / 244 GB RAM Aurora MySQL 5.6 MySQL 5.7 Five times higher throughput than stock MySQL based on industry standard benchmarks.

7 Scaling with instance sizes WRITE PERFORMANCE READ PERFORMANCE Aurora MySQL 5.6 MySQL 5.7 Aurora scales with instance size for both read and write.

8 Beyond benchmarks If only real world applications saw benchmark performance POSSIBLE DISTORTIONS Real world requests contend with each other Real world metadata rarely fits in data dictionary cache Real world data rarely fits in buffer cache Real world production databases need to run with HA enabled

9 Scaling User Connections Connections Amazon Aurora RDS MySQL w/ 30K IOPS 50 40,000 10, ,000 21,000 8x U P T O FA S T E R 5, ,000 13,000 SysBench OLTP Workload 250 tables

10 Scaling Table Count Tables Amazon Aurora MySQL I2.8XL local SSD MySQL I2.8XL RAM disk RDS MySQL w/ 30K IOPS (single AZ) 10 60,000 18,000 22,000 25, ,000 19,000 24,000 23,000 11x U P T O FA S T E R 1,000 64,000 7,000 18,000 8,000 10,000 54,000 4,000 8,000 5,000 SysBench write-only workload Measuring writes per second 1,000 connections

11 Scaling Data Size SYSBENCH WRITE-ONLY CLOUDHARMONY TPC-C DB Size Amazon Aurora RDS MySQL w/ 30K IOPS DB Size Amazon Aurora RDS MySQL w/ 30K IOPS 1GB 107,000 8,400 10GB 107,000 2,400 80GB 12, GB 9, GB 101,000 1,500 1TB 26,000 1,200 U P T O 21x FA S T E R U P T O 136x FA S T E R

12 Real-life data gaming workload Aurora vs. RDS MySQL r3.4xl, MAZ Aurora 3X faster on r3.4xlarge

13 How did we achieve this? DO LESS WORK Do fewer IOs Minimize network packets Cache prior results Offload the database engine BE MORE EFFICIENT Process asynchronously Reduce latency path Use lock-free data structures Batch operations together DATABASES ARE ALL ABOUT I/O NETWORK-ATTACHED STORAGE IS ALL ABOUT PACKETS/SECOND HIGH-THROUGHPUT PROCESSING DOES NOT ALLOW CONTEXT SWITCHES

14 IO traffic in MySQL MYSQL WITH REPLICA AZ 1 AZ 2 Primary Instance 1 Amazon Elastic Block Store (EBS) 3 Replica Instance EBS 4 IO FLOW Issue write to EBS EBS issues to mirror, ack when both done Stage write to standby instance through DRBD Issue write to EBS on standby instance OBSERVATIONS Steps 1, 3, 5 are sequential and synchronous This amplifies both latency and jitter Many types of writes for each user operation Have to write data blocks twice to avoid torn writes 2 5 PERFORMANCE Amazon S3 EBS mirror EBS mirror 780K transactions 7,388K I/Os per million txns (excludes mirroring, standby) Average 7.4 I/Os per transaction 30 minute SysBench writeonly workload, 100GB dataset, RDS MultiAZ, 30K PIOPS T YPE OF WRIT E LOG BINLOG DATA DOUBLE-WRITE FRM FILES

15 IO traffic in Aurora AMAZON AURORA AZ 1 AZ 2 AZ 3 Primary Instance Replica Instance Replica Instance IO FLOW Boxcar redo log records fully ordered by LSN Shuffle to appropriate segments partially ordered Boxcar to storage nodes and issue writes ASYNC 4/6 QUORUM Amazon S3 DISTRIBUTED WRITES OBSERVATIONS Only write redo log records; all steps asynchronous No data block writes (checkpoint, cache replacement) 6X more log writes, but 9X less network traffic Tolerant of network and storage outlier latency PERFORMANCE 27,378K transactions 35X MORE 950K I/Os per 1M txns (6X amplification) 7.7X LESS T YPE OF WRIT E LOG BINLOG DATA DOUBLE-WRITE FRM FILES

16 IO traffic in Aurora (Storage Node) STORAGE NODE IO FLOW Primary Instance Peer Storage Nodes LOG RECORDS 4 ACK INCOMING QUEUE 2 PEER TO PEER GOSSIP SORT GROUP UPDATE QUEUE HOT LOG 3 1 COALESCE 5 POINT IN TIME SNAPSHOT 6 S3 BACKUP GC DATA BLOCKS 7 SCRUB 8 1 Receive record and add to in-memory queue 2 Persist record and ACK 3 Organize records and identify gaps in log 4 Gossip with peers to fill in holes 5 Coalesce log records into new data block versions 6 Periodically stage log and new block versions to S3 7 Periodically garbage collect old versions 8 Periodically validate CRC codes on blocks OBSERVATIONS All steps are asynchronous Only steps 1 and 2 are in foreground latency path Input queue is 46X less than MySQL (unamplified, per node) Favor latency-sensitive operations Use disk space to buffer against spikes in activity

17 Scaling updates with replicas Updates per second Amazon Aurora RDS MySQL 30K IOPS (single AZ) 1, ms 0 s 2, ms 1 s 5, ms 60 s 500x U P T O L O W E R L A G 10, ms 300 s SysBench Writeonly Workload 250 tables

18 Real-life data - read replica latency In RDS MySQL, we saw replica lag spike to almost 12 minutes which is almost absurd from an application s perspective. The maximum read replica lag across 4 replicas never exceeded beyond 20 ms.

19 IO traffic in Aurora Replicas MYSQL READ SCALING AMAZON AURORA READ SCALING MySQL Master 70% Write SINGLE-THREADED BINLOG APPLY MySQL Replica 70% Write Aurora Master 70% Write PAGE CACHE UPDATE Aurora Replica 100% New Reads 30% Read 30% New Reads 30% Read Data Volume Data Volume Shared Multi-AZ Storage Logical: Ship SQL statements to Replica Write workload similar on both instances Independent storage Can result in data drift between Master and Replica Physical: Ship redo from Master to Replica Replica shares storage. No writes performed Cached pages have redo applied Advance read view when all commits seen

20 Performance only matters if your database is up

21 Storage durability Storage volume automatically grows up to 64 TB Quorum system for read/write; latency tolerant AZ 1 AZ 2 AZ 3 Peer to peer gossip replication to fill in holes Continuous backup to S3 (built for 11 9s durability) Continuous monitoring of nodes and disks for repair 10GB segments as unit of repair or hotspot rebalance Quorum membership changes do not stall writes Amazon S3

Fault-tolerant storage Six copies across three availability zones 4 out 6 write quorum; 3 out of 6 read quorum Peer-to-peer replication for repairs Volume

22 Fault-tolerant storage Six copies across three availability zones 4 out 6 write quorum; 3 out of 6 read quorum Peer-to-peer replication for repairs Volume striped across hundreds of storage nodes AZ 1 AZ 2 AZ 3 SQL Transaction Caching AZ 1 AZ 2 AZ 3 SQL Transaction Caching Read availability Read and write availability

23 Continuous backup Segment snapshot Log records Segment 1 Segment 2 Segment 3 Recovery point Time Take periodic snapshot of each segment in parallel; stream the redo logs to Amazon S3 Backup happens continuously without performance or availability impact At restore, retrieve the appropriate segment snapshots and log streams to storage nodes Apply log streams to segment snapshots in parallel and asynchronously

24 Survivable caches We moved the cache out of the database process Caching process is outside the DB process and remains warm across a database restart Cache remains warm in the event of database restart SQL SQL SQL Lets you resume fully loaded operations much faster Transactions Caching Transactions Caching Transactions Caching Instant crash recovery + survivable cache = quick and easy recovery from DB failures

25 Instant crash recovery Traditional Databases Have to replay logs since the last checkpoint Amazon Aurora Underlying storage replays redo records on demand as part of a disk read Typically 5 minutes between checkpoints Parallel, distributed, asynchronous Single-threaded in MySQL; requires a large number of disk accesses Crash at T 0 requires a re-application of the SQL in the redo log since last checkpoint No replay for startup Crash at T 0 will result in redo logs being applied to each segment on demand, in parallel, asynchronously Checkpointed Data Redo Log T 0 T 0

26 Read replicas are failover targets Aurora cluster contains primary node and up to fifteen secondary nodes Failing database nodes are automatically detected and replaced AZ 1 AZ 2 AZ 3 Primary Primary Node Primary Node Node Primary Primary Secondary Node Node Node Primary Primary Secondary Node Node Node Failing database processes are automatically detected and recycled Secondary nodes automatically promoted on persistent outage, no single point of failure Customer application may scale-out read traffic across secondary nodes Customer specifiable fail-over order Read balancing across read replicas

27 Faster failover MYSQL DB Failure Failure Detection DNS Propagation App Running Recovery Recovery AURORA WITH MARIADB DRIVER DB Failure Failure Detection DNS Propagation s e c Recovery s e c App Running

28 Real-life data fail-over time In RDS MySQL, it took minutes or sometimes tens of minutes to failover. It s pretty awesome that you can failover/restart within less than a minute.

29 Simulate failures using SQL To cause the failure of a component at the database node: ALTER SYSTEM CRASH [{INSTANCE DISPATCHER NODE}] To simulate the failure of disks: ALTER SYSTEM SIMULATE percent_failure DISK failure_type IN [DISK index NODE index] FOR INTERVAL interval To simulate the failure of networking: ALTER SYSTEM SIMULATE percent_failure NETWORK failure_type [TO {ALL read_replica availability_zone}] FOR INTERVAL interval

31 Compatible with the MySQL ecosystem

32 Well established MySQL ecosystem We ran our compatibility test suites against Amazon Aurora and everything just worked." - Dan Jewett, Vice President of Product Management at Tableau Business Intelligence Data Integration Query and Monitoring SI and Consulting Source: Amazon

33 Many 3 rd party monitoring tools

34 Monitoring Aurora with Datadog Just add read-only AWS credentials and select the services you wish to monitor (e.g. RDS)

35 Aurora enhanced metrics Monitor the new RDS enhanced metrics for high-resolution system-level metrics

36 Monitoring the whole stack Correlate Aurora metrics with metrics and events from the rest of your infrastructure

37 Q&A

Amazon Aurora Deep Dive

Amazon Aurora Deep Dive Enterprise-class database for the cloud Damián Arregui, Solutions Architect, AWS October 27 th, 2016 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Enterprise