Deep Dive on MySQL Databases on Amazon RDS. Chayan Biswas Sr. Product Manager Amazon RDS

Size: px

Start display at page:

Download "Deep Dive on MySQL Databases on Amazon RDS. Chayan Biswas Sr. Product Manager Amazon RDS"

Sylvia Washington
5 years ago
Views:

1 Deep Dive on MySQL Databases on Amazon RDS Chayan Biswas Sr. Product Manager Amazon RDS

2 Amazon RDS is... Cloud native engine Open source engines Commercial engines RDS platform Automatic fail-over Backup & recovery X-region replication Isolation & security Industry compliance Automated patching Advanced monitoring Routine maintenance Push-button scaling

Automatic fail-over Backup & recovery X-region replication Isolation &

3 Amazon RDS for open source MySQL engines are... Cloud native engine Open source engines Commercial engines RDS platform Automatic fail-over Backup & recovery X-region replication Isolation & security Industry compliance Automated patching Advanced monitoring Routine maintenance Push-button scaling

4 Agenda Why run MySQL Why run managed MySQL on Amazon RDS Migrating to RDS Tips and tricks from one of the world s largest MySQL operators

6 1. Popular 2. Innovative 3. Flexible Image credit: By Mackphillips - Own work, CC BY-SA 4.0,

7 1. Popular 2. Innovative 3. Flexible Image credit: By Mackphillips - Own work, CC BY-SA 4.0,

8 MySQL is the world s most popular database All respondents Source: Stack Overflow Developer Survey Results 2018 ( )

9 MySQL is the world s most popular database Professional Developers Source: Stack Overflow Developer Survey Results 2018 ( )

10 Most popular buys you... Highly exercised, stable code Large ecosystem of ISVs, Tools, Implementation and Support Partners Large community of users and community-driven resources and a larger DBA talent pool

11 1. Popular 2. Innovative 3. Flexible Image credit: By Mackphillips - Own work, CC BY-SA 4.0,

12 MySQL 8.0 highlights - FUNCTIONALITY Common Table Expressions Window functions JSON improvements 5108 Spatial Reference Systems utf8mb4

13 Common Table Expressions (CTEs; WITH statement) Is more readable Can be referenced more than once Can be chained Has better performance Can be recursive Common Table Expressions WITH derived AS (subquery) SELECT... FROM derived, t1... vs. Derived Tables SELECT... FROM (subquery) AS derived, t1...

14 JSON improvements JSON_TABLE(): accepts JSON, returns relational table ->> (inline path) operator: Equivalent to JSON_UNQUOTE() + JSON_EXTRACT() JSON aggregation functions: JSON_ARRAYAGG() and JSON_OBJECTAGG() JSON_MERGE_PATCH(): accepts two JSON objects, merges them into one Range support in XPath expressions: for example $[1 to 5], $[last] Utility functions: JSON_PRETTY(), JSON_STORAGE_SIZE(), JSON_STORAGE_FREE()

15 MySQL 8.0 highlights - AVAILABILITY Instant ADD COLUMN Unified, transactional data dictionary Crash-safe, atomic DDL

16 Instant ADD COLUMN ALTER TABLE table_name [alter_specification], ALGORITHM=INSTANT; Only metadata changes are made in the data dictionary No need to acquire metadata lock during SE changes and we don t touch the data of the table Currently supports the following operations (with some limitations*): Change index option Rename table (in ALTER way) SET/DROP DEFAULT MODIFY COLUMN Add/drop virtual columns Add columns (non-generated) also called instant ADD COLUMN Fails fast for operations that do not support INSTANT algorithm

17 Unified, transactional metadata dictionary Benefits Simplicity Centralized, uniform data dictionary Removal of file-based metadata storage Uniform and centralized caching for dictionary objects Transactional, crash-safe storage of dictionary data Simpler, improved implementation for INFORMATION_SCHEMA tables Atomic DDL

18 MySQL 8.0 highlights - PERFORMANCE 2x-5x higher performance versus MySQL 5.7 Hot-spot management Descending indexes Invisible indexes Improved optimizer cost model Resource Groups Improved replication

19 2x-5x higher performance versus MySQL 5.7 Multi-AZ configuration, r4.16xlarge, 30K IOPS, 1TB database 6000 Transactions per second, with varying number of threads MySQL 5.6 MySQL 5.7 MySQL 8.0

20 Resource groups CREATE RESOURCE GROUP Reporting TYPE = USER VCPU = 5-6 THREAD_PRIORITY = 6; Dynamically control resource consumption by threads in a group Currently supports vcpus as a manageable resource CREATE RESOURCE GROUP Batch TYPE = USER VCPU = 7-8 THREAD_PRIORITY = 4; SET RESOURCE GROUP Reporting FOR 1866; SELECT /*+ RESOURCE_GROUP(Reporting) */ <col> FROM ALTER RESOURCE GROUP Reporting VCPU = 5 THREAD_PRIORITY = 19;

21 MySQL 8.0 highlights SECURITY, MANAGEABILITY Roles Password strength Open SSL as default TLS/SSL library Enhanced observability

22 MariaDB 10.2 highlights Common Table Expressions Window Functions JSON and GeoJSON functions Replication improvements Delayed replication; throttling; compressed binlog Oracle compatibility improvements Multi-trigger; CHECK constraint expressions; EXECUTE IMMEDIATE, 38-digit DECIMAL etc. Image credit: By Mackphillips - Own work, CC BY-SA 4.0,

23 MariaDB 10.3 highlights Oracle compatibility PL/SQL compatibility parser Sequences INTERSECT and EXCEPT to complement UNION New ROW type and TYPE OF stored functions Invisible Columns Cursor with parameters Temporal data processing User-defined aggregates ADD INSTANT COLUMN for InnoDB Image credit: By Mackphillips - Own work, CC BY-SA 4.0,

24 Oracle compatibility - PL/SQL compatibility parser Easier migration from Oracle to MariaDB No need to migrate Oracle PL/SQL to SQLP/PM Simply SET sql_mode= oracle to create a stored procedure using Oracle PL/SQL syntax Parser addresses syntax differences between a large subset of PL/SQL and SQL/PSM Stored Procedures and Stored Functions Cursors LOOP Variables Exceptions BEGIN Blocks ELSIF, UNIQUE, TRUNCATE Functions Prepared Statements Synonyms for Basic SQL Types Packages NULL Handling Oracle and MariaDB syntax based stored procedures can be used in the same database

25 1. Popular 2. Innovative 3. Flexible Image credit: By Mackphillips - Own work, CC BY-SA 4.0,

27 Amazon RDS for open source MySQL engines are... Cloud native engine Open source engines Commercial engines RDS platform Automatic fail-over Backup & recovery X-region replication Isolation & security Industry compliance Automated patching Advanced monitoring Routine maintenance Push-button scaling

28 1. Popular 2. Innovative 3. Flexible

29 1. Popular 2. Innovative 3. Flexible

30 Amazon RDS: hundreds of thousands of customers

31 Popular buys you... Highly exercised, stable code Unrivalled operational excellence

32 Who can you trust? Automated remediation World s most experienced operators Automation

33 1. Popular 2. Innovative 3. Flexible Image credit: By Mackphillips - Own work, CC BY-SA 4.0,

34 Amazon RDS highlights - AVAILABILITY Automated, 0-RPO failover across AZs Managed x-region replicas for DR Automated backups, manual snapshots Point-in-time recovery Elastic volumes up to 16 TB

35 Automated, 0-RPO failover across AZs Each host manages set of EBS volumes with a full copy of the data Availability zone 1 Availability zone 2 Primary Standby Instances are monitored by an external observer to maintain consensus over quorum EBS #1 EC2 #1 DNS EBS #2 EC2 #2 App

36 Automated, 0-RPO failover across AZs Each host manages set of EBS volumes with a full copy of the data Availability zone 1 Availability zone 2 Primary Standby Instances are monitored by an external observer to maintain consensus over quorum Failover initiated by automation or through RDS API EBS #1 EC2 #1 DNS EBS #2 EC2 #2 App

37 Automated, 0-RPO failover across AZs Each host manages set of EBS volumes with a full copy of the data Availability zone 1 Availability zone 2 Primary Instances are monitored by an external observer to maintain consensus over quorum Failover initiated by automation or through RDS API EBS #1 EC2 #1 DNS EBS #2 EC2 #2 Redirection to the new primary instance is provided through DNS App

Automated, 0-RPO failover across AZs Each host manages set of EBS volumes with a full copy of the data Availability zone 1 Availability zone 2 Standby Primary Instances are monitored by an

38 Automated, 0-RPO failover across AZs Each host manages set of EBS volumes with a full copy of the data Availability zone 1 Availability zone 2 Standby Primary Instances are monitored by an external observer to maintain consensus over quorum Failover initiated by automation or through RDS API EBS #1 EC2 #1 DNS EBS #2 EC2 #2 Redirection to the new primary instance is provided through DNS App

39 Backups, Snapshots, and Point-in-time restore Region 1 Two options automated backups and manual snapshots EBS snapshots stored in Amazon S3 Transaction logs stored every 5 minutes in S3 to support Point in Time Recovery Availability zone 2 Availability zone 1 EBS EBS Vol vol. EBS Snapshot Transaction Logs Amazon S3 No performance penalty for backups Snapshots can be copied across regions or shared with other accounts EBS vol. Region 2 EBS vol. App

40 Amazon RDS highlights SECURITY, MANAGEABILITY IAM DB Authentication Industry compliance Automated OS and database upgrades Push-button scaling Managed binlog replication Log upload to CloudWatch Logs Recommendations

41 Recommendations Example issues flagged by the feature: Engine version outdated Pending maintenance available Automated backups disabled Enhanced Monitoring disabled Encryption disabled

42 Amazon RDS highlights PERFORMANCE New! M5 database instance family Performance insights

43 Performance Insights Measures DB Load Identifies bottlenecks (top SQL, wait events) Adjustable time frame (hour, day, week, longer)

44 1. Popular 2. Innovative 3. Flexible Image credit: By Mackphillips - Own work, CC BY-SA 4.0,

45 All we have to see Is I don t belong to you And you don t belong to me. George Michael Freedom! 90

47 Migration paths to RDS MySQL MySQL Tools & Amazon S3 AWS Data Migration Service On-premise MySQL/MySQL-compatible Commercial DBs Heterogeneous Non-relational 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

48 MySQL/MySQL-compatible Non MySQL-compatible Amazon EC2 On-premise Commercial DBs Non-relational Backup MySQL Tools AWS DMS Supported Recommended Unsupported Usable 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

49 MySQL databases to RDS with Backup Replication Backup Managed restore MySQL S3 Ingestion from S3 of Percona XtraBackup files High performance Faster restores with binary backups Binlog replication for near-zero downtime migration EC2 or On-premises 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

50 MySQL/MySQL-compatible Non MySQL-compatible Amazon EC2 On-premise Commercial DBs Non-relational Backup MySQL Tools AWS DMS Supported Recommended Unsupported Usable 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

MySQL databases to RDS with MySQL tools Replication Logical dump Logical import Source DB Source-side Client S3 Client (EC2 instance) MySQL tools MySQL commands mysqldump /

51 MySQL databases to RDS with MySQL tools Replication Logical dump Logical import Source DB Source-side Client S3 Client (EC2 instance) MySQL tools MySQL commands mysqldump / mysqlimport: MySQL-native tools. SELECT INTO OUTFILE, LOAD DATA INFILE mydumper, myloader: Multi-threaded, flexible 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

52 Client location Close proximity to database servers Logical dump Client Instance Logical import Same VPC and AZ as the server Source Database Two client machines If the source database is in a different VPC or on-premises 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

53 Client instance considerations 1 CPU/export or import thread for real-time processing e.g. compression/decompression Use EC2 instance types that support enhanced networking Ensure client instances and underlying EBS volume can sustain I/O throughput Rely on OS diagnostics to detect CPU, storage or network I/O bottlenecks 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

54 RDS instance considerations Provision enough capacity to ingest or dump data 2 CPU/thread for imports, and 1 CPU/thread for exports Point-and-click compute scaling through AWS console to adjust compute capacity 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

55 Large table exports and imports Use mydumper for a multithreaded export of table data. For large tables, specify the --rows parameter to break it into chunks --rows : Split table into chunks of these many rows, default unlimited Use myloader for multithreaded imports Adjust the number of threads with the --thread parameter for instances larger than r4.2xlarge 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

56 Client performance tweaks Enable Receive Packet Steering (RPS) sudo sh -c 'for x in /sys/class/net/eth0/queues/rx-*; do echo ffffffff, ffffffff > $x/rps_cpus; done' sudo sh -c "echo 4096 > /sys/class/net/eth0/queues/rx-0/rps_flow_cnt" sudo sh -c "echo 4096 > /sys/class/net/eth0/queues/rx-1/rps_flow_cnt Enable Receive Flow Steering (RFS) sudo sh -c "echo > /proc/sys/net/core/rps_sock_flow_entries" 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

58 MySQL/MySQL-compatible Non MySQL-compatible Amazon EC2 On-premise Commercial DBs Non-relational Backup MySQL Tools AWS DMS Supported Recommended Unsupported Usable 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

59 AWS Database Migration Service (AWS DMS) DMS migrates databases to AWS easily and securely with minimal downtime. It can migrate your data to and from most widely used commercial and open-source databases and for as little a $3 for TB DB. Amazon Aurora

Native Tool Data Source DB AWS DMS 2017, Amazon Web

60 Database migration process Step 1: Convert or Copy your Schema AWS SCT Source DB Step 2: Move your data Native Tool Data Source DB AWS DMS 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

source and target databases Create Tasks Application users Let AWS DMS create tables,

61 How does DMS work? AWS DMS Customer premises VPN AWS Internet Start a replication instance Connect to source and target databases Create Tasks Application users Let AWS DMS create tables, load data, and keep them in sync Switch applications over to the target at your convenience 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

63 6 frequently asked questions... Which instance family should I choose? When should I use Multi-AZ vs. Read Replicas? When should I use automated backups vs. snapshots? How do I secure my database? How do I monitor my database? Can I know when service events happen?

64 Which instance family should I choose? T2/T3 M2/M4/M5 R3/R4 Burstable General purpose Memory optimized 1vCPU, 1GiB RAM to 8vCPU, 32 GiB RAM Moderate networking 2vCPU, 8GiB RAM to 96vCPU, 384 GiB RAM High perf. networking 2 vcpu, 16 GiB RAM to 64 vcpu 488 GiB RAM High perf. networking Dev/test and smaller workloads CPU intensive workloads (for example, WordPress) Query intensive workloads with high connection count

65 When should I use Multi-AZ vs. Read Replicas? Multi-AZ Synchronous replication highly durable Secondary is idle Backups can be taken from secondary Always in two AZs within a Region DB engine upgrades on primary Automatic failover when issue is detected Read-replica General purpose Replica can serve read workload No backups configured by default Can be within AZ, cross-az, or cross-region DB engine upgrades independent of primary Manual promotion to standalone database

When should I use backups vs. snapshots?

window (35 day maximum) or instance is deleted Supports Point in Time Restore

CLI or API Kept until you delete them Restores to saved snapshot Good for

66 When should I use backups vs. snapshots? Automated backups Specify retention window (7 day default) Kept until out of window (35 day maximum) or instance is deleted Supports Point in Time Restore Good for disaster recovery Manual snapshots Create manually via AWS console, CLI or API Kept until you delete them Restores to saved snapshot Good for data-retention compliance, checkpoint before making large changes, spawning test databases, and final copy before deleting a database

67 How do I secure my database? Amazon VPC: network Isolation AWS IAM: resource-level permission controls AWS KMS: encryption at rest SSL: protection for data in transit

68 How do I monitor my database? Amazon CloudWatch metrics: CPU, storage, memory, swap usage, I/O, latency, throughput, replica lag Amazon CloudWatch logs: Audit log, General log, Slowquery log, and Error Amazon CloudWatch alarms: Similar to on-premises monitoring tools Enhanced monitoring: 50+ CPU, memory, file system and disk I/O metrics Other: Performance Insights, 3 rd party tools

69 Can I know when service events happen? Amazon SNS: , text, call to HTTP endpoint Amazon CloudWatch Events: run AWS Lambda or Amazon ECS tasks. 6 different source types: DB Instance, DB Parameter Group, DB Security Group, DB Snapshot DB Cluster, DB Cluster Snapshot 17 different event categories: availability, backup, configuration change, etc.

70 Thank you! Chayan Biswas

71 Rate My Session

Aurora, RDS, or On-Prem, Which is right for you

Aurora, RDS, or On-Prem, Which is right for you Kathy Gibbs Database Specialist TAM Katgibbs@amazon.com Santa Clara, California April 23th 25th, 2018 Agenda RDS Aurora EC2 On-Premise Wrap-up/Recommendation