Deep Dive on Amazon Relational Database Service

Size: px

Start display at page:

Download "Deep Dive on Amazon Relational Database Service"

Everett Simon
6 years ago
Views:

2 What to expect Amazon RDS overview Security Customer story Migrating to RDS Metrics and monitoring Scaling on RDS Backups and snapshots High availability

3 Amazon Relational Database Service (Amazon RDS) No infrastructure management Instant provisioning Application compatibility Cost-effective Scale up/down

4 Amazon RDS engines Commercial Open source Amazon Aurora

5 Amazon Aurora vs. MySQL Feature RDS Aurora RDS MySQL Number of replicas Up to 15 Up to 5 Replication type Asynchronous (milliseconds) Asynchronous (seconds) Replication performance impact on primary Low Replica can act as failover target Yes (no data loss) Yes (potentially minutes of loss) Storage Up to 64 TB, auto growth Up to 6 TB, specify storage limit Automated failover Yes, to replica Yes, to standby User-defined replication delay No Yes Replica support for different data or schema vs. primary Cross-region replication No Yes Data cache survives Yes No No High Yes

Trade-offs with a managed service Fully managed host and OS No access to the database host operating system Limited ability to modify configuration that is managed on the host operating system No

6 Trade-offs with a managed service Fully managed host and OS No access to the database host operating system Limited ability to modify configuration that is managed on the host operating system No functions that rely on configuration from the host OS Fully managed storage Max storage limits SQL Server 4 TB MySQL, MariaDB, PostgreSQL, Oracle 6 TB Aurora 64 TB Growing your database is a process

7 Selected Amazon RDS customers

8 Security

9 Amazon Virtual Private Cloud (Amazon VPC) Securely control network configuration /16 Manage connectivity /24 AWS Direct Connect VPN Connection VPC Peering Routing Rules Internet Gateway Availability Zone AWS Region

10 Security groups Database IP firewall protection Corporate address admins Protocol Port Range Source TCP /16 TCP 3306 Application security group Application tier

11 Compliance Singapore MTCS 27001/ /27018

12 Compliance MySQL, Oracle, PostgreSQL SOC 1, 2, and 3 ISO 27001/9001 ISO 27017/27018 PCI DSS FedRamp HIPAA BAA UK government programs Singapore MTCS Germany C5 SQL Server SOC 1, 2, and 3 ISO 27001/9001 ISO 27017/27018 PCI DSS UK government programs Singapore MTCS Germany C5 Aurora SOC 1, 2, and 3 ISO 27001/9001 ISO 27017/27018 PCI DSS Germany C5

$myinstance.c123xyz.rds-eu-west-1.amazonaws \ --ssl-ca=rds-combined-ca-bundle.$

13 SSL Using SSL to encrypt a connection to a DB instance Available for all six engines mysql -h myinstance.c123xyz.rds-eu-west-1.amazonaws \ --ssl-ca=rds-combined-ca-bundle.pem --ssl-verify-server-cert

14 At-rest encryption DB instance storage Automated backups Read Replicas Snapshots Available for all six engines No additional cost Support compliance requirements

AWS KMS RDS standard encryption Two-tiered key hierarchy using

KMS master keys encrypt data keys Customer Master Key(s)

performance for encrypting large data Easier to manage small

access and audit of key activity Data Key 1 Data Key 2 Data Key

15 AWS KMS RDS standard encryption Two-tiered key hierarchy using envelope encryption Unique data key encrypts customer data AWS KMS master keys encrypt data keys Customer Master Key(s) Benefits: Limits risk of compromised data key Better performance for encrypting large data Easier to manage small number of master keys than millions of data keys Centralized access and audit of key activity Data Key 1 Data Key 2 Data Key 3 Data Key 4 Amazon S3 Object Amazon EBS Volume Amazon Redshift Cluster Custom Application

16 Enabling encryption AWS Command Line Interface (AWS CLI) aws rds create-db-instance --region eu-west-1 --db-instanceidentifier sg-cli-test \ --allocated-storage 20 --storage-encrypted \ --db-instance-class db.m4.large --engine mysql \ --master-username myawsuser --master-user-password myawsuser aws rds create-db-instance --region eu-west-1 --db-instanceidentifier sg-cli-test1 \ --allocated-storage 20 \ --storage-encrypted \ --kms-key-id xxxxxxxxxxxxxxxxxx \ --db-instance-class db.m4.large --engine mysql \ --master-username myawsuser --master-user-password myawsuser

17 Amazon RDS + AWS KMS useful hints You can only encrypt on new database creation Encryption cannot be removed Master and Read Replica must be encrypted Unencrypted snapshots cannot be restored to encrypted DB Cannot restore MySQL to Aurora or Aurora to MySQL You can now copy encrypted or unencrypted snapshots across regions

18 IAM managed access You can use AWS Identity and Access Management (IAM) to control who can perform actions on RDS Controlled with database grants Controlled with IAM Applications Users and DBA Applications DBA and Ops Your database RDS

19 IAM policies for RDS Policies Read Only "Action": [ "rds:describe*", "rds:listtagsforresource", "ec2:describeaccountattributes", "ec2:describeavailabilityzones", "ec2:describesecuritygroups", "ec2:describevpcs, "cloudwatch:getmetricstatistics", "logs:describelogstreams", "logs:getlogevents" ], "Effect": "Allow", "Resource": "*" Full Access "Action": [ "rds:*", "cloudwatch:describealarms", "cloudwatch:getmetricstatistics", "ec2:describeaccountattributes", "ec2:describeavailabilityzones", "ec2:describesecuritygroups", "ec2:describesubnets", "ec2:describevpcs", "sns:listsubscriptions", "sns:listtopics", "logs:describelogstreams", "logs:getlogevents" ], "Effect": "Allow", "Resource": "*"

20 NEW: IAM DB auth for MySQL and Aurora You can now also use AWS Identity and Access Management (IAM) to control access to the database Controlled with IAM Applications Users and DBA DBA and Ops Your database RDS

21 IAM DB Auth for MySQL and Amazon Aurora 1. Create RDS DB instance with IAM DB auth enabled 2. Create the user in the DB 3. Attach an IAM policy to the IAM user or role 4. Get an authentication token 5. Connect to DB using IAM DB auth For more details: MDBAuth.html

22 RDS Deep Dive Steve Blake - CTO steve.blake@sportpursuit.co.uk

23 SportPursuit business overview Founded in 2011 Flash sale business Unbeatable deals on sports and outdoor gear from the world s leading sports brands Mission Give access to and inspire sports enthusiasts to discover gear that they ll fall in love with 3.5m+ Members the UK s largest private shopping club for sports & outdoor enthusiasts of the world s best sports brands / 40% of which are on-uk 7 Languages / 8 Currencies / Shipping to 40+ countries 40% YoY growth

24 2-way proposition: brands & customers - Marketing to huge audience of sports enthusiasts - Channel for clearing excess stock, without compromising brand identity - Access to the best discounts on sports and outdoor gear from the world s leading sports brands

25 Technical architecture 2011 e-commerce Platform MySQL Database EC2 EC2

26 Technical architecture 2017 Cloudfront Varnish e-commerce Platform ERP Platform Memcache / Redis ELB Application Server Redis RDS - PostgreSQL ELB API Elasticsearch RDS - MySQL Redshift RDS MySQL Analytics Platform

27 RDS estate statistics ~ 25 RDS instances (incl. replicas) Volume of RDS data Production environment: ~2TB (incl. replicas) Entire estate: ~7TB Throughput of data e-commerce platform Peak: ~100 MB/S Average: ~30 MB/S

28 Challenges before RDS Manual database administration Creating slave replicas Solving replication errors Version upgrades Backups Time consuming as amount of data increase Refreshing staging environments Time consuming: export -> anonymise -> create -> import

29 RDS positives Reduction in manual database administration Create many read replicas with ease Painless version upgrades Multi-AZ for production Daily automated backups out of the box Weekly refresh of staging environments Automated snapshot -> anonymise -> restore Feature request: anonymisation as a service - serverless

30 Lessons learned Fix replication errors on slaves/replicas ASAP Binary log disk usage Don t use MyISAM tables Restrictions in sub-accounts Can t use automated snapshots Database creation: automated -> manual -> restore

31 Sticking points Adjusting to cloud philosophy Destroy & re-create failing component: Pets / Cattle Mindset change Health of the overall platform, not on a single element Time taken to make changes / upgrades Reboot required for option group changes

32 Upcoming projects Aurora Plan to benchmark against Aurora vs RDS MySQL Test to ensure compatibility with e-commerce platform BI / reporting RedShift for enterprise data warehouse Data Pipeline for data ingestion

33 BI / Reporting Data warehouse Redshift Board BI tool Python Django Data Pipeline RDS > Redshift Python Django API integration to 3 rd parties

34 Thank you! Steve Blake - CTO steve.blake@sportpursuit.co.uk

35 Migrating to RDS

36 Historically, Migration = Cost, Time Commercial data migration and replication software Complex to setup and manage Legacy schema objects, PL/SQL or T-SQL code Application downtime

37 Database Migration 2 Steps

38 Step 1: Schema Conversion Tool Overview

39 ü Move data to the same or different database engine AWS Database Migration Service ü Keep your apps running during the migration ü Start your first migration in 10 minutes or less ü Replicate within, to, or from Amazon EC2 or RDS

Keep your apps running during the migration Customer

Connect to source and target database Select tables,

Database Migration Service create tables, load data, and

40 Keep your apps running during the migration Customer premises VPN AWS Internet Start a replication instance Connect to source and target database Select tables, schemas, or databases Application Users Let the AWS Database Migration Service create tables, load data, and keep them in sync Switch applications over to the target at your convenience

41 Flexible migration approach Target Multiple targets Source Replication instance Target Target Source Multiple sources Source Replication instance Target Source Source Target Selective Replication instance instance L

42 Metrics and monitoring

43 Accessing Amazon RDS metrics

44 Amazon RDS standard metrics Change Time Period 45 Metrics Dive Deeper Create Alarms

45 Amazon RDS Enhanced Monitoring Access to over 50 metrics in 7 categories: Memory, I/O, CPU, File system, Load, Swap Processes

46 Amazon RDS Event Notifications Get notified when events occur on your database instances 17 different event categories (availability, backup, configuration change and so on) Uses Amazon Simple Notification Service (Amazon SNS)

47 Scaling on RDS

Bring data close to your customer s applications in different regions Promote a Read Replica

48 Scale out with Read Replicas Relieve pressure on your master node for supporting reads and writes. Bring data close to your customer s applications in different regions Promote a Read Replica to a master for faster recovery in the event of disaster Replicas within and crossregion MySQL, MariaDB, PostgreSQL Aurora Engines Needing Other Tools Oracle Microsoft SQL Server

49 Creating and promoting Read Replica Read Replica creation and promotion are accessed from the Instance Actions button in the RDS console

50 Creating and promoting Read Replicas with CLI create-db-instance-read-replica--db-instance-identifier <value> --source-db-instance-identifier <value>

51 Creating and promoting Read Replicas With CLI create-db-instance-read-replica--db-instance-identifier <value> --source-db-instance-identifier <value> [--db-instance-class <value>] [--availability-zone <value>] [--port <value>] [--auto-minor-version-upgrade --no-auto-minor-version-upgrade] [--iops <value>][--option-group-name <value>] [--publicly-accessible --no-publicly-accessible] [--tags <value>][--db-subnet-group-name <value>] [--storage-type <value>] [--copy-tags-to-snapshot --no-copy-tags-to-snapshot] [--monitoring-interval <value>] [--monitoring-role-arn <value>] [--kms-key-id <value>] [--pre-signed-url <value>] [--enable-iam-database-authentication --no-enable-iam-databaseauthentication] [--source-region <value>] [--cli-input-json <value>] [--generate-cli-skeleton <value>]

52 Scaling up and down Handle higher load or lower usage Control costs

53 Scaling Up and Down Console

54 NEW Stop & Start DB Instances

$stop-db-instance \$

55 Stop your RDS database instance aws rds stop-db-instance \ --db-instance-identifier mydbinstance

56 and start it again aws rds start-db-instance \ --db-instance-identifier mydbinstance

57 Backups and snapshots

not with multi-az deployment) Multiple copies in each AZ where you have instances for a deployment Aurora

58 RDS backups MySQL, PostgreSQL, MariaDB, Oracle, SQL Server Scheduled daily backup of entire instance Archive database change logs Up to 35 day retention for backups I/O suspension as backup is initiated (but not with multi-az deployment) Multiple copies in each AZ where you have instances for a deployment Aurora Automatic, continuous, incremental backups Point-in-time restore No impact on database performance 35 day retention

59 RDS restore Restoring creates an entire new database instance You define all the instance configuration just like a new instance

Snapshots Full copies of your Amazon RDS database that are different from your scheduled backups Backed by Amazon S3 Typical use cases Resolve production

60 Snapshots Full copies of your Amazon RDS database that are different from your scheduled backups Backed by Amazon S3 Typical use cases Resolve production issues Nonproduction environments Point-in-time restore Final copy before terminating a database Disaster recovery Cross-region copy Copy between accounts

61 High availability

62 Minimal deployment: Single AZ / /24 Amazon Elastic Block Store Volume Availability Zone AWS Region

63 High availability: Multi-AZ / /24 Same instance type as master /24 Replicated storage Availability Zone A Availability Zone B AWS Region

64 High availability Multi-AZ to DNS dbinstancename us-west-2.rds.amazonaws.com:3006

Aurora high availability Aurora cluster contains primary node and up to 15 secondary nodes Failing database nodes are automatically detected and replaced Failing database processes are automatically

65 Aurora high availability Aurora cluster contains primary node and up to 15 secondary nodes Failing database nodes are automatically detected and replaced Failing database processes are automatically detected and recycled Secondary nodes automatically promoted on persistent outage, no single point of failure Customer application can scale out read traffic across secondary nodes AZ 1 AZ 2 AZ 3 Primary Primary Node Primary Node Node Primary Primary Secondary Node Node Node Primary Primary Secondary Node Node Node Amazon S3

WITH MARIADB DRIVER Driver benefits DB Failure Failure

66 Aurora-DNS Failover MYSQL DB Failure Failure Detection DNS Propagation App Running sec Recovery Recovery AURORA WITH MARIADB DRIVER Driver benefits DB Failure Failure Detection DNS Propagation sec Recovery 5-20 sec App Running

Cloud Computing /AWS Course Content

Cloud Computing /AWS Course Content 1. Amazon VPC What is Amazon VPC? How to Get Started with Amazon VPC Create New VPC Launch an instance (Server) to use this VPC Security in Your VPC Networking in Your