Migrating and living on RDS/Aurora. life after Datacenters

Transcription

1 Migrating and living on RDS/Aurora life after Datacenters

2 Why to migrate to RDS - It is AWS native - A lot of complexity is handled by Amazon - It is Someone Else s Problem (SEP ) - You have someone to blame - It just works!

3 Why not migrate to RDS - It s not always the same approach as you d choose - You can t access certain parts of your system - When you have your own enlightned toolset which you want to use - You are not happy to use a black box system, when you just hope it will work somehow

4 What is Aurora

5 RDS/Aurora - MySQL compatible HA database cluster - No replication between the nodes, they share a common EBS storage, and the redo logs got shared - This means there s no real replication lag between the write node and the readers, in our case it is between around 12-16ms - The cluster has a writer and a reader endpoint, the writer is not getting read

6 Creating cluster

7 Using the web interface - DON T! - Just don t. - It is not reproducible - There is no clean code you can read and handle to someone else

8 Awscli / boto3 - Amazon native - Flexible - Relatively easy - Not the most reusable

9 Puppet/ Chef / Ansible We tried to use puppet, but it wasn't a success - A lot of recreations happened instead of modify - The newest API was not supported - It worked, but with limitations

10 Terraform - No vendor lock - It needs an Atlas server to work properly - We had bad experiences - Constantly changes - Not always the latest API in use - It always recreated and destroyed the cluster instead of modifying

11 CloudFormation - Native Amazon solution - Could be written in YAML or in JSON - Works with changesets - The Amazon infrastructure fully supports it - With CodePipeline you can create a real neat pipeline. :(

12 Node sizes - Always check Amazon s node parameters, because with MySQL not often the CPU is the bottleneck, but the storage speed (Aurora using EBS) or the network could be!

13 Loading data

14 MySQLDump - --single-transaction - --master-data=2 - --hex-dump - Don t forget the interactive_timeout parameter! - When uploading to s3 you have to take care of splicing up

15 Mydumper / Myloader - Parallel dumper, way faster than mysqldump (in my case the data loading was around 5hrs instead of 26! - No --single-transaction, so you have to take care of it - My recomendation: do it on a replica, because you ll have the binlog file name and position as well (STOP SLAVE; mydumper ; START SLAVE)

16 S3 Since 1.8 you are able to load data directly from s3 with LOAD DATA FROM S3 command. (These command is similiar to LOAD DATA INFILE regarding the usage and data formats. To create a compatible dump, use SELECT INTO OUTFILE ) More info:

17 MySQLPump It s like mysqldump but able to work parallel The load time will not affected, because the data load is still single-threaded - in exchange, it can do --single-transaction dump

18 Compatibility

19 MariaDB? - No, it is not. - It seems they forked MariaDB (version?) but since not all the feature changes were implemented, there s no guarantee that it will work - For example, we had to break GTID replication, because MariaDB don t uses the same GTID implementation as native MySQL could, but it can fail back there - theoretically. Aurora can t.

20 Time Zones - Aurora databases are in UTC time zone - You can change it - But if you do it, and replicates to an external db be aware if the mysql timezones are installed - this can break replication l.html

21 Replication

22 Not designed to replicate - You can replicate, in or out, but I don t recommend it - My recommendation is to use replication only for migration - It s not crash safe - I had broken and unfixable replication because of a node restart (just because a config change happened!)

23 RDS managed tables in MySQL schema - They are not appear in a mysqldump - When you create a new host, create these tables manually, (SHOW CREATE TABLE ) or your replication will break

24 Aurora (RDS) as master - Check the hostname of the cluster before setting up replication, it might be too long for data dictionary (in mysql 5.6 there are a VARCHAR(60) field hostnames - Just create a CNAME record, or use ip address instead - I never tested what happens during/after a failover

25 Time Zones - Aurora databases are in UTC time zone - You can change it - But if you do it, and replicates to an external db be aware if the mysql timezones are installed - this can break replication l.html

26 Security & Access

27 Internet facing cluster - Try to not expose it on internet - If so, always use SSL - You can force SSL connection with mysql - GRANT USAGE ON *.* TO 'someuser'@'%' REQUIRE SSL;

28 Management node - It is good to have a management node where you can access your database - Preconfigured work environment, plenty of disk space for dumps, preconfigured connection parameters etc. - Shouldn t be an expensive node (unless you want to utilize a lot of CPU or disk IO - Should be spin up quickly and be destroyed when not needed (hey, this is cloud, right?)

29 IAM Users - You can authenticate IAM users, and grant privileges inside of RDS but WHY WOULD YOU DO THIS?! - I think it is way better to separate db users from IAM users - Yes, this applies to AD as well

30 Create your own users - Don t use the cluster s own user just for administrative purposes - Restrict your app users* - Separate Read/Write and administrative users * This is not Aurora/RDS only!

31 Scaling Aurora

32 Native autoscaling? - Sadly there is no autoscaling built in, however it would be nice to have one - You can create your own one - You can add node with API - You can delete node with API - You can have events generated by CPU usage or number of queries etc.

33 Cluster addresses - One dns address for read/write - One dns address for read - Each hosts has it s own dns name - If you remove a node, it s address will be in dns for 60 seconds (default TTL for Aurora cluster addresses)

34 Bring your own Autoscaler (BYOA) - Endless possibilities! - Haproxy/ProxySQL/DNS frontend - My solution is a simple dns based autoscaler, it manages the cluster address in Route53, when I add a node it creates a node, and puts the nodes address to dns when it is available, when I remove a node, it cleans it out from dns first, and after ttl starts to delete the node

35 Living without SUPER

36 Stored procedures (a lot) mysql.rds_set_external_master mysql.rds_reset_external_master mysql.rds_start_replication mysql.rds_stop_replication mysql.rds_skip_repl_error mysql.rds_next_master_log mysql.rds_innodb_buffer_pool_dump_now mysql.rds_innodb_buffer_pool_load_now mysql.rds_innodb_buffer_pool_load_abort mysql.rds_set_configuration mysql.rds_show_configuration mysql.rds_kill mysql.rds_kill_query mysql.rds_rotate_general_log mysql.rds_rotate_slow_log mysql.rds_enable_gsh_collector mysql.rds_set_gsh_collector mysql.rds_disable_gsh_collector mysql.rds_collect_global_status_history mysql.rds_enable_gsh_rotation mysql.rds_set_gsh_rotation mysql.rds_disable_gsh_rotation mysql.rds_rotate_global_status_history

37 Restricted commands - SET GLOBAL READ_ONLY = 0/1 - This is managed by RDS

38 Read-only variables - All variables which needs SUPER to change acts like read-only variables - You can change them in the parameter group - And restart the node to the change take place

39 Configuration

40 DB parameter group/cluster parameter group - Nodes can have different DB parameter groups, not all of them has to be the same (Like read_only) - Cluster parameter groups have options which has to be the same along the entire cluster (Like timezone, character encoding)

41 Read only parameters and cloudformation - When you change a read only parameter, the cluster node gets in a pending change state and you can reboot it manually to apply the changes - When the cluster/node has associated parameter group in cloudformation, and you change a read only value, your node will be rebooted automatically - To avoid this, don t associate parameter groups with instances/clusters via cloudformation - You can have blue/green param group pairs

42 Backup

43 Automated backups - They re created via snapshotting the instances - It is in S3 as backend, but you can t access it as object, just from RDS - You can t keep them forever - You can t set retention policy and storage class - Quick -> Slow -> Long term (Glacier)

44 Long term backups - Create logical dumps (mysqldump/mysqlpump/mydumper/select INTO OUTFILE, SELECT INTO S3) - You can provision a host for this, do the backup, and copy it to s3, and remove the backup host (awscli/boto3) - You can set up storage class for the long term backups - WARNING: Glacier is freaking expensive if you want to access your backup. No joke.

45 ALWAYS TEST YOUR BACKUPS IF YOUR BACKUP IS NOT TESTED WITH RESTORE, THAT BACKUP IS A NON-EXISTENT BACKUP. PERIOD. IF YOUR BACKUP IS NOT TESTED WITH RESTORE, THAT BACKUP IS A NON-EXISTENT BACKUP. PERIOD. IF YOUR BACKUP IS NOT TESTED WITH RESTORE, THAT BACKUP IS A NON-EXISTENT BACKUP. PERIOD.

46 Monitoring & Logging

47 Cloudwatch - You can access host metrics from cloudwatch - like CPU, Memory, Disk IO - Not all the MySQL metrics are accessible - You can create nice dashboards - As well as nice alerting based on events

48 Third party monitoring tools - Vividcortex - PMM - Newrelic These tools can connect to mysql port, and collect metrics via performance_schema. Check their documentation about the details, be security-aware. If you can monitor via a host or a container I believe the second one is better. (KISS)

49 Monitoring - Cloudwatch is good, but not all the metrics can be reached by it. - Vividcortex should connect via mysql protocol using performance_schema

50 Logging - Hell. - You can access them from the web interface - You can access them with awscli - (maybe there s a logger backend somewhere?)

51 Thank you for your time!