PostgreSQL migration from AWS RDS to EC2

Size: px

Start display at page:

Download "PostgreSQL migration from AWS RDS to EC2"

Poppy Lester
5 years ago
Views:

1 PostgreSQL migration from AWS RDS to EC2

2 Technology lover Worked as Software Engineer, Team lead, DevOps, DBA, Data analyst Sr. Tech Architect at Coverfox me at Tweet me Everything is possible but, it takes time - Hitul Mistry

4 Database Evolution Self hosted database AWS RDS Why we migrated from Database As Service to self hosted database Challenges in migration How we planned migration and point to be noted Current Architecture Demo

6 Simple GUI HA, Multi AZ. Encryption, Backup, Recovery, Disaster recovery, Security, Compliance SLAs Performance optimizations by self GUI for version upgrades

7 AWS RDS Postgresql DB functionality is similar to postgresql Cannot install extra extensions other then provided Cannot do replication to self hosted server Cannot install custom plugin to logical decoding Cannot install custom data-types Upgrades can be done at few clicks on GUI Self hosted PostgreSQL Postgresql DB functionality will be as postgresql original behaviour 100% control on functionality Upgrade needs to be done manually

8 AWS RDS HA, Fault Tolerance, Disaster recovery, Backups can be implemented at few GUI clicks You will have to monitor common parameters when postgresql can crash. It can crash when disk is full or CPU usage is high or other parameters. Postgresql will be auto restarted, disk is full. SLA Self hosted PostgreSQL HA, Fault Tolerance, Disaster recovery, Backups can be implemented by configuring postgresql by self We will have to monitor all the threats which can occur and be ready to fix those things Upgrades can be done by self SLA

9 AWS RDS Everything is available in GUI Postgresql usage knowledge and some architectural knowledge required AWS controlled performance tuning; not use-case dependent Self hosted PostgreSQL Postgresql Expert knowledge required Everything needs to be done by setting up configs

10 AWS RDS Few performance parameters can be tuned via GUI Cannot use custom hardware for performance Self hosted PostgreSQL Performance can be tuned via basic postgresql config and also other parameters like kernel, os, disk etc. Lot of performance parameters available to tune as per the application

11 AWS RDS To identify fault in postgresql you will be provided GUI where all the postgresql logs will be printed. New version upgrades can be done with few clicks Cannot go deep beyond DAAS service provides Self hosted PostgreSQL Can directly see logs of postgresql Can go deep as much want to go

12 AWS RDS Operating environment cannot be changed Self hosted PostgreSQL It can be moved to any operating environment

13 Cost to scale vertically or horizontally on aws is high Many open-source plugins required by the application cannot be installed on RDS New Logical decoding plugin for replication or other use-cases is not supported by RDS AWS takes time to upgrade to latest postgres versions Almost zero downtime server upgrades possible with self hosting Database auto scaling Performance tuning as per application needs

14 AWS RDS Cost (On Demand) Instance Type CPU RAM(GiB) Pricing/Yr m4.2xlarge 8 32 $ m4.4xlarge $ m4.10xlarge $ AWS EC2 Cost (On Demand) Instance Type CPU RAM(GiB) Pricing/Yr m4.2xlarge 8 32 $ m4.4xlarge $ m4.10xlarge $ 18396

15 If you we buy multi-az setup then cost will be doubled Reserved instance can save cost from 12-64% Rack servers and different cloud infrastructures usage for cost cutting Zero Downtime Upgrades Migration required extra hands, but self-hosted maintenance has not increased load on DevOPs team!

16 RDS supports limited plugins. Just now they added wal2json. Database Operation INSERT INTO data(data) VALUES('1'); INSERT INTO data(data) VALUES('2'); Format inserts table public.data: INSERT: id[integer]:1 data[text]:'1' table public.data: INSERT: id[integer]:2 data[text]:'2'

17 BEGIN table public.core_tracker: UPDATE: id[bigint]: session_key[text]:'w84fhz6c8b5jpc1ufesnegbxrfmnehh8' user_id[bigint]:23573 extra[text]:'{"h":100,"no_show":true}' created[timestamp with time zone]:' :03: :30' fd_id[integer]:null COMMIT BEGIN table public.core_tracker: UPDATE: id[bigint]: session_key[text]:'w84fhz6c8b5jpc1ufesnegbxrfmnehh8' user_id[bigint]:23573 extra[text]:'{"h":100,"no_show":true,"hello-world":{"sfs":"sdf\"2''3"}}' created[timestamp with time zone]:' :03: :30' fd_id[integer]:null COMMIT 89285

22 5M unique quotes a month 45M unique quotes from insurance companies 5GB write on DB and logs combined

23 Downtime allowed Daily: 8.6s Weekly: 1m 0.5s Monthly: 4m 23.0s Yearly: 52m 35.7s

24 Almost zero downtime Stable database after migration and should work as older one SLAs 99.99

25 Shared disk failover File system level Transaction log(wal) Trigger based Statement based

26 Most Reliable We cannot access pg_hba.conf You don t have enough permission to execute pg_start_backup

27 PgBadger CURRENT_TIMESTAMP, random, sequences will be affected Huge change in codebase

28 Postgresql utility to create postgresql dump and restore it Time consuming

29 Functions, Indexes, Constraints, are not migrated JSON considered as CLOB and truncated values Varchar, character varying values truncated DDL ignored Does not replicate partitioned tables

30 Does not replicate DDL Tough to parse output and output is not standard

31 Documentation and support was really limited If tools failed then whole replication will fail

32 All tables must have modified date All the tables must have primary key but, some tables had non non numeric primary keys

33 Mongodb + GoLang + Trigger based replication + pg_listen + pg_notify

34 Disable foreign key validations on self hosted postgresql db Create triggers on AWS RDS database Take backup of postgresql RDS

37 { MongoDB Schema "table_name":"schema_name.table_name", "primary_key":"{primary_key_value}", "created_at":"timestamp", "operation":"insert/update/delete" }

40 Reset sequence Enable foreign key validations Stop AWS RDS instance Run basic Sanity scripts which will verify data on sample Stop website and it can be opened from internal users only Run QA tests Take backup of AWS RDS

44 How much downtime can be accepted? SLA What is the worst thing that can happen? Services which can be affected? How soon we can recover? How much data will be lost and can be recovered? What will be long term ROI? What data will be affected?

45 Plan should be like steps to execute Execute plan once on sandbox environment before going live Plan should include rollback strategy

46 CPU Disk type Disk size Connections Future plans and traffic RAID Power usage Network

47 Buffer size Kernel parameters Work memory Checkpoint

49 Our rollback strategy was similar Logical Decoding replication for 2 weeks

50 Write deployment scripts to setup database Write script for things as much you can

51 Sanity scripts QA team scripts and approval

54 High availability Fault tolerance Disaster recovery Backup & Recovery Hardware & Software updates Security Monitoring Testing Rollback Compliance

56 Promote master pg_ctl promote -D /data-dir-path/ Add cluster to PostgreSQL rm -rf /data-dir-path/ && pg_basebackup. Run pg_rewind after promote pg_rewind -D /data-dir-path --source-server=... host=...

57 Service discovery is the automatic detection of devices and services offered by these devices on a computer network. - Wikipedia

59 Service Discovery Health Checking KV store Mutli Data Center

60 Fork of Governer Developed at Zalando Used with Consul, ZooKeeper, Etcd

64 Image:

67 service/coverfox/optime/leader service/coverfox/members/master-a Leader node name Member of cluster service/coverfox/members/master-b Member of cluster

70 Errors data corruption system failure (including hardware failure) human error natural disaster

71 Tool for disaster recovery, backups and recovery by 2ndcondrant Remote backup Remote restore WAL Logs recovery

76 Barman Backup /usr/bin/barman backup --jobs 6 mumbai-master-a Barman restore db barman recover --target-time " :22:00" --remote-ssh-command "ssh postgres@x.x.x.x" mumbai-master-c T /pg-data-dir-path/ -j 10

77 What to monitor? RAM/CPU Usage DIskIO Process info Bandwidth Vacuum running DB Space Active Connections Active Transactions Open files Replication diff PgBouncer client connections PgBouncer stats

78 Grafana InfluxDB Twilio Slack

PGConf.Russia 2019, Moscow. Alexander Kukushkin

PGConf.Russia 2019, Moscow. Alexander Kukushkin Типичные ошибки при построении высокодоступных кластеров и как их избежать PGConf.Russia 2019, Moscow Alexander Kukushkin 06-02-2018 ABOUT ME Alexander Kukushkin Database Engineer @ZalandoTech The Patroni