MySQL usage of web applications from 1 user to 100 million. Peter Boros RAMP conference 2013

Transcription

1 MySQL usage of web applications from 1 user to 100 million Peter Boros RAMP conference 2013

2 Why MySQL? It's easy to start small, basic installation well under 15 minutes. Very popular, supported by a lot of frameworks. Development focus is under the engine cover, not necessarily SQL level features. Keeps up with performance requirements, very large deployments exist.

3 Agenda The architecture, which is the best for every application (the silver bullet) Backups Online schema changes High availability and scalability Sharding basics

4 Agenda The architecture, which is the best for every application (the silver bullet) Backups Online schema changes High availability and scalability Sharding basics

5 Backups: early days Mysqldump Single threaded logical backup Mydumper Table level parallelism Easy to implement, logical backups are easy to manage, restore takes a of time. Capable of point in time recovery

6 Logical backup tuning Parallel backup Table level Parallel SELECT INTO OUTFILE Restore tables with only primary keys, add secondary keys later In case of MyISAM disable indexes

7 Binary backups When logical backup takes too long My daily backup takes almost 30 hours When mean time to restore matters Snapshot based backups Volume manager, intelligent storage, EBS Percona Xtrabackup

8 Copy on write snapshot internals O O O O O O O O O O O: original block F: free block lvcreate -s... F F F F F F F F F F Initially snapshot is empty and it doesn't take up disk space

9 LVM snapshot internals Write changed content C to one of the blocks O C O O O O O O O O O O: original block F: free block C: changed block lvcreate -s... F O F F F F F F F F F Changed original content (O) of the block is written to the snapshot

10 Snapshot performance

11 XtraBackup Pages in datafile O O O O O O Backup O O Redo log copied continously O: original page C: changed page during backup Blue: copied by XtraBackup

12 XtraBackup Pages in datafile O O O O O O Backup O O O C O O O O C Redo log copied continously LSN x, C(1) O: original page C: changed page during backup Blue: copied by XtraBackup

13 XtraBackup Pages in datafile O O O O O O C O O O O C Backup O O O O O C apply-log, roll forward C O O O O C Redo log copied continously LSN x, C(1) LSN y, C(2) C O O O O C O: original page C: changed page during backup Blue: copied by XtraBackup

14 Agenda The architecture, which is the best for every application (the silver bullet) Backups Online schema changes High availability and scalability Sharding basics

15 Early days Either no high availability (relying on backups), or basic high availability with shared storage or DRBD + pacemaker MySQL's built-in replication: high availability and scaling reads

16 Some other HA options Dual master with manual failover Replication managers MMM, MHA, PRM Percona XtraDB Cluster

17 Pacemaker + DRBD Storage block level replication Can be synchronous and asynchronous (using asynchronous here is not recommended) Write IO response time increased The standby server is cold MySQL is not running, the data storage is not mounted Crash recovery at non-graceful failover

18 Dual masters with manual failover Popular because of simplicity Database runs on both instances Database cache is cold on the standby instance If both instances are written, you have to deal with conflict resolution auto_increment_increment and auto_increment_offset doesn't resolve every conflict (think about UPDATEs)

19 Cold cache issue If the buffer pool is cold on the standby side Server becomes IO bound It could mean that the database is unusable for a while This can increase failover time How we solved this by continuously keeping standby's cache warm

20 How we solved Groupon

21 Workload warmed up with itself

22 Workload warmed up with the previous chunk

23 Replication managers Introducing an automation layer on top of replication Automating VIP switch (if VIP exists) Automating choosing the slave to promote Automating repositioning the other slaves Cold cache issue can make things really worse when failover is automated

24 Replication manager topology I. Used by MMM Dedicated standby master

25 Replication manager topology II. Used by PRM/MHA On master failure, the most recent slave is promoted to master

26 Replication and durability MySQL's built-in replication is asynchronous Binary logs are written to disk in an asynchronous fashion sync_binlog From MySQL 5.5: semi-sync replication guarantees that at least one slave has the data in the relay log, when the client gets commit acknowledgement

27 Replication issues Replication is asynchronous, data can be different on the master and slaves A user with SUPER privilege can write a database with read_only=1 Verifying integrity: pt-table-checksum Fixing replication: pt-table-sync Syncs based on pt-table-checksum's results table New pt-table-checksum in Percona Toolkit 2.x, uses resources in a smart way

28 Replication integrity check ~]# pt-table-checksum h=localhost,u=percona,p=percona --chunk-size=10000 TS ERRORS DIFFS ROWS CHUNKS SKIPPED TIME TABLE 07-10T09:03: mysql.columns_priv 07-10T09:03: mysql.db 07-10T09:03: mysql.event 07-10T09:03: mysql.func 07-10T09:03: mysql.help_category 07-10T09:03: mysql.help_keyword 07-10T09:03: mysql.help_relation 07-10T09:03: mysql.help_topic 07-10T09:03: mysql.host 07-10T09:03: mysql.ndb_binlog_index 07-10T09:03: mysql.plugin 07-10T09:03: mysql.proc 07-10T09:03: mysql.procs_priv 07-10T09:03: mysql.proxies_priv 07-10T09:03: mysql.servers 07-10T09:03: mysql.tables_priv 07-10T09:03: mysql.time_zone 07-10T09:03: mysql.time_zone_leap_second 07-10T09:03: mysql.time_zone_name 07-10T09:03: mysql.time_zone_transition 07-10T09:03: mysql.time_zone_transition_type 07-10T09:03: mysql.user 07-10T09:04: sbtest.sbtest1

29 Replication integrity check mysql> update sbtest1 set c='nasty_update' order by rand() limit 1; Query OK, 1 row affected, 2 warnings (0.11 sec) Rows matched: 1 Changed: 1 Warnings: 2 [root@ps1 ~]# pt-table-checksum h=localhost,u=percona,p=percona --chunk-size= databases=sbtest TS ERRORS DIFFS ROWS CHUNKS SKIPPED TIME TABLE 07-10T09:07: sbtest.sbtest1 mysql> SELECT db, tbl, SUM(this_cnt) AS total_rows, COUNT(*) AS chunks -> FROM percona.checksums -> WHERE ( -> master_cnt <> this_cnt -> OR master_crc <> this_crc -> OR ISNULL(master_crc) <> ISNULL(this_crc)) -> GROUP BY db, tbl; db tbl total_rows chunks sbtest sbtest row in set (0.00 sec)

30 Replication integrity check mysql> select * from percona.checksums where tbl='sbtest1' and this_crc!=master_crc\g *************************** 1. row *************************** db: sbtest tbl: sbtest1 chunk: 2 chunk_time: chunk_index: PRIMARY lower_boundary: upper_boundary: this_crc: 897f461b this_cnt: master_crc: 49b06b87 master_cnt: ts: :07:12 1 row in set (0.00 sec) mysql> select * into outfile '/tmp/sbtest1.master.tsv' from sbtest1 where id between and 20000; Query OK, rows affected (0.02 sec) mysql> select * into outfile '/tmp/sbtes1.slave.csv' from sbtest1 where id between and 20000; Query OK, rows affected (0.02 sec)

31 Replication integrity check tmp]# diff sbtest1.master.tsv sbtest1.slave.tsv 2881c2881 < > nasty_update mysql> show create table sbtest1\g *************************** 1. row *************************** Table: sbtest1 Create Table: CREATE TABLE `sbtest1` ( `id` int(10) unsigned NOT NULL AUTO_INCREMENT, `k` int(10) unsigned NOT NULL DEFAULT '0', `c` char(120) NOT NULL DEFAULT '', `pad` char(60) NOT NULL DEFAULT '', PRIMARY KEY (`id`), KEY `k_1` (`k`) ) ENGINE=InnoDB AUTO_INCREMENT= DEFAULT CHARSET=latin1 MAX_ROWS= row in set (0.00 sec)

32 Percona XtraDB Cluster Real parallel, synchronous replication Using write set replication Based on academic research, state of the art replication Galera: wsrep provider library made by Codership Percona XtraDB Cluster: Percona Server + Galera

33 PXC Single data center Any node is writable Application server typically use some load balancer to connect No VIPs needed Each node sends each write set to all the other nodes in parallel Bidirectional arrows represent group communication

34 PXC virtually synchronous replication

35 Multi data center PXC Real nodes in every DC 2 nodes + arbitrator

36 PXC common caveats InnoDB only MyISAM support is experimental On node failure, writes are stalled for suspect timeout Conflicting transactions can be an issue when writing multiple nodes Long running transactions are more painful than in the single node case Always test carefully with your workload!

37 PXC common caveats InnoDB only MyISAM support is experimental On node failure, writes are stalled for suspect timeout Conflicting transactions can be an issue when writing multiple nodes Long running transactions are more painful than in the single node case Always test carefully with your workload!

38 Agenda The architecture, which is the best for every application (the silver bullet) Backups Online schema changes High availability and scalability Sharding basics

39 Possible methods Using replication, and altering boxes one by one Using pt-online-schema-change Using wsrep_osu_method='rsu' in PXC Built-in method in Percona XtraDB cluster

40 Online schema changes using replication ALTER the standby server (SET sql_log_bin=0) ALTER all the slaves Flip to a slave or to the passive master or promote something which has the new schema manually Flip the former active master

41 Agenda The architecture, which is the best for every application (the silver bullet) Backups Online schema changes High availability and scalability Sharding basics

42 Sharding types Table level sharding Geographical sharding Static sharding Dynamic sharding Static and dynamic

43 Sharding pros and cons Write scalability No cross-shard transactions No cross-shard joins Application level logic to implement and maintain

44 Static sharding Simple hash / modulo function No central dictionary needed What if we have to add a node?

45 Rebalancing shards Adding a new node Adding more powerful nodes Replacing old nodes with more powerful ones Change in usage patterns

46 Dynamic sharding Central dictionary service Knows where should data go and where data is Key-value store, easy to scale

47 Should I shard? If you don't know for sure, no. A lot of big applications are operating without sharding. Really big applications are doing a mix of static and dynamic sharding.

48 Help writes before sharding: archiving Archive less frequently accessed data More compact indexes Better cache utilization Faster writes A good archiving strategy based on data access patterns can postpone sharding

49 Help writes before sharding: partitioning Can be helpful if you have a single larger table A partitioned table is practically multiple physical tables under the hood Smaller tables -> smaller indexes -> faster writes Faster reads by partitioning function Slower reads if it's not by partitioning function It has it's limitations as well

50 Agenda The architecture, which is the best for every application (the silver bullet) Backups Online schema changes High availability and scalability Sharding basics

51 Silver bullet Doesn't exist.

52 Q&A

53 Thanks for attention.

54 See you next time