Scale out Read Only Workload by sharing data files of InnoDB. Zhai weixiang Alibaba Cloud

Transcription

1 Scale out Read Only Workload by sharing data files of InnoDB Zhai weixiang Alibaba Cloud

2 Who Am I - My Name is Zhai Weixiang - I joined in Alibaba in 2011 and has been working on MySQL since then - Mainly focus on InnoDB: bugfix, performance tuning, develop new features. - Contact Me: - zhaiwx1987@hotmail.com - Facebook Page: - Twitter: 2

3 Agenda - Background - The New Architecture - Deep Dive into implementation 3

4 Background 4

5 Background: Scale out read - If your system is completely read only: - Turn on --innodb-read-only - And start multiple instances with same data directory - But it s not suitable for general scenarios!!! Read Request Read Request Read Request Instance Instance Instance Innodb-read-only=1 Innodb-read-only=1 Innodb-read-only=1 Data 5

6 Background: Scale out read - A typical way to scale out read workload is to run SELECT statements on slave instance - Add a new slave to extend the read ability Read/Writ e Master Slave Data && Log Read Only Data && Log Slave Data && Log Read Only Slave Data && Log Read Only 6

7 Background: Scale out read But adding new slave is time consuming!! - Backup && restore a full copy of data - Need to apply delayed logs after restoring - Sometimes the SELECTs on slave is sensitive to replicating rate - Binary replication is not effective - It may slow down the master instance if there are too many dump requests 7

8 Background: New Goals Provide ability to: - Startup a new read only instance without copying data from master instance - The new instance can provide read-only services in less than 1(or more?) minutes - The master instance still provides all types of workload that a normal MySQL allows 8

9 Background: New Goals How to? - Code Base: InnoDB Storage engine, With physical Replication - Refer to my talk at last year s percona live: eplication-based-innodb - Update the cache of read-only node by replaying redo log - Make use of Shared Storage - Run multiple instances based on same copy of data && redo files 9

10 New Architecture: Overview Region 1 Ship redo log Region 2 Primary Standby Exchange state Read/Writ e Shared Storage Data Log Data Log Read/Writ e Read Read Read Replica Replica Replica Primary: Read/Write Standby: Read Only, has its own data set Replica: Read Only 10

11 New Architecture: Primary->Replica What is replica? - Only afford Read-Only ability, and the statements that may change data will be rejected - Share data/log files with the Primary Server. - No need to maintain flush list. - All Files are open in read only mode. 11

12 New Architecture: Primary->Replica A more detailed view with all involved threads: Primary Ack Receiver Msg Sender Exchange state Ack Sender Msg Receiver Replica wakeup wakeup Log-Parser thread Session1 Session 2 Group Commit LGWR Write redo log Shared Storage Log Data Read log/data Parse Log && store to hash table Hash table Read Worker thread Hash table Read Worker thread Page cleaners Flush dirty pages to disk Update Cache Update Cache Buffer pool 12

13 New Architecture: Primary->Replica Key Issues: - Control page flushing/eviction - Startup a replica node on fly - Support MVCC on Replica - How to Replay DDL - Failover between Primary and Replica 13

14 Deep dive into implementation 14

15 Control page flushing/eviction - Problem #1: Avoid reading a page which is being modified Primary Flush page to disk Updated 0 4kb Read a corrupted page 16kb Replica 15

16 Control page flushing/eviction - Problem #2: Avoid reading pages in future or in past Split Page Flush Page 1 to disk Primary Replica Data File Old.2 Old.3 New.1 Page 1-> 4 User client 16

17 Control page flushing/eviction Let s make some rules: - The replica applies all logs in memory, no matter if the page is in buffer pool or not. - The primary instance has to make sure that a page has been read into buffer pool of all connected replicas before flushing it to disk - The replica instance only allows to evict pages that the in-memory changes on replica have been flushed to disk by Primary 17

18 Control page flushing/eviction - Basic Algorithm: - Flush page on Primary - primary.block.oldest_modification <= replicas.smallest_applied_lsn() - The LSN on block will be transmitted to all replicas(disk.block.lsn) Page on Primary Firstly modified at LSN = 10 Log Applied LSN on Replica Can be evicted Can t be flushed Can t be evicted - Evict page from bp on Replica: - Replica.block.newest_modification = 0 - Or equal to disk.block.lsn LSN = 20 Flush page to disk Can be flushed Flush done Can be evicted - Drawback: - Page flushing on Primary is subject to the replication rate of all connected Replicas 18

19 Speedup Replication Rate - Way#1: Apply logs in parallel - Create multiple threads to apply log - hashed by fold(space_id, page_no) % Thread_CNT; Log parser Store Log to Hash Hash Table Hash Table read Hash Table read Worker Worker Apply Apply 19

20 Speedup Replication Rate - Way#2: Remove redundant log paring (Bug#82937) - InnoDB will first fully parse a log entry to find the end of it. And then store the log to hash - The log entry will be parsed again while applying it. - Fix: Add two bytes to store the entry length in log header Type Space ID Page No Log Entry Type Space ID Page No Length Log Entry 20

21 Speedup Replication Rate - Way#3: Reuse dummy index to save CPU times(bug#82176) - The dummy index is used for helping replay operations on b-tree - These structures can be reused if the Btree has same number of fields - FIXME: - Cache the created dummy index for each log-apply threads 21

22 Simulated Crash Recovery on Replica - The replica can t read from disk directly as the primary server may still has some dirty pages in buffer pool that hasn t been flushed to disk - Rebuild these pages in buffer pool before accepting requests Primary Make a checkpoint Register replica Connect to primary Oldest_LSN in buffer pool Newest_LSN Transaction State Log file info: (lsn, offset), file size Exchange state Exchange state Replica Initialize InnoDB - Read log from oldest_lsn, and apply up to newest_lsn - resurrect transaction state Initialize other components Accept user requests 22

23 After starting replica Exchange state 1. Latest Log LSN on disk 2. Flushed Page Set (Page_id, LSN) Primary Replica 1. Applied LSN 2. Oldest Read View 23

24 MVCC On Replica On Primary - Store ID of all uncommitted RW transactions in global array - Use undo log to build a consistent snapshot of the record is not visible On Replica - Should track uncommitted transactions - Don t clear undo log if it may still be referenced by a active read view 24

25 MVCC On Replica Resurrect Transaction state: - The initial transaction state is copied from Primary during crash recovery - Then update global transaction IDs array via redo log - MLOG_TRX_START: Mark the start point of a read-write transaction after assigning a trx_id, logged in order - MLOG_TRX_COMMIT: Mark the end point of a transaction after marking undo state as committed - Use a lazy-build strategy to resurrect global transaction state on replica: - The global transaction state is updated after applying one batch of redo log Max_trx_id = 11 Trx_ids={} Read View Start 11 Start 12 low_trx_id = 11 Up_limit_id=11 Trx_ids={} Max_trx_id = 13 Max_trx_id = 14 Trx_ids={12} Trx_ids={12, 13} Max_trx_id = 15 Commit Commit Start 13 Start 14 Trx_ids={13, 14} Apply Apply Apply low_trx_id = 13 Up_limit_id=12 Trx_ids={12} low_trx_id = 14 Up_limit_id=12 Trx_ids={12, 13} 25

26 Purge Control - The purge thread will create a snapshot based on - The oldest read view on Primary - The oldest read view on each replica - Drawback: undo tablespace may grow extremely large! - Luckily MySQL5.7 can truncate undo tablespace online 26

27 Replay DDL on Replica - The tablespace shouldn t be touched by user clients on replica if its structure is being changed. - The replica needs to know when DDL starts/ends. - And also update the corresponding table cache. - FIXME: - Let the replica knows what happens on primary by adding new logs: - MLOG_META_CHANGE_START: Write After acquiring Exclusive MDL lock - MLOG_META_CHANGE_END: Write before releasing Exclusive MDL lock - DDL is a synchronous operation - MLOG_META_CHANGE_START should be applied on all replicas before CREATE/DELETE/RENAME the IBD file 27

28 Failover - To reduce downtime, we have to allow a replica node to take over quickly if the primary node is found to be offline or unreachable - If the Primary is down, a Replica is promoted to Primary - Promise no data loss - If the whole data center is down, a Standby will be promoted to Primary - Promise no data loss if synchronous replication is turned on 28

29 Failover Main Steps: Replica Reopen all files in read-write mode Check if any block is corrupted by traversing double write buffer Write dirty blocks to disk by scanning LRU list Detect the LSN of last redo log written by Primary Change its state to STANDBY Standby Apply all logs up to latest LSN, and change the server state to Primary Reinitialize some in-memory objects, such as RSEG, change buffer, etc Rollback uncommitted transactions Accept read-write workload 29

30 Thank You &&QA Please WATCH && STAR our public branch on github And Next talk at this room is from my colleague Lixun Peng: Flashback - Rolling back a MySQL/MariaDB instance, database or table to a previous snapshot 30