Using the MySQL Binary Log as a Change Stream

Size: px

Start display at page:

Download "Using the MySQL Binary Log as a Change Stream"

Maximillian Richards
5 years ago
Views:

2 Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, timing, and pricing of any features or functionality described for Oracle s products may change and remains at the sole discretion of Oracle Corporation. Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 2

3 Agenda MySQL Replication 101 The Binary Log Change Capture Use Cases Advanced Experiments and Plugins Tips and Tricks Conclusion Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 3

5 MySQL Replication 101 This section will present: MySQL replication basic architecture. Different building blocks of MySQL replication. Different topologies that can built with MySQL replication. Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 5

6 MySQL Database Replication: Overview App INSERT... binary log relay log binary log Server A INSERT... Comm. Framework INSERT... Server B INSERT... Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 6

binary log Server A .. Comm. Framework .

7 MySQL Database Replication: Overview App INSERT... Persistent log buffer Persistent log buffer Threaded applier binary log relay log binary log Server A INSERT... Comm. Framework INSERT... Server B INSERT... Capture statements or data changes. Send, Receive, ACK, NACK, Heartbeating,... Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 7

8 MySQL Database Replication: Overview App INSERT... binary log relay log binary log Server A INSERT... Comm. Framework INSERT... Server B INSERT... Master generates a binary log, which contains changes. Slave may also generate a binary log. Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 8

9 MySQL Database Replication: Overview Coordination Between Servers A transactions B asynchronous (native) transactions A acks B semi-synchronous (plugin) A B C And in MySQL 8 as of Since group replication (plugin) Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 9

18 MySQL Database Replication: Topologies / Setups But why is this important? And many other combinations of these topologies together this EP is especially important in change capture to show how easy and flexbile it is to deploy a MySQL replica dedicated to capturing changes and P expose S them in Sits binary log, instead of doing it on a primary or other more important secondaries. Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 18

20 The Binary Log This section will present: What is the binary log and how it is organized. How changes are captured and then stored in the binary log. How to start a MySQL server. How to find out which files is the binary log stored in. How to inspect the contents of the binary log through a MySQL session. How to inspect the contents of the binary log using the mysqlbinlog command line tool. What are the contents of the binary log. Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 20

21 The Binary Log: What is it? Sequential logical stream recording changes happening on the server. Persisted on disk in a set of files. Row or statement based format (may be intermixed). Row format enables change capture. Each transaction is split into groups of events, but all events of a transaction are flushed to the log file together. Control events: Rotate, Format Description, Gtid, and more. Data events: Query_log_event, Update_rows_log_events,... GTID BEGIN E1 E2... COMMIT GTID BEGIN E1 E2... COMMIT Layout of the Binary Log. Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 21

22 The Binary Log: What is it used for? App Server A INSERT... binary log INSERT... Not only replication... For instance: Point-in-time recovery Integration with other technologies Rolling upgrades And more! Row based format is rich in change capture data. We will only focus on ROW format in this tutorial. Master generates a binary log, which records changes on the server. Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 22

23 The Binary Log: How is it created? Simplified Capture Flow Transaction Begins Parse next Statement Commit? No Yes Execute Statement Changes records? Yes No Is next record null? No Yes ha_update_row ha_delete_row ha_write_row Write record images to cache Cache empty? No Yes Prepare Storage engine Notify before_commit Flush to the binary log and commit to storage engine Notify after_commit Transaction Ends Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 23

24 The Binary Log: How is it created? Simplified Capture Flow Transaction Begins Parse next Statement Capture is an in-memory operation (with overspill to temporary file if needed). Commit? No Yes Execute Statement Changes records? Yes No Is next record null? No Yes ha_update_row ha_delete_row ha_write_row Write record images to cache Cache empty? No Yes Prepare Storage engine Notify before_commit Flush to the binary log and commit to storage engine Notify after_commit Transaction Ends Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 24

25 The Binary Log: How is it created? Simplified Capture Flow Transaction Begins Parse next Statement Commit? No Yes Execute Statement Changes records? Yes No Is next record null? No Yes ha_update_row ha_delete_row ha_write_row Write record images to cache Cache empty? No Yes Prepare Storage engine Notify before_commit Flush to the binary log and commit to storage engine Notify after_commit Transaction Ends Captured changes are written to disk transactionally. Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 25

The Binary Log: Where is it persisted? Disk layout Collections of files content files (the file extension is incremented by one on rotation) one index file listing content files binlog.

26 The Binary Log: Where is it persisted? Disk layout Collections of files content files (the file extension is incremented by one on rotation) one index file listing content files binlog binlog binlog binlog Binary Log binlog.index binlog binlog binlog binlog Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 26

The Binary Log: Hands-on Initializing a data directory Note: we used --initialize-insecure only to not clutter this presentation with passwords and credentials management. tut6318 $ mysql-8.0.

27 The Binary Log: Hands-on Initializing a data directory Note: we used --initialize-insecure only to not clutter this presentation with passwords and credentials management. tut6318 $ mysql /bin/mysqld --initialize-insecure --basedir=$pwd/mysql datadir=$pwd/vardir/master <TIMESTAMP> 0 [Warning] [MY ] [Server] 'Disabling symbolic links using --skip-symboliclinks (or equivalent) is the default. Consider not using this option as it' is deprecated and will be removed in a future release. <TIMESTAMP> 0 [System] [MY ] [Server] /ssd2/lsoares/tut6318/server/mysql /bin/mysqld (mysqld ) initializing of server in progress as process <TIMESTAMP> 5 [Warning] [MY ] [Server] root@localhost is created with an empty password! Please consider switching off the --initialize-insecure option. <TIMESTAMP> 0 [System] [MY ] [Server] /ssd2/lsoares/tut6318/server/mysql /bin/mysqld (mysqld ) initializing of server has completed Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 27

28 The Binary Log: Hands-on Launching the server Start the server (the binary log is enabled by default in MySQL 8). tut6318 $ mysql /bin/mysqld --datadir=$pwd/vardir/master \ --socket=$pwd/vardir/master/master.sock <TIMESTAMP> 0 [Warning] [MY ] [Server] 'Disabling symbolic links using --skip-symbolic-links (or equivalent) is the default. Consider not using this option as it' is deprecated and will be removed in a future release. <TIMESTAMP> 0 [System] [MY ] [Server] /ssd2/lsoares/tut6318/server/mysql /bin/mysqld (mysqld ) starting as process <TIMESTAMP> 0 [Warning] [MY ] [Server] CA certificate ca.pem is self signed. <TIMESTAMP> 0 [Warning] [MY ] [Server] Insecure configuration for --pid-file: Location '/ssd2' in the path is accessible to all OS users. Consider choosing a different directory. <TIMESTAMP> 0 [System] [MY ] [Server] /ssd2/lsoares/tut6318/server/mysql /bin/mysqld: ready for connections. Version: '8.0.13' socket: '/ssd2/lsoares/tut6318/server/vardir/master.sock' port: 3306 MySQL Community Server - GPL. <TIMESTAMP> 0 [ERROR] [MY ] [Server] Plugin mysqlx reported: 'Setup of socket: '/tmp/mysqlx.sock' failed, can't open lock file /tmp/mysqlx.sock.lock' <TIMESTAMP> 0 [System] [MY ] [Server] X Plugin ready for connections. Bind-address: '::' port: Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 28

29 The Binary Log: Hands-on Connecting to the server and listing the binary logs tut6318 $ mysql /bin/mysql -u root -h P 3306 Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 8 Server version: MySQL Community Server - GPL Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> SHOW BINARY LOGS; Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 29

30 The Binary Log: Hands-on Connecting to the server and listing the binary logs tut6318 $ mysql /bin/mysql -u root -h P 3306 Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 8 Server version: MySQL Community Server - GPL Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> SHOW BINARY LOGS; Log_name File_size File offset where the next entry shall be written to. binlog row in set (0,00 sec) The binary log file name. Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 30

The Binary Log: Hands-on Inspecting the contents of the binary log mysql> SHOW BINLOG EVENTS;

Info +---------------+-----+----------------+-----------+-------------+-----------------------------------+ binlog.

000001 124 Previous_gtids 1 155 +---------------+-----+----------------+-----------+-------------+-----------------------------------+ 2 rows in set

32 The Binary Log: Hands-on Inspecting the contents of the binary log mysql> SHOW BINLOG EVENTS; Log_name Pos Event_type Server_id End_log_pos Info binlog Format_desc Server ver: , Binlog ver: 4 binlog Previous_gtids rows in set (0,00 sec) mysql> Format description event is important. It describes the binlog encoding in the stream. Therefore, parsing a stream requires a format description event for that stream. A The few binary control log events file name. only. Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 32

33 The Binary Log: Hands-on Forcing a log rotation mysql> FLUSH BINARY LOGS; Query OK, 0 rows affected (0,00 sec) mysql> SHOW BINARY LOGS; Log_name File_size binlog binlog rows in set (0,00 sec) mysql> [1]+ Stopped [ ]/mysql /bin/mysql -u root -h P 3306 tut6318 $ cat $PWD/vardir/master/binlog.index./binlog /binlog tut6318 $ fg [ ]/mysql /bin/mysql -u root -h P 3306 mysql> Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 33

34 The Binary Log: Hands-on Forcing a log rotation mysql> FLUSH BINARY LOGS; Query OK, 0 rows affected (0,00 sec) mysql> SHOW BINARY LOGS; Log_name File_size binlog binlog rows in set (0,00 sec) mysql> [1]+ Stopped [ ]/mysql /bin/mysql -u root -h P 3306 tut6318 $ cat $PWD/vardir/master/binlog.index./binlog /binlog tut6318 $ fg [ ]/mysql /bin/mysql -u root -h P 3306 mysql> New log file: binlog Checking the index file in the filesystem shows twofiles as well. You should not mess around with this file yourself as this is an internal file. This is presented here only to demonstrate how the filesystem layout is organized. Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 34

35 The Binary Log: Hands-on Creating a schema and a table. mysql> CREATE DATABASE test; Query OK, 1 row affected (0,07 sec) mysql> use test Database changed mysql> CREATE TABLE t1 (c1 INT PRIMARY KEY); Query OK, 0 rows affected (0,04 sec) mysql> Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 35

transaction specific Transaction specific metadata. metadata. mysql> SHOW BINLOG EVENTS IN binlog.

36 The Binary Log: Hands-on Creating a schema and a table and inspecting the binlog now. mysql> CREATE DATABASE test; Query OK, 1 row affected (0,07 sec) mysql> use test Database changed mysql> CREATE TABLE t1 (c1 INT PRIMARY KEY); Query OK, 0 rows affected (0,04 sec) Contains transaction specific Transaction specific metadata. metadata. mysql> SHOW BINLOG EVENTS IN binlog ; Log_name Pos Event_type Server_id End_log_pos Info binlog Format_desc Server ver: , Binlog ver: 4 binlog Previous_gtids binlog Anonymous_Gtid 'ANONYMOUS' binlog Query CREATE DATABASE test /* xid=8 */ binlog Anonymous_Gtid 'ANONYMOUS' binlog Query use `test`; CREATE TABLE t1 (c1 INT PRIMARY KEY) /* xid=13 */ rows in set (0,00 sec) New Actual log changes file: binlog recorded. Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 36

37 The Binary Log: Hands-on Populating the table and printing the changes captured. mysql> FLUSH LOGS; Query OK, 0 rows affected (0,01 sec) mysql> INSERT INTO t1 VALUES (1), (2); Query OK, 2 rows affected (0,28 sec) Records: 2 Duplicates: 0 Warnings: 0 Table metadata encoded in the change stream. Specifies properties of the table that was modified. MySQL 8 extended this metadata. More on that later. mysql> SHOW BINLOG EVENTS in 'binlog '; Log_name Pos Event_type Server_id End_log_pos Info binlog Format_desc Server ver: , Binlog ver: 4 binlog Previous_gtids binlog Anonymous_Gtid 'ANONYMOUS' binlog Query BEGIN binlog Table_map table_id: 89 (test.t1) binlog Write_rows table_id: 89 flags: STMT_END_F binlog Xid COMMIT /* xid=17 */ rows in set (0,00 sec) The records inserted. Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 37

38 The Binary Log: Hands-on Populating the table and printing the changes captured. mysql> UPDATE t1 SET c1 = 3 WHERE c1 = 1; Query OK, 1 row affected (0,05 sec) Rows matched: 1 Changed: 1 Warnings: 0 mysql> DELETE FROM t1 WHERE c1 = 2; Query OK, 1 row affected (0,06 sec) mysql> SHOW BINLOG EVENTS in 'binlog '; [...] binlog Anonymous_Gtid 'ANONYMOUS' binlog Query BEGIN binlog Table_map table_id: 89 (test.t1) binlog Update_rows table_id: 89 flags: STMT_END_F binlog Xid COMMIT /* xid=20 */ binlog Anonymous_Gtid 'ANONYMOUS' binlog Query BEGIN binlog Table_map table_id: 89 (test.t1) binlog Delete_rows table_id: 89 flags: STMT_END_F binlog Xid COMMIT /* xid=21 */ rows in set (0,01 sec) Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 38

39 The Binary Log: Hands-on Command line tool: mysqlbinlog Decodes events in the binary log files; Good for debugging, recovery and even auditing; Outputs valid SQL syntax that a MySQL server can process: When piped through mysql client program can replay binary logs; Can be used as a tool for implementing point-in-time recovery; Can even act as a fake slave enables complex workflows. Old example, but illustrates the potential: Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 39

40 The Binary Log: Hands-on Command line tool: mysqlbinlog tut6318 $ mysqlbinlog vv $PWD/vardir/master/binlog # at 4 # :24:25 server id 1 end_log_pos 124 CRC32 0x4f Start: binlog v 4, server v created :24:25 # Warning: this binlog is either in use or was not closed properly. BINLOG ' GaPEWw8BAAAAeAAAAHwAAAABAAQAOC4wLjEzAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAEwANAAgAAAAABAAEAAAAYAAEGggAAAAICAgCAAAACgoKKioAEjQA CgFIFglP '/*!*/; The format description event. Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 40

The Binary Log: Hands-on Command line tool: mysqlbinlog tut6318 $ mysqlbinlog vv $PWD/vardir/master/binlog.000003 [.

41 The Binary Log: Hands-on Command line tool: mysqlbinlog tut6318 $ mysqlbinlog vv $PWD/vardir/master/binlog [...] # at 124 # :24:25 server id 1 end_log_pos 155 CRC32 0x0e5119bb Previous-GTIDs # [empty] An event containing information about transaction identifiers stored in preceding binary log files. Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 41

immediate_commit_timestamp=1539613476111048 transaction_length=276 /*!50718 SET TRANSACTION ISOLATION LEVEL READ COMMITTED*//*!*/; # original_commit_timestamp=1539613476111048 (2018-10-15 16:24:36.

GTID_NEXT= 'ANONYMOUS'/*!*/; [...] # at 307 #181015 16:24:35 server id 1 end_log_pos 355 CRC32 0x2fce6489 Table_map: `test`.

42 The Binary Log: Hands-on Command line tool: mysqlbinlog [...] # at 155 # :24:35 server id 1 end_log_pos 230 CRC32 0xa63f6ba6 Anonymous_GTID last_committed=0 sequence_number=1 rbr_only=yes original_committed_timestamp= immediate_commit_timestamp= transaction_length=276 /*!50718 SET TRANSACTION ISOLATION LEVEL READ COMMITTED*//*!*/; # original_commit_timestamp= ( :24: CEST) # immediate_commit_timestamp= ( :24: CEST) /*! 'ANONYMOUS'/*!*/; [...] # at 307 # :24:35 server id 1 end_log_pos 355 CRC32 0x2fce6489 Table_map: `test`.`t1` mapped to number 89 # at 355 # :24:35 server id 1 end_log_pos 400 CRC32 0xa03a40a8 Write_rows: table id 89 flags: STMT_END_F [...] ### INSERT INTO `test`.`t1` ### SET /* INT meta=0 nullable=0 is_null=0 */ ### INSERT INTO `test`.`t1` ### SET /* INT meta=0 nullable=0 is_null=0 */ Transaction specific metadata, that allows the user to know when it was originally committed and when it was committed in a relay master. Moreover, it contains information about the transaction identififer and commit sequence number. The actual values inserted (1 and 2). Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 42

43 The Binary Log: Hands-on Command line tool: mysqlbinlog [...] # at 431 # :31:19 server id 1 end_log_pos 506 CRC32 0x4b322c42 Anonymous_GTID last_committed=1 sequence_number=2 rbr_only=yes original_committed_timestamp= immediate_commit_timestamp= transaction_length=286 /*!50718 SET TRANSACTION ISOLATION LEVEL READ COMMITTED*//*!*/; # original_commit_timestamp= ( :31: CEST) # immediate_commit_timestamp= ( :31: CEST) /*! 'ANONYMOUS'/*!*/; # at 506 # :31:19 server id 1 end_log_pos 592 CRC32 0x51bfc235 Query thread_id=8 exec_time=0 error_code=0 SET TIMESTAMP= /*!*/; /*!80013 BEGIN /*!*/; # at 592 # :31:19 server id 1 end_log_pos 640 CRC32 0xc968fe53 Table_map: `test`.`t1` mapped to number 89 # at 640 # :31:19 server id 1 end_log_pos 686 CRC32 0xad89e32a Update_rows: table id 89 flags: STMT_END_F ### UPDATE `test`.`t1` ### WHERE /* INT meta=0 nullable=0 is_null=0 */ ### SET /* INT meta=0 nullable=0 is_null=0 */ The actual row updated (1 becomes 3). Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 43

44 The Binary Log: Hands-on Command line tool: mysqlbinlog [...] # at 717 # :31:43 server id 1 end_log_pos 792 CRC32 0xb654a2a4 Anonymous_GTID last_committed=2 sequence_number=3 rbr_only=yes original_committed_timestamp= immediate_commit_timestamp= transaction_length=271 /*!50718 SET TRANSACTION ISOLATION LEVEL READ COMMITTED*//*!*/; # original_commit_timestamp= ( :31: CEST) # immediate_commit_timestamp= ( :31: CEST) /*! 'ANONYMOUS'/*!*/; # at 792 # :31:43 server id 1 end_log_pos 869 CRC32 0xfef26634 Query thread_id=8 exec_time=0 error_code=0 SET TIMESTAMP= /*!*/; /*!80013 BEGIN /*!*/; # at 869 # :31:43 server id 1 end_log_pos 917 CRC32 0xa4deb2e5 Table_map: `test`.`t1` mapped to number 89 # at 917 # :31:43 server id 1 end_log_pos 957 CRC32 0x16f6ca45 Delete_rows: table id 89 flags: STMT_END_F [...] ### DELETE FROM `test`.`t1` ### WHERE /* INT meta=0 nullable=0 is_null=0 */ The actual row deleted (2). Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 44

45 The Binary Log: Hands-on Extended table metadata MySQL 8 can capture more table metadata if instructed to do it. This is governed by the following option: Property Value Name binlog_row_metadata Scope GLOBAL Type ENUM Dynamic Yes Values MINIMAL, FULL Default MINIMAL Details: Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 45

$The Binary Log: Hands-on Command line tool: mysqlbinlog mysql> SET GLOBAL binlog_row_metadata = FULL; Query OK, 0 rows affected (0,00 sec) mysql> \q Bye tut6318 $ $PWD/mysql-8.0.13/bin/mysql -u root -h 127.$

46 The Binary Log: Hands-on Command line tool: mysqlbinlog mysql> SET GLOBAL binlog_row_metadata = FULL; Query OK, 0 rows affected (0,00 sec) mysql> \q Bye tut6318 $ $PWD/mysql /bin/mysql -u root -h P 3306 [...] mysql> use test Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A Explicit switch to display additional metadata. Database changed mysql> INSERT INTO t1 VALUES (1000); Query OK, 1 row affected (0,08 sec) mysql> \q Bye tut6318 $ mysql /bin/mysqlbinlog -vv --print-table-metadata $PWD/vardir/master/binlog Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 46

47 The Binary Log: Hands-on Command line tool: mysqlbinlog [...] # at 1140 # :12:48 server id 1 end_log_pos 1196 CRC32 0xed25a26b Table_map: `test`.`t1` mapped to number 89 # Columns(`c1` INT NOT NULL) # Primary Key(c1) # at 1196 # :12:48 server id 1 end_log_pos 1236 CRC32 0xe49b76d8 Write_rows: table id 89 flags: STMT_END_F BINLOG ' ck7ewxmbaaaaoaaaakweaaaaafkaaaaaaaeabhrlc3qaanqxaaedaaabaqaeawjjmqgbaguije0= ck7ewx4baaaakaaaanqeaaaaafkaaaaaaaeaagab/wdoawaa2hab5a== '/*!*/; ### INSERT INTO `test`.`t1` ### SET /* INT meta=0 nullable=0 is_null=0 */ [...] In this case, column names and primary key information is displayed. Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 47

48 The Binary Log: Hands-on Command line tool: mysqlbinlog mysqlbinlog can decode the contents of the binary log files or even from a server directly, if it connects to it acting as a fake slave. mysqlbinlog relies on a general purpose decoder that is included in libbinlogevents. Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 48

49 The Binary Log: New Metadata in MySQL 8 MySQL 8 Table Metadata: UNSIGNED flag of numeric columns Default character set of string columns Character set of string columns Column name [requires binlog_row_metadata=full] String value of SET columns [requires binlog_row_metadata=full] String value of ENUM columns [requires binlog_row_metadata=full] Primary key without prefix [requires binlog_row_metadata=full] Primary key with prefix [requires binlog_row_metadata=full] Geometry type of geometry columns Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 49

50 The Binary Log: New Metadata in MySQL 8 MySQL 8 Transaction Metadata: Original execution commit timestamp in seconds since the epoch Relay server commit timestamp in seconds since the epoch Full transaction length Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 50

52 Change Capture Use Cases This section will present: How the replication stream can be used for other use cases than just plain replication. Other technologies that rely on MySQL replication stream and use it to implement: change data capture extract, transform and load external triggers point-in-time recovery and more. Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 52

53 Change Capture Use Cases Disclaimer Some technologies presented in this section are Community projects and not owned by Oracle. ProxySQL, Gh-ost, Maxwell They are here to show how some of the MySQL users are relying on MySQL infrastructure to build their own tooling and automation. Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 53

55 Change Tracking: ProxySQL s consistent reads case Track which changes are where at a given point in time. Route application traffic based on that information. The ProxySQL case (ProxySQL is a MySQL Proxy): Knows which transactions a session has seen. A process listens to the binary log stream and captures transaction identifiers. Forwards that information to ProxySQL instances. ProxySQL routes incoming requests based on the knowledge of which transactions have been applied to which servers. Read your writes consistency can then be implemented by routing transactions to servers that already have the expected changes applied. Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 55

57 Transaction Identifiers Change Tracking: ProxySQL s consistent reads case App. ProxySQL A B C Binlog reader Binlog reader Binlog reader Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 57

requiremetns and the state of each server. App.

have been applied on each server A B C Binlog reader Binlog reader

58 Transaction Identifiers Change Tracking: ProxySQL s consistent reads Case Query routing is done according to the consistency requiremetns and the state of each server. App. ProxySQL Extract from the binary log stream, which transactions have been applied on each server A B C Binlog reader Binlog reader Binlog reader Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 58

60 Userland Online Schema Changes: Github s gh-ost Github Online Schema Changes (gh-ost) Userland tool to migrates a table from schema V to schema V+1. Operations: (preferred mode, but there are others) PRIMARY SECONDARY Binary Log Replication Load records from V Copies records to V+1 Gh-ost (Convert records to V+1) Reads the Binary Log stream for changes to V while loading is taking place. Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 60

com/github/gh-ost Userland tool to migrates a table from schema V to schema V+1.

table with version v+1. SECONDARY Orchestrator transforms Load records the records.

V+1) Binary log used as a change stream, from where changes are extracted.

61 Userland Online Schema Changes: Github s gh-ost Github Online Schema Changes (gh-ost) Userland tool to migrates a table from schema V to schema V+1. Records are loaded Operations: (preferred mode, but there are others) PRIMARY into table with version v+1. SECONDARY Orchestrator transforms Load records the records. from V Copies records to V+1 Binary Log Replication Gh-ost (Convert records to V+1) Binary log used as a change stream, from where changes are extracted. Reads the Binary Log stream for changes to V while loading is taking place. Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 61

63 Online Rolling Upgrades Upgrades are operationally expensive Requires automation Always on, even when upgrading The binary log, as a change stream, and MySQL replication, prove themselves invaluable to meet these requirements. Setup a secondary (version V+1) that replicates from the primary (version V). Cut application traffic on the primary, let secondary catch up. Switch application over to the secondary, which becomes the new primary. Nuke or upgrade old primary. Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 63

64 Online Rolling Upgrades New Version V+1 Old Version V App App App P S S P S S P* S S S S S App P S S S App P S S S Rolling upgrades means no new features used until upgrade is done (usually). Whatch out for defaults and changed behavior. Test, Test, Test. Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 64

66 Data Integration: Oracle Golden Gate Replicate data from MySQL to Oracle and vice-versa. Use the binary log stream (row format) as a change log. Self-contained metadata in the binary log stream. Contains enough metadata to infer the original table structure. Contains enough metadata notifying table structure modifications. Metadata has been continuously enhaced. New metadata fields in MySQL 8. Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 66

67 Data Integration: Oracle Golden Gate Apply to MySQL. Apply to Oracle. MySQL Oracle Oracle GoldenGate Capture changes from the binary log stream. Capture from Oracle DB. Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 67

stream. Kafka Publish to Kafka Capture from Oracle DB.

68 Data Integration: Oracle Golden Gate Apply to MySQL. Apply to Oracle. MySQL Oracle Oracle GoldenGate Capture changes from the binary log stream. Kafka Publish to Kafka Capture from Oracle DB. Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 68

69 Change Listener: Maxwell s Daemon Reads the binary log stream Writes row changes, as JSON, to a streaming infrastructure: Kafka Kinesis RabbitMQ Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 69

70 Change Listener: Maxwell s Daemon App insert into `test`.`maxwell` set id = 1, daemon = 'Stanislaw Lem'; Maxwell MySQL Example taken from: maxwell: { "database": "test", "table": "maxwell", "type": "insert", "ts": , "xid": , "commit": true, "data": { "id":1, daemon": "Stanislaw Lem } } Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 70

73 Point-in-time Recovery Binary logs capture incremental changes to the server. Backup then restore and apply binary logs to get state up to a certain [logical] point in time. Backup Restore Primary Crash Primary Copy of the Primary time Backup Backup Copy of the Primary Incremental Copyright 2018, Oracle and/or its affiliates. All rights reserved. 73

74 Point-in-time Recovery # Starts replaying binlog from 16:31:43 to 17:12:48 of 15-Oct-2018 tut6318 $ mysql /bin/mysqlbinlog --start-datetime= :31:43 --stop-datetime=" :12:48" $PWD/vardir/master/binlog mysql u root P 3306 h [...] # Starts replaying binlog from offset 431 to 988 tut6318 $ mysql /bin/mysqlbinlog --start-position=431 --stop-position=988 $PWD/vardir/master/binlog mysql u root P 3306 h [...] # Replay only the following transactions tut6318 $ mysql /bin/mysqlbinlog include-gtids= c-9c3f-4842-ae40-173a99ab45c9:3-10 $PWD/vardir/master/binlog mysql u root P 3306 h [...] Copyright 2018, Oracle and/or its affiliates. All rights reserved. 74

76 Undelete One or More Transactions The binary log captures the record images for inserts, deletes and updates. If one replays them in reverse order, in some cases, one is able to go back to a point in time and undo an error. Some people have developed their own tooling to do it. Examples: Limited applicability (e.g., can t revert DDL operations), but in some cases it may be useful to undo a specific transaction instead of re-imaging a server. Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 76

Undelete One or More Transactions App sends

User applies the modified binlog as if it

GTID BEGIN BI AI COMMIT GTID BEGIN AI BI

images in the binary log for the Offending

78 Undelete One or More Transactions App sends offending transaction. User applies the modified binlog as if it was doing a pointin-time recovery. GTID BEGIN BI AI COMMIT GTID BEGIN AI BI COMMIT User reverts the order of the change images in the binary log for the offending transaction. Offending transaction was compensated. Database is back to the previous state. Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 78

80 Advanced Experiments and Plugins The next section will: Show how some plugins mine the binary log caches to intersect the captured changes. Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 80

81 Advanced Experiments and Plugins Over-simplified Capture Flow Refreshing our mind! Transaction Begins Parse next Statement Capture is an in-memory operation (with overspill to temporary file if needed). Commit? No Yes Execute Statement Changes records? Yes No Is next record null? No Yes ha_update_row ha_delete_row ha_write_row Write record images to cache Cache empty? No Yes Prepare Storage engine Notify before_commit Flush to the binary log and commit to storage engine Notify after_commit Transaction Ends Captured changes are written to disk transactionally. Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 81

83 Change listener plugin: The Group Replication Plugin Case MySQL Group Replication plugin is... Built on top of proven technology! Shares many pieces of MySQL Replication. Built on reusable components! Layered implementation approach. Interface driven development. Decoupled from the server core. Plugin registers as listener to server events. Reuses the capture procedure from regular replication. Provides further decoupling from the communication infrasctructure. Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 83

84 Change listener plugin: The Group Replication Plugin Case MySQL Group Replication plugin is... Applications P P P P P Replication Group Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 84

85 Change listener plugin: The Group Replication Plugin Case MySQL Group Replication plugin is... Applications MySQL Server P P P P P API Replication Plugin API Replication Group Group Com. Engine Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 85

86 Plugin Change listener plugin: The Group Replication Plugin Case Listens to a notification that a transaction is about to commit! Performance Schema Tables: Monitoring MySQL Server API Replication Plugin API InnoDB MySQL APIs: Lifecycle / Capture / Applier Capture Conflicts Handler Applier Replication Protocol Group Com. API Recovery Group Com. Engine Group Com. Binding Group Com. Engine Network Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 86

87 Plugin Change listener plugin: The Group Replication Plugin Case Listens to a notification that a transaction is about to commit! Performance Schema Tables: Monitoring MySQL Server API Replication Plugin API InnoDB MySQL APIs: Lifecycle / Capture / Applier Capture Conflicts Handler Applier Replication Protocol Group Com. API Recovery Group Com. Engine Intercepts the change buffer and sends it out to the group. Group Com. Binding Group Com. Engine Network Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 87

Change listener plugin: The Group Replication Plugin Case Over-simplified Capture Flow Interception point Transaction Begins Parse next Statement Capture is an in-memory operation (with overspill to

88 Change listener plugin: The Group Replication Plugin Case Over-simplified Capture Flow Interception point Transaction Begins Parse next Statement Capture is an in-memory operation (with overspill to temporary file if needed). Commit? No Yes Execute Statement Changes records? Yes No Is next record null? No Yes ha_update_row ha_delete_row ha_write_row Write record images to cache Cache empty? No Yes Prepare Storage engine Notify before_commit Flush to the binary log and commit to storage engine Notify after_commit Transaction Ends This is where the interception takes place. Note that at this point the changes captured are still in the capture cache. Captured changes are written to disk transactionally. Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 88

Change listener plugin: The Group Replication Plugin Case Capture Interception Point Begin RW Trx Next stmt execute capture encode buffer Com mit?

The three stages of the binlog group commit. no yes Replicate OK? yes Prepare OK?

89 Change listener plugin: The Group Replication Plugin Case Capture Interception Point Begin RW Trx Next stmt execute capture encode buffer Com mit? no Rollback Trx yes Commit Trx Panic / Server RO yes Commit In SE OK? no Local commit. Local externalization. yes Flush and sync OK? Binary Log Persistence. The three stages of the binlog group commit. no yes Replicate OK? yes Prepare OK? no More specifically, during the before_commit commit phase, the transaction is prepared Distributed Commit. locally and then propagated Global externalization. to the group. Group Replication Propagation Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 89 no

90 Change listener plugin: The Group Replication Plugin Case Internal APIs used by the plugin Callback definition /** This callback is called before transaction commit This callback is called right before write binlog cache to binary param The parameter for transaction 0 1 Failure */ typedef int (*before_commit_t)(trans_param *param); Part of trans_param definition /** Transaction observer parameter */ typedef struct Trans_param { [...] /* Set on before_commit hook. */ Binlog_cache_storage *trx_cache_log; Binlog_cache_storage *stmt_cache_log; [...] } Trans_param; Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 90

91 Change listener plugin: The Group Replication Plugin Case before_commit hook implementation int group_replication_trans_before_commit(trans_param *param) { [...] // Broadcast the Transaction Message send_error = gcs_module->send_message(transaction_msg); [...] } Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 91

92 Change listener plugin: The Group Replication Plugin Case Take aways Interception happens in the capture pipeline. Right before the transaction terminates and is written to the log file. The cache is shared with the plugin. The plugin eventually sends the transaction changes to the group. Other servers in the receive and apply the changes. Key point: Capture and extraction are the same as for regular replication, except that it is intercepted and exposed at a different point in time. Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 92

94 Plugin Change listener plugin: An Experiment Listens to a notification that a transaction is about to commit! Performance Schema Tables: Monitoring MySQL Server API Replication Plugin API InnoDB MySQL APIs: Lifecycle / Capture / Applier Capture Conflicts Handler Applier Replication Protocol Group Com. API Recovery Group Com. Engine Group Com. Binding Group Com. Engine Network Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 94

Observer InnoDB MySQL APIs: Lifecycle / Capture / Applier Capture

95 Plugin Change listener plugin: An Observer Plugin Observing Changes Performance Schema Tables: Monitoring MySQL Server Observer InnoDB MySQL APIs: Lifecycle / Capture / Applier Capture Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 95

96 Change listener plugin: An Observer Plugin before_commit hook implementation unsigned long before_commit_counter; static int trans_before_commit(trans_param *param MY_ATTRIBUTE((unused))) { before_commit_counter++; LogPluginErr( INFORMATION_LEVEL, ER_LOG_PRINTF_MSG, before_commit_counter = %s, before_commit_counter); } Trans_observer trans_observer = { sizeof(trans_observer), }; trans_before_dml, trans_after_commit, trans_before_commit, trans_before_rollback, trans_after_rollback, static int replication_observers_example_plugin_init(mysql_plugin plugin_info) { [...] register_trans_observer(&trans_observer, (void *)plugin_info_ptr); [...] } Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 96

97 Change listener plugin: An Observer Plugin before_commit hook implementation unsigned long before_commit_counter; static int trans_before_commit(trans_param *param MY_ATTRIBUTE((unused))) { before_commit_counter++; LogPluginErr( INFORMATION_LEVEL, ER_LOG_PRINTF_MSG, before_commit_counter = %s, before_commit_counter); } before_commit hook implementation that counts the number of times the before_commit hooks was called and prints the counter. Trans_observer trans_observer = { sizeof(trans_observer), }; trans_before_dml, trans_after_commit, trans_before_commit, trans_before_rollback, trans_after_rollback, static int replication_observers_example_plugin_init(mysql_plugin plugin_info) { [...] register_trans_observer(&trans_observer, (void *)plugin_info_ptr); [...] } Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 97

98 Change listener plugin: An Observer Plugin before_commit hook implementation unsigned long before_commit_counter; static int trans_before_commit(trans_param *param MY_ATTRIBUTE((unused))) { before_commit_counter++; LogPluginErr( INFORMATION_LEVEL, ER_LOG_PRINTF_MSG, before_commit_counter = %s, before_commit_counter); } Trans_observer trans_observer = { sizeof(trans_observer), }; trans_before_dml, trans_after_commit, trans_before_commit, trans_before_rollback, trans_after_rollback, before_commit hook implementation that counts the number of times the before_commit hooks was called and prints the counter. Transaction observer hooks setup. static int replication_observers_example_plugin_init(mysql_plugin plugin_info) { [...] register_trans_observer(&trans_observer, (void *)plugin_info_ptr); [...] } Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 98

$Change listener plugin: An Observer Plugin before_commit hook implementation unsigned long before_commit_counter; static int trans_before_commit(trans_param *param MY_ATTRIBUTE((unused))) {$ $called and prints the counter. Transaction observer hooks setup. static int replication_observers_example_plugin_init(mysql_plugin plugin_info) { [.$

99 Change listener plugin: An Observer Plugin before_commit hook implementation unsigned long before_commit_counter; static int trans_before_commit(trans_param *param MY_ATTRIBUTE((unused))) { before_commit_counter++; LogPluginErr( INFORMATION_LEVEL, ER_LOG_PRINTF_MSG, before_commit_counter = %s, before_commit_counter); } Trans_observer trans_observer = { sizeof(trans_observer), }; trans_before_dml, trans_after_commit, trans_before_commit, trans_before_rollback, trans_after_rollback, before_commit hook implementation that counts the number of times the before_commit hooks was called and prints the counter. Transaction observer hooks setup. static int replication_observers_example_plugin_init(mysql_plugin plugin_info) { [...] register_trans_observer(&trans_observer, (void *)plugin_info_ptr); [...] } Registering the hooks when the plugin is initialized. Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 99

100 Change listener plugin: An Observer Plugin before_commit hook implementation unsigned long before_commit_counter; static int trans_before_commit(trans_param *param MY_ATTRIBUTE((unused))) { before_commit_counter++; LogPluginErr( INFORMATION_LEVEL, ER_LOG_PRINTF_MSG, before_commit_counter = %s, before_commit_counter); } Trans_observer trans_observer = { sizeof(trans_observer), }; For a complete example/mock, see: plugin/replication_observers_example/replication_observers_example.cc ( trans_before_dml, trans_after_commit, trans_before_commit, trans_before_rollback, trans_after_rollback, static int replication_observers_example_plugin_init(mysql_plugin plugin_info) { [...] register_trans_observer(&trans_observer, (void *)plugin_info_ptr); [...] } Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 100

102 Capture Interfaces Some of the APIs shown here are really internal hooks. These should be made APIs registered in the new service infrastructure that has debutted in MySQL 8. Promote them to proper public APIs. Make them properly documented. Allows everyone to extend MySQL more easily. But the basic infrastructure is there and already allows interesting things. Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 102

104 Tips and Tricks This section will present: How to control whether or not the original statement should be included as part of the changes captured. How to control the amount of data logged in the record s change images. How to control whether to log full documents or just logical diff of a document change. How to control the amount of table metadata captured. Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 104

105 Tip #1: Capture Partial Rows Two optimizations: RBR can be configured to exclude BLOBS only (when not needed); RBR can also work with partial rows in addition to full rows: Before image: only fields required to find the row; After image: only fields that actually changed; Reduces memory footprint; Reduces network bandwidth usage; Reduces binary log files size; Default configuration: capture full row. This is the most interesting setup in a change capture scenario. Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 105

106 Tip #1: Capture Partial Rows Before Image After Image Full Rows Blob Rows without Blobs Minimal Rows Primary Key Changed Columns Copyright 2018, Oracle and/or its affiliates. All rights reserved. Oracle Code 106

Replication features of 2011

FOSDEM 2012 Replication features of 2011 What they were How to get them How to use them Sergey Petrunya MariaDB MySQL Replication in 2011: overview Notable events, chronologically: MySQL 5.5 GA (Dec 2010)