MySQL 8.0: Atomic DDLs Implementation and Impact

Transcription

1 MySQL 8.0: Atomic DDLs Implementation and Impact Ståle Deraas, Senior Development Manager Oracle, MySQL 26 Sept 2017 Copyright 2017, Oracle and/or its its affiliates. All All rights reserved.

2 Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle s products remains at the sole discretion of Oracle. 2

3 Agenda Requirements for a Data Dictionary Storage Engine Requirements for an SE implementing Atomic DDLs Effects of new INFORMATION_SCHEMA in MySQL 8.0

4 Requirements for a Data Dictionary Storage Engine

5 MySQL Data Dictionary before MySQL 8.0 Data Dictionary Files FRM TRG OPT File system SQL System Tables (mysql. ) user events proc MyISAM InnoDB System Tables InnoDB 5

6 SQL Transactional Data Dictionary in MySQL 8.0 Data Dictionary DD DD Table DD Table DD SE 6

7 Requirements for Data Dictionary Storage Engine Some special care need to be taken at initialize/bootstrap Must support transactions Must support attachable transactions To open a table, the DD SE needs to open DD tables In MySQL 8.0 InnoDB has dual role MySQL Data Dictionary Storage Engine (DD SE) Storage Engine that supports Atomic DDL 7

8 The Data Dictionary Storage Engine API An addition to the Storage Engine API Not strictly separated from SE API Some examples of functionality that the DD SE must implement: - Initialize dictionary support - Initiate recovery of dictionary transactions - Reset of the DD SE internal cache of meta data 8

9 Data Dictionary Storage Engine Client SQL statement Result Query Executor Parser Optimizer MySQL Server Data Dictionary Storage Engine DD Storage Engine 9

10 Example - DROP SCHEMA at high level MySQL 5.7 Delete tables Metadata, TRN/TRG/FRM files Data, InnoDB tables Delete stored programs Metadata, rows in MyISAM (nontransactional) Delete schema Metadata, DB.OPT file Mix of filesystem, nontransactional/transactional storage and multiple commits MySQL 8.0 Delete tables Metadata, rows in InnoDB Data, InnoDB tables Delete stored programs Metadata, rows in InnoDB Delete schema Metadata, rows in InnoDB Updates to transactional storage, one commit 10

11 Requirements for an SE implementing Atomic DDLs

12 Why Atomic DDL support in MySQL 8.0 Ensure we have a consistent state after a DDL operation Prevent slave drift Prevent internal inconsistencies At last possible due to transactional storage of meta data Enables implemention of crash-safe DDL Enables addressing longstanding issues 12

13 User visible changes to Atomic TABLE DDL in MySQL 8.0 DROP TABLES, all tables dropped or none DROP SCHEMA, all entities in the schema are dropped, or none Note that atomic DDL statements will be rolled back or committed even in case of crash, e.g. RENAME TABLES CREATE TABLE would be successfully committed or rolled back (no orphan ibd left) TRUNCATE TABLE (including InnoDB tables with FTS AUX tables) would be successfully committed or rolled back RENAME TABLES, all or none ALTER TABLE successful or not done 13

14 Atomic DDL: Storing SE private data (prerequisite) An SE which will support atomic DDL can store private data in SE private columns of Data Dictionary tables InnoDB has chosen to store key-value pairs in the SE private data Examples: index root page number, InnoDB table ID, tablespace ID etc. 14

15 How do we implement Atomic and Crash-Safe TABLE DDL At high level, create a single atomic transaction: Do updates to the Data Dictionary (DD SE) Do necessary changes in the Storage Engine Do necessary writes to binary log Since it is a single transaction, all updates to Data Dictionary can be rolled back and are crash safe 15

16 How do we implement Atomic and Crash-Safe TABLE DDL At more detailed level: Make sure there are no intermediate commits on SQL layer during DDL SE methods should have no intermediate commits, and register as part of trx SE can do redo/rollback of DDL, implemented with DDL_LOG in InnoDB SQL layer will invoke a post_ddl() hook, so SE can do post commit/rollback work As part of the transaction, write to binary log Introduced the SE property HTON_SUPPORTS_ATOMIC_DDL, that can be tested for in the source code 16

17 InnoDB: Features in SE for Atomic and Crash-Safe TABLE DDL DDL_LOG Protected table in mysql tablespace, so no DDL and no user DML allowed Add trx Ids to all entries, which will be deleted by post_ddl() hook Changes to DDL_LOG are persisted ASAP. Will prevent data files updates and no flushing of DDL_LOG entries (exempted from innodb_flush_log_at_trx_commit control for delayed flush) Issue with physical files and index tree: Before MySQL 8.0, could leave orphan InnoDB ibd files, orphan index trees/pages Orphan files/trees cause problems for subsequent DDLs and waste tablespace With MySQL 8.0, DDLs crash safe, no ibd files/index trees orphaned if no commit At commit, the old ibd files/index trees will be dropped 17

18 Atomic DDL: Use Case - CREATE TABLE, SQL Perspective MySQL 5.7 At SQL layer: create table obj Store obj in DD tables, commit «Open» table, construct all internal structures Call to SE handler::create(name,...) MySQL 8.0 At SQL layer: create table obj Call to SE, adding SE private data Store obj in DD tables * «Open» table, construct all internal structures Call to SE handler::create(name,...), which can update SE private data Store obj in DD tables * * = commit for non-atomic SE 18

19 Atomic DDL: Use Case - CREATE TABLE, SQL Perspective CONT MySQL 5.7 SE creates physical files, its own artifacts such as InnoDB tablespaces. InnoDB stores data in own system tables. Write statement to binary log MySQL 8.0 Write statement to binary log Commit/Rollback, post_ddl() hook 19

20 Atomic DDL: Use Case - CREATE TABLE, SE perspective MySQL 5.7 Call to SE handler::create(name,.) SE creates the physical tablespaces/files(file-per-table) or Cluster index tree and other index trees SE starts its own transaction, insert the new table/indexes's metadata to InnoDB own System tables MySQL 8.0 Call to SE handler::create(name,...) SE creates the physical tablespaces/files(file-per-table) or Cluster index tree and other index trees SE logs above physical file and indexes creation Note: No separate SE transaction, all info for new tablespace/indexes are passed back to server with DD objects 20

21 Atomic DDL: Use Case - CREATE TABLE, SE perspective cont MySQL 5.7 Note: If crash happens before above trx commit, there could be orphan files or index trees left MySQL 8.0 SQL layer commits or rollback Call to SE post_ddl(). If rollback, the post_ddl() physically delete the tablespace/ibd (file-per-table) and drop the index trees for the table Note: If crash, SE will recover info in DDL_LOG to delete the ibd/index trees 21

22 Atomic DDL: Use Case - DROP TABLES, SQL Perspective MySQL 5.7 For each table in table list Call SE handler:ha_delete_table() and if error do appropriate action Remove for Data Dictionary and commit Write up to 3 artifical DROP TABLES to binary log for dropped tables (temp tables *2 + all base tables) MySQL 8.0 For each table in table list Check for 5 types of tables Err out if non-existing tables and no IF EXISTS clause Handle tables in non-atomic DDL SE Delete Table in SE, and del in DD, commit Handle tables in atomic DDL SE Delete Table in SE, and del in DD, no commit Write DROP TABLES statment to binary log incl all table (in «no GTID» mode) Commit/rollback, post_ddl() hook Concurrent DDL on table blocked 22

23 Atomic DDL: Use Case - DROP TABLES, SQL Perspective cont MySQL 5.7 MySQL 8.0 If GTID mode, and write to binlog postponed, write DROP TABLES stmt with all tables we dropped to binlog To handle GTID mode from older servers For each temp table in nontransactional SE Drop table, call SE handler::delete_table() Write DROP TEMPORARY TABLES for all temp tables above to binary log For each temp table in transactional SE Drop table, call SE handler::delete_table() Write DROP TEMPORARY TABLES for all temp tables above to binary log 23

24 Atomic DDL: Use Case - DROP TABLES, SE perspective MySQL 5.7 Call to SE handler:ha_delete_table() InnoDB starts its own transaction and delete metadata from the InnoDB own system tables InnoDB commit trx InnoDB deletes physical files/ibd (in file-per-table case) Note: If crash happens after InnoDB commit trx, there could be orphaned files/ibd or trees MySQL 8.0 Call to SE handler:ha_delete_table() InnoDB only logs actions to delete ibd file or index trees in DDL_LOG, so no physical action (No InnoDB system tables needs to be updated) Back to SQL layer to commit - Delete metadata in DD system tables and commit 24

25 Atomic DDL: Use Case - DROP TABLES, SE perspective cont MySQL 5.7 MySQL 8.0 SQL layer calls post_ddl(). If commit, InnoDB physically deletes ibd and drop index trees. If rollback, delete entries in DDL_LOG, files and trees left as they are Note: If crash and DROP trx committed, recover info from DDL_log, and delete files/trees during crash recovery 25

26 How do we implement Atomic and Crash-Safe non-table DDL At slightly detailed level (this is all at SQL layer): Make sure there are no intermediate commits during DDL Write to binary log as part of the DDL transaction Ensure caches for Data Dictionary, routines, events and UDFs are consistent with DDL status Ensure user visible behaviour is atomic NOTE: Behaviour of DROP VIEW changes, for list of views Pre-MySQL 8.0: Giving an error for non-existent views and removing existing views MySQL 8.0: Giving an error for non-existent views 26

27 Atomic DDL Non-TABLE: Use case - CREATE ROUTINE MySQL 5.7 Create routine obj Store obj in DD tables, commit For SF, find VIEWs usage and update view cols metadata for each view. Commit for each view. Invalidate cache Write stored routine statement to binary log MySQL 8.0 Create routine obj Store obj in DD tables For SF, find VIEWs usage and update view cols metadata for each view Write stored routine event to binary log Commit and invalidate cache, or on error, rollback and give msg 27

28 Atomic DDL for user management: Overview MySQL 5.7 Partial execution is allowed Make persistent changes to ACL tables/caches Throw error Error propagation to binary log Required for HA setup Other nodes expect same error while executing the statement MySQL 8.0 Execution is atomic A statement either fully succeeds or is rolled back No error propagation to binary log Only a successful statment makes it way to binary log Simpler handling in HA setup 28

29 Atomic DDL for user management: Use Case - CREATE USER CREATE USER `foo`@`mysql.com` IDENTIFIED BY haha, `bar`@`mysql.com` IDENTIFIED WITH non_existing_plugin BY hoho ; MySQL 5.7 Result foo@mysql.com is created Error is thrown for bar@mysql.com for invalid authentication plugin Binary log Statement is written to binary log with password transformation Expected error for bar@localhost.com is written MySQL 8.0 Result Error because of invalid authentication plugin for bar@localhost.com foo@mysql.com is not created Binary log No statement is written to binary log 29

30 Atomic DDL for user management: Use Case - GRANT GRANT SELECT ON *.* TO `foo`@`mysql.com`, `non_existing@mysql.com`; MySQL 5.7 Result foo@mysql.com get ability to SELECT from all tables/views. Error is thrown for non existing user Binary log Statement is written to binary log Expected error for `non_existing`@`mysql.com` MySQL 8.0 Result Error because of invalid user foo@mysql.com is not granted SELECT Binary log No statement is written to binary log 30

31 Atomic DDL for user management: Note that... IF EXISTS extensions (ALTER USER DROP USER) Will continue to ignore errors related to non-existing authorization IDs All other errors will cause statement rollback IF NOT EXISTS extensions ( CREATE USER CREATE ROLE) Will continue to ignore errors related to existing authorization IDs All other errors will cause statement rollback New MDL type to protect access to in-memory caches for user management 31

32 Effects of new INFORMATION_SCHEMA in MySQL 8.0

33 NEW INFORMATION_SCHEMA in MySQL 8.0 Uniform, simpler implemention makes it a lot faster MySQL Client MySQL Client I_S Query Results I_S Query Results MySQL Server MySQL Server Create temporary table. Return rows to user. TEMP TABLE File system / MyISAM / InnoDB engine Optimizer prepares execution plan. Return rows to user. InnoDB storage engine Heuristic optimization. Executor reads metadata from data dictionary tables. Read metadata from File system or from MyISAM/InnoDB engine.. INFORMATION_SCHEMA in 5.7 INFORMATION_SCHEMA in

34 INFORMATION_SCHEMA Performance and Scalability I_S queries scale, both with database size and query load Typically 30X performance improvements over MySQL 5.7 More than 100X for some queries like: List all InnoDB table columns List all InnoDB tables columns 5k tables List all InnoDB tables columns 10k tables MySQL 8.0 MySQL Time in Seconds (Lower is better) 34

35 INFORMATION_SCHEMA Performance 100 schemas times 50 tables (5000 tables) Count All Schemas Schema aggregate size stats All Dynamic Table Info All Static Table Info Auto Increments Near Limit MySQL 8.0 MySQL 5.7 Count All Columns Count All Indexes Time in Seconds (Lower is better) 35

36 INFORMATION_SCHEMA in MySQL 8.0, community feedback In MySQL 8.0 DMRs : Option information_schema_stats introduced TABLES.TABLE_ROWS, TABLES.DATA_FREE, Handled by opening table and retrieving data from SE in MySQL 5.7 expensive!!! Bugreport: Nicolai Plum bug#83957 and Peter Brawley bug#87548 Feedback from several MySQL ACEs, that found inital solution confusing For MySQL RC1 We have created an improved solution 36

37 INFORMATION_SCHEMA in MySQL New option: information_schema_stats_expiry Default 24h Cache dynamic values in mysql.index_stats and mysql.table_stats Set to 0, always fetch data from SE and no «stats» will be stored ANALYZE will update mysql.index_stats and mysql.table_stats Improved performance fetching dynamic data from SE (InnoDB) In DMR, typically 2-3 times faster than 5.7 In RC1, typically 4-5 times faster than 5.7 Option information_schema_stats removed 37

38 Try for yourself! Downloadable RC1 dev.mysql.com Enjoy and give us your feedback! Thank you for listening For details on atomic DDL: WL#7743, WL#9173, WL#9045, WL#9175, WL#9045, WL#