Wednesday, May 15, 13

Transcription

1 1

2 <Insert Picture Here> Whats New With MySQL 5.6 Ligaya Turmelle Principle Technical Support Engineer - MySQL

3 <Insert Picture Here> About Me MySQL Support Engineer MySQL ACE ~10 3

4 The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle s products remains at the sole discretion of Oracle. 4 Safe harbor statement

5 MySQL world's most popular open source database software a key part of LAMP (Linux, Apache, MySQL, PHP / Perl / Python) Site: Developer Zone: mysql.org or dev.mysql.com Download: Online Manual: 5.6/en/index.html

6 Agenda Security InnoDB Optimizer Replication Performance Schema 6 Newest GA version In each of the various areas I am only going to be going over a few of the (in my opinion) choicer improvements or enhancements. To learn more you can check out the manual page What Is New in MySQL 5.6

7 Security 7

8 User Account Security 8

9 User Account Security Optional SHA-256 password hashing 9 Plugin supports stronger encryption for user account passwords via built in sha256_password plugin. See

10 User Account Security Optional SHA-256 password hashing Example: mysql> CREATE USER IDENTIFIED WITH sha256_password; Query OK, 0 rows affected (0.00 sec) mysql> SET old_passwords = 2; Query OK, 0 rows affected (0.00 sec) mysql> SET PASSWORD FOR 'sha256user'@'localhost' = PASSWORD('sha256P@ss'); Query OK, 0 rows affected (0.09 sec) 10 old_passwords value sets the password hashing format See mysql> SELECT plugin FROM mysql.user WHERE User = 'sha256user'; plugin sha256_password row in set (0.00 sec)

11 User Account Security mysql.user.password_expired 11 default value is 'N', but can be set to 'Y' with ALTER USER statement. (Think cron job to set it) Built in. NOT for an application password. After expired, all operations performed in *subsequent* connections to the server using the account - result in an error until the user issues a SET PASSWORD statement. ALTER USER man page -

12 User Account Security mysql.user.password_expired Ex: Expire a users password mysql> CREATE USER 'test'@'localhost'; Query OK, 0 rows affected (0.00 sec) mysql> ALTER USER 'test'@'localhost' PASSWORD EXPIRE; Query OK, 0 rows affected (0.00 sec) 12 default value is 'N', but can be set to 'Y' with ALTER USER statement. (Think cron job to set it) After an account's password has been expired, all operations performed in subsequent connections to the server using the account result in an error until the user issues a SET PASSWORD statement to establish a new account password. ALTER USER man page -

13 User Account Security mysql.user.password_expired Ex: Connecting as that user mysql> SELECT 1; ERROR 1820 (HY000): You must SET PASSWORD before executing this statement mysql> SET PASSWORD = PASSWORD('new_password'); Query OK, 0 rows affected (0.00 sec) mysql> SELECT 1; row in set (0.00 sec) After an account's password has been expired, all operations performed in subsequent connections to the server using the account result in an error until the user issues a SET PASSWORD statement to establish a new account password. ALTER USER man page

14 User Account Security Password strength Validate_password plugin 3 Policies Low, Medium, Strong 7 server settings available to control options 14 Plugin Only checks passwords supplied as a cleartext value. This affects the CREATE USER, GRANT, and SET PASSWORD statements. Passwords given as arguments to the PASSWORD() and OLD_PASSWORD() functions are checked as well. 0 or LOW Length (min 8) 1 or MEDIUM Length; numeric (1), (1)lowercase/(1)uppercase, and (1)special characters 2 or STRONG Length; numeric (1), (1)lowercase/(1)uppercase, and (1)special characters; password substrings of length 4 or longer must not match dictionary file --validate_password: how the plugin is loaded --validate_password_dictionary_file: path name of the dictionary file --validate_password_length: minimum number of characters -- validate_password_mixed_case_count: minimum number of lowercase and uppercase characters --validate_password_number_count: minimum number of numeric (digit) characters --validate_password_policy: password policy enforced See

15 InnoDB 15

16 InnoDB Enhancements Fulltext Online DDL Tablespace Management Dump, Restore/Warm Buffer Pool NoSQL interface Improved Performance & Scalability 16 Data Definition Language

17 InnoDB: Fulltext Not just for MyISAM any more Physically represented as a InnoDB table Uses MATCH()... AGAINST syntax in SELECT Fully transactional, fast look up Natural language/boolean modes It can do a lot more then what I mention, but I am not strong on Fulltext matching to explain it all. See the manual. 17

18 InnoDB: Online DDL Performing the operation in-place Improves responsiveness and availability Concurrency vs. Responsiveness LOCK=EXCLUSIVE LOCK=SHARED LOCK=NONE Avoids temp increases in disk space usage/io DML operations (INSERT, UPDATE, DELETE, etc) and queries run on the table while the DDL (Ex: ALTER) is in progress - for example if you were modify its indexes or column definitions - you don t have to wait for the table to be rebuilt. - You control these (C&L) with the LOCK clause. - omit the LOCK clause or specify LOCK=DEFAULT, MySQL allows as much concurrency as possible depending on the *type of operation*. - If the specified LOCK clause for an operation is not available - it acts as an assertion and errors out. - block access to the table entirely (LOCK=EXCLUSIVE clause), - allow SELECT but not DML (LOCK=SHARED clause), - allow full query and DML access to the table (LOCK=NONE clause). - By doing the changes in-place where possible, rather than creating a new copy of the table, it avoids temporary increases in disk space usage and the I/O overhead of copying the table and reconstructing all the secondary indexes. 18

19 InnoDB: Tablespace Management Works with innodb_file_per_table Easier to reclaim space Specific tables on separate devices Transportable tablespaces Compression for tables and indexes Dynamic Row Format 19 Historically, all InnoDB tables and indexes were stored in the system tablespace. InnoDB's file-pertable mode is a more flexible alternative, where you store each InnoDB table and its indexes in a separate file. Each such.ibd file represents a separate tablespace. Advantages: - can reclaim disk space when truncating or dropping a table. - store specific tables on separate storage devices (Ex: SSD), for I/O optimization, space management, or backup purposes. - copy individual InnoDB tables from one MySQL instance to another. Todd Farmer has a blog post on how to copy over a table working only with the tablespace. FLUSH TABLE... FOR EXPORT, UNLOCK TABLE, ALTER TABLE DISCARD TABLESPACE, ALTER TABLE IMPORT TABLESPACE See - Compress to save space and IO - used with SSD, compressed OLTP workloads are now possible - If it fits, the entire row is stored in the index node. Otherwise long data values are stored off-page.

20 Bullet 3: - Although the buffer pool itself could be many gigabytes in size, the data that InnoDB saves on disk to restore the buffer pool is tiny by comparison: just the tablespace and page IDs necessary to locate the appropriate pages on disk. InnoDB: Dump, Restore/Warm Buffer Shortened Buffer Warmup At shutdown/startup or manually Examples: - Specify that a dump should be taken at shutdown: SET innodb_buffer_pool_dump_at_shutdown=on; - Specify that a dump should be loaded at startup: SET innodb_buffer_pool_load_at_startup=on; - Trigger a dump of the buffer pool manually: SET innodb_buffer_pool_dump_now=on; - Trigger a load of the buffer pool manually: SET innodb_buffer_pool_load_now=on; Small footprint on disk Bullet 1: After you restart a busy server, there is typically a warmup period with steadily increasing throughput, as disk pages that were in the InnoDB buffer pool are brought back into memory as the same data is queried, updated, and so on. Once the buffer pool holds a similar set of pages as before the restart, many operations are performed in memory rather than involving disk I/O, and throughput stabilizes at a high level. This feature shortens the warmup period by immediately reloading disk pages that were in the buffer pool before the restart, rather than waiting for DML operations to access the corresponding rows. The I/O requests can be performed in large batches, making the overall I/O faster. The page loading happens in the background, and does not delay the database startup. Especially important with large buffer pools - Warmup can go from hours to minutes Bullet 2: - Control of when to dump and load the buffer pool is controlled by server options. 20

21 InnoDB: NoSQL Interface Direct access to InnoDB tables Key-value access via the memcached API Data Accessible via SQL mysqlnd_memcache PHP extension - bypassing the SQL parser, the optimizer, and even the Handler API layer. - you use the memcached protocol to read and write data directly to InnoDB, and let MySQL manage the in-memory caching through the InnoDB buffer pool. - You can still access the underlying table through SQL, for reporting, analysis, ad hoc queries, bulk loading, set operations such as union and intersection, and other operations well suited to the expressiveness and flexibility of SQL. - mysqlnd_memcache PHP extension transparently translating SQL into requests for the MySQL InnoDB Memcached Daemon Plugin (server plugin). Still early so you are warned. 21

22 InnoDB: Performance & Scalability Internal Improvements Persistent Optimizer Statistics Increased plan stability, accurate statistics Better user control, automatic/manual SSD Optimizations 4/8K page sizes.ibd files can be outside the datadir separate tablespace for undo logs Bullet 1: reduced contention by splitting the kernel mutex, moving flushing operations from the main thread to a separate thread, enabling multiple purge threads, and reducing contention for the buffer pool on large-memory systems. Bullet 2: sudden changes in query execution plans can cause huge performance issues with a system. You now can control how much sampling is done and when it will be done. - On by default, enabled by the configuration option innodb_stats_persistent. - How much done: innodb_stats_persistent_sample_pages and innodb_stats_transient_sample_pages - When it is done: ANALYZE TABLE and innodb_stats_auto_recalc - determines whether the statistics are calculated automatically whenever a table undergoes substantial changes (to more than 10% of the rows) - To revert to the previous method of collecting statistics that are periodically erased, run the command ALTER TABLE tbl_name STATS_PERSISTENT=0. Bullet 3: SSDs are finding there way into more and more systems - Innodb page size default is 16K. The 4/8K sizes can work better with SSD. Set with innodb_page_size config - Tables with high levels of random read/writes may have improved performance on SSDs - the IO patterns for the undo log (rollback segments) work well with SSD 22

23 <Insert Picture Here> More Information on InnoDB Improvements 23 A lot more then what I covered was done. To learn more, be sure to check out the web site.

24 Optimizer 24

25 EXPLAIN Plan Now also works with DELETE INSERT UPDATE REPLACE Optional JSON output 25 EXPLAIN EXTENDED FORMAT=JSON UPDATE...

26 ORDER BY.. Limit Optimization Stop sorting when X first rows of sorted result found How it works If ordering done by index - fast If filesort All rows that match the query without the LIMIT clause are selected Most or all of them are sorted, until the first X are found. Sort stops If able to fit - will do it all in the sort_buffer_size 26 Avoids merge file and does it all in memory.

27 Multi-Range Read Reduce random disk accesses How it works Range scan of secondary index Collect PK values of chosen rows Order the PK values retrieve base table data by ordered PK Extra shows Using MRR 27 leads to a more sequential scan of the base table

28 Index Condition Pushdown To understand the index condition pushdown optimization you have to have a basic understanding of MySQL s API architecture. A query comes in (SQL interface), - is parsed and the syntax is checked (Parser), - an execution plan is determined (optimizer), - execution plan sent to the query processor. (Caches & Buffers would be something like the query cache.). * query processor then ** takes the query plan and generates a queue of method calls (Handler calls to the API interface of the various storage engines). - The storage engine handles the actual data/file/disk access. Different storage engines handle these requirements differently. For example the MyISAM storage engine caches (key buffer), the indexes of its tables but leaves the caching of the actual data to the operating system. InnoDB on the other hand tries to cache everything within the buffer pool. If it can t it uses various techniques to be smart about your needs. 28

29 Index Condition Pushdown Works with your indexes Reduce accesses to base table Potentially reduce rows returned and handled in the server How does it work Example without Example with Without: 1. In the storage engine, get the next row, first by reading the index tuple, and then by using the index tuple to locate and read the full table row. 2. Pass the full table row out of the storage engine back to the MySQL Server and test the part of the WHERE condition that applies to this table. Accept or reject the row based on the test result. With: 1. In the storage engine, get the next row's index tuple (but not the full table row). 2. In the storage engine, test the part of the WHERE condition that applies to this table *and* can be checked using only index columns. - If the condition is not satisfied, proceed to the index tuple for the next row. - If the condition is satisfied, use the index tuple to locate and read the full table row. 3. Pass this out of the storage engine back to the MySQL Server and test the remaining part of the WHERE condition that applies to this table. Accept or reject the row based on the test result. 29

30 Tracing the Optimizer Primarily for developers See how the optimizer chooses the execution plan Works for SELECT INSERT / REPLACE UPDATE DELETE 30 SET OPTIMIZER_TRACE="enabled=on",END_MARKERS_IN_JSON=on;

31 Replication 31

32 Global Transaction ID Unique Identifier across all servers pair of coordinates Ex: source_id:transaction_id Transaction based No need for log files or positions anymore Preserved between master and slave Easier failover * source_id = originating server - typically server_uuid * transaction_id = sequence number determined by the order in which the transaction was committed on the source_id server - can not be 0 - can be given as a range. Ex: 1-5 * All data for synchronizing replication is taken from the replication stream. No more MASTER_LOG_FILE and MASTER_LOG_POS in CHANGE MASTER. Just use MASTER_AUTO_POSITION * Can always determine the source for any transaction applied on any slave. Also once a transaction with a given GTID is committed on a given server, any subsequent transaction having the same GTID is ignored by that server. * the mysqlfailover utility 32

33 Setup Replication with GTID Key Steps Synchronize the servers & read only 33 Cold start - replication for the first time or able to stop the servers - Step 1: = ON;

34 Setup Replication with GTID Key Steps Synchronize the servers & read only Stop the servers 34 - Step 2: mysqladmin -u user -p shutdown Cold start - replication for the first time or able to stop the servers - Step 1: = ON;

35 Setup Replication with GTID Key Steps Synchronize the servers & read only Stop the servers Configure & restart both servers grid_mode=on log-bin log-slave-updates enforce-gtid-consistancy read-only skip-slave-start - Step 3: Bare minimum shell> mysqld_safe --gtid_mode=on --log-bin --log-slave-updates --enforce-gtid-consistency & Additionally read-only to prevent updates and skip-slave-start since we have not told it about the master yet 35 Cold start - replication for the first time or able to stop the servers - Step 1: = ON; - Step 2: mysqladmin -u user -p shutdown

36 Setup Replication with GTID Key Steps Synchronize the servers & read only Stop the servers Configure & restart both servers grid_mode=on log-bin log-slave-updates enforce-gtid-consistancy read-only skip-slave-start CHANGE MASTER & restart slave 36 - Step 4: mysql> CHANGE MASTER TO > MASTER_HOST = host, > MASTER_PORT = port, > MASTER_USER = user, > MASTER_PASSWORD = password, > MASTER_AUTO_POSITION = 1; mysql> START SLAVE; Cold start - replication for the first time or able to stop the servers - Step 1: = ON; - Step 2: mysqladmin -u user -p shutdown - Step 3: Bare minimum shell> mysqld_safe --gtid_mode=on --log-bin --log-slave-updates --enforce-gtid-consistency & Additionally read-only to prevent updates and skip-slave-start since we have not told it about the master yet

37 Setup Replication with GTID Key Steps Synchronize the servers & read only Stop the servers Configure & restart both servers grid_mode=on log-bin log-slave-updates enforce-gtid-consistancy read-only skip-slave-start CHANGE MASTER & restart slave Disable read-only 37 - Step 5: mysql> = OFF; Cold start - replication for the first time or able to stop the servers - Step 1: = ON; - Step 2: mysqladmin -u user -p shutdown - Step 3: Bare minimum shell> mysqld_safe --gtid_mode=on --log-bin --log-slave-updates --enforce-gtid-consistency & Additionally read-only to prevent updates and skip-slave-start since we have not told it about the master yet - Step 4: mysql> CHANGE MASTER TO > MASTER_HOST = host, > MASTER_PORT = port, > MASTER_USER = user, > MASTER_PASSWORD = password, > MASTER_AUTO_POSITION = 1; mysql> START SLAVE;

38 mysqlfailover Utility Python Replication health monitoring Handles Failover auto elect fail Automatic!== Instantaneous 38 - Only works with GTIDs - 3 failover modes - auto = failover to the list of slave candidates. Can t find any viable candidate, look at all slaves attached to master for a viable candidate, still none - error out. - elect = failover to the list of slave candidates. Can t find any viable candidate - error out - fail = error out Stupid easy

39 Multi-threaded Slaves Increases slave throughput and reduces lag Break up concurrent SQL threads by database Assumes Data and updates partitioned by database Updates occur in same relative order as on master not necessary to coordinate transactions between DBs # threads controlled by slave_parrallel_workers Since transactions on different databases can occur in a different order on the slave than on the master, simply checking for the most recently executed transaction is not a guarantee that all previous transactions on the master have been executed on the slave. This has implications for logging and recovery when using a multi-threaded slave. 39

40 Row Image Control Works with RBR Save disk space, network & memory Control over what is placed in binlog binlog_row_image minimal full noblob PK Changed Columns MySQL RBR originally always logged the entire row that was changed. This lead to the potential of RBR binary logs being much larger then SBR binary logs. - minimal = log only required columns - full = log all columns - noblob = log all columns except unneeded blob/text columns 40

41 Event Checksums Ensures replicated data is correct, consistent and accessible Detects corrupt events before they re applied Returns an error Protects entire replication path Memory Disk Network 41 to write checksums for the events - binlog_checksum=crc32 to read checksums from the binary log - master_verify_checksum =ON SQL thread to read checksums from relay log -slave-sql-verify-checksum=1

42 Performance Schema 42

43 General Information Monitor MySQL Server execution at a low level Can be configured dynamically Data held in performance_schema DB Can be queried with SELECT Views/Temp Tables - no persistance Activation does not alter server behavior Some loss of performance - configure dynamically by updating tables in the P_S DB with SQL. Affects collection immediately - There are various blog posts/benchmarks and bug reports on it. I have heard of values as low as 5% and as high as 28%. You can disable if you really need it. 43

44 New Instrumentation Table and Index IO Statements and stages within statements Table Locks Users/Hosts/Accounts Network IO 44

45 Using Performance Schema Controlling it What can be watched: mysql> SELECT * FROM performance_schema.setup_instruments; Turn on watchers mysql> UPDATE setup_instruments SET enabled = YES WHERE NAME LIKE 'wait/io/file/innodb/%'; Turn off watchers mysql> UPDATE setup_instruments SET enabled = NO WHERE NAME LIKE 'wait/io/file/innodb/%'; 45

46 Using Performance Schema ps_helper Created by Mark Leith Collection of views and procedures Examples: View: top_io_by_thread View: statements_with_full_table_scans View: schema_tables_with_full_table_scans View: schema_unused_indexes * top_io_by_thread: Show the top IO consumers by thread, ordered by total latency. (5.5) * statements_with_full_table_scans: Lists all normalized statements that use have done a full table scan ordered by the percentage of times a full scan was done, then by the number of times the statement executed (5.6) * schema_tables_with_full_table_scans: Find tables that are being accessed by full table scans, ordering by the number of rows scanned descending * schema_unused_indexes: Find indexes that have had no events against them (and hence, no usage) for the the lifetime of the server or since statistics were last truncated (5.6) 46

47 Conclusion Lot of changes Security InnoDB Optimizer Replication Performance Schema In each of the various areas I am only going to be going over a few of the (in my opinion) choicer improvements or enhancements. To learn more you can check out the manual page What Is New in MySQL

48 Questions? 48

49 San Francisco, September Additional Day of Tutorials Oracle.com/mysqlconnect Early Bird Discount: Register Now and Save US$500! Keynotes Conferences Sessions Birds-of-a-feather sessions Tutorials Hands-on Labs Exhibition Hall Demo Pods Receptions

50 50