Using Percona Server in Production. Peter Zaitsev CEO, Percona Inc Percona Live, New York Oct 1, 2012

Transcription

1 Using Percona Server in Production Peter Zaitsev CEO, Percona Inc Percona Live, New York Oct 1, 2012

2 Agenda Percona Server History and Philosophy Installation and Configuration Using Percona Server features

3 Lets start with Phylosophy Performance, Scalability, Transparency, Operational Ease of use Exactly what you need for Running MySQL at Scale! Driven by real customer production challenges Quality. Low risk replacement for high end installations

4 Compatibility 99.99% Compatibility. Everyone claiming 100% is misleading Only transparent changes (such as performance) enabled by default Any behavior changes have to be explicitly enabled

5 Development Focus on Small, Safe high value changes Based on MySQL, follow version convention. Percona Server Predictable release schedule Within 1 month of MySQL release Focus on Customer Needs. Give back to Community. Curated development. Only high value good quality contributions are accepted.

6 Percona Server Origins Set of Patches for MySQL 5.0 July 16, Great Performance Gains

7 XtraDB Drop in Replacement for Innodb Graduating from set of patches to full blown storage engine for MySQL 5.1 Dec 16,2008

8 Stable Performance The performance is only as good as it is in worse of the times

9 MariaDB uses XtraDB Aug 9, 2009 MariaDB uses XtraDB as replacement for Innodb storage engine

10 Performance Improvements for Fast (Flash) storage Innodb was not ready for Flash when it came XtraDB Offered Dramatic Improvements

11 Percona Server Many great changes beyond Storage Engine Released April,

12 Percona Server 5.5 GA Relese April 28, Less than 6 month after MySQL 5.5 GA A lot of Performance Fixes in MySQL 5.5 The Performance gap reduced in many cases

13 Stability and Flash A lot of work done on Stable Performance Flash Optimizations work with Vendors

14 Future Working already on Percona Server 5.6 Expect even less delay with release Better Performance, Scalability, Support for Large Scale deployments and Cloud Percona XtraDB Cluster Simple Scalability and High Availability for MySQL

15 Early benchmarks for MySQL 5.6 Percona Server vs MySQL 5.6 Very fast storage, Small buffer pool Blue PS; Green MySQL 8 BP; Red MySQL 1 Innodb buffer pool.

16 Percona Server Installation Use latest GA version for new installs If replacing MySQL may wish to use same base release PS to replace MySQL Consider using Packages repository for your OS Do not forget to install Shared-Compat and Debuginfo packages.

17 Percona Yum Repository rpm -Uhv

18 Percona Apt Repository Install Percona Key: $ gpg --keyserver hkp://keys.gnupg.net --recv-keys 1C4CBDCDCD2EFD2A... [some output removed]... gpg: imported: 1 $ gpg -a --export CD2EFD2A sudo apt-key add - Add repository to /etc/apt/sources.list deb VERSION main deb-src VERSION main

19 Validation Any changes have risk! You can reduce risks upgrading. Upgrading MySQL Best Practices Webinar recording Download Slides

20 Configuring Percona Server can get you going with starting configuration

21 MySQL Settings to verify Max_connections Set according to your application needs Max_connect_errors Set or so Skip_name_resolve Do not relay on Reverse DNS Back_log Set 1000 or so for high connects/sec

22 MySQL Settings to Verify Thread_cache 64 is good general value Key_cache_size Using MyISAM? Think again! Table_definition_cache Around of number of tables in the system Table_open_cache 4096 can be good value to start

23 MySQL Settings to Verify Open_files_limit Set enough to cover connections and open tables. 64K is good start. Tmpdir Consider using tmpfs (/dev/shm) unless need a lot of space

24 Slow Query Log Slow_query_log Have it configured Log_slow_admin_statements You do not want some slow statements omitted, do not you? Long_query_time Set to some large value (10?) lower when more verbose logging is needed

25 Basic Innodb Settings Innodb_buffer_pool_size Most of your memory spent here Innodb_buffer_pool_instances Set 4-16 depending number of cores Innodb_log_file_size Larger better performance, longer recovery Innodb_log_buffer_size 64M-256M

26 Basic Innodb Settings Innodb_flush_method O_DIRECT best in most cases Innodb_flush_log_at_trx_commit 1 if need full Durability; 2 if not Innodb_file_per_table Enable unless have tables

27 Not so Basic Innodb Settings Innodb_io_capacity Set to number of IOs your system handles Innodb_old_blocks_time Set to Avoid table scans washing away BP Innodb_stats_on_metadata Disable. Magically improve INFORMATION_SCHEMA performance

28 MySQL Configuration More details available Optimizing MyQL Configuration Webinar: Slides:

29 Percona Server Extended Configuration Options

30 Fast Index Creation Fast_index_creation Keep enabled unless have problems with feature. Now pretty stable Expand_fast_index_creation Enable for faster ALTER, OPTIMIZE innodb_merge_sort_block_size Increase for faster index build. 64M good default.

31 Fast Index Creation Reading Selection of Blog posts

32 Potential in One image Adding column on large database ALTER TABLE execution time Red expand_fast_index_creation=1

33 Flushing Configuration innodb_adaptive_flushing_method Multiple methods for different workloads innodb_flush_neighbor_pages cont good for hard drives, none for SSDs innodb_checkpoint_age_target Control effective log file size without restarts

34 Innodb_flush_neighbor_pages Check out Vadim s research

35 Problem Unsolved Percona Server can be tuned to do well in most cases. Rare workloads is still a challenge

36 Fast Warmup innodb_buffer_pool_restore_at_startup Set to 300 or so (time in seconds) innodb_blocking_buffer_pool_restore Do not accept connections until warmup complete.

37 Use latest Percona Server Fast Warmup Older versions could keep Lock for too long while doing the dump Gavin Towey s research

38 Advanced IO Configuration innodb_flush_method =ALL_O_DIRECT Reduce pressure on OS cache. innodb_log_block_size Block size for log files. 4K can be faster for SSDs (breaks data compatibility) innodb_fast_checksum Use different, much faster checksums Breaks data compatibility

39 Innodb Log Block Size Benchmarks on FusionIO Storage

40 IO Configuration for SSDs Innodb_doublewrite_file Storing on conventional RAID (with BBU) to reduce SSD writes Innodb_page_size 4K and 8K pages are better for some workloads. Innodb_ibuf_max_size Limit size if using SSD. Set 256M or less.

41 Other Innodb Settings Innodb_adaptive_hash_index_partitions Reduced contention on Adaptive Hash Latch Innodb_corrupt_table_action=warn Do not assert on corrupted Innodb tables Innodb_lazy_drop_table Reduce cost of dropping Innodb tablespace Drop/truncate w innodb_file_per_table

42 Innodb_lazy_drop_table info Complete solution still in works Number of good Blog posts

43 Misc Innodb Settings Innodb_kill_idle_transactions Kill transactions idle more than specified number of seconds. Innodb_dict_size_limit Restrict internal Innodb dictionary size. Use when have Innodb tables innodb_stats_auto_update=0 Disable automatic table stats update. May help with unstable query plans.

44 Memory usage w Many Tables

45 Misc Innodb Settings Innodb_recovery_update_relay_log Crash Safe Replication on the slave if only Innodb engine is used Innodb_stats_update_need_lock Reduce contention on &dict_operation_lock but makes table information less exact Innodb_show_verbose_locks More information about locks shown

46 Misc Innodb Settings innodb_use_global_flush_log_at_trx_commit Disable to change durability per session Quite handy if different applications are mixed on same DB innodb_use_sys_stats_table Store index cardinality statistics in table instead of sampling Stable plans between restarts A lot faster warmup with large number of tables Can use large innodb_stats_sample_pages to have more exact statistics

47 Special Purpose Settings Innodb_fake_changes Allows using highly efficient replication prefetch by Facebook innodb_import_table_from_xtrabackup Allows moving tables between servers with Percona Xtrabackup innodb_recovery_stats Shows details about Innodb recovery process

48 Innodb_recovery_stats Example text from Error log file Recovery process statistics Checked pages by doublewrite buffer: 128 Overwritten pages from doublewrite: 0 Recovered pages by io_thread: Recovered pages by main thread: 0 Parsed log records to apply: Sum of the length: Applied log records: Sum of the length: Pages which are already new enough: 1855 (It may not be accurate, if turns > 1) Oldest page's LSN: Newest page's LSN:

49 Query Response Time Maintains Query Response time Histogram Very useful as performance fingerprint Query_response_time_stats Enables feature Query_response_time_range_base Defines the exponential step for histogram

50 Example Query Response time Stats Example mysql> select * from query_response_time; time count total TOO LONG 0 TOO LONG rows in set (0.00 sec)

51 User Statistics Userstat Enable user statistics collection Populates USER_STATISTICS, TABLE_STATISTICS, INDEX_STATISTICS, CLIENT_STATISTICS Thread_statistics Tracks per thread statistics What Current and Old threads were doing Beware, can use lots of memory!

52 USER_STATISTICS mysql> select * from user_statistics limit 1 \G *************************** 1. row *************************** USER: root TOTAL_CONNECTIONS: 66 CONCURRENT_CONNECTIONS: 0 CONNECTED_TIME: BUSY_TIME: CPU_TIME: 328 BYTES_RECEIVED: BYTES_SENT: BINLOG_BYTES_WRITTEN: ROWS_FETCHED: ROWS_UPDATED: TABLE_ROWS_READ: SELECT_COMMANDS: 8 UPDATE_COMMANDS: 0 OTHER_COMMANDS: COMMIT_TRANSACTIONS: ROLLBACK_TRANSACTIONS: 0 DENIED_CONNECTIONS: 0 LOST_CONNECTIONS: 0 ACCESS_DENIED: 0 EMPTY_QUERIES: 0 TOTAL_SSL_CONNECTIONS: 0 1 row in set (0.00 sec)

53 CLIENT_STATISTICS mysql> select * from client_statistics limit 1 \G *************************** 1. row *************************** CLIENT: TOTAL_CONNECTIONS: 126 CONCURRENT_CONNECTIONS: 0 CONNECTED_TIME: 0 BUSY_TIME: 0 CPU_TIME: 0 BYTES_RECEIVED: 1455 BYTES_SENT: BINLOG_BYTES_WRITTEN: 0 ROWS_FETCHED: 2289 ROWS_UPDATED: 0 TABLE_ROWS_READ: 0 SELECT_COMMANDS: 3 UPDATE_COMMANDS: 0 OTHER_COMMANDS: 18 COMMIT_TRANSACTIONS: 0 ROLLBACK_TRANSACTIONS: 0 DENIED_CONNECTIONS: 0 LOST_CONNECTIONS: 0 ACCESS_DENIED: 0 EMPTY_QUERIES: 0 TOTAL_SSL_CONNECTIONS: 0 1 row in set (0.01 sec)

54 TABLE_STATISTICS See most actively used tables mysql> select * from table_statistics order by rows_read desc limit 10; TABLE_SCHEMA TABLE_NAME ROWS_READ ROWS_CHANGED ROWS_CHANGED_X_INDEXES sbtest sbtest cacti poller_time cacti poller_item cacti data_template_rrd cacti data_input_fields cacti poller_output cacti host cacti settings cacti poller_reindex mysql user rows in set (0.00 sec)

55 INDEX_STATISTICS See most actively used Indexes mysql> select * from index_statistics order by rows_read desc limit 10; TABLE_SCHEMA TABLE_NAME INDEX_NAME ROWS_READ sbtest sbtest PRIMARY cacti data_input_fields PRIMARY 743 cacti data_template_rrd duplicate_dsname_contraint 563 cacti poller_item rrd_next_step 188 cacti poller_item PRIMARY 188 cacti data_template_rrd local_data_id 180 cacti poller_output PRIMARY 172 cacti poller_reindex PRIMARY 24 cacti host disabled 16 cacti settings PRIMARY rows in set (0.00 sec)

56 Find Unused Indexes Let server long enough for representative load mysql> SELECT INNODB_INDEX_STATS.TABLE_NAME, INNODB_INDEX_STATS.INDEX_NAME from INNODB_INDEX_STATS WHERE CONCAT(INNODB_INDEX_STATS.index_name, INNODB_INDEX_STATS.table_name)NOT IN(SELECT CONCAT(index_statistics.index_name, index_statistics.table_name) FROM index_statistics) AND INNODB_INDEX_STATS.TABLE_SCHEMA='sbtest' AND INNODB_INDEX_STATS.INDEX_NAME<>'GEN_CLUST_INDEX'; TABLE_NAME INDEX_NAME sbtest k sbtest c sbtest pad rows in set (0.00 sec)

57 THREAD_STATISTICS mysql> select * from thread_statistics limit 1 \G *************************** 1. row *************************** THREAD_ID: 54 TOTAL_CONNECTIONS: 1 CONCURRENT_CONNECTIONS: 0 CONNECTED_TIME: 1375 BUSY_TIME: 1326 CPU_TIME: 11 BYTES_RECEIVED: BYTES_SENT: 880 BINLOG_BYTES_WRITTEN: ROWS_FETCHED: ROWS_UPDATED: TABLE_ROWS_READ: SELECT_COMMANDS: 0 UPDATE_COMMANDS: 0 OTHER_COMMANDS: 2532 COMMIT_TRANSACTIONS: 2531 ROLLBACK_TRANSACTIONS: 0 DENIED_CONNECTIONS: 0 LOST_CONNECTIONS: 0 ACCESS_DENIED: 0 EMPTY_QUERIES: 0 TOTAL_SSL_CONNECTIONS: 0 1 row in set (0.00 sec)

58 query_cache_strip_comments First ensure Query Cache is right for you at all! MySQL Query Cache considers queries with different comments different Can kill efficiency with dynamic comments

59 Slow Query Log Settings log_slow_rate_limit=n Log ony 1/Nth of the queries log_slow_rate_type Session log by session Query log by query

60 Slow Query Log Settings log_slow_slave_statements Log statements from slave thread log_slow_sp_statements Log statements contained in stored procedures log_slow_verbosity=full Full level of details in slow query log

61 Slow Query Log Settings slow_query_log_timestamp_always Print timestamp with each query slow_query_log_timestamp_precision =microsecond Microsecond level of accuracy for timestamps slow_query_log_use_global_control=all Change all logging options globally

62 Enforce_storage_engine Enforce_storage_engine=Innodb Ensure users do not create MyISAM tables by accident

63 Using the Features

64 Some are Transparent Performance and Scalability improvements Most are enabled automatically

65 Percona Monitoring Plugins Will use information from some of the features automatically.

66 Memory usage Details

67 Query Response Times

68 Query Numbers

69 Status Counters Innodb_adaptive_hash_cells Innodb_adaptive_hash_heap_buffers 0 Innodb_adaptive_hash_hash_searches 49 Innodb_adaptive_hash_non_hash_searches 966 Innodb_background_log_sync 127 Innodb_buffer_pool_pages_data 455 Innodb_buffer_pool_pages_dirty 0 Innodb_buffer_pool_pages_flushed 0 Innodb_buffer_pool_pages_LRU_flushed 0 Innodb_buffer_pool_pages_free 7735 Innodb_buffer_pool_pages_made_not_young 0 Innodb_buffer_pool_pages_made_young 0 Innodb_buffer_pool_pages_misc 1 Innodb_buffer_pool_pages_old 0 Innodb_buffer_pool_pages_total 8191 Innodb_buffer_pool_read_ahead_rnd 0 Innodb_buffer_pool_read_ahead 0 Innodb_buffer_pool_read_ahead_evicted 0 Innodb_buffer_pool_read_requests 6168 Innodb_buffer_pool_reads 456 Innodb_buffer_pool_wait_free 0 Innodb_buffer_pool_write_requests 0 Innodb_checkpoint_age 0 Innodb_checkpoint_max_age Innodb_checkpoint_target_age

70 SHOW ENGINE INNODB STATUS A lot more information available! BUFFER POOL AND MEMORY Total memory allocated ; in additional pool allocated 0 Internal hash tables (constant factor + variable factor) Adaptive hash index ( ) Page hash (buffer pool 0 only) Dictionary cache ( ) File system ( ) Lock system ( ) Recovery system 0 (0 + 0) Dictionary memory allocated Buffer pool size 8191 Buffer pool size, bytes

71 Large Innodb Log Sizes Can impact Performance AND its Stability

72 Slow Query Log Analyses One of most important features! Works together with pt-query-digest From Percona Toolkit Enable Slow Query Log as Described Logging with long_query_time=0 is important Pt-query_digest slow.log > report.txt

73 It is All Level of Details MySQL Level of Details # Time: :02:26 # User@Host: localhost [] # Query_time: Lock_time: Rows_sent: 1 Rows_examined: use test; SET timestamp= ; select count(*) from auto_inc;

74 More Details Better! Percona Server Level of Details # Time: :11:26 # User@Host: localhost [] # Thread_id: 1 Schema: test Last_errno: 0 Killed: 0 # Query_time: Lock_time: Rows_sent: 1 Rows_examined: Rows_affected: 0 Rows_read: # Bytes_sent: 68 Tmp_tables: 0 Tmp_disk_tables: 0 Tmp_table_sizes: 0 # InnoDB_trx_id: F00 # QC_Hit: No Full_scan: Yes Full_join: No Tmp_table: No Tmp_table_on_disk: No # Filesort: No Filesort_on_disk: No Merge_passes: 0 # InnoDB_IO_r_ops: 984 InnoDB_IO_r_bytes: InnoDB_IO_r_wait: # InnoDB_rec_lock_wait: InnoDB_queue_wait: # InnoDB_pages_distinct: 973 SET timestamp= ; select count(*) from auto_inc;

75 Workload Summary # 8.1s user time, 60ms system time, 26.23M rss, 62.49M vsz # Current date: Thu Dec 29 07:09: # Hostname: somehost.net # Files: slow-query.log.1 # Overall: 20.08k total, 167 unique, QPS, 0.01x concurrency # Time range: :42:47 to 19:03:39 # Attribute total min max avg 95% stddev median # ============ ======= ======= ======= ======= ======= ======= ======= # Exec time 8s 1us 44ms 403us 541us 2ms 98us # Lock time 968ms 0 11ms 48us 119us 134us 36us # Rows sent k # Rows examine k k # Rows affecte 1.34k # Rows read k # Bytes sent 46.63M k 2.38k 6.63k 11.24k # Merge passes # Tmp tables 1.37k # Tmp disk tbl # Tmp tbl size 72.52M k 3.70k k 0 # Query size 3.50M k

76 Workload Summary (2) # InnoDB: # IO r bytes 96.00k k # IO r ops # IO r wait 64ms 0 26ms 13us 0 530us 0 # pages distin 28.96k # queue wait # rec lock wai # Boolean: # Filesort 4% yes, 95% no # Filesort on 0% yes, 99% no # Full scan 4% yes, 95% no # QC Hit 0% yes, 99% no # Tmp table 4% yes, 95% no # Tmp table on 2% yes, 97% no

77 Profile # Profile # Rank Query ID Response time Calls R/Call Apdx V/M Item # ==== ================== ============= ===== ====== ==== ===== ========== # 1 0x92F3B1B361FB0E5B % SELECT wp_options # 2 0xE71D28F50D128F0F % SELECT poller_output poller # 3 0x211901BF2E1C351E % SELECT poller_time # 4 0xA766EE8F7AB % SELECT wp_terms wp_term_taxo # 5 0xA3EEB63EFBA42E9B % SELECT UNION wp_pp_daily_sum # 6 0x94350EA2AB8AAC % UPDATE wp_options # 7 0x7AEDF19FDD3A33F % SELECT wp_options # 8 0x4C FD8EDB % SELECT film # 9 0xCFC0642B5BBD9AC % SELECT UNION wp_pp_daily_su # 10 0x88BA308B9C0EB % SELECT poller_item

78 Query Information # Query 1: 0.26 QPS, 0.00x concurrency, ID 0x92F3B1B361FB0E5B at byte # This item is included in the report because it matches --limit. # Scores: Apdex = 1.00 [1.0], V/M = 0.00 # Query_time sparkline: _^ # Time range: :42:47 to 19:03:10 # Attribute pct total min max avg 95% stddev median # ============ === ======= ======= ======= ======= ======= ======= ======= # Count # Exec time 50 4s 5ms 25ms 13ms 20ms 4ms 12ms # Lock time 3 32ms 43us 163us 103us 131us 19us 98us # Rows sent k # Rows examine k # Rows affecte # Rows read k # Bytes sent M 46.52k 84.36k 81.56k 83.83k 7.31k 79.83k # Merge passes # Tmp tables # Tmp disk tbl # Tmp tbl size # Query size k

79 Query Information 2 # InnoDB: # IO r bytes # IO r ops # IO r wait # pages distin k # queue wait # rec lock wai # Boolean: # Full scan 100% yes, 0% no # String: # Databases wp_blog_one (264/84%), wp_blog_tw (36/11%)... 1 more # Hosts # InnoDB trxid 86B40B (1/0%), 86B430 (1/0%), 86B44A (1/0%) more # Last errno 0 # Users wp_blog_one (264/84%), wp_blog_two (36/11%)... 1 more # Query_time distribution # 1us # 10us # 100us # 1ms ################# # 10ms ################################################################ # 100ms # 1s # 10s+

80 Query Information 3 # Tables # SHOW TABLE STATUS FROM `wp_blog_one ` LIKE 'wp_options'\g # SHOW CREATE TABLE `wp_blog_one `.`wp_options`\g # EXPLAIN /*!50100 PARTITIONS*/ SELECT option_name, option_value FROM wp_options WHERE autoload = 'yes'\g

81 Graphical Tools Available Query Digest UI Box Anemometer

82 Screenshot Example

83 Internal Table Information mysql> select * from innodb_sys_tables limit 5; TABLE_ID SCHEMA NAME FLAG N_COLS SPACE SYS_FOREIGN SYS_FOREIGN_COLS sbtest sbtest test t#p#p test t#p#p rows in set (0.00 sec) mysql> select * from innodb_sys_indexes limit 5; INDEX_ID NAME TABLE_ID TYPE N_FIELDS PAGE_NO SPACE ID_IND FOR_IND REF_IND ID_IND PRIMARY rows in set (0.00 sec)

84 Innodb Table Statistics mysql> select * from innodb_table_stats where table_schema='sbtest'; table_schema table_name rows clust_size other_size modified sbtest sbtest row in set (0.01 sec) mysql> select * from innodb_index_stats where table_schema='sbtest'; table_schema table_name index_name fields rows_per_key index_total_pages index_leaf_pages sbtest sbtest PRIMARY sbtest sbtest k , rows in set (0.00 sec)

85 Buffer Pool Information Innodb_buffer_pool_pages Innodb_buffer_pool_pages_index Innodb_buffer_pool_pages_blob mysql> select * from innodb_buffer_pool_pages_index limit 1; index_id space_id page_no n_recs data_size hashed access_time modified dirty old lru_position fix_count flush_type row in set (0.02 sec)

86 How Well Tables are Cached table_schema table_name index_name cnt dirty hashed fit_pct test a c test a PRIMARY sbtest sbtest#p#p1 PRIMARY sbtest sbtest#p#p0 PRIMARY sbtest sbtest#p#p2 PRIMARY sbtest sbtest#p#p3 PRIMARY sbtest sbtest#p#p3 k sbtest sbtest#p#p2 k sbtest sbtest#p#p1 k sbtest sbtest#p#p0 k stats tables PRIMARY stats tables TABLE_SCHEMA percona transactions PRIMARY rows in set (0.04 sec)

87 Temporary Tables Are no more invisible mysql> select * from global_temporary_tables \G *************************** 1. row *************************** SESSION_ID: 46 TABLE_SCHEMA: test TABLE_NAME: a ENGINE: InnoDB NAME: #sqlf32_2e_0 TABLE_ROWS: 4 AVG_ROW_LENGTH: 4096 DATA_LENGTH: INDEX_LENGTH: 0 CREATE_TIME: NULL UPDATE_TIME: NULL 1 rows in set (0.00 sec)

88 Improved Memory Engine Take less space for VARCHAR columns Support BLOB/TEXT Is not (yet) used for Implicit Memory Tables mysql> create table m(b blob) engine=memory; Query OK, 0 rows affected (0.03 sec)a

89 PAM Authentication Plugin Open Source Alternative to MySQL Enterprise Security More Info

90 Thank You Slides will be published on the conference website for follow-up questions

91 Percona Live New York Sponsors Platinum Sponsors Diamond Sponsors

92 Percona Live New York Sponsors Exhibitor Sponsors Media Sponsors Additional Sponsors

93 Percona Live London London UK, Dec 4-5 Registration is Open! Visit:

94 Annual Percona Live MySQL Conference and Expo April 22nd-25th, 2013 and Call for Papers are Open! Registration Visit: The Hyatt Regency Hotel, Santa Clara, CA