Detecting MySQL IO problems on Linux at different abstraction layers. Nickolay Ihalainen Percona Live London 2011

Transcription

1 Detecting MySQL IO problems on Linux at different abstraction layers Nickolay Ihalainen Percona Live London 2011

2 Agenda Dataflow layers OS tools MySQL instrumentation Inside InnoDB: story of one insert

3 Layers Hardware Level On disk queue NCQ, TCQ and cache RAID controllers queue, caches OS Level BLK device level IO scheduler File system Page cache

4 Software Layers AIO Kernel AIO (for win and Linux with O_DIRECT) Glibc AIO (not used) Normal read/pread and write/pwrite Directories operations MySQL Data, InnoDb simulated AIO Dictionaries (table cache and dictionary cache)

5 High-level tools vmstat procs memory swap io---- -system cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa iostat Device: rrqm/s wrqm/s r/s w/s rmb/s wmb/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda pt-diskstats

6 MySQL IO tracing features Slow log, high verbosity with Percona Server Innodb status, look for pending* information_schema.innodb_io_pattern pt-mext

7 Debug IO PMP strace, syscalls per sec and req per sec

8 Block level read load blktrace /dev/sdg -a issue -a complete -w o - blkiomon -I 10 -h - sizes read (bytes): num 3203, min 4096, max , avg d2c read (usec): num 3203, min 85, max 36013, avg throughput read (bytes/msec): num 3203, min 147, max , avg sizes histogram (bytes): 0: : : : : : : : : : : : 0 d2c histogram (usec): 64: 0 128: : : : : : : : : : : 0

9 Block level write load blktrace /dev/sdg -a issue -a complete -w o - blkiomon -I 10 -h - sizes write (bytes): num 1141, min 4096, max , avg d2c write (usec): num 1141, min 192, max 54462, avg throughput write (bytes/msec): num 1141, min 75, max , avg sizes histogram (bytes): 0: : : : : : : : : : : : 0 d2c histogram (usec): 64: 0 128: 2 256: : : : : : : : : : 0

10 File System level perf stat -e 'ext4:*' -p `pgrep -x mysqld` 1,303 ext4:ext4_mark_inode_dirty 3,357 ext4:ext4_da_write_begin 3,357 ext4:ext4_da_write_end 634 ext4:ext4_da_writepages 1,328 ext4:ext4_da_write_pages 634 ext4:ext4_da_writepages_result 634 ext4:ext4_sync_file_enter 634 ext4:ext4_sync_file_exit 10 ext4:ext4_da_reserve_space 580 ext4:ext4_ext_map_blocks_enter 580 ext4:ext4_ext_map_blocks_exit seconds time elapsed

11 File System level cont. systemtap, tools like disktop.stp, io_submit.stp, and more

12 Understand how mysql & innodb works Use non-stripped sources with -g enabled GDB Full strace -ff listing systemtap with ubacktrace

13 Avoid high IO load More RAM Avoid randomness Random primary keys Useless updates/inserts Redundant indexes and fields Decrease number of rows fetched by using correct indexes More disks in raid and/or SSD disks Always test RAID adapters

14 Benchmarking Bare-metal disk with dumb RAID Raid adapter (stripe size and etc) File system Tools dd, dbench iozone sysbench Replaying general log/queries from tcpdump Mysql replication slave

15 Story of one INSERT Preparation #!/usr/bin/env stap # use: innodbio.stp --vp d /usr/sbin/mysqld --ldd -x `pgrep -x mysqld` c++filt tee -a innodbio.log probe process("mysqld").function("os_file*") { printf("%d %d %s\n",gettimeofday_s(),tid(), probefunc() ); print_ustack(ubacktrace()); printf("\n"); } probe process("mysqld").function("os_aio*") { printf("%d %d %s\n",gettimeofday_s(),tid(), probefunc() ); print_ustack(ubacktrace()); printf("\n"); }

16 Story of one INSERT A long way to open table os_aio 0x7f8bd7113b60 : os_aio+0x0/0x550 [/usr/sbin/mysqld] 0x7f8bd70d813d : fil_io+0x1dd/0x4b0 [/usr/sbin/mysqld] 0x7f8bd70bc3f5 : buf_read_page_low+0xd5/0x210 [/usr/sbin/mysqld] 0x7f8bd70bc755 : buf_read_page+0x225/0x3e0 [/usr/sbin/mysqld] 0x7f8bd70b33d7 : buf_page_get_gen+0x147/0xa50 [/usr/sbin/mysqld] 0x7f8bd70a0d69 : btr_cur_search_to_nth_level+0x399/0xf40 [/usr/sbin/mysqld] 0x7f8bd : btr_pcur_open_on_user_rec+0x64/0x2e0 [/usr/sbin/mysqld] 0x7f8bd70c77b1 : dict_load_table +0x1f1/0x2450 [/usr/sbin/mysqld] 0x7f8bd70c65a0 : dict_table_get+0x160/0x190 [/usr/sbin/mysqld] 0x7f8bd709cd0d : ha_innobase::open(char const*, int, unsigned int)+0x1cd/0x6a0 [/usr/sbin/mysqld]

17 Story of one INSERT A long way to open table os_aio 0x7f8bd7113b60 : os_aio+0x0/0x550 [/usr/sbin/mysqld] 0x7f8bd70d813d : fil_io+0x1dd/0x4b0 [/usr/sbin/mysqld] 0x7f8bd70bc3f5 : buf_read_page_low+0xd5/0x210 [/usr/sbin/mysqld] 0x7f8bd70bc755 : buf_read_page+0x225/0x3e0 [/usr/sbin/mysqld] 0x7f8bd70b33d7 : buf_page_get_gen+0x147/0xa50 [/usr/sbin/mysqld] 0x7f8bd70a0d69 : btr_cur_search_to_nth_level+0x399/0xf40 [/usr/sbin/mysqld] 0x7f8bd : btr_pcur_open_on_user_rec+0x64/0x2e0 [/usr/sbin/mysqld] 0x7f8bd70ca5ba : dict_load_indexes+0x26a/0x13e0 [/usr/sbin/mysqld] 0x7f8bd70c82d4 : dict_load_table+0xd14/0x2450 [/usr/sbin/mysqld] 0x7f8bd70c65a0 : dict_table_get+0x160/0x190 [/usr/sbin/mysqld] 0x7f8bd709cd0d : ha_innobase::open(char const*, int, unsigned int)+0x1cd/0x6a0 0x7f8bd6fde56d : handler::ha_open(st_table*, char const*, int, int)+0x3d/0x180

18 Story of one INSERT A long way to open table: statistics os_file_create_simple_no_error_handling 0x7f8bd : os_file_create_simple_no_error_handling+0x0/0x1a0 [/usr/sbin/mysqld] 0x7f8bd70cf8c0 : fil_node_open_file+0x170/0x440 [/usr/sbin/mysqld] 0x7f8bd70cfbe9 : fil_node_prepare_for_io+0x59/0x190 [/usr/sbin/mysqld] 0x7f8bd70d8986 : fil_space_get_size+0xe6/0x140 [/usr/sbin/mysqld] 0x7f8bd70bc5ce : buf_read_page+0x9e/0x3e0 [/usr/sbin/mysqld] 0x7f8bd70b33d7 : buf_page_get_gen+0x147/0xa50 [/usr/sbin/mysqld] 0x7f8bd7170c0f : btr_get_size+0x1af/0x330 [/usr/sbin/mysqld] 0x7f8bd70c631b : dict_update_statistics_low +0x4b/0x170 [/usr/sbin/mysqld] 0x7f8bd70c6554 : dict_table_get+0x114/0x190 [/usr/sbin/mysqld] 0x7f8bd709cd0d : ha_innobase::open(char const*, int, unsigned int)+0x1cd/0x6a0 [/usr/sbin/mysqld]

19 Story of one INSERT Statistics is not finished: os_file_pread os_file_close os_file_create os_file_lock os_aio reads Insert buffer operations: ibuf_merge_or_delete_for_page Reserved pages fseg_n_reserved_pages

20 Story of one INSERT Start transaction os_aio 0x7f8bd70b33d7 : buf_page_get_gen+0x147/0xa50 [/usr/sbin/mysqld] 0x7f8bd715f34b : trx_sys_flush_max_trx_id+0x9b/0xe0 [/usr/sbin/mysqld] 0x7f8bd7161e05 : trx_start_low+0xe5/0x1c0 [/usr/sbin/mysqld] 0x7f8bd7161f4f : trx_start+0x6f/0xe0 [/usr/sbin/mysqld] Read required pages before transaction start Save rowid in header file page dict_hdr_flush_row_id

21 Story of one INSERT Commit XA two phase fixation os_aio 0x7f8bd7113b60 : os_aio+0x0/0x550 [/usr/sbin/mysqld] 0x7f8bd70d813d : fil_io+0x1dd/0x4b0 [/usr/sbin/mysqld] 0x7f8bd70ffd6b : log_group_write_buf+0x1ab/0x370 [/usr/sbin/mysqld] 0x7f8bd710037e : log_write_up_to+0x44e/0x7d0 [/usr/sbin/mysqld] 0x7f8bd71644c9 : trx_prepare_off_kernel+0x2d9/0x330 [/usr/sbin/mysqld] 0x7f8bd : trx_prepare_for_mysql+0x69/0x180 [/usr/sbin/mysqld] 0x7f8bd709830b : innobase_xa_prepare(handlerton*, THD*, bool)+0xeb/0x150 [/usr/sbin/mysqld] And the same for normal logs

22 Story of one INSERT Periodic operations: Write dirty pages os_aio os_aio 0x7f8bd7113b60 : os_aio+0x0/0x550 [/usr/sbin/mysqld] 0x7f8bd70d813d : fil_io+0x1dd/0x4b0 [/usr/sbin/mysqld] 0x7f8bd70b69ba : buf_flush_buffered_writes+0x26a/0x620 [/usr/sbin/mysqld] 0x7f8bd70b75f7 : buf_flush_batch+0x157/0x11f0 [/usr/sbin/mysqld] 0x7f8bd714b666 : srv_master_thread+0xbe6/0xc60 [/usr/sbin/mysqld] And log_checkpoint is called from the same thread

23 Thank You to Our Sponsors Platinum Sponsor Gold Sponsor Silver Sponsors

24 Percona Live London Sponsors Exhibitor Sponsors Friends of Percona Sponsors Media Sponsors

25 Annual MySQL Users Conference Presented by Percona Live The Hyatt Regency Hotel, Santa Clara, CA April 10th-12th, 2012 Featured Speakers Mark Callaghan, Facebook Jeremy Zawodny, Craigslist Marten Mickos, Eucalyptus Systems Sarah Novotny, Blue Gecko Peter Zaitsev, Percona Baron Schwartz, Percona The Call for Papers is Now Open! Visit /live/mysql-conference-2012/

26 We're Hiring! /about-us/careers/

27 /live