XtraDB 5.7: Key Performance Algorithms. Laurynas Biveinis Alexey Stroganov Percona

Transcription

1 XtraDB 5.7: Key Performance Algorithms Laurynas Biveinis Alexey Stroganov Percona

2 XtraDB 5.7 Key Performance Algorithms Focus on the buffer pool, flushing, the doublewrite buffer Talk assumes familiarity, but feel free to interrupt What we learned What we did How we did it

3 InnoDB buffer pool Memory cache of disk data pages In-memory data pages accessible through several data structures 1) Page hash for lookup Hash array Data page lists (space_id; page_id)? Fold

4 InnoDB buffer pool 2) flush list for dirty page management. Dirtying: Clean page INSERT INTO foo VALUES(bar) Dirty page, LSN = 42 Flush list tail: Dirty page, LSN = 25 Dirty page, LSN = 32 Dirty page, LSN = 42

5 InnoDB buffer pool 2) flush list for dirty page management. Flushing: Flush list head: Dirty page, LSN = 5 Dirty page, LSN = 7 Dirty page, LSN = 12 Flush up to LSN 10 Clean page Clean page Flush list head: Dirty page, LSN = 12

6 InnoDB buffer pool 3) LRU list for deciding which pages to evict Preventing eviction for recently-used pages (making them young): Clean page Dirty page Dirty page Clean page Clean page Page access Clean page Clean page Dirty page Clean page Dirty page

7 InnoDB buffer pool 4) free list for having free space in the buffer pool to read currently non-present pages. Reading: Free page Free page Free page Free page Free page Preparing to read a page Empty page Free page Free page Free page Free page

8 InnoDB buffer pool 3/4) Evicting/flushing pages from the LRU list and putting them on the free list: Clean page Dirty page Dirty page Clean page Clean page Free page Free page Free page Free page Dirty page Dirty page Clean page Clean page Free page Free page Free page Free page Free page

9 The doublewrite buffer Step 1 Data page Step 3 Add Write Step 2 Doublewrite buffer in memory Flush Doublewrite buffer in disk Data file

10 Buffer pool concurrency page hash latch flush list mutex buffer pool mutex LRU list page hash flush list free list misc.

11 Buffer pool instances page hash latch page hash flush list mutex flush list buffer pool mutex LRU list buffer pool instance 0 free list misc. page hash latch page hash flush list mutex flush list buffer pool mutex LRU list buffer pool instance 1 free list misc.

12 Buffer pool instances Problem: some instances are cold and some are hot First the accesses to the buffer pools is in no way evenly spread out. Six year-old quote, still relevant the same today

13 Concurrency in XtraDB page hash latch flush list mutex LRU list mutex free list mutex misc mutex / atomics page hash flush list LRU list free list misc Patch contributed to MySQL, and merged in

14 Concurrency solutions are compatible page hash latch flush list mutex LRU list mutex free list mutex misc mutex / atomics page hash flush list LRU list free list misc buffer pool instance 0 page hash latch flush list mutex LRU list mutex free list mutex misc mutex / atomics page hash flush list LRU list free list misc buffer pool instance 1

15 Buffer pool mutexes are so 5.5 Improvement by the buffer pool mutex split Improvement by adaptive flushing

16 5.6+ changed things In 5.5 and earlier: reduce mutex contention by X%, observe TPS increase by ~X% Changing flushing heuristics is driven by performance stability, not necessarily by peak performance Pre-release Percona Server 5.6: reduce mutex contention by X%, observe TPS increase by ~0% What happened? InnoDB cleaner thread happened

17 Buffer pool / flushing concurrency in 5.5 Time Master thread Query thread 1 Query thread 2 flush list flush flush list flush make page young make page young make page young LRU list flush flush list flush LRU list flush make page young

18 Buffer pool / flushing concurrency in 5.6+ Time Cleaner thread Query thread 1 Query thread 2 LRU list flush flush list flush LRU list flush flush list flush make page young make page young make page young LRU list flush flush list flush make page young

19 Buffer pool / flushing concurrency in 5.6+ In 5.6+, code-level changes to reduce locking granularity are still important, but Increasing thread specialization means that flushing - including LRU - heuristics are very important now

20 MySQL 5.7 multi-threaded flushing coordinator thread LRU instance #0 flush list instance #0 LRU worker thread #0 LRU instance #1 flush list instance #1 LRU worker thread #1 LRU instance #2 flush list instance #2 LRU time 0 s 1 s

21 MySQL OLTP_RW

22 PFS data is incomplete

23 MySQL OLTP_RW 660 pthread_cond_wait,enter (ib0mutex.h:850), buf_dblwr_write_single_page (ib0mutex.h:850),buf_flush_write_block_low(buf0flu.cc:1096),buf_flush_page (buf0flu.cc1096),buf_flush_single_page_from_lru (buf0flu.cc:2217), buf_lru_get_free_block(buf0lru.cc:1401), pthread_cond_wait,buf_dblwr_write_single_page (buf0dblwr.cc:1213), buf_flush_write_block_low(buf0flu.cc:1096),buf_flush_page (buf0flu.cc:1096), buf_flush_single_page_from_lru (buf0flu.cc:2217), buf_lru_get_free_block(buf0lru.cc:1401), pthread_cond_wait,policymutex<ttaseventmutex<genericpolicy> (ut0mutex.ic:89),get_next_redo_rseg (trx0trx.cc:1185), trx_assign_rseg_low(trx0trx.cc:1278),trx_set_rw_mode (trx0trx.cc:1278), lock_table(lock0lock.cc:4076),...

24 631 pthread_cond_wait,buf_dblwr_write_single_page

25 Single-page flushing Query thread needs a free page Yes Is free page available? No Take a free page from the free list Single-page flush Single-page doublewrite Query thread has a free page

26 XtraDB innodb_empty_free_list_algorithm=backoff Query thread needs a free page Yes Is free page available? No Wait Take a free page from the free list Single-page flush Single-page doublewrite Query thread has a free page

27 MySQL 5.7 multi-threaded flushing Single page flushes! coordinator thread free pages worker thread #0 free pages worker thread #1 LRU instance #0 flush list instance #0 LRU instance #1 flush list instance #1 LRU instance #2 flush list instance #2 LRU LRU LRU time 0 s 1 s

28 Percona Server 5.7 multithreaded flushing LRU flusher #0 LRU instance #0 LRU instance #0 LRU free pages LRU flusher #1 free pages coordinator worker #0 LRU instance #1 flush list instance #0 flush list instance #1 LRU flush flush time 0 s 1 s

29 Percona Server OLTP_RW

30 Percona Server OLTP_RW 2678 nanosleep (libpthread.so.0),,buf_lru_get_free_block (buf0lru.cc:1435), pthread_cond_wait,...,log_write_up_to(log0log.cc:1293), pthread_cond_wait,, mtr_t::s_lock(sync0rw.ic:433), btr_cur_search_to_nth_level(btr0cur.cc:1022), libaio::??(libaio.so.1),linuxaiohandler::collect (os0file.cc: 2325), poll(libc.so. 6),...,Protocol_classic::read_packet(protocol_classic.cc:810),...

31 2678 nanosleep,, buf_lru_get_free_block

32 Percona Server OLTP_RW flushers only

33 Legacy doublewrite buffer: adding pages

34 Percona Server OLTP_RW flushers only 139 libaio::??(libaio.so.1),linuxaiohandler::collect (os0file.cc:2448), LinuxAIOHandler::poll(os0file.cc:2594), pthread_cond_wait,,buf_dblwr_add_to_batch (buf0dblwr.cc:1111),,buf_flush_lru_list_batch (buf0flu.cc:1555),...,buf_lru_manager(buf0flu.cc:2334), pthread_cond_wait,,os_event_wait_low (os0event.cc:534),buf_flush_page_cleaner_worker(buf0flu.cc:3482), pthread_cond_wait,, PolicyMutex<TTASEventMutex<GenericPolicy> (ut0mutex.ic:89),buf_page_io_complete (buf0buf.cc:5966), fil_aio_wait(fil0fil.cc:5754),io_handler_thread(srv0start.cc:330),... 8 pthread_cond_timedwait,,buf_flush_page_cleaner_coordinator (buf0flu.cc:2726),...

35 56 pthread_cond_wait,, buf_dblwr_add_to_batch

36 Legacy doublewrite buffer: flushing buffer

37 Parallel doublewrite buffer: adding pages

38 Parallel doublewrite buffer: flushing buffers

39 Percona Server OLTP_RW flushers only

40 Percona Server OLTP_RW flushers only 112 libaio::??(libaio.so.1),linuxaiohandler::collect (os0file.cc:2455),...,io_handler_thread(srv0start.cc:330), pthread_cond_wait,,buf_dblwr_flush_buffered_writes (buf0dblwr.cc:1287),,buf_flush_lru_list (buf0flu.cc:2341),buf_lru_manager(buf0flu.cc:2341), pthread_cond_wait,, PolicyMutex<TTASEventMutex<GenericPolicy> (ut0mutex.ic:89), buf_page_io_complete(buf0buf.cc:5986),, io_handler_thread(srv0start.cc:330), pthread_cond_wait,...,buf_flush_page_cleaner_worker(buf0flu.cc:3489), pthread_cond_wait,,enter(ib0mutex.h:845), buf_lru_block_free_non_file_page(ib0mutex.h:845), buf_lru_block_free_hashed_page(buf0lru.cc:2567),,buf_page_io_complete(buf0buf.cc:6070),,io_handler_thread (srv0start.cc:330),...

41 Percona Server 5.7 OLTP_RW

42 Percona Server 5.7 OLTP_RW

43 Summary: 5.7 story I/O-bound workloads: high demand for free pages, provided by LRU batch flushing or single-page flushing Single-page flushes are bad, w/ and w/o doublewrite Removed it Made batch LRU flusher truly parallel Doublewrite buffer negates parallel flushing gains Made it parallel too