Analyzing and Optimizing Linux Kernel for PostgreSQL. Sangwook Kim PGConf.Asia 2017

Transcription

1 Analyzing and Optimizing Linux Kernel for PostgreSQL Sangwook Kim PGConf.Asia 2017

2 Sangwook Kim Co-founder and Apposha Ph. D. in Computer Science Cloud/Virtualization SMP scheduling [ASPLOS 13, VEE 14] Group-based memory management [JPDC 14] Database/Storage Non-volatile cache management [USENIX ATC 15, ApSys 16] Request-centric I/O prioritization [FAST 17, HotStorage 17] 2

3 What We Do for Fully-managed & performance-optimized for MySQL and PostgreSQL Fully-managed & performance-optimized for MongoDB No special H/W support No code changes for DB engine 3

4 Contents Background tasks Full page writes Parallel query 4

5 PostgreSQL Architecture Request Database performance Client I/O T1 Response T2 T3 I/O I/O I/O Operating System Storage Device PostgreSQL T4 * PostgreSQL s processes - Backend (foreground) - Checkpointer - Autovacuum workers - WAL writer - Writer - 5

6 PostgreSQL Architecture Client Request Response T1 T2 T3 T4 I/O I/O I/O I/O Operating System Storage Device PostgreSQL * PostgreSQL s processes - Backend (foreground) - Checkpointer - Autovacuum workers - WAL writer - Writer - 5

7 PostgreSQL Checkpoint 6

8 Impact of Checkpointer Max trx latency (ms) Default 1.1 sec Elapsed time (sec) Dell Poweredge R cores 132GB DRAM 1 SAS SSD PostgreSQL v GB shared_buf 10GB max_wal TPC-C workload 50GB dataset 1 hour run 50 clients 7

9 PostgreSQL Autovacuum Free Space Map 8

10 Impact of Autovacuum Workers Trx thput (trx/sec) Default Aggressive AV 40% # of clients autovacuum_max_workers 3 => 6 autovacuum_naptime 1min => 15s autovacuum_vacuum_threshold 50 => 25 autovacuum_analyze_threshold 50 => 10 autovacuum_vacuum_scale_factor 0.2 => 0.1 autovacuum_analyze_scale_factor 0.1 => 0.05 autovacuum_vacuum_cost_delay 20ms => -1 autovacuum_vacuum_cost_limit -1 =>

11 Impact of Autovacuum Workers 2500 Default Aggressive AV 2.5 sec Max trx latency (ms) Elapsed time (sec) 10

12 Why Background Tasks Matter Multiple independent layers Abstraction Application Caching Layer File System Layer Block Layer Storage Device read() write() FG Buffer Cache BG BG BG reorder FG admission control FG BG reorder BG admission control admission control 11

13 Why Background Tasks Matter I/O priority inversion Application FG wait FG wait user var wake BG wait I/O Caching Layer File System Layer Block Layer FG lock BG wait I/O FG wait wait var wake BG Storage Device 12

14 Request-Centric I/O Prioritization Solution v1 (reactive) Request-aware I/O processing in the I/O path [FAST 17] I/O priority inheritance [USENIX ATC 15, FAST 17] Locks FG lock BG FG BG submit I/O FG BG complete Condition variables inherit wake register BG FG wait submit CV CV BG I/O FG CV BG complete [USENIX ATC 15] Request-Oriented Durable Write Caching for Application Performance [FAST 17] Enlightening the I/O Path: A Holistic Approach for Application Performance 13

15 Request-Centric I/O Prioritization Solution v1 (problem) linux/include/linux/mutex.h linux/include/linux/pagemap.h linux/include/linux/rtmutex.h linux/include/linux/rwsem.h linux/include/uapi/linux/sem.h linux/include/linux/wait.h linux/kernel/sched/wait.c linux/kernel/locking/rwsem.c linux/kernel/futex.c linux/kernel/locking/mutex.h linux/kernel/locking/mutex.c linux/kernel/locking/rtmutex.c linux/kernel/locking/rwsem-xadd.c Synchronization Application Interface Caching Layer File System Layer Block Layer linux/kernel/fork.c linux/kernel/sys.c linux/kernel/sysctl.c linux/include/linux/sched.h linux/include/linux/writeback.h linux/mm/page-writeback.c linux/include/linux/mm_types.h linux/fs/buffer.c linux/fs/buffer.c linux/fs/ext4/extents.c linux/fs/ext4/inode.c linux/include/linux/jbd2.h linux/fs/jbd2/commit.c linux/fs/jbd2/ journal.c linux/fs/jbd2/ transaction.c linux/include/linux/blk_types.h linux/include/linux/blkdev.h linux/include/linux/buffer_head.h linux/include/linux/caq.h linux/block/blk-core.c linux/block/blk-flush.c linux/block/blk-lib.c linux/block/blk-mq.c linux/block/caq-iosched.c linux/block/cfq-iosched.c linux/block/elevator.c 14

16 Request-Centric I/O Prioritization Solution v2 (proactive) VFS Apposha Front-End File System Page Cache - Control writes at the upper layer - Tag low-priority for background I/Os Ext4 XFS F2FS Etc Block Layer Noop CFQ Deadline Apposha I/O Scheduler Device Driver Linux I/O Stack - Priority-aware I/O scheduling - Control device-level congestion 15

17 Request-Centric I/O Prioritization V12 Engine User-level library - Classify task priority, optimize WAL accesses MongoDB Library PostgreSQL Library Front-End File System I/O Scheduler Kernel Modules V12-M V12-P 16

18 Request-Centric I/O Prioritization 2500 Default Aggressive AV V12-P Max trx latency (ms) sec Elapsed time (sec) V12-P: V12 Engine for PostgreSQL 17

19 Request-Centric I/O Prioritization 2500 Default Aggressive AV V12-P v9.6 New feature since v9.6: checkpoint_flush_after Max trx latency (ms) sec Elapsed time (sec) V12-P: V12 Engine for PostgreSQL 17

20 Contents Background tasks Full page writes Parallel query 18

21 Full Page Writes Tuning PostgreSQL for High Write Workloads Grant McAlister 19

22 Full Page Writes Impact on WAL size WAL size (GB) Default PostgreSQL v GB shared_buffer 1GB max_wal TPC-C workload 50GB dataset 10 min run 200 clients Elapsed time (sec) 20

23 WAL Compression Impact on WAL size WAL size (GB) Default WAL compression PostgreSQL v GB shared_buffer 1GB max_wal TPC-C workload 50GB dataset 10 min run 200 clients Elapsed time (sec) 21

24 WAL Compression Impact on performance Trx thput (trx/sec) Default WAL compression 22

25 WAL Compression Impact on performance Trx thput (trx/sec) Default WAL compression What if we can safely disable full_page_writes w/o special H/W support? 22

26 Atomic Write Support In memory On disk PostgreSQL (full_page_writes off) write() Mapping Table Linux Kernel (w/ V12 Engine) Data Block Data Block 23

27 Atomic Write Support In memory On disk PostgreSQL (full_page_writes off) fsync() Mapping Table Linux Kernel (w/ V12 Engine) writeback Data Block Data Block Data Block 23

28 Atomic Write Support In memory On disk PostgreSQL (full_page_writes off) fsync() Mapping Table Linux Kernel (w/ V12 Engine) atomic update Data Block Data Block Data Block 23

29 Atomic Write Support In memory On disk PostgreSQL (full_page_writes off) fsync() Mapping Table Linux Kernel (w/ V12 Engine) Data Block Data Block Data Block 23

30 Atomic Write Support PostgreSQL (full_page_writes off) In memory free On disk Mapping Table Linux Kernel (w/ V12 Engine) Data Block Data Block Data Block 23

31 Atomic Write Support PostgreSQL (full_page_writes off) In memory On disk Mapping Table Linux Kernel (w/ V12 Engine) Data Block Data Block 23

32 Atomic Write Support Impact on WAL size 14 Default WAL compression V12-P 12 WAL size (GB) Elapsed time (sec) V12-P: V12 Engine for PostgreSQL 24

33 Atomic Write Support Impact on performance Trx thput (trx/sec) Default WAL compression V12-P 2X V12-P: V12 Engine for PostgreSQL 25

34 Contents Background tasks Full page writes Parallel query 26

35 Parallel Query User query Scan, aggregate, join, Backend Worker Worker Worker Worker 27

36 Parallel Query User query Slow response Fast response Backend Worker Worker Worker Worker cache storage hit cache miss! storage cache miss! storage cache storage hit 27

37 Parallel Query Problem inside Linux kernel Query1 Query2 10sec 18sec 10sec I/O Scheduling 20sec Fair-based (communism) 28

38 Parallel Query Problem inside Linux kernel Query1 Query2 Query3 10sec 10sec 24sec 10sec I/O Scheduling 26sec 30sec Fair-based (communism) 28

39 Parallel Query Problem inside Linux kernel Query1 Query2 Query3 QueryN 10sec 10sec? 10sec 10sec I/O Scheduling?? Fair-based (communism) 28

40 Parallel Query Problem inside Linux kernel Query1 Query2 10sec 18sec 10sec 10sec I/O Scheduling 20sec 20sec Fair-based (communism) Task-aware (utilitarianism) 29

41 Optimizing Parallel Query Illustrative experiment pgbench i s 1000 test1 pgbench i s 1000 test2 Q1: SELECT * FROM pgbench_accounts WHERE filler LIKE %x% -d test1 Q2: SELECT * FROM pgbench_accounts WHERE filler LIKE %x% -d test2 QUERY PLAN Gather (cost= rows=1 width=97) Workers Planned: 7 -> Parallel Seq Scan on pgbench_accounts (cost= rows=1 width=97) Filter: (filler ~~ '%x%'::text) 30

42 Optimizing Parallel Query Illustrative experiment Query latency (ms) Q1 Q2 Q3 Q4 Deadline CFQ NOOP Taskaware 31

43 Recall Multiple independent layers Application read() write() Abstraction Caching Layer File System Layer Block Layer Storage Device Buffer Cache FG BG FG reorder BG BG FG BG 32

44 Optimizing Parallel Query Illustrative experiment (device queue off) Query latency (ms) Q1 Q2 Q3 Q4 Q1 73% Avg 23% 0 Deadline CFQ NOOP Taskaware 33

45 Optimizing Parallel Query Illustrative experiment (4ms idle_slice) Query latency (ms) Q1 Q2 Q3 Q4 Avg 17% Q1 2.8X Deadline CFQ NOOP Taskaware Taskaware (Idle 4ms) 34

46 Optimizing Parallel Query Ongoing work Handling heavy tasks Extending to CPU scheduling 35

47 H F M sangwook@apposha.io 47