Troubleshooting PostgreSQL with pgcenter. Alexey Lesovsky
|
|
|
- Philip McDaniel
- 6 years ago
- Views:
Transcription
1 Troubleshooting PostgreSQL with pgcenter Alexey Lesovsky
2 01 02 Introduction What is pgcenter. Goals and quick overview. Troubleshooting cases Health checks and load spikes. Hidden problems. Client's activity. Streaming replication.
3 01 Introduction
4 01 Goals PgCenter overview. How to use pgcenter. How to troubleshoot PostgreSQL with pgcenter.
5 01 What is pgcenter Top-like admin tool for Postgres stats. Quick Postgres administration. Available on RHEL/CentOS/Fedora/ALT Linux/Ubuntu... Hosted on GitHub. Work in progress.
6 01 PostgreSQL Performance Views
7 01 Features PostgreSQL 9.x support. Support of the most stats views. Support of the system stats. PostgreSQL admin tasks: configs, logs, reload, cancel/kill backends.
8 01 Interface overview pgcenter: :40:28, load average: 1.59, 0.74, 0.42 conn1 [ok]: /run/postgresql:5433 (ver: 10devel, up 2 days 20:29:27) %cpu: 14.3 us, 3.9 sy, 0.0 ni, 45.8 id, 36.0 wa, 0.0 hi, 0.0 si, 0.0 st activity: 8/100 conns, 0/0 prepared, 1 idle, 0 idle_xact, 2 active, 6 waiting, 0 others MiB mem: 7836 total, 1582 free, 3910 used, 2344 buff/cached autovacuum: 0/3 workers/max, 0 manual, 0 wraparound, 00:00:00 vac_maxtime MiB swap: 5119 total, 5088 free, 31 used, 26/0 dirty/writeback statements: 7050 stmt/s, stmt_avgtime, 00:00:00 xact_maxtime, 00:00:00 prep_maxtime user database t_all_t t_read_t t_write_t t_cpu_t all_t read_t write_t cpu_t calls queryid query postgres pgbench 00:05:31 00:00:00 00:00:00 00:05: ce UPDATE pgbench_accounts SET abalance = abalance + $N WHERE aid = $N postgres pgbench 00:00:09 00:00:00 00:00:00 00:00: c5345de88 UPDATE pgbench_branches SET bbalance = bbalance + $N WHERE bid = $N postgres pgbench 00:00:07 00:00:00 00:00:00 00:00: cdaf0baaf UPDATE pgbench_tellers SET tbalance = tbalance + $N WHERE tid = $N postgres pgbench 00:00:04 00:00:00 00:00:00 00:00: f72672d89 SELECT abalance FROM pgbench_accounts WHERE aid = $N postgres pgbench 00:00:04 00:00:00 00:00:00 00:00: fe4e1b copy pgbench_accounts from stdin postgres pgbench 00:00:03 00:00:00 00:00:00 00:00: f1faff SELECT datname, xact_commit AS commit, xact_rollback AS rollback, b postgres pgbench 00:00:02 00:00:00 00:00:00 00:00: f1f154154f INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES ($ postgres postgres 00:00:02 00:00:00 00:00:00 00:00: f21a96353 SELECT a.rolname AS user, d.datname AS database, date_trunc($n, rou postgres postgres 00:00:01 00:00:00 00:00:00 00:00: b0cfaf67a8 create database pgbench postgres postgres 00:00:01 00:00:00 00:00:00 00:00: d14dc3 SELECT datname, xact_commit AS commit, xact_rollback AS rollback, b Device: rrqm/s wrqm/s r/s w/s rmb/s wmb/s avgrq-sz avgqu-sz await r_await w_await %util sda: sda1: sda2: sdb: sdb1: sdb2: dm-0: dm-1: dm-2:
9 01 Interface overview pgcenter: :40:28, load average: 1.59, 0.74, 0.42 conn1 [ok]: /run/postgresql:5433 (ver: 10devel, up 2 days 20:29:27) %cpu: 14.3 us, 3.9 sy, 0.0 ni, 45.8 id, 36.0 wa, 0.0 hi, 0.0 si, 0.0 st activity: 8/100 conns, 0/0 prepared, 1 idle, 0 idle_xact, 2 active, 6 waiting, 0 others MiB mem: 7836 total, 1582 free, 3910 used, 2344 buff/cached autovacuum: 0/3 workers/max, 0 manual, 0 wraparound, 00:00:00 vac_maxtime MiB swap: 5119 total, 5088 free, 31 used, 26/0 dirty/writeback statements: 7050 stmt/s, stmt_avgtime, 00:00:00 xact_maxtime, 00:00:00 prep_maxtime user database t_all_t t_read_t t_write_t t_cpu_t all_t read_t write_t cpu_t calls queryid query postgres pgbench 00:05:31 00:00:00 00:00:00 00:05: ce UPDATE pgbench_accounts SET abalance = abalance + $N WHERE aid = $N postgres pgbench 00:00:09 00:00:00 00:00:00 00:00: c5345de88 UPDATE pgbench_branches SET bbalance = bbalance + $N WHERE bid = $N postgres pgbench 00:00:07 00:00:00 00:00:00 00:00: cdaf0baaf UPDATE pgbench_tellers SET tbalance = tbalance + $N WHERE tid = $N postgres pgbench 00:00:04 00:00:00 00:00:00 00:00: f72672d89 SELECT abalance FROM pgbench_accounts WHERE aid = $N postgres pgbench 00:00:04 00:00:00 00:00:00 00:00: fe4e1b copy pgbench_accounts from stdin postgres pgbench 00:00:03 00:00:00 00:00:00 00:00: f1faff SELECT datname, xact_commit AS commit, xact_rollback AS rollback, b postgres pgbench 00:00:02 00:00:00 00:00:00 00:00: f1f154154f INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES ($ postgres postgres 00:00:02 00:00:00 00:00:00 00:00: f21a96353 SELECT a.rolname AS user, d.datname AS database, date_trunc($n, rou postgres postgres 00:00:01 00:00:00 00:00:00 00:00: b0cfaf67a8 create database pgbench postgres postgres 00:00:01 00:00:00 00:00:00 00:00: d14dc3 SELECT datname, xact_commit AS commit, xact_rollback AS rollback, b Device: rrqm/s wrqm/s r/s w/s rmb/s wmb/s avgrq-sz avgqu-sz await r_await w_await %util sda: sda1: sda2: sdb: sdb1: sdb2: dm-0: dm-1: dm-2:
10 01 Interface overview pgcenter: :40:28, load average: 1.59, 0.74, 0.42 conn1 [ok]: /run/postgresql:5433 (ver: 10devel, up 2 days 20:29:27) %cpu: 14.3 us, 3.9 sy, 0.0 ni, 45.8 id, 36.0 wa, 0.0 hi, 0.0 si, 0.0 st activity: 8/100 conns, 0/0 prepared, 1 idle, 0 idle_xact, 2 active, 6 waiting, 0 others MiB mem: 7836 total, 1582 free, 3910 used, 2344 buff/cached autovacuum: 0/3 workers/max, 0 manual, 0 wraparound, 00:00:00 vac_maxtime MiB swap: 5119 total, 5088 free, 31 used, 26/0 dirty/writeback statements: 7050 stmt/s, stmt_avgtime, 00:00:00 xact_maxtime, 00:00:00 prep_maxtime user database t_all_t t_read_t t_write_t t_cpu_t all_t read_t write_t cpu_t calls queryid query postgres pgbench 00:05:31 00:00:00 00:00:00 00:05: ce UPDATE pgbench_accounts SET abalance = abalance + $N WHERE aid = $N postgres pgbench 00:00:09 00:00:00 00:00:00 00:00: c5345de88 UPDATE pgbench_branches SET bbalance = bbalance + $N WHERE bid = $N postgres pgbench 00:00:07 00:00:00 00:00:00 00:00: cdaf0baaf UPDATE pgbench_tellers SET tbalance = tbalance + $N WHERE tid = $N postgres pgbench 00:00:04 00:00:00 00:00:00 00:00: f72672d89 SELECT abalance FROM pgbench_accounts WHERE aid = $N postgres pgbench 00:00:04 00:00:00 00:00:00 00:00: fe4e1b copy pgbench_accounts from stdin postgres pgbench 00:00:03 00:00:00 00:00:00 00:00: f1faff SELECT datname, xact_commit AS commit, xact_rollback AS rollback, b postgres pgbench 00:00:02 00:00:00 00:00:00 00:00: f1f154154f INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES ($ postgres postgres 00:00:02 00:00:00 00:00:00 00:00: f21a96353 SELECT a.rolname AS user, d.datname AS database, date_trunc($n, rou postgres postgres 00:00:01 00:00:00 00:00:00 00:00: b0cfaf67a8 create database pgbench postgres postgres 00:00:01 00:00:00 00:00:00 00:00: d14dc3 SELECT datname, xact_commit AS commit, xact_rollback AS rollback, b Device: rrqm/s wrqm/s r/s w/s rmb/s wmb/s avgrq-sz avgqu-sz await r_await w_await %util sda: sda1: sda2: sdb: sdb1: sdb2: dm-0: dm-1: dm-2:
11 01 Interface overview pgcenter: :40:28, load average: 1.59, 0.74, 0.42 conn1 [ok]: /run/postgresql:5433 (ver: 10devel, up 2 days 20:29:27) %cpu: 14.3 us, 3.9 sy, 0.0 ni, 45.8 id, 36.0 wa, 0.0 hi, 0.0 si, 0.0 st activity: 8/100 conns, 0/0 prepared, 1 idle, 0 idle_xact, 2 active, 6 waiting, 0 others MiB mem: 7836 total, 1582 free, 3910 used, 2344 buff/cached autovacuum: 0/3 workers/max, 0 manual, 0 wraparound, 00:00:00 vac_maxtime MiB swap: 5119 total, 5088 free, 31 used, 26/0 dirty/writeback statements: 7050 stmt/s, stmt_avgtime, 00:00:00 xact_maxtime, 00:00:00 prep_maxtime user database t_all_t t_read_t t_write_t t_cpu_t all_t read_t write_t cpu_t calls queryid query postgres pgbench 00:05:31 00:00:00 00:00:00 00:05: ce UPDATE pgbench_accounts SET abalance = abalance + $N WHERE aid = $N postgres pgbench 00:00:09 00:00:00 00:00:00 00:00: c5345de88 UPDATE pgbench_branches SET bbalance = bbalance + $N WHERE bid = $N postgres pgbench 00:00:07 00:00:00 00:00:00 00:00: cdaf0baaf UPDATE pgbench_tellers SET tbalance = tbalance + $N WHERE tid = $N postgres pgbench 00:00:04 00:00:00 00:00:00 00:00: f72672d89 SELECT abalance FROM pgbench_accounts WHERE aid = $N postgres pgbench 00:00:04 00:00:00 00:00:00 00:00: fe4e1b copy pgbench_accounts from stdin postgres pgbench 00:00:03 00:00:00 00:00:00 00:00: f1faff SELECT datname, xact_commit AS commit, xact_rollback AS rollback, b postgres pgbench 00:00:02 00:00:00 00:00:00 00:00: f1f154154f INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES ($ postgres postgres 00:00:02 00:00:00 00:00:00 00:00: f21a96353 SELECT a.rolname AS user, d.datname AS database, date_trunc($n, rou postgres postgres 00:00:01 00:00:00 00:00:00 00:00: b0cfaf67a8 create database pgbench postgres postgres 00:00:01 00:00:00 00:00:00 00:00: d14dc3 SELECT datname, xact_commit AS commit, xact_rollback AS rollback, b Device: rrqm/s wrqm/s r/s w/s rmb/s wmb/s avgrq-sz avgqu-sz await r_await w_await %util sda: sda1: sda2: sdb: sdb1: sdb2: dm-0: dm-1: dm-2:
12 01 Interface overview pgcenter: :40:28, load average: 1.59, 0.74, 0.42 conn1 [ok]: /run/postgresql:5433 (ver: 10devel, up 2 days 20:29:27) %cpu: 14.3 us, 3.9 sy, 0.0 ni, 45.8 id, 36.0 wa, 0.0 hi, 0.0 si, 0.0 st activity: 8/100 conns, 0/0 prepared, 1 idle, 0 idle_xact, 2 active, 6 waiting, 0 others MiB mem: 7836 total, 1582 free, 3910 used, 2344 buff/cached autovacuum: 0/3 workers/max, 0 manual, 0 wraparound, 00:00:00 vac_maxtime MiB swap: 5119 total, 5088 free, 31 used, 26/0 dirty/writeback statements: 7050 stmt/s, stmt_avgtime, 00:00:00 xact_maxtime, 00:00:00 prep_maxtime user database t_all_t t_read_t t_write_t t_cpu_t all_t read_t write_t cpu_t calls queryid query postgres pgbench 00:05:31 00:00:00 00:00:00 00:05: ce UPDATE pgbench_accounts SET abalance = abalance + $N WHERE aid = $N postgres pgbench 00:00:09 00:00:00 00:00:00 00:00: c5345de88 UPDATE pgbench_branches SET bbalance = bbalance + $N WHERE bid = $N postgres pgbench 00:00:07 00:00:00 00:00:00 00:00: cdaf0baaf UPDATE pgbench_tellers SET tbalance = tbalance + $N WHERE tid = $N postgres pgbench 00:00:04 00:00:00 00:00:00 00:00: f72672d89 SELECT abalance FROM pgbench_accounts WHERE aid = $N postgres pgbench 00:00:04 00:00:00 00:00:00 00:00: fe4e1b copy pgbench_accounts from stdin postgres pgbench 00:00:03 00:00:00 00:00:00 00:00: f1faff SELECT datname, xact_commit AS commit, xact_rollback AS rollback, b postgres pgbench 00:00:02 00:00:00 00:00:00 00:00: f1f154154f INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES ($ postgres postgres 00:00:02 00:00:00 00:00:00 00:00: f21a96353 SELECT a.rolname AS user, d.datname AS database, date_trunc($n, rou postgres postgres 00:00:01 00:00:00 00:00:00 00:00: b0cfaf67a8 create database pgbench postgres postgres 00:00:01 00:00:00 00:00:00 00:00: d14dc3 SELECT datname, xact_commit AS commit, xact_rollback AS rollback, b Device: rrqm/s wrqm/s r/s w/s rmb/s wmb/s avgrq-sz avgqu-sz await r_await w_await %util sda: sda1: sda2: sdb: sdb1: sdb2: dm-0: dm-1: dm-2:
13 01 Supported stats pg_stat_database. pg_stat_replication. pg_stat_user_tables, pg_statio_user_tables. pg_stat_user_indexes, pg_statio_user_indexes. pg_stat_user_functions. pg_stat_activity, pg_stat_progress_vacuum. pg_stat_statement.
14 01 Additional stats Iostat storage devices stats. Nicstat network interfaces stats. Log tail latest lines from postgres logs.
15 Troubleshooting cases
16 Troubleshooting cases Health checks and load spikes. Hidden problems. Clients' activity. Streaming replication.
17 Health checks or load spikes Check whether is everything OK? or When something really goes wrong.
18 Health checks and load spikes Run pgcenter. Check system resources (cpu, memory/swap, storage, network). Check postgres activity (conn's states, xact age). Check iostat/nicstat if required.
19 Health checks and load spikes pgcenter: :50:36, load average: 4.14, 0.98, 0.46 conn1 [ok]: /var/run/postgresql:5432 (ver: 9.6.2, up 1 day 04:02:57) %cpu: 2.9 us, 0.7 sy, 0.0 ni, 11.4 id, 85.0 wa, 0.0 hi, 0.0 si, 0.0 st activity: 13/100 conns, 0/0 prepared, 0 idle, 0 idle_xact, 13 active, 0 waiting, 0 others MiB mem: total, 6228 free, used, 432 buff/cached autovacuum: 0/3 workers/max, 0 manual, 0 wraparound, 00:00:00 vac_maxtime MiB swap: total, free, 0 used, 1/0 dirty/writeback statements: 3519 stmt/s, stmt_avgtime, 00:00:00 xact_maxtime, 00:00:00 prep_maxtime datname commit rollback reads hits returned fetched inserts updates deletes conflicts deadlocks tmp_files tmp_bytes read_t write_t pgbench_64gb pgbench_6gb postgres template Device: rrqm/s wrqm/s r/s w/s rmb/s wmb/s avgrq-sz avgqu-sz await r_await w_await %util sda: sda1: sda2: sdb: sdb1:
20 Health checks and load spikes pgcenter: :50:36, load average: 4.14, 0.98, 0.46 conn1 %cpu: 2.9 us, 0.7 sy, 0.0 ni, 11.4 id, 85.0 wa, 0.0 hi, 0.0 si, 0.0 st act MiB mem: total, 6228 free, used, 432 buff/cached autov MiB swap: total, free, 0 used, 1/0 dirty/writeback state datname commit rollback reads hits returned fetched inserts updates delete pgbench_64gb pgbench_6gb postgres template
21 Health checks and load spikes conn1 [ok]: /var/run/postgresql:5432 (ver: 9.6.2, up 1 day 04:02:57) activity: 13/100 conns, 0/0 prepared, 0 idle, 0 idle_xact, 13 active, 0 waiting, 0 others autovacuum: 0/3 workers/max, 0 manual, 0 wraparound, 00:00:00 vac_maxtime statements: 3519 stmt/s, stmt_avgtime, 00:00:00 xact_maxtime, 00:00:00 prep_maxtime s deletes conflicts deadlocks tmp_files tmp_bytes read_t write_t
22 Health checks and load spikes Device: rrqm/s wrqm/s r/s w/s rmb/s wmb/s avgrq-sz avgqu-sz await r_await w_await %util sda: sda1: sda2: sdb: sdb1:
23 Go into details Which queries cause the load?
24 pg_stat_statements pg_stat_statements is a contrib module. GUCs: shared_preload_libraries = 'pg_stat_statements' track_io_timing = on Available stats in pgcenter: timings - how long queries have run; general - number of calls and rows; IO - buffers, temp, local IO.
25 Health checks and load spikes pgcenter: :50:36, load average: 4.14, 0.98, 0.46 conn1 [ok]: /var/run/postgresql:5432 (ver: 9.6.2, up 1 day 04:02:57) %cpu: 2.9 us, 0.7 sy, 0.0 ni, 11.4 id, 85.0 wa, 0.0 hi, 0.0 si, 0.0 st activity: 13/100 conns, 0/0 prepared, 0 idle, 0 idle_xact, 13 active, 0 waiting, 0 others MiB mem: total, 6228 free, used, 432 buff/cached autovacuum: 0/3 workers/max, 0 manual, 0 wraparound, 00:00:00 vac_maxtime MiB swap: total, free, 0 used, 1/0 dirty/writeback statements: 3519 stmt/s, stmt_avgtime, 00:00:00 xact_maxtime, 00:00:00 prep_maxtime user database t_all_t t_read_t t_write_t t_cpu_t all_t read_t write_t cpu_t calls queryid query postgres pgbench_64gb 47:38:04 47:33:07 00:00:00 00:04: ffafcb30 UPDATE pgbench_accounts SET abalance = abalance +? WHERE aid = postgres pgbench_64gb 23:51:51 23:48:08 00:00:00 00:03: df3ad65 SELECT abalance FROM pgbench_accounts WHERE aid =?; postgres pgbench_6gb 02:14:23 00:02:45 00:00:00 02:11: e8e1cdb2da SELECT abalance FROM pgbench_accounts WHERE aid = ; postgres pgbench_64gb 00:01:31 00:00:18 00:00:00 00:01: b6b13d62 UPDATE pgbench_tellers SET tbalance = tbalance +? WHERE tid =? postgres pgbench_6gb 01:07:36 00:00:16 00:00:00 01:07: e099 UPDATE pgbench_accounts SET abalance = abalance +? WHERE aid = postgres pgbench_64gb 00:01:22 00:00:08 00:00:00 00:01: e192b43faf UPDATE pgbench_branches SET bbalance = bbalance +? WHERE bid = okagent pgbench_6gb 00:00:00 00:00:00 00:00:00 00:00: f67e66660a select * from pg_stat_user_tables s limit? postgres pgbench_6gb 00:00:00 00:00:00 00:00:00 00:00: f6241e76ed vacuum pgbench_branches postgres pgbench_64gb 00:00:00 00:00:00 00:00:00 00:00: eaf8d3d459 truncate pgbench_history postgres pgbench_64gb 00:00:42 00:00:00 00:00:00 00:00: e7cc INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES postgres pgbench_6gb 00:30:17 00:00:00 00:00:00 00:30: e7766dd7c6 UPDATE pgbench_branches SET bbalance = bbalance +? WHERE bid = okagent pgbench_64gb 00:00:02 00:00:00 00:00:00 00:00: de77700d35 SELECT schemaname, relname, indexrelname, coalesce(pg_relation_ okagent pgbench_64gb 00:00:02 00:00:00 00:00:00 00:00: d1bcc9180b SELECT schemaname, relname, coalesce(heap_blks_read,?) as heap_ postgres pgbench_64gb 00:00:00 00:00:00 00:00:00 00:00: d101ae1052 vacuum pgbench_branches
26 Health checks and load spikes oad average: 4.14, 0.98, 0.46 conn1 [ok]: /var/run/postgresql:5432 (ver: 9.6.2, up 1 day 04:02:57) ni, 11.4 id, 85.0 wa, 0.0 hi, 0.0 si, 0.0 st activity: 13/100 conns, 0/0 prepared, 0 idle, 0 idle_xact, 13 active, 0 waiting, 0 others ree, used, 432 buff/cached autovacuum: 0/3 workers/max, 0 manual, 0 wraparound, 00:00:00 vac_maxtime ree, 0 used, 1/0 dirty/writeback statements: 3519 stmt/s, stmt_avgtime, 00:00:00 xact_maxtime, 00:00:00 prep_maxtime t_read_t t_write_t t_cpu_t all_t read_t write_t cpu_t calls queryid query 47:33:07 00:00:00 00:04: ffafcb30 UPDATE pgbench_accounts SET abalance = abalance +? WHERE aid = 23:48:08 00:00:00 00:03: df3ad65 SELECT abalance FROM pgbench_accounts WHERE aid =?; 00:02:45 00:00:00 02:11: e8e1cdb2da SELECT abalance FROM pgbench_accounts WHERE aid = ; 00:00:18 00:00:00 00:01: b6b13d62 UPDATE pgbench_tellers SET tbalance = tbalance +? WHERE tid =? 00:00:16 00:00:00 01:07: e099 UPDATE pgbench_accounts SET abalance = abalance +? WHERE aid = 00:00:08 00:00:00 00:01: e192b43faf UPDATE pgbench_branches SET bbalance = bbalance +? WHERE bid = 00:00:00 00:00:00 00:00: f67e66660a select * from pg_stat_user_tables s limit? 00:00:00 00:00:00 00:00: f6241e76ed vacuum pgbench_branches 00:00:00 00:00:00 00:00: eaf8d3d459 truncate pgbench_history 00:00:00 00:00:00 00:00: e7cc INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES 00:00:00 00:00:00 00:30: e7766dd7c6 UPDATE pgbench_branches SET bbalance = bbalance +? WHERE bid = 00:00:00 00:00:00 00:00: de77700d35 SELECT schemaname, relname, indexrelname, coalesce(pg_relation_ 00:00:00 00:00:00 00:00: d1bcc9180b SELECT schemaname, relname, coalesce(heap_blks_read,?) as heap_ 00:00:00 00:00:00 00:00: d101ae1052 vacuum pgbench_branches
27 Health checks and load spikes Another example.
28 Health checks and load spikes pgcenter: :52:54, load average: 10.65, 4.65, 1.89 conn1 [ok]: /var/run/postgresql:5432 (ver: 9.6.2, up 1 day 04:05:15) %cpu: 77.4 us, 9.6 sy, 0.0 ni, 12.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st activity: 73/100 conns, 0/0 prepared, 12 idle, 0 idle_xact, 61 active, 0 waiting, 0 others MiB mem: total, 215 free, used, 6440 buff/cached autovacuum: 0/3 workers/max, 0 manual, 0 wraparound, 00:00:00 vac_maxtime MiB swap: total, free, 0 used, 0/0 dirty/writeback statements: stmt/s, stmt_avgtime, 00:00:00 xact_maxtime, 00:00:00 prep_maxtime user database t_all_t t_read_t t_write_t t_cpu_t all_t read_t write_t cpu_t calls queryid query postgres pgbench_6gb 02:14:30 00:02:45 00:00:00 02:11: e8e1cdb2da SELECT abalance FROM pgbench_accounts WHERE aid = ; postgres pgbench_6gb 01:07:36 00:00:16 00:00:00 01:07: e099 UPDATE pgbench_accounts SET abalance = abalance +? WHERE aid = postgres pgbench_6gb 00:30:17 00:00:00 00:00:00 00:30: e7766dd7c6 UPDATE pgbench_branches SET bbalance = bbalance +? WHERE bid = postgres pgbench_6gb 00:29:07 00:00:00 00:00:00 00:29: cff023 UPDATE pgbench_tellers SET tbalance = tbalance +? WHERE tid =? postgres pgbench_6gb 00:16:05 00:00:00 00:00:00 00:16: a4d34a918 INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES postgres pgbench_64gb 47:40:16 47:35:18 00:00:00 00:04: ffafcb30 UPDATE pgbench_accounts SET abalance = abalance +? WHERE aid = postgres pgbench_64gb 23:51:51 23:48:08 00:00:00 00:03: df3ad65 SELECT abalance FROM pgbench_accounts WHERE aid =?; postgres pgbench_64gb 00:01:22 00:00:08 00:00:00 00:01: e192b43faf UPDATE pgbench_branches SET bbalance = bbalance +? WHERE bid = postgres pgbench_64gb 00:01:31 00:00:18 00:00:00 00:01: b6b13d62 UPDATE pgbench_tellers SET tbalance = tbalance +? WHERE tid =? postgres pgbench_64gb 00:00:42 00:00:00 00:00:00 00:00: e7cc INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES postgres pgbench_6gb 00:00:41 00:00:00 00:00:00 00:00: cbfcd7045 END; postgres pgbench_6gb 00:00:35 00:00:00 00:00:00 00:00: dbbb915 BEGIN; Device: rrqm/s wrqm/s r/s w/s rmb/s wmb/s avgrq-sz avgqu-sz await r_await w_await %util sda: sda1: sda2: sdb: sdb1:
29 pg_stat_statements reports Reports based on pg_stat_statements. Use queryid and "G" hotkey.
30 IO consuming query summary: total_time: 75:59:21, cpu_time: 04:50:58, io_time: 71:08:23 (ALL: %, CPU: 6.38%, IO: 93.62%), total queries: 1,090,573,273 query info: usename: postgres, datname: pgbench_64gb, calls (relative to all queries): 4,826,787 (0.44%), rows (relative to all queries): 4,826,787 (0.53%), total time (relative to all queries): 47:21:38 (ALL: 62.3%, CPU: 1.7%, IO: 66.5%), average time (only for this query): 35.32ms, cpu_time: 0.06ms, io_time: 35.26ms, (ALL: 100.0%, CPU: 0.2%, IO: 99.8%), query text (id: 94ffafcb30): UPDATE pgbench_accounts SET abalance = abalance +? WHERE aid =?;
31 CPU consuming query summary: total_time: 75:56:34, cpu_time: 04:50:04, io_time: 71:06:29 (ALL: %, CPU: 6.37%, IO: 93.63%), total queries: 1,087,323,630 query info: usename: postgres, datname: pgbench_6gb, calls (relative to all queries): 531,640,449 (48.89%), rows (relative to all queries): 531,640,449 (58.59%), total time (relative to all queries): 02:12:35 (ALL: 2.9%, CPU: 44.8%, IO: 0.1%), average time (only for this query): 0.01ms, cpu_time: 0.01ms, io_time: 0.00ms, (ALL: 100.0%, CPU: 100.0%, IO: 0.0%), query text (id: e8e1cdb2da): SELECT abalance FROM pgbench_accounts WHERE aid = ;
32 pg_stat_statements summary using pg_stat_statements you able to find: frequently called queries; queries that return highest number of rows; queries caused buffers IO; cpu hog queries; disk hog queries (read/write); queries with IO on temp files; queries with IO on temp tables; use sorting and filtering; reports.
33 Troubleshooting cases Health checks and load spikes. Hidden problems. Clients' activity. Streaming replication.
34 Hidden problems Sequential scans. Database s unwanted events. Unused indexes. Tables' sizes.
35 Sequential scans Sequential scans: might increase time of query execution; extra IO impact on large tables. But OK for OLAP workload.
36 Sequential scans pgcenter: :10:14, load average: 3.09, 1.03, 0.37 conn1 [ok]: /var/run/postgresql:5432 %cpu: 91.8 us, 0.2 sy, 0.0 ni, 8.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st activity: 12/100 conns, 0/0 prepa MiB mem: total, 291 free, used, 6362 buff/cached autovacuum: 0/3 workers/max, 0 man MiB swap: total, free, 0 used, 0/0 dirty/writeback statements: 0 stmt/s, stmt_a relation seq_scan seq_read idx_scan idx_fetch inserts updates deletes hot_updates live dead public.ycsb_metrics public.pgbench_tellers public.pgbench_history public.pgbench_branches public.pgbench_accounts
37 Sequential scans pgcenter: :10:50, load average: 6.73, 2.17, 0.78 conn1 [ok]: /var/run/postgresql:5432 %cpu: 91.6 us, 0.2 sy, 0.0 ni, 8.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st activity: 12/100 conns, 0/0 prepa MiB mem: total, 290 free, used, 6363 buff/cached autovacuum: 0/3 workers/max, 0 man MiB swap: total, free, 0 used, 0/0 dirty/writeback statements: 0 stmt/s, stmt_a relation heap_read heap_hit idx_read idx_hit toast_read toast_hit tidx_read tidx_hit public.ycsb_metrics public.pgbench_tellers public.pgbench_history public.pgbench_branches public.pgbench_accounts
38 Sequential scans pgcenter: :13:02, load average: 10.66, 5.43, 2.18 conn1 [ok]: /var/run/postgresql:5432 (ver: 9.6.2, up 1 day %cpu: 91.9 us, 0.3 sy, 0.0 ni, 7.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st activity: 12/100 conns, 0/0 prepared, 0 idle, 0 idle_xact, 12 active, MiB mem: total, 286 free, used, 6363 buff/cached autovacuum: 0/3 workers/max, 0 manual, 0 wraparound, 00:00:00 MiB swap: total, free, 0 used, 0/0 dirty/writeback statements: 160 stmt/s, stmt_avgtime, 00:00:00 xact_maxtime, 00:00:00 user database t_calls t_rows calls rows queryid query postgres pgbench_6gb select id,inserted_at,severity,payload from ycsb_metrics where ycsb_sensor =?; postgres pgbench_6gb e55c44ee4a select id,ycsb_sensor,extra1 from ycsb_metrics where severity =? order by inserted_at postgres pgbench_6gb d47af42e select id,inserted_at,severity,payload,extra1,extra2,extra3 from ycsb_metrics where postgres pgbench_6gb cadff41ef SELECT a.rolname AS user, d.datname AS database, sum(p.calls) AS t_calls, sum(p.rows) postgres pgbench_6gb e0adb select ycsb_sensor,inserted_at,severity,payload from ycsb_metrics where id =?;
39 Unwanted events Rollbacks. Deadlocks. Temporary files. Recovery conflicts.
40 Unwanted events pgcenter: :40:07, load average: 10.56, 4.80, 2.28 conn1 [ok]: /var/run/postgresql:5432 (ver: 9.6.2, up 1 day %cpu: 91.9 us, 2.6 sy, 0.0 ni, 5.5 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st activity: 12/100 conns, 0/0 prepared, 7 idle, 0 idle_xact, 5 active, MiB mem: total, 267 free, used, 6382 buff/cached autovacuum: 0/3 workers/max, 0 manual, 0 wraparound, 00:00:00 MiB swap: total, free, 0 used, 0/0 dirty/writeback statements: 3535 stmt/s, stmt_avgtime, 00:00:00 xact_maxtime, datname commit rollback reads hits returned fetched inserts updates deletes conflicts deadlocks tmp_files tmp_bytes read_t write_t pgbench_64gb pgbench_6gb postgres template template
41 Unwanted events summary Rollbacks - check postgres logs and application. Deadlocks - enable log_lock_waits, check logs. Temp files - enable log_temp_files, check logs, pg_stat_statements. Recovery conflicts - check postgres logs, standby s queries.
42 Unused indexes Unused indexes are bad: consume disk space; extra writes operations; slow down INSERT/UPDATE/DELETEs; extra work for (auto)vacuum.
43 Unused indexes pgcenter: :45:20, load average: 0.09, 2.08, 1.87 conn1 [ok]: /var/run/postgresql:5432 postgre %cpu: 0.2 us, 0.0 sy, 0.0 ni, 99.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st activity: 1/100 conns, 0/0 prepared, 0 MiB mem: total, 282 free, used, 6380 buff/cached autovacuum: 0/3 workers/max, 0 manual, 0 MiB swap: total, free, 0 used, 0/0 dirty/writeback statements: 6 stmt/s, stmt_avgtime, relation index idx_scan idx_tup_read idx_tup_fetch idx_read idx_hit public.ycsb_metrics ycsb_metrics_ycsb_sensor_idx public.ycsb_metrics ycsb_metrics_severity_idx public.ycsb_metrics ycsb_metrics_pkey public.ycsb_metrics ycsb_metrics_inserted_at_idx public.pgbench_tellers pgbench_tellers_pkey public.pgbench_branches pgbench_branches_pkey public.pgbench_accounts pgbench_accounts_pkey
44 Tables' sizes How fast do tables and their indexes grow? pg_relation_size(); pg_total_relation_size(); pg_stat_user_tables.
45 Table s sizes pgcenter: :03:55, load average: 3.29, 0.85, 0.81 %cpu: 79.1 us, 8.8 sy, 0.0 ni, 12.1 id, 0.0 wa, 0.0 hi, 0.1 si, 0.0 st MiB mem: total, 230 free, used, 6423 buff/cached MiB swap: total, free, 0 used, 29/0 dirty/writeback relation total_size rel_size idx_size total_change rel_change idx_change public.ycsb_metrics public.pgbench_tellers public.pgbench_history public.pgbench_branches public.pgbench_accounts
46 Hidden problems summary These are not all cases. There are many other use cases.
47 Troubleshooting cases Health checks and load spikes. Hidden problems. Clients' activity. Streaming replication.
48 Clients' activity A lot of active clients. Idle transactions. Waiting clients. Age of the longest transaction. Autovacuum.
49 Clients' activity pgcenter: :02:32, load average: 12.65, 10.19, 5.50 conn1 [ok]: /var/run/postgresql:5432 (ver: 9.6.2, up 1 day 20:14:53) %cpu: 69.0 us, 10.5 sy, 0.0 ni, 20.3 id, 0.2 wa, 0.0 hi, 0.0 si, 0.0 st activity: 11/100 conns, 0/0 prepared, 0 idle, 0 idle_xact, 11 active, 3 waiting, 0 others MiB mem: total, 224 free, used, 6434 buff/cached autovacuum: 0/3 workers/max, 0 manual, 0 wraparound, 00:00:00 vac_maxtime MiB swap: total, free, 0 used, 1/0 dirty/writeback statements: stmt/s, stmt_avgtime, 00:00:00 xact_maxtime, 00:00:00 prep_maxtime pid cl_addr cl_port datname usename state wait_etype wait_event xact_age query_age change_age query pgbench_6gb postgres active 00:00:00 00:00:00 00:00:00 UPDATE pgbench_accounts SET abalance = abalance WH pgbench_6gb postgres active 00:00:00 00:00:00 00:00:00 UPDATE pgbench_accounts SET abalance = abalance WH pgbench_6gb postgres active 00:00:00 00:00:00 00:00:00 UPDATE pgbench_accounts SET abalance = abalance WH pgbench_6gb postgres active 00:00:00 00:00:00 00:00:00 UPDATE pgbench_accounts SET abalance = abalance WH pgbench_6gb postgres active LWLockNamed XidGenLock 00:00:00 00:00:00 00:00:00 UPDATE pgbench_accounts SET abalance = abalance WHE pgbench_6gb postgres active 00:00:00 00:00:00 00:00:00 UPDATE pgbench_accounts SET abalance = abalance WHE pgbench_6gb postgres active LWLockTranche buffer_content 00:00:00 00:00:00 00:00:00 UPDATE pgbench_accounts SET abalance = abalance WH pgbench_6gb postgres active 00:00:00 00:00:00 00:00:00 UPDATE pgbench_accounts SET abalance = abalance W pgbench_6gb postgres active 00:00:00 00:00:00 00:00:00 UPDATE pgbench_accounts SET abalance = abalance W pgbench_6gb postgres active LWLockNamed XidGenLock 00:00:00 00:00:00 00:00:00 UPDATE pgbench_accounts SET abalance = abalance WH
50 Clients' activity summary Avoid: idle and long transactions cause bloat; deadlocks frustrate users. Possible solutuions: cancel or terminate them; inspect an application.
51 Troubleshooting cases Health checks and load spikes. Hidden problems. Clients' activity. Streaming replication.
52 Streaming replication Replication issues: replication lag; standby s queries.
53 pg_stat_replication & co pgcenter: :13:49, load average: 0.04, 3.10, 4.61 conn1 [ok]: /var/run/postgresql:5432 %cpu: 0.0 us, 0.1 sy, 0.0 ni, 99.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st activity: 1/100 conns, 0/0 prepared, 0 idle, MiB mem: total, 232 free, used, 6431 buff/cached autovacuum: 0/3 workers/max, 0 manual, 0 wra MiB swap: total, free, 0 used, 0/0 dirty/writeback statements: 6 stmt/s, stmt_avgtime, 00: client user name state mode xlog pending write flush replay total_lag xact_age time_age replica walreceiver streaming async :02: replica walreceiver streaming async replica walreceiver streaming async :00:01
54 Troubleshooting cases Health checks and load spikes. Hidden problems. Clients activity. Streaming replication.
55 Summary Postgres offers a lot of stats. pgcenter offers handy stats' representation. pgcenter helps to solve many types of issues. It's still work in progress.
56 Links pgcener Slides DataEgret DataEgret Blog
57 Thanks for watching!
Monitoring and Debugging PostgreSQL
Monitoring and Debugging PostgreSQL Ashutosh Sharma 2018.02.22 2013 EDB All rights reserved. 1 Agenda Database Logs The PostgreSQL Statistics Collector Statistics Views Wait Events Monitoring query performance
Monitoring PostgreSQL At
Monitoring PostgreSQL At Scale pganalyze Statistics That Matter Two Tables To Remember Breaking Down High-Level Statistics Log Events Worth Knowing Fingerprinting & Tracing Queries Statistics That Matter
Troubleshooting performance issues in PostgreSQL
Troubleshooting performance issues in PostgreSQL Camille Baldock, Salesforce LISA2017 2017, Camille Baldock All rights including distribution and publication are reserved 1 Hello T6: Troubleshooting performance
(auto)vacuum and You. Gabrielle Roth
(auto)vacuum and You. Gabrielle Roth EnterpriseDB @gorthx The Plan. Part I: Intro Part II: VACUUM Part II: Autovacuum Part III: Adjusting autovacuum parameters Part I: Intro. shared by creative commons
(Auto)Vacuum and You. Gabrielle Roth
(Auto)Vacuum and You Gabrielle Roth Me. PDXPUG! @gorthx twitter, gmail, wordpress "I use Postgres because I don't have to care." Topics Vacuum & autovacuum A little bit about ANALYZE A little bit about
Introduction to pgbench
2ndQuadrant US 04/05/2009 About this presentation The master source for these slides is http://projects.2ndquadrant.com Slides are released under the Creative Commons Attribution 3.0 United States License
VMware vrealize operations Management Pack FOR. PostgreSQL. User Guide
VMware vrealize operations Management Pack FOR PostgreSQL User Guide TABLE OF CONTENTS 1. Purpose... 3 2. Introduction to the Management Pack... 3 2.1 How the Management Pack Collects Data... 3 2.2 Data
Performance Tuning of the Network Performance Server in Network Node Manager i December 8, 2016
Performance Tuning of the Network Performance Server in Network Node Manager i December 8, 2016 Brought to you by Vivit Network Management SIG Leaders Wendy Wheeler Chris Powers Hosted By Wendy Wheeler
Professional PostgreSQL monitoring made easy. Kaarel Moppel Kaarel Moppel
Professional PostgreSQL monitoring made easy Kaarel Moppel Kaarel Moppel Why to monitor Failure / Downtime detection Slowness / Performance analysis Proactive predictions Maybe wasting money? Kaarel Moppel
Detecting MySQL IO problems on Linux at different abstraction layers. Nickolay Ihalainen Percona Live London 2011
Detecting MySQL IO problems on Linux at different abstraction layers Nickolay Ihalainen Percona Live London 2011 Agenda Dataflow layers OS tools MySQL instrumentation Inside InnoDB: story of one insert
MGRID. Implementing MGRID. CHAR(11), Cambridge, July 12, 2011
MGRID Implementing MGRID CHAR(11), Cambridge, July 12, 2011 Portavita Chronic disease management Largest online electronic health record (EHR) in Netherlands Largest telemedicine project in Europe 2/34
PostgreSQL Query Optimization. Step by step techniques. Ilya Kosmodemiansky
PostgreSQL Query Optimization Step by step techniques Ilya Kosmodemiansky (ik@) Agenda 2 1. What is a slow query? 2. How to chose queries to optimize? 3. What is a query plan? 4. Optimization tools 5.
Mastering the art of indexing
Mastering the art of indexing Yoshinori Matsunobu Lead of MySQL Professional Services APAC Sun Microsystems Yoshinori.Matsunobu@sun.com 1 Table of contents Speeding up Selects B+TREE index structure Index
Benchmarking In PostgreSQL
Benchmarking In PostgreSQL Lessons learned Kuntal Ghosh (Senior Software Engineer) Rafia Sabih (Software Engineer) 2017 EnterpriseDB Corporation. All rights reserved. 1 Overview Why benchmarking on PostgreSQL
Introduction to Linux features for disk I/O
Martin Kammerer 3/22/11 Introduction to Linux features for disk I/O visit us at http://www.ibm.com/developerworks/linux/linux390/perf/index.html Linux on System z Performance Evaluation Considerations
In-core compression: how to shrink your database size in several times. Aleksander Alekseev Anastasia Lubennikova.
In-core compression: how to shrink your database size in several times Aleksander Alekseev Anastasia Lubennikova www.postgrespro.ru What does Postgres store? Agenda A couple of words about storage internals
High availability and analysis of PostgreSQL
High availability and analysis of PostgreSQL Sergey Kalinin 18-19 of April 2012, dcache Workshop, Zeuthen Content There is a lot you can do with PG. This talk concentrates on backup, high availability
PostreSQL Monitoring Enhancements
PostreSQL Monitoring Enhancements -Rahila Syed 1 2015 EnterpriseDB Corporation. All rights reserved. # Progress Reporting API Progress reporting of utility commands Parameters reported 10 64-bit counters
Inside the PostgreSQL Shared Buffer Cache
Truviso 07/07/2008 About this presentation The master source for these slides is http://www.westnet.com/ gsmith/content/postgresql You can also find a machine-usable version of the source code to the later
PostgreSQL Performance The basics
PostgreSQL Performance The basics Joshua D. Drake jd@commandprompt.com Command Prompt, Inc. United States PostgreSQL Software in the Public Interest The dumb simple RAID 1 or 10 (RAID 5 is for chumps)
Autovacuum, explained for engineers PGCon 2016, Ottawa. Ilya Kosmodemiansky
Autovacuum, explained for engineers PGCon 2016, Ottawa Ilya Kosmodemiansky ik@postgresql-consulting.com Outline What is it and why is it so important? Aggressiveness of autovacuum What else important can
Database Hardware Benchmarking
Truviso 05/21/2009 About this presentation The master source for these slides is http://www.westnet.com/~gsmith/content/postgresql Slides are released under the Creative Commons Attribution 3.0 United
Michael Glaesemann. myyearbook.com.
Michael Glaesemann myyearbook.com michael.glaesemann@myyearbook.com We were somewhere around Barstow on the edge of the desert when the drugs began to take hold Visualizing Postgres PGCon 2009 Ottawa
Faster and Durable Hash Indexes
Faster and Durable Hash Indexes Amit Kapila 2017.05.25 2013 EDB All rights reserved. 1 Contents Hash indexes Locking improvements Improved usage of space Code re-factoring to enable write-ahead-logging
PostgreSQL what's new
PostgreSQL 9.1 - what's new PGDay.IT 2011 Prato, Italy Magnus Hagander magnus@hagander.net @magnushagander PRODUCTS CONSULTING APPLICATION MANAGEMENT IT OPERATIONS SUPPORT TRAINING PostgreSQL 9.1 Released
Performance Enhancements In PostgreSQL 8.4
Performance Enhancements In PostgreSQL 8.4 PGDay.EU 2009 Paris, France Magnus Hagander Redpill Linpro AB PostgreSQL 8.4 Released July 2009 8.4.1 released September 2009 Major upgrade from 8.3 New features
pg_statsinfo - More useful information for DBAs -
pg_statsinfo - More useful information for DBAs - Tatsuhito Kasahara NTT OSS Center PGCon 2010 Ottawa Copyright(c) 2010 NTT, Inc. All Rights Reserved. Agenda Background What s the pg_statsinfo? Goal Overview
Mvcc Unmasked BRUCE MOMJIAN
Mvcc Unmasked BRUCE MOMJIAN This talk explains how Multiversion Concurrency Control (MVCC) is implemented in Postgres, and highlights optimizations which minimize the downsides of MVCC. Creative Commons
Load Testing Tools. for Troubleshooting MySQL Concurrency Issues. May, 23, 2018 Sveta Smirnova
Load Testing Tools for Troubleshooting MySQL Concurrency Issues May, 23, 2018 Sveta Smirnova Introduction This is very personal webinar No intended use No best practices No QA-specific tools Real life
MONITORING 101: POSTGRESQL JASON YEE, DATADOG
MONITORING 101: POSTGRESQL JASON YEE, DATADOG @gitbisect COLLECTING DATA IS CHEAP; NOT HAVING IT WHEN YOU NEED IT CAN BE EXPENSIVE SO INSTRUMENT ALL THE THINGS! @gitbisect Technical Writer/Evangelist
Keep an eye on your PostgreSQL clusters
Keep an eye on your PostgreSQL clusters Open PostgreSQL Monitoring & PostgreSQL Workload Analyzer Julien Rouhaud Dalibo - www.dalibo.org pgconf.ru 2015 - February, 6th [ 1 / 37 ] Monitoring? Service availability
Professional PostgreSQL monitoring made easy. Kaarel Moppel - p2d2.cz 2019 Prague
Professional PostgreSQL monitoring made easy Kaarel Moppel - p2d2.cz 2019 Prague Who? Kaarel Moppel Senior Database Consultant km@cybertec.at PostgreSQL Database Services Training Consulting High Availability
PgBench Work in Progress
1 / 29 Work in Progress Fabien Coelho MINES ParisTech, PSL Research University PostgreSQL Session #9, Paris November 17, 2017 2 / 29 Work in Progress Talk Outline 1 3 2 Costs Costs 3 / 29 Pgbench 4 / 29
Database Hardware Benchmarking
2ndQuadrant US 03/25/2009 About this presentation The master source for these slides is http://www.2ndquadrant.com/talks Slides are released under the Creative Commons Attribution 3.0 United States License
Just In Time Compilation in PostgreSQL 11 and onward
Just In Time Compilation in PostgreSQL 11 and onward Andres Freund PostgreSQL Developer & Committer Email: andres@anarazel.de Email: andres.freund@enterprisedb.com Twitter: @AndresFreundTec anarazel.de/talks/2018-09-07-pgopen-jit/jit.pdf
Всё, что вы хотели узнать про автовакуум в PostgreSQL. Ilya Kosmodemiansky
Всё, что вы хотели узнать про автовакуум в PostgreSQL Ilya Kosmodemiansky ik@postgresql-consulting.com Outline What is it and why is it so important? Aggressiveness of autovacuum What else important can
How have PostgreSQL community and Market evolved in Japan. Tatsuo Ishii SRA OSS, Inc. Japan
How have community and Market evolved in Japan Tatsuo Ishii SRA OSS, Inc. Japan 2 Who am I? OSS developer Community activity committer pgpool-ii community lead Co founder of Japan Users Group Secretary
VMWARE VREALIZE OPERATIONS MANAGEMENT PACK FOR. MongoDB. User Guide
VMWARE VREALIZE OPERATIONS MANAGEMENT PACK FOR MongoDB User Guide TABLE OF CONTENTS 1. Purpose... 3 2. Introduction to the Management Pack... 3 2.1 How the Management Pack Collects Data... 3 2.2 Data the
PostgreSQL Replication 2.0
PostgreSQL Replication 2.0 NTT OSS Center Masahiko Sawada PGConf.ASIA 2017 Copyright 2017 NTT corp. All Rights Reserved. Who am I Masahiko Sawada @sawada_masahiko NTT Open Source Software Center PostgreSQL
Practical PostgreSQL
Practical PostgreSQL at myyearbook.com michael.glaesemann@myyearbook.com CS 500 Database Theory Department of Computer Science Drexel University 2010 08 05 myyearboook.com mind/body failure myb topology
PoWA Documentation. Release Dalibo
PoWA Documentation Release 1.2.1 Dalibo May 04, 2015 Contents 1 PostgreSQL Workload Analyzer detailled installation guide 3 1.1 Download PoWA from the website................................... 3 1.2
IT Best Practices Audit TCS offers a wide range of IT Best Practices Audit content covering 15 subjects and over 2200 topics, including:
IT Best Practices Audit TCS offers a wide range of IT Best Practices Audit content covering 15 subjects and over 2200 topics, including: 1. IT Cost Containment 84 topics 2. Cloud Computing Readiness 225
PostgreSQL Solution 1.1
vrealize Operations Manager This document supports the version of each product listed and supports all subsequent versions until the document is replaced by a new edition. To check for more recent editions
Effective Testing for Live Applications. March, 29, 2018 Sveta Smirnova
Effective Testing for Live Applications March, 29, 2018 Sveta Smirnova Table of Contents Sometimes You Have to Test on Production Wrong Data SELECT Returns Nonsense Wrong Data in the Database Performance
PostgreSQL West Scaling PostgreSQL with Hot Standby
PostgreSQL West 2010 Scaling PostgreSQL with Hot Standby Hot Standby New in PostgreSQL 9.0 Allows connections to server while archive recovery is in progress Connections will not be terminated if/when
JBoss Enterprise Application Platform: Troubleshooting Guide
JBoss Enterprise Application Platform: Troubleshooting Guide JBoss Enterprise Application Platform includes one of the leading Java EE certified application server which is used by many fortune companies
MySQL Performance Optimization and Troubleshooting with PMM. Peter Zaitsev, CEO, Percona
MySQL Performance Optimization and Troubleshooting with PMM Peter Zaitsev, CEO, Percona In the Presentation Practical approach to deal with some of the common MySQL Issues 2 Assumptions You re looking
What s New in PostgreSQL 9.6, by PostgreSQL contributor
What s New in PostgreSQL 9.6, by PostgreSQL contributor NTT OpenSource Software Center Masahiko Sawada @PGConf.ASIA 2016 (2 Dec) Who am I? Masahiko Sawada Twitter : @sawada_masahiko PostgreSQL Contributor
What is the Future of PostgreSQL?
What is the Future of PostgreSQL? Robert Haas 2013 EDB All rights reserved. 1 PostgreSQL Popularity By The Numbers Date Rating Increase vs. Prior Year % Increase January 2016 282.401 +27.913 +11% January
PostgreSQL monitoring with pgwatch2. Kaarel Moppel / PostgresConf US 2018
PostgreSQL monitoring with pgwatch2 Why to monitor Failure / Downtime detection Slowness / Performance analysis Proactive predictions Maybe wasting money? Different levels of Database monitoring Service
Postgres Cluster and Multimaster
Postgres Cluster and Multimaster postgrespro.ru Ivan Panchenko Postgres Pro Cluster definition: several DBs working as one Redundancy Sharding Parallel query processing Failover Dynamic reconfiguration
PostgreSQL 10. PGConf.Asia 2017 Tokyo, Japan. Magnus Hagander
PostgreSQL 10 PGConf.Asia 2017 Tokyo, Japan Magnus Hagander magnus@hagander.net Magnus Hagander Redpill Linpro Principal database consultant PostgreSQL Core Team member Committer PostgreSQL Europe PostgreSQL
Using PostgreSQL in Tantan - From 0 to 350bn rows in 2 years
Using PostgreSQL in Tantan - From 0 to 350bn rows in 2 years Victor Blomqvist vb@viblo.se Tantan ( 探探 ) December 2, PGConf Asia 2016 in Tokyo tantanapp.com 1 Sweden - Tantan - Tokyo 10 Million 11 Million
<Insert Picture Here> Looking at Performance - What s new in MySQL Workbench 6.2
Looking at Performance - What s new in MySQL Workbench 6.2 Mario Beck MySQL Sales Consulting Manager EMEA The following is intended to outline our general product direction. It is
New Feature Grand Prix
Josh Berkus pgcon 2012 Read Scalability Write Performance Cascading Replication Index-only Scans Start Other Features JSON DDL Migrations Range Types Admin Tools SP-GiST New Feature Grand Prix START YOUR
VACUUM Strategy: Autovacuum, FSM, Visibility Map. Selena Deckelmann End Point
VACUUM Strategy: Autovacuum, FSM, Visibility Map Selena Deckelmann End Point Corporation @selenamarie Hi! Postgres consultant for End Point User Group Liaison for postgresql.org Conference Organizer What
CS 5614: Transaction Processing 121. Transaction = Unit of Work Recall ACID Properties (from Module 1)
CS 5614: Transaction Processing 121 Module 3: Transaction Processing Transaction = Unit of Work Recall ACID Properties (from Module 1) Requirements of Transactions in a DBMS 7-by-24 access Concurrency
Breaking PostgreSQL at Scale. Christophe Pettus PostgreSQL Experts FOSDEM 2019
Breaking PostgreSQL at Scale. Christophe Pettus PostgreSQL Experts FOSDEM 2019 Christophe Pettus CEO, PostgreSQL Experts, Inc. christophe.pettus@pgexperts.com thebuild.com twitter @xof So, what is this?
MONITORING 101: POSTGRESQL JASON YEE, DATADOG
MONITORING 101: POSTGRESQL JASON YEE, DATADOG @gitbisect @gitbisect Technical Writer/Evangelist Docs & Talks Travel Hacker & Whiskey Hunter @datadoghq SaaS-based monitoring Trillions of data points per
MySQL Performance Optimization and Troubleshooting with PMM. Peter Zaitsev, CEO, Percona Percona Technical Webinars 9 May 2018
MySQL Performance Optimization and Troubleshooting with PMM Peter Zaitsev, CEO, Percona Percona Technical Webinars 9 May 2018 Few words about Percona Monitoring and Management (PMM) 100% Free, Open Source
Taking advantage of custom bgworkers
Taking advantage of custom bgworkers Postgres Open 2013, Chicago, 2013/09/16 Michael Paquier, Member of Technical Staff PostgreSQL 2013 VMware Inc. All rights reserved About your lecturer Michael Paquier
Utils Commands CHAPTER
CHAPTER 9 Published: October 31, 2011, This chapter contains the following utils commands: utils auditd disable utils auditd enable utils auditd status utils create report hardware utils create report
Postgres-XC PostgreSQL Conference Michael PAQUIER Tokyo, 2012/02/24
Postgres-XC PostgreSQL Conference 2012 Michael PAQUIER Tokyo, 2012/02/24 Agenda Self-introduction Highlights of Postgres-XC Core architecture overview Performance High-availability Release status Copyright
Postgres-XC PG session #3. Michael PAQUIER Paris, 2012/02/02
Postgres-XC PG session #3 Michael PAQUIER Paris, 2012/02/02 Agenda Self-introduction Highlights of Postgres-XC Core architecture overview Performance High-availability Release status 2 Self-introduction
Chapter 4: Transaction Models
Chapter 4: Transaction Models Handout #19 Overview simple transactions (flat) atomicity & spheres of control non-flat transactions CS346 - Transaction Processing Markus Breunig - 4 / 1 - Atomic Actions
PL/Profiler. Presented by Jan Wieck, OpenSCG Inc.
PL/Profiler Presented by Jan Wieck, OpenSCG Inc. What is Profiling? In software engineering, profiling ( program profiling, software profiling ) is a form of dynamic program analysis that measures, for
Linux OS Fundamentals for the SQL Admin. Anthony E. Nocentino
Linux OS Fundamentals for the SQL Admin Anthony E. Nocentino aen@centinosystems.com Anthony E. Nocentino Consultant and Trainer Founder and President of Centino Systems Specialize in system architecture
Utils Commands CHAPTER
CHAPTER 9 Published: October 26, 2010, This chapter contains the following utils commands: utils fior utils firewall utils iostat utils iothrottle enable utils iothrottle disable utils iothrottle status
Oracle GoldenGate Performance Best Practices O R A C L E W H I T E P A P E R M A Y
Oracle GoldenGate Performance Best Practices O R A C L E W H I T E P A P E R M A Y 2 0 1 7 Table of Contents Table of Contents 2 Introduction 4 Oracle Software 5 Database Configuration 5 Configuring the
Advanced Topics on the Mirth Connect Interface Engine. July 6, 2016
Advanced Topics on the Mirth Connect Interface Engine July 6, 2016 You have been automatically muted. Please use the Q&A panel to submit questions during the presentation PRESENTER Nate Bessa Technical
16 Sharing Main Memory Segmentation and Paging
Operating Systems 64 16 Sharing Main Memory Segmentation and Paging Readings for this topic: Anderson/Dahlin Chapter 8 9; Siberschatz/Galvin Chapter 8 9 Simple uniprogramming with a single segment per
PolarDB. Cloud Native Alibaba. Lixun Peng Inaam Rana Alibaba Cloud Team
PolarDB Cloud Native DB @ Alibaba Lixun Peng Inaam Rana Alibaba Cloud Team Agenda Context Architecture Internals HA Context PolarDB is a cloud native DB offering Based on MySQL-5.6 Uses shared storage
Introducing PostgreSQL Enterprise Consortium activities
Introducing PostgreSQL Enterprise Consortium activities Hitoshi Hemmi / NIPPON TELEGRAPH AND TELEPHONE CORPORATION Tatsuo Ishii / SRA OSS, Inc. Japan Executive board member of PostgreSQL Enterprise Consortium
Server Status Dashboard
The Cisco Prime Network Registrar server status dashboard in the web user interface (web UI) presents a graphical view of the system status, using graphs, charts, and tables, to help in tracking and diagnosis.
Build ETL efficiently (10x) with Minimal Logging
Simon Cho Build ETL efficiently (10x) with Minimal Logging SQL Saturday #696- Redmond 2/10/2018 Simon Cho Chapter leader of SQLAngeles.com SQL Community Speaker Visa Inc Database Engineer Blog : Simonsql.com
White Paper ETERNUS AF series Best Suited for Databases. White Paper
White Paper ETERNUS AF series All-Flash Arrays Best Suited for Databases Typically, complicated RAID designs are required for hard disk arrays to secure high-speed processing of databases. However, all-flash
Mega-scale Postgres How to run 1,000,000 Postgres Databases
Mega-scale Postgres How to run 1,000,000 Postgres Databases Program What is Heroku & Heroku Postgres? Organizing principles for mega-scale operations Heroku Postgres Code deployment is good, but what
FLAT DATACENTER STORAGE. Paper-3 Presenter-Pratik Bhatt fx6568
FLAT DATACENTER STORAGE Paper-3 Presenter-Pratik Bhatt fx6568 FDS Main discussion points A cluster storage system Stores giant "blobs" - 128-bit ID, multi-megabyte content Clients and servers connected
VMWARE VREALIZE OPERATIONS MANAGEMENT PACK FOR. Amazon Aurora. User Guide
VMWARE VREALIZE OPERATIONS MANAGEMENT PACK FOR User Guide TABLE OF CONTENTS 1. Purpose...3 2. Introduction to the Management Pack...3 2.1 How the Management Pack Collects Data...3 2.2 Data the Management
15 Sharing Main Memory Segmentation and Paging
Operating Systems 58 15 Sharing Main Memory Segmentation and Paging Readings for this topic: Anderson/Dahlin Chapter 8 9; Siberschatz/Galvin Chapter 8 9 Simple uniprogramming with a single segment per
Table of Contents. Copyright Pivotal Software Inc,
Table of Contents Table of Contents Greenplum Command Center User Guide Dashboard Query Monitor Host Metrics Cluster Metrics Monitoring Multiple Greenplum Database Clusters Historical Queries & Metrics
pgconf.de 2018 Berlin, Germany Magnus Hagander
A look at the Elephants Trunk PostgreSQL 11 pgconf.de 2018 Berlin, Germany Magnus Hagander magnus@hagander.net Magnus Hagander Redpill Linpro Principal database consultant PostgreSQL Core Team member Committer
Rdb features for high performance application
Rdb features for high performance application Philippe Vigier Oracle New England Development Center Copyright 2001, 2003 Oracle Corporation Oracle Rdb Buffer Management 1 Use Global Buffers Use Fast Commit
Adding Analytical Behavioral Intelligence to Block Storage Layer
Adding Analytical Behavioral Intelligence to Block Storage Layer andy mills, co-founder/ceo June 2nd 2016 www.enmotus.com Topics Need for a fast, efficient storage device virtualization (SDV) layer Behavioral
Linux OS Fundamentals for the SQL Admin. Anthony E. Nocentino
Linux OS Fundamentals for the SQL Admin Anthony E. Nocentino aen@centinosystems.com Anthony E. Nocentino Consultant and Trainer Founder and President of Centino Systems Specialize in system architecture
Crash Consistency: FSCK and Journaling. Dongkun Shin, SKKU
Crash Consistency: FSCK and Journaling 1 Crash-consistency problem File system data structures must persist stored on HDD/SSD despite power loss or system crash Crash-consistency problem The system may
Examples of Physical Query Plan Alternatives. Selected Material from Chapters 12, 14 and 15
Examples of Physical Query Plan Alternatives Selected Material from Chapters 12, 14 and 15 1 Query Optimization NOTE: SQL provides many ways to express a query. HENCE: System has many options for evaluating
V. Primary & Secondary Memory!
V. Primary & Secondary Memory! Computer Architecture and Operating Systems & Operating Systems: 725G84 Ahmed Rezine 1 Memory Technology Static RAM (SRAM) 0.5ns 2.5ns, $2000 $5000 per GB Dynamic RAM (DRAM)
MySQL Performance Troubleshooting
MySQL Performance Troubleshooting Best Practices Francisco Bordenave - Architect, Percona Agenda Who am I? Introduction Identifying the source of problem We know where the problem is, now what? Best practices
ALTER TABLE Improvements in MARIADB Server. Marko Mäkelä Lead Developer InnoDB MariaDB Corporation
ALTER TABLE Improvements in MARIADB Server Marko Mäkelä Lead Developer InnoDB MariaDB Corporation Generic ALTER TABLE in MariaDB CREATE TABLE ; INSERT SELECT; RENAME ; DROP TABLE ; Retroactively named
Conceptual Modeling on Tencent s Distributed Database Systems. Pan Anqun, Wang Xiaoyu, Li Haixiang Tencent Inc.
Conceptual Modeling on Tencent s Distributed Database Systems Pan Anqun, Wang Xiaoyu, Li Haixiang Tencent Inc. Outline Introduction System overview of TDSQL Conceptual Modeling on TDSQL Applications Conclusion
Anthony AWR report INTERPRETATION PART I
Anthony AWR report INTERPRETATION PART I What is AWR? AWR stands for Automatically workload repository, Though there could be many types of database performance issues, but when whole database is slow,
Percona XtraDB Cluster ProxySQL. For your high availability and clustering needs
Percona XtraDB Cluster-5.7 + ProxySQL For your high availability and clustering needs Ramesh Sivaraman Krunal Bauskar Agenda What is Good HA eco-system? Understanding PXC-5.7 Understanding ProxySQL PXC
Scale out Read Only Workload by sharing data files of InnoDB. Zhai weixiang Alibaba Cloud
Scale out Read Only Workload by sharing data files of InnoDB Zhai weixiang Alibaba Cloud Who Am I - My Name is Zhai Weixiang - I joined in Alibaba in 2011 and has been working on MySQL since then - Mainly
Checking Resource Usage in Fedora (Linux)
Lab 5C Checking Resource Usage in Fedora (Linux) Objective In this exercise, the student will learn how to check the resources on a Fedora system. This lab covers the following commands: df du top Equipment
Pgpool-II Development Status Updates. Pgpool-II Global Development Group
Pgpool-II Development Status Updates Pgpool-II Global Development Group 2 We are moved! Due to closing of pgfoundry, we have moved to new web site(november 2011) http://www.pgpool.net Hosted by SRA OSS,
Overview of Transaction Management
Overview of Transaction Management Chapter 16 Comp 521 Files and Databases Fall 2010 1 Database Transactions A transaction is the DBMS s abstract view of a user program: a sequence of database commands;
Logical Decoding : - Amit Khandekar. Replicate or do anything you want EnterpriseDB Corporation. All rights reserved. 1
Logical Decoding : Replicate or do anything you want - Amit Khandekar 2014 EnterpriseDB Corporation. All rights reserved. 1 Agenda Background Logical decoding Architecture Configuration Use cases 2016
6.830 Lecture Recovery 10/30/2017
6.830 Lecture 14 -- Recovery 10/30/2017 Have been talking about transactions Transactions -- what do they do? Awesomely powerful abstraction -- programmer can run arbitrary mixture of commands that read
6.830 Problem Set 3 Assigned: 10/28 Due: 11/30
6.830 Problem Set 3 1 Assigned: 10/28 Due: 11/30 6.830 Problem Set 3 The purpose of this problem set is to give you some practice with concepts related to query optimization and concurrency control and