PostgreSQL West Scaling PostgreSQL with Hot Standby

Transcription

1 PostgreSQL West 2010 Scaling PostgreSQL with Hot Standby

2 Hot Standby New in PostgreSQL 9.0 Allows connections to server while archive recovery is in progress Connections will not be terminated if/when we move from recovery to normal running at switchover/failover time hot_standby = on (not the default) Works with either streaming or file-based replication

3 User's Overview Connections work normally no special options Transactions will be read only Cannot write new data to the database Cannot write new WAL data Cannot allocate a TransactionId Cannot hold any table lock higher than AccessShare No row level locking No temporary tables No two-phase commit (prepared transactions) No access to sequences (i.e. no nextval())

4 What works?!? SELECTs, Cursors including sort files MVCC works normally Recovery changes made by a single writer Don't normally block readers LOAD so many plugins work normally SET, RESET Serializable transactions, savepoints etc

5 Use Cases High availability with active monitoring Offload long running reports Materialize views Load balance small read-only queries

6 Prioritizations Keep the standby current for failover (good) Long running reports on the standby (fast) Avoid adding overhead to the master (cheap) Good, fast, cheap: pick two

7 Query Conflicts Primary: Drop Database X Standby: Query on database X Cannot do both Action on primary has already happened, so whatever occurs, WAL recovery must always win

8 Causes of Query Conflicts Cleanup records ( Snapshot too old error) Access Exclusive Locks (LOCK, DDL) Drop Database Drop Tablespace Very, very long queries (wraparound issue) Buffer cleanup (VACUUM, HOT) Others can normally be avoided; this cannot.

9 Standard query visibility MVCC keeps multiple row versions Active query transactions are noted Row cleanup avoids touching active data

10 Hot Standby Query Visibility Queries executing on standby are independent Master does not know what is running DROP TABLE will not respect open locks VACUUM cleanup will prune active rows

11 vacuum_defer_cleanup_age Set in master postgresql.conf Units are transactions Monitor txid_current() to estimate times Also delays HOT cleanup Will cause cleanup bloat on the master

12 max_standby_*_delay Set in standby postgresql.conf max_standby_archive_delay Changes from a shipped archive file max_standby_streaming_delay Changes from Streaming Replication Defaults are both 30 seconds Similar to statement_timeout Changes eventually must be applied Maximum amount of time to wait for them

13 max_standby_*_delay 0 always cancels queries immediately Small values prioritize quick failover -1 means wait forever for queries to finish Large values or -1 for reporting server Keep master and standby clocks synchronized

14 Query Conflict Avoidance Connect to primary via dblink Hold open a snapshot while query executes SELECT sleep(60); Similar to how a running query impacts MVCC

15 Optimizing for long queries Avoid cleanup - vacuum_defer_cleanup_age Will bloat the master Set a large value for max_standby_*_delay Will cause standby to fall behind Hold an open snapshot Bloats the master only when running

16 Optimizing for fast failover Low value for max_standby_*_delay

17 Optimizing for load balancing Low max_standby_*_delay for fresh results Application should expect cancellation

18 Routing reads and writes Writes must go to master Reads execute against master or any standby Application may know Application servers may support this concept JDBC Django Database proxy servers can do this routing pgpool-ii 3.0 tastes queries Hard to solve in all cases Where do functions (stored procedures) go?

19 Architecture: Read Scaling Many read copies with slight lag Each is also a potential failover node Not suitable for long reports Writes pgpool-ii router Reads Hot Read-Only Node Primary Node Hot Read-Only Node

20 Architecture: Reporting Server Rolling Reporting Server(s) Live servers runs queries Other servers provide failover capability for Primary Hot Read-Only Reporting Node Primary Node Archive Failover Node

21 Architecture: Relay Server Archive data streamed to a standby Ship the result to a second layer standby pg_streamrecv Hot Standby Hot Standby Primary Node

22 Work in progress Automatic snapshot export to master pg-blender: projects.2ndquadrant.com Easier provisioning/failover repmgr: projects.2ndquadrant.com Improved monitoring Synchronous replication Expected for PostgreSQL 9.1 Master and standby no longer decoupled

23 For more information