Streaming Replication: Switchover/Switchback in EDB Postgres Advanced Server

Transcription

1 Streaming Replication: Switchover/Switchback in EDB Postgres Advanced Server EPAS Version 9.5 July 15, 2016

2 by EnterpriseDB Corporation Copyright 2016 EnterpriseDB Corporation. All rights reserved. EnterpriseDB Corporation, 34 Crosby Drive, Suite 100, Bedford, MA 01730, USA T F E info@enterprisedb.com Copyright 2016 EnterpriseDB Corporation. All rights reserved. 2

3 Table of Contents 1 Introduction Assumptions Typographical Conventions Used in this Guide Configuring Streaming Replication Modifying the Configuration Files Configuring the Master and Standby Nodes Creating a User with Replication Privilege Modifying the pg_hba.conf File Modifying the recovery.conf File Restarting the server Example: Creating a Streaming Replication Cluster with Three Nodes Configuring the Master (node 1) for Streaming Replication Configuring the Standby (node 2) for Streaming Replication Configuring the Standby (node 3) for Streaming Replication Example: Performing a Switchover to a Standby (node 2) Configuring the Master (node 1) for Switchover Configuring the Standby (node 2) for Switchover Configuring the Standby (node 3) for Switchover Example: Performing a Switchback to a Standby (node 1) Configuring the Master (node 2) for Switchback Configuring the Standby (node 1) for Switchback Configuring the Standby (node 3) after switchback Appendix archive_command archive_mode full_page_writes hot_standby max_replication_slots max_wal_senders wal_buffers wal_level wal_log_hints Copyright 2016 EnterpriseDB Corporation. All rights reserved. 3

4 1 Introduction The purpose of this paper is to provide a guide for configuring streaming replication and switchover on an EDB Postgres Advanced Server. This document uses the following key terms and concepts: A failover operation happens when users unexpectedly lose connection to the Master because of network, hardware, or software problems. Failover is the mechanism that promotes a server from Standby to Master. The potential for some loss of data exists resulting from an abrupt failure of the Master. A switchover (or graceful switchover) is a routine, planned activity. During a switchover, a Standby substitutes for the Master to avoid any disruption in service. Normal database operations continue following a small outage when the Standby becomes the new Master. Downtime is minimal, and there is no loss of data. A switchback is a switchover (in the reverse order) that restores the database activity to the original Master server. High availability (HA) solutions include multiple servers configured for replication. The goal of high availability is to sustain uninterrupted services with minimal to no downtime. Streaming replication is the log-based process of updating the Standby with WAL entries. Streaming replication allows a Standby to stay in sync with the primary server. The Master streams WAL records to a Standby as they happen, without waiting for the WAL file to fill. This process operates asynchronously; there is a brief interval before the Standby receives an update from the Master. Though this delay is often negligible, it is never instantaneous. EDB Postgres Advanced Server uses write-ahead logging (WAL) to continuously archive database transactions and record changes to a database. For each change made to data files, WAL writes an entry in a log file. The system uses these log entries to perform point-in-time restoration from archives and to keep the Standby in alignment. Advanced Server maintains the write-ahead log in the pg_xlog/ subdirectory of the cluster data directory. A hot standby system is when data replicates on a Standby in real time: the Standby database automatically activates and immediately replaces the Master in the event of a failure. For more information about replication options, please see the PostgreSQL documentation, available at: Copyright 2016 EnterpriseDB Corporation. All rights reserved. 4

5 In this document we will refer to the primary server as the the Master, and refer to a secondary server as a Standby. A primary server can accept read and write transactions, while a Standby server is a read-only replica. 1.1 Assumptions The subsequent discussion of streaming replication and switchover assumes the following: Each server within your replication scenario is running the same version of EDB Postgres Advanced Server (version 9.4.x or greater). For information about downloading and installing Advanced Server, please visit the EnterpriseDB website at: This guide includes commands and paths for Linux 6.5; if necessary, adjust the commands for your Linux version. Your EDB Postgres instances must be able to communicate with each other. Ensure that the appropriate ports are open and modify configuration settings for firewalls as necessary. You know the IP address of each server. Configuring EFM (EDB Failover Manager) is out of the scope of this document. It is recommended that you refer to the following link for more details about EFM: Copyright 2016 EnterpriseDB Corporation. All rights reserved. 5

6 1.2 Typographical Conventions Used in this Guide Certain typographical conventions are used in this manual to clarify the meaning and usage of various commands, statements, programs, examples, etc. This section provides a summary of these conventions. In the following descriptions a term refers to any word or group of words that are language keywords, user-supplied values, literals, etc. A term s exact meaning depends on the context in which it is used. Italic font introduces a new term, typically, in the sentence that defines it for the first time. Fixed-width (mono-spaced) font is used for terms that must be given literally such as SQL commands, specific table and column names used in the examples, programming language keywords, etc. For example, SELECT * FROM emp; Italic fixed-width font is used for terms for which the user must substitute values in actual usage. For example, DELETE FROM table_name; A vertical pipe denotes a choice between the terms on either side of the pipe. A vertical pipe is used to separate two or more alternative terms within square brackets (optional choices) or braces (one mandatory choice). Square brackets [ ] denote that one or none of the enclosed terms may be substituted. For example, [ a b ] means choose one of a or b or neither of the two. Braces {} denote that exactly one of the enclosed alternatives must be specified. For example, { a b } means exactly one of a or b must be specified. Ellipses... denote that the preceding term may be repeated. For example, [ a b ]... means that you may have the sequence, b a a b a. Copyright 2016 EnterpriseDB Corporation. All rights reserved. 6

7 2 Configuring Streaming Replication Advanced Server streaming replication clusters are extremely robust. Streaming replication is a core capability of Advanced Server that combines high throughput with low latency. A streaming replication scenario consists of one Master instance and one or more replica nodes. The Master streams write-ahead log (WAL) records to a Standby node as each WAL record is generated, ensuring that the replica is kept up-to-date with changes to the Master. The Master can accept read/write transactions, while a Standby can accept read-only transactions, providing high-availability and/or disaster recovery (see Figure 2.1). Standbys can reside on-site or be regionally distributed. Figure 2.1 Streaming replication from Master to Standby. Copyright 2016 EnterpriseDB Corporation. All rights reserved. 7

8 2.1 Modifying the Configuration Files Configuring Switchover In a default Advanced Server installation, the postgresql.conf file resides in the data directory; use your editor of choice to modify the configuration parameters on the master and standby nodes Configuring the Master and Standby Nodes Configuration parameters in the postgresql.conf file specify server behavior for auditing, authentication, encryption, and other behaviors. Each configuration parameter is set using a name/value pair. Parameter names are caseinsensitive. The parameter name is typically separated from its value by an optional equals sign (=). Parameters that are preceded by a pound sign (#) are set to their default value (as shown in the parameter setting). To change a parameter value, remove the pound sign and enter a new value. When you have modified the configuration file, save the file on the remote server: select Save from the File menu, or click the Save toolbar icon. After setting or changing a parameter, you must either reload or restart the server for a new parameter value to take effect. To configure streaming replication, set the following parameters by editing the postgresql.conf file. Specify the following parameters for all nodes: hot_standby = on This parameter specifies whether a user will be able to connect and run queries during recovery. The default value is off. The Master will ignore this parameter because it is necessarily in read/write mode. Set this value to on. wal_level = hot_standby The wal_level parameter determines how much information is written to the WAL (write-ahead log). hot_standby adds information required to run readonly queries on a Standby server. Please note each level includes the information logged at all lower levels. This parameter can only be set at server start. Set this value to hot_standby. Copyright 2016 EnterpriseDB Corporation. All rights reserved. 8

9 max_wal_senders = at least (number of standbys +1) This parameter limits the number of concurrent connections, or the maximum number of simultaneously running WAL sender processes, from standby servers or streaming base backup clients. This parameter is equal to or lower than the value of max_connections. An abrupt streaming client disconnection might cause an orphaned connection slot until the next timeout, so this parameter should be adjusted higher than the maximum number of expected clients to enable reconnection. The default is zero, disabled replication. This parameter can only be set at server start. Our example uses a database architecture with two Standbys; set the value of this parameter equal to 3, or one more than the number of Standbys. archive_mode = on When archive_mode is on, completed WAL segments are sent to an archive storage by setting the archive_command parameter (see the following entry). There is no significant difference between the on and always values, except that the WAL archiver is enabled during both archive recovery and standby modes when the value is set to always. archive_mode cannot be enabled when wal_level is set to minimal. This parameter can only be set at server start. archive_mode and archive_command are distinct parameters. You can change the archive_command parameter without exiting an archive mode. Set this value to on to disable re-archiving of WAL files at Standby side. archive_command = 'rsync -a %p <location of archive directory>/%f' This parameter is ignored unless archive_mode (see preceding entry) is set at server start to archive completed WAL segments. Set this parameter equal to rsync -a %p <location of archive directory>/%f' to archive completed WAL segments. wal_log_hints = on When this value is set to on, the server writes the entire content of each disk page to WAL after a checkpoint and during the first modification of that page, even for non-critical modifications of so-called hint bits. Change this parameter to on; the default value is off. Copyright 2016 EnterpriseDB Corporation. All rights reserved. 9

10 max_replication_slots = at least (number of standbys + 1) This parameter specifies the maximum number of replication slots that the server can support. The default is zero. Setting this number to a lower value than currently existing replication slots will prevent the server from starting. This parameter can only be set at server start. Our example uses a database architecture with a 3 node cluster; set the value of this parameter equal to 3 (1 Master plus 2 replica databases). full_page_writes = on If this parameter is set to on, the PostgreSQL server writes the entire content of each disk page to WAL during the first modification of that page after a checkpoint. A page write concurrent with an operating system crash might be partially completed, resulting in an on-disk page that contains a mixture of both old and new data. The row-level change data stored in WAL cannot guarantee a complete restoration of this page during post-crash recovery. Storing the full page image guarantees that the page can be correctly restored, but increases the size of data storage archives. (One way to reduce the cost of full-page writes is to increase the checkpoint interval parameters because WAL replay always starts from a checkpoint.) Set this parameter equal to on Creating a User with Replication Privilege To enable streaming replication between the Master and Standby, a user must have special permissions for replication activities or be a superuser. To assign a user replication privileges, you can use the REPLICATION SQL clause. The REPLICATION clause allows a role to initiate streaming replication and to put the system in and out of backup mode. A role having the REPLICATION attribute is a highly privileged role, and should be assigned carefully. The following command creates a user with the REPLICATION privilege: CREATE USER repuser PASSWORD '<PASSWORD>' REPLICATION; Copyright 2016 EnterpriseDB Corporation. All rights reserved. 10

11 2.1.3 Modifying the pg_hba.conf File Configuring Switchover The configuration file named pg_hba.conf, usually found in the data directory of your postgres installation, administers client authentication. (The hba stands for hostbased authentication.) The pg_hba.conf file contains a set of records with a one record per line format. Each record is composed of several fields separated by spaces or tabs; a record specifies a connection type, a client IP address range, a database name, a username, and the authentication method to be used for any connection matching the record s parameters. When reading the pg_hba.conf file, the server ignores blank lines and any text after the # comment character. Fields may contain white space only if the field value is doublequoted. Records in the pg_hba.conf file follow the format: #host database user address auth-method Where: the database name in the above entry is replication; the user is repuser and repuser is allowed to connect to the Master for replication; the address is the Standby IP address (<standby server ip>/<cidr>). For example, to allow connections from a user named repuser connecting from a host on a network with the prefix of , add the following record: host replication repuser /24 md5 After modifying the pg_hba.conf file, you must restart the database server to apply the changes Modifying the recovery.conf File With your choice of text editor, create a file named recovery.conf in the data directory under your postgres installation that includes the following specifications: standby_mode = on This parameter specifies whether to start the Advanced Server as a Standby. If this parameter is on, the server will not stop recovery upon reaching the end of Copyright 2016 EnterpriseDB Corporation. All rights reserved. 11

12 archived WAL; it will keep trying to continue recovery by fetching new WAL segments using the restore_command and/or by connecting to the Master server as specified by the primary_conninfo parameter setting. primary_conninfo = 'primary connection info' This parameter specifies the connection string that will be used by the Standby when connecting with the Master. This string follows the format mentioned in the following link: CONNSTRING If any option remains unspecified in this string, then the corresponding environment variable will be used. For more information on environment variables, please refer to the following link: recovery_target_timeline = 'latest' This parameter specifies a particular timeline for recovery. The default is to recover along the same timeline current during the base backup. Setting this parameter to latest recovers the latest timeline found in the archive. primary_slot_name = '<slot name>' This parameter specifies the use of an existing replication slot when connecting to the Master, via streaming replication, to control resource removal on the upstream node. Note: This setting has no effect if primary_conninfo is not set. Before the release of Advanced Server version 9.4, the Master did not preserve WAL for a Standby that lagged behind. If the Standby lagged too far behind, the Master deleted WAL segments before the Standby replayed them, and segments were unrecoverable. To prevent this, a user was expected to configure continuous archiving and provide a restore_command to give the Standby access to the archive, or be willing to work with a less-than-current Standby. An alternative technique, though less reliable, sets wal_keep_segments to a high value; this ensures the Standby never lags too far behind the Master. However, high enough entails guesswork and so this technique offers less security than continuous archiving. Copyright 2016 EnterpriseDB Corporation. All rights reserved. 12

13 From version 9.4 onwards, replication slots allow the Master to hold WAL segments until the Standby secures all WAL segments. Along with the replication_slot parameter, you will be using the restore_command to configure recovery. The restore_command in the recovery.conf file provides a fail safe option for any situation in which the Standby is unable to receive WAL segments from the Master. trigger_file = '<trigger file>' This parameter specifies a trigger file that ends the recovery process for the Standby. Even if this value is not set, you can use pg_ctl to promote the Standby. This setting has no effect if standby_mode is off. restore_command = 'rsync -a <location of archive command>/%f %p' This parameter is required for archive recovery, but is optional for streaming replication. Any %f in the string is replaced by the name of the file you wish to retrieve from the archive, and any %p is replaced by the destination path of the replicated file on the server. It is important for the command to return a zero exit status when the command succeeds; if the command is prompted for file names that are not present in the archive, it must return nonzero. Copyright 2016 EnterpriseDB Corporation. All rights reserved. 13

14 2.2 Restarting the server After modifying the configuration parameters, you must restart the server. If your installation of Advanced Server resides on version 7.x of RHEL or CentOS, you must use the systemctl command to control the Advanced Server service and supporting components. The systemctl command must be in your search path and must be invoked with superuser privileges. To use the command, open a command line, and enter: systemctl restart ppas-9.x If your installation of Advanced Server resides on version 6.x of RHEL or CentOS, you can use the service command to restart the server: service ppas-9.x restart The Linux service command invokes a script (with the same name as the service) that resides in /etc/init.d. If your Linux distribution does not support the service command, you can call the script directly by entering: /etc/init.d/ppas-9.5 restart Copyright 2016 EnterpriseDB Corporation. All rights reserved. 14

15 3 Example: Creating a Streaming Replication Cluster with Three Nodes In this example, you will use three nodes to create a streaming replication cluster on three different machines of Advanced Server instances (see Figure 3.1). Figure 3.1 A three node system. Copyright 2016 EnterpriseDB Corporation. All rights reserved. 15

16 The Master and Standbys use the following IP addresses: Node 1: Node 2: Node 3: Follow the steps below to create a replication cluster: Configuring Switchover 3.1 Configuring the Master (node 1) for Streaming Replication 1. Use the following command to create a primary cluster: /usr/ppas-9.5/bin/initdb -D /data/node 2. Modify the postgresql.conf file of the primary data cluster, using the editor of your choice: port = 5444 wal_level = hot_standby wal_log_hints = on archive_mode = on archive_command = 'rsync -a %p /data/archived_wals/%f' max_wal_senders = 3 max_replication_slots = 3 hot_standby = on Please remember to change the archive_command to reflect your archive WAL file directory in your environment. 3. You must modify the pg_hba.conf file to allow communication between the nodes of the replication scenario. The following example modifies the pg_hba.conf file, using a CIDR address to allow communication between all of the nodes that reside on a network with a prefix of : host replication repuser /24 md5 For more information about modifying the pg_hba.conf file, see: Copyright 2016 EnterpriseDB Corporation. All rights reserved. 16

17 Start the primary cluster using the following pg_ctl command: ~]$ /usr/ppas-9.5/bin/pg_ctl -- pgdata=/data/node -w start :49:24 EDT LOG: redirecting log output to logging collector process :49:24 EDT HINT: Future log output will appear in directory "pg_log". 4. Connect to the edb database and create a database user named repuser with REPLICATION privileges: [VibhorKumar@LDAPKerb node1]$ psql -U VibhorKumar -c "CREATE USER repuser PASSWORD 'repuser' REPLICATION;" -p d edb CREATE ROLE 5. Create two replication slots for Standby node2 and Standby node3: [VibhorKumar@LDAPKerb node1]$ psql -U VibhorKumar -c "select pg_create_physical_replication_slot('node2')" -p 5444 pg_create_physical_replication_slot (node2,) (1 row) [VibhorKumar@LDAPKerb node1]$ psql -U VibhorKumar -c "select pg_create_physical_replication_slot('node3')" -p 5444 pg_create_physical_replication_slot (node3,) (1 row) Copyright 2016 EnterpriseDB Corporation. All rights reserved. 17

18 3.2 Configuring the Standby (node 2) for Streaming Replication 1. Using the pg_basebackup command, create a base backup of the primary cluster: [VibhorKumar@LDAPKerb node1]$ /usr/ppas-9.5/bin/pg_basebackup --pgdata=/data/node --format=p --write-recovery-conf --xlogmethod=stream --dbname "host= port=5444 user=repuser password=repuser" If you want to hide the password for security reasons, you can modify the.pgpass file or you can enter a password at the pg_basebackup password prompt. The Standby data directory is /data/node. 2. Modify the postgresql.conf file with the following values: port = 5444 wal_level = hot_standby wal_log_hints = on archive_mode = on archive_command = 'rsync -a %p /data/archived_wals/%f' max_wal_senders = 3 max_replication_slots = 3 hot_standby = on 3. Modify or create the recovery.conf file in the data directory of Standby (node 2) with the following parameters: standby_mode = 'on' primary_conninfo = 'user=repuser password=repuser host= port=5444 sslmode=prefer sslcompression=1 krbsrvname=postgres' recovery_target_timeline = 'latest' primary_slot_name = 'node2' trigger_file = '/tmp/node2' restore_command = 'rsync -a /data/archived_wals/%f %p' 4. Start the Standby (node 2) using the pg_ctl command: /usr/ppas-9.5/bin/pg_ctl --pgdata=/data/node -w start [VibhorKumar@LDAPKerb ~]$ /usr/ppas-9.5/bin/pg_ctl -- pgdata=/data/node -w start :28:26 EDT LOG: redirecting log output to logging collector process :28:26 EDT HINT: Future log output will appear in directory "pg_log". Copyright 2016 EnterpriseDB Corporation. All rights reserved. 18

19 After the command executes, connect to the Master (node 1) and use the following command to confirm replication: ~]$ psql -U VibhorKumar -x -c "select * from pg_stat_replication" -p 5444 As the output below shows, the Standby (node 2) is connected and receiving data from the Master. If the connection between the Standby and the Master is unsuccessful, there will be no record of the connection. -[ RECORD 1 ] pid usesysid usename repuser application_name walreceiver client_addr client_hostname client_port backend_start 30-APR-16 08:28: :00 backend_xmin state streaming sent_location 0/ write_location 0/ flush_location 0/ replay_location 0/ sync_priority 0 sync_state async The parameter client_hostname is empty (by default, the log_hostname parameter is not enabled). To display the client_hostname in the pg_stat_replication output, set the log_hostname parameter in the postgresql.conf file; setting this parameter may slow performance. 5. Create two replication slots, one for Master (node 1) and one for Standby (node 3), to prevent recovery conflicts when there is a switchover: [VibhorKumar@LDAPKerb ~]$ psql -U VibhorKumar -c "select pg_create_physical_replication_slot('node1')" -p 5444 pg_create_physical_replication_slot (node1,) (1 row) [VibhorKumar@LDAPKerb ~]$ psql -U VibhorKumar -c "select pg_create_physical_replication_slot('node3')" -p 5444 pg_create_physical_replication_slot (node3,) (1 row) Copyright 2016 EnterpriseDB Corporation. All rights reserved. 19

20 3.3 Configuring the Standby (node 3) for Streaming Replication 1. Using the pg_basebackup command, create a base backup of the primary cluster: [VibhorKumar@LDAPKerb node1]$ /usr/ppas-9.5/bin/pg_basebackup --pgdata=/data/node --format=p --write-recovery-conf --xlogmethod=stream --dbname "host= port=5444 user=repuser password=repuser" If you want to hide the password for security reasons, you can modify the.pgpass file or you can enter a password at the pg_basebackup password prompt. The Standby data directory is /data/node. 2. Modify the postgresql.conf file with the following values: port = 5444 wal_level = hot_standby wal_log_hints = on archive_mode = on archive_command = 'rsync -a %p /data/archived_wals/%f' max_wal_senders = 3 max_replication_slots = 3 hot_standby = on 3. Modify or create the recovery.conf file in the data directory of Standby (node 3) with the following parameters: standby_mode = 'on' primary_conninfo = 'user=repuser password=repuser host= port=5444 sslmode=prefer sslcompression=1 krbsrvname=postgres' recovery_target_timeline = 'latest' primary_slot_name = 'node3' trigger_file = '/tmp/node3' restore_command = 'rsync -a /data/archived_wals/%f %p' 4. Start the Standby (node 3) using the pg_ctl command: /usr/ppas-9.5/bin/pg_ctl --pgdata=/data/node -w start [VibhorKumar@LDAPKerb ~]$ /usr/ppas-9.5/bin/pg_ctl -- pgdata=/data/node -w start :28:26 EDT LOG: redirecting log output to logging collector process :28:26 EDT HINT: Future log output will appear in directory "pg_log". Copyright 2016 EnterpriseDB Corporation. All rights reserved. 20

21 After the command executes, connect to the Master (node 1) and use the following command to confirm replication: ~]$ psql -U VibhorKumar -x -c "select * from pg_stat_replication" -p 5444 As the output below shows, the Standby (node 3) is connected and receiving data from the Master. If the connection between the Standby and the Master is unsuccessful, there will be no record of the second connection. You now have a working three node connection. -[ RECORD 1 ] pid usesysid usename repuser application_name walreceiver client_addr client_hostname client_port backend_start 30-APR-16 08:28: :00 backend_xmin state streaming sent_location 0/A write_location 0/A flush_location 0/A replay_location 0/A sync_priority 0 sync_state async -[ RECORD 2 ] pid usesysid usename repuser application_name walreceiver client_addr client_hostname client_port backend_start 30-APR-16 14:26: :00 backend_xmin state streaming sent_location 0/A write_location 0/A flush_location 0/A replay_location 0/A sync_priority 0 sync_state async 5. Create two replication slots, one for Master (node 1) and one for Standby (node 2), to prevent recovery conflicts if there is a switchover to Standby (node 3): Copyright 2016 EnterpriseDB Corporation. All rights reserved. 21

22 ~]$ psql -U VibhorKumar -c "select pg_create_physical_replication_slot('node1')" -p 5444 pg_create_physical_replication_slot (node1,) (1 row) ~]$ psql -U VibhorKumar -c "select pg_create_physical_replication_slot('node2')" -p 5444 pg_create_physical_replication_slot (node2,) (1 row) Copyright 2016 EnterpriseDB Corporation. All rights reserved. 22

23 4 Example: Performing a Switchover to a Standby (node 2) Follow the steps in this section to perform a switchover (see Figure 4.1): Figure 4.1 Switchover to node 2. Copyright 2016 EnterpriseDB Corporation. All rights reserved. 23

24 4.1 Configuring the Master (node 1) for Switchover 1. Perform a clean shutdown on the Master (node 1); use pg_ctl or the following service script: /usr/ppas-9.5/bin/pg_ctl --pgdata=/data/node --mode=fast stop 2. Create a recovery.conf file in the Master data directory and include the following parameters: standby_mode = 'on' primary_conninfo = 'user=repuser password=repuser host= port=5444 sslmode=prefer sslcompression=1 krbsrvname=postgres' recovery_target_timeline = 'latest' primary_slot_name = 'node1' trigger_file = '/tmp/node1' restore_command = 'rsync -a /data/archived_wals/%f %p' Please note that the value set for primary_conninfo is the Standby (node 2) IP address. The recovery.conf file ensures that the node 1 will start in recovery mode and will not accept any writes. 3. Use the following pg_ctl command to start the Master (node 1) : [VibhorKumar@LDAPKerb ~]$ /usr/ppas-9.5/bin/pg_ctl -- pgdata=/data/node start [VibhorKumar@LDAPKerb ~]$ :27:11 EDT LOG: redirecting log output to logging collector process :27:11 EDT HINT: Future log output will appear in directory "pg_log". All three nodes are now running in recovery mode; none are accepting writes. 4. Connect to the Master (node 1) to verify the replication status: [VibhorKumar@LDAPKerb ~]$ psql -U VibhorKumar -x -c "select * from pg_stat_replication" -p 5444 As the output shows, the Standby (node 2) and the Standby (node 3) are streaming WAL segments from the Master (node 1) and are fully caught up. -[ RECORD 1 ] pid usesysid usename repuser Copyright 2016 EnterpriseDB Corporation. All rights reserved. 24

25 application_name walreceiver client_addr client_hostname client_port backend_start 02-MAY-16 10:29: :00 backend_xmin state streaming sent_location 0/B write_location 0/B flush_location 0/B replay_location 0/B sync_priority 0 sync_state async -[ RECORD 2 ] pid usesysid usename repuser application_name walreceiver client_addr client_hostname client_port backend_start 02-MAY-16 10:29: :00 backend_xmin state streaming sent_location 0/B write_location 0/B flush_location 0/B replay_location 0/B sync_priority 0 sync_state async 5. Connect to the Standby (node 2) to verify its replication status: [VibhorKumar@LDAPKerb ~]$ psql -U VibhorKumar -x -c "select * from pg_stat_replication" -p 5444 As the output shows, the prior Master (node 1) is now behaving as a Standby for the Standby (node 2). -[ RECORD 1 ] pid usesysid usename repuser application_name walreceiver client_addr client_hostname client_port backend_start 02-MAY-16 10:29: :00 backend_xmin state streaming sent_location 0/B Copyright 2016 EnterpriseDB Corporation. All rights reserved. 25

26 write_location 0/B flush_location 0/B replay_location 0/B sync_priority 0 sync_state async Copyright 2016 EnterpriseDB Corporation. All rights reserved. 26

27 4.2 Configuring the Standby (node 2) for Switchover 1. Promote the Standby (node 2) to Master by creating the trigger file referenced in its recovery.conf file. Use the following command: touch /tmp/node2 2. Connect to the Standby (node 2) to verify its recovery: ~]$ psql -U VibhorKumar -x -c "SELECT pg_is_in_recovery()" -p 5444 When the output is false or 'f', it means the Standby (node 2) is not in recovery and now the Master; it can receive read/write activity. -[ RECORD 1 ] pg_is_in_recovery f 3. Connect to the new Master (node 2) to verify replication between the new Master (node 2) and the old Master (node 1): [VibhorKumar@LDAPKerb ~]$ psql -U VibhorKumar -x -c "select * from pg_stat_replication" -p 5444 As the output shows, the old Master (node 1) is connected to the new Master (node 2) and streaming data. -[ RECORD 1 ] pid usesysid usename repuser application_name walreceiver client_addr client_hostname client_port backend_start 02-MAY-16 10:29: :00 backend_xmin state streaming sent_location 0/B0001A8 write_location 0/B0001A8 flush_location 0/B0001A8 replay_location 0/B0001A8 sync_priority 0 sync_state async Copyright 2016 EnterpriseDB Corporation. All rights reserved. 27

28 4. Connect to the old Master (node 1) to verify Standby (node 3) is streaming: ~]$ psql -U VibhorKumar -x -c "select * from pg_stat_replication" -p 5444 As the output shows the Standby (node 3) is still receiving data from the old Master (node 1): -[ RECORD 1 ] pid usesysid usename repuser application_name walreceiver client_addr client_hostname client_port backend_start 02-MAY-16 10:29: :00 backend_xmin state streaming sent_location 0/B0001A8 write_location 0/B0001A8 flush_location 0/B0001A8 replay_location 0/B0001A8 sync_priority 0 sync_state async 5. Drop the replication slot (node2) on the old Master (node 1) and recreate it: [VibhorKumar@LDAPKerb ~]$ psql -U VibhorKumar -c "select pg_drop_replication_slot('node2')" -p 5444 pg_drop_replication_slot (1 row) [VibhorKumar@LDAPKerb ~]$ psql -U VibhorKumar -c "select pg_create_physical_replication_slot('node2')" -p 5444 pg_create_physical_replication_slot (node2,) (1 row) Copyright 2016 EnterpriseDB Corporation. All rights reserved. 28

29 4.3 Configuring the Standby (node 3) for Switchover 1. Modify the recovery.conf file of the Standby (node 3) and change the following parameters: standby_mode = 'on' primary_conninfo = 'user=repuser password=repuser host= port=5444 sslmode=prefer sslcompression=1 krbsrvname=postgres' recovery_target_timeline = 'latest' primary_slot_name = 'node3' trigger_file = '/tmp/node3' restore_command = 'rsync -a /data/archived_wals/%f %p' Note that the host value for the primary_conninfo parameter belongs to the new Master s (node 2) IP address. 2. Restart the Standby (node 3), using either pg_ctl or the following service script: /usr/ppas-9.5/bin/pg_ctl --pgdata=/data/node --mode=fast restart [VibhorKumar@LDAPKerb ~]$ /usr/ppas-9.5/bin/pg_ctl -- pgdata=/data/node --mode=fast restart [VibhorKumar@LDAPKerb ~]$ :49:16 EDT LOG: redirecting log output to logging collector process :49:16 EDT HINT: Future log output will appear in directory "pg_log". 3. Connect to the new Master (node 2). Verify its streaming replication status using the following command: [VibhorKumar@LDAPKerb ~]$ psql -U VibhorKumar -x -c "select * from pg_stat_replication" -p 5444 As the output shows, the old Master (node 1) and the Standby (node 3) are now connected to new Master (node 2) and are both streaming data. -[ RECORD 1 ] pid usesysid usename repuser application_name walreceiver client_addr client_hostname client_port backend_start 02-MAY-16 10:47: :00 backend_xmin state streaming Copyright 2016 EnterpriseDB Corporation. All rights reserved. 29

30 sent_location 0/B write_location 0/B flush_location 0/B replay_location 0/B sync_priority 0 sync_state async -[ RECORD 2 ] pid usesysid usename repuser application_name walreceiver client_addr client_hostname client_port backend_start 02-MAY-16 10:49: :00 backend_xmin state streaming sent_location 0/B write_location 0/B flush_location 0/B replay_location 0/B sync_priority 0 sync_state async 4. Connect to the old Master and execute the following command to drop replication slot node3 and recreate it: [VibhorKumar@LDAPKerb pg_xlog]$ psql -U VibhorKumar -c "select pg_drop_replication_slot('node3')" -p 5444 pg_drop_replication_slot (1 row) [VibhorKumar@LDAPKerb pg_xlog]$ psql -U VibhorKumar -c "select pg_create_physical_replication_slot('node3')" -p 5444 pg_create_physical_replication_slot (node3,) (1 row) Copyright 2016 EnterpriseDB Corporation. All rights reserved. 30

31 5 Example: Performing a Switchback to a Standby (node 1) A switchback is a switchover (in the reverse order) that restores the database activity from the promoted Standby to the original Master server. Follow the steps in Section 5 to perform a switchback, returning a Master node to its original role within a replication scenario after a failover occurs. 5.1 Configuring the Master (node 2) for Switchback 1. Perform a clean shutdown on the Master (node 2); use pg_ctl or the following service script: /usr/ppas-9.5/bin/pg_ctl --pgdata=/data/node --mode=fast stop 2. Create a recovery.conf file in the Master data directory that includes the following parameters: standby_mode = 'on' primary_conninfo = 'user=repuser password=repuser host= port=5444 sslmode=prefer sslcompression=1 krbsrvname=postgres' recovery_target_timeline = 'latest' primary_slot_name = 'node1' trigger_file = '/tmp/node1' restore_command = 'rsync -a /data/archived_wals/%f %p' Please note that the value set for primary_conninfo is the IP address of the Standby (node 1). The recovery.conf file ensures that the node2 will start in recovery mode and will not accept any transactions that write to the database. 3. Use the following pg_ctl command to start the Master (node 2) : [VibhorKumar@LDAPKerb ~]$ /usr/ppas-9.5/bin/pg_ctl -- pgdata=/data/node start [VibhorKumar@LDAPKerb ~]$ :27:11 EDT LOG: redirecting log output to logging collector process :27:11 EDT HINT: Future log output will appear in directory "pg_log". At this point, all three nodes are running in recovery mode; none of the nodes will accept a transaction that writes to the database. Copyright 2016 EnterpriseDB Corporation. All rights reserved. 31

32 4. Connect to the Master (node 2) to verify the replication status: Configuring Switchover ~]$ psql -U VibhorKumar -x -c "select * from pg_stat_replication" -p 5444 As the output shows, the Standby (node 1) and the Standby (node 3) are streaming WAL segments from the Master, and are up-to-date. -[ RECORD 1 ] pid usesysid usename repuser application_name walreceiver client_addr client_hostname client_port backend_start 02-MAY-16 10:29: :00 backend_xmin state streaming sent_location 0/B write_location 0/B flush_location 0/B replay_location 0/B sync_priority 0 sync_state async -[ RECORD 2 ] pid usesysid usename repuser application_name walreceiver client_addr client_hostname client_port backend_start 02-MAY-16 10:29: :00 backend_xmin state streaming sent_location 0/B write_location 0/B flush_location 0/B replay_location 0/B sync_priority 0 sync_state async 5. Connect to the Standby (node 1) to verify its replication status: [VibhorKumar@LDAPKerb ~]$ psql -U VibhorKumar -x -c "select * from pg_stat_replication" -p 5444 As the output shows, the prior Master (node 2) is now behaving as a Standby for the Standby (node 1). Copyright 2016 EnterpriseDB Corporation. All rights reserved. 32

33 -[ RECORD 1 ] pid usesysid usename repuser application_name walreceiver client_addr client_hostname client_port backend_start 02-MAY-16 10:29: :00 backend_xmin state streaming sent_location 0/B write_location 0/B flush_location 0/B replay_location 0/B sync_priority 0 sync_state async Copyright 2016 EnterpriseDB Corporation. All rights reserved. 33

34 5.2 Configuring the Standby (node 1) for Switchback 1. Promote the Standby (node 1) to Master by creating the trigger file referenced in its recovery.conf file. Use the following command: touch /tmp/node1 2. Connect to the Standby (node 1) to verify its recovery: ~]$ psql -U VibhorKumar -x -c "SELECT pg_is_in_recovery()" -p 5444 When the output is false or 'f', it means the Standby (node 1) is not in recovery and now the Master can receive read/write activity. -[ RECORD 1 ] pg_is_in_recovery f 3. Connect to the new Master (node 1) to verify replication between the new Master (node 1) and the old Master (node 2): [VibhorKumar@LDAPKerb ~]$ psql -U VibhorKumar -x -c "select * from pg_stat_replication" -p 5444 As the output shows, the old Master (node 1) is connected to the new Master (node 1) and streaming data. -[ RECORD 1 ] pid usesysid usename repuser application_name walreceiver client_addr client_hostname client_port backend_start 02-MAY-16 10:29: :00 backend_xmin state streaming sent_location 0/B0001A9 write_location 0/B0001A9 flush_location 0/B0001A9 replay_location 0/B0001A9 sync_priority 0 sync_state async 4. Connect to the old Master (node 2) to verify Standby (node 3) is streaming: [VibhorKumar@LDAPKerb ~]$ psql -U VibhorKumar -x -c "select * from pg_stat_replication" -p 5444 Copyright 2016 EnterpriseDB Corporation. All rights reserved. 34

35 As the output shows the Standby (node 3) is still receiving data from the old Master (node 1): -[ RECORD 1 ] pid usesysid usename repuser application_name walreceiver client_addr client_hostname client_port backend_start 02-MAY-16 10:29: :00 backend_xmin state streaming sent_location 0/B0001A9 write_location 0/B0001A9 flush_location 0/B0001A9 replay_location 0/B0001A9 sync_priority 0 sync_state async 5. Drop the replication slot (node1) on the old Master (node 2) and recreate it: [VibhorKumar@LDAPKerb ~]$ psql -U VibhorKumar -c "select pg_drop_replication_slot('node1')" -p 5444 pg_drop_replication_slot (1 row) [VibhorKumar@LDAPKerb ~]$ psql -U VibhorKumar -c "select pg_create_physical_replication_slot('node1')" -p 5444 pg_create_physical_replication_slot (node1,) (1 row) Copyright 2016 EnterpriseDB Corporation. All rights reserved. 35

36 5.3 Configuring the Standby (node 3) after switchback 1. Modify the recovery.conf file of the Standby (node 3), updating the following parameters: standby_mode = 'on' primary_conninfo = 'user=repuser password=repuser host= port=5444 sslmode=prefer sslcompression=1 krbsrvname=postgres' recovery_target_timeline = 'latest' primary_slot_name = 'node3' trigger_file = '/tmp/node3' restore_command = 'rsync -a /data/archived_wals/%f %p' Note that the host value for the primary_conninfo parameter belongs to the new Master s (node 1) IP address. 2. Restart the Standby (node 3), using either pg_ctl or the following service script: /usr/ppas-9.5/bin/pg_ctl --pgdata=/data/node --mode=fast restart [VibhorKumar@LDAPKerb ~]$ /usr/ppas-9.5/bin/pg_ctl -- pgdata=/data/node --mode=fast restart [VibhorKumar@LDAPKerb ~]$ :49:16 EDT LOG: redirecting log output to logging collector process :49:16 EDT HINT: Future log output will appear in directory "pg_log". 3. Connect to the new Master (node 1), and verify its status with the following command: [VibhorKumar@LDAPKerb ~]$ psql -U VibhorKumar -x -c "select * from pg_stat_replication" -p 5444 As the output shows, the old Master (node 2) and the Standby (node 3) are now connected to new Master (node 1) and are both streaming data: -[ RECORD 1 ] pid usesysid usename repuser application_name walreceiver client_addr client_hostname client_port backend_start 02-MAY-16 10:47: :00 backend_xmin state streaming Copyright 2016 EnterpriseDB Corporation. All rights reserved. 36

37 sent_location 0/B write_location 0/B flush_location 0/B replay_location 0/B sync_priority 0 sync_state async -[ RECORD 2 ] pid usesysid usename repuser application_name walreceiver client_addr client_hostname client_port backend_start 02-MAY-16 10:49: :00 backend_xmin state streaming sent_location 0/B write_location 0/B flush_location 0/B replay_location 0/B sync_priority 0 sync_state async 4. Connect to the old Master and execute the following command to drop replication slot node3 and recreate it: [VibhorKumar@LDAPKerb pg_xlog]$ psql -U VibhorKumar -c "select pg_drop_replication_slot('node3')" -p 5444 pg_drop_replication_slot (1 row) [VibhorKumar@LDAPKerb pg_xlog]$ psql -U VibhorKumar -c "select pg_create_physical_replication_slot('node3')" -p 5444 pg_create_physical_replication_slot (node3,) (1 row) Copyright 2016 EnterpriseDB Corporation. All rights reserved. 37

38 6 Appendix This section lists the postgresql.conf parameters used in this guide along with a number of key attributes. These attributes are described as follows: Parameter. Configuration parameter name. Default Value. The setting assigned to the parameter if no other value is set. Range. The range of values accepted by the parameter. Minimum Scope of Effect. Scope of effect of the configuration parameter setting. Cluster Setting affects the entire database cluster (that is, all databases managed by the database server instance). Database Setting can vary by database and is established when the database is created. Applies to a small number of parameters related to locale settings. Session Setting can vary down to the granularity of individual sessions. In other words, different settings can be made for the following entities whereby the latter settings in this list override prior ones: a) the entire database cluster, b) specific databases in the database cluster, c) specific roles, d) specific roles when connected to specific databases, e) a specific session. When Value Changes Take Effect. When a changed parameter setting takes effect. Preset Established when the Advanced Server product is built or a particular database is created. This is a read-only parameter and cannot be changed. Restart Database server must be restarted. Reload Configuration file must be reloaded (or the database server can be restarted). Immediate Immediately effective in a session if the PGOPTIONS environment variable or the SET command is used to change the setting in the current session. Effective in new sessions if ALTER DATABASE, ALTER ROLE, or ALTER ROLE IN DATABASE commands are used to change the setting. Required Authorization to Activate. Type of operating system account or database role that must be used to put the parameter setting into effect. PPAS service account EDB Postgres Advanced Server service account (enterprisedb for an installation compatible with Oracle databases, postgres for a PostgreSQL compatible mode installation). Superuser Database role with superuser privileges. User Any database role with permissions on the affected database object (the database or role to be altered with the ALTER command). n/a Parameter setting cannot be changed by any user. Description. Brief description of the configuration parameter. For more information about the following parameters, see the PostgreSQL Core documentation, available at: Copyright 2016 EnterpriseDB Corporation. All rights reserved. 38

39 6.1 archive_command Parameter Type: String Default Value: Empty String Minimum Scope of Effect: Cluster When Value Changes Take Effect: Reload Required Authorization to Activate: PPAS service account Description: Sets the shell command that will be called to archive a WAL file. 6.2 archive_mode Parameter Type: Enum Default Value: off Range: off, on, or always off: This value disables archiving completed WAL segments. on: This value enables archiving completed WAL segments using archive_command. always: There is no significant difference between the on and always values, except that the WAL archiver is enabled during both archive recovery and standby modes when the value is set to always. Minimum Scope of Effect: Cluster When Value Changes Take Effect: Restart Required Authorization to Activate: PPAS service account Description: Allows archiving of WAL files using archive_command. Copyright 2016 EnterpriseDB Corporation. All rights reserved. 39