Setting up irods High Availability using Pgpool in NorStore

Transcription

1 Setting up irods High Availability using Pgpool in NorStore This is a slightly modified version of: irods_high_avaliability Note: I call Server1 throughout the document the icat enabled, irods server running the PgPool service. This is also the master icat host for the whole zone. Suggestion for the name of the zone: NorStore Important to note that if each node uses a different username for the irods admin user (which they probably should), those usernames (and passwords) needs to be created using iadmin on the master node (Server1). I also assume that the unix username running the irods server is irods (both on the master and slave nodes, not mandatory), not to be confused with the irods admin above. I.Setting up the master Node (Server1) 1. Standard irods installation using irodssetup, below are the important configuration parameters: Zone = NorStore (this must be the same on all the slaves) Postgres Port = 5431 (notice that is not the default 5432 ) Database User = irods (I have set the same username on the slave nodes as well) irods Installation Directory = <irods_home> (full path to the irods install dir, must not be necessarily the same on all the nodes) Postgres Directory = <PSQL_HOME> (full path to the postgresql install dir, can be different on the nodes, but the online recovery scripts from the site above, assume that they are the same ) 2. Some postgresql configuration needs to be done next: 2.1 in <PSQL_HOME>/pgsql/data/postgresql.conf, uncomment and set the line as below: listen_addresses = '*' 2.2 modify <PSQL_HOME>pgsql/data/pg_hba.conf, we need to set all md5 to trust (MD5 authentication is not supported in pgpool replication mode) and also add one line for each slave node as below: host ICAT irods <node_host_ip_address>/32 trust

2 3. Take a backup of the ICAT database and then drop it, commands: pg_dump ICAT > ICAT_DUMP dropdb ICAT 4. Stop the irods server. 5. Install PgPool mkdir <PGPOOL_INSTALL_DIR> chown -R irods:irods <PGPOOL_INSTALL_DIR> su - irods cd <PGPOOL_INSTALL_DIR> mkdir pgpool mkdir logs tar -xvf pgpool-ii-xx.tar.gz cd pgpool-ii-xx./configure --prefix= <PGPOOL_INSTALL_DIR> --with-pgsql= <PSQL_HOME>/pgsql make make install 6. Configure PgPool Create the 3 configuration files from the templates in <PGPOOL_INSTALL_DIR>/pgpool: cd <PGPOOL_INSTALL_DIR>/pgpool/etc cp pcp.conf.sample pcp.conf cp pgpool.conf.sample pgpool.conf cp pool_hba.conf.sample pool_hba.conf Changes: in pcp.conf add: irods:f0cedadasdasdasdad3cf434488eec5 here the string after : is the md5 hash of the irods database user password, obtained using the pg_md5 pgtool utility.

3 in pool_hba.conf add the following line for each node: host ICAT irods <node_host_ip_address>/32 trust in pgpool.conf (main configuration), the following entries need to be modified listen_addresses = '*' ----Listen to all Network cards port = 5432 pcp_port = 9898 #Pgpool Port #Pgpool Control(admin) Port logdir = '<PGPOOL_INSTALL_DIR>/ logs' pid_file_name = '/<PGPOOL_INSTALL_DIR>/logs/pgpool.pid' replication_mode = true load_balance_mode = true insert_lock = false backend_hostname0 = 'localhost backend_port0 = 5431 backend_weight0 = 1 In addition the following lines need to be added for each slave node (increment the index accordingly): backend_hostname1 = 'Hostname of ip address of the slave node backend_port1 = 5432 backend_weight1 = 1 7. Start PgPool Make sure that the postgresql server is running on each node, then run: pgpool -d -n 2>&1 > <PGPOOL_INSTALL_DIR>/ logs/pgpool.log see the above webpage for setting up logrotation using cronolog. 8. Recreate the ICAT database on Server1 from the dump we created earlier createdb p 5432 ICAT psql ICAT < ICAT_DUMP this should replicate the database on all the slave nodes. 9. Before starting the irods server, change: port = 5432 in <PSQL_HOME>/pgsql/etc/odbc.ini

4 Port = 5432 in ~irods/.odbc.ini 10. Start the irods server and create the administrator users for each slave irods server (if they differ from the irods admin on Server1). II.Setting up a slave Node (Server2) 1. Default irods installation using irodssetup, below are the important configuration parameters: Zone = norstore-test (this must be the same on all the slaves) Postgres Port = 5432 (default ) Database User = irods irods Installation Directory = <irods_home> (full path to the irods install dir, must not be necessarily the same on all the nodes) Postgres Directory = <PSQL_HOME> (preferably the same as on the master node) 2. Stop the irods server irodsctl istop 3. Drop ICAT database using dropdb ICAT and stop postgresql irodsctl dbstop 4. Some postgresql configuration needs to be done next: 4.1 in <PSQL_HOME>/pgsql/data/postgresql.conf, uncomment and set the line as below: listen_addresses = '*' 4.2 modify <PSQL_HOME>pgsql/data/pg_hba.conf, we need to set all md5 to trust (MD5 authentication is not supported in pgpool replication mode) and also add the line to grant access from the master node as below: host ICAT irods /32 trust 5. Start Postgres with

5 irodsctl dbstart 6. Edit file <irods_home>/server/config/server.config as icathost slaveicathost localhost 7. Modify the file <irods_home>/server/config/irodshost if there are any special network addresses configured for the irods host. 8. Start irods irodsctl istart 9. Notify the master node administrator to create your irods administrator username/ password. 10. Create any users, resources as normally using the iadmin command. III.Setting up online recovery 1. Setup ssh access using public keys among all the nodes. Use: ssh-keygen -t dsa (as irods) then the public key (typically in ~irods/.ssh/id_dsa.pub) should be added to ~irods/.ssh/ authorized_keys on all the nodes. 2. Enable WAL archiving We will use the directory <PSQL_HOME> section I above in the following. Create a directory pgsql_wal_archive as: mkdir <PSQL_HOME>/pgsql_wal_archive Enable WAL archiving in postgresql, edit the file <PSQL_HOME>/pgsql/data/ postgresql.conf archive_mode = on archive_command = 'test! -f <PSQL_HOME>/backup_in_progress rsync -a %p <PSQL_HOME>/pgsql_wal_archive/%f' archive_timeout = 300 wal_level = archive 3. Restart postgresql: <irods_home>/irodsctl dbrestart (node should be detached from pgpool on the master before this). 4. Setup template1 function

6 For this download and unpack the file pgpool-ii tar.gz from download.php?f=pgpool-ii tar.gz. tar zxvf pgpool-ii tar.gz cd pgpool-ii-3.1.3/sql/pgpool-recovery/ Edit Makefile (change the <PSQL_HOME> value for your setup, and comment out USE_PGXS = true as below): MODULES = pgpool-recovery DATA_built = pgpool-recovery.sql DATA = uninstall_pgpool-recovery.sql # if you are using PostgreSQL 8.0 or later, # using pg_config is recommended. # if you are not, comment out following line and... #USE_PGXS = true # set top_builddir to the PostgreSQL build source tree top. # (for example /usr/local/src/postgresql-8.4) top_builddir = <PSQL_HOME>/postgresql-9.0.3/ ifdef USE_PGXS PGXS = $(shell /home/irods/psql/pgsql/bin/pg_config --pgxs) include $(PGXS) else subdir = contrib/pgpool-recovery #top_builddir =../.. include $(top_builddir)/src/makefile.global include $(top_srcdir)/contrib/contrib-global.mk endif then do make install <PSQL_HOME>/pgsql/bin/psql -p f pgpool-recovery.sql template1 irodsctl dbstop 5. Edit pgpool.conf file (should be done only on the master node) and add the following: health_check_period = 30 health_check_timeout = 60 failover_command = failback_command = backend_data_directory1 = '<PSQL_HOME1>/pgsql/data' backend_data_directory2 = '<PSQL_HOME2>/pgsql/data' recovery_user = 'irods' recovery_password = '****'

7 recovery_1st_stage_command = 'copy_base_backup' recovery_2nd_stage_command = 'pgpool_recovery_pitr' 6. Copy copy_base_backup, pgpool_recovery_pitr, pgpool_remote_start to <PSQL_HOME>/pgsql/data on each node. These files are listed below, the entries with <> should be replaced by the corresponding paths on each node. cat copy_base_backup!#!/bin/sh DATA=$1 RECOVERY_TARGET=$2 RECOVERY_DATA=$3 WAL_ARCHIVE= <PSQL_HOME>/pgsql_wal_archive echo "$DATA $RECOVERY_TARGET $RECOVERY_DATA " > /tmp/recovery.log rm -f $WAL_ARCHIVE/* ssh -T $RECOVERY_TARGET "find "${RECOVERY_DATA}"/pg_xlog/ -type f -exec rm -f {} \;" >> /tmp/recovery.log 2>&1 touch <PSQL_HOME>/backup_in_progress RECOVERY_LABEL=`date "+ %Y-%m-%d_%H:%M:%S"` psql -c "select pg_start_backup('${recovery_label}')" postgres >> /tmp/recovery.log 2>&1 echo "restore_command = 'rsync -pog $HOSTNAME:$WAL_ARCHIVE/%f %p'" > /tmp/ recovery.conf rsync -avh --delete -e ssh --exclude backup_label --exclude postgresql.conf --exclude pg_hba.config --exclude pg_xlog --exclude recovery.conf --exclude postmaster.pid -- exclude postmaster.opts $DATA/ $RECOVERY_TARGET:$RECOVERY_DATA/ >> /tmp/ recovery.log 2>&1 psql -c "select pg_stop_backup()" postgres >> /tmp/recovery.log 2>&1 scp /tmp/recovery.conf $RECOVERY_TARGET:$RECOVERY_DATA/. >> /tmp/recovery.log 2>&1 rm -f /tmp/recovery.conf exit 0 cat pgpool_recovery_pitr

8 #! /bin/sh psql -c 'select pg_switch_xlog()' postgres cat pgpool_remote_start #! /bin/sh if [ $# -ne 2 ] then echo "pgpool_remote_start remote_host remote_datadir" exit 1 fi DEST=$1 DESTDIR=$2 echo "$DEST $DESTDIR" >> /tmp/recovery.log PSQL_HOME= <PSQL_HOME> irodsctl=irodsctl rm -f $PSQL_HOME/backup_in_progress >> /tmp/recovery.log 2>&1 ssh -T $DEST $irodsctl dbstart >> /tmp/recovery.log 2>&1 7. chmod +x copy_base_backup pgpool_recovery_pitr pgpool_remote_start 8. Modify $PATH on each node to include the location of <irods_home>, we will use the irodsctl script to start icat remotely.