Stand-by server iprotect 8.03

Transcription

1 Stand-by server iprotect 8.03 o Functional Description o Implementation Guide lines o Cold- and Warm stand-by o Automatic fail-over

2 Publication May, 2016 Keyprocessor BV Paasheuvelweg BJ Amsterdam Tel.: This manual represents the knowledge at the above mentioned date. Keyprocessor works nonstop to improve her products. For the most recent technologic information please contact your consultant or dealer. Content overview Publication... 2 Content overview Introduction Stand-by functionality Stand-by configuration License Cold stand-by configuration Events Warm stand-by configuration Events Automatic fail-over configuartion Fail-over setups server setup server setup Configuration Primary server setup Backup server setup Witness server setup Important remarks Stopping iprotect Upgrading iprotect Using the proper IP address Fail back Lines Expected behavior... 12

3 1 Introduction This document describes the stand-by functionality of iprotect. The cold stand-by functionality is already available for many years in iprotect and from the iprotect 8.03 release also a warm stand-by functionality has been implemented. 1.1 Stand-by functionality Primary iprotect Server The primary iprotect server is the iprotect server that controls the hardware in the system and where the users log into. It is the operational system. Backup iprotect Server The backup iprotect server is a second iprotect server that is periodically updated with the same data as the primary iprotect server, but it does not communicate with the hardware in the system. Usually, the users do not log into this system, because they can t perform any actions. Cold Stand-by A Cold Stand-by server is a backup iprotect server that receives once every 24-hours the full database from the primary iprotect server. It immediately restores this database and has all the iprotect processes up and running. However there is no communication with the hardware and users. When the primary iprotect server fails an operator can activate the cold stand-by server and from that moment on this server starts the communication with the hardware. Because the server contains the data from the backup, this will become the actual state of the system. All changes between the moment of the backup and the moment of the switch over will be lost. The cold stand-by functionality and the installation instructions for this feature are described in a separate document: Stand-by server, Installation Instructions, version 1.2/1.4. Warm Stand-by In the warm stand-by configuration the backup iprotect server receives updates from the primary iprotect server at short intervals, typically every minute. It immediately restores these updates and has all the iprotect processes up and running, except for the GUI. Once every 24 hours the backup server receives a full backup of the primary iprotect database (as in the cold stand-by configuration). There is no communication with the hardware and users. When the primary iprotect server fails the warm stand-by server is automatically activated and from that moment on this server starts the communication with the hardware. Because the server contains the data from the backup and the intermediate updates, only a very limited amount of data will be lost, i.e. only the changes within the last minute before the failure. The automatic fail-over mechanism is described in chapter 3.

4 2 Stand-by configuration 2.1 License The Standby functionality becomes available with a special license: License number 5080: Standby Server If the value of this license is 1 the Cold Stand-by functionality is available, if the value is 2 the Warm stand-by functionality is available. Note! This license must be installed on both the primary and the backup server. 2.2 Cold stand-by configuration Note! The primary- and the standby server MUST be the same iprotect version. On the primary server we must define the ip-address and the atlas account of the backup server. This can be defined with the menu option: Installation Settings System parameters. On the tab Backup/Backup server the parameters can be filled in. 1. The Daily backup time defines the time at which the daily backup will be made. After this backup has been made the backup is also transferred to the backup server. 2. Server IP Address: This is the IP-address of the backup server 3. Password: This is the password of the administrative atlas account on the backup server. The backup server can be configured by starting a backup on the primary server via the maintenance page. When the installation on the backup server is completed the license file (with the standby server license) must be installed on the backup server. This is done via the Install Configuration Upload license in the maintenance page of the backup server. Then, a backup of the backup iprotect server must be made through the maintenance page. This stores the license file of the backup server in the /home/backup directory. After the backup server has been installed, the synchronization between the primary server can be started by making a backup on the primary server. A backup can be made through the maintenance page of the primary server (iprotect Backup General Start). The progress can be monitored via the last transaction form in iprotect. The state of the server can be verified by checking the Database status in the Installation Settings System parameters menu option. For the primary server this should be Operational and for the standby server this should be Standby. Also in the lower left corner of the user interface of the backup server the message Standby!!!! should be visible Events When the transfer of the full backup starts from the primary to the backup iprotect server, the

5 primary iprotect server logs an event: Start standby upgrade. When the backup server has successfully restored the backup, the event Standby upgrade done is logged with the message Cold standby dataset restore done. The backup server restarts the iprotect application after the backup from the primary server has been restored. 2.3 Warm stand-by configuration On the primary server we must define the ip-address and the atlas account of the backup server. This can be defined with the menu option: Installation Settings System parameters. On the tab Backup/Backup server the parameters can be filled in. 1. The Daily backup time defines the time at which the daily backup will be made. After this backup has been made the backup is also transferred to the backup server. 2. Server IP Address: This is the IP-address of the backup server 3. Password: This is the password of the administrative atlas account on the backup server. 4. On the tab Installation Settings System parameters Hardware the checkbox Deactivate lines on backup/restore should be UNCHECKED. Otherwise, the lines are not activated after the standby server becomes the primary server. On the backup iprotect server nothing has to be installed. The synchronization between the primary server and the backup server can be started by making a backup on the primary server. A backup can be made through the maintenance page of the primary server (iprotect Backup General Start). The progress can be monitored via the last transaction form in iprotect. After the transfer of the backup to the backup server, the license file (with the backup server license) must be installed on the backup server. This can be done via the upload license button in the maintenance page. On the backup server NO user interface is running. The user interface will be started when the primary server switches over to the backup server. To check if the warm backup process is running correctly, you can check the /home/backup/increment folder on the primary iprotect server. In this folder all changes in the database of the last minute are stored before they are transmitted to the backup server. These files should change every minute Events When the transfer of the full backup starts from the primary to the backup iprotect server, the primary iprotect server logs an event: Start standby upgrade. When the backup server has successfully restored the backup, the event Standby upgrade done is logged with the message Warm standby dataset restore done. The fail over of the iprotect system will be logged by the transaction: State change database => Server state: Operative Give this event priority hard if you want an alarm on failover. The backup server restarts the iprotect application after the backup from the primary serverhas been restored. After the increments are restored the backup server is not restarted.

6 3 Automatic fail-over configuartion For the warm standby functionality also the automatic fail-over functionality must be configured. This functionality makes sure that when the primary iprotect server fails, the backup server automatically becomes primary server. This fail-over mechanism uses an additional (virtual) IP-address. This extra IP-address will be used as the IP-address that is used in iprotect (for users and devices). So we are not using the real IP-address of the iprotect servers for this. When the fail-over takes place, the virtual IPaddress of the primary server is transferred to the backup server so all the external interfaces with the iprotect server remain operational and the users can use the same URL to log into the iprotect server after fail-over. Thus, we need at least 3 IP-addresses in the same 100Mbps LAN for this functionality: The real IP-address of the primary iprotect server ( in figure) The real IP-address of the backup server ( in figure) The virtual IP-address that will be used in the iprotect URL and by the devices. ( in figure) For this functionality we use the keep-alived package of Linux. The latest version of this package can be installed with the ipuntu-failover-keepalived taz package. An extra package with the fail-over configuration files (ipuntu-failover-config.taz) needs to be installed.

7 3.1 Fail-over setups iprotect support two different kind of fail-over setups: A 2-server solution, with the primary and the backup iprotect server. A 3-server solution, that has a separate witness server, besides the 2 iprotect servers server setup In the 2-server setup the backup server monitors the availability of the primary iprotect server. If the backup server loses contact with the primary iprotect server, it switches from standby mode to operational mode and becomes the new primary iprotect server. The fail-over can be caused by: A crash of the iprotect software on the primary server A crash of the primary server hardware Loss of network between the backup server and the primary server server setup In the 3-server setup, a third server is used. This is the so-called witness server. In case loss of network between the primary iprotect server and the backup server it is uncertain for the backup server if the primary server is still running. It could be the case that the network interface of the standby server is broken and that the iprotect system is functioning normally. This situation can be solved by the introduction of a witness server that checks the availability of both the primary and backup server. In case of conflicts between backup and primary server: for example the backup server thinks the primary server is down, but the primary server is up, the witness server decides what the correct situation is.

8 3.2 Configuration Primary server setup First we must define the virtual IP-address in the failover configuration files. Login to the primary iprotect server using putty. Go the the failover directory: cd /home/failover Edit both the keepalived.master.conf and the keepaliver.backup.conf file Change the following part of the file: virtual_ipaddress { /24 brd dev eth0 } Replace the first IP-address ( ) with the IP-address that will be used for the virtual IP-address. The second number (/24) is the type of network this IP-address is on (/24 for a class C network). The third number ( ) is the broadcast address on this network: For class C network the first 3 numbers are the same as the IP-address and the 4th number is 255. So if our virtual IP-address is on a class C network the line should be changed to: /24 brd dev eth0 Save the file and do the same for the keepaliver.backup.conf file. In the keepalived.master.conf and the keepalived.backup.conf files we also specify the conditions for the fail over. vrrp_script failover { script "/home/failover/failover.pl check" interval 15 fall 4 } This configuration defines that every 15 seconds the status of the system will be checked and that after 4 unsuccessful checks the failover shall occur. This means that a failover shall occur if there is no contact for more than 1 minute. In the Maintenance page, for the 2-server setup, only the Keepalived part needs to be configured. This functionality is configured in the Server Services Configuration area of the Maintenance page of the iprotect server. The Role of this server is Master, and with Activate set to Yes the failover functionality is started. The iprotect system is now available on the virtual IP-address that has been configured in the keepalived configuration files. Note! Always start the Keepalived service on the master first!! Otherwise the standby server will immediately become the master server. Note! When starting the keepalived service the automatic activation of iprotect after booting the server is disabled. This is to prevent problems with 2 running iprotect databases after the master server has been restarted.

9 3.2.2 Backup server setup First we must define the virtual IP-address in the failover configuration files. Login to the backup iprotect server using putty. Go the the failover directory: cd /home/failover Edit both the keepalived.master.conf and the keepaliver.backup.conf file. Change the following part of the file: virtual_ipaddress { /24 brd dev eth0 } Replace the first IP-address ( ) with the IP-address that will be used for the virtual IP-address. The second number (/24) is the type of network this IP-address is on (/24 for a class C network). The third number ( ) is the broadcast address on this network: For class C network the first 3 numbers are the same as the IP-address and the 4th number is 255. So if our virtual IP-address is on a class C network the line should be changed to: /24 brd dev eth0 Save the file and do the same for the keepaliver.backup.conf file. In the Maintenance page, for the 2-server setup, only the Keepalived part needs to be configured. This functionality is configured in the Server Services Configuration area of the Maintenance page of the iprotect server. The Role of this server is Backup, and with Activate set to Yes the failover functionality is started. The backup iprotect system can only be reached on the URL of the maintenance page with the real IP-address of the backup server. The iprotect GUI is NOT running! Witness server setup The witness server can be a small ipuntu server (4GB memory is more than enough). The maintenance page should be installed on this server but iprotect must not be running. It can be disabled permanently with the disable77 command. To configure the server as a witness server the Server->Services->Configuration area of the Maintenance page is used. Witness server For the witness server the Role should be set to Server. The Protocol that will be used to test if the master and backup server are running should be set to Pong, which is basically HTTPS. Furthermore, the real IP-address of the master and the backup server must be entered. If the IP-address of the witness server is the setup in the example from becomes: Role: Server Protocol: Pong IP-address Master: IP-address Backup: Master + Backup serverserver For the master and the backup server the Role should be set to Client. The IP-address of the witness server must be entered. Also the username and password of the maintenance page

10 of the witness server must be entered. So in our example: Role: Client IP-address witness: Username: atlas Password: kp4700

11 4 Important remarks The warm standby and failover features of the iprotect system cause the system to be have different from a normal iprotect system. In this chapter a number of important issues are addressed when dealing with a warm standby system. 4.1 Stopping iprotect Because we have a failover mechanism, we cannot simply stop the primary iprotect system when we want to stop iprotect. The backup iprotect server would take over the control of the lines. When you want to stop the iprotect system the following procedure should be followed: 1. Stop the keepalived process on the backup server by selecting activate NO in the maintenance box 2. Stop the keepalived process on the primary server by selecting activate NO in the maintenance box 3. Perform a kill77 on the primary server 4.2 Upgrading iprotect When upgrading the iprotect system to a version with the same major release number (no new license required), it is enough to upgrade only the primary iprotect server. The backup server will be upgraded automatically with the next full backup. When the system is upgraded to an iprotect version with a different major version number (new license is required), the new license file for the backup server must be uploaded via the maintenance page of the backup server. 4.3 Using the proper IP address In the warm standby setup the iprotect system uses 3 or 4 IP-addresses in the same network. It is important to always use the correct IP-address. The users of the iprotect system should use the Virtual IP-address, so that when the system switches from primary to standby server they still can use the same URL. External interfaces to the database should use the Virtual IP-address, so that when the system switches from primary to standby server it can connect to the same URL. Maintenance on the primary and backup server via the maintenance page or puttyshould use the Real IP-address of the server. To list all the IP-adresses from an interface (eth0), including the virtual IP-address you can use the command: ip addr show eth0. You will get something like: 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 08:00:27:83:49:e2 brd ff:ff:ff:ff:ff:ff inet /24 brd scope global eth0 inet /24 brd scope global secondary eth0 inet6 fe80::a00:27ff:fe83:49e2/64 scope link valid_lft forever preferred_lft forever

12 This shows that is present as the secondary (virtual ) IP-address. If necessary you can remove this secondary address by: sudo ip addr del /24 dev eth0 4.4 Fail back The fail-over mechanism is a one-time mechanism: after a fail-over has occurred the system needs to be reconfigured again to activate this mechanism again. To enable the warm standby again it is best to make the old primary server the backup of the old backup server (now primary). The procedure for this is: 1. Deactivate the keepalived service on the old primary server. 2. Deactivate the keepalived service on the new primary server. 3. A warm standby license (5080) should be present on the new primary server. 4. Add the old primary server as the backup server in iprotect. So in our example: should be filled in as the Backup server in the Backup tab of Installation->Settings->System parameters. 5. Make a backup on the new primary server. This is automatically transferred to the new backup server which becomes the warm standby server. 6. Switch the master and backup IP address in the maintenance page of the witness server. 7. Activate the new primary server as master 8. Activate the new backup server as backup. 4.5 Lines In iprotect the lines with underlying nodes automatically fail-over to the new primary iprotect server. Because the keystores are also transferred with the database, they can continue to communicate securely with the iprotect system. If the lines are connected via a separate network interface on the iprotect server (technical network), no special fail-over configuration is required for this interface. However, it is required that the backup server also has a second network interface in the same technical network. 4.6 Expected behavior With warm standby many different scenarios can occur, and it can be difficult to predict the behavior of the system. Here we describe a number of likely scenario s and the response of the system. iprotect process crashes When one of the iprotect processes crashes (kp77db, kp77ln, kp77usr, kp77trans) keepalived will detect this and the primary system will go into FAULT status. The other iprotect processes on this system will be terminated. This is detected by the standby system and this system will go into MASTER state and iprotect on this system will go from standby to operational. The virtual IP address of the system will go over from the primary server to the standby server. The old primary server will go into BACKUP state. If the old primary server is rebooted, or if by accident the iprotect software is started again by start77, the keepalived software will prevent it from becoming operational.

13 Primary server stops When the primary server stops the keep-alived process on the standby server will detect this and this system will go into MASTER state and iprotect on this system will go from standby to operational. The virtual IP address of the system will go over from the primary server to the standby server. If the old primary server is started again the keep-alived software will prevent it from becoming operational. Primary server freezes temporarily When the primary server freezes temporarily (i.e. due to a snapshot on the Hyper-V server) it will not respond in time to the standby server. This is detected by the standby system and this system will go into MASTER state and iprotect on this system will go from standby to operational. The virtual IP address of the system will go over from the primary server to the standby server. If the primary server then unfreezes it is still in MASTER state, but it will detect that there is another master server in the network and it will go to BACKUP state and deactivate the iprotect processes. The network between primary and standby is interrupted When the network between the primary and standby server is interrupted, it will not respond in time to the standby server. This is detected by the standby system. When there is a witness server in the system the standby system will query the witness whether it sees the master serv. If the witness sees the master, nothing will happen. If the witness server also cannot see the master the standby server will go into MASTER state and iprotect on this system will go from standby to operational. The virtual IP address of the system will go over from the primary server to the standby server. If the network connection between primary and standby is restored again, the primary server is still in MASTER state, but it will detect that there is another master server in the network and it will go to BACKUP state and deactivate the iprotect processes.