APM High Availability

Transcription

1 APM High Availability Version 1.0 Written by Ben Stern Executive IT Specialist: Application Performance Management Best Practices

2 Contents Contents... 2 Overview... 4 Set up DB Prepare your firewall scripts... 9 Create your firewall scripts... 9 enableprimary.sh... 9 enablestandby.sh Test your firewall scripts Set up a reverse proxy httpd.conf file Set up a forwarding proxy httpd.conf file Install your Performance Management servers Install a standby Performance Management server Install a primary Performance Management server Install your monitoring agents Configure and test failover Set up customizations and synchronize the content Understand what data is not backed up Test failover and failback Automate the synchronization and failover processes Complete optional configuration tasks Configure HTTPS Traffic from the agents... 31

3 Synchronize the Performance Management Servers using backup and restore 31 Define a backup schedule Appendix A: Setting up high availability with DB2 HADR Cluster Setting up the DB2 HADR Cluster Setting up the Performance Management database in DB2 HADR Primary node setup Standby DB2 node setup Primary DB2 node setup Appendix B: Prompts for installing a standby Performance Management server Appendix C: Prompts for installing a primary Performance Management server Appendix D: Dropping databases in a DB2 HADR Environment Appendix E: Setting up Performance Management without a VIP Reconfigure kafka to use the database cluster Reconfigure the XML toolkit to use the DB2 HADR environment Reconfigure SCR to use the DB2 HADR environment Modify the summarization & pruning to use the DB2 HADR environment Notices... 63

4 Overview This document describes how to set up a high availability configuration for IBM Performance Management version 8.x. The instructions are written for Performance Management version 8.1.3, but also apply to subsequent minor releases. The introduction provides an overview of the deployment and describes at a high level how failover is achieved. Detailed instructions are provided in the subsequent sections of this document. The high-level steps are as follows: Set up a remote DB2 server. The server can either be set up as a stand-alone DB2 server or as a DB2 HADR clustered environment. The remote DB2 server is used as the backend database for your Performance Management environment. DB2 HADR is the mechanism that is used to achieve high availability within DB2. Prepare some scripts to set up a firewall on the database servers so that only one of the two Performance Management servers can communicate with the databases. During the installation and execution of your Performance Management environment, configure the firewall so that the active Performance Management server can communicate with DB2, but the other Performance Management server is blocked. Set up two Performance Management servers. Each of the Performance Management servers is configured to communicate with the same DB2 server or DB2 cluster. o You initially set up the standby Performance Management server using the shared DB2 server or cluster. o After you install the standby Performance Management server, drop and re-create the DB2 database by using the steps that are documented for a remote DB2 server. o Install the primary Performance Management server using the shared DB2 server or cluster. o Both Performance Management servers are active and running, but you must block communications from one of the two Performance Management servers to the DB2 server. During normal operations, traffic from the standby Performance Management server to the DB2 servers is blocked. During failover operations, traffic from the primary Performance Management server to the DB2 servers is blocked. This setup ensures that only one of the two Performance Management servers is writing to the database.

5 Set up a reverse proxy so that all user interface (UI) communications is sent through a reverse proxy to the currently active Performance Management server. Set up a forwarding proxy so that all agent traffic is sent through a forwarding proxy to the currently active Performance Management server. Periodically, run the backup.sh script on the primary Performance Management server and restore the backed-up configuration to the standby Performance Management server using the restore.sh script. Back up the database on the database server by following best practices for your database configuration. These processes differ depending on whether you are using a single-server database, an HADR database with replication, or a database cluster. During a failover, complete these steps: o First, reconfigure the firewall on the DB2 server to block traffic from the primary Performance Management and enable traffic from the standby Performance Management server. This step allows the standby Performance Management server to be fully functional. o Second, reconfigure the reverse proxy so that UI traffic is routed to the standby Performance Management server. o Finally, reconfigure the forwarding proxy to route agent traffic to the standby Performance Management server. When you patch the environment, patch one of the two Performance Management servers at a time and use the failover steps to ensure that only one Performance Management server can communicate with the DB2 server. To ensure that the two servers are in sync in case there is a failover, follow the recommended set of steps and schedules that are provided in this document for backing up the configuration and event content on the primary Performance Management server and restoring the content to the standby Performance Management server. Depending on your preferred method of DB2 high availability, go to the appropriate section of the document for detailed instructions. Most of the documentation references the DB2 server. Information about DB2 HADR configurations is included in appendices A, D, and E. Appendix A: Setting up high availability with DB2 HADR Cluster: This appendix documents how to set up the Performance Management database in a DB2 HADR configuration. It does not include information on setting up a VIP.

6 Appendix D: Dropping databases in a DB2 HADR Environment: This appendix gives detailed information about dropping databases in a DB2 HADR environment. If you follow the recommendations in this document, you should not need to use this section. Appendix E: Setting up Performance Management without a VIP: This appendix details how to configure the Performance Management components so that they use a primary and standby database. The failover processes outlined in this document include manual steps, but it is possible to detect failures and automate these steps. Figure 1 shows how the components communicate with each other. In a single node DB2 deployment, the DB2 server replaces the DB2 virtual IP address (VIP) component. Figure 1 Deployment Diagram

7 Set up DB2 Set up DB2 based on the level of high availability you require. DB2 can be set up as a single node or it can be set up in a DB2 HADR cluster. If you choose to set up DB2 as a single node configuration, you experience a small amount of downtime as you set up a new DB2 server and restore the backed-up database. If you set up a DB2 HADR cluster, you will experience no downtime during many failure conditions. IMPORTANT: The IBM Performance Management software requires DB2 Advanced Enterprise Edition 10.5 or higher. There are features that are only available in the Advanced edition that are used by the Performance Management software. When you set up the DB2 server, use the DB2 Advanced Enterprise edition. When you install DB2, the following accounts are set up on the DB2 server. Later, you will need to know and use the passwords for these accounts. These user names are hardcoded in the IBM Performance Management installer, so do not attempt to change the names. Technically, it is possible to use a different instance owner name on the Performance Management server than on the database server, but it adds much complexity to the installation process. It is highly recommended that you set up your environment exactly as is described here. Set up DB2 with db2apm as the database instance. The Linux account db2apm is the instance owner. The DB2 fenced user is db2fenc1. The DB2 Administration Server (DAS) user is dasusr1. In addition, set up an itmuser OS account on the database server. This account is used to access the data in the database. When you complete the setup, you will have these 4 Linux OS accounts on the database server: db2apm db2fenc1 dasusr1 itmuser Make note of the passwords for these four accounts. They are used during the installation of the Performance Management server. With DB2, a good backup and restore strategy is important to ensure that the data can be restored in case any software or hardware failures occur. Typically, this includes setting up periodic full backups and incremental backups.

8 Set up the Performance Management databases by following the instructions in the Performance Management Knowledge Center for setting up a remote DB2 database: stall_server_mydb2.htm If you are using a DB2 HADR cluster or DB2 HADR cluster with VIP, set up the three databases on the primary DB2 node and then follow normal practices for configuring the databases in a cluster. After you follow the instructions, three databases are created and tuned with some of the schemas populated. The three databases are: WAREHOUS SCR32 DATAMART You can confirm that the databases are created by entering the following command as the db2apm user or another user with database privileges: db2 list database directory The command lists the three databases and their home directories. DB2 HADR clustering is normally set up after the databases are created. Detailed information on setting up a DB2 HADR Cluster is in Appendix A: Setting up high availability with DB2 HADR Cluster and you can find more instructions online. If you are setting up a standard high availability DB2 HADR clustered environment without a VIP, complete the entire IBM Performance Management server installation by using the primary database node. Then, configure the HADR cluster after all the software is installed. This is easier than setting up DB2 HADR and then installing the software. However, if you are using a DB2 HADR VIP, the VIP must be set up before the Performance Management software is installed. Otherwise, the Performance Management software is configured with the wrong host name for the DB2 server.

9 Prepare your firewall scripts In this high availability configuration, you want only one of the two Performance Management servers to communicate with the DB2 database. Create your firewall scripts Create two scripts and place them on the DB2 servers. Run the appropriate script that allows traffic on the database port from the chosen Performance Management server. In a typical DB2 environment, the database is listening on port 50,000. The sample scripts that are provided assume that the port is the default. If your database is set up for a different port, change the port number in the scripts. In addition, the IP addresses listed in the scripts are based on the IP addresses in the deployment diagram. Replace the IP addresses with the addresses of your database servers and Performance Management servers. The first script is called enableprimary.sh. This script blocks traffic to the standby Performance Management server and allows traffic to the primary Performance Management server. In addition, in a clustered environment, it allows for communication between the two nodes in the database cluster. You can configure your firewall in other ways, but the script that is provided works. If you have a single database node, put one entry in the database section with the IP address of the database server. #!/bin/bash enableprimary.sh # Add port to the firewall configuration firewall-cmd --zone=internal --add-port=50000/tcp # Block the standby Performance Management server firewall-cmd --zone=internal --remove-source= /24 # Allow traffic from the primary Performance Management server firewall-cmd --zone=internal --add-source= /24 # Allow traffic between the two database nodes and VIP firewall-cmd --zone=internal --add-source= /24 firewall-cmd --zone=internal --add-source= /24

10 firewall-cmd --zone=internal --add-source= /24 The second script is enablestandby.sh. This script allows traffic between the database servers and the standby Performance Management server and blocks traffic to the primary Performance Management server. #!/bin/bash enablestandby.sh # Add port to the firewall configuration firewall-cmd --zone=internal --add-port=50000/tcp # Block the primary Performance Management server firewall-cmd --zone=internal --remove-source= /24 # Allow traffic from the standby Performance Management server firewall-cmd --zone=internal --add-source= /24 # Allow traffic between the two database nodes and VIP firewall-cmd --zone=internal --add-source= /24 firewall-cmd --zone=internal --add-source= /24 firewall-cmd --zone=internal --add-source= /24 Test your firewall scripts Before you move to the next step in your setup, test to make sure that the scripts are working properly. This test can easily be done by using the telnet command. If telnet is not installed on your systems, it can be installed by using yum install telnet or you can test the connectivity by using the tool of your choice. First, run the enableprimary.sh script. On the primary Performance Management server, run the following command, specifying the host name or IP address of your database server. For an HADR cluster, use the VIP. Use the port number of your database server.

11 telnet <IP address of database server> <port number of database server> Example: telnet If your firewall is configured correctly, the primary Performance Management server can connect to the database server and you see a message like the following example: Trying Connected to Escape character is '^]'. Type Ctrl ] and then quit to return to the command prompt. Next, try to telnet from the standby Performance Management server to the database server: telnet <IP address of database server> <port number of database server> Example: telnet If your firewall is configured correctly, you see something like this: Trying telnet: connect to address : No route to host Next, test the enablestandby.sh script. Use the same commands. You will see the opposite results. The primary Performance Management server should not be able to connect to database server and the standby Performance Management server should succeed in connecting over port 50,000.

12 Set up a reverse proxy When you set up the reverse proxy, a few things are important to note. This section provides an example of a working Apache 2.2 httpd.conf file. This example does not include the entire configuration, but highlights the required items. In the configuration settings that are provided, you ll notice a few key entries. First, the mod_proxy and mod_rewrite modules are loaded. These modules are required for the reverse proxy. In the configuration that is defined, all user interface communications use HTTPS. Before you start, generate your own certificate for the reverse proxy. Later, you import those certificates into the Performance Management servers. In the VirtualHost section, you configured the reverse proxy to disable SSL v2, SSL v3, and all certificate validation against the backend Performance Management servers. We are using the proxypreservehost to ensure that host names are preserved. The following are some sample httpd.conf file settings. If you choose to copy and paste the contents that are provided, replace the following host names with the fully qualified host names in your environment: ipm2.ibm.com proxy1.ibm.com (standby Performance Management server) (reverse proxy server) You ll notice that initially the reverse proxy is configured to proxy traffic to the standby Performance Management server (ipm2.ibm.com). The reason is because we will initially be installing the standby Performance Management server. Later, you will reconfigure the reverse proxy so that it can proxy traffic to the primary Performance Management server. During an HA failover of IBM Performance Management, reconfiguring the reverse proxy is the way you ensure that traffic is being routed to the active Performance Management server. # Key modules to load httpd.conf file LoadModule rewrite_module modules/mod_rewrite.so LoadModule proxy_module modules/mod_proxy.so LoadModule proxy_http_module modules/mod_proxy_http.so # The Reverse Proxy server must listen on both port 9443 and 8099

13 Listen 9443 Listen 8099 # The first VirtualHost defines the communications that are used on port 9443 <VirtualHost *:9443> ServerName proxy1.ibm.com ServerAlias *.proxy1.ibm.com SSLEngine on SSLProtocol all -SSLv2 -SSLv3 SSLCertificateKeyFile /etc/pki/tls/private/proxy1.key SSLCertificateFile /etc/pki/tls/certs/proxy1.crt SSLProxyEngine On SSLProxyVerify none SSLProxyCheckPeerCN off ProxyRequests off ProxyPreserveHost On ProxyPass / ProxyPassReverse / </VirtualHost> # The second VirtualHost defines the communications on port 8099 <VirtualHost *:8099> ServerName proxy1.ibm.com ServerAlias *.proxy1.ibm.com SSLEngine on

14 SSLProtocol all -SSLv2 -SSLv3 SSLCertificateKeyFile /etc/pki/tls/private/proxy1.key SSLCertificateFile /etc/pki/tls/certs/proxy1.crt SSLProxyEngine On SSLProxyVerify none SSLProxyCheckPeerCN off ProxyRequests off ProxyPreserveHost On ProxyPass / ProxyPassReverse / </VirtualHost>

15 Set up a forwarding proxy The forwarding proxy is used for communication between the agents and the Performance Management server. All agent traffic is sent to the forwarding proxy. The traffic is then proxied to the Performance Management server. When the environment is up and running, reconfiguration of the forwarding proxy allows the agents to connect and send data to the active Performance Management server. In the example that is provided, the forwarding proxy is set up for HTTP communication from the agents. This is the default configuration for IBM Performance Management. The assumption is that all communication is within the LAN and it is not necessary to encrypt the traffic. It is possible to set up the Performance Management server for HTTPS communication. If you do, you must configure the forwarding proxy for HTTPS communication and ensure that the certificates match. You must configure the global.environment file to use the forwarding proxy address for HTTPS. In the following example, you see the configuration settings for an Apache 2.x HTTP server that is set up as a forwarding proxy. You can set up any forwarding proxy that you want. The following file does not represent the entire httpd.conf file, but does represent the key parameters to load the modules for the proxy, define the listening ports, and to set up the forwarding proxy configuration. In the following configuration, replace with the hostname of your primary Performance Management server. During a failover of Performance Management, modify the httpd.conf to use the standby Performance Management server and recycle the HTTP server. # Key modules to load httpd.conf file LoadModule proxy_module modules/mod_proxy.so LoadModule proxy_http_module modules/mod_proxy_http.so # The Forwarding Proxy server must listen on port 80 for Agent communications. If you set up your Performance Management environment for HTTPS, change the port to 443 Listen 80 <IfModule mod_proxy.c>

16 ProxyRequests On ProxyVia On <Proxy *> Order deny,allow Allow from /24 </Proxy> ProxyPass / </IfModule> Notice in the configuration that communication from any systems in the subnet is allowed. Set up that entry so that you allow connectivity from the appropriate subnets in your environment only.

17 Install your Performance Management servers The next step is to install the two Performance Management servers. One is set up as the primary server and the other is set up as the standby server. Their functions and software are identical. But, only one Performance Management server can actively be used to monitor the agents that are connected to the environment. The other server is active and ready to take over the workload in case the primary server goes down. Begin by installing the standby Performance Management server. Install a standby Performance Management server Before you start the installation, run the enablestandby.sh script to block database traffic to the primary Performance Management server. Extract the IBM Performance Management server software. The media is delivered as a tar file that can be extracted. After you extract the tar file, you see an install.sh file. Do not run the install.sh script yet. Some extra steps are required when you use a remote DB2 server. Since you are connecting to a remote DB2 server, you must install either the IBM Data Server Client or the DB2 server software on the Performance Management server system. You do not need to create any databases. Ensure that the following operating system accounts exist on the server and that their passwords match the passwords that are set on the DB2 database server. db2apm itmuser db2fenc1 dasusr1 After you install the DB2 software, change directory to the location where you extracted the IBM Performance Management server software. Before you start the installation, you must ensure that the passwords that are set in the install.properties file match the passwords that are set on your DB2 database server and the local accounts. db2apm.password=passw0rd1! itmuser.password= Passw0rd2!

18 db2fenc1.password= Passw0rd3! dasusr1.password= Passw0rd4! If you want to review an interactive example of the installation process on the standby Performance Management server, go to Appendix B: Prompts for installing a standby Performance Management server. Begin the installation by typing:./install.sh Follow the normal installation procedure for installing the IBM Performance Management server with a few exceptions: Do not configure the agent packages during the installation. That step is executed when the primary IPM node is installed. If you are using DB2 HADR, during the database configuration section, specify the host name or IP address of the DB2 HADR VIP (ipmdb.ibm.com). Otherwise, specify the host name or IP address of the database server. You are asked to confirm the host name and IP address of the server that is used in a browser to connect to the Performance Management server. In this section, you specify the host name and IP address of the reverse proxy as shown in the example. Do NOT accept the default values. If you accidentally accept the default values, you can run the /opt/ibm/ccm/configure_server_address.sh script following the installation. o Fully qualified domain name: proxy1.ibm.com o Short host name: proxy1 o IP: After you install the standby Performance Management server, upgrade the server to the latest interim fix level that is available. You do not have to specify any unique configuration options during the upgrade. Before you can log in to the Performance Management server, you must import the SSL certificates from your reverse proxy into the Performance Management server. Export the *.crt file from your reverse proxy HTTP server. In the example, the file is named proxy1.crt. Copy the *.crt file to the Performance Management server and place it in a temporary directory such as /tmp

19 Back up the existing truststore: cp /opt/ibm/wlp/usr/shared/resources/security/trust.jks /opt/ibm/wlp/usr/shared/resources/security/trust.jks.sav List the current contents of the truststore: keytool -list -keystore /opt/ibm/wlp/usr/shared/resources/security/trust.jks -storepass ccmr0cks! Import the certificate file into the Performance Management server by using the following command: keytool -import -trustcacerts -alias proxy1 -file /tmp/proxy1.crt -keystore /opt/ibm/wlp/usr/shared/resources/security/trust.jks -storepass ccmr0cks! List the current contents of the truststore to confirm that the certificate was imported. keytool -list -keystore /opt/ibm/wlp/usr/shared/resources/security/trust.jks -storepass ccmr0cks! After you import the certificate, restart the APM UI service by using the following command: apm restart apmui Log in to the standby Performance Management server s APM UI and ensure that it is functioning properly. When you log in to the server, you specify a URL using the reverse proxy. Because you configured the Performance Management server with the fully qualified host name and short name of the reverse proxy, that is what the UI is expecting when requests come to the server. You can NOT connect directly to the UI by specifying the host name of the standby Performance Management server. Example: When prompted, enter the administrator user name and password. The user name is apmadmin. The default password is apmpass. If you specified a different password during the installation, use the password that you specified. If you don t see the log in screen or are unable to log on to the APM server, check the following: First, confirm that the APM services are all running by typing: apm status If the APM services are running and you are unable to see the log in screen or are unable to log in, it is probably due to a configuration problem with the reverse proxy.

20 Install a primary Performance Management server Before you start the installation, run the enableprimary.sh script to block database traffic to the standby Performance Management server. You can leave the standby Performance Management server up and running, but it is not fully functional because it does not have access to the DB2 databases. The first step is to drop the existing DB2 databases. The Performance Management installation will not run successfully if the databases have extra content that is populated during the installation. To drop the existing databases, go to the DB2 server and run the following commands. The commands are slightly different if you have an HADR clustered DB2 configuration. If it is an HADR cluster, see the instructions in Appendix D: Dropping databases in a DB2 HADR Environment. For a single node DB2 environment, run the following commands as the db2apm user: db2 drop database WAREHOUS db2 drop database DATAMART db2 drop database SCR32 Next, set up the Performance Management databases by following the instructions in the IBM Performance Management knowledge center for setting up a remote DB2 database. stall_server_mydb2.htm If you are using a DB2 HADR clustered environment, see the additional information in Appendix A: Setting up high availability with DB2 HADR Cluster. After you follow the instructions, three databases are created and tuned with some of the schemas populated. The three databases are: WAREHOUS SCR32 DATAMART You can confirm that the databases are created by running the following command as the db2apm user or another user with database privileges. db2 list database directory

21 You should see three databases and their home directories. On the primary Performance Management server, extract the IBM Performance Management server software. The media is delivered as a tar file that can be extracted. After you extract the tar file, you will see an install.sh file. Do not run the install.sh script yet. Some extra steps are required when you use a remote DB2 server. Since you are connecting to a remote DB2 server, you must install either the IBM Data Server Client or the DB2 server software on the Performance Management server system. You do not need to create any databases. Ensure that the following operating system accounts exist on the server and that their passwords match the passwords that are set on the DB2 database server. db2apm itmuser db2fenc1 dasusr1 After you install the DB2 software, change directory to the location where you extracted the IBM Performance Management server software. Before you start the installation, you must ensure that the passwords that are set in the install.properties file match the passwords that are set on your DB2 database server and the local accounts. db2apm.password=passw0rd1! itmuser.password= Passw0rd2! db2fenc1.password= Passw0rd3! dasusr1.password= Passw0rd4! If you want to review an interactive example of the installation process on the Primary Performance Management server, go to Appendix B: Prompts for installing a standby Performance Management server. Begin the installation by typing:./install.sh

22 Follow the normal installation procedure for installing the IBM Performance Management server with a few exceptions: This time, you DO want to configure the agent packages. When prompted, specify the host name or IP address of the forwarding proxy. This action configures the agent packages so that they send their traffic to the forwarding proxy (proxy2.ibm.com). If you are using DB2 HADR, during the database configuration section, specify the host name or IP address of the DB2 HADR VIP (ipmdb.ibm.com). Otherwise, specify the host name or IP address of the database server. You are asked to confirm the host name and IP address of the server that is used in a browser to connect to the Performance Management server. In this section, specify the host name and IP address of the reverse proxy as displayed in the example. Do NOT accept the default values. If you accidentally accept the default values, you can run the /opt/ibm/ccm/configure_server_address.sh script after the installation. o Fully qualified domain name: proxy1.ibm.com o Short hostname: proxy1 o IP: After you install of the primary Performance Management server, upgrade the server to the latest interim fix level that is available. You do not have to specify any unique configuration options during the upgrade. Before you can log in to the Performance Management server, you must import the SSL certificates from your reverse proxy into the Performance Management server. Export the *.crt file from your reverse proxy HTTP server. In the example, the file is named proxy1.crt. Copy the *.crt file to the Performance Management server and place it in a temporary directory such as /tmp Back up the existing truststore: cp /opt/ibm/wlp/usr/shared/resources/security/trust.jks /opt/ibm/wlp/usr/shared/resources/security/trust.jks.sav List the current contents of the truststore:

23 keytool -list -keystore /opt/ibm/wlp/usr/shared/resources/security/trust.jks -storepass ccmr0cks! Import the certificate file into the Performance Management server by using the following command: keytool -import -trustcacerts -alias proxy1 -file /tmp/proxy1.crt -keystore /opt/ibm/wlp/usr/shared/resources/security/trust.jks -storepass ccmr0cks! List the current contents of the trust store to confirm that the certificate was imported. keytool -list -keystore /opt/ibm/wlp/usr/shared/resources/security/trust.jks -storepass ccmr0cks! After you import the certificate, restart the APM UI service using the following command: apm restart apmui Log in to the standby Performance Management server s APM UI and ensure that it is functioning properly. When you log in to the server, you specify a URL using the reverse proxy. Because we configured the Performance Management server with the fully qualified host name and short name of the reverse proxy, that is what the UI is expecting when requests come to the server. You can NOT connect directly to the UI by specifying the host name of the standby Performance Management server. Example: When prompted, enter the administrator user name and password. The user name is apmadmin. The default password is apmpass. If you specified a different password during the installation, use the password that you specified. If you don t see the log in screen or are unable to log in to the APM server, look at two things: First, confirm that the APM services are all running by typing: apm status If the APM services are running and you are unable to see the log in screen or are unable to log in, it is probably due to a configuration problem with the reverse proxy.

24 Install your monitoring agents You are now ready to install a small number of monitoring agents. The agent media is preconfigured to connect to the forwarding proxy. Using the agent media in /opt/ibm/ccm/depot, install one or more agents. Note: Do NOT use the agent media that you downloaded from Passport Advantage. You must use the agent media that was reconfigured during the installation. The media that was found in /opt/ibm/ccm/depot was configured to communicate with the forwarding proxy. When an agent is installed, you will want to verify a couple of things: 1. Confirm that the agent is successfully connecting to the primary Performance Management server and is visible in the APM UI. Depending on the size of your environment, you might need to create a business application and add the agents to the application before they become visible. 2. Edit the /opt/ibm/apm/agent/localconfig/<2 letter code>/<2 letter code>_asfserver.xml file on one of the servers and check that the URL points to the forwarding proxy. o For the Linux OS Agent, this is /opt/ibm/apm/agent/localconfig/lz/lz_asfserver.xml You are now ready to configure and test the failover between the two servers.

25 Configure and test failover Now that the two Performance Management servers are up and running, verify that failover and failback is working properly. After you ensure that the failover and failback are working properly, create automation to automate as much of the process as possible. Set up customizations and synchronize the content To show that content is being synchronized properly between the two servers, it is important to make some changes. For example, you might want to make the following types of changes: Create a new threshold Create a new resource group Create a new business application or modify the contents of an existing application Customize agent configuration settings that can be modified centrally using the UI, for example, define some log file monitoring or change the transaction tracking settings for an agent. Define some new Role Based Access Control (RBAC) settings Now, you need to synchronize content between the servers. Some of the content is automatically synchronized because you re using a common DB2 server. But, some content must be synchronized by using the backup.sh/restore.sh scripts. Since the primary Performance Management server is now the active server, run the /opt/ibm/ccm/backup.sh script. This script backs up content and saves a file similar to the following: o /opt/ibm/backups/backup_ _ tar Your actual file name is different, but can be identified when you run the backup.sh command. Copy the tar file to the standby Performance Management server. The file can be placed in any directory. Before you restore the backup on the standby Performance Management server, you must modify the restore.sh script. The restore.sh script attempts to restore content for several components including mongodb, oslc, and more. One of the components that it restores, by

26 default, is the SCR component. You do NOT want to restore the SCR component for the following reasons: o First, there is no need to restore the SCR component because it is a shared database on the DB2 server. It already has the most current content. o Second, if you attempt to restore the SCR component in the current state, it will fail because the standby Performance Management server can t connect to DB2. Traffic is being blocked by the firewall. So, you must edit the /opt/ibm/ccm/restore.sh script and make the following change. Look for COMPONENT_LIST near the start of the file. Make a copy of that entry and comment out one of the entries. This allows you restore SCR if you want at some later date. In the entry that is uncommented, remove the entry for scr. The COMPONENT_LIST entry must be changed from: o COMPONENT_LIST="install supports itmcdp db2 oauth2 oidc min_config min spark mongodb scr apmui cscs ccs oslc biagent" to o COMPONENT_LIST="install supports itmcdp db2 oauth2 oidc min_config min spark mongodb apmui cscs ccs oslc biagent Edit the COMPONENT_LIST in the restore.sh script on both the primary and standby Performance Management servers to allow you to synchronize content in either direction. You are now ready to restore the backup content. Run the command /opt/ibm/ccm/restore.sh <path to backup tar file>, where the path to the backup tar file is the file that you copied from the primary Performance Management server. o For example: /opt/ibm/ccm/restore.sh -f /tmp/backup_ _ tar After a period, the restore will complete. Pay attention to how long the restore takes. Later, as you automate the process, you ll need to understand how long this takes. However, the amount of time grows as more agents are added to the environment and more customization is added. You are now ready to perform a failover.

27 Understand what data is not backed up The backup and restore procedures will back up the critical content to ensure that you can fail over to the standby Performance Management server. However, some data is not backed up. The DB2 server and the three databases are not backed up using the backup scripts. They should be backed up by your DBA according to your desired schedule. For example, weekly full backups and incremental backups. Deep dive diagnostic data is stored on the agents and is not accessible within the APM server for backup and restore. However, during a failover, this data is not lost because it is still available on the agent. Transaction tracking data is stored within the MongoDB database. There is a large volume of transaction tracking data stored in the MongoDB database and it is not backed up by the backup.sh script. If you want, create a backup script to back up and restore the transaction tracking data that is stored in MongoDB. Test failover and failback You are now ready to test the failover of the Performance Management server. Use the following steps to trigger a failover. First, reconfigure the firewall on the DB2 servers so that traffic is now blocked from the primary Performance Management server and allowed from the standby Performance Management server. Run the enablestandby.sh script on the DB2 server. Next, reconfigure the forwarding proxy so that agent traffic is routed to the standby Performance Management server. Change only one line in the httpd.conf file, as follows: o Change: ProxyPass / o To: ProxyPass / o Then, recycle the HTTP server for the changes to take effect. Finally, reconfigure the reverse proxy so that UI traffic is routed to the standby Performance Management server. Change four entries in the file. You ll find two entries for proxypass and two entries for proxypassreverse. Replace the host name of the primary Performance Management server with the host name of the standby Performance Management server. Then, recycle the HTTP server for the changes to take effect. Everything should be fully functional following the failover. You might need to log out and back in to the Performance Management user interface.

28 Note: If you are using a DB2 HADR cluster, you can also test the failover between the two database nodes. Use the DB2 commands to fail over the database nodes. You can find information here: in.ha.doc/doc/c html After failing over, confirm the following: o Confirm that you can log in to the Performance Management server. You might want to shut down the primary Performance Management server to prove that you are accessing the correct server. o Confirm that key features and functions are working properly. o Confirm that agents are still sending data to the Performance Management server even though you are now using the standby server. You might experience a small gap in your data from the time that you reconfigured the firewall and the time that the DNS changes took effect. o Confirm that the configuration modifications that you made on the primary Performance Management server are showing up on the standby Performance Management server. For example, verify that you see the new thresholds and resource groups that you created. Verify that you see the RBAC changes that you made. If you want, you can trigger a failback to the primary Performance Management server by reversing the steps: o Back up the content on the standby Performance Management server by running the backup.sh script on the standby server. o Copy the backup tar file to the primary Performance Management server. o Restore the content to the primary Performance Management server by running the restore.sh script. o Trigger the failback by doing these three things: Run the enableprimary.sh script on the DB2 server(s) to block traffic to the standby Performance Management server. Reconfigure the reverse proxy so that UI traffic flows to the primary Performance Management server

29 Reconfigure the forwarding proxy so that agent traffic flows to the primary Performance Management server. Automate the synchronization and failover processes You now understand how data is backed up and restored between the two Performance Management servers. You also understand what needs to change in the environment during a failover. It is now time to automate as much of that process as possible. On the primary Performance Management server, create a cron job that periodically backs up the server by running the /opt/ibm/ccm/backup.sh script. Base the frequency of the cron job on how long it takes to run the backup script. This duration increases as more agents are added to the environment and more customization is added. So, be conservative initially until you have a good sense of how long the backups are going to take. A suggestion is to back up every 2 hours initially. Using the mechanism of your choice, automate the process of copying the backup tar files to the standby Performance Management server. Using something like rsync to synchronize the /opt/ibm/backups directory works well. On the standby Performance Management server, create a cron job that will periodically restore the content that was copied from the primary Performance Management server. The cron job will run the /opt/ibm/ccm/restore.sh f <filename> command. You must create a script that ensures that you restore the most recent backup tar file. A suggestion is to restore no more than every 2 hours, but this interval depends on the need for currency and duration of the restore.sh script your environment. If you want, set up automation to detect an outage on the primary server and trigger the failover process. If a failover is triggered, the automation must: o Run the enablestandby.sh script on the database servers. o Reconfigure the reverse proxy so that UI traffic is routed to the standby Performance Management server. On an HTTP server, the easiest approach is to create two different httpd.conf files. During a failover, copy the standby httpd.conf file into place and recycle the HTTP server. o Reconfigure the forwarding proxy so that UI traffic is routed to the standby Performance Management server. On an HTTP server, the easiest approach is to

30 create two different httpd.conf files. During a failover, copy the standby httpd.conf file into place and recycle the HTTP server.

31 Complete optional configuration tasks Some optional configuration tasks that can be done as part of this high availability configuration. Configure HTTPS Traffic from the agents If you decide to configure HTTPS traffic between the agents and the Performance Management server, follow the instructions for setting up certificates at the following link: em_config_cont_certificates.htm Keep in mind that the certificate will also need to be added to the forwarding proxy. The agents connect to the forwarding proxy using the certificate and the forwarding proxy connects to the Performance Management server using the certificate. Synchronize the Performance Management Servers using backup and restore This section recommends how to keep the two Performance Management servers in sync to ensure that minimal data is lost during the failover from the primary to the standby Performance Management server. You use the backup.sh script that is provided with the product to periodically back up the configuration and event content on the primary Performance Management server. The restore.sh script is used to restore the configuration and event content to the standby Performance Management server. Define a backup schedule Define a backup schedule that can be reliably executed in your environment. This is based on how long it takes to run the backup.sh and restore.sh scripts. More than likely, the restore.sh script will require longer than the backup.sh scripts. The duration of the backup is based on the quantity of data that is stored in the various repositories. Information includes data that is collected from the agents and configuration data. As more configuration data and agent data is added to the environment, the restore takes longer. So, monitor the duration that the restore.sh takes to run and adjust your automation as required.

32 Appendix A: Setting up high availability with DB2 HADR Cluster Setting up a DB2 HADR cluster is the preferred method for achieving database high availability with IBM Performance Management. If your DBA is familiar with setting up DB2 in a clustered configuration with a virtual IP address (VIP), then the configuration and failover of Performance Management is straightforward. When you use a VIP, the high availability of the database server is transparent to the Performance Management servers. If a VIP is not used, then extra configuration steps are required to configure the Performance Management components so that they use a primary and secondary database. Setting up the DB2 HADR Cluster Complete these steps to set up Performance Management with a DB2 HADR Cluster configuration. First, set up DB2 in a clustered configuration. Follow the DB2 documentation and configure DB2 as needed. DB2 has multiple HADR configurations. Documentation can be found here: ha.doc/doc/c html You can achieve high availability in multiple ways including clustering where the standby node takes over the IP address of the primary node or through a Virtual IP address. Configuring Performance Management to use a standard HADR cluster is complex because you must manually configure some of the components to use the primary and secondary database nodes. Setting up DB2 HADR with a Virtual IP (VIP) is much simpler. When you use a VIP, the Performance Management server treats the database servers the same as it would a single database node. It is highly recommended that you use a VIP for your DB2 HADR configuration. You can find information about setting up a VIP at these links: hadrdb2haicu/HADR_db2haicu.pdf tivoli/index.html Either way, you must set up a cluster so that your database clients, in this case the Performance Management servers, can connect to a single IP address without knowledge of the backend DB2 servers. From an Performance Management server perspective, both Performance Management servers will connect to a single host

33 name/ip address, and behind the scenes, DB2 ensures that the single host name/ip address is connected to the active database server. The Performance Management servers are connected to the virtual IP address (VIP). At any point in time, the VIP connects only to one of the two clustered database servers. The recommendation for data synchronization is to set up the DB2 synchronization mode as NEARSYNC, but other options are supported. The level of synchronization that you choose depends on the amount of risk you want to take in possibly losing a transaction. The higher the protection, the higher the overhead is on the database transactions. NEARSYNC is a good compromise that offers good performance with minimal risk of losing transactions. In the diagram, notice that the following host names are used. These names are used in the example commands used throughout this document: DNS Alias for the Performance Management server: ipm.ibm.com Host name of the primary Performance Management server: ipm1.ibm.com Host name of the standby Performance Management server: ipm2.ibm.com Host name or the VIP for the database cluster: ipmdb.ibm.com Host name of the primary DB2 node: ipmdb_1.ibm.com Host name of the standby DB2 node: ipmdb_2.ibm.com Ensure that the DB2 Cluster is setup with a VIP (Virtual IP address). This ensures that the Performance Management server does not need any knowledge of the backend database topology and state. The Performance Management servers are configured using the VIP. Ensure that both DB2 nodes and all three databases are configured for online backups. Instructions for configuring online backups are documented at the following link: min_backup_online_dbs.htm Setting up the Performance Management database in DB2 HADR The follwing are examples of commands used to set up the DB2 database in a DB2 HADR clustered environment: