Note: Before using this information and the product it supports, read the information in Notices.

Load Balancing and High Availability UI Data Provider Connections to Netcool/Impact Author: Brian R. Fabec, Advisory Software Engineer and Plamen Tzvetkov, Software Engineer October, 2016 Note: Before using this information and the product it supports, read the information in Notices. 1

Copyright International Business Machines Corporation 2015. US Government Users Restricted Rights Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corporation 2

Contents Introduction... 3 Overview... 3 Prerequisites... 4 Load balancing for Dashboard Application Services Hub... 5 Netcool/Impact Clustering Architecture... 5 Netcool/Impact Clustering Architecture with UI data provider High Availability... 6 IBM HTTP Server... 6 Installing IBM HTTP Server 8.5...7 Configuring the Web Server Plugin for SSL Connectivity...15 Configuring the Web Server Plug-in...20 Configure the IBM HTTP Server...24 Configuring Dashboard Connections to the Load Balancer... 26 Conclusion... 30 Troubleshooting... 30 References... 31 Trademarks...35 Introduction This paper is a step-by-step solution guide to enable load balancing and high availability connections for the Jazz for Service Management (JazzSM) Dashboard component to the Netcool/Impact data provider cluster. Netcool/Impact 6.1.1 and higher provide out of the box solution capabilities by exposing Netcool/Impact data types and policy variables (user output parameters) through its user interface (UI) data provider to visualize its data through dashboard widgets. By using an IBM HTTP server to roundrobin HTTP(S) connections, we can improve performance and guarantee availability, scaling horizontally by increasing Netcool/Impact GUI servers and backend servers. Overview 3

In the environment described in the following diagram, the user connects to the Jazz for Service Management Dashboard server component through the IBM HTTP server which load balances the connections and provides high availability to multiple Dash servers, each pointing to the same database server in a failover type configuration. The Dashboard servers have connections which are configured to the various data providers that supply the Dashboard widgets with various data to visualize. Instead of having the connections point directly to the data provider, the connection to Netcool/Impact points to an IBM HTTP server which load balances the connection and provides high availability for the suite of UI data provider connections. The UI data provider connections IBM HTTP server points to multiple Netcool/Impact GUI servers, where the UI data provider resides. The Netcool/Impact GUI servers each point to the primary backend server that is also in a cluster with the Netcool/Impact out of the box clustering solution. While this example describes connections to a Netcool/Impact data provider, other data providers that do not support an out of the box type connection failover can potentially use this same type of architecture, such as IBM Tivoli Monitoring (ITM). Note: This paper is not a procedure for load balancing Netcool/Impact GUI interfaces. It focuses on load balancing and high availability UI data provider connections from Netcool/Impact to the Jazz for Service Management Dashboard. Prerequisites The following three products are required for this integration: 1. Netcool/Impact 7.1 or higher 4

2. Jazz for Service Management 3.1.0.2 or higher Dashboard Component 3. IBM HTTP Server 8.5 or higher To provide high availability, a minimum of three servers should be available for this setup for Netcool/Impact: two for a combination installation including both GUI and backend servers, and one for the IBM HTTP server. Additional servers can be utilized to scale horizontally or to separate the GUI and backend servers by placing them each on their own individual server. Optionally, if load balancing and high availability of the Dashboard servers are required, then a minimum of three additional servers should be available: two for the Dashboard and database servers, and one for the IBM HTTP server. Additional servers can be used to scale horizontally or to separate the Dashboard and database servers by placing them each on their own individual server. Note: For the basis of this paper, we are using Jazz for Service Management 3.1.1.0 and Netcool/Impact 7.1 FP2. Load balancing for Dashboard Application Services Hub The Jazz for Services Management Dashboard Hub component supports load balancing through a cluster of console nodes with identical configurations which evenly distribute user sessions, so when a Dashboard server becomes unavailable, new user connections are directed to Dashboard servers that are available. This paper focuses solely on the setup of UI data provider connections for high availability. However, the procedure to configure load balancing for Dashboard servers is documented with great detail and recommended as part of your configuration. For more information on configuring Dashboard servers for load balancing, refer to the following link: http://www- 01.ibm.com/support/knowledgecenter/SSEKCU_1.1.1.0/com.ibm.psc.doc_1.1.1.0/tip_original/ctip_confi g_ha_ovw.html?lang=en Netcool/Impact Clustering Architecture Netcool/Impact has two distinct types of servers: a GUI and a backend server. Netcool/Impact provides an out of the box high availability clustering solution with its backend servers by replicating the data sources, data types, policies, and so forth. Additionally, the backend servers can perform load balancing by offloading event processing work to the various available backend servers. The backend servers can be in one of two states: primary mode or secondary mode; and only one backend server can be in primary mode at any given time. Netcool/Impact connections to certain data sources can also be setup for failover and failback type configurations. 5

The GUI server connects to the primary backend server. When the primary backend server becomes unavailable, a secondary backend server will be promoted to primary, and the GUI server will reconnect to this new backend server. The user (or the dashboard widget) connects to the Netcool/Impact GUI server, and when the GUI server becomes unavailable, any future connection attempts to the GUI server fails, the user must either wait for that GUI server to be available again, or connect to another GUI server that is available. In addition, any new user (or the dashboard widget) requesting data will point to the same DASH-configured GUI server, resulting in the UI data provider being overused, while other UI data provider servers in the cluster will be underutilized. For more information on Netcool/Impact clustering, refer to the following link: http://www- 01.ibm.com/support/knowledgecenter/SSSHYH_7.1.0.2/com.ibm.netcoolimpact.doc_7.1.0.2/admin/imag _cluster_c.html Netcool/Impact Clustering Architecture with UI data provider High Availability The proposed architecture in this paper is an option to resolve the lack of failover and load-balancing of the UI data providers (component of GUI server) in an Impact/Netcool cluster. The GUI server doesn t have the notion of primary or secondary status and all GUI server cluster members are with the same status. With the introduction of the HTTP Server, the user (or the dashboard widget) connects to the IBM HTTP Server, instead of connecting to the UI data provider directly, and enables the load-balancing and failover for the GUI server cluster-members. In case of UI data provider failure, any future HTTP requests from the user (or the dashboard widget) will be re-routed by the HTTP Server to the next available GUI data provider in a new HTTP session. This will provide the UI data provider failover functionality. In this case, we may see in the logs a one-time benign message informing for potential Cross-Site-Request-Forgery attempt. Each http session generates a unique token, and in failover the new HTTP session generates a new token passed as a parameter. In case of a new user (or new dashboard widget) making HTTP request for data, a new HTTP session will be established with the next available GUI servers in the cluster, in a round-robin fashion. This will provide the UI data provider load-balancing functionality, which is happening on per session, not per request basis. IBM HTTP Server To provide load balancing for the connections from Jazz for Service Management dashboards to the Netcool/Impact GUI servers, an IBM HTTP Server must be installed and configured. The mode that we describe in this paper is a round-robin type configuration. In a round-robin type configuration, when a connection from the Dashboard is made to the HTTP server, it is directed to one of the Netcool/Impact GUI servers. When another connection is made, it is directed to one of the other Netcool/Impact GUI 6

servers. The IBM HTTP Server is installed between the Netcool/Impact GUI servers and the Jazz for Service Management dashboard server. Jazz for Service Management bundles the WebSphere Application Server Version 8.5 Supplements installation media, which contains the installation packages for IBM HTTP Server. If you do not have the DVDs, you can download the electronic images for Jazz for Service Management from IBM Passport Advantage. http://www-01.ibm.com/software/passportadvantage/ Installing IBM HTTP Server 8.5 On the server where the IBM HTTP Server will reside, extract the WebSphere Application Server Version 8.5 Supplements. To install the IBM HTTP Server, IBM Installation Manager must be installed first. If IBM Installation Management is not already installed on the server, you can obtain the installation media from the following link: http://www.ibm.com/support/entry/portal/downloads/software/rational/ibm_installation_manager 1. Launch the IBM Installation Management GUI by running the following command: $IM_HOME/eclipse/launcher 2. Select File -> Preferences and add the extracted supplement directory as a repository location. 7

3. Select Install on the main IBM Installation Manager screen. 4. Select the IBM HTTP Server for WebSphere Application Server, Web Server Plug-ins for IBM WebSphere Application Server, and the WebSphere Customization Toolkit (WCT) and select Next. 8

9 5. After reading and accepting the license, select Next.

10 6. Choose an installation location and select Next.

11 7. Select the Architecture type and select Next.

12 8. Choose the HTTP port which the IBM HTTP Server will communicate on and select Next.

13 9. Select Install to start the installation.

14 10. Select Finish to complete the installation process.

Configuring the Web Server Plugin for SSL Connectivity The keystore used by the web server plug-in must be a CMS keystore. The JKS keystore that is created by the Liberty profile and used by Netcool/Impact cannot be used. The CMS keystore must be created using the ikeyman utility and certificates exchanged between the web server plug-in CMS keystore and the Netcool/Impact JKS keystore. 1. For each Impact GUI server in the environment, export the SSL certificate from the Netcool/Impact JKS keystore using the keytool command: $IMPACT_HOME/sdk/bin/keytool -export -alias default -file <filename> -keystore $IMPACT_HOME/wlp/usr/servers/ImpactUI/resources/security/key.jks For example: keytool -export -alias default -file /tmp/impact71devlin.crt -keystore /opt/ibm/tivoli/impact_ha/wlp/usr/servers/impactui/resources/security/key.jks 2. Copy all the exported certificates to the IBM HTTP Web server. 3. Start the ikeyman Java utility tool on the local machine. The tool can be located under the WebSphere Customization Toolkit directory: $WCT_HOME/java/jre/bin/ikeyman or under IBM 15

HTTP Server : $HTTP_SERVER_HOME/java/jre/bin/ikeyman. In both locations, the tool is functionally identical. E.g. /opt/ibm/httpserver/java/jre/bin//ikeyman 4. Select the Create a new key database file icon. 5. Select the CMS key database type, and provide a location and filename for the database file, and select OK. Note: This location will be used throughout the rest of the configuration. In this paper, we used the location /opt/ibm/httpserver/conf/plugin.kdb to store the database keyfile. 16

6. Enter a keystore password and ensure the Stash password to a file checkbox is selected. Select OK. 7. Create at least one personal self-signed certificate. 17

18 8. Select Signing Certificates from the Key database content drop down list.

19 9. For each exported SSL certificate performed in Step 1, add them to the key database.

10. Close the ikeyman utility. Configuring the Web Server Plug-in A web server plug-in is used to forward HTTP requests from the IBM HTTP Server to one or more application servers, including WebSphere Liberty which is the application server platform Netcool/Impact servers run on top of. The plug-in takes the request and based on the configuration inside the plugincfg.xml file, maps the URI for the HTTP request to the host name and port number of an application server, and finally forwards the request to the specified application server. 1. Login to any server in the environment that is currently a host for an Impact GUI server. 2. Add the following plugin Configuration element in the $IM- PACT_HOME/wlp/usr/servers/ImpactUI/server.xml file in between the <server> </server> section: <pluginconfiguration webserverport="80" webserversecureport="443" sslkeyringlocation="/opt/ibm/httpserver/conf/plugin.kdb" sslstashfilelocation="/opt/ibm/httpserver/conf/plugin.sth" sslcertlabel="impactui"/> 3. Add the following feature element in the $IMPACT_HOME/wlp/usr/shared/config/features.xml file in between the <featuremanager> </featuremanager> section. <feature>localconnector1.0</feature> 4. Restart the Netcool/Impact GUI server. Ensure that it is currently online and connections are available. 5. Start the jconsole utility under the $IMPACT_HOME/sdk/bin directory. 20

6. Select Local Processes and choose the ImpactUI process and select the Connect button. Note: The connection operation can take several minutes. 21

22

23 7. Select the MBeans tab and locate the com.ibm.ws.jmx.mbeans.generatepluginconfig MBean. Under operations, select the generatedefaultpluginconfig operation to generate the plugin.

24 8. Repeat the steps for each Impact GUI server in the environment. 9. The plugin-cfg.xml is generated under the $IMPACT_HOME/wlp/usr/servers/ImpactUI directory. Copy all of the generated plugin-cfg.xml files to the JazzSM Dashboard Component server. 10. In the $JAZZSM/profile/bin directory, use the plugincfgmerge utility to merge all of the generated plugin-cfg.xml files that were copied. The following generates the plugin-cfg.xml file that will be used within the IBM HTTP Server by merging plug-cfg1.xml and plugin-cfg2.xml files. i.e../plugincfgmerge.sh -sortvhostgrp -debug plugin-cfg1.xml plugin-cfg2.xml plugin-cfg.xml 11. Copy the generated plugin-cfg.xml file to the server that hosts the IBM HTTP Server under the configuration directory. For example /opt/ibm/httpserver/conf Configure the IBM HTTP Server 1. Add the following lines to the bottom of the IBM HTTP server configuration file (httpd.conf), where the mod_was_app22_http.so has the correct path to the plugin location.

LoadModule was_ap22_module "/opt/ibm/websphere/plugins/bin/64bits/mod_was_ap22_http.so" WebSpherePluginConfig "/opt/ibm/httpserver/conf/plugin-cfg.xml" 2. To enable SSL on the IBM HTTP Web Server, add the following section to the bottom of the httpd.conf file: LoadModule ibm_ssl_module modules/mod_ibm_ssl.so Listen 443 <VirtualHost *:443> SSLEnable </VirtualHost> KeyFile /opt/ibm/httpserver/conf/plugin.kdb SSLDisable 3. Start the IBM HTTP Web Server:./apachectl start Note: You will not be able to successfully login to the Netcool/Impact GUI using the load balancer. However, HTTP and HTTPS UI data provider connections from the Dashboard will connect successfully. 25

Configuring Dashboard Connections to the Load Balancer If SSL UI data provider connections are required, the SSL certificate from the load balancer server needs to be imported into the truststore of the Dashboard server. 1. Login to the Dashboard server and select Settings -> WebSphere Administrative Console. 2. Expand Security and select SSL certificate and key management. 3. Under Related Items, select Key stores and certificates. 26

4. Select the NodeDefaultTrustStore. 5. Under Additional Properties, select Signer certificates. 6. Select the Retrieve from port button. 27

7. Enter the IBM HTTP Server Hostname, the SSL port number for the load balancer (for example 443) and an Alias to describe the certificate. Select Retrieve signer information and select OK. 8. Select Save the configuration and restart the Dashboard server for the changes to take effect. Once SSL certificate from the IBM HTTP Server is loaded into the truststore of the Dashboard server and the Dashboard server is restarted, an SSL or non-ssl connection to the IBM HTTP Server can be created. 1. Login to the Dashboard server and select Settings -> Connections. 28

2. Select the Create new remote provider icon. 3. Provide the Protocol (HTTP/HTTPS-SSL), the IBM HTTP Server hostname, the IBM HTTP Server port number (443), and the Netcool/Impact username and password. 4. Select the Search button which populates the providers list. Select the Impact_<CLUSTERNAME> radio button and select OK to create the new provider connection. 5. The new provider connection appears in the Connections list. 29

6. Dashboard pages can now be created using the IBM HTTP Server providing load balancing and high availability connections to the Netcool/Impact UI data provider GUI servers. Conclusion In this paper, we provided a detailed step-by-step guide to enabling load balancing and high availability connections between the Jazz for Service Management (JazzSM) Dashboard component to the Netcool/Impact GUI server data providers. By utilizing an IBM HTTP Server front end for Dashboard connections, when a Netcool/Impact GUI server becomes unavailable, the IBM HTTP Server requests will continue to make connections to other available Netcool/Impact GUI servers that are available in the environment, scaling horizontally across all of servers to provide high availability. Additionally, HTTP connections are round-robin so that various Netcool/Impact GUI servers, distribute the workload across the cluster. While this example describes connections to a Netcool/Impact data provider, other data providers that do not support an out of the box type connection failover can potentially utilize this same type of architecture, such as IBM Tivoli Monitoring (ITM). Troubleshooting To enable additional logging for the web server plugin, edit the plugin-cfg.xml file in the configuration directory of the IBM HTTP Server and update the Log element with the correct location and log level for the plugin logs. For example: <Log LogLevel="Trace" Name="/opt/IBM/HTTPServer/logs/http-plugin.log"/> 30

References Netcool/Impact Clustering Overview: http://www- 01.ibm.com/support/knowledgecenter/SSSHYH_7.1.0/com.ibm.netcoolimpact.doc_7.1/admin/imag_cluster_c.html Load balancing for Dashboard Application Services Hub: http://www- 01.ibm.com/support/knowledgecenter/SSEKCU_1.1.0.3/com.ibm.psc.doc_1.1.0.3/tip_original/ctip_config_ha_ovw. html?lang=en Configuring a web server plug-in for the Liberty profile: http://www- 01.ibm.com/support/knowledgecenter/SSAW57_8.5.5/com.ibm.websphere.wlp.nd.doc/ae/twlp_admin_webserver_p lugin.html?cp=ssaw57_8.5.5%2f1-3-11-0-3-3-6 Understanding IBM HTTP Server plug-in Load Balancing in a clustered environment: http://www- 01.ibm.com/support/docview.wss?uid=swg21219567 31

32

Notices This information was developed for products and services offered in the U.S.A. IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation of any non-ibm product, program, or service. IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing IBM Corporation North Castle Drive Armonk, NY 10504-1785 U.S.A. For license inquiries regarding double-byte (DBCS) information, contact the IBM Intellectual Property Department in your country or send inquiries, in writing, to: Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan, Ltd. 1623-14, Shimotsuruma, Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement might not apply to you. This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice. Any references in this information to non-ibm websites are provided for convenience only and do not in any manner serve as an endorsement of those websites. The materials at those websites are not part of the materials for this IBM product and use of those websites is at your own risk. IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you. Licensees of this program who wish to have information about it for the purpose of enabling: (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged, should contact: 33

IBM Corporation 2Z4A/101 11400 Burnet Road Austin, TX 78758 U.S.A. Such information may be available, subject to appropriate terms and conditions, including in some cases payment of a fee. The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement, IBM International Program License Agreement or any equivalent agreement between us. Any performance data contained herein was determined in a controlled environment. Therefore, the results obtained in other operating environments may vary significantly. Some measurements may have been made on developmentlevel systems and there is no guarantee that these measurements will be the same on generally available systems. Furthermore, some measurement may have been estimated through extrapolation. Actual results may vary. Users of this document should verify the applicable data for their specific environment. Information concerning non-ibm products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products and cannot confirm the accuracy of performance, compatibility or any other claims related to non-ibm products. Questions on the capabilities of nonibm products should be addressed to the suppliers of those products. All statements regarding IBM's future direction or intent are subject to change or withdrawal without notice, and represent goals and objectives only. This information is for planning purposes only. The information herein is subject to change before the products described become available. This information contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the examples include the names of individuals, companies, brands, and products. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental. COPYRIGHT LICENSE: This information contains sample application programs in source language, which illustrate programming techniques on various operating platforms. You may copy, modify, and distribute these sample programs in any form without payment to IBM, for the purposes of developing, using, marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore, cannot guarantee or imply reliability, serviceability, or function of these programs. You may copy, modify, and distribute these sample programs in any form without payment to IBM for the purposes of developing, using, marketing, or distributing application programs conforming to IBM s application programming interfaces. Each copy or any portion of these sample programs or any derivative work, must include a copyright notice as follows: (your company name) (year). Portions of this code are derived from IBM Corp. Sample Programs. Copyright IBM Corp. _enter the year or years_. All rights reserved. If you are viewing this information in softcopy form, the photographs and color illustrations might not be displayed. 34

Trademarks IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at Copyright and trademark information at www.ibm.com/legal/copytrade.shtml. Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/or its affiliates. Other company, product, or service names may be trademarks or service marks of others. 35