Mediation Pack for BMC TSIM

Transcription

1 Operations Analytics Predictive Insights IBM Operations Analytics Predictive Insights Version 1.2.0

2 Operations Analytics Predictive Insights Edition notice This edition applies to IBM Operations Analytics Predictive Insights version and to all subsequent releases. Please refer to the IBM developerworks terms of use for more information. Also reference the IBM Operations Analytics Development Community ( for additional information on Predictive Insights.

3 CONTENTS Overview... 4 Prerequisite Information... 5 BMC TSIM Metrics... 6 Metrics Json File... 6 Samples of BMC Data CSV Files... 6 BMC Devices Installing and Configuring... 9 Mediation Packs Contents... 9 Configure the Mediation Pack Configure BMC TSIM Metric Collection Configure Data Modeling Setup the Predictive Insights Topic Deploy the Defined Model in Predictive Insights Test Mediation Pack and Data Training Process Review Results and Anomalies Support 24 iii

4 Overview IBM Operations Analytics Predictive Insights analyzes data and learns the normal behavior of a system through KPI metric data streams. This includes data from various sources including metrics gathered by industry performance and network monitoring solutions. A mediation pack helps users quickly integrate the KPI data streams by providing a default model to integrate a particular performance manager. IBM Operations Analytics Predictive Insights provides a pre-built model for data ingestion. Predictive Insights creates a performance model that it uses to detect or forecast behavior outside the modeled range, including generating an alarm when an anomalous behavior occurs. Doing so enables organizations to pro-actively avoid system and solution outages before users are impacted. The provides the capability to ingest performance data and metrics from BMC TSIM into IBM Operations Analytics Predictive Insights to perform analysis. This document will cover the versions of the BMC TSIM and Predictive Insights support, along with steps to setup the Mediation Pack and adjust the data stream for deployment in a customer environment. BMC TSIM API calls Mediation Pack development considers all possible integration options to determine the optimal configuration. Options are reviewed based on the source systems ability to present data using multiple methods. The selected methods are derived based on many factors including but not limited to flexibility, scalability, and availability of KPI s (Key Performance Indicators). Additional considerations are based on the source systems majority configuration and maturity of such methods at the time of development. Requests to enhance mediation packs utilizing additional methods supported by the source system can be reviewed with our professional services team. 4

5 Prerequisite Information Name Item Predictive Insights supported versions Predictive Insights Information V1.3.3, v1.3.5, and v Description IBM Operations Analytics (demo, trial, resources) IBM Operations Analytics Community (docs, videos, mediation packs) IBM Operations Analytics Predictive Insights documentation BMC TSIM supported versions BMC TSIM Information Filename Data source BMC TSIM Version v5.4.5 and above BMC TSIM Product Information BMC_MedPack_v1.2.0.zip This pack creates csv files by a java client that uses API calls to the BMC TSIM. These csv files are deposited on the Predictive Insights server itself. The csv files are generated using a java script from the /bin directory in the mediation pack. For more information, please refer to the section titled Configure Metric Collection Additional Information BMC TSIM needs to be configured for historical collection for metrics and collecting performance metric data for every 5 min interval. 5

6 BMC TSIM Metrics BMC TSIM provides metric data in CSV format for which the Mediation Pack will use to formulate content into Predictive Insights. BMC TSIM provides APIs which are utilized to retrieve the performance metric data. A java client is employed to call the various APIs and format the metric data into the corresponding CSV files. Metrics Json File The Pre-identified metrics are listed into a Json file as key value pairs, which is later used to read each metric dynamically through the java client to get the performance data and write to the csv files. The Json structure should be followed as shown in the example below. Each block of metrics for different OS systems should be written separately. The above metrics file is generated through a python script using the available metadata rest api call from the BMC TSIM environment. The Response is parsed and written to a metrics.json file in the above mentioned format. The script must be executed with the Rest API url as Parameter. For example: Python metrics.py <url to get the metadata> 6

7 Samples of BMC TSIM Data CSV Files The following are samples of the BMC TSIM CSV file contents. Each CSV represents a different metric group that has been extracted from BMC TSIM via an API call and then formatted into a CSV file using a java utility. CPUprcrProcessorTimePercent NETniNetworkUtilization SYSsysProcessorQueueLength Memory Usage timestamp host name Memory Usage xx.xx.xx.xx yy.yy.yy.yy PDpdDiskTimePercent timestamp host name PDpdDiskTimePercent xx.xx.xx.xx yy.yy.yy.yy ProcessorUtilization timestamp host name ProcessorUtilization xx.xx.xx.xx yy.yy.yy.yy

8 LDldDiskTimePercent timestamp host name LDldDiskTimePercent xx.xx.xx.xx yy.yy.yy.yy 0.05 LDldFreeSpacePercent timestamp host name LDldFreeSpacePercent xx.xx.xx.xx yy.yy.yy.yy BMC TSIM Metrics BMC TSIM device list is also configured as a Input json file to the java scrapper where the devices are written as HostName:IP address format. The devices.json file will be generated through a Presentation server Rest API call. The Rest API used to fetch the device list is as below. <rest api to get the device list> If the Presentation Server environment is not available, the json file must be generated manually with the above-mentioned format with each type of devices in an individual block. Refer the below sample devices.json file 8

9 Installing and Configuring The following section provides information for installing and configuring the Mediation Pack on your IBM Operations Analytics Predictive Insights system. Mediation Packs Contents The following directories are contained in the Mediation Pack: BMC_MedPack_v1.2.0.zip /config Pack organization /config /docs /license /bin /pamodel /etc /sample Directory for configuration files if needed by the Mediation Pack. This directory contains the BMC_config file which is needed for the java utility. /docs Users Guide for deploying and configuring the Mediation Pack. Including a readme text file. BMC_MediationPack_UserGuide_v1.2.0.pdf This is a deployment procedural guide to deploy the BMC TSIM mediation pack into the PI environment. BMC_MediationPack_v1.2.0._ReadMe /license Text license files for governing the use of the Mediation pack. /bin Any binary compiled files needed by procedures in the Mediation Pack. BMCMediationUtility.jar This file contains the java code which retrieves the BMC TSIM metrics using its REST API, which receives the metrics in json format and then converts them into a.csv file for every five minute intervals. 9

10 Executing Jar: The jar file need to be executed with all the required parameters from the configuration file. The execution of the jar file must be validated as below example. The jar file should be prefixed with java path. /path to java bin folder/java BMCMediationUtility.jar -Dprotocol=$protocol - Dhostname=$hostname -Dport=$port -Dusername=$username -Dpassword=$password - Dlatency=$latency -Dfilepath=$filepath -DfilterFilePath=$filterFilePath - DdeviceFilePath=$devicefilepath -DmetricFilePath=$metricfilePath BMC.sh This file is the shell script that executes the BMCMediationUtility.jar The script file is used in a cron job for executing the jar files in a regular interval of five minutes. The shell script in turn reads the contents of the configuration file and appends all the required params to the jar file. The parameters should be enclosed in double quotes if shell script is executed on a linux environment. For example Java BMCMediationUtility.jar -Dprotocol= $protocol -Dport= $port Like shown above, include all the values with double quotes. Also the content of the script file is commands to read the values from the config file and to call the jar file with java and all the required params. /pamodel Core files used in the Mediation Pack representing the Predictive Insights model consisting of all attributes and metric definitions. BMC_2.pamodel The BMC_2.pamodel file is a PI model file which consists of all attributes and metric definitions. This file consists of all metric group definitions. Import this file into mediation tool. /sample Contains a sample CSV file representing the captured data streams that can be used to try the Mediation Pack output in the Operations Analytics Predictive Insights product. There is one CSV file per group per interval. A group is a collection of data like the metrics that exist in a database table or csv exported to one group. Groups provide a way to distinguish a metric like Response Time which is ambiguous, and frequently used in many contexts. What is an interval? The time for which the data is collected. Typically, data is collected every five or fifteen minutes. File naming format: The filename format can be described using the following regular expression: ([a-za-z]\w*) ([0-9\.]+)\.csv Example: MYTABLE csv Which is composed of three regular expression capturing groups: 1. The first is the name of the group containing only characters: a-za-z_0-9 10

11 2. The second is the start time of the interval in UTC. The time formats for the start and end times should follow the java SimpleDateFormat: yyyymmdd.hhmm Commented [RC1]: Example: #"_time", "rnc_name","node_name", "response time", "CSSR" , rnc123,abcczstr1, 1636, , rnc123,abcczstr2, 9320, , rnc456,abcrostr2, 973,

12 Configure the Mediation Pack Use the following steps to add the Mediation Pack to your Operations Analytics Predictive Insights system and configure the required views, modeling and data stream training. Configure BMC TSIM Metric Collection BMC TSIM Java Utility A java utility is used for fetching the data using REST apis in regular intervals. The utility contains different methods for each metric group mentioned below. The data is fetched in json format and converted into csv file into the specified path. To run the java utility you will need to have the following prerequisites in place. IBM JRE 1.7 and 1.8. Oracle JRE 1.8 and 1.9. Configuration of the BMC TSIM Java Utility A configuration file named BMC_config is included in the /config directory of the unzipped BMC_MedPack_v1.2.0.zip file. Before you execute the BMC TSIM Java utility you must edit this file to set the correct parameters that are read by the java utility program. The following configuration items in the BMC_config file that must be set. protocol= (must be set to either http or https depending on if ssl is utilized) Hostname= (must be set to the ip address of the BMC TSIM server) port= (must be set to the port number that is used for the API call) apikey= (must be set to the BMC api key if not using the username and password authentication) username= (must be set to the username for the API access to BMC TSIM) password= (must be set to the password of the username used for API access) filepath= (must be set to the path where the csv files are to be written) jarpath=(must be set to the file path where the BMCMediationUtility.jar resides. ) devicefile = (a file with all the device list) metricfile (a file with all the required metrics to be polled) Preparing the BMC TSIM Java Utility 1. Upload the BMC_MedPack_v1.2.0.zip to the Operations Predictive Insights Server. 2. Uncompress the BMC_MedPack_v1.2.0.zip. A script called BMC.sh has been created for you located in the /bin directory of the unzipped pack contents. You must make sure that the path to your java jre is set correctly. 12

13 Running the BMC TSIM Java Utility to collect metrics The BMC TSIM java utility needs to be executed at regular intervals to fetch data for all the required metric groups. In order to accomplish regularly scheduled execution of the java utility, cron is used. Setting Up a Cron Job Using crontab Step1: Open a Terminal Window (Command Line) in Linux. Step 2: Give the shell script the correct permission. For example, if the script is called bmc.sh, set permission as follows: chmod 755 bmc.sh Step 3: Add new cron job to crontab: crontab e This opens the vi editor for you if your default editor is vi. You can set a different editor using the following: export EDITOR=emacs Create the cron command using the following syntax: */5 * * * * /path_to_bmc.sh/bmc.sh Change the directory to the path that you had defined in the BMC_config filepath parameter. In our example we will use the /home/scadmin/bmcdatafiles directory on the Operations Analytics Predictive Insights system. After the java utility has retrieved the metric information from BMC TSIM you should see the csv files created in the /home/scadmin/bmcdatafiles directory as shown below. The naming pattern will be metricgroupname_starttime of the interval 1. To insure that the data is populating properly in the csv files verify the csv files contents by selecting a csv file and run the cat command. (example: cat CPUprcrProcessorTimePercent_ csv An example of the cat command output appears below: 13

14 " timestamp","host name","cpuprcrprocessortimepercent" ,TPCSRV, ,TPCSRV, ,TPCSRV, ,TPCSRV, ,TPCSRV,26.49 Configure Data Modeling Use the following steps to configure the data modeling or as application setup the REST Interface. Note: This model comes with certain prediefined metric groups and metrics. If user s environment has different metrics or metric this model should be edited for proper data mediation. Please refer Predictive Insights user guide for further details about how to add metrics/metric groups to the model. 1. Copy the BMC_2.pamodel file to the Predictive Insight server where the mediation tool is installed. 2. Run $PI_HOME/bin/mediationtool.sh to launch mediation tool 3. Go to File menu and click on the import button. 14

15 4. Select the Predictive Insights Model in the menu and click on Next 5. Select the BMC_TSIM.pamodel file and click on ok 15

16 6. Click on finish to complete the import process. 16

17 7. Enter the BMC File System details as shown below. For further information please refer to the following url: min_guide/t_tasp_admin_createfilesystemdatasource.html To preview the data in the mediation tool click on the icon shown circled in red. The data appears at the bottom of the screen. 17

18 Setup the Predictive Insights Topic Use the following steps to define a topic for handling the data flow in the Mediation Pack. 1. Login into the Predictive Insight server as scadmin 2. Change to the $PI_HOME/bin directory 3. Create the topic using the./admin.sh tool as shown below../admin.sh create_topic BMCTSIM 4. Run command below to set the topic options../admin.sh set -t= BMCTSIM system.aggregation.interval 5. Note: Typically the aggregation interval will depend on data collection interval in the monitoring source. In the command above the interval is set for 5 minutes. It is required for Java client that the monitoring intervals are set to 5 minutes collection intervals. For more information on properties for the admin.sh refer to the following: in_guide/r_tsaa_adminguide_cli_show_instance_properties.html Refer to the following for the set parameter: in_guide/r_tsaa_adminguide_cli_set_property.html Refer to the following on the properties that can be set: in_guide/c_tsaa_adminguide_config_properties.html Deploy the Defined Model in Predictive Insights Follow these steps to deploy the defined model for representing the data flow in Predictive Insights. 1. Click on the model(s) to be deployed and multi-select if multiple models are to be deployed to the same topic. Then Right click on the selection and select the Deploy Model option. 2. Gather the hostname, database and port number of Predictive Insights database. Tip: grep connection. $PI_HOME/config/tasp.connection.properties 18

19 Deploy model BMC_2.pamodel to Predictive Insights system 19

20 3. Enter the password for scadmin and update the JDBC URL with the hostname and port number. Deploy model bmc_2.pamodel to Predictive Insights system hostname sss 4. Click the box for counting KPI counts. Click on Yes to deploy the model. 20

21 Test Mediation Pack and Data Training Process Follow these steps to start the training process and test the mediation pack 1. Login into the Predictive Insight server as scadmin 2. Change to the $PI_HOME/bin directory 3. Run /start.sh -t= BMCTSIM (to start the topic) 4. Start the training using the command below. Note: -s start time is based on your datasource collection. Command:./admin.sh run_extractor_instance t= BMCTSIM s= Review the Training Process 1. Once the training has started you will see the events below in the Active event list screen capture. 2. Login in to the console using the following url: 3. User name : ncoadmin 4. Enter the ncoadmin password 5. You can see list events showing status about training process in Active event list Review Results and Anomalies Follow these steps to review the detected anomalies 1. Login in to the console using the following url: 2. User name: ncoadmin 3. Enter the ncoadmin password 4. Navigate to the Detected Anomalies page 5. Review the Anomalies 21

22 6. Right click on an anomaly and select ServiceDiagnosis tool 22

23 7. Review the effected metrics and behavior of the metrics for the time period. 23

24 Support To get assistance from the IBM support team, you may open a Problem Management Record (PMR). The PMR needs to be opened for IBM Operations Analytics Predictive Insights or IBM Operations Analytics Advanced Insights indicating the. Describe the problem encountered providing the following information that must be attached to the PMR: Product Name: IBM Operations Analytics Predictive Insights or IBM Operations Analytics Advanced Insights Product Version: or higher Mediation Pack: Mediation Pack Version: Include the following: Sample extracted CSV files, which are being ingested into Predictive Insights. Attach the output of the command $PI_HOME/bin/collect.sh -t=<topicname>. Found in folder $PI_HOME/var/spool/topics/<topicname>/extracted/ Data source connectivity status screenshot Predictive Insights log files ($PI_HOME/log/topicname/) when the problem occurred while using the mediation pack. Use the command $PI_HOME/bin/collect.sh -t=<topicname> A screen shot of the error message that is generated while using the mediation pack. A specific use case scenario(s) where problem occurred while using the mediation pack. 24

25 Copyright IBM Corporation 2017 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. Howev er, it is the user's responsibility to evaluate and verify the operation of any non-ibm product, program, or service. IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing IBM Corporation North Castle Drive Armonk, NY U.S.A. For license inquiries regarding double-byte (DBCS) information, contact the IBM Intellectual Property Department in your country or send inquiries, in writing, to: Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan, Ltd , Shimotsuruma, Yamato-shi Kanagawa Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you. This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice. Any references in this information to non-ibm Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk. IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you. Licensees of this program who wish to have information about it for the purpose of enabling: (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged, should contact: IBM Corporation 2Z4A/ Burnet Road Austin, TX U.S.A. 25

26 Such information may be available, subject to appropriate terms and conditions, including in some cases payment of a fee. The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement, IBM International Program License Agreement or any equivalent agreement between us. Any performance data contained herein was determined in a controlled environment. Therefore, the results obtained in other operating environments may vary significantly. Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems. Furthermore, some measurement may have been estimated through extrapolation. Actual results may vary. Users of this document should verify the applicable data for their specific environment. Information concerning non-ibm products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products and cannot confirm the accuracy of performance, compatibility or any other claims related to non-ibm products. Questions on the capabilities of non-ibm products should be addressed to the suppliers of those products. All statements regarding IBM's future direction or intent are subject to change or withdrawal without notice, and represent goals and objectives only. This information contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the examples include the names of individuals, companies, brands, and products. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental. If you are viewing this information in softcopy form, the photographs and color illustrations might not be displayed. Copyright License This information contains sample application programs in source language, which illustrate programming techniques on various operating platforms. You may copy modify, and distribute these sample programs in any form without payment to IBM, for the purposes of developing, using, marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore, cannot guarantee or imply reliability, serviceability, or function of these programs. The sample programs are provided "AS IS", without warranty of any kind. IBM shall not be liable for any damages arising out of your use of the sample programs. Privacy Policy Considerations IBM Software products, including software as service solutions, ( Software Offerings ) may use cookies or other technologies to collect product usage information, to help improve the end user experience, to tailor interactions with the end user, or for other purposes. In many cases, no personally identifiable information is collected by the Software Offerings. Some of our Software Offerings can help enable you to collect personally identifiable information. If this Software Offering uses cookies to collect personally identifiable information, specific information about this offering s use of cookies is set forth below. Depending on the configuration that is deployed in the application server, this Software Offering may use session cookies for session management. These cookies can optionally be enabled or disabled, but disabling the cookies will also disable the functionality that they enable. See the application server documentation for additional information. Optionally, you can also use functionality, such as typeahead, to store data locally on the browser. The data that is stored locally could contain personally identifiable information: Data validation. If the configurations that are deployed for this Software Offering provide you as customer the ability to collect personally identifiable information from end users by using cookies and other technologies, you should seek your own legal advice about any laws that are applicable to such data collection, including any requirements for notice and consent. For more information about the use of various technologies, including cookies, for these purposes, see IBM s Privacy Policy at and IBM's Online Privacy Statement at in the section entitled Cookies, Web Beacons and Other Technologies and the "IBM Software Products and Software-as-a-Service Privacy Statement" at Trademarks IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at Copyright and trademark information at 26

27 Adobe, Acrobat, PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, other countries, or both. Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/or its affiliates. Linux is a trademark of Linus Torvalds in the United States, other countries, or both. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. UNIX is a registered trademark of The Open Group in the United States and other countries. Other product and service names might be trademarks of IBM or other companies. 27