IBM Tivoli Netcool Performance Manager Wireline Component Document Revision R2E1. Troubleshooting Guide

Transcription

1 IBM Tivoli Netcool Performance Manager Wireline Component Document Revision R2E1 Troubleshooting Guide

2 Note Before using this information and the product it supports, read the information in Notices on page 175. Copyright IBM Corporation 2011, US Government Users Restricted Rights Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

3 Contents About this publication v Intended audience v What this publication contains v Tivoli Netcool Performance Manager - Wireline Component v Network and Service Assurance community on Service Management Connect vi Tivoli technical training vii Support information vii Conventions used in this publication vii Typeface conventions vii Chapter 1. Troubleshooting Tivoli Netcool Performance Manager Troubleshooting a problem Troubleshooting checklist for Tivoli Netcool Performance Manager Known problems and solutions Troubleshooting tasks Real-time charts do not work as expected Error: SQLCODE -542 :unique constraint (PV_ADMIN.PK_SEGM) violated MDE memory constraint Incomplete SNMPv3 metric collection After upgrading Tivoli Common Reporting, it is not possible to log into Tivoli Integrated Portal.. 6 Collectors swapping from idle to running at startup Specifying the database home directory path in the non-default DB2 installation Changing DB_USERNAME for installing UBA Alcatel-Lucent 5620 SAM Searching knowledge bases Chapter 2. Logs (Wireline Component) 11 Overview Logs by component Installation log files DataChannel logs DataLoad logs DataMart logs DataView logs Database log Logs messages format Logging configuration and information utilities.. 17 DataChannel logs configuration DataView logs configuration statget utility Configuring trace and logging Default logging level Trace logging for DataView The configure command Troubleshooting Event IDs Chapter 3. Contacting IBM support.. 25 Exchanging information with IBM Sending information to IBM Support Chapter 4. Introduction SNMP Inventory Overview Discovery Metrics and Properties Inventory Synchronization and Change Management Change Management for Elements Change Management for Sub-Elements Grouping Sub-Elements Where to Go From Here Chapter 5. SNMP inventory troubleshooting Overview Discovery Troubleshooting Discovery Does Not Start Discovery Starts But Issues Warning Messages.. 39 Discovery Seems to Hang or Never Finishes.. 46 Synchronization Troubleshooting Synchronization (Elements) Synchronization (Sub-elements) Grouping Troubleshooting Monitoring the Tivoli Netcool Performance Manager Log File Tivoli Netcool Performance Manager Log Messages 58 Burned subelements Scenario 1 - Instance Shift Causes Disconnect.. 58 Scenario 2 - Instance Shift Causes Burn Where to Go From Here Chapter 6. SNMP inventory management Regular monitoring Routine SNMP inventory management tasks Finding Elements and subelements about to reach their retry limit Finding Elements and Sub-elements That Have Been Retired Where to go from here Chapter 7. Messages DataChannel error messages DataLoad error messages DataView operational messages Notices Copyright IBM Corp. 2011, 2013 iii

4 iv IBM Tivoli Netcool Performance Manager: Troubleshooting Guide

5 About this publication Intended audience IBM Tivoli Netcool Performance Manager consists of two main components: a wireline component (formerly Tivoli Netcool/Proviso), and a wireless component (formerly Tivoli Netcool Performance Manager for Wireless). It is assumed that you are a network administrator or operations specialist having the knowledge of database management. The Troubleshooting Guide helps you to troubleshoot problems with Tivoli Netcool Performance Manager. The audience for this guide is the Tivoli Netcool Performance Manager administrator who is trying to troubleshoot problems with Tivoli Netcool Performance Manager components. What this publication contains This publication contains the following sections: v Chapter 1, Troubleshooting Tivoli Netcool Performance Manager, on page 1 v Chapter 2, Logs (Wireline Component), on page 11 v Chapter 3, Contacting IBM support, on page 25 v Chapter 4, Introduction SNMP Inventory, on page 27 v Chapter 5, SNMP inventory troubleshooting, on page 31 v Chapter 6, SNMP inventory management, on page 61 v Chapter 7, Messages, on page 65 Tivoli Netcool Performance Manager - Wireline Component IBM Tivoli Netcool Performance Manager consists of a wireline component (formerly Netcool/Proviso) and a wireless component (formerly Tivoli Netcool Performance Manager for Wireless). Tivoli Netcool Performance Manager - Wireline Component consists of the following subcomponents: v v v v v DataMart is a set of management, configuration, and troubleshooting GUIs. The Tivoli Netcool Performance Manager System Administrator uses the GUIs to define policies and configuration, and to verify and troubleshoot operations. DataLoad provides flexible, distributed data collection and data import of SNMP and non-snmp data to a centralized database. DataChannel aggregates the data collected through Tivoli Netcool Performance Manager DataLoad for use by the Tivoli Netcool Performance Manager DataView reporting functions. It also processes online calculations and detects real-time threshold violations. DataView is a reliable application server for on-demand, web-based network reports. Technology Packs extend the Tivoli Netcool Performance Manager system with service-ready reports for network operations, business development, and customer viewing. Copyright IBM Corp. 2011, 2013 v

6 The following figure shows the different Tivoli Netcool Performance Manager modules. Tivoli Netcool Performance Manager documentation consists of the following: v Release notes v Configuration recommendations v User guides v Technical notes v Online help The documentation is available for viewing and downloading on the information center at com.ibm.tnpm.doc/welcome_tnpm.html. Network and Service Assurance community on Service Management Connect Connect, learn, and share with Service Management professionals: product support technical experts who provide their perspectives and expertise. Access Service Management Connect at servicemanagement/nsa/index.html. Use Service Management Connect in the following ways: v Become involved with transparent development, an ongoing, open engagement between other users and IBM developers of Tivoli products. You can access early designs, sprint demonstrations, product roadmaps, and prerelease code. vi IBM Tivoli Netcool Performance Manager: Troubleshooting Guide

7 Tivoli technical training v v v Support information Connect one-on-one with the experts to collaborate and network about Tivoli and the Network and Service Assurance community. Read blogs to benefit from the expertise and experience of others. Use wikis and forums to collaborate with the broader user community. For Tivoli technical training information, see the following IBM Tivoli Education website at If you have a problem with your IBM software, you want to resolve it quickly. IBM provides the following ways for you to obtain the support you need: Online Access the IBM Software Support site at support/probsub.html. IBM Support Assistant The IBM Support Assistant is a free local software serviceability workbench that helps you resolve questions and problems with IBM software products. The Support Assistant provides quick access to support-related information and serviceability tools for problem determination. To install the Support Assistant software, go to support/isa. Troubleshooting Guide For more information about resolving problems, see the problem determination information for this product. Conventions used in this publication Several conventions are used in this publication for special terms, actions, commands, and paths that are dependent on your operating system Typeface conventions This publication uses the following typeface conventions: Bold Italic v v v v v v Lowercase commands and mixed case commands that are otherwise difficult to distinguish from surrounding text Interface controls (check boxes, push buttons, radio buttons, spin buttons, fields, folders, icons, list boxes, items inside list boxes, multicolumn lists, containers, menu choices, menu names, tabs, property sheets), labels (such as Tip:, and Operating system considerations:) Keywords and parameters in text Citations (examples: titles of publications, diskettes, and CDs Words defined in text (example: a nonswitched line is called a point-to-point line) Emphasis of words and letters (words as words example: "Use the word that to introduce a restrictive clause."; letters as letters example: "The LUN address must start with the letter L.") About this publication vii

8 v New terms in text (except in a definition list): a view is a frame in a workspace that contains data. v Variables and values you must provide:... where myname represents... Monospace v Examples and code examples v File names, programming keywords, and other elements that are difficult to distinguish from surrounding text v Message text and prompts addressed to the user v Text that the user must type v Values for arguments or command options Bold monospace v Command names, and names of macros and utilities that you can type as commands v Environment variable names in text v Keywords v Parameter names in text: API structure parameters, command parameters and arguments, and configuration parameters v Process names v Registry variable names in text v Script names viii IBM Tivoli Netcool Performance Manager: Troubleshooting Guide

9 Chapter 1. Troubleshooting Tivoli Netcool Performance Manager Troubleshooting a problem Troubleshooting is a systematic approach to solving a problem. The goal of troubleshooting is to determine why something does not work as expected and how to resolve the problem. The first step in the troubleshooting process is to describe the problem completely. Problem descriptions help you and the IBM technical-support representative know where to start to find the cause of the problem. This step includes asking yourself basic questions: v What are the symptoms of the problem? v Where does the problem occur? v When does the problem occur? v Under which conditions does the problem occur? v Can the problem be reproduced? The answers to these questions typically lead to a good description of the problem, which can then lead you a problem resolution. What are the symptoms of the problem? When starting to describe a problem, the most obvious question is What is the problem? This question might seem straightforward; however, you can break it down into several more-focused questions that create a more descriptive picture of the problem. These questions can include: v Who, or what, is reporting the problem? v What are the error codes and messages? v How does the system fail? For example, is it a loop, hang, crash, performance degradation, or incorrect result? Where does the problem occur? Determining where the problem originates is not always easy, but it is one of the most important steps in resolving a problem. Many layers of technology can exist between the reporting and failing components. Networks, disks, and drivers are only a few of the components to consider when you are investigating problems. The following questions help you to focus on where the problem occurs to isolate the problem layer: v v Is the problem specific to one platform or operating system, or is it common across multiple platforms or operating systems? Is the current environment and configuration supported? If one layer reports the problem, the problem does not necessarily originate in that layer. Part of identifying where a problem originates is understanding the environment in which it exists. Take some time to completely describe the problem Copyright IBM Corp. 2011,

10 environment, including the operating system and version, all corresponding software and versions, and hardware information. Confirm that you are running within an environment that is a supported configuration; many problems can be traced back to incompatible levels of software that are not intended to run together or have not been fully tested together. When does the problem occur? Develop a detailed timeline of events leading up to a failure, especially for those cases that are one-time occurrences. You can most easily develop a timeline by working backward: Start at the time an error was reported (as precisely as possible, even down to the millisecond), and work backward through the available logs and information. Typically, you need to look only as far as the first suspicious event that you find in a diagnostic log. To develop a detailed timeline of events, answer these questions: v Does the problem happen only at a certain time of day or night? v How often does the problem happen? v What sequence of events leads up to the time that the problem is reported? v Does the problem happen after an environment change, such as upgrading or installing software or hardware? Responding to these types of questions can give you a frame of reference in which to investigate the problem. Under which conditions does the problem occur? Knowing which systems and applications are running at the time that a problem occurs is an important part of troubleshooting. These questions about your environment can help you to identify the root cause of the problem: v Does the problem always occur when the same task is being performed? v Does a certain sequence of events need to occur for the problem to surface? v Do any other applications fail at the same time? Answering these types of questions can help you explain the environment in which the problem occurs and correlate any dependencies. Remember that just because multiple problems might have occurred around the same time, the problems are not necessarily related. Can the problem be reproduced? From a troubleshooting standpoint, the ideal problem is one that can be reproduced. Typically, when a problem can be reproduced you have a larger set of tools or procedures at your disposal to help you investigate. Consequently, problems that you can reproduce are often easier to debug and solve. However, problems that you can reproduce can have a disadvantage: If the problem is of significant business impact, you do not want it to recur. If possible, re-create the problem in a test or development environment, which typically offers you more flexibility and control during your investigation. v Can the problem be re-created on a test system? v Are multiple users or applications encountering the same type of problem? v Can the problem be re-created by running a single command, a set of commands, or a particular application? 2 IBM Tivoli Netcool Performance Manager: Troubleshooting Guide

11 Searching knowledge bases on page 9 You can often find solutions to problems by searching IBM knowledge bases. You can optimize your results by using available resources, support tools, and search methods. Troubleshooting checklist for Tivoli Netcool Performance Manager By answering a set of questions that are structured into a checklist, you can sometimes identify the cause of a problem and find a resolution to the problem on your own. Answering the following questions can help you to identify the source of a problem that is occurring with Tivoli Netcool Performance Manager: 1. Is your issue a known problem? 2. Is the configuration supported? 3. What are you doing when the problem occurs? v Installing, upgrading, or migrating the product v Doing administration tasks v Doing authorization tasks v Networking v Using the product 4. What, if any, error messages or error codes were issued? 5. If the checklist does not guide you to a resolution, collect additional diagnostic data. This data is necessary for an IBM technical-support representative to effectively troubleshoot and assist you in resolving the problem. Known problems and solutions Troubleshooting tasks A list of known problems and their solutions are described here. For a list of known problems, visit the following Web site: Known Issues with Tivoli Netcool Performance Manager Wireline Component Some troubleshooting tasks in Tivoli Netcool Performance Manager are described here. Real-time charts do not work as expected Symptoms Sometimes when you restart the Tivoli Netcool Performance Manager Channel Name Server (CNS) or Channel Manager (CMGR) component, your real-time charts no longer work as expected. Real-time charts require a valid reference to the real-time subscriber object. A restart of these components can cause the Channel Name Server (CNS) to no longer provide a valid reference to the real-time subscriber object. Resolving the problem To restore the proper operation of your real-time charts in this situation: User response: Chapter 1. Troubleshooting Tivoli Netcool Performance Manager 3

12 1. On the DataChannel host, change to the $DCHOME/bin directory. For example: cd /opt/datachannel/bin 2. Find the PID of the DataChannel CNS component../findvisual grep CNS_visual You can see an output like the following: pvuser Apr 14? 129:46 /opt/datachannel/bin/cns_visual -nologo /opt/datachannel/bin/dc.im -a CNS -f /o Note: The PID is the first number after your login ID. For example, in the previous output, the PID is Stop CNS by using the following command: kill -9 CNS_pid 4. Find the PID of the DataChannel CMGR component by using the following command:./findvisual grep CMGR_visual 5. Stop CMGR by using the following command: kill -9 CMGR_pid 6. On the DataView host, stop the Tivoli Integrated Portal server by using the following command: stopserver server1 -user tip_user -password tip_password 7. On the DataChannel host, change to the $DCHOME/bin directory. For example: cd /opt/datachannel/bin 8. Restart CNS and CMGR by running the following commands in the following order:./cnsw./cmgrw 9. On the DataView host, restart the Tivoli Integrated Portal server by using the following command: startserver server1 -user tip_user -password tip_password Error: SQLCODE -542 :unique constraint (PV_ADMIN.PK_SEGM) violated An unique constraint error is generated during inventory and grouping. Symptoms The following message is received in inventory and grouping output: -Error: SQLCODE -542:unique constraint (PV_ADMIN.PK_SEGM) violated -Reason: a static group membership link exists between a sub-elements and a group Causes A user uses the resmgr command to insert a resource into a group and then runs inventory grouping. If inventory grouping tries to place the same resource into the group, a unique constraint error is generated. Resolving the problem Using the resmgr command, delete the grouping link. MDE memory constraint An unique constraint error is generated when running an MDE query. Symptoms The following message is received when you attempt to run MDE query: 4 IBM Tivoli Netcool Performance Manager: Troubleshooting Guide

13 GYMVD0002E: Unable to retrieve information from the data source" error. Causes You are constrained by the available space in PV_LOIS_TEMP. This is a TEMP tablespace that is shared across all MDE sessions. A user uses the resmgr command to insert a resource into a group and then runs inventory grouping. If inventory grouping tries to place the same resource into the group, a unique constraint error is generated. Resolving the problem This error can be addressed with the following suggestions: v Reduce the time period used in the MDE query. If the time range is halved then the space required should be halved. v Reduce the number of metrics requested. If the original request was for 5 metrics and you can reduce this by 1 metric, it should require approximately 20% less space. v If neither option is possible, then you can increase the size of the PV_LOIS_TEMP tablespace. Incomplete SNMPv3 metric collection Some data loss can be observed from the Expected measures and Produced measures in the SNMP collector log message. Symptoms Some data loss can be observed from the Expected measures and Produced measures in the SNMP collector log message, especially when there is a sudden drop in Produced measures when compared with past hours of collection. For example: DL31066 I DL_PERF_SUMMARY Hour: <1s:hour> subelmts: <2s:ses> metrics: <3s:metrics> requests: <4s:requests> expmeas: <5s:expmeas> prodmeas: <6s:prodmeas> GYMDL30287W: SNMPJob jobdescr can not start in a dedicated thread. Internal error is err. Causes This is a thread-related issue that corresponds to insufficient resources to create another thread, or there is a system-imposed limit on the number of threads to be created. Resolving the problem This resolution is applicable only if the following warning message is displayed first to indicate that this is a thread-specific issue as there are many other configuration issues that can cause data losses. DL30287 W SNMPJOB_THREAD_ERR SNMPJob <1s:jobdescr> can not start in a dedicated thread. Interna If the system resources cannot be increased, decrease the concurrency in the collector. Note: This decreases the overall performance of the collector. Concurrency is controlled by the GLOBAL.SNMP.MAXASYNC parameter, which is given the default value of 256. The default value can be changed by adding a Chapter 1. Troubleshooting Tivoli Netcool Performance Manager 5

14 Custom Datachannel parameter in the TNPM topology. This parameter can be added using the Topology Editor: GLOBAL.SNMP.MAXASYNC=<value> where <value> is a number less than 256. After upgrading Tivoli Common Reporting, it is not possible to log into Tivoli Integrated Portal Symptoms After upgrading Tivoli Common Reporting, it is not possible to log into Tivoli Integrated Portal. Resolving the problem After upgrade of Tivoli Common Reporting, if you are experiencing problems logging into Tivoli Integrated Portal, do the following you need to and : User response: 1. Clear your Browser cache for the Tivoli Integrated Portal server For example: v In Firefox (v3.6): a. Click Tools > Options > Privacy b. Remove individual cookies. c. Search using your Tivoli Integrated Portal server name and remove those cookies. 2. Restart your browser. 3. log in to Tivoli Integrated Portal. Collectors swapping from idle to running at startup Symptoms The situation may occur that a collector, upon startup, waits a period of time, then transitions itself to running state only to be swapped to an idle state by the High Availability Manager (HAM). Causes At startup the Collector sits in 'Idle' state waiting to determine what it should do next ( if anything ). The HAM (High Availability Manager) should probe the Collector, discover its state, and instruct it to 'start' or to 'stay in idle' or to 'load' Start leads to running (active collection), load leads to ready (a sort of hot-spare mode ) and staying in idle means keep waiting. If a network disconnect occurred between the collector and HAM, the Collector may have a configuration (channel & collector number) from its previous run state. The result would be that the collector waits a period of time, then transitions itself to running state only to be swapped to an idle state by the HAM. The symptom is a result of the timing of the HAM's probing, and the Collector's idle-timeout. Resolving the problem User response: If you this behaviour a problem, you update either the polling interval of the HAM or the idle timeout of the collector: v Increase the Collector's Idle Timeout by editing DataChannels > Global DataChannel Properties > Advanced Properties > IDLETIMEOUT, or 6 IBM Tivoli Netcool Performance Manager: Troubleshooting Guide

15 v Decrease the HAM's probe interval by editing DataChannels > Administrative Components > High Availibility Managers > <HAM identifier> > Properties > POLL_INTERVAL Specifying the database home directory path in the non-default DB2 installation Change in the default value of DATABASE_HOME path does not update other required directory paths in the Topology Editor. Symptoms After you install DB2 in a non-default location, you change the default path of DATABASE_HOME directory while you add new host in the Topology Editor. However, it does not correctly update the other required directory paths. For example, following are the directory paths where the non-default DATABASE_HOME directory path is not updated. v DATABASE_HOME_ROOT in Host s properties v TARGET_DATABASE_HOME and DB2_BASE field in Database Configurations' properties Resolving the problem Following procedure shows how to update other required directory paths in the Topology Editor in non-default DB2 installation. User response: 1. The default DB2_BASE directory path is /opt/db2 For example, your DB2 home directory is /opt/db23 2. The default DATABASE_HOME directory path is opt/db2/product/10.1.0/. For example, the directory where you install DB2 is /opt/db23/product/ a. Install and start the Topology Editor. For more information about installing and starting the Topology editor, see IBM Tivoli Netcool Performance Manager Wireline Component: Installation Guide b. Create new topology. c. Ensure to specify the DATABASE_HOME path /opt/db23/product/ when you add a host. d. Continue to add required components into the topology. e. In Topology Editor, manually update the following values: v Update the value of DB2_BASE in Database Configurations' properties to /opt/db23 v Update the value of DATABASE_HOME_ROOT in host s properties to /opt/db23/product v Update the value of DATABASE_HOME in Database Configurations' properties to /opt/db23/product/ All the required directories in the Topology Editor get updated after you change the default DB2 directory path. Changing DB_USERNAME for installing UBA Alcatel-Lucent 5620 SAM Changing DB_USERNAME from PV_INVENTORY to PV_ADMIN in datachannel properties in xml file that is generated from Topology Editor. Chapter 1. Troubleshooting Tivoli Netcool Performance Manager 7

16 Symptoms When you install UBA Alcatel-Lucent 5620 SAM with default configurations, for SAMIF component DB_USERNAME is PV_INVENTORY. Since in DB2, user name with more than 8 characters cannot have system level access like working with sysibm tables, you need to change the user PV_INVENTORY to PV_ADMIN. Resolving the problem Following procedure shows how to change DB_USERNAME for UBA Alcatel-Lucent 5620 SAM in xml file. User response: 1. The default configuration for UBA Alcatel-Lucent 5620 SAM contains DB_USERNAME as PV_INVENTORY. 2. Save topology. 3. Open Topology xml file in an editor, for example, vi, search for PV_INVENTORY. Results are <DB_USERNAME>PV_INVENTORY </DB_USERNAME> Change this line to <DB_USERNAME> PV_ADMIN </DB_USERNAME>. 4. Save the xml file. 5. Open the same topology file in topology editor, it appears something as, 8 IBM Tivoli Netcool Performance Manager: Troubleshooting Guide

17 6. Run the deployer for installation. Searching knowledge bases You can often find solutions to problems by searching IBM knowledge bases. You can optimize your results by using available resources, support tools, and search methods. About this task You can find useful information by searching the information center for Tivoli Netcool Performance Manager, but sometimes you need to look beyond the information center to answer your questions or resolve problems. Procedure To search knowledge bases for information that you need, use one or more of the following approaches: v Search for content by using the IBM Support Assistant (ISA). ISA is a no-charge software serviceability workbench that helps you answer questions and resolve problems with IBM software products. You can find instructions for downloading and installing ISA on the ISA website. v Find the content that you need by using the IBM Support Portal. The IBM Support Portal is a unified, centralized view of all technical support tools and information for all IBM systems, software, and services. The IBM Support Portal lets you access the IBM electronic support portfolio from one place. You can tailor the pages to focus on the information and resources that you need for problem prevention and faster problem resolution. Familiarize yourself with the IBM Support Portal by viewing the demo videos ( about this tool. These videos introduce you to the IBM Support Portal, explore troubleshooting and other resources, and demonstrate how you can tailor the page by moving, adding, and deleting portlets. v Search for content about Product X by using one of the following additional technical resources: Tivoli Netcool Performance Manager technotes Chapter 1. Troubleshooting Tivoli Netcool Performance Manager 9

18 v v Tivoli Netcool Performance Manager Support Website Tivoli support communities (forums and newsgroups) Search for content by using the IBM masthead search. You can use the IBM masthead search by typing your search string into the Search field at the top of any ibm.com page. Search for content by using any external search engine, such as Google, Yahoo, or Bing. If you use an external search engine, your results are more likely to include information that is outside the ibm.com domain. However, sometimes you can find useful problem-solving information about IBM products in newsgroups, forums, and blogs that are not on ibm.com. Tip: Include IBM and the name of the product in your search if you are looking for information about an IBM product. 10 IBM Tivoli Netcool Performance Manager: Troubleshooting Guide

19 Chapter 2. Logs (Wireline Component) Overview Tivoli Netcool Performance Manager has various logs that can be used to examine processing results and problems. The following table provides a high-level description of Tivoli Netcool Performance Manager logs. Table 1. The Tivoli Netcool Performance Manager logs by component Component Description DataChannel DataChannel manages the primary Tivoli Netcool Performance Manager proviso.log log file. This log file collects data from each DataChannel component, and from the SNMP and BCOL collectors. DataChannel also creates the tnpmlog.log log file that contains all logs of interest to the user. DataLoad DataView DataMart Database Separate log files capture events about the daemon start-stop sequence, SNMP activity, and watchdog queries to determine that status of the daemon. Log data is also written to the primary DataChannel log file. Records data about web transactions and database calls. A number of log files containing messages related to inventory, internal Tivoli Netcool Performance Manager communications, and database status. The standard DB2 database log file. Logs by component A description of Tivoli Netcool Performance Manager logs organized by component. Installation log files A list of log files created by the Tivoli Netcool Performance Manager Wireline Components during installation. Where relevant, recommendations regarding the deletion of log files is provided. Database v /opt/proviso/* (do not delete these files as they are used during upgrades and for maintenance. Only delete these files on uninstallation.) v /var/tmp/pvinstall/install.cfg v /var/tmp/pvinstall/install.log DataChannel DataChannel installer logs to stdout and stderr so there are no logs resulting from the installation process. Copyright IBM Corp. 2011,

20 Tivoli Integrated Portal $home represents the root location for the Tivoli Integrated Portal. v Install logs $home/ia-tipinstall-xx.log $home/tcr13installtrace00.log $home/tcr13installmessage00.log v Uninstall logs $home//ia-tipuninstall-xx.log There are also logs created for the Deployment Engine, these are not entirely Tivoli Integrated Portal or Tivoli Common Reporting related. DataView v $home/dataview_installlog.log DataLoad v /tmp/dlsetup_install_`date +%Y.%m.%d` DataMart v /var/tmp/pvinstall/install.log Installation The log resulting from the installation of the Topology Editor is located at: <topology Editor install location>/topology_editor_installlog.log. The logs resulting from any run of the Deployer are created in /tmp/provisoconsumer. The main log file is: v /tmp/provisoconsumer/log.txt Deployment plan logs: v /tmp/provisoconsumer/plan/logs/[install_<time_stamp> Deployer Ant logs: v /tmp/provisoconsumer/plan/machineplan_<machine_name>/logs The best cleanup method for installation logs is to remove the /tmp/provisoconsumer directory when the system has been running long enough to validate all aspects of a working installation. A ProvisoConsumer directory is created in the /tmp directory of each server in the installation, not just for the primary host. Topology Editor logs are located at: <topology Editor install location>/topologyeditor/topologyeditortrace.log Note: A new log file is created each time the Topology Editor is run. 12 IBM Tivoli Netcool Performance Manager: Troubleshooting Guide

21 DataChannel logs The DataChannel logs include proviso.log, tnpmlog.log, walkback logs, and logs for individual DataChannel components. The default location of these logs is /DCHOME/log, where DCHOME is the location of DataChannel on your system. Note: DataChannel component logs are generated only when you enable dual-logging for the component. When enabled, dual-logging writes information to proviso.log and tnpmlog.log, and to the individual log for the DataChannel component. For information about dual-logging, see DataChannel logs configuration on page 18. Proviso.log proviso.log is the core Tivoli Netcool Performance Manager log file. It is always running and continuously processing data. The location of this log file is determined by the LOG.FILE configuration setting. For more information, see Topology Editor log settings on page 18. At the end of the day, a timestamp is added to the file name (for example, _ _proviso.log) and a new proviso.log file is created for current processing. The key performance indicators of Tivoli Netcool Performance Manager are derived from Proviso log files. The statistics are analyzed and an hourly system summary is provided per Tivoli Netcool Performance Manager component. For example, UTC UBA :12997 PERF_INPUT_PROCESSING GYMDC10021I SAM_INVENTORY-sam_inventory_ csv: processed 3145 records in sec ( records/min), rejected 0 records. This log file contains information about all DataChannel components and SNMP collectors. The types of events that are logged are determined by the LOG_FILTER setting, as described in Topology Editor log settings on page 18. tnpmlog.log tnpmlog.log contains a subset of the messages written to the proviso.log file. The subset of messages placed in the tnpmlog.log file are only those messages that are of interest to the user. The location of this log is determined by the LOG.LOG.FILE configuration setting (for more information, see Topology Editor log settings on page 18). DataChannel log format The current log format for log messages written to proviso.log and tnpmlog.log: <version and sequence number><utc time stamp><component>-pid:t ID <GYMXXXXXSeverity><Category><Message Text> Message Components Explanation of each component of the log format for the previous example: v <version and sequence number>: V1:1234 v <time stamp>: UTC v <Component>-PID:T ID: CME :3456 v <GYMXXXXXSeverity>:Unique ID format, for example GYMDC0412W, which consists of Chapter 2. Logs (Wireline Component) 13

22 TNPM product identifier: "GYM" Component: DC Message identifier: 0412 Severity level character: W=warning (can also be I=information, E=error) v Category: DISCARDED_RECORDS v Message Text: Got 4 duplicate records, Discarded 0, Total of 4 for mid: 2206 rid: If log messages do not have a message ID, then only the severity is shown. Log message example: V1: UTC CME :3456 GYMDC0412W DISCARDED_RECORDS Got 4 duplicate records, Discarded 0, Total of 4 for mid: 2206 rid: Walkback logs Walkback logs are generated when a Tivoli Netcool Performance Manager application encounters serious problems. In most cases, walkback logs are produced just before the application shuts down because of the error. These logs are crucial for problem determination by IBM Technical Support. The name of the file begins with walkback- and includes the DataChannel component and timestamp (for example, walkback-uba log). Unless you have advanced knowledge of the DataChannel, only the first few lines of a walkback log are useful. The following log entry is an example: EXCEPTION: DB2 not available shared memory realm does not exist SVR4 Error: 2: No such file or directory FACILITY_NAME: LDR Release: R2E2 Build: Pico.27 ORIGINATOR: 'a DB2ThreadedConnection( hsvcctx = 245F374 )' PARAMETER: OrderedCollection (a DB2 Error) TEXT: DB2 not available shared memory realm does not exist SVR4 Error: 2: No such file or directory Note: Walkback files must be manually deleted. Due to the DataChannel cron settings, a process that fails is repeated every 5 minutes. Since a new walkback log is generated upon every failure, the result can be many unneeded files. DataLoad logs DataLoad logs include the SNMP.log, pvmdmgr.log, and WatchDog.log logs. The default location of these logs is the /DLHOME/log directory, where DLHOME is the location of DataLoad on your system. SNMP log The SNMP log file contains detailed messages about all SNMP requests. All SNMP log files are created with the date in their name. Also, the name of the local SNMP log file also includes the collector number, for example: SNMP.1.1.log By default, SNMP logs include the following recurring events: v Close hour events v Debug level changes v Start and stop messages 14 IBM Tivoli Netcool Performance Manager: Troubleshooting Guide

23 Note: Events reported in this log are also reported to the DataChannel log. Only the current debug level can be set by using the Collector Information Tool as described in the IBM Tivoli Netcool Performance Manager: DataMart Operation Guide. On restart, the collector switches back to the permanent debug level (which by default is Fatal + Warning + Info messages). The permanent debug level is set through the Topology Editor. Changing the debug level is useful when a specific network device or device group does not respond correctly to SNMP requests. Pvmdmgr.log The pvmdmgr.log file stores events about the start and stop sequence of the daemon. Occasionally, the PVM Collecting daemon is running message is displayed before the process is in complete run mode. WatchDog logs The daily DataLoad WatchDog.log file contains entries about pings sent to the collector to ensure that the daemon is still running. The name of this log begins with the date for which events are records, for example WatchDog.log. DataMart logs DataMart does not have a central log. Instead, DataMart information is written to log files, such as logfile.pvm or TraceInventory.log, that are associated with individual components or actions, such as Inventory. The default location of these logs is the /DMHOME/log directory, where DMHOME is the location of DataMart on your system. Note: The inventory process does NOT automatically create a log. If inventory is runasacron entry, you must redirect the data to a specified log file, otherwise no log data is stored. To show time stamps in inventory log initiated by cron, you must add the following lines to the datamart.env file: PVM_LOG_DATE=1 export PVM_LOG_DATE TraceInventory.log This log is created when the SNMP Inventory GUI is used. The log file contains messages that sequentially indicate the processing status of an inventory. The following example shows a typical Discover_Analyze entry in this log. logfile.pollprofile.{collector ID} This log contains messages related to bulk inventory, and one file is produced for each bulk collector profile. The file suffix follows the format.bulk_n, where n is the number of the collector. Chapter 2. Logs (Wireline Component) 15

24 logfile.* This type of log file records minor events related to GUI or module function and is generated as required. Examples of this type of log file include logfile.pollinventory and logfile.resmgr. provisoinfod*.log This log contains messages about internal communications and is of limited use for troubleshooting. At the end of each day, a UNIX timestamp is appended to the file name (for example, provisoinfod log). NotifyDBSpace*.log file The NotifyDBSpace*.log file is a daily automatic file containing messages about the status of the database. At the end of each day, a UNIX timestamp is appended to the file name (for example, NotifyDBSpace log). DataView logs DataView does not have a central log. Instead, DataView writes DataView traces to the Tivoli Integrated Portal. Note: DataView log messages and configuration options are covered in the IBM Tivoli Netcool Performance Manager: DataView User and Administrator Guide, under Configuring trace and logging in Chapter 4: Administration tasks. Database log The Tivoli Netcool Performance Manager database uses the DB2 supplied log. DB2 has two logs. These logs can be used based on your requirement. Notification log Notification logs are readable. These logs log database activities/events, and also database start/stop events. The default file is instance_name.nfy (db2.nfy). Diagnostic log Diagnostic logs are not easy to read. These logs log all the logs from notification log plus some more error/warning logs. You can use them primarily for IBM support. The default file is db2diag.log. The file name for these logs rotates. For example, db2diag.n.log and <instance>.n.nfy files, where n is an integer. 0 is default, when DIAGSIZE parameter is set to non-zero. [db2@tnpminlnx10 db2dump]$ db2 get dbm cfg grep -i DIAGSIZE Size of rotating db2diag & notify logs (MB) (DIAGSIZE) = 9 Set the following path in DIAGPATH parameter. [db2@tnpminlnx10 db2dump]$ db2 get dbm cfg grep -i DIAGPATH Where, v Diagnostic data directory path DIAGPATH = /opt/db2/sqllib/db2dump/ v Current member resolved DIAGPATH = /opt/db2/sqllib/db2dump/ 16 IBM Tivoli Netcool Performance Manager: Troubleshooting Guide

25 Logs messages format v Alternative diagnostic data directory path ALT_DIAGPATH = alternative path is used when the primary diagnostic data path, DIAGPATH is unavailable. For more information, see index.jsp?topic=%2fcom.ibm.db2.luw.admin.trb.doc%2fdoc%2fc html In general, each log message indicates the date, time, Tivoli Netcool Performance Manager component, severity code, event ID, and event description. The following table describes log message elements. Table 2. Log message elements. Field Description Date and Time Timezone Component Severity Code Date and time using the following format: <year>.<month>.<day>-<hour>.<minute>.<second> Always UTC. Name of Tivoli Netcool Performance Manager component and its process ID separated by a dash (-). Names of Tivoli Netcool Performance Manager components are defined privately for each subsystem. DataChannel uses channel-based naming conventions (for example, CME.1.1), other subsystems can develop their own conventions. Some components include both the process ID and thread ID separated by a colon (for example, CME :415). Event severity code. For more information, see the description of the LOG_FILTER setting in Topology Editor log settings on page 18. Event ID Event identifier. For more information, see Event IDs on page 24. Description Description corresponding to the Event ID. Logging configuration and information utilities You can configure logging behavior or use information utilities to help troubleshoot Tivoli Netcool Performance Manager components. Chapter 2. Logs (Wireline Component) 17

26 DataChannel logs configuration Configuration options that govern DataChannel logging behavior are set by using the Topology Editor and are maintained in the database. Logging behavior can be set at three levels: DataChannel, specific DataChannel components, and all DataChannel components or global'. Logging settings are controlled by the log configuration. Table 3. DataChannel log configuration components. Level Description DataChannel Specify logging behavior for DataChannel, including the Channel Manager (CMGR), Channel Name Server (CNS), and Application Manager (AMGR). They override any conflicting options set using GLOBAL. To specify, use the following syntax: LOG.<option>=<value> where <option> = log configuration option (see XREF) <value> = value for the configuration option Example: LOG.ROOT_DIRECTORY=/opt/datachannel DataChannel Component Global Specify logging behavior for DataChannel components (UBA, FTE, CME, LDR, DLDR). They override any conflicting options set using GLOBAL. To specify, use the following syntax: <component>.<channel>.<collector>.<option>=<value> where <component> = 3- or 4-character string for the component (UBA, CME, FTE, LDR, DLDR) <channel> = DataChannel number. <collector> = Collector number. <option> = Configuration option (see XREF) <value> = Value Example: CME DUAL_LOGGING=true Specify logging behavior for all DataChannel components. To specify, use the following syntax: GLOBAL.<option>=<value> where <option> = Configuration option (see XREF) <value> = Value Example: GLOBAL.FC_RETENTION_HOURS=48 Logging configuration changes in Tivoli Netcool Performance Manager can only be made by the Installer. All components must be restarted to apply the changes. The settings include enabling/disabling central and local logging and tracing and changing the log levels for local/remote logging/tracing. This behavior is consistent with the previous releases. Topology Editor log settings Log options that can be specified in the Topology Editor are described in the following table. 18 IBM Tivoli Netcool Performance Manager: Troubleshooting Guide

27 Table 4. Log options in Topology Editor Option Levels Description DUAL_LOGGING GLOBAL and Component Use true or false to turn dual logging on or off. When set to false at the GLOBAL level, only DataChannel logs are generated. When set to true at the GLOBAL level, individual logs for all DataChannel components are generated, in addition to the DataChannel log. Default is false. Example: UBA DUAL_LOGGING=true LOG_PORT GLOBAL Port number of the log server for common log and trace files. Example: GLOBAL.LOG_PORT=25000 LOG_SERVER GLOBAL Host name of the log server for common log and trace log files. Example: GLOBAL.LOG_SERVER=cme4 MAX_LOGS GLOBAL Retention period for local trace files in days. Example: CME.1.1. MAX_LOGS=7 LOG_PORT GLOBAL Port number to use for logging. Example: GLOBAL.LOG_PORT=25000 LOG_SERVER GLOBAL Host name of the log server. Example: GLOBAL.LOG_SERVER=burlington.acme.com RENDER_MESSAGE_ GLOBAL Example: CME.1.1.RENDER_MESSAGE_ARGUMENTS=false ARGUMENTS SYSLOG_FACILITY GLOBAL Enter 128 to set the syslog facility to localhost. Example: GLOBAL.SYSLOG_FACILITY=128 Note: The syslog daemon must be running locally to have access to the log host on the network. FILTER or LOG_FILTER ALL The event types to log: v F = Failure (a hard process error) v E = Error (process termination or disk space problem) v W = Warning (frequent messages that require no specific action) v I = Information only messages v 1 = Level 1 debugging information v 2 = Level 2 debugging information v 3 = Level 3 debugging information Example: UBA LOG_FILTER=FE LOG_FILE LOG Name of the common log file. Example: LOG.LOG_FILE=/opt/datachannel/log/tnpmlog.log LOG_MAX_LOGS LOG Retention period for common log and trace files in days. Example: LOG.LOG_MAX_LOGS=7 LOG_RENDER_ MESSAGE_ ARGUMENTS LOG Write message parameters to log file. Example: LOG.LOG_RENDER_MESSAGE_ARGUMENTS=FALSE FILE LOG Name and location of the DataChannel log file. Example: LOG.FILE=/opt/datachannel/log/proviso.log TRAP_HOST LOG Host to send traps generated from log rules. Example: LOG.TRAP_HOST= TRAP_PORT LOG Port to send traps generated from log rules. Example: LOG.TRAP_PORT=162 SMTP_HOST LOG Host to send s generated from log rules. Example: LOG.SMTP_HOST= SMTP_PORT LOG Port to send s generated from log rules. Example: LOG.SMTP_PORT=162 Chapter 2. Logs (Wireline Component) 19

28 Table 4. Log options in Topology Editor (continued) Option Levels Description SMTP_TO LOG To address for s generated from log rules. LOG_FORWARD LOG Use true or false to enable/disable syslog forwarding. Example: LOG.LOG_FORWARD=false LOG_FORWARD_ FILTER LOG_FORWARD_ PORT LOG_FORWARD_ SERVER LOG LOG LOG Filter to use for forwarded log messages. Example: LOG.LOG_FORWARD_FILTER=FEWI 123 UDP port used by the host defined in LOG_FORWARD_SERVER. Example: LOG.LOG_FORWARD_PORT=514 Host name of the syslog server to forward log messages. Example: LOG.LOG_FORWARD_SERVER=localhost LOG_TRAP_HOST LOG Host where SNMP traps for specific types of log messages are sent. Example: LOG.LOG_TRAP_HOST=localhost Note: The rules file defining the message types must be installed and loaded. LOG_TRAP_PORT LOG Port used by the host defined in LOG_TRAP_HOST. Example: LOG.LOG_TRAP_PORT=162 Note: The rules file defining the message types must be installed and loaded. LOG_TRAPS If set to TRUE, traps sent by the CME as the result of threshold violations are added to the CME log file. Example: CME.1.1.LOG_TRAPS=TRUE MAX_LOGS ALL Maximum number of days to retain log files. Example: UBA MAX_LOGS=3 ROOT_DIRECTORY LOG Root directory where DataChannel logs are generated. The logs are located in thelog directory directly under this root. Example: LOG.ROOT_DIRECTORY=/opt/datachannel SUPPRESS_ TIMESTAMP_ ON_FORWARD LOG Suppresses the timestamp. Example: LOG.SUPPRESS_TIMESTAMP_ON_FORWARD=true Note: GLOBAL settings are used by all applications. LOG settings are application-specific settings that affect only that application. ALL settings can be used as both global and application-specific settings. DataView logs configuration Note: DataView log messages and configuration options are covered in the IBM Tivoli Netcool Performance Manager: DataView User and Administrator Guide, under Configuring trace and logging in Chapter 4: Administration tasks. statget utility statget is a utility that is located at each collector and provides DataLoad statistics like the statistics that are accessible from the Collector Information Tool GUI. statget can be run on any local server, and is located in the following default locations: v v ~/opt/dataload/bin/ ~/opt/datamart/bin/ 20 IBM Tivoli Netcool Performance Manager: Troubleshooting Guide

29 Note: For remote systems, you must use the Collector Information Tool. For more information, see the IBM Tivoli Netcool Performance Manager: DataMart Operation Guide. Syntax statget [-l {objects instances counters stats requests}] [-o <object>] [-i <instance>] [-c <counter>] [-D <debuglevel>] [-S <servername>] [-P <portnumber>] [-T <connecttimeout>] [-T2 <dialogtimeout>] [-?] [-v] Options Table 5. Options for statget utility Option Description [-l <objecttype>] <objecttype> is one of the following items: v objects - Lists all main classes of statistics counters. v instances - Lists all instances of a specific statistic class. v counters - Lists all counters names of a specific statistic class. v stats - Lists all counters values for a specific class, instance, and so on. v requests - List all requests configured inside the scheduler. The requests objecttype is the default objecttype if no -l option is specified. [-o <object>] <object> is a flag that filters the result set for -l stats and -l instances. -i <instance> <instance> is a flag that filters the result set for -l stats and -l counters. -c <counter> <counter> is a flag that filters the result set for -l stats. [-D <debuglevel>] <debuglevel> is a number from 0-6,where 0 specifies no debugging, and 6 specifies verbose debugging. [-S <servername>] <servername> is the name of the server that hosts a specific SNMP collector. If the flag is undefined, the PVM_SSDADDRESS environment variable is used. If the environment variable is also undefined, the value localhost' is used. [-P <portnumber>] <portnumber> is the number of the listening port number for the collector. If the flag is undefined, the PVM_SSDPORT environment variable is used. If the environment variable is also undefined, the value is used. [-T <connecttimeout>] <connecttimeout> is the amount of time permitted to establish a connection before a timeout. If the flag is undefined, a default value of 20 seconds is used. [-T2 <dialogtimeout>] <dialogtimeout> is the amount of time permitted for a connection response. If the flag is undefined, a default value of 7200 seconds (2 hours) is used. -? Displays the statget command reference page. -v Displays the build version string. Examples Table 6. Examples of statget usage Use Dump all pending and current SNMP requests. Syntax statget Chapter 2. Logs (Wireline Component) 21

30 Table 6. Examples of statget usage (continued) Use Syntax Get all classes of statistics counters. statget -l objects Get all possible instances of the class of statget -l instances -o Targets counters Targets'. Get values of all statistics counters of class statget -l stats -o Targets -i _Total Targets, for instance _Total'. Get all currently configured requests in statget -l requests DataLoad. Configuring trace and logging The default logging level can be set by using the configure command. You can manage logs and trace in the Tivoli Integrated Portal from the Websphere Administrative Console from the Settings > Websphere Administrative Console option. Default logging level You use the configure command to configure the Tivoli Integrated Portal logging level for Tivoli Netcool Performance Manager packages or components. You must restart the application server for your changes to take effect. When you have restarted the server, the logging level you have selected becomes the default logging level. Trace logging for DataView Note: DataView log messages and configuration options are covered in the IBM Tivoli Netcool Performance Manager: DataView User and Administrator Guide, under Configuring trace and logging in Chapter 4: Administration tasks. The configure command Configures the database connection information and the Tivoli Integrated Portal logging level for the Tivoli Netcool Performance Manager installation. You must restart the application server for your changes to take effect. When you have restarted the server, the logging level you have selected becomes the default logging level. Location <tip_location>/products/tnpm/dataview/bin Syntax configure.sh -tipuser <tip_username> -tippassword <tip_password> -type jdbc [-driverhome <driver_home>] [-jdbcurl <jdbc_url>] [-jdbcuser <jdbc_username>] [-jdbcpassword <jdbc_password>] configure.sh -tipuser <tip_username> -tippassword <tip_password> -type logging [-level <level>] [-package <package_name>] [-module <module_name>] 22 IBM Tivoli Netcool Performance Manager: Troubleshooting Guide

31 configure.sh -tipuser <tip_username> -tippassword <tip_password> -type debug [-state <on off>] Parameters <tip_location> The Tivoli Integrated Portal installation directory, by default /opt/ibm/tivoli/tipv2. <tip_username> A Tivoli Integrated Portal user name for the local Tivoli Integrated Portal. <tip_password> The Tivoli Integrated Portal user password for the local Tivoli Integrated Portal. <jdbc logging debug> The three types of configuration options. jdbc Configures the JDBC database connection information. logging Configures the Tivoli Integrated Portal logging level for the Tivoli Netcool Performance Manager installation. debug Configures the remote debugging. Optional parameters <driver_home> The JDBC driver location. <jdbc_url> The database URL. <jdbc_username> The database user name. <jdbc_password> The database password. <level> Set the level of logging detail: fatal, severe, warning, audit, info, config, detail, fine, finer, finest, or all. <package_name> Set logging for this software package. Wildcards * and? are supported. <module_name> Set logging for this software component. Wildcards * and? are supported. <on off> This is the remote debugging state. on The remote debugging state is on. off The remote debugging state is off. Examples The following command sets a Tivoli Integrated Portal logging level of detail. A wildcard selects all of the com.ibm.tivoli.tnpm.dal packages. Chapter 2. Logs (Wireline Component) 23

32 Troubleshooting Event IDs configure.sh -tipuser <tip_username> -tippassword <tip_password> logging -level detail -package com.ibm.tivoli.tnpm.dal.* The following command sets the JDBC URL to jdbc:db2:// host1.company.com:50000/pv: configure.sh -tipuser <tip_username> -tippassword <tip_password> jdbc -driverhome "/root/directory/tnpm.dataview" -jdbcurl jdbc:db2:// host1.company.com:50000/pv -jdbcuser <jdbc_username> -jdbcpassword <jdbc_password> You can use logs for a number of troubleshooting tasks. v Use DUAL_LOGGING to write component-specific log files in addition to proviso.log. v CME BUILD_TREE messages indicate whether Formula Requests are deployed and for how many subelements (Debug Level 2). v UBA SCANNED_INPUT, PERF_ACQUIRE_ALL, START_INPUT, and METRIC_STREAM_INFO messages indicate if metric input is being retrieved and processed. v UBA PERF_INVFLUSH messages indicate when discovered elements and subelements are written to the database. v Use StatGet to obtain SNMP DataLoad information. For information about Event IDs that are used in Tivoli Netcool Performance Manager logs, see Error Messages section in this guide.. 24 IBM Tivoli Netcool Performance Manager: Troubleshooting Guide

33 Chapter 3. Contacting IBM support IBM Support provides assistance with product defects, answering FAQs, and performing rediscovery. Before you begin After trying to find your answer or solution by using other self-help options such as technical notes, you can contact IBM Support. Before contacting IBM Support, your company must have an active IBM maintenance contract, and you must be authorized to submit problems to IBM. For information about the types of available support, see the Support portfolio topic in the Software Support Handbook. Procedure Complete the following steps to contact IBM Support with a problem: 1. Define the problem, gather background information, and determine the severity of the problem. For more information, see the Getting IBM support topic in the Software Support Handbook. 2. Gather diagnostic information. 3. Submit the problem to IBM Support in one of the following ways: v Using IBM Support Assistant (ISA): v Online through the IBM Support Portal: You can open, update, and view all your Service Requests from the Service Request portlet on the Service Request page. v By phone: For the phone number to call in your country, see the Directory of worldwide contacts web page. Results If the problem that you submit is for a software defect or for missing or inaccurate documentation, IBM Support creates an Authorized Program Analysis Report (APAR). The APAR describes the problem in detail. Whenever possible, IBM Support provides a workaround that you can implement until the APAR is resolved and a fix is delivered. IBM publishes resolved APARs on the IBM Support website daily, so that other users who experience the same problem can benefit from the same resolution. Exchanging information with IBM To diagnose or identify a problem, you might need to provide IBM Support with data and information from your system. In other cases, IBM Support might provide you with tools or utilities to use for problem determination. Copyright IBM Corp. 2011,

34 Sending information to IBM Support To reduce the time that it takes to resolve your problem, you can send trace and diagnostic information to IBM Support. Procedure To submit diagnostic information to IBM Support: 1. Open a problem management record (PMR). 2. Collect the diagnostic data that you need. Diagnostic data helps reduce the time that it takes to resolve your PMR. You can collect the diagnostic data manually or automatically: v Collect the data manually. v Collect the data automatically. 3. Compress the files by using the ZIP or TAR format. 4. Transfer the files to IBM. You can use one of the following methods to transfer the files to IBM: v IBM Support Assistant v The Service Request tool v Standard data upload methods: FTP, HTTP v Secure data upload methods: FTPS, SFTP, HTTPS v All of these data exchange methods are explained on the IBM Support site. 26 IBM Tivoli Netcool Performance Manager: Troubleshooting Guide

35 Chapter 4. Introduction SNMP Inventory This chapter provides an introduction to the Tivoli Netcool Performance Manager SNMP Inventory. Overview Tivoli Netcool Performance Manager allows the operator to decide how much the Tivoli Netcool Performance Manager DataMart will rely upon the OSS Inventory system. The Inventory system can be virtually anything from a full-featured commercial Inventory package, to an EMS or Node Manager like HP Open View, to a flat file like /etc/hosts. The minimum required is a list of the IP addresses of resources to monitor. Tivoli Netcool Performance Manager can discover both elements (resources that have an IP address, such as a router or a switch), and the sub-elements associated or contained with them, such as an interface or a port. Tivoli Netcool Performance Manager supports the following three modes of element and sub-element discovery: Mode Inventory Contains Tivoli Netcool Performance Manager Discovers 1 Nothing Elements, sub-elements 2 Elements Sub-elements 3 Elements, Sub-Elements Nothing Most Tivoli Netcool Performance Manager deployments are in mode two. In this mode, Tivoli Netcool Performance Manager imports a list of elements and then walks through the MIB to discover the sub-elements. In the first mode, Tivoli Netcool Performance Manager sweeps the network to discover the elements and their associated sub-elements. Discovery Tivoli Netcool Performance Manager's Discovery capabilities include some powerful and flexible tools that allow you to determine exactly what Tivoli Netcool Performance Manager will monitor, and how the sub-elements will be labeled and grouped. These capabilities make it possible to initiate automatically data collection, threshold monitoring, and reporting on discovered elements. Using a formula language, Tivoli Netcool Performance Manager can be configured to walk through an element's MIBs to discover particular MIBs representing users, tunnels, protocols, service classes or other sub-elements. Particular OIDs can be used to automatically create a label for the sub-element. Copyright IBM Corp. 2011,

36 For example, the sub-element label could be a combination of the element name, the interface, the port and the customer name, all taken from the MIB. Metrics and Properties In addition to the identifier of the sub-element and the metrics collected for it, Tivoli Netcool Performance Manager allows the operator to create any number of user-defined properties. There are two main differences between metrics and properties. Metrics come from a monitored resource and are used to calculate statistics that are the basis of performance reports and alarm thresholds. Metrics are generally numeric values that change frequently, like the number of packets transmitted or a resource's availability. Properties, by contrast, are values that change less frequently, such as the CIR (committed information rate) or the location of the element. Properties consist of metadata-like identifiers or labels for such things as the customer and/or the services using a particular sub-element. The values for properties can be discovered automatically from the monitored resource, or they can be imported from Inventory, provisioning or from another OSS component. Inventory Synchronization and Change Management Sub-element properties such as the CIR or customer name can change. Tivoli Netcool Performance Manager tracks the change and the time of the change, so that reports are displayed correctly. For example, utilization may be calculated against CIR. After the CIR is updated, reports must reflect the new value for utilization calculations. But reports that show dates prior to the CIR update must use the old CIR value. Tivoli Netcool Performance Manager manages this without error. If a sub-element is assigned to a new customer, the customer property will change. If the sub-element is in a particular customer's group, this can cause the sub-element to move to a new group. This can change the collection, alarm thresholds and reporting for that sub-element, automatically. Change Management for Elements The Inventory must track changes so that continuity of meta-data associated with the elements can be maintained. Unfortunately, Inventory is not as simple as sweeping a range of IP addresses to identify the network elements. That is just the beginning of the process. The Inventory must track changes so that continuity of meta-data associated with the elements (such as associations to customers, VPNs and services) can be maintained. At least one additional challenge remains to keep the element Inventory accurate, as shown with these two problem statements: v IP Address changes v Problem: If you are tracking a router by its IP address, and you discover a router at a new IP address, how do you know if it is a new router, or an existing router with a changed IP address? 28 IBM Tivoli Netcool Performance Manager: Troubleshooting Guide

37 v v Tivoli Netcool Performance Manager solves this problem by associating additional properties with each element which provide additional continuity and trace-ability in the face of IP address changes. These additional properties can be discovered from the device itself, like SNMP sysname, or gathered externally, like the name resolved from the IP address of the element's management interface. Name changes Problem: If you are tracking a router by its name, and you discover that the name has changed, how do you know if it is a new router with that IP address, or an existing router with a changed name? Tivoli Netcool Performance Manager does not track elements by their name or any other single property. Instead, by tracking a combination of properties, Tivoli Netcool Performance Manager is able to provide continuity to inventory even when any of these properties change. By automatically tracking changes to an element, rather than discovering it as a new element or forcing the operator to manually update the database, Tivoli Netcool Performance Manager helps reduce operating costs as follows: v v v v v Performance and trend reports for the element show the entire history of the element, without interruption. Changes to the element are shown in historical reports so they can be correlated to problems or changes in performance. Meta-data, such as location, community string, or other properties remains associated with the element, saving the operator from having to re-enter this. Inventory accuracy is improved because the update operation is automatic, not manual, eliminating errors. Inventory accuracy is improved because synchronization is automated, eliminating manual delays. Change Management for Sub-Elements In addition to the challenge of detecting and correctly managing changes on sub-elements, it is important to display this information correctly on reports. From an external (customer) point of view, subelement changes should be invisible. From an internal (network operations) perspective, the change must be visible. Tivoli Netcool Performance Manager manages all of this automatically. There are many reasons why the identifier (in SNMP, the Object Identifier, or OID) might change for a particular sub-element. Assuming that the sub-element is a port or virtual circuit residing on an interface, some of the changes will be due to failure and recovery scenarios, or network reconfigurations due to growth: v v v Adding or removing an interface card can cause the SNMP indexes to shift for other sub-elements. The interface the sub-element resides upon might fail, forcing the service associated with the sub-element to be moved to another interface. The service may be moved to a currently unused sub-element.the service may be moved to a sub-element in use, and the service currently on the sub-element is moved to another sub-element Most network changes should be invisible to customers. Their reports should reflect the quality of their service, and moves and changes to the network to Chapter 4. Introduction SNMP Inventory 29

38 Grouping Sub-Elements Where to Go From Here preserve their service should be invisible to them. This is particularly important for SLA reporting. You certainly want to avoid forcing the customer to view two reports, one for the original NIC and a second report for the replacement NIC. Throughout the network changes, network operations and engineering staff must have an accurate view of the actual sub-elements. For troubleshooting and capacity planning purposes, they should have a historical view of performance and traffic on a particular port, with information on changes that have occurred. Properties can be used to automatically group sub-elements. For example, sub-elements can be grouped according to technology, customer, service or site. Groups can be hierarchical, so it is possible to create structures like the following: v Site/Technology, to see all ATM SVCs in the New York POP. v v Customer/Service, to show all of the services a particular customer has subscribed to. Technology/Site, to see which sites are generating the most Frame Relay activity. Sub-elements can exist in multiple groups simultaneously. For example, a sub-element might be part of a network operations group and a particular customer's group. Relevant information. For information on troubleshooting tasks to perform after a new SNMP Inventory has been run, see See SNMP Inventory Troubleshooting. For information on periodic administrative tasks to perform, see See SNMP Inventory Management. 30 IBM Tivoli Netcool Performance Manager: Troubleshooting Guide

39 Chapter 5. SNMP inventory troubleshooting Overview This section discusses SNMP Inventory troubleshooting for Tivoli Netcool Performance Manager. The major phases of SNMP inventory. The Tivoli Netcool Performance Manager SNMP Inventory consists of the following three major phases, which usually happen sequentially: v SNMP Discovery Detects all resources on a target network and creates a virtual image of the network. v Synchronization Compares the virtual network image generated by the Discovery with the records in the Tivoli Netcool Performance Manager database that were created by the previous Inventory run. Any modifications (new, missing, or renamed resources, for example) are then synchronized through the application of various algorithms, and the new network image is written to the database. v Grouping Updates the grouping structure in the database, which determines the kind of information that is to be collected on each resource, element, sub-element, and so forth. In almost all cases, Tivoli Netcool Performance Manager's SNMP Inventory requires virtually no operator intervention. However, under certain circumstances, problems arise which you will need to address. The following sections discuss the more common problems you are likely to encounter and - where possible -- provide suggestions for remedial actions. It is strongly advised that you monitor the logs for potential error messages by doing one of the following: v Running a Discovery from the command line. If you run a Discovery from the command line, redirect STDER to a log file, as follows: inventory -nox -action discovery -name lowell >output 2>error_log For a complete list of error messages written to the Tivoli Netcool Performance Manager log file, see Messages section of this guide. For more information on using the Tivoli Netcool Performance Manager log file, see See Monitoring the Log File. v Running a Discovery from the DataMart GUI If you use the DataMart GUI to initiate a Discovery, error messages will appear on the DataMart GUI > Resource tab > Inventory Tool iconlive Information tab. Copyright IBM Corp. 2011,

40 Figure 1. Errors Displayed in the DataMart GUI The Inventory Tool prints out messages like the following every five seconds: 2005/12/09 13:46:52 [PL2DBS1, 238 sec, IP done.1/ SNMP done.1/ Elmt 0.1.0/ SubElmt 0.0.0] These messages explain the progress of the discovery as follows: IP done.1 Indicates that the IP phase of the discovery process has completed. SNMP done.1 Indicates that the SNMP phase of the discovery process has completed. Elmt Indicates that progress of discovered elements, using the following syntax:numberofobjectsininputqueue.numberofthreadsrunning.number OfElementsDiscovered SubElmt Indicates that progress of discovered sub-elements, using the following syntax: numberofobjectsininputqueue.numberofthreadsrunning.number OfSubElementsDiscovered If after two minutes there is no change in these messages, the Inventory Tool displays a more detailed message like the following: 2005/12/09 13:46:57 Current /12/09 13:46:57 Stage: IP done /12/09 13:46:57 Stage: SNMP done /12/09 13:46:57 Stage: Elmt /12/09 13:46:57 W: R00004/ /12/09 13:46:57 Stage: SubElmt The line that includes the run number and IP address (2005/12/09 13:46:57 W: R00004/ , for example) can be used to troubleshoot possible problems, as explained in See Discovery Seems to Hang or Never Finishes. 32 IBM Tivoli Netcool Performance Manager: Troubleshooting Guide

41 Discovery Troubleshooting The following sections address the more common problems that arise during Discovery. Discovery Does Not Start The following sections offer the most common solutions to problems with Discovery not starting. Discovery Fails Because Discovery Server Does Not Run What do to if the discovery fails. About this task If the Discovery server fails to start, an error message like the following is returned: IIOP: couldn t connect to :34024: couldn t open socket: connection refused Error: StartInventory Failed for Discovery Server : IDL:omg.org/CORBA/INTF_REPOS:1.0 {minor 0 completion_status COMPLETED_NO} To troubleshoot this problem, do the following: Procedure 1. Log in as pvuser on the system where the channel manager and log server are installed. 2. Change your working directory to the $DC_HOME/bin directory, by entering the following command. Note that $DC_HOME is defined as /opt/datachannel by default. cd $DC_HOME/bin 3. Verify that the Discovery server is not running by entering the following command: $ dccmd -action status -pattern DISC.*.* If the Discovery server is not running, the dccmd command returns output like the following: NUMBER FACILITY HOST STATUS ES DURATION EXTENDED STATUS 1 DISC unresponsive What to do next ACTION: If the Discovery server is not running, do the following: 1. Restart the Discovery server by entering a command like the following, specifying the Discovery server for your deployment (in this example we use DISC.DEV19.1 ): dccmd -action bounce -pattern DISC.DEV Verify that the Discovery server is running by entering the following command: dccmd -action status -pattern DISC.*.* If the Discovery server is running, the dccmd command returns output like the following: Chapter 5. SNMP inventory troubleshooting 33

42 NUMBER FACILITY HOST STATUS ES DURATION EXTENDED STATUS 1 DISC DEV19.QUALLA running 1 running For more information on using the dccmd command, see the Netcool/Proviso Command Line Interface Guide. Discovery Fails Because Collector Stops During Discovery About this task If the collector stops during a Discovery, several different error messages are logged. The most common error messages are the following: v Error: Aborted at March 14, :21:58 pm v Error: Connection refused v Error: Discovery Server : Status of lowell : invalid CLIENTERR [DC1] R00015 Connection refused (-I 682 -D 0 -profil lowell -collector dev19.quallaby.com:3002 -nbgetifaddress 100 -invfiletxt /opt/datamart/conf/inventory_subelements.txt -vname {} -intcollector 1) To troubleshoot this problem, do the following: Procedure 1. Log in as pvuser (or the user name that you specified during installation) on the system where DataMart is installed. 2. (Optional) Ensure that the DB2 database and Listener are running. For more information, see the Tivoli Netcool Performance Manager Installation Guide. 3. Enter the following command, replacing DATAMART_ROOT with the root DataMart directory ( /opt/datamart by default): DATAMART_ROOT/bin/pvm The DataMart GUI appears. Figure 2. DataMart GUI 4. Click on the Collector Information icon. The Collector Information Tool appears. 34 IBM Tivoli Netcool Performance Manager: Troubleshooting Guide

43 Figure 3. Collector Information Tool The Collectors Tab lists all the collectors loaded from the database. 5. Select the collector that you are troubleshooting. You look for the collector name in the output of the Live Information tab of the Discovery trace, as follows (in this example, the collector is dev19.quallaby.com): BeforeDiscovery for lowell : nothing to do Running Discovery on profile : lowell (mode 1) Enforce SE invariant uniqueness per element : active (from profile ) Duplicate IpAddress: active with collector number: 1 (dev19.quallaby.com:3002) (from profile ) Sub-element configuration file: /opt/datamart/conf/inventory_subelements.txt (from profile ) Discovery collector number: 1 (dev19.quallaby.com:3002) (from profile ) Start Reload formulas on dev19.quallaby.com: Chapter 5. SNMP inventory troubleshooting 35

44 Figure 4. Stopped Collector What to do next ACTION: Restart the collector by entering the following command on the Collector server: /opt/dataload/bin/pvmdmgr start Note: The collector cannot be restarted from the GUI. Discovery does not start because inventory is locked How to troubleshoot Tivoli Netcool Performance Manager if discovery does not start because inventory is locked About this task To troubleshoot this problem, do the following: Procedure 1. Log in as pvuser (or the user name that you specified during installation) on the system where DataMart is installed. 2. (Optional) Ensure that the DB2 database and Listener are running. For more information, see the Tivoli Netcool Performance Manager Installation Guide. 3. Enter the following command, replacing DATAMART_ROOT with the root DataMart directory ( /opt/datamart by default): DATAMART_ROOT/bin/pvm The DataMart GUI appears, as shown in See DataMart GUI. 4. Click on the Resource tab > Inventory Tool icon. The Inventory Tool appears. Figure 5. Inventory Tool 36 IBM Tivoli Netcool Performance Manager: Troubleshooting Guide

45 5. Click on the Live Information tab. If the profile is locked, you will see a warning message like the one displayed in See Inventory Tool. 6. To confirm that the profile is locked, return to the DataMart GUI and click on the Monitor tab->datamart Status icon. The DataMart Status Tool appears. Figure 6. DataMart Status Tool In this example, the Inventory profile lowell is locked by an Inventory process with a PID of Determine if the lock is valid and if the process is still running, by entering the following command, replacing LOCK_PROCESS_PID with the PID of the locking process (13458 in our example): ps -aef grep LOCK_PROCESS_PID What to do next ACTION: If the process is active, the lock is valid and you should wait until the Inventory completes before running another Inventory. ACTION: If the process is inactive, remove the lock and start another Inventory. To remove the lock, click on the Remove Locks icon the toolbar. located in the upper left of The qping Utility Has Incorrect Settings As part of Discovery in Inventory Mode 1, Tivoli Netcool Performance Manager scans IP ranges using the DataLoad qping utility. This utility will fail if group, user, and execute permissions are not set About this task The qping utility will fail with the following error messages if group, user, and execute permissions are not set correctly on the file: Inventory Tool Chapter 5. SNMP inventory troubleshooting 37

46 To troubleshoot this problem, do the following: Procedure 1. Log in as pvuser (or the user name that you specified during installation) on the system where the DataLoad server is installed. 2. Change your working directory to $PVMHOME/bin ( /opt/dataload/bin by default), and enter the following command to check the settings on the qping utility: cd $PVMHOME/bin ls -l./qping The correct settings are as follows: -r-sr-sr-- 1 root pvusers Jan 14 15:31 /opt/dataload/bin/qping 3. If the settings are not correct, su to root and enter the following commands: chown root $PVMHOME/bin/qPing chmod 6554 $PVMHOME/bin/qPing chgrp pvusers $PVMHOME/bin/qPing 4. Verify the settings by entering the ls -l command again Note: The qping utility and the DataLoad user must belong to the same group (in our example, pvusers ). The sticky bit setting ("s" in the ls -l line) allows the utility to be executed as root by any UNIX user belonging to the pvusers group. 38 IBM Tivoli Netcool Performance Manager: Troubleshooting Guide

47 Discovery Starts But Issues Warning Messages The following sections offer the most common solutions to warning messages that occur during Discovery. IP Addresses Are Rejected If an IP address is rejected during Discovery, errors will be written to the log or will display on the Information Tab of the Inventory Tool. About this task The errors created are like the following: Warning: IP Address rejected because this IP address has been excluded. Warning: IP Address rejected because this IP address has been excluded. In most cases, this is not a problem. To troubleshoot this error message, do the following: Procedure 1. Log in as pvuser (or the user name that you specified during installation) on the system where DataMart is installed. 2. (Optional) Ensure that the DB2 database and Listener are running. For more information, see the Tivoli Netcool Performance Manager Installation Guide. 3. Enter the following command, replacing DATAMART_ROOT with the root DataMart directory ( /opt/datamart by default): DATAMART_ROOT/bin/pvm The DataMart GUI appears, as shown in See DataMart GUI. 4. Invoke the Discovery Tool Wizard by doing the following: a. Select the Resource Tab->Inventory Tool icon. The Inventory Tool appears with the Configuration Tab displayed, as shown in See Inventory Tool Configuration Tab Chapter 5. SNMP inventory troubleshooting 39

48 Figure 7. Inventory Tool Configuration Tab b. Highlight a profile and then select Edit > Profile or click on the edit icon. The Inventory Tool Wizard appears, as shown in See Inventory Tool Wizard. Figure 8. Inventory Tool Wizard c. Click the Next button twice to navigate to the Discovery Tool Wizard, as shown in See Discovery Tool Wizard. 40 IBM Tivoli Netcool Performance Manager: Troubleshooting Guide

49 Figure 9. Discovery Tool Wizard 5. As shown in See Discovery Tool Wizard, check to see if the rejected IP addresses - either individually or within a specified range - have been intentionally excluded. If they have not, contact Micromuse support. Duplicate Elements Are Found If the same element, a router for example, has more than one IP address associated with it, Tivoli Netcool Performance Manager will "discover" the element multiple times, reject the duplicates discoveries, and write warning messages to the log. About this task See Rejected Duplicate Elements, for example, shows Tivoli Netcool Performance Manager rejecting three duplicate elements. Figure 10. Rejected Duplicate Elements This is expected behavior. If these warning messages appear often, Discovery performance may be degraded, since Tivoli Netcool Performance Manager must spend a lot of time calculating and eliminating duplicate elements. You may therefore want to exclude the duplicate addresses from the Discovery by doing the following: Chapter 5. SNMP inventory troubleshooting 41

50 Procedure 1. Invoke the Discovery Tool Wizard (DTW). For instructions on how to invoke the DTW, see See Invoke the Discovery Tool Wizard by doing the following:. 2. Add the duplicated IP addresses to the IP address exclude area of the DTW, as shown in the following figure: Figure 11. IP address exclude area of the DTW Elements Are Not Identified During Discovery If DataLoad did not receive an SNMP answer using the community name configured in the Tivoli Netcool Performance Manager profile, warning messages are created. About this task Warning messages like the following are written to the log and displayed on the Live Information Tab of the Inventory Tool: Warning: Unidentified Agents= { , , , , , , , , , } Warning: Unidentified Agents= { , , , , , , , , , } Devices may fail to respond for reasons like the following: v The device is not reachable from DataLoad collector (for example, no network route). v The SNMP agent was not started in the device. v Tivoli Netcool Performance Manager has the wrong SNMP community name for the device. 42 IBM Tivoli Netcool Performance Manager: Troubleshooting Guide

51 v v The device Access List is preventing Tivoli Netcool Performance Manager DataLoad to act as an SNMP Manager for this device. The firewall configuration is preventing SNMP traffic with the device. To troubleshoot this problem, follow these steps: Procedure 1. Log in as pvuser (or the user name that you specified during installation) on the system where DataMart is installed. 2. (Optional) Ensure that the DB2 database and Listener are running. For more information, see the Tivoli Netcool Performance Manager Installation Guide. 3. Change your working directory to DATAMART_ROOT/bin, replacing DATAMART_ROOT with the root DataMart directory ( /opt/datamart by default). 4. Perform an Internet Control Message Protocol (ICMP) query on the device, by entering the following command, replacing IP_ADDRESS_OF_UNIDENTIFIED_DEVICE with the IP address of the unidentified device, and NAME_OF_COLLECTOR_SERVER_DOING_PING with the name of the collector server doing the ping: qping IP_ADDRESS_OF_UNIDENTIFIED_DEVICE -S NAME_OF_COLLECTOR_SERVER_DOING_PING If successful, the qping command returns the IP address of the unidentified device, as follows: $ qping S dev :10 5. Connect to the device and verify whether or not the SNMP agent is enabled and running. 6. Connect to the device and verify the community name. 7. (Optional) You might have to change the community name in Tivoli Netcool Performance Manager, by using the DataMart->Inventory->Discovery Wizard to add an "alternate" community name to the profile. To change the community name, follow these steps: a. Invoke the Discovery Tool Wizard (DTW). For instructions on how to invoke the DTW, see See Invoke the Discovery Tool Wizard by doing the following:. b. Add an alternate community name in the specified text box. c. Click the Add button to confirm your choice, as shown in the following figure: Chapter 5. SNMP inventory troubleshooting 43

52 Figure 12. Discovery Tool Wizard Elements Skipped Because No Related Sub-Elements When Tivoli Netcool Performance Manager discovers an element but cannot discover any related sub-elements, warning messages are written to the log and displayed on the Information Tab of the Inventory Tool: About this task The warning messages are like the following: Skipping 3 elements (Set ( _jeffs _default _jeffs1 )) in output file, because they don t have related subelements This is not expected behavior, and should be resolved immediately. To troubleshoot this problem, follow these steps: Procedure 1. Log in as pvuser (or the user name that you specified during installation) on the system where DataChannel is installed. 2. Change your working directory to $DC_HOME/log ( /opt/datachannel/log by default), by entering the following command: cd $DC_HOME/log 3. Enter the following command to search the proviso.log for the Discovery formula of the skipped element, replacing EL_IP_AD, with the IP address of the skipped element: grep DISC proviso.log grep R00020 grep ELS_IP_AD grep CHKDISCOVERY 4. If he grep command returns no output. 44 IBM Tivoli Netcool Performance Manager: Troubleshooting Guide

53 If the grep command does not return output, the content of the following files in the $PVMHOME/conf directory ( $PVMHOME is defined as /opt/datamart by default): v $PVMHOME/conf/inventory_elements.txt v $PVMHOME/conf/inventory_sub_elements.txt Contact Micromuse support. 5. If the grep command returns output like the following: UTC DISC.DEV CHKDISCOVERY R00020/ Family:Generic~Agent, try discoveryformula Formula (Basic_Element string) UTC DISC.DEV CHKDISCOVERY R00020/ Family:1213_Device, try discoveryformula Formula (1213_Device ) UTC DISC.DEV CHKDISCOVERY R00020/ Family:IETF_IF, try discoveryformula Formula (IETF_IF ) The following may be attempted: a. If you believe that the device should in fact respond, try to discover the device manually and monitor the trace, using the DataMart->Metric- >Formula Editor, as shown in the following figure: Figure 13. Formula Editor b. Activate the relevant portion of the device MIB (for example SAA in the Cisco router). c. In some cases, SNMP attributes do not respond properly, and the Discovery formula for those devices must be changed. In this event, contact Micromuse support. Chapter 5. SNMP inventory troubleshooting 45

54 Discovery Seems to Hang or Never Finishes If the Discovery server shows no progress for several minutes it has probably encountered problems with a Discovery formula. About this task If the Discovery server shows no progress for several minutes (up to thirty minutes in some cases), as shown in the following trace log, it has probably encountered problems with a Discovery formula: Figure 14. Discovery formula Problems - real or apparent - with Discovery formulas can be the result of the following: v The SNMP agent is slow to respond. v The network latency is very long. v The Discovery formula is not well-suited to the SNMP agent. v Collector performance issues. When Tivoli Netcool Performance Manager is in such a state one of two things result: v The Discovery formula finally succeeds and the Inventory continues. v The Discovery hangs until the Inventory timeout occurs (two hours) when the following error message is written to the log file: Error: Profile is not progressing during the last 7201 seconds, aborting Contact Micromuse support and provide them with the DATACHANNEL_ROOT/ datachannel/log/[yyyy.mm.dd]snmp.log However, before contacting Micromuse support, and before the Discovery finishes or times out, do the following: Procedure 1. Determine which Discovery formula is causing the problem by doing the following: a. Log in as pvuser (or the user name that you specified during installation) on the system where DataChannel is installed. 46 IBM Tivoli Netcool Performance Manager: Troubleshooting Guide

55 b. Change your working directory to DATACHANNEL_ROOT/log, replacing DATACHANNEL_ROOT with the root DataMart directory ( /opt/datachannel by default). c. Enter the following command to search the proviso.log for the Discovery formula: grep DISC proviso.log grep R00010 tail UTC DISC.PVDEMO CHKDISCOVERY R00010/ Family:Cisco_CBQoS_Action, try discoveryformula Formula (Cisco_CBQoS_Action ) In this example, the Discovery formula is Cisco_CBQoS_Action. 2. Determine the internal request ID that is stuck by doing the following: a. Log in as pvuser (or the user name that you specified during installation) on the system where the DataMart or DataLoad server is installed. b. Change your working directory to DATAMART_ROOT/log, replacing DATAMART_ROOT with the root DataMart directory ( /opt/datamart by default). c. Enter the following command to find the internal request ID, replacing SERVER_NAME with the name of the Tivoli Netcool Performance Manager DataMart server: statget -S SERVER_NAME grep -i once + [33] ID 96627,{CAL none (ONCE)(next=2005/03/15 20:32:16)}(P3) ACTIVE (LastExec): ServiceForm:(Trgt=(string) )(Form=(form)Cisco_CBQoS_Action)(Inst=)(RComm=public) + [40] ID 96683,{CAL none (ONCE)(next=2005/03/15 20:48:19)}(P1) ACTIVE (LastExec): ServiceSTAT (LONG) In this example, the internal request ID is Important: If the command returns no output, contact Micromuse support. 3. Enable limited debugging on the collector server in order to populate the log with additional information by doing the following: a. Change your working directory to /opt/dataload/contribs. b. Enter the following command, replacing TASKID with the Internal request ID you found in See Determine the internal request ID that is stuck by doing the following:. dialogtest2 Debug 6.TASKID Tivoli Netcool Performance Manager outputs a message like the following: Set debug level to 6 for taskid Debug configuration: > Global Level= 1; Mask=FW > ID Level= 6; Mask=FWI1234 WARNING: DO NOT USE THE GLOBAL COLLECTOR DEBUGGER. Chapter 5. SNMP inventory troubleshooting 47

56 Synchronization Troubleshooting The following sections address the more common problems that arise during Synchronization. Synchronization (Elements) During the synchronization phase, Tivoli Netcool Performance Manager compares the virtual network image generated by the Discovery with the records in the Tivoli Netcool Performance Manager database that were created by the previous Inventory run. Any modifications (new, missing, or renamed resources, for example) are then synchronized through the application of various algorithms, and the new network image is written to the database. In order to track elements and sub-elements through subsequent inventories, Tivoli Netcool Performance Manager identifies them with a unique, never-changing identifier, called an invariant. The default element invariant, for example, is a concatenation of the following three attributes: v The MIB II sysname v v The first IP address responding to the ICMP scan (in Inventory mode 1) or the one given in the the mode 2 Inventory file The fist valid physaddress (MAC address) in the MIB II iftable See Synchronization Invariant Logic illustrates the logic used to determine if a "newly-discovered" element is actually new or is an existing element that has moved or changed. Synchronization Invariant Logic Element Attributes in Database MIB II sysname IP Address physaddress New Element If... Defined Defined Defined Two or three attributes have changed. Empty Defined Defined Two attributes have changed. Empty Empty Empty The resolved name is different from resolved name in the database. If an element does not respond to one of these three MIB II attributes, contact Micromuse support. To avoid synchronization errors, we strongly recommend that you limit device configuration changes to one attribute between two runs of an Inventory (Discovery and Synchronization). 48 IBM Tivoli Netcool Performance Manager: Troubleshooting Guide

57 Pre-synchronization Summary As Tivoli Netcool Performance Manager prepares for synchronization, it compiles a list of elements The list contains the following types of elements: v New v Updated v Burned v Unchanged v Not Found v Histo Reject See Pre-synchronization Summary shows the display written to the DataMart->Inventory Tool->Live Information tab, with this summary information highlighted: Figure 15. Pre-synchronization Summary The following sections discuss this summary in more detail. Identifying not found elements If the Pre-Synchronization summary lists any contact Micromuse support. An example of not found elements can be found in See Pre-synchronization Summary, which contains three examples. Chapter 5. SNMP inventory troubleshooting 49

58 Identifying burned elements How to identify burned elements In cases where the MIB II sysname, IP address, and physaddress are all defined for an element (the first row in See Synchronization Invariant Logic), Tivoli Netcool Performance Manager may encounter a special situation if the MIB II sysname and phyaddress have changed, but the IP address remains unchanged. In this special case, Tivoli Netcool Performance Manager creates a new element and writes it to the database. However, since the IP address for "both" elements is the same, so too is the resolved name, which means that the elements would be seen by Tivoli Netcool Performance Manager as the same element. To prevent this situation, Tivoli Netcool Performance Manager renames the initial elment with a "Burned" prefix, as shown in See Burned Element: Figure 16. Burned Element When Synchronization completes, a list of burned elements is written to the following file: PROFILE_HOME/PROFILE_NAME/synchro/e_burned.dat e_burned.dat file If you have an e_burned.dat file, contact Micromuse support. The burned elements and their sub-elements are duplicated, and the statistics attached to these resources are not continuous in reports. Preventing Burned Elements Tivoli Netcool Performance Manager relies quite heavily on the sysname when calculating invariants. When using sysname you need to prevent burned elements. About this task To prevent burned elements, so the following: 50 IBM Tivoli Netcool Performance Manager: Troubleshooting Guide

59 Procedure 1. Do NOT change the sysname on a device when the lowest physaddress has also changed. 2. Ensure that the sysname is defined and not left empty. 3. Ensure that the sysname is defined with a unique value. 4. Minimize any changes to the sysname once they have been set, unless you want Tivoli Netcool Performance Manager to see the device as new and interrupt the continuity of statistics. Detecting Too Many New Elements If there has been extensive configuration changes before an Inventory run (for example, if two attributes were changed simultaneously on a router), Tivoli Netcool Performance Manager will discover and create a new element in the database. About this task The consequences are severe: v The element and its sub-elements are duplicated. v The statistics attached to these resources are not continuous in the reports. v Double polling might happen on "both", until the "first" device is "retired" by the Inventory. Aside from limiting device configuration changes to one attribute between two runs of an Inventory, there is no way to prevent this situation from occurring. When Synchronization completes, a list of new elements is written to the following file. Note that $PVMHOME is defined as /opt/datamart, by default: $PVMHOME/importexport/PROFILE_NAME/synchro/new_e.dat Procedure 1. If you have a new_e.dat file, check to ensure that the number of new elements is not unexpected. 2. If the number is unexpected, contact Micromuse support. Note: A large new_e.dat file is to be expected the first time a network is discovered. Detecting Different Elements Resolved With the Same Name If the name resolution system (for example, DNS) assigns identical names to the IP addresses of different devices - a configuration mistake - Tivoli Netcool Performance Manager will an error during Synchronization. About this task The error will look like the following: Warning: IP Address rejected Warning: IP Address rejected When Synchronization completes, a list of rejected elements is written to the following file: PROFILE_HOME_DIRECTORY/PROFILE_NAME/synchro/duplicateElement_e.reject If you have a duplicateelement_e.reject file, do the following: Chapter 5. SNMP inventory troubleshooting 51

60 Procedure 1. From the DataMart server, resolve the name of both conflicting IP addresses, in order to confirm the identical naming resolution. 2. Fix the naming resolution system and start the same Inventory again. Do not delete incorrectly Named Elements That Define SNMP Community Names Configuring Tivoli Netcool Performance Manager involves associating elements with their SNMP Community name. About this task When initially configuring Tivoli Netcool Performance Manager, you use the SNMP Configuration Tool, as shown in See Configuring SNMP Community Names, to associate elements with their SNMP Community name so that Tivoli Netcool Performance Manager can read from the devices. Figure 17. Configuring SNMP Community Names The SNMP Configuration Tool writes this information to the database. During the next Inventory run, the synchronization process sees the elements that were newly entered into the database by the SNMP Configuration Tool as missing all three invariant attributes (see row three in See Synchronization Invariant Logic). 52 IBM Tivoli Netcool Performance Manager: Troubleshooting Guide

61 Tivoli Netcool Performance Manager then attempts to determine if a new element needs to be created by comparing the resolved names of the devices, with the following results: v v If the name that was entered into the SNMP Configuration Tool was the correct resolved name of the device, Tivoli Netcool Performance Manager will not create a new element and will instead update the attributes for the existing element with the discovered values. If the name that was entered into the SNMP Configuration Tool was the incorrect resolved name of the device, Tivoli Netcool Performance Manager will create a new element. On subsequent Discoveries, Tivoli Netcool Performance Manager will rediscover and eventually update this element and ignore the element with the incorrect resolved name. Procedure Do NOT remove the element with the wrong resolved name from the database. Results Tivoli Netcool Performance Manager continues to use that entry for SNMP Community name mapping. If the entry is removed, Tivoli Netcool Performance Manager will lose the SNMP Community mapping and will not be able to read from that resource. Failed Synchronization Pay particular attention to failed elements while monitoring the Synchronization process About this task While monitoring the Synchronization process, pay particular attention to failed elements. Chapter 5. SNMP inventory troubleshooting 53

62 Figure 18. Failed Elements and the Inventory Tool Procedure If Synchronization reports failed elements (greater than 0), contact Micromuse support immediately. Synchronization (Sub-elements) The sub-element invariant is defined in the Discovery Formulas referenced in the file $PVMHOME/conf/inventory_sub_elements.txt. Note that $PVMHOME is defined as /opt/datamart by default. This invariant is usually defined differently per object type (for example, an SAA probe) and vendor (Cisco, for example). The logic used to decide if a "newly-discovered" sub-element is in fact new, depends on the following: v If the invariant is not defined, the sub-element name (usually the concatenation of the element name and the SNMP instance) is used to determine if a new sub-element should be created or an existing one should be updated. Any change to the name between two subsequent Discovery runs triggers the creation of a new sub-element. 54 IBM Tivoli Netcool Performance Manager: Troubleshooting Guide

63 An SNMP index (instance) change will trigger the creation of a new sub-element (because the SNMP index is part of the name), which will cause reporting discontinuity. v If an invariant is defined, the parameter " Enforce Sub-Element Invariant uniqueness per element " is used, which is set in the DataMart > Inventory > Discovery Wizard, as shown in the following figure: If the option highlighted in the figure is checked, uniqueness is verified on a per element basis; if left unchecked, uniqueness is verified across the entire profile. Identical Sub-element Invariant If an invariant is defined for a particular type of sub-element (for example, Cisco SAA), Tivoli Netcool Performance Manager will check the uniqeness of that invariant during an SNMP Inventory. About this task If Tivoli Netcool Performance Manager detects two sub-elements with the same invariant, it displays an error message like the following: Warning: Identical invariant detected for invariant _CBQoS class-default-Match any -output-1 ( ,CBQoS<1107><1123> and ,CBQoS<1107><1177>). Note: When identical invariants are detected, neither sub-element is stored in the database. To resolve this situation, follow these steps: Procedure 1. Use the DataMart->Metric->Mib Browser utility to verify that the two invariants are in fact identical. 2. If they are identical, the following needs to be done: v The Discovery formula needs to be changed to ensure the uniqueness of either the element or the profile. This is usually done by concatenating more OIDs to the invariant. v The SNMP OID value, used for building the invariant, needs to be changed. For example, if you use the Cisco ifalias settable OID, its value might not be Chapter 5. SNMP inventory troubleshooting 55

64 unique (among other things, a configuration error may exist due to a non-unique Customer contract number, which should be fixed. v Discovery should then be re-triggered, so that both sub-elements can be discovered. 3. Contact Micromuse support for assistance in making these changes. Checking the TraceInventory.log File Viewing the summary information in the $PVMHOME/log/TraceInventory.log file. Note that $PVMHOME is defined as /opt/datamart by default. Figure 19. TraceInventory.log File You should review the $PVMHOME/log/TraceInventory.log file and check for the following: Large number of sub-elements not found: A large mount of "missing" sub-elements (discovered before, but not rediscovered), may be because a large portion of your network is not reachable. About this task If the same sub-elements are continually not found during subsequent Inventories, they will eventually be retired. For more information, see See Finding Elements and Sub-elements About to Reach Their Retry Limit. Procedure If your network is reachable, the sub-elements exist, and you are still receiving error messages, contact Micromuse support. Rejected Sub-elements: The log may report types of rejected sub-elements. Procedure If the log reports any of the following types of rejected sub-elements, contact Micromuse support: v histo reject v dup reject Burned sub-elements: Procedure for dealing with burned sub-elements. 56 IBM Tivoli Netcool Performance Manager: Troubleshooting Guide

65 About this task A burned sub-element is determined as follows: v The sub-element has an invariant in the database (for example, inv1). v The sub-element has a name (for example, name1). v During an Inventory, a new sub-element is discovered and created with the same name (for example, name1) and a different invariant for example, inv2). In this case, two different sub-elements are about to be created in the database with the same name. To prevent this duplication, Tivoli Netcool Performance Manager renames the initial sub-element with a "Burned" prefix, so that each sub-element can be stored in the database with a unique name. A list of burned sub-elements is written to the following file: PROFILE_HOME/PROFILE_NAME/synchro/burned_seinv.dat1 Procedure If the log reports any burned sub-elements, contact Micromuse support. Disconnected sub-elements: Procedure for dealing with disconnected sub-elements. About this task A disconnected sub-element is determined as follows: v The sub-element has an invariant in the database (for example, inv1). v The sub-element has a name (for example, name1). v During an Inventory, a sub-element is discovered with the same name (for example, name1) and an empty invariant. An empty invariant typically happens when an invariant is used to assign physical resources (ports, for example) to customers. In this case, the empty invariant is interpreted as a "disconnection" between the customer and the resource. If a sub-element is determined to be disconnected, no new sub-element is created, and the initial sub-element is marked as disconnected. A list of disconnected sub-elements is written to the following file: PROFILE_HOME/PROFILE_NAME/synchro/disconnected_seinv.dat1 Procedure If the log reports any disconnected sub-elements, contact Micromuse support. Chapter 5. SNMP inventory troubleshooting 57

66 Grouping Troubleshooting If all problems with Discovery and Synchronization have been solved - or none arise - it is unlikely that you will encounter problems with Grouping. Monitoring the Tivoli Netcool Performance Manager Log File The Tivoli Netcool Performance Manager log file is located in the DataChannelInstallROOT/log directory (/opt/datachannel/log by default) and should be monitored on a regular basis. Currently, only DataChannel and DataLoad write to the Tivoli Netcool Performance Manager log. Log files for other components can be found in $PVMHOME/log ( /opt/datamart/log, by default) and SILVERSTREAM_HOME/log ( /opt/silverstream/log, by default). You can write a program to do monitor the Tivoli Netcool Performance Manager log file (recommended) or you can check the Tivoli Netcool Performance Manager log file by hand, using the grep command. For example, the following command checks the Tivoli Netcool Performance Manager log file for a fatal loader error: grep CMGR proviso.log grep -w F CMGR F ERROR There was a problem loading the domain database Tivoli Netcool Performance Manager Log Messages Burned subelements For a complete list of log messages written to the Tivoli Netcool Performance Manager log, see the Troubleshooting Tivoli Netcool Performance Manager Guide. Reasons why a subelement (SE) can be in a `burned' state There are two reasons a subelement (SE) can be in a `burned' state. v If its Element is burned v An SE is not in Discovery and another SE with Invariant is discovered with the same Element and Instance Scenario 1 - Instance Shift Causes Disconnect Initial Discovery finds three sub-elements, all with Invariant. Elt dbindex Elt Name Instance SE dbindex SE Name Invariant State Missing Count alm elt01 <04> alm elt01.seinv04 I:01:04 on alm elt01 <05> alm elt01.seinv05 I:01:05 on alm elt01 <06> alm elt01.seinv06 I:01:06 on 0 During a subsequent Discovery the following occurs: v The SE with Invariant I:01:05 is not found v The SE with Invariant I:01:06 is found, but at Instance <05> 58 IBM Tivoli Netcool Performance Manager: Troubleshooting Guide

67 Result: v The SE with Invariant I:01:05 is marked missing because it is not in the Discovery v The SE with Invariant I:01:05 becomes Disconnected because it is not in Discovery and another SE was discovered with the same Element and Instance v The SE with Invariant I:01:06 is updated (Instance, Name, Label, etc.), but retains the same dbindex and Invariant Elt dbindex Elt Name Instance SE dbindex SE Name Invariant State Missing Count alm elt01 <04> alm elt01.seinv04 I:01:04 on alm elt01 <05> alm elt01.seinv05 I:01:06 on alm elt01 Disconnected_ _<05> I:01:05 off 1 Disconnected_ _alm elt01.seinv05 In theory, a Disconnected resource could come back; the Invariant is the unique identifier to indicate the return of the SE. The Disconnected resource attempting to return would also have to have a different Instance in order to return. Element Name and Instance are required to be unique across all Sub-elements. Scenario 2 - Instance Shift Causes Burn Initial Discovery finds three sub-elements without Invariant and three sub-elements with Invariant. Elt dbindex Elt Name Instance SE dbindex SE Name Invariant State Missing Count alm elt02 <01> alm elt02.seinv01 on alm elt02 <02> alm elt02.seinv02 on alm elt02 <03> alm elt02.seinv03 on alm elt02 <04> alm elt02.seinv04 I:02:04 on alm elt02 <05> alm elt02.seinv05 I:02:05 on alm elt02 <06> alm elt02.seinv06 I:02:06 on 0 During a subsequent Discovery the following occurs: v The SE that was at Instance <03> is not found in Discovery v The SE with Invariant I:01:06 is found, but at Instance <03> Result: v The SE that was at Instance <03> is marked missing because it is not present in Discovery v The SE that was at Instance <03> is Burned because it is not present in Discovery and another SE in Discovery has the same Element and Instance v The SE with Invariant I:01:06 is updated (Instance, Name, Label, etc.), but retains the same dbindex and Invariant Elt dbindex Elt Name Instance SE dbindex SE Name Invariant State Missing Count alm elt02 <01> alm elt02.seinv01 on alm elt02 <02> alm elt02.seinv02 on alm elt02 <03> alm elt02.seinv03 I:02:06 on alm elt02 <04> alm elt02.seinv04 I:02:04 on 0 Chapter 5. SNMP inventory troubleshooting 59

68 Elt dbindex Elt Name Instance SE dbindex SE Name Invariant State Missing Count alm elt02 <05> alm elt02.seinv05 I:02:05 on alm elt02 Burned_ _<03> off 1 Burned_ _alm elt02.se03 Where to Go From Here Burned resources cannot come back, there is no Invariant to uniquely identify the SE in Discovery as that which was Burned. If the Burned resource attempted to come back without Invariant and with the same Instance, Inventory would reject both because of a duplication on Element Name and Instance that cannot be resolved. Relevant information For information on periodic administrative tasks to perform, see See SNMP Inventory Management 60 IBM Tivoli Netcool Performance Manager: Troubleshooting Guide

69 Chapter 6. SNMP inventory management Regular monitoring This section discusses SNMP Inventory troubleshooting for Tivoli Netcool Performance Manager. Monitoring actions have to be performed regularly to keep Tivoli Netcool Performance Manager's SNMP Inventory process performing optimally. If your SNMP Inventory is triggered every day, we recommend performing these monitoring actions at least once a week. If your SNMP Inventory is triggered several times a day (for example, every six hours), we recommend performing these monitoring actions at least once a day. Routine SNMP inventory management tasks The following sections discuss routine SNMP Inventory management tasks: Finding Elements and subelements about to reach their retry limit When elements and sub-elements reach their retry limit, they are retired from the Inventory, which means that Tivoli Netcool Performance Manager stops collecting on them and removes them from some groups. About this task The retry limit is set and managed in the DataMart > Inventory Tool > Edit Profile > Synchronization Wizard. The default is three retries and a resource that is older than seven days, as shown in the following figure. BOTH limits must be met in order for a sub-element to be retired. Copyright IBM Corp. 2011,

70 Figure 20. Synchronization Wizard To find elements and sub-elements about to reach their retry limit, do the following: Procedure 1. Log in as pvuser (or the user name that you specified during installation) on the system where DataMart is installed. 2. Enter the following commands, noting that $PVMHOME is defined as /opt/datamart by default. The commands look for elements and sub-elements approaching their retry counter, using greater than 2 (>2) for the retry count and 3 (the default) to specify the age of the resource. v Elements: v $PVMHOME/bin/resmgr/-noHead -export elt -colnames "dbindex origin name profil state missing" -filterrule "%(elt.profil) not like %bulk% AND %(state)= on AND %(missing) >2" The resmgr command returns output like the following: _ inventory _ skywalker.quallaby.com-1 _ lowell _ on _ 8 _ _ inventory _ ducks.quallaby.com-1 _ lowell _ on _ 4 _ _ inventory _ _ lowell _ on _ 7 _ Sub-elements: $PVMHOME/bin/resmgr -nohead -export se -colnames "dbindex origin name label elt.profil invariant state missing" -filterrule "%(elt.profil) not like %bulk% AND %(state)= on AND %(missing) >2" The resmgr command returns output like the following: _ inventory _ bbpser170-1_processid<12074> _ bbpser170-1_"bbpser170" "AMGR_visual" pid _ unix _ AMGR_visual _ on _ 3 _ For more information on the resmgr command, see the Tivoli Netcool Performance Manager Command Line Interface Guide. 62 IBM Tivoli Netcool Performance Manager: Troubleshooting Guide