IBM. PDF file of IBM Knowledge Center topics. IBM Operations Analytics for z Systems. Version 2 Release 2

IBM Operations Analytics for z Systems IBM PDF file of IBM Knowledge Center topics Version 2 Release 2

IBM Operations Analytics for z Systems IBM PDF file of IBM Knowledge Center topics Version 2 Release 2

ii Operations Analytics for z Systems: PDF file of IBM Knowledge Center topics

Contents What's new in Version 2.2.0...... 1 Conventions used in this information.. 7 Operations Analytics for z Systems overview.............. 9 z/os Insight Packs............ 11 IBM Operations Analytics - Log Analysis.... 12 Extensions to Log Analysis for z/os Problem Insights and client-side Expert Advice..... 13 z/os Log Forwarder........... 15 Installation and configuration checklists..... 15 Log Analysis checklist.......... 16 z/os Insight Packs and extensions checklist.. 17 z/os Log Forwarder checklist....... 17 Planning for installation and configuration of Operations Analytics for z Systems............ 21 System requirements........... 21 z/os Insight Pack requirements...... 21 z/os Log Forwarder requirements..... 22 Planning for installation of Log Analysis.... 24 Planning for Log Analysis to support LDAP interaction with RACF for sign-on authentication.. 24 Planning for installation of the z/os Log Forwarder 25 Planning for configuration of the z/os Log Forwarder............... 26 z/os Log Forwarder program files..... 26 Required authorities for z/os Log Forwarder operations.............. 27 z/os Log Forwarder start procedure..... 28 Authorization with APF......... 28 Data sources............. 29 Scalable data collection architecture..... 33 Data collection............ 35 Data forwarding............ 50 Installing Operations Analytics for z Systems.............. 53 Installing Log Analysis.......... 53 Installing the z/os Insight Packs and extensions.. 55 Using the manual method of installation... 57 Uninstalling the Insight Packs........ 58 Removing data from Log Analysis....... 59 Uninstalling Log Analysis......... 60 Upgrading Operations Analytics for z Systems.............. 63 Upgrading to V2.2.0.1 Interim Feature 2, including upgrading to Log Analysis V1.3.3 Fix Pack 1... 63 Upgrading to V2.2.0.1 Interim Feature 2 without upgrading Log Analysis.......... 67 Upgrading the z/os Insight Packs...... 69 Using the manual method of upgrading.... 71 Updating the z/os Log Forwarder configuration.. 73 Configuring the z/os Log Forwarder 75 Preparing to configure the z/os Log Forwarder.. 76 Creating the z/os Log Forwarder start procedure.............. 76 Preparing the z/os NetView message provider for use............... 77 Creating the z/os SMF real-time data provider start procedure............ 78 Preparing the Log Forwarder Configuration Assistant for use............. 78 Preparing the Log Forwarder Configuration Assistant for sysplex-wide configuration.... 78 Customizing the Log Forwarder Configuration Assistant start program......... 79 Starting the Log Forwarder Configuration Assistant 80 Defining the z/os Log Forwarder environment.. 80 Defining the sources of log data....... 81 Correlation of the data to be analyzed with the associated data gatherer types, data source types, and dashboards............ 82 Data configuration file overview...... 85 Manually configuring the z/os Log Forwarder to forward data............ 89 Encrypting passwords in the data configuration file................ 94 Verifying file path values for rolling logs... 95 Defining multiple data gatherers in a single z/os Job Log gatherer definition........ 96 Loading historical data in batch mode for analysis............... 98 Verifying the identity of the target server for forwarding data............. 99 Preparing to analyze z/os log data 103 Starting and stopping the z/os Log Forwarder.. 103 Starting and stopping the NetView message provider............... 104 Starting and stopping the SMF real-time data provider............... 104 Logging in to Log Analysis......... 104 Use of cookies in the Log Analysis UI.... 105 Grouping data sources to optimize troubleshooting in your IT environment.......... 105 Extending troubleshooting capability with Custom Search Dashboard applications....... 106 Customizing the dashboard applications... 106 Running the dashboard applications..... 111 Getting started with Problem Insights for z/os.. 112 Getting started with client-side Expert Advice.. 112 Copyright IBM Corp. 2014, 2016 iii

Troubleshooting Operations Analytics for z Systems........... 115 Log files............... 115 Enabling tracing............ 115 Enabling static tracing for the z/os Log Forwarder............. 116 Enabling dynamic tracing for the z/os Log Forwarder............. 116 z/os Log Forwarder user ID has insufficient authority............... 118 BPX messages precede GLA messages in the z/os SYSLOG............... 118 Cannot shut down the z/os Log Forwarder by using the STOP command......... 119 When z/os Log Forwarder starts, configuration changes do not seem to be implemented.... 120 APPLID values for CICS Transaction Server might not be correct in Log Analysis user interface... 121 DB2 or MQ command prefix values might not be correct in Log Analysis user interface..... 121 Log record skipped and not available in Log Analysis............... 122 z/os log data is missing in Log Analysis search results................ 122 Cannot establish a secure connection between the z/os Log Forwarder and the Logstash or load balancer server............. 125 NetView for z/os problems........ 126 z/os Log Forwarder message states that PPI issued return code 24.......... 126 NetView message provider GLANETV issues message GLAL004E with return code 15... 126 NetView message provider GLANETV issues message GLAL006E.......... 127 SMF problems............. 127 SMF data is not showing in Log Analysis... 127 SMF message explanations........ 128 Reference............. 131 Command reference for the z/os Log Forwarder 131 DISPLAY command for data gatherers.... 131 START command for data gatherers..... 132 STOP command for data gatherers..... 133 Dashboards.............. 135 WebSphere Application Server for z/os Custom Search Dashboard applications...... 135 z/os Network Custom Search Dashboard applications............. 135 z/os SMF Custom Search Dashboard applications............. 136 z/os SYSLOG Custom Search Dashboard applications............. 140 WebSphere Application Server for z/os data source types.............. 145 zos-was-sysout data source type...... 145 zos-was-sysprint data source type..... 148 zos-was-hpel data source type...... 150 z/os Network data source types....... 153 zos-netview data source type....... 154 z/os SMF data source types........ 157 zos-smf30-annotate annotations...... 157 zos-smf80-annotate annotations...... 158 zos-smf110_e-annotate annotations..... 161 zos-smf110_s_10-annotate annotations.... 162 zos-smf120-annotate annotations...... 164 z/os SYSLOG data source types....... 168 zos-syslog-console data source type.... 169 zos-syslog-sdsf data source type..... 172 zos-syslogd data source type....... 175 zos-cics-msgusr data source type: three variations............. 177 zos-cics-eyulog data source type: three variations............. 178 zos-cics-annotate annotations...... 179 Property reference for the data configuration file 181 System properties........... 181 Log receiver properties......... 183 Data gatherer properties......... 186 Sample searches............ 200 WebSphere Application Server for z/os sample searches.............. 200 z/os Network sample searches...... 201 z/os SMF sample searches........ 202 z/os SYSLOG sample searches...... 204 SMF type 80-related records that are created by the SMF real-time data provider........ 208 SMF80_COMMAND record type........ 208 SMF80_LOGON record type......... 209 SMF80_OMVS_RES record types....... 210 SMF80_OMVS_SEC record types....... 210 SMF80_OPERATION record type....... 211 SMF80_RESOURCE record type....... 213 Variable reference for the environment configuration file............ 214 Variable reference for the z/os Log Forwarder start procedure............. 216 Variable reference for the z/os SMF real-time data provider start procedure.......... 216 Notices.............. 219 Terms and conditions for product documentation 220 Trademarks.............. 221 iv Operations Analytics for z Systems: PDF file of IBM Knowledge Center topics

What's new in Version 2.2.0 Review the new features that are available in Version 2.2.0 of IBM Operations Analytics for z Systems to understand the potential benefits to your z/os-based IT operations environment. v V2.2.0.1 Interim Feature 2 (June 2016) v V2.2.0.1 Interim Feature 1 (March 2016) on page 3 v V2.2.0.0 features (October 2015) on page 3 V2.2.0.1 Interim Feature 2 (June 2016) Table 1. New features in IBM Operations Analytics for z Systems V2.2.0.1 Interim Feature 2 Feature description The Problem Insights extension to IBM Operations Analytics - Log Analysis provides real-time insight into problems in your IT environment, with suggested actions to help resolve the problems. The z/os Log Forwarder can be configured to use the scalable data collection architecture in IBM Operations Analytics - Log Analysis to send data to the IBM Operations Analytics - Log Analysis server. Support for gathering data from a Virtual Storage Access Method (VSAM) cluster that is an entry-sequenced data set (ESDS) More information v Extensions to Log Analysis for z/os Problem Insights and client-side Expert Advice on page 13 v z/os Insight Pack requirements on page 21 v Installing Log Analysis on page 53 v v v Installing the z/os Insight Packs and extensions on page 55 Getting started with Problem Insights for z/os on page 112 Operations Analytics for z Systems overview on page 9 v Creation of data sources on page 29 v Scalable data collection architecture on page 33 v Data forwarding on page 50 v Log receiver properties on page 183 v Log Analysis receiver properties on page 183 v Logstash receiver properties on page 185 v Configuring the data destination on page 93 v Data gatherer properties on page 186 v z/os Data Set gatherer properties on page 190 v Data collection from paired data sets on page 50 Copyright IBM Corp. 2014, 2016 1

Table 1. New features in IBM Operations Analytics for z Systems V2.2.0.1 Interim Feature 2 (continued) Feature description Support for gathering and analyzing SMF record type 120 subtype 9 data, which is request activity data that is produced by WebSphere Application Server for z/os The client-side Expert Advice extension to IBM Operations Analytics - Log Analysis gives you access to expert advice even if the IBM Operations Analytics - Log Analysis server does not have access to the Internet. New z/os Log Forwarder commands for managing data gatherers without having to stop and restart the z/os Log Forwarder Capability to search for data based on sysplex or system names More information v z/os Insight Packs on page 11 v SMF real-time data provider on page 39 v v Requirements for enabling the generation of SMF data at the source on page 41 Correlation of the data to be analyzed with the associated data gatherer types, data source types, and dashboards on page 82 v z/os SMF data source types on page 157 v zos-smf120-annotate annotations on page 164 v z/os SMF Custom Search Dashboard applications on page 136 v z/os UNIX Log File gatherer properties on page 198 v z/os SMF sample searches on page 202 v Extensions to Log Analysis for z/os Problem Insights and client-side Expert Advice on page 13 v z/os Insight Pack requirements on page 21 v Installing Log Analysis on page 53 v v Installing the z/os Insight Packs and extensions on page 55 Getting started with client-side Expert Advice on page 112 v DISPLAY command for data gatherers on page 131 v START command for data gatherers on page 132 v STOP command for data gatherers on page 133 v zos-was-sysout-annotate annotations on page 146 v zos-was-sysprint-annotate annotations on page 149 v zos-was-hpel-annotate annotations on page 151 v zos-netview-annotate annotations on page 155 v zos-smf30-annotate annotations on page 157 v zos-smf80-annotate annotations on page 158 v zos-smf110_e-annotate annotations on page 161 v zos-smf110_s_10-annotate annotations on page 162 v zos-smf120-annotate annotations on page 164 v zos-syslog-console-annotate annotations on page 170 v zos-syslog-sdsf-annotate annotations on page 173 v zos-syslogd-annotate annotations on page 176 v zos-cics-annotate annotations on page 179 2 Operations Analytics for z Systems: PDF file of IBM Knowledge Center topics

V2.2.0.1 Interim Feature 1 (March 2016) Table 2. New features in IBM Operations Analytics for z Systems V2.2.0.1 Interim Feature 1 Feature description Support for gathering and analyzing SMF record type 80 data, which is produced during Resource Access Control Facility (RACF) processing More information v z/os Insight Packs on page 11 v SMF real-time data provider on page 39 v v Requirements for enabling the generation of SMF data at the source on page 41 Correlation of the data to be analyzed with the associated data gatherer types, data source types, and dashboards on page 82 v z/os SMF data source types on page 157 v zos-smf80-annotate annotations on page 158 v z/os SMF Custom Search Dashboard applications on page 136 v z/os UNIX Log File gatherer properties on page 198 v z/os SMF sample searches on page 202 v SMF type 80-related records that are created by the SMF real-time data provider on page 208 V2.2.0.0 features (October 2015) Table 3 indicates changes to some terminology. Tip: If you are upgrading to IBM Operations Analytics for z Systems Version 2.2.0.0, see Updating the z/os Log Forwarder configuration on page 73 for information about how these terminology changes affect the z/os Log Forwarder configuration. Table 3. Terminology changes Previous name log gatherer log path source type New name data gatherer file path data source type Table 4 on page 4 lists the new features and indicates where you can find more information. What's new in Version 2.2.0 3

Table 4. New features in IBM Operations Analytics for z Systems V2.2.0.0 Feature description Support for gathering and analyzing NetView for z/os messages Support for gathering and analyzing the following SMF record type 110 data for CICS Transaction Server for z/os: v Global transaction manager statistics data v Monitoring exceptions data Domain insight for z/os security A new dashboard and sample searches are available for Resource Access Control Facility (RACF ) messages in the z/os SYSLOG. The new dashboard can also highlight other intrusion-related indicators. More information v Operations Analytics for z Systems overview on page 9 z/os Insight Packs on page 11 z/os Log Forwarder on page 15 v z/os Log Forwarder requirements on page 22 v Data sources on page 29 Time zone information for z/os log records on page 31 v NetView message provider on page 45 v Defining the sources of log data on page 81 v z/os Network data source types on page 153 v zos-netview data source type on page 154 - zos-netview-annotate annotations on page 155 z/os Network Custom Search Dashboard applications on page 135 v z/os Network sample searches on page 201 v Data gatherer properties on page 186 v z/os NetView Message gatherer properties on page 195 Operations Analytics for z Systems overview on page 9 z/os Insight Packs on page 11 z/os Log Forwarder on page 15 v Data sources on page 29 v Time zone information for z/os log records on page 31 Correlation of the data to be analyzed with the associated data gatherer types, data source types, and dashboards on page 82 v z/os SMF data source types on page 157 v zos-smf110_e-annotate annotations on page 161 zos-smf110_s_10-annotate annotations on page 162 z/os SMF Custom Search Dashboard applications on page 136 v z/os SMF sample searches on page 202 v z/os UNIX Log File gatherer properties on page 198 v Correlation of the data to be analyzed with the associated data gatherer types, data source types, and dashboards on page 82 v Security for z/os Dashboard on page 144 v Security for z/os sample searches on page 206 4 Operations Analytics for z Systems: PDF file of IBM Knowledge Center topics

Table 4. New features in IBM Operations Analytics for z Systems V2.2.0.0 (continued) Feature description Support for gathering and analyzing CICS Transaction Server for z/os EYULOG data More information In the z/os SYSLOG Insight Pack, three new data source types are provided for EYULOG data. The log record annotator zos-cics-annotate annotates the log records for both EYULOG and MSGUSR log data. In earlier releases of IBM Operations Analytics for z Systems, zos-cics-annotate was named zos-cics-msgusr-annotate. New DB2 for z/os sample searches If you collect data from a z/os job log, you can now use wildcard characters in the job name to define multiple data gatherers in a single z/os Job Log gatherer definition. There is no longer a requirement to set the Java stack size on the IBM Operations Analytics - Log Analysis server to 6144 KB. The following topics provide more information: v Operations Analytics for z Systems overview on page 9 z/os Insight Packs on page 11 z/os Log Forwarder on page 15 v Data sources on page 29 v Time zone information for z/os log records on page 31 Correlation of the data to be analyzed with the associated data gatherer types, data source types, and dashboards on page 82 v z/os SYSLOG data source types on page 168 zos-cics-eyulog data source type: three variations on page 178 v Data gatherer properties on page 186 z/os Job Log gatherer properties on page 192 z/os UNIX Log File gatherer properties on page 198 v DB2 for z/os sample searches on page 204 v Defining multiple data gatherers in a single z/os Job Log gatherer definition on page 96 v z/os Job Log gatherer properties on page 192 Not applicable What's new in Version 2.2.0 5

6 Operations Analytics for z Systems: PDF file of IBM Knowledge Center topics

Conventions used in this information This information describes conventions that are used in the IBM Operations Analytics for z Systems documentation. LA_INSTALL_DIR directory LA_INSTALL_DIR is the directory where IBM Operations Analytics - Log Analysis is installed. The default location for installing Log Analysis is the /home/user/ibm/loganalysis directory, where user represents the user ID of the non-root user who installs Log Analysis. Copyright IBM Corp. 2014, 2016 7

8 Operations Analytics for z Systems: PDF file of IBM Knowledge Center topics

Operations Analytics for z Systems overview IBM Operations Analytics for z Systems extends the capabilities of IBM Operations Analytics - Log Analysis to help you more quickly identify, isolate, and resolve problems in a z/os-based IT operations environment. IBM Operations Analytics for z Systems provides the following capabilities: v Capability to collect z/os logs across the System z enterprise and to forward them to the IBM Operations Analytics - Log Analysis server for analysis v Capability to index, search, and analyze application, middleware, and infrastructure log data across the System z enterprise v Capability to quickly search and visualize errors across thousands of log records v Expert advice that is based on linking search results to available troubleshooting information, such as best practices and previously documented solutions v Continuous streaming of z/os logs Relationship between IBM Operations Analytics for z Systems and IBM Operations Analytics - Log Analysis IBM Operations Analytics - Log Analysis is a cross-platform solution for identifying and resolving problems in an IT operations environment. In IBM Operations Analytics - Log Analysis, an Insight Pack is software that extends the capabilities of IBM Operations Analytics - Log Analysis to provide support for loading and analyzing data from sources that share common characteristics. Examples include log sources for a specific operating system or for a specific application, such as IBM WebSphere Application Server. IBM Operations Analytics for z Systems includes z/os Insight Packs that extend IBM Operations Analytics - Log Analysis function to analyze the various types of z/os log data. IBM Operations Analytics for z Systems also includes a version of IBM Operations Analytics - Log Analysis. Flow of source data Figure 1 on page 10 illustrates the flow of data among the following primary components of IBM Operations Analytics for z Systems: v z/os Insight Packs on page 11 v IBM Operations Analytics - Log Analysis on page 12 v z/os Log Forwarder on page 15 Copyright IBM Corp. 2014, 2016 9

z/os LPAR 1 Log Log z/os Log Forwarder Log Analysis server User Interface Log z/os LPAR 2 1 2 z/os Generic Insight Packs Receiver 3 Log Log z/os Log Forwarder Log Figure 1. Flow of source data among IBM Operations Analytics for z Systems components The following steps describe the data flow among components, which is indicated by arrows in the illustration: 1. In each z/os logical partition (LPAR), the z/os Log Forwarder retrieves the data from the respective source and forwards it to the IBM Operations Analytics - Log Analysis server. A source can be any of the following entities: v v v v v Any job log Any z/os UNIX log file or entry-sequenced Virtual Storage Access Method (VSAM) cluster that meets the following criteria: Is encoded in Extended Binary Coded Decimal Interchange Code (EBCDIC) Is record-based Has a time stamp in each record z/os SYSLOG console NetView for z/os messages WebSphere Application Server for z/os High Performance Extensible Logging (HPEL) logs The z/os Log Forwarder forwards data to the IBM Operations Analytics - Log Analysis server either directly or by using the scalable data collection architecture in IBM Operations Analytics - Log Analysis. If the scalable data collection architecture is used, the z/os Log Forwarder forwards the data to a Logstash server, and another Logstash server forwards the data to the IBM Operations Analytics - Log Analysis server. 2. The source data is processed by the respective z/os Insight Pack. Insight is provided for data from the following source types: v v v v z/os system log (SYSLOG) CICS Transaction Server for z/os EYULOG or MSGUSR log data Network data, such as data from UNIX System Services system log (syslogd) or z/os Communications Server NetView for z/os message data 10 Operations Analytics for z Systems: PDF file of IBM Knowledge Center topics

v z/os Insight Packs v SMF data WebSphere Application Server for z/os logs that include SYSOUT, SYSPRINT, or HPEL log data Tip: Although IBM Operations Analytics for z Systems provides support for gathering data from a VSAM cluster that is an entry-sequenced data set (ESDS), no z/os Insight Packs are provided for analyzing data from VSAM data sets. You can create your own insight pack for this purpose, or use a generic insight pack that is provided by IBM Operations Analytics - Log Analysis. 3. The user can analyze the data that is shown in the IBM Operations Analytics - Log Analysis user interface. Each z/os Insight Pack in IBM Operations Analytics for z Systems extends IBM Operations Analytics - Log Analysis function to analyze a unique type of z/os log data. The following z/os Insight Packs are available: WebSphere Application Server for z/os Insight Pack This Insight Pack enables Log Analysis to ingest, and perform searches against, logging data that is retrieved from IBM WebSphere Application Server for z/os. Summary: Install this Insight Pack if you want to analyze WebSphere Application Server for z/os log data. z/os Network Insight Pack This Insight Pack enables Log Analysis to perform searches against z/os network data. It also enables Log Analysis to ingest, and perform searches against, IBM Tivoli NetView for z/os message data. To analyze z/os network data, you must use both the z/os Network Insight Pack and the z/os SYSLOG Insight Pack. The z/os SYSLOG Insight Pack enables Log Analysis to ingest z/os network data. Summary: Install this Insight Pack if you want to analyze any of the following data: v Network data, such as data from UNIX System Services system log (syslogd) or z/os Communications Server v NetView for z/os message data z/os SMF Insight Pack This Insight Pack enables Log Analysis to ingest, and perform searches against, data that is retrieved from IBM z/os System Management Facilities (SMF). Summary: Install this Insight Pack if you want to analyze SMF data. z/os SYSLOG Insight Pack This Insight Pack enables Log Analysis to ingest, and perform searches against, data that is retrieved from the following sources: v z/os SYSLOG console, which includes data from the following software: IBM CICS Transaction Server for z/os IBM DB2 for z/os Operations Analytics for z Systems overview 11

IBM IMS for z/os IBM MQ for z/os IBM Resource Access Control Facility (RACF) z/os Communications Server v CICS Transaction Server for z/os EYULOG and MSGUSR log v UNIX System Services system log (syslogd) Summary: Install this Insight Pack if you want to analyze any of the following data: v z/os SYSLOG console data v CICS Transaction Server for z/os EYULOG or MSGUSR log data v UNIX System Services system log (syslogd) v SMF data Table 5 lists each z/os Insight Pack with its associated installation package file. Table 5. Installation package files for z/os Insight Packs Insight Pack WebSphere Application Server for z/os Insight Pack z/os Network Insight Pack Important: To analyze network data, install both the z/os SYSLOG Insight Pack and this insight pack. z/os SMF Insight Pack Important: To analyze SMF data, install both the z/os SYSLOG Insight Pack and this insight pack. z/os SYSLOG Insight Pack Installation package WASforzOSInsightPack_v2.2.0.2.zip zosnetworkinsightpack_v1.1.0.2.zip SMFforzOSInsightPack_v1.1.0.3.zip SYSLOGforzOSInsightPack_v2.2.0.2.zip IBM Operations Analytics - Log Analysis IBM Operations Analytics - Log Analysis is used with the z/os Log Forwarder and the respective z/os Insight Pack to provide support for z/os log data. IBM Operations Analytics - Log Analysis includes the following functions that you can configure in the Log Analysis UI: Role-based access control You can create and modify users and roles to assign role-based access control to individual users. For more information, see Users and roles in the Log Analysis documentation. Alerting You can create alerts that are based on events, and you can define the actions for the system to take when an alert is triggered. For example, you can define an action to send an email notification to one or more people when an alert is triggered. For more information, see Managing alerts in the Log Analysis documentation. 12 Operations Analytics for z Systems: PDF file of IBM Knowledge Center topics

Some components that are preinstalled with Log Analysis Although many components are preinstalled with IBM Operations Analytics - Log Analysis, the following components might be especially useful with your z/os Insight Packs: Expert advice custom search dashboard The Expert Advice provides links to contextually relevant information to help you resolve problems quickly. Using this application, you can select any column or cells in the Grid view and can launch a search of the IBM Support Portal. The application searches for matches to unique terms that are contained in the column that you select. You can start the Expert Advice application by clicking Search Dashboards in the navigation pane of the Search workspace. For more information about this component, see Custom Search Dashboards in the Log Analysis documentation. Tip: To use this Expert Advice, the Log Analysis server must have access to the Internet. With the client-side Expert Advice extension that is provided by IBM Operations Analytics for z Systems, you can access expert advice even if the Log Analysis server does not have access to the Internet. For more information about the client-side Expert Advice, see Extensions to Log Analysis for z/os Problem Insights and client-side Expert Advice. WebSphere Application Server Insight Pack This Insight Pack is different from the WebSphere Application Server for z/os Insight Pack. It is intended for use with WebSphere Application Server on distributed platforms. It does not support native logging formats for WebSphere Application Server for z/os. However, if you configure your WebSphere Application Server for z/os environment to use a distributed logging format, this Insight Pack provides you with the required annotation and indexing capabilities. For more information about this component, see WebSphere Application Server Insight Pack in the Log Analysis documentation. Extensions to Log Analysis for z/os Problem Insights and client-side Expert Advice IBM Operations Analytics for z Systems V2.2.0.1 Interim Feature 2 provides extensions to IBM Operations Analytics - Log Analysis for z/os Problem Insights and client-side Expert Advice. To use these extensions, you must have IBM Operations Analytics - Log Analysis Version 1.3.3 Fix Pack 1 installed. Problem Insights The Problem Insights extension provides real-time insight into problems in your IT environment, with suggested actions to help resolve the problems. If the Problem Insights component of IBM Operations Analytics for z Systems is installed, the Log Analysis UI includes a new tab that is titled Problem Insights. For each sysplex from which data is being forwarded to the Log Analysis server, the Problem Insights page includes insight about certain problems that are identified in the ingested data. Operations Analytics for z Systems overview 13

You can select the time range for which you want to see insights. For example, if you want to know about certain problems that were identified in the last hour, you can select the last hour as the time range. The default time range is the last 15 minutes. You can also reload the Problem Insights page and associated data by clicking the Refresh button that is located in the area of the UI where you select the time range. Each sysplex in the monitored environment is represented by a button that indicates the number of problems that are found in that sysplex. Click a sysplex button to show or hide the associated sysplex data in the Problem Insights and Suggested Actions table. You can show the data for multiple sysplexes in the table at the same time, and you can toggle the showing or hiding of data in the table by clicking the sysplex buttons. The table includes the following information about each problem that is discovered: Severity The indication of the severity of the problem. Sysplex The sysplex where the problem occurred. System The system where the problem occurred. Subsystem The subsystem or system resources manager where the problem occurred. Time The last time that the problem occurred in the selected time range. For example, if you select the last 15 minutes as the time range, this column shows the last time that the problem occurred in the last 15 minutes. Problem Summary A summary of the problem that provides insight into the cause. Count The total number of occurrences of the problem in the selected time range. Suggested Actions An extension of the problem summary that provides more insight about the problem, includes one or more actions that you can take to resolve the problem, and links to other sources of information that might help you resolve the problem, such as relevant topics in the IBM Knowledge Center. Evidence The message number of the message that identifies the problem, which you can click to view more information. Client-side Expert Advice After you install the IBM Operations Analytics for z Systems extensions, the following applications are shown under Expert Advice in the Custom Search Dashboards panel of the left navigation pane of the Search workspace: v IBMSupportPortal-ExpertAdvice, which is the default Expert Advice in IBM Operations Analytics - Log Analysis. When this Expert Advice is launched, the Log Analysis server sends search requests to the IBM Support Portal and displays the query search results in the Log Analysis UI. 14 Operations Analytics for z Systems: PDF file of IBM Knowledge Center topics

z/os Log Forwarder v IBMSupportPortal-ExpertAdvice on Client, which is the extension for client-side Expert Advice that is provided by IBM Operations Analytics for z Systems. When this Expert Advice is launched, the client browser sends search requests directly to the IBM Support Portal and opens a new browser tab to display the query search results. The z/os Log Forwarder can automatically monitor and forward z/os data to the IBM Operations Analytics - Log Analysis server. In this context, z/os data includes any of the following data: v z/os SYSLOG console data v NetView for z/os message data v SMF data v Data from any job log or UNIX System Services log, including the following logs: CICS Transaction Server for z/os EYULOG and MSGUSR log UNIX System Services system log (syslogd) WebSphere Application Server for z/os logs v Data from any entry-sequenced data set (ESDS) that is a Virtual Storage Access Method (VSAM) cluster. Each logical partition (LPAR) must have its own instance of the z/os Log Forwarder. In each LPAR, you must configure the z/os Log Forwarder to specify which data is to be forwarded to the Log Analysis server. In each LPAR where you want to collect the following data, you must also install or configure the respective mechanism for providing this data to the z/os Log Forwarder: NetView for z/os message data You must configure the NetView message provider. For more information, see NetView message provider on page 45. SMF data You must configure the SMF real-time data provider. For more information, see SMF real-time data provider on page 39. z/os SYSLOG data You must install either the GLASYSG or GLAMDBG user exit. Installation and configuration checklists For more information about the user exits, see User exits for collecting z/os SYSLOG data on page 36. To help you get started, these installation and configuration checklists summarize the important steps in installing and configuring the Operations Analytics for z Systems components. Before you can use Operations Analytics for z Systems to analyze log data, you must complete the following tasks: Operations Analytics for z Systems overview 15

1. Install IBM Operations Analytics - Log Analysis Version 1.3.0, 1.3.1, 1.3.2, or 1.3.3. See the Log Analysis checklist. 2. On the Log Analysis server, install the z/os Insight Packs that you want to use. See the z/os Insight Packs and extensions checklist on page 17. 3. Install the z/os Log Forwarder in each z/os logical partition (LPAR) for which you want to process z/os log messages. a. In each LPAR, configure the z/os Log Forwarder to collect and forward z/os log data to the Log Analysis server. See the z/os Log Forwarder checklist on page 17. 4. Ensure that z/os data sources are configured and grouped appropriately in Log Analysis. See Preparing to analyze z/os log data on page 103. Log Analysis checklist This checklist summarizes the important steps in installing IBM Operations Analytics - Log Analysis. Preparing to install Log Analysis Complete the following steps before you install IBM Operations Analytics - Log Analysis: 1. If you do not have a previously installed instance of Log Analysis Version 1.3.0, 1.3.1, 1.3.2, or 1.3.3, plan to install both Log Analysis Version 1.3.3 Standard Edition and Fix Pack 1 of Version 1.3.3. More information: v z/os Insight Pack requirements on page 21 2. Verify that the system requirements for Log Analysis, including the installation of the prerequisite software, are met. More information: v Hardware and software requirements 3. Using the ulimit command, tune the Linux operating system so that the number of concurrent files is 4096, and the virtual memory is unlimited. More information: v Hardware and software requirements 4. To install and run Log Analysis, you must be logged in to the Linux computer system with a non-root user ID. Either create this non-root user ID, or use an existing non-root user ID. More information: v Planning for installation of Log Analysis on page 24 5. Review the network port assignments for Log Analysis. If the default ports are not available, determine alternative port assignments. More information: v Default ports 6. Decide whether you want to use the scalable data collection architecture. 16 Operations Analytics for z Systems: PDF file of IBM Knowledge Center topics

More information: v Scalable data collection architecture on page 33 7. The IBM Tivoli Monitoring Log File Agent is an optional component that is provided with Log Analysis and other IBM products. This component is not required for analyzing z/os log data. Decide whether you want to install this agent as part of the Log Analysis installation. More information: v Installing and configuring the IBM Tivoli Monitoring Log File Agent Installing Log Analysis Verify that you are logged in to the Linux computer system with the non-root user ID, and complete the steps in Installing Log Analysis on page 53. z/os Insight Packs and extensions checklist This checklist summarizes the important steps in installing z/os Insight Packs and extensions. Installing z/os Insight Packs and extensions 1. Verify that Log Analysis is running. More information: v Installing the z/os Insight Packs and extensions on page 55 2. Verify that you are logged in to the Linux computer system with the non-root user ID that was used to install Log Analysis. More information: v Installing the z/os Insight Packs and extensions on page 55 3. Install the z/os Insight Packs with sample searches and the extensions for Problem Insights and client-side Expert Advice by using the self-extracting installer file. More information: v Installing the z/os Insight Packs and extensions on page 55 z/os Log Forwarder checklist This checklist summarizes the important steps in installing and configuring the z/os Log Forwarder. The following terms are used in the context of the z/os Log Forwarder installation: driving system The system on which you run SMP/E for z/os (SMP/E) to install the z/os Log Forwarder. target system The system on which you configure and run the z/os Log Forwarder. Installing the z/os Log Forwarder 1. Verify that the software requirements for the driving system are met. 2. To install the z/os Log Forwarder, use SMP/E for z/os (SMP/E). Operations Analytics for z Systems overview 17

Assumption: The SMP/E relative files (RELFILES) for the z/os Log Forwarder are available on the driving system. For more information, see the IBM Operations Analytics for z Systems Program Directory. Preparing to configure the z/os Log Forwarder On each target system, complete the following steps before you configure the z/os Log Forwarder: 1. Verify that the software requirements for each target system are met. More information: v z/os Log Forwarder requirements on page 22 2. Verify that each target system has a properly configured TCP/IP stack. More information: v Requirements for running the z/os Log Forwarder on page 23 3. Verify that each target system can communicate with the Log Analysis server on the port that is defined during the Log Analysis installation. More information: v Planning for installation of Log Analysis on page 24 4. Determine the appropriate location for the z/os Log Forwarder configuration files (the environment configuration file and the data configuration file). More information: v z/os Log Forwarder program files on page 26 5. Determine which user ID to use for running the z/os Log Forwarder, and give that user ID the required authorities for file access and for issuing console messages. More information: v Required authorities for z/os Log Forwarder operations on page 27 6. Determine the sources from which you want to collect log data. More information: v Data sources on page 29 v Data collection on page 35 7. If you want to collect z/os SYSLOG data, install either the GLASYSG or GLAMDBG user exit for providing this data to the z/os Log Forwarder. More information: v User exits for collecting z/os SYSLOG data on page 36 8. If you want to collect NetView for z/os message data, associate the NetView message provider with a NetView autotask that has the required permissions. More information: v NetView message provider on page 45 18 Operations Analytics for z Systems: PDF file of IBM Knowledge Center topics

9. If you want to collect System Management Facilities (SMF) data, complete the following steps: a. Verify the following items: 1) 2) 3) On each target system, verify that SMF is active. On each target system, verify that SMF is configured to use log streams and to collect SMF data. Verify that each target system has a hierarchical file system (HFS) with enough available space to contain the z/os SMF real-time data provider temporary files. b. Determine which user ID to use for running the z/os SMF real-time data provider, and give that user ID the required authorities for SMF log stream and HFS file access. More information: v SMF real-time data provider on page 39 Configuring the z/os Log Forwarder to collect data 1. Create the z/os Log Forwarder start procedure by updating the sample procedure GLAPROC. More information: v Creating the z/os Log Forwarder start procedure on page 76 2. If you are collecting NetView for z/os message data, prepare the z/os NetView message provider for use. More information: v Preparing the z/os NetView message provider for use on page 77 3. If you are collecting SMF data, create the z/os SMF real-time data provider start procedure by updating the sample procedure GLASMF. More information: v Creating the z/os SMF real-time data provider start procedure on page 78 4. If necessary, customize the start program for the Log Forwarder Configuration Assistant. For example, you might want to use the sysplex-wide configuration option or to customize variable values in the environment configuration file. More information: v Preparing the Log Forwarder Configuration Assistant for use on page 78 5. Start the Log Forwarder Configuration Assistant to define the z/os Log Forwarder environment (environment configuration file) and data gatherers (data configuration file). Tips: v Leave the value of the z/os host name blank so that the z/os Log Forwarder can discover this host name. v If you are not using the scalable data collection architecture, let the z/os Log Forwarder automatically create data sources in Log Analysis. Operations Analytics for z Systems overview 19

v To prevent unauthorized use of the passwords that are stored in the z/os Log Forwarder data configuration file, encrypt the passwords. See Encrypting passwords in the data configuration file on page 94. More information: v Starting the Log Forwarder Configuration Assistant on page 80 6. If you are not using the scalable data collection architecture, you might want to secure communication between the z/os Log Forwarder and the Log Analysis server. More information: v Verifying the identity of the target server for forwarding data on page 99 20 Operations Analytics for z Systems: PDF file of IBM Knowledge Center topics

Planning for installation and configuration of Operations Analytics for z Systems System requirements In planning for the installation of IBM Operations Analytics for z Systems, you must also plan for the installation of IBM Operations Analytics - Log Analysis and the installation and configuration of the z/os Log Forwarder. Ensure that your environment meets the system requirements for IBM Operations Analytics for z Systems, which include the system requirements for IBM Operations Analytics - Log Analysis and the z/os Log Forwarder. Log Analysis requirements For information about the system requirements for Log Analysis, see Hardware and software requirements. Restriction: If you deploy Log Analysis on Linux for System z, you can use the IBM InfoSphere BigInsights Hadoop Version 3.0 service, but at this time, the Hadoop Distributed File System (HDFS) must be installed on an x86 Linux system. The support for integrating Log Analysis on Linux for System z with other Hadoop distributions is dependent on the platform support that is provided by the respective Hadoop distribution. z/os Insight Pack requirements You must install the z/os Insight Packs on the IBM Operations Analytics - Log Analysis server. Hardware requirements For the Log Analysis hardware requirements, see Hardware and software requirements. Software requirements The z/os Insight Packs must be installed on a Log Analysis server with Log Analysis Version 1.3.0, 1.3.1, 1.3.2, or 1.3.3. The IBM Operations Analytics for z Systems Version 2.2.0.0 package includes IBM Operations Analytics - Log Analysis Version 1.3.1 Standard Edition. The IBM Operations Analytics for z Systems Version 2.2.0.1 Interim Feature 2 package includes IBM Operations Analytics - Log Analysis Version 1.3.3 Standard Edition. Tip: To use the Problem Insights and client-side Expert Advice extensions that are available in IBM Operations Analytics for z Systems Version 2.2.0.1 Interim Feature 2, you must use Log Analysis Version 1.3.3 Fix Pack 1. To get Log Analysis Version 1.3.3 Fix Pack 1, complete the following steps: 1. Go to IBM Fix Central. 2. In the Product selector field, start typing IBM Operations Analytics - Log Analysis, and when the correct product name is shown in the resulting list, select it. More fields are then shown. Copyright IBM Corp. 2014, 2016 21

3. In the Installed Version field, select 1.3.3. 4. In the Platform field, select All. 5. Click Continue. 6. In the resulting Identify fixes window, select Browse for fixes, and click Continue. 7. In the Select fixes window, you should see the fix pack 1.3.3-TIV-IOALA- FP001, which you can select and download. For installation instructions, see the readme file. Table 6 indicates the software versions that are supported by each z/os Insight Pack. Table 6. Software versions that are supported by each z/os Insight Pack Insight Pack Supports data ingestion from this software z/os Network Insight Pack v z/os 1.13, 2.1, and 2.2 v NetView for z/os 6.1, 6.2, and 6.2.1 z/os SMF Insight Pack v z/os 1.13, 2.1, and 2.2 z/os SYSLOG Insight Pack v z/os 1.13, 2.1, and 2.2 WebSphere Application Server for z/os Insight Pack v DB2 for z/os 10.1 and 11.1 v CICS Transaction Server for z/os 4.2, 5.1, and 5.2 v IMS for z/os 11.1, 12.1, and 13.1 v MQ for z/os 7.1 and 8.0 WebSphere Application Server for z/os 7.0, 8.0, and 8.5.5 z/os Log Forwarder requirements You must run the z/os Log Forwarder in each z/os logical partition (LPAR) for which you want to process z/os log messages. Hardware requirements The z/os Log Forwarder can run on any hardware that supports its operating system and other software requirements. Software requirements The z/os Log Forwarder requires the following software to be installed: v z/os V1.13 or later v z/os Communications Server v Resource Access Control Facility (RACF) v JES2 or JES3 v IBM SDK for z/os, Java Technology Edition, Version 6, 7, or 8. Both 31-bit and 64-bit versions are supported. Important considerations: Use the latest available service release of the version of IBM SDK for z/os, Java Technology Edition, that you choose, and apply fix packs as soon as possible after they are released. To find the latest service release or fix pack, see IBM Java Standard Edition Products on z/os. 22 Operations Analytics for z Systems: PDF file of IBM Knowledge Center topics

When the z/os Log Forwarder runs on IBM SDK for z/os, Java Technology Edition, Version 8, it can use the Transport Layer Security (TLS) Version 1.1 and 1.2 protocols to secure communication with the IBM Operations Analytics - Log Analysis server. Log data traffic is then more secure than it is with the TLS Version 1 protocol. If the security of log data transmission is a concern, consider running the z/os Log Forwarder on IBM SDK for z/os, Java Technology Edition, Version 8. Although the z/os Log Forwarder runs equally well on IBM SDK for z/os, Java Technology Edition, Versions 6, 7 and 8, some tests indicate that CPU usage might increase when you run the z/os Log Forwarder on Version 7. If CPU usage of the z/os Log Forwarder is a concern, consider running the z/os Log Forwarder on IBM SDK for z/os, Java Technology Edition, Version 6 or Version 8. Requirements for running the z/os Log Forwarder Before you start the z/os Log Forwarder, you must verify that the z/os environment is set up correctly so that the z/os Log Forwarder can resolve host names and can access the TCP/IP resolver configuration file. Host name resolution To operate, the z/os Log Forwarder must determine the fully qualified domain name (FQDN) of the system on which it is running. Therefore, the networking and name resolution services that are configured in the system for use by the z/os Log Forwarder should be activated before the z/os Log Forwarder is started. When networking and name resolution services are not activated: When the z/os Log Forwarder is started automatically at IPL time, the initialization of these services might not be complete. In this situation, the z/os Log Forwarder might retrieve only a loopback address. The z/os Log Forwarder then tries again every 30 seconds to determine the FQDN until it successfully retrieves the local host address. Search order for the TCP/IP resolver configuration file The z/os Log Forwarder is a z/os UNIX System Services program. It uses TCP/IP functions that require access to the TCP/IP resolver configuration file. This access is provided by using a resolver search order. The resolver search order for z/os UNIX System Services programs is documented in the topic about resolver configuration files in the z/os Communications Server: IP Configuration Guide. The following list summarizes the resolver search order: 1. GLOBALTCPIPDATA 2. The RESOLVER_CONFIG environment variable in the z/os Log Forwarder environment configuration file 3. The /etc/resolv.conf file 4. The SYSTCPD DD statement in the z/os Log Forwarder start procedure 5. userid.tcpip.data, where userid is the user ID that is associated with the z/os Log Forwarder start procedure 6. SYS1.TCPPARMS(TCPDATA) 7. DEFAULTTCPIPDATA 8. TCPIP.TCPIP.DATA Planning for installation and configuration 23