EView/390z Insight for Splunk v7.1 EView/390z Insight Overview (IBM Mainframe environment) Technical Details By leveraging the foundation EView Intelligent Agent technology to power EView/390z Insight for Splunk, enterprises have an end-to-end enterprise view of the IT infrastructure to include the IBM Mainframe environment data. The EView/Splunk combination enables the ability to control all data through a single, easy-to-use interface, and integrate and automate processes for better security, compliance, and log analysis. EView/390z Insight for Splunk is a scalable solution to analyze the terabytes of big data from your IT operations. Turn the thousands of various types of messages generated from the mainframe, into data that is relevant and understandable. The seamless integration into Splunk enables you to get the information to search quickly across massive amounts of mainframe data, providing the Operational Intelligence and insights that you can act on immediately, and predict problems before they occur. A Custom Message Interface makes it easy to extend applications, batch jobs and installation automation rules to send customer messages to Splunk. With the EView custom message interface, EView Insight s ability to gather, report, and analyze any mainframe is nearly limitless. 1
About EView/390z Architecture and Data Flow EView/390z consists of two main components: the EView Intelligent Agent component that runs on the z/os mainframe, and the server component that runs on the EView Splunk forwarding server. Events and performance data are forwarded from the agent to the EView Splunk forwarding server and written to a file that is monitored by a standard Splunk forwarder. The Splunk forwarder sends data to the Splunk server where the EView/390z Splunk app maps data from common event fields. The EView/390z Splunk app contains dashboards to help get you started in viewing z/os event and performance data. Figure 1: shows the data flow between the z/os mainframe, the EView/390 Splunk forwarding server and the Splunk server. What the EView/390z Agent Does The EView/390z Agent operates as a z/os started task. Mainframe messages are collected by the EView Intelligent Agent from several sources, which will be further detailed in this document. Pre-defined messages filters identify important messages that are then packaged into a common data structure and forwarded via TCP/IP to the Splunk server for processing. Forwarding z/os Messages By capturing any z/os SYSLOG message that comes across the z/os console; the powerful, intelligent, EView Agent has the ability to capture the thousands of message types which are generated by the mainframe (z/os) system. Since all enterprise environments are different and unique, the powerful and flexible EView Custom Message Interface provides the ability to extend applications, batch jobs and installation automation rules to send customer messages 2
to Splunk. With the EView custom message interface, EView Insight s ability to gather, report, and analyze any mainframe is nearly limitless. Messages can include information from the following: Operating System DB2 (DataBase2) JES2 (JobEntrySubsystem2) RACF (SECURITY) MQSeries (Message Queuing Series) CICS (Customer Information Central System) utilizing an EView/390 exit program in the o CICS address space WebSphere SMF types Detailed Examples Forwarding VTAM Messages The z/os network task, VTAM, issues messages regarding the mainframe SNA network. The EView/390z agent collects these VTAM messages through the VTAM PPO interface (or PPI interface if IBM NetView is installed on the LPAR). Forwarding DB2 Management Data EView/390z provides the ability to monitor DB2 messages that are sent to the z/os system console. Forwarding RACF Security EView/390z provides the ability to monitor RACF security messages that are sent to the z/os system console. Forwarding SMF Data An interface to the IBM System Monitoring Facility (SMF) is provided to collect and forward performance information to the EView Splunk forwarding server. Forwarding Performance Data An interface to the IBM Resource Monitoring Facility (RMF) is provided to collect and forward performance information to the EView Splunk forwarding server. Forwarding WebSphere Management Data EView/390z provides the ability to monitor WebSphere messages that are sent to the z/os system console. Forwarding CICS Event and Transaction Data EView/390z contains CICS exits that may be configured to send transient data queue CICS messages to the console where the agent console task will be able to send these messages to the EView Splunk forwarding server. Another exit is available to monitor transaction response 3
of transactions against a configured threshold. When the response time of a configured transaction exceeds the threshold, a message is created that can be sent to Splunk. Forwarding Custom Message Data EView/390z provides a module to send custom messages to the agent which are then sent to the EView/390z Splunk forwarding server. The module can be used in batch jobs, REXX programs or application programs (including CICS programs) and SMF type information. Event and Message Buffering If event, message or performance data cannot be sent to the EView Splunk forwarding server for any reason, the EView/390z agent can be configured to save or buffer the data until the connection from the Splunk forwarding server is available. This ensures that important data will not be lost. EView Insight - Splunk Dashboards The EView/390z Splunk app contains several out of the box default dashboards to provide examples of different ways mainframe data can be viewed as the EView/390z information is seamlessly integrated into Splunk. Since EView Insight is seamlessly integrated into Splunk, the simplicity of building your own custom dashboards is already there. And, EView Technology provides the detailed information in the EView/390z Insight: Installation and Customization Guide, to provide the foundation for creating a powerful IT Operations Intelligence Splunk Platform, which integrates the IBM mainframe (z/os) environment. For iseries (AS/400) environments, the EView/400i Insight: Installation and Customization Guide is available. Examples Security The EView Dashboard shows RACF Security Messages. A Splunk Operator can easily drill down and get to the root cause of issues, identify potential threats, etc 4
Messaging/Communication The EView Dashboard shows MQ Series message totals, overall mainframe message totals and totals by z/os mainframe source hosts. CICS Transactions Exceeding Threshold Dashboard This dashboard shows transactions that have exceeded the configured response time threshold. 5
Performance Data Dashboard The performance data dashboard shows the latest reported CPU utilization and graphs of CPU utilization along with memory related metrics. SMF-Types Dashboards SMF Type 14 and 15 dataset records 6
SMF Type 30 Job Completion SMF Type 30 Job Data 7
SMF Type 80 RACF RMF Performance Data 8
Hardware Requirements Splunk Forwarding Server EView/390z requires appropriate Ethernet hardware on the client to communicate via TCP/IP. All other hardware requirements are the same as the requirements for a Splunk forwarding server. z/os Operating System EView/390z requires the appropriate Ethernet hardware on the zseries to allow for TCP/IP communication with the Splunk forwarding server. In addition, make sure that the Splunk forwarding server and z/os partitions meet the disk space requirements described in the following table. Platform Splunk Forwarding Server zseries Mainframe Disk Space 5MB 60 tracks of 3390 DASD Software Requirements On the Splunk Forwarding Server: Windows Client: o Microsoft Windows 2008 R2 or later Linux Client: o Linux 64-bit kernel Version 2.6.24 or later o Perl Version 5.8 or later o glibc Version 2.7 or later The TCP/IP network protocol stack must be active. All other software requirements are the same as the requirements for a Splunk forwarding server. On the zseries mainframe: z/os V1R10 or later The TCP/IP network protocol stack (V3R1 or higher) must be active. 9