New monitoring method for enterprise critical applications

Size: px

Start display at page:

Download "New monitoring method for enterprise critical applications"

Clara Linette Hodge
5 years ago
Views:

1 New monitoring method for enterprise critical applications Dr Tomasz Cieplak SystemWork GmbH 07/11/2017 OC

2 Agenda 1. Application monitoring 2. Facts about SMF records 3. Software for processing SMF records information 4. Long term SMF records reporter 5. Use cases; a. Important TCP/IP transactions; b. CPU resource usage; c. Datasets usage; d. Resources sizing; e. Accounting/billing.

3 Someone cannot work scenario It can take hours

4 Time saving solution: application monitoring What is necessary to develop a real time application monitoring: Find transactions from monitored applications logged in SMF records. It can be: TCP/IP; CICS; IMS (from version 14); WebShpere; DB2 SQL; Identify an enterprise dashboard or monitoring system (Icigna; Splunk; ) Define a communication protocol; Design a software architecture and develop software which: Reads SMF records in real time; Parses/interprets them; Communicates with event processing software.

5 Cargo asynchronous process

6 CAM timetable transactions for monitoring The following CICS transitions are counted: "Processing order of messages (ZAB) and LB to PVG (OES and ÜS) Send" - FV01; "Processed messages for trains arriving from PVG" - FVA0; "Processed messages for trains leaving from PVG" - FVA1; "Processing sequence message from PVG (end marshaling)" - FVA3. The amount of these transactions per time interval (one minute) is counted form 110 SMF subtype 1 "CICS - Monitoring which has been read simultaneously from SMF logger structures.

7 Why this transactions are critical for freight trains? ~5.000 freight trains every day in Germany Freight train time tables much more dynamic than for passenger trains. Created from Honeywell handhelds (and soon tablets) flowing information to CICS Few hours interrupt will lead to Timetable not update Cargo trains stopped Tracks blocked Passenger trains stopped Penalty from harbors for delayed containers

8 Application monitoring - crisis case :02 00:23 00:41 00:58 01:15 01:35 01:53 02:13 02:32 02:53 04:05 05:21 06:06 06:31 07:09 07:32 08:01 08:37 09:11 09:49 10:09 10:39 11:08 11:25 11:42 11:59 12:16 12:34 12:51 13:08 13:26 13:43 14:01 14:18 14:35 14:53 15:11 15:28 15:45 16:02 16:19 16:36 16:53 17:11 17:28 17:45 18:03 18:21 18:39 18:56 19:14 19:31 19:48 20:05 20:22 20:39 20:56 21:13 21:30 21:48 22:05 22:22 22:39 22:56 23:13 23:30 23:48 FV01 FVA0 FVA1 FVA3

9 Icinga screen

10 Application monitoring - architecture Mainframe Host Coupling Facility- SMF Records Application Processes Mainframe Monitor Agent TcpIp Port 5668 Icinga Server

11 Installation of application monitor on z/os Jars: javax.json jar AppMonitor.jar Started task procedure : //JAVAJVM EXEC PGM=JVMLDM71,REGION=2047M, // PARM='/-jar AppMonitor.jar' //SYSPRINT DD SYSOUT=* < System stdout //SYSOUT DD SYSOUT=* < System stderr //STDOUT DD SYSOUT=* < Java System.out //STDERR DD SYSOUT=* < Java System.err //CEEDUMP DD SYSOUT=* //CSV DD DISP=SHR,DSN=SYS2.APPMON.CSV //MAINARGS DD DISP=SHR,DSN=SYS2.APPMON.PARM(ARGS&SYSNAME.) //STDENV DD DISP=SHR,DSN=SYS2.APPMON.PARM(STDENV) Parametes: IFASMF.BLP7.DO.DFLT FV01;FVA0;FVA1;FVA3

13 Query { TransName : * BEMT, DateFrom : * HH:MM, Duration : MMMM } Json Communicators Dialog Response { ReturnCode : , Message : AMS010E Not collected transaction AMS011E not collected time AMS012E Syntax error AMS013E Unrecognized data input, DateFrom : yyyy.mm.dd hh:mm, DateTo : yyyy.mm.dd hh:mm, Transactions :[ TransName : amount integer ] }

14 Modify command interface

15 Response

16 SMF records Big Data? System Management Facility records is a central logging technique that is unique for z/os system. There are two kinds of records: Event oriented describing single system events; Statistical periodically written with collected information. Inside of SMF records a lot of answers for z/os system relevant questions are hidden. The usage of Big Data processing methods can help to find the answers. Frequently the information in records are stored redundantly. It is important to chose the most suited records and data fields for processing.

17 SMF processing methods Traditional process x x x Extra SMF ct Dump Transfor Data mwarehouse Load Reporting closed ETL System Big data process SMF72 SMF32 SMF Dump SMF4U software Extract, Transform, Load Dynamic tables On Demand analysis

18 SMF direct fron RAW files and Relational DB Information from SMF Records :00 00:05 01: RAW % RDB %

19 Long term SMF records reporter The Long Term SMF records reporter is designed for processing huge amounts of dumped/archived SMF records in short time (parser efficiency - 68 MB per second or almost 200,000 records per second). This program parses defined SMF data fields and stores them in tables or in JSON file. This program can be used on demand for solving current questions/problems or permanently for loading data to sophisticated, platform independent, analyzing tools, reporting databases or accounting software. This software is written in 100% in Java therefore can use only ziip processors and can be used on distributed platforms especially if SMF records are archived in enterprise storage services.

20 Parameters json file

21 Selection section

22 Runtime section

23 SMF types/subtypes

25 Results table

26 Installing and running Long Term Analysis The Long Term Analysis can be installed and run under: z/os as: Batch job using JZOS framework or comparable method. The installation and JCL are almost the same like applications Monitor. See slide 11 Shell script under USS java -jar LongTermAnalysis.jar --inpfile="//your MVS dataset'" --tablespath=. MS Windows; Linux; in this same way like under USS. Input file need to be downloaded for example with FTP as binary, with the SITE RDW parameter turned on. If you have downloaded SMF records for IBM SCRT reports you can use them at once. Currently System Work offers this software free of charge.

27 Accounting / billing Traditionally accounting in corporations with own datacenters is based on distributing cost to departments which used IT services. Typically processor time normalized to MSUs or MIPS is used for charging/reporting. In current Z system technology the processor time is not really proportional to the amount of processed information. Management frequently expects to be charged according to the amount of processed business transactions - sold tickets. The new feature of SMF 30 Counter data section opened interesting opportunities for charging on the base of real MIPS. The numbers of executed instructions is proportional to processing information.

28 Recent questions Real questions recently received: We are invested in Crypto cart. How effectively are we using this hardware? Maybe do we need more of them? We don t use currently ziip processors. Which performance benefits can take us buying of this processors? How much could be optimal? To find answers with classic reporting tools usual is impossible even if newest SMF records are available. The dynamically defined reports in Long Term Analysis will deliver the answers precisely.

29 Conclusion Application disruptions can have heavy impact on business operations State-of-the-art Application Monitoring on z/os can therefore save time and money SMF records can provide valuable information to monitor business applications Feel free to try the Long Term Analysis application

30 Thank you Dr Tomasz Cieplak Chief of Development

uk/2017/feedback/oc Paper feedback forms

31 Session feedback Please submit your feedback online at: Paper feedback forms also available from the Chair person This session is OC

The Power to Stream z IT Operational Data to the Analytic Engine of Your Choice

The Power to Stream z IT Operational Data to the Analytic Engine of Your Choice Domenico D Alterio IBM November 2018 Session OK Agenda Business challenges IBM Common Data Provider for z Systems Overview