CICS Instrumentation Data for Cloud and Mobile

Transcription

1 GUIDE SHARE EUROPE UK REGION GSE UK Conference 2015 z Systems: The Cloud has a silver lining CICS Instrumentation Data for Cloud and Mobile Ezriel Gross Circle Software Incorporated Tuesday 3 November Session GD Copyright Circle Software Incorporated 2015 Session feedback

2 Agenda Background z/os as a component of cloud applications A requirement Wider access to z/os instrumentation data A general solution Extract, transform, and load (ETL) to off-host analytics software A specific example ETL CICS performance data to open-source ELK stack analytics software CICS performance data presented in interactive browser-based dashboards Beyond the dashboard Deep dive into instrumentation data for z/os-specific issues 2

3 Terminology 3

4 What s a log? Time-based data; data that consists of timestamped records Each record contains a time stamp and some other fields: Log record Time stamp field2 field3 field4 Not always a simple sequence. Can contain structures, such as repeating groups of fields. This definition includes data that, in a z/os context, might not typically be referred to as a log: SMF, such as CICS monitoring facility performance class (SMF 110) records CICS trace Also known as instrumentation data, to distinguish it from application data such as customer or product details 4

5 5 A dashboard is a user interface that graphically presents key performance indicators in a set of visualizations, such as bar charts

6 Background z/os as a component of cloud applications 6

7 Cloud applications from a user s perspective User It just works! Mobile device 7

8 Cloud applications from a provider s perspective Web service Instrumentation data (logs) Users Web application server z/os? Analytics tools e.g. dashboards Mobile device Web service 8

9 There s a cloud of subsystems in z/os, too z/os CICS IMS DB2 MQ WAS SMF A single transaction can span multiple subsystems, with activity recorded in separate, subsystem-specific logs 9

10 A requirement Wider access to z/os logs 10

11 Access to application data versus logs Cloud developers can access application data without z/os skills 11 RESTful APIs and other abstract APIs that hide z/os implementation details but the corresponding activity logs are inaccessible z/os Application data Logs Don t want it all! Key performance fields Fields for correlation with off-host logs Identification fields to pass back to z/os experts for deep-dive analysis

12 Log types on z/os In this presentation, we ll focus on this data IMS CICS DB2 MQ, WAS z/os IMS log and trace CMF performance class (SMF 110) DB2 log MQ log extract SMF IMS monitor CICS trace (DFHAUXT, GTF) DB2 accounting MQ statistics (SMF 115-1, -2) OPERLOG CQS log stream VSAM forward recovery journal log stream DB2 performance trace (IFCIDs) MQ accounting (SMF 116) z/os Connect (SMF ) IMS Connect events data (collected by IMS Connect Extensions) Near Term History (collected by OMEGAMON XE for DB2) WAS request activity performance statistics (SMF 120-9) OMEGAMON ATF IRLM long lock detection (SMF 79-15) 12 Just a thought: wouldn t it be great to have a tool that could analyze all of these log types in depth on z/os, and also extract the data you want, in the format you need, for off-z/os analytics?

13 A general solution Extract, transform, and load (ETL) z/os logs to off-host analytics software 13

14 What we will demonstrate Given that you already have this: We demonstrate this: z/os Linux Web browser CICS SMF 110 records in log stream or dumped MANx files Analytics software Dashboard 14

15 Getting log data off z/os Various specific methods and tools A general solution: use a scheduled batch job to extract the data you want from the data sets or log streams that your z/os subsystems already write to: z/os Subsystems (CICS, DB2, SMF, etc.) Data sets Batch job Scheduled daily? Hourly? Triggered by an event? Up to you This general solution allows many options for the specific methods and tools for transferring the extracted data off z/os (we ll describe some later) 15

16 Questions to consider What data do we want to ETL? Which records? For only some CICS transactions? What are the selection criteria? Which fields in those records? Which performance indicators do we want to chart? Which fields can we use for correlation with off-host log data? Which identification fields might we need to communicate back to a z/os subject-matter expert for deep dive problem diagnosis? Which data format(s) can our analytics software ingest? Which format is best? How will we transfer the data? What metadata (information about the data, and what to do with it) does our analytics software require? 16

17 A specific example ETL of CICS performance data to the ELK stack 17

18 What you will need Function Extract the records and fields you want Transform into a data format that can be ingested by your analytics software Load into your analytics software, including transfer from z/os Analytics Dashboard What we used IBM Transaction Analysis Workbench for z/os ( Workbench ) Workbench, Logstash Redis, Logstash Elasticsearch Kibana Elasticsearch, Logstash, and Kibana are collectively known as the ELK stack 18

19 IBM Transaction Analysis Workbench for z/os z/os Other (e.g. Linux) Supported log types Metadata Logstash config HCatalog Log forwarding SFTP ELK stack Kibana Elasticsearch Workbench Flexible selection criteria to extract only the data you want Log data Schema JSON CSV Co:Z Toolkit fromfile Redis Logstash Hadoop Hive BigSheets, etc 19

20 ELK stack Open source, on GitHub All (now) developed by Elastic Windows, Linux, OS X Scalable to petabytes Very popular Active development Kibana Web-browser based dashboard RESTful HTTP APIs Elasticsearch Distributed, scalable search and analytics, based on Apache Lucene Data Logstash config Logstash Log transformation and forwarding 20

21 Example: CICS data in a Kibana dashboard CICS performance data (SMF 110 records) from here displayed in a dashboard here Kibana Elasticsearch SMF records Workbench Config Logstash z/os batch job JSON log data Redis client shipper Redis server Linux 21

22 Dashboards in Kibana Creating charts, then arranging them into a dashboard 22

23 Creating a dashboard Step 1. Create and save visualizations (bar charts, line charts, area charts, pie charts, etc.) 23

24 24 Step 2. Drag n drop saved visualizations into a new dashboard

25 Dashboards are interactive Drag over a time-based series to zoom in on a time interval 25

26 or set the time interval to a specific relative or absolute range You can quickly zoom from viewing months or years of data to a fraction of a second 26

27 Dashboards can also contain tables You define the search criteria and which fields to display as columns You can expand each row to show all fields 27

28 Mixing log types in a dashboard CMF (SMF 110) DB2 accounting (SMF 101, IFCID 3) 28

29 You re not restricted to using Kibana Kibana is great, but what if it doesn t offer exactly the visualizations you need? Kibana is just a front end to Elasticsearch Kibana uses the Elasticsearch RESTful HTTP APIs Search results are returned as JSON You can develop your own apps using these APIs Kibana Web-browser based dashboard RESTful HTTP APIs You can use these APIs, too Elasticsearch Distributed, scalable search and analytics, based on Apache Lucene 29

30 Recap Extract, transform, load: what we did, what we used 30

31 Extract z/os subsystems CICS IMS DB2 From the logs you have Workbench reads logs that subsystems already write to Logs As much detail as you want Workbench lets you select from all significant fields in each log type, not just a small subset Workbench Batch job Only what you want Workbench offers flexible filter criteria to select only the log records that you are interested in 31 As often as you like Scheduled, ad hoc: up to you

32 Transform Logs CSV or JSON Workbench outputs the two most common formats ingested by analytics software, with various options to suit specific requirements Data Workbench Metadata Specific support for ELK and Hadoop Workbench creates Logstash configs.hcatalog files General-purpose metadata with detailed field descriptions 32

33 Load Redis offloads data from z/os fast z/os Linux Data Redis client shipper TCP/IP Redis server Logstash Elasticsearch Multiple shipper threads Set the number of threads used to forward data, for optimal performance Queues Multiple Logstash instances Logstash config specifies the number of Logstash instances to start (to read Redis queues) 33

34 Beyond the dashboard What if your dashboard indicates a problem on z/os? 34

35 Deep-dive analysis of z/os-specific issues Your dashboard indicates an issue on z/os For example, one of the key performance indicators that you have offloaded, such CICS transaction response time, indicates an issue on z/os. Who are you going to call? You need a z/os subject matter expert to investigate the issue. To examine the issue in detail, that expert might require more log data than you have chosen to offload for performance and high-level diagnosis. What are they going to use? There are many subsystem-specific (CICS, DB2, IMS, etc.) tools for z/osbased problem diagnosis. We are aware of only one tool that offers both offload and deep-dive diagnosis of z/os transactions across z/os subsystems 35

36 Workbench: ETL off z/os, deep dive on z/os 2. Perform deep-dive analysis using the detailed log data on z/os, not just the fields you selected to ETL. z/os Off-host Original logs Deep-dive log analysis Workbench ETL Selected z/os log data Analytics software Log extracts 1. The z/os log data you selected to ETL indicates a problem on z/os. Typically, you will need more detail for diagnosis. What now? 36

37 Tracking finds and fits the pieces together 37 Use values from your analytics software to identify a matching record on z/os e.g. CICS region, task number, time Workbench finds related records across all log types Workbench ISPF dialog log browser CMF record CICS trace records DB2 trace records DB2 log records showing the transaction lifecycle on z/os in time sequence Original logs or extracts by Workbench, e.g. SMF file or log stream DB2 log CICS trace: DFHAUX or GTF

38 Deep dive: Workbench log browser Long response time is caused by DB2 stored procedure FBOSP006 (called by program DSNCUEXT) performing many get page requests 38 BROWSE JCH.FUWDEMO.P D T CICS Record More: < > Command ===> TRACE 1 Scroll ===> CSR Navigate < > Date/Time / Tracking Tuesday Time (Elapsed) TX 6E13 CICS Transaction TranCode=FB66 Task= TranCode=FB66 Program=FBOCCP66 Userid=TWM LTerm=SC0TCP07 Terminal=CP07 RecToken=FUWTCIC/CC145FF1F9E7C984 Resp= CPU= DB2=2 ACCT=FTS3.SC0TCP07.145FF1F9E7C9 Task=251 AP 1940 APLI ENTRY START_PROGRAM PROGRAM(FBOCCP66) AP E160 EXEC ENTRY LINK PROGRAM('FBODCP06') AP 1940 APLI ENTRY START_PROGRAM PROGRAM(FBODCP06) AP 1940 APLI ENTRY START_PROGRAM PROGRAM(DSNCUEXT) AP E160 EXEC ENTRY RETURN ASM Thread allocate FBODCP06 DBA Package allocation FBODCP06 DBA SP entry FBOSP006 STMT= DBA Package allocation FBOSP006 DBA SP exit FBOSP006 SQLCODE=0 STMT= DBA AP E160 EXEC ENTRY RETURN COBOLII STMT_#(00311) AP E161 EXEC EXIT LINK PROGRAM('FBODCP06') AP E160 EXEC ENTRY RETURN TRANSID('FB66') Package accounting-sp DBA Thread accounting DBA TranCode=FB66 Userid=TWM ClientID=FUWTCIC ET1= ET2= CPU1= CPU2= Suspend= ACCT=FTS3.SC0TCP07.145FF1F9E7C9 Source=CICS SEL=1 UPD=1 CAL=1 LogRecs=6 GetPage=14616 UpdPage=1 MaxLock=2 LUWID=FTS3/DBA6LU/CC145FF1FAB4/0002 ******************************* Bottom of Data ********************************

39 Complete, formatted log record details QW0058ID... Scan information Scan type... 'SEQD' Rows processed Rows examined Rows qualified after stage Rows qualified after stage Rows inserted Rows updated Rows deleted Pages scanned Pages scanned (RI) Rows deleted (RI) Pages scanned (LOB) Pages updated (LOB) QW0058ID... 'SEQD' Scan type Off QW0058IX... 'INDX' Index On QW0058SD... 'SEQD' Sequential data Off QW0058SW... 'SEQW' Sequential data workfile A0 QW0058TY... Statement-level information +01A0 SQL statement type A8 Statement ID B0 Sync reads Getpages C0 Rows examined C8 Rows processed Sorts D8 Index scans E0 Table space scans E8 Buffer writes F0 Parallel groups F8 In-DB2 elapsed This example shows a DB2 accounting record (SMF 101, IFCID 3)

40 CICS-VSAM transaction CICS-related data sources Supported by Workbench include CMF, CICS trace, and VSAM forward recovery journal log stream 40 BROWSE JCH.CTR.VSAM.EXTRACT0 + Record More: < > Command ===> Scroll ===> PAGE Navigate < > Date/Time / Tracking Tuesday Time (Elapsed) 6E13 CICS Transaction TranCode=EVSM Task= AP 1940 APLI ENTRY START_PROGRAM PROGRAM(EXVSAM01) AP E160 EXEC ENTRY WRITEQ TD QUEUE('CESE') FROM(' AP E161 EXEC EXIT WRITEQ TD QUEUE('CESE') FROM(' AP E160 EXEC ENTRY WRITE FILE('EXMPLOG') AP E161 EXEC EXIT WRITE FILE('EXMPLOG') AP E160 EXEC ENTRY WRITE FILE('EXMPCAT') VSAM file EXMPCAT write add VSAM file EXMPCAT write add complete Delay AP E161 EXEC EXIT WRITE FILE('EXMPCAT') AP E160 EXEC ENTRY WRITE FILE('EXMPLOG') AP E161 EXEC EXIT WRITE FILE('EXMPLOG') AP E160 EXEC ENTRY READ FILE('EXMPCAT') VSAM file EXMPCAT read update AP E161 EXEC EXIT READ FILE('EXMPCAT') AP E160 EXEC ENTRY WRITE FILE('EXMPLOG') AP E161 EXEC EXIT WRITE FILE('EXMPLOG') AP E160 EXEC ENTRY REWRITE FILE('EXMPCAT') VSAM file EXMPCAT write update AP E161 EXEC EXIT REWRITE FILE('EXMPCAT')

41 Conclusion 41

42 Conclusion 42 ELK is open source ( free ), scalable, easy to install and configure, has serious users, increasing adoption, and is being actively developed: just add data How? One solution: a tool to extract, transform, and load (ETL) your z/os logs One example: IBM Transaction Analysis Workbench for z/os When choosing a solution, consider: What ETL coverage the solution offers across the various log types from different z/os subsystems (CICS, DB2, IMS, MQ, etc.) How you will perform deep-dive analysis - requiring more detail than you have offloaded - for issues that are identified as z/os-specific How many different tools does the solution involve? Do they offer an integrated approach across z/os log types?

43 Session feedback Please give your feedback on this session at: 43

44 Questions? 44

45 Supplementary slides 45

46 Interactive filtering 1. Click an item, such as a bar, in a visualization 2. The dashboard includes only records with that value, indicated by a green filter lozenge 3. You can toggle a filter to include or exclude 4. The filter lozenge turns red, indicating exclude 46

47 Aggregations: percentiles Percentiles are useful for service level agreements (SLAs) 47

48 48 Aggregations: ranges

49 Visualizations with nested aggregations Multilevel pie charts can provide useful insights, but you need to understand what you re looking at! CICS applid tran userid Who is running what, and where? From inner to outer ring: Top 5 applids Top 5 transaction IDs (on those applids) Top 5 users (of those transactions, on those applids) Again, you can interactively create filters just by clicking: 49

50 ELK tips Change the default Elasticsearch index property value for string fields from analyzed to not_analyzed. Otherwise, searching, matching, and aggregating based on string field values can have unexpected results. For convenience, use all lowercase field names (not mixed case), especially if you want to write your own ad hoc Lucene queries Kibana uses field names as labels on charts; as of Kibana 4.1.2, with the exception of custom labels for filter expressions, Kibana does not allow custom labels in place of field names. By default, Kibana formats numbers to 3 decimal places. You might want to increase this; for example, to show elapsed time values to microseconds: set format:numberprecision (under Settings Advanced) to 6. In common with many off-host analytics platforms, time stamp precision is limited to milliseconds (0.001). 50

51 Coverage across z/os subsystems IMS CICS DB2 MQ WAS z/os Consolidation of different logs in a single user interface IBM Transaction Analysis Workbench for z/os Session manager Automated file selection Interactive log browsing Batch processing and reporting Transformation (Elasticsearch/ Hadoop) Collaboration between experts and other users First responders 51 System programmers Subsystem admins Application developers Other subjectmatter experts

52 Automated file selection DB2 logs IMS logs (SLDS) IMS Connect Extensions journal SMF log streams or (dumped MANx) files OMEGAMON XE for DB2 PE near-term history DB2 bootstrap data set (BSDS) RECON data sets DB2 print log map utility DBRC API Transaction Analysis Workbench automated file selection utility 52 System definition repository