Customer Experiences:

Transcription

1 IBM z Systems Customer Experiences: Monitoring and Managing z/vm, Linux on z Systems and LinuxONE Tracy Dean IBM tld1@us.ibm.com Mike Sine IBM sine@us.ibm.com June Customer Experiences Managing z/vm and Linux on z Systems

2 Notices and disclaimers Copyright 2017 by International Business Machines Corporation (IBM). No part of this document may be reproduced or transmitted in any form without written permission from IBM. U.S. Government Users Restricted Rights use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM. Information in these presentations (including information relating to products that have not yet been announced by IBM) has been reviewed for accuracy as of the date of initial publication and could include unintentional technical or typographical errors. IBM shall have no responsibility to update this information. This document is distributed as is without any warranty, either express or implied. In no event shall IBM be liable for any damage arising from the use of this information, including but not limited to, loss of data, business interruption, loss of profit or loss of opportunity. IBM products and services are warranted according to the terms and conditions of the agreements under which they are provided. IBM products are manufactured from new parts or new and used parts. In some cases, a product may not be new and may have been previously installed. Regardless, our warranty terms apply. Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without notice. Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual performance, cost, savings or other results in other operating environments may vary. References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services available in all countries in which IBM operates or does business. Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not necessarily reflect the views of IBM. All materials and discussions are provided for informational purposes only, and are neither intended to, nor shall constitute legal or other guidance or advice to any individual participant or their specific situation. It is the customer s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal counsel as to the identification and interpretation of any relevant laws and regulatory requirements that may affect the customer s business and any actions the customer may need to take to comply with such laws. IBM does not provide legal advice or represent or warrant that its services or products will ensure that the customer is in compliance with any law. 2 Customer Experiences Managing z/vm and Linux on z Systems

3 Notices and disclaimers continued Information concerning non-ibm products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products in connection with this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-ibm products. Questions on the capabilities of non-ibm products should be addressed to the suppliers of those products. IBM does not warrant the quality of any third-party products, or the ability of any such third-party products to interoperate with IBM s products. IBM expressly disclaims all warranties, expressed or implied, including but not limited to, the implied warranties of merchantability and fitness for a particular, purpose. The provision of the information contained herein is not intended to, and does not, grant any right or license under any IBM patents, copyrights, trademarks or other intellectual property right. IBM, the IBM logo, ibm.com, Aspera, Bluemix, Blueworks Live, CICS, Clearcase, Cognos, DOORS, Emptoris, Enterprise Document Management System, FASP, FileNet, Global Business Services, Global Technology Services, IBM ExperienceOne, IBM SmartCloud, IBM Social Business, Information on Demand, ILOG, Maximo, MQIntegrator, MQSeries, Netcool, OMEGAMON, OpenPower, PureAnalytics, PureApplication, purecluster, PureCoverage, PureData, PureExperience, PureFlex, purequery, purescale, PureSystems, QRadar, Rational, Rhapsody, Smarter Commerce, SoDA, SPSS, Sterling Commerce, StoredIQ, Tealeaf, Tivoli Trusteer, Unica, urban{code}, Watson, WebSphere, Worklight, X-Force and System z Z/OS, are trademarks of International Business Machines Corporation, registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at: 3 Customer Experiences Managing z/vm and Linux on z Systems

4 Agenda A little fun What does managing include? What tools or products can you use? Customer scenarios Operational monitoring and automation Performance monitoring Backup and recovery Summary and reference information 4 Customer Experiences Managing z/vm and Linux on z Systems

5 What is Managing and What Tools Can I Use? 5 Customer Experiences Managing z/vm and Linux on z Systems

6 Administration and Provisioning Administer Linux guests/servers via GUI View of all servers graphically Run shell scripts against a server or group of servers Activate or deactivate a server or group of servers Login to server directly from GUI View and modify network connections Provision Linux guests/servers Across LPARs or machines Memory and CPU Network connect to Guest LANs or VSWITCHes Storage based on admin-defined device pools Customize first boot before TCPI/IP initialized Customize cloning via REXX scripts Real time monitoring High level view of system status via dashboard gauges View storage utilization 6 Customer Experiences Managing z/vm and Linux on z Systems

7 Administration and Provisioning Manage and administer Linux guests/servers via GUI View of all servers graphically Run shell scripts against a server or group of servers Activate or deactivate server or group of servers Login to server directly from GUI View and modify network connections IBM Wave for z/vm Vendor solutions Provision Linux guests/servers Across LPARs or machines Homegrown Servers - memory tools and CPU Network connect to Guest LANs or VSWITCHes Storage based on admin-define device pools Customize first boot before TCPI/IP initialized Customize cloning via REXX scripts Real time monitoring High level view of system status via dashboard gauges View storage utilization 7 Customer Experiences Managing z/vm and Linux on z Systems

8 Performance Monitoring and Automation Monitor performance based on best practices Virtual CPU for each guest z/vm processor utilization Spin lock wait Virtual disk utilization Virtual storage utilization with V/R memory ratio Formation and size of eligible list Page and spool space utilization and I/O rates DASD I/O and minidisk cache usage Resource constraint analysis Use historical data to Understand capacity Size Linux guests for best performance in a hosted (shared) environment 8 Customer Experiences Managing z/vm and Linux on z Systems

9 Performance Monitoring and Automation Monitor performance based on best practices Virtual CPU for each guest z/vm processor utilization Spin lock wait Virtual IBM disk utilization OMEGAMON Vendor Virtual XE on storage z/vm utilization with V/R solutions memory ratio Formation and Linuxand size of eligible list IBM Page and spool space Performance utilization and I/O rates DASD I/O and minidisk Toolkit cache for usage Resource constraint z/vm analysis Homegrown tools Use historical data to Understand capacity Size Linux guests for best performance in a hosted environment 9 Customer Experiences Managing z/vm and Linux on z Systems

10 Operational Monitoring and Automation Console monitoring and viewing Operations staff monitoring a central console of alerts System programmers debugging a problem on a guest or service machine Console log data available for audits or future reference Generate alerts and/or automatically recover from Abend, termination, or error messages Service machine disks approaching full Critical user IDs or guests being logged off or entering error state Spool and/or page space approaching full Schedule automated system maintenance procedures Spool cleanup based on policies Minidisk cleanup (from logs), including archiving Orderly startup and shutdown Relocation of critical guests to another SSI member Backups of z/vm system 10 Customer Experiences Managing z/vm and Linux on z Systems

11 Operational Monitoring and Automation View & issue commands on consoles of Linux guests and CMS service machines Operations staff monitoring multiple consoles or a central console of alerts System programmers debugging a problem on a guest or service machine Generate alerts and/or automatically recover from IBM Abend, termination, Operations or error messages Vendor Service Manager machine for disks approaching full solutions Critical user z/vm IDs or guests IBM being Wakeup, logged off or entering error state Spool and/or page space approaching PROP, full *VMEVENT, *MSG, etc Homegrown tools Spool cleanup based on policies Schedule automated system maintenance procedures Minidisk cleanup (from logs), including archiving Orderly startup and shutdown Relocation of critical guests to another SSI member Backups of z/vm system 11 Customer Experiences Managing z/vm and Linux on z Systems

12 Backup and Recovery of z/vm and Linux Image level backup of z/vm Operating system File level backup of z/vm data Directory information Configuration files Log files Tools REXX EXECs, automation scripts, etc. Image level backup of Linux guests Operating system Applications Application data (maybe) File level backup of Linux guests Configuration files Log files Tools 12 Customer Experiences Managing z/vm and Linux on z Systems

13 Backup and Recovery of z/vm and Linux Image of file level backup of z/vm Operating system IBM DDR IBM Spectrum Protect (aka Image level backup of Linux guests Operating system Homegrown Applications tools Application data (maybe) Tivoli Storage Manager) File level backup of z/vm data Directory information Configuration filesvendor Log files solutions IBM Backup Tools REXX EXECs, automation scripts, etc. and Restore Manager for z/vm File level backup of Linux guests Configuration files Log files Tools 13 Customer Experiences Managing z/vm and Linux on z Systems

14 Complete Solution for administration and management of the z/vm and Linux on z Systems environment IBM Infrastructure Suite for z/vm and Linux OMEGAMON XE on z/vm and Linux Performance monitoring of z/vm and Linux guests Spectrum Protect (formerly TSM) File Level backup and recovery for Linux Virtual Machines IBM Wave for z/vm Simple, intuitive, graphical z/vm administration tool, including provisioning Single PID 5698-IS2 OTC 5698-IS1 Annual S&S Operations Manager for z/vm Facilitate operational monitoring and automated operations, take action based on events Backup and Restore Manager for z/vm Image and file level backup and restore of z/vm environment Image level backup and restore of Linux Add Tape Manager for z/vm (5697-J08) for customers backing up from z/vm to tape 14 Customer Experiences Managing z/vm and Linux on z Systems

15 Customer Scenarios Operational Monitoring and Automation Performance Monitoring and Troubleshooting Backup and Recovery 15 Customer Experiences Managing z/vm and Linux on z Systems

16 Error Messages on Linux IPL The Situation: During boot process, Linux file system is read-only Application needs read/write But sometimes not until hours or days after boot Error discovered hours or days later when application fails Operations Manager Initial solution Write homegrown tool Scan logs on a daily basis looking for error messages Final solution Console monitoring tool Write a rule looking for error message during boot process and take action immediately 16 Customer Experiences Managing z/vm and Linux on z Systems

17 Error Message on z/vm IPL The Situation: Error messages on z/vm IPL Reason unknown to customer (new to z/vm) Operations Manager Initial solution Final solution None Took photo of HMC with smartphone Show IBM and ask for help EREP & Accounting disks full No knowledge of impact Monitoring tool Simple monitor setup Automatically monitor percent full someone who can follow documented procedures to save/archive data 17 Customer Experiences Managing z/vm and Linux on z Systems

18 Why Was an Application Running Slow The Situation: Application owner asks z/vm system programmer why application was running slow yesterday afternoon Application owner doesn t have the data he needs to research the problem OMEGAMON Initial Solution Look at performance data for the Linux guest Performance data in logs for the Linux operating system No application data Final solution One performance monitoring solution for all layers Hypervisor Linux operating system Application 18 Customer Experiences Managing z/vm and Linux on z Systems

19 Why Was an Application Running Slow The Situation: Application owner asks z/vm system programmer why application was running slow yesterday afternoon Application owner doesn t have the data he needs to research the problem Drill down to each layer within a specified time Initial Solution window Final solution Look at performance data for the Linux guest One performance monitoring solution for all layers Performance data in logs for the Linux operating system No application data Hypervisor Linux operating system Application 19 Customer Experiences Managing z/vm and Linux on z Systems

20 Why Was an Application Running Slow z/vm Linux on z Systems Notice an anomaly at the z/vm workload level Link to the Linux Process view Link to one or more DB2 views DB2 UDB Agent 20 Customer Experiences Managing z/vm and Linux on z Systems

21 Coordinate Application Shutdown with z/os The Situation: Database on z/os Application server on Linux on z Shutdown of database necessitates shutdown of app server Operations Manager Initial solution Manual coordination of shutdown Inconvenient for system programmers/operations during non-business hours Final solution Console monitoring tools System Automation on z/os sends message to z/vm Automation on z/vm triggers application server shutdown Automation on z/vm sends message to z/os when ready 21 Customer Experiences Managing z/vm and Linux on z Systems

22 Resource Utilization Reports The Situation: Linux admins misinterpret utilization of their virtual servers Overwhelm support with (unnecessary) demands for additional resources Sysadmin tools don t show correct breakdown in a virtual server OMEGAMON Initial solution SysAdmin Tools Tools like TOP and others don t reflect the virtualized environment. Users get mixed information and make wrong conclusions. Misunderstanding between application owners, Linux admins, and system providers Final solution Monitoring tool Develop reports CPU utilization max and average Monthly memory utilization breakdown Linux admins and application owners satisfied they are getting necessary resources 22 Customer Experiences Managing z/vm and Linux on z Systems

23 Maximum and Average CPU example 23 Customer Experiences Managing z/vm and Linux on z Systems

24 Average Linux Memory Breakdown Example 24 Customer Experiences Managing z/vm and Linux on z Systems

25 Shared Monitoring and Automation Across LPARs The Situation: Multiple z/vm LPARs not in same SSI cluster Similar monitoring and automation configuration on all LPARs Operations Manager Initial solution Manual processing Common configuration information maintained on one system Shared within SSI cluster Manually sent and reloaded on other LPARs Final solution Automated real-time sharing Shared read/only disk across non-ssi members Update configuration from single LPAR Automatically reload on all SSI and non-ssi systems 25 Customer Experiences Managing z/vm and Linux on z Systems

26 Hypervisor Using 25% of CPU The Situation: Most monitoring focuses on CPU utilization overall OMEGAMON Missing focus on CP s % of CPU as a separate metric How much is the hypervisor using? Best Practice is to investigate if hypervisor using > 10% of CPU One morning found CP% at 25%, simple drill down revealed cause Initial solution None System CPU measured, while hypervisor (CP) specific numbers omitted Reactive steps only taken when performance issue arose Final solution Monitoring tool Automatically monitor CP % for threshold of 10% Once threshold is alerted, simple proactive drill down in tool reveals impact often before downstream performance impact is noticed 26 Customer Experiences Managing z/vm and Linux on z Systems

27 System Processor Utilization Workspace 27 Customer Experiences Managing z/vm and Linux on z Systems

28 z/vm Workload Workspace Sort virtual machines by CP s % of CPU 28 Customer Experiences Managing z/vm and Linux on z Systems

29 Including Performance Data with z/os Processes The Situation: Collecting performance data on z/vm (Performance Toolkit) All mainframe performance data processed on z/os Want to include z/vm and Linux data Operations Manager Initial Solution Manual processing each morning Login and run commands to Summarize PerfKit data FTP file to z/os Erase file from z/vm Final solution Automated processing overnight Schedule commands to summarize data Message to z/os z/os FTP file from z/vm z/os FTP message to z/vm indicating successful file retrieval z/vm erase the file 29 Customer Experiences Managing z/vm and Linux on z Systems

30 System Abend with No Console Data The Situation: Operations Legacy best practice of spooling consoles Manager System abends IPL with warm start unsuccessful or not possible No console data to review what happened leading up to abend Dump data only Initial solution IPL cold start and hope for the best Or IPL cold start and dig through dump data Final solution Console monitoring tool IPL cold start and review console data written in one log file on disk 30 Customer Experiences Managing z/vm and Linux on z Systems

31 Spool and Page Space Full The Situation: Spool and pagespace fill up System abends Unplanned outage Operations Manager Initial solution Homegrown tool Create a service machine running WAKEUP Check spool and page space percent full on regular intervals Maintain service machine and code Final solution Monitoring tool Simple monitor setup Watch for percent full to be within threshold range Watch for sudden growth Take action Easily add or change threshold or frequency 31 Customer Experiences Managing z/vm and Linux on z Systems

32 Graceful Shutdown of z/vm from GDPS The Situation: Shutdown of z/vm LPAR included in GDPS processing Shutdown of Linux guests handled by GDPS Need graceful shutdown of z/vmwithout triggering monitoring and automation Operations Manager Initial solution None GDPS handled shutdown of guests Shutdown of z/vm interfered with monitoring and automation Final solution Automated graceful shutdown GDPS signal triggers automation Runtime monitors and automation suspended Shutdown monitors and automation resumed 32 Customer Experiences Managing z/vm and Linux on z Systems

33 Alert for Excessive CPU by Virtual Server The Situation: Dev/test environment 2 IFLs Using Performance Toolkit Single guest is looping, impacting other guests No alerting until someone complained OMEGAMON Operations Manager Initial Solution Final solution Set thresholds in Performance Toolkit Data about CPU utilization in PerfKit logs Threshold alert written to PerfKit log No detailed Linux data Use automation solution to send alerts to central alert system Long term, still need enterprise performance monitor for z/vm and Linux 33 Customer Experiences Managing z/vm and Linux on z Systems

34 Central Operations Console Already have z/os console in operations center Alerts, important messages, etc. for operations staff Want one console for all z/vm LPARs and Linux guests Operations staff sees only important messages on central console When needed can also look at full console of any specific user ID or guest Can expand to include more LPARs as environment grows Still a single console Operations Manager 34 Customer Experiences Managing z/vm and Linux on z Systems

35 Creating a Central Console Operations Console OPERATOR OPERATOR OP Message 1b OP Message 2b OP Mount Message 3b CP SECUSER CP SECUSER OP Message 1b OP Message 2b OP Mount Message 3b Console Monitor (Operations Mgr) CP MSGNOH or TCP/IP CP MSGNOH Console Monitor (Operations Mgr) DIRMAINT DIR Message 1b DIR Abend Msg 2b DIR Message 3b OPERMSGS Urgent Message 1a Abend Message 2b Mount Message 3a Urgent Message 1b Abend Message 2a Mount Message 3b DIRMAINT DIR Message 1b DIR Abend Msg 2b DIR Message 3b z/vm A LINUX01 LX2 Urgent Msg 1b LX2 Message 2b LX2 Message 3b z/vm B ADMIN1 LINUX02 LX2 Urgent Msg 1b LX2 Message 2b LX2 Message 3b 35 Customer Experiences Managing z/vm and Linux on z Systems

36 Perform Weekly System Healthcheck The Situation: Need to monitor system to verify not approaching a threshold Spool space filling up Paging space filling up Disk full for several z/vm service machines or guests Operations Manager Initial solution Logon weekly and go through checklist manually Check disk space Check page space Check spool space Final solution Automate regular monitoring and alerts team if anything approaches threshold 36 Customer Experiences Managing z/vm and Linux on z Systems

37 Perform Weekly System Healthcheck The Situation: Need to monitor system to verify not approaching a threshold Disk full for several z/vm service machines or guests Spool space filling up Paging space filling up Add additional automation to automatically clean up the disk Back up or archive data Erase files Initial solution Logon weekly and go through checklist manually Check disk space Check page space Check spool space Final solution Automate regular monitoring and alerts team if anything approaches threshold 37 Customer Experiences Managing z/vm and Linux on z Systems

38 Unidentified Change in Performance The Situation: System performed at normal level for period of time CPU utilization Over several days, steady increase until new normal No new applications or virtual servers Unknown cause OMEGAMON Initial solution Accept the new normal Or Dig through performance log data (MONITOR records or Performance ToolKit) Final solution Change control, historical data collection and access Lock in on the period of time of the increase Find the specific virtual servers contributing to increase Review change control records 38 Customer Experiences Managing z/vm and Linux on z Systems

39 On-Demand: Persistent Historical Views 39 Customer Experiences Managing z/vm and Linux on z Systems

40 History On-Demand with Persistent Historical Views Easier to see anomalies, or match spikes Capturing performance data as a base line is a must: General history data business as usual Detailed raw monitor data prior to and following any major changes Ability to review attributes of a past incident through the enterprise view! On-Demand through the Portal or Batch 40 Customer Experiences Managing z/vm and Linux on z Systems

41 Send z/vm and Linux Alerts to z/os The Situation: Extensive automation for alerts already running on z/os Automation and operations teams trained there Want all mainframe alerts to be handled this way Needz/VM andlinux on z alerts included Operations Manager Initial solution None z/vm and Linux alerts sent via only Mainframe operations team not able to participate Final solution Monitoring/automation tool Trigger alerts for z/vm and Linux events, messages, etc. Send via syslog writer to z/os USS syslog Configure USS syslog to send all alerts from z/vm to z/os syslog 41 Customer Experiences Managing z/vm and Linux on z Systems

42 Painful Recovery of Critical z/vm Files The Situation: Backups of z/vm volumes done from z/os Operational issue (aka user error) corrupts a configuration file Recovery is tedious and error-prone process Restoring whole volume Mapping a new minidisk to the right location on the volume Recovery very difficult if corrupted file is USER DIRECT Backup Manager Initial solution Train peopleto make backup copies before updating a file Final solution File level backup and recovery Weekly full backups and daily incrementals of all z/vm files 42 Customer Experiences Managing z/vm and Linux on z Systems

43 Summary and Reference Information Production (and dev/test) systems need Monitoring operational and performance Automation Backup and recovery Real situations need to be addressed Learn from others Solutions exist Demos available Contact Tracy Dean, Mike Sine, 43 Customer Experiences Managing z/vm and Linux on z Systems

44 IBM Infrastructure Suite for z/vm and Linux Bundle/suite of IBM products Tools needed to manage the z/vm and Linux on z Systems infrastructure Wave for z/vm OMEGAMON XE on z/vm and Linux Operations Manager for z/vm Backup and Restore Manager for z/vm Order Tape Manager for z/vm separately if plan to back up to tape Spectrum Protect Extended Edition (formerly Tivoli Storage Manager) Discounted price as a bundle Website: DeveloperWorks Wiki videos of product use/demos IBMVM Mailing list: 44 Customer Experiences Managing z/vm and Linux on z Systems

45 Live Demos 45 Customer Experiences Managing z/vm and Linux on z Systems

46 Automation Demos Available 1. View consoles of Linux guests, Linux syslog data, and CMS user IDs or service machines 2. Send an based on a console message 3. Send an alert to Netcool/OMNIbus based on a console message, hold and unhold messages a. Using POSTZMSG interface to Netcool/OMNIbus b. Using SNMP interface to Netcool/OMNIbus 4. Send a message or if spool approaches full a. Send a message if spool usage is too high on any member of an SSI Cluster see how spool files appear in SSI b. Send an if spool usage is too high on a single system 5. View and clean up spool files 6. Automated spool cleanup 7. Back up or archive DIRMAINT s log files when disk gets full 8. Process a file of test messages as a console 9. Process Linux syslog data as a console 10. Create a central operations console on one z/vm system 11. Create a central operations console across multiple z/vm systems a. When the systems are in an SSI cluster b. When the systems are not in an SSI cluster 12. Integration with OMEGAMON XE on z/vm and Linux - take action based on CPU usage of Linux guest 13. Monitor service machines for logoff and autolog them 14. Send an if page space approaches full 15. Monitor SSI connectivity between 2 cluster members 16. Suppress passwords on Linux consoles 17. Autolog a Linux guest and send message if doesn t start successfully 46 Customer Experiences Managing z/vm and Linux on z Systems

47 Backup and Recovery Demos Available A. Performing an incremental backup B. Restoring files from backup C. Back up and restore single and multiconfiguration users in an SSI environment D. Scheduling image backups of Linux guests E. Suspend and resume a guest as part of backup F. Reviewing a disaster recovery backup G. Reviewing data in the backup catalog for recovery 47 Customer Experiences Managing z/vm and Linux on z Systems

48 Hindi Russian Traditional Chinese Ndzi khense ngopfu Tsonga Korean Gracias Spanish Arabic Ke a leboha Grazie Italian Thank English You Danke German Obrigado Brazilian Portuguese Merci Tswana Simplified Chinese French Tamil Japanese Thai 48 Customer Experiences Managing z/vm and Linux on z Systems