EMC GDDR for SRDF /S with ConGroup Version 3.1

Transcription

1 EMC GDDR for SRDF /S with ConGroup Version 3.1 Product Guide P/N REV A01 EMC Corporation Corporate Headquarters: Hopkinton, MA

2 Copyright EMC Corporation. All rights reserved. Published September, 2009 EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. THE INFORMATION IN THIS PUBLICATION IS PROVIDED AS IS. EMC CORPORATION MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com. All other trademarks used herein are the property of their respective owners. For the most up-to-date regulatory document for your product line, go to the Technical Documentation and Advisories section on EMC Powerlink. 2 EMC GDDR Product Guide

3 Contents Preface Chapter 1 Chapter 2 Product Overview What is EMC GDDR? Major features Situational awareness Survivor recognition Restart coordination Additional capabilities Types of environment Supported business continuity configurations SRDF/S with ConGroup configuration The EMC GDDR Complex EMC GDDR fundamentals Control systems Workload location Managed workloads External workloads Excluded systems HMC-only systems EMC GDDR processes EMC GDDR components Parameters User interface Events Monitors Message rules EMC GDDR supported scripts Planned event management Unplanned event management Resumption after planned or unplanned outages Parameter wizard: Telling EMC GDDR what to manage Installing EMC GDDR Preinstallation tasks Mainframe environment requirements Minimum software requirements Minimum hardware requirements EMC GDDR Product Guide 3

4 Contents DASD support Installation procedure Before you begin Gather EMC GDDR installation information Install EMC GDDR Run the installation jobs Post-installation tasks Chapter 3 Integrating EMC GDDR Overview Update system parameter files Authorize the EMC Consistency Group started task to use the trip API Perform ConGroup configuration Perform ConGroup Started Task automated startup Specify EMC GDDR security EMC GDDR RACF functional groups Summary of RACF permissions EMC GDDR user interface security RACF authorization for OMVS RACF authorization for HMC LPAR actions Mainframe Enablers EMCSAFI security interface authorization Install EMC GDDR C-System started procedures Define GDDR data in virtual datasets Customize EMC z/os Console Monitor started procedures Customize member GDDRPROC Allocate the parameter backup dataset and your parameter wizard work dataset Customize CA-OPS/MVS for EMC GDDR Step 1: Include EMC GDDR libraries in OPSVIEW REXX exec Step 2: Merge CA-OPS/MVS user applications (optional) Step 3: Make EMC GDDR AOF rules available to CA-OPS/MVS Step 4: Set REXX and TSO transaction limits in the CA-OPS/MVS OPSMAIN parameters Step 5: Enable SMF Support in CA-OPS/MVS Step 6: Change CA-OPS/MVS access rules Step 7: Customize EMC GDDR user exit 7 (optional) Step 8: Update CA-OPS/MVS started procedure OPSOSF Step 9: Update the UNIX system service directory Step 10: Define the EMC GDDR monitoring started tasks to CA-OPS/MVS SSM Step 11: Update CA-OPS/MVS CCI parameters Step 12: Apply CA-OPSMVS Usermod Modify CA-OPS/MVS to use the GDDRMSG table Configure EMC GDDR Step 1: Customize the GDDR invocation REXX exec Step 2: Collect and review input parameter information Step 3: Update your personal GDDR ISPF profile Step 4: Define initial parameters Step 5: Configure the EMC GDDR HMC interface Step 6: Modify EMC GDDR user exits (optional) EMC GDDR Product Guide

5 Contents Chapter 4 Using EMC GDDR Online Facilities Primary Options Menu Option P: Profile Update Personal GDDR ISPF Profile Option M: Maintenance GDDR Setup and Maintenance Option P: Manage GDDR Parameters Option D: Message, Debug and Trace Options Option Q: Manage GDDR Internal Command Queue Option H: Perform HMC Discovery Option R: Refresh GDDR Message Table Option S: Manage GDDR System Variables Option G: GDDR Config View GDDR Configuration Option R: Roles Manage Site Roles Option C: Checkup Perform Pre-script Checkup Health Check monitoring Health Check monitoring exception notification Additional pre-script environment checks Option S: Scripts Run GDDR Scripts Option T: Timing View GDDR Script Statistics Option A: Actions Perform GDDR Actions Option H: Perform HMC Discovery Option L: Perform HMC LPAR Actions Option CBU: Perform HMC CBU actions Option S: Manage Couple Datasets Option O: OPS Access CA-OPS/MVS Using OPSVIEW facilities for EMC GDDR administration Ensuring MSF connections between C-Systems Chapter 5 Performing Script Operations Running scripts Call overrides Rerunning a script WTOR messages Planned script operations Abandon Site DC1 (site swap) Restart production at DC2 after site swap Perform test IPL from BCVs at DC Perform test IPL from R2s at DC Unplanned script operations Recover after loss of DC1 (LDR) Resume replication after loss of DC Resumption operations Resume after test IPL from BCVs at DC Resume after test IPL from R2s at DC Resume replication after link failure Chapter 6 Handling Unplanned Events Introduction Consistency group trips Local disaster operations Confirm loss of DC System failure operations EMC GDDR C-System failure Production system failure EMC GDDR Master Function transfer EMC GDDR Product Guide 5

6 Contents Chapter 7 Performing Maintenance Procedures Setting up a new EMC GDDR C-System Renaming an existing EMC GDDR C-System Adding a new production system or sysplex to EMC GDDR Changing the Consistency Group name Adding new RDF groups to EMC GDDR Adding new devices to EMC GDDR Removing an RDF group from EMC GDDR control Removing devices from EMC GDDR control Removing a system or a sysplex from EMC GDDR Special cases Non-LOGR couple datasets Chapter 8 Chapter 9 Appendix A Using the Audit Monitoring Facility Overview Global variable changes Messages State changes CA-OPS/MVS environment changes Implementation tasks Message logging implementation GDDR SAY and state monitoring implementation Global variable monitoring implementation EMC GDDR audit monitoring SMF extract and report JCL SMF audit data flow Sample output CA-OPS/MVS environment monitoring AOFEVENT segment SMFRULEDISABLE segment OSFTERM segment Summary section Troubleshooting Detecting and resolving problems Using the GDDRXCMD batch utility To print the current queue To clear the current queue EMC GDDR User Exits User exit programming considerations Sample procedure Built-in routines available to exits Exit specifications GDDRUX GDDRUX GDDRUX GDDRUX GDDRUX GDDRUX GDDRUX EMC GDDR Product Guide

7 Contents Appendix B Appendix C EMC GDDR z/os Console Monitor Introduction z/os Console Monitor GDDRPBAL z/os operator console commands BAL command processor BALC BAL CSC ports CSC RTokens Using GDDRMAIN and the Heartbeat Monitor Starting/Stopping GDDRMAIN Stop command (P) Modify command (F) GDDRMAIN EXEC parameters GDDRGVX utility DSPLIST DIVLIST DSPSAVE RELOAD EMC GDDR system variable integrity and access Index lock Update lock Starting/Stopping the Heartbeat Monitor Glossary EMC GDDR Product Guide 7

8 Contents 8 EMC GDDR Product Guide

9 Figures Title Page 1 SRDF/S with ConGroup environment EMC GDDR Complex EMC JCL customization utility EMC JCL customization utility completed panel Primary Options Menu Change GDDR ISPF Profile Variable Values panel Setup and Maintenance Menu Parameter Management Options Menu Select Parameter input dataset for parameter review Reviewer's version of the Parameter Management Options Menu Manage GDDR Parameter Backups panel Select Dataset for GDDR Parameter Backup Select Parameter Input Dataset panel Prepare Work Dataset for Parameter Load confirmation panel Prepare Work Dataset status panel Parameter Management Options Menu with parameter load input selection Define Configuration Basics panel Define Configuration Features panel Define GDDR Configuration Features panel Define C-Systems panel Define GDDR Datasets panel Define Site Roles and Groups panel Define Storage Objects panel Define SRDF Device Ranges panel Define SRDF/S GNS Groups Define TimeFinder Device Ranges panel Define SDDF Clean Utility Gatekeepers panel Define Host Objects panel Define Managed Systems panel Define Managed LPARs panel Define Managed CPCs panel Define IPL Parameters panel Define Managed HMCs panel Define HMC Community Names panel Define Managed Couple Datasets panel 1 of Define Managed Couple Datasets panel 2 of Define Managed CF Structures panel 1 of Define Managed CF Structures panel 2 of Define External Workloads panel Define EMC MF Enablers STCs panel EMC GDDR Product Guide 9

10 Figures Title Page 41 Specify GDDR Options panel Specify Default Script Call Overrides panel Script Sysplex Options panel Script AUTOCBU Options panel Script JCL Parameters panel Utility Parameters panel Messaging and SMF Options panel Tuning Values panel Validate GDDR Parameter Set panel Activate GDDR Parameter Set panel Set Output Message Levels by Program panel Add Program to MsgLevel/Debug/Trace List panel Manage GDDR Internal Command Queue panel HMC object discovery panel HMC Discovery Results panel Message table refresh indicator Manage GDDR System Variables panel Manage GDDR System Variables panel Manage GDDR System Variables panel Manage GDDR System Variables - Detail panel Manage GDDR System Variables - Detail panel View GDDR Configuration panel Manage Site Roles panel Master C-System Transfer panel Perform Health Check panel Consistency group status display output Select Script to Run panel Script Selection for Status panel GDDR Actions Menu HMC object discovery panel HMC Discovery Results panel Perform HMC LPAR Actions panel Perform CBU Actions panel Manage Couple Datasets panel CA-OPS/MVS OPSVIEW Primary Options panel Specify Parameters for Initial Script Run panel Specify Call Overrides panel (screen 1 of 2) Specify Call Overrides panel (screen 2 of 2) Confirm Job Submission panel EMC GDDR Product Guide

11 Tables Title Page 1 Mainframe environment requirements Minimum hardware requirements Installation tasks RIMLIB library contents RACF functional groups RACF permissions RACF permissions, OPERCMDS class Summary of GDDR ISPF RACF permissions Defining Managed Couple Datasets Monitoring events EMC GDDR call overrides Possible lock states EMC GDDR Product Guide 11

12 Tables 12 EMC GDDR Product Guide

13 Preface As part of an effort to improve and enhance the performance and capabilities of its product lines, EMC periodically releases revisions of its hardware and software. Therefore, some functions described in this document may not be supported by all versions of the software or hardware currently in use. For the most up-to-date information on product features, refer to your product release notes. If a product does not function properly or does not function as described in this document, please contact your EMC representative. Note: This document was accurate as of the time of publication. However, as information is added, new versions of this document may be released to the EMC Powerlink website. Check the Powerlink website to ensure that you are using the latest version of this document. Audience Related documentation This document is part of the EMC Geographically Dispersed Disaster Restart (EMC GDDR) documentation set, and is intended for use by EMC GDDR systems administrators and computer operators. This document describes the basic concepts of EMC Geographically Dispersed Disaster Restart (EMC GDDR), how to install it, and how to implement its major features and facilities. Readers of this document are expected to be familiar with the following topics: IBM z/os operating environments IBM parallel sysplex Unicenter CA-OPS/MVS EMC software products: SRDF, ResourcePak Base, Consistency Group, and AutoSwap Related documents include: EMC GDDR Release Notes EMC GDDR Message and Code Guide EMC Mainframe Enablers Installation and Customization Guide EMC ResourcePak Base for z/os Product Guide EMC Symmetrix SRDF Host Component for z/os Product Guide EMC Symmetrix Remote Data Facility Product Guide EMC AutoSwap Product Guide EMC Consistency Group for z/os Product Guide EMC GDDR Product Guide 13

14 Preface EMC TimeFinder/Mirror for z/os Product Guide EMC TimeFinder/Clone Mainframe SNAP Facility Product Guide EMC REXX Interface Programmer s Reference Guide Unicenter CA-OPS/MVS for EMC Geographically Dispersed Disaster Restart Documentation CD Conventions used in this document EMC uses the following conventions for special notices. Note: A note presents information that is important, but not hazard-related.! CAUTION A caution contains information essential to avoid data loss or damage to the system or equipment. The caution may apply to hardware or software.! IMPORTANT An important notice contains information essential to operation of the software. The important notice applies only to software. EMC GDDR This document uses the acronym EMC GDDR in place of full product name, EMC Geographically Dispersed Disaster Restart. Typographical conventions EMC uses the following type style conventions in this document: Normal Bold: Italic: Courier: Courier bold: Used in running (nonprocedural) text for: Names of interface elements (such as names of windows, dialog boxes, buttons, fields, and menus) Names of resources, attributes, pools, Boolean expressions, buttons, DQL statements, keywords, clauses, environment variables, filenames, functions, utilities URLs, pathnames, filenames, directory names, computer names, links, groups, service keys, file systems, notifications Used in running (nonprocedural) text for: Names of commands, daemons, options, programs, processes, services, applications, utilities, kernels, notifications, system calls, man pages Used in procedures for: Names of interface elements (such as names of windows, dialog boxes, buttons, fields, and menus) What user specifically selects, clicks, presses, or types Used in all text (including procedures) for: Full titles of publications referenced in text Emphasis (for example a new term) Variables Used for: System output, such as an error message or script URLs, complete paths, filenames, prompts, and syntax when shown outside of running text Used for: Specific user input (such as commands) 14 EMC GDDR Product Guide

15 Preface Courier italic: Used in procedures for: Variables on command line User input variables < > Angle brackets enclose parameter or variable values supplied by the user [ ] Square brackets enclose optional values Vertical bar indicates alternate selections - the bar means or { } Braces indicate content that you must specify (that is, x or y or z)... Ellipses indicate nonessential information omitted from the example Where to get help EMC support, product, and licensing information can be obtained as follows. Product information- For documentation, release notes, software updates, or for information about EMC products, licensing, and service, go to the EMC Powerlink website (registration required) at: Technical support- For technical support, go to EMC Customer Service on Powerlink. To open a service request through Powerlink, you must have a valid support agreement. Please contact your EMC sales representative for details about obtaining a valid support agreement or to answer any questions about your account. Your comments Your suggestions will help us continue to improve the accuracy, organization, and overall quality of the user publications. Please send your opinion of this document to: techpub_comments@emc.com If you have issues, comments, or questions about specific information or procedures, please include the title and, if available, the part number, the revision (for example, A01), the page numbers, and any other details that will help us locate the subject you are addressing. EMC GDDR Product Guide 15

16 Preface 16 EMC GDDR Product Guide

17 Invisible Body Tag 1 Product Overview This chapter presents an overview of EMC GDDR and its capabilities. What is EMC GDDR? Major features Supported business continuity configurations The EMC GDDR Complex EMC GDDR fundamentals EMC GDDR components EMC GDDR supported scripts Parameter wizard: Telling EMC GDDR what to manage Product Overview 17

18 Product Overview What is EMC GDDR? EMC Geographically Dispersed Disaster Restart (EMC GDDR) is a mainframe software product that automates business recovery following both planned outages and disaster situations, including the total loss of a data center. EMC GDDR achieves this goal by providing monitoring, automation and quality controls to the functionality of many EMC and third-party hardware and software products required for business restart. Because EMC GDDR restarts production systems following disasters, it does not reside on the same servers that it is seeking to protect. EMC GDDR resides on separate logical partitions (LPARs) from the host servers that run your application workloads. You install EMC GDDR on a control LPAR at each site. Each EMC GDDR node is aware of the other EMC GDDR nodes through network connections between each site. This awareness allows EMC GDDR to: Detect disasters Identify survivors To achieve the task of business restart, EMC GDDR automation extends well beyond the disk level and into the host operating system level. It is at this level that sufficient controls and access to third party software and hardware products exist to enable EMC to provide automated recovery capabilities. EMC GDDR s main activities include: Managing planned site swaps (workload and DASD) between the primary and secondary sites. Active monitoring of the managed environment and responding to exception conditions. 18 EMC GDDR Product Guide

19 Product Overview Major features EMC GDDR successfully undertakes these activities by exploiting the following major features: Situational awareness Survivor recognition Situational awareness EMC GDDR can distinguish normal operational disruptions from disasters and respond accordingly. For example, EMC GDDR is able to distinguish between network outages (SRDF link drop) and real disasters. This awareness is achieved by periodic exchange of dual-direction heartbeats between the EMC GDDR LPARs. Survivor recognition EMC GDDR can determine which sites and systems have survived a disaster. Unlike the foundation technologies (such as TimeFinder /Mirror or TimeFinder/Clone Mainframe SNAP Facility), EMC GDDR has built-in intelligence to monitor other EMC GDDR systems. EMC GDDR constantly checks for disaster situations and constantly ensures that other GDDR systems are healthy. This checking allows EMC GDDR to recognize, and act on, potential disaster situations, even if only one EMC GDDR system survives. Split brain problems associated with cluster technologies are avoided through operator prompts. Upon the initial recognition stage, EMC GDDR issues messages to the operator console seeking confirmation of the event and, further, confirmation of restart actions required. Restart coordination If a primary site disaster occurs, the EMC GDDR Master C-System located at the secondary site will execute the recovery. The EMC GDDR Master C-System operates in a Master Owner/ No-Owner role for other EMC GDDR control LPARs. Changes to EMC GDDR configuration information can only be made on the EMC GDDR Master Control System (C-System). EMC GDDR propagates these changes to the subordinate EMC GDDR systems using the CA-OPS/MVS MSF (multi-system facility) inter-system communications feature. Restart procedures following disasters are coordinated from the EMC GDDR Master C-System. EMC GDDR coordinates and executes predetermined processes to: Restart the enterprise at the desired surviving site in the event of a disaster Automate a planned site swap Major features 19

20 Product Overview Additional capabilities As part of the planned site swap process and as part of the recovery process after an unplanned site swap, EMC GDDR can optionally perform the following tasks: Trigger stopping or starting distributed workloads Trigger stopping or starting z/os workloads Types of environment EMC GDDR can manage environments that are comprised of the following elements: Multiple z/os systems Multiple sysplexes Multiple Symmetrix controllers Intermix of CKD and FBA/FBAM DASD and BCVs 20 EMC GDDR Product Guide

21 Product Overview Supported business continuity configurations EMC GDDR is available in the following configurations: SRDF/S with ConGroup The 2-site SRDF/S with ConGroup configuration provides disaster restart capabilities at site DC2. SRDF/S with AutoSwap The 2-site SRDF/S with AutoSwap configuration provides for near-continuous availability through device failover between DC1 and DC2. SRDF/A The 2-site SRDF/A configuration provides disaster restart capabilities at site DC3. SRDF/Star The 3-site SRDF/Star configuration provides disaster restart capabilities at either DC2 or DC3. Concurrent and Cascaded SRDF support further minimize the DC3 recovery time objective. SRDF/Star with AutoSwap The 3-site SRDF/Star with AutoSwap configuration provides for near-continuous availability through device failover between DC1 and DC2 as well as disaster restart capabilities at DC3. Concurrent and Cascaded SRDF support further minimize the DC3 recovery time objective. EMC GDDR has been designed to be customized to operate in any of these configurations. EMC GDDR functionality is controlled by a parameter library. During EMC GDDR implementation, this parameter library is customized to reflect: The prerequisite software stack The desired data center topology (two-site versus three-site, synchronous or asynchronous). The data centers are referred to as sites DC1 and DC2. Usage of these data center sites is described in Sites DC1 and DC2 on page 23. EMC GDDR is able to control multiple sysplexes from a single control LPAR. This document discusses the EMC GDDR SRDF/S with ConGroup configuration. Documentation for other EMC GDDR configurations is available on the EMC Powerlink website at: SRDF/S with ConGroup configuration The 2-site SRDF/S with ConGroup configuration provides disaster restart capabilities at site DC2. Figure 1 on page 22 illustrates EMC GDDR operation in the SRDF/S with Consistency Group environment. Supported business continuity configurations 21

22 Product Overview DC1 DC2 EMC GDDR EMC GDDR ConGroup ConGroup R1 SRDF/S R2 GDDR heartbeat communication Active Escon/Ficon channels Standby Escon/Ficon channels Active SRDF links SYM Figure 1 SRDF/S with ConGroup environment As Figure 1 shows, the relationship between the DC1 and DC2 sites is maintained through SRDF/S replication of primary disk images at DC1 to DC2. Both open systems (FBA) and mainframe (CKD) disk images can be replicated. Figure 1 shows the two EMC GDDR C-Systems with their heartbeat communication paths, separate from the production disk and computer facilities. Each of the DC1 and DC2 production z/os LPARs has EMC Consistency Group (ConGroup) software installed. EMC GDDR does not have a requirement to freeze I/O to obtain a point of consistency. SRDF/S and ConGroup provide the mechanism. At the point that EMC GDDR receives notification of an unplanned or failure event, a point of consistency is already achieved through these foundation technologies. In this environment, EMC GDDR can do the following: Manage planned site swaps Restart processing at the secondary site following unplanned primary site events Perform standard operational tasks: IPL, system reset, activate, deactivate Trigger stop/start of business workloads Actively monitor for unplanned/failure events Sites Systems Loss of SRDF/S ConGroup trip Inter-site communication failure 22 EMC GDDR Product Guide

23 Product Overview The EMC GDDR Complex An EMC GDDR Complex consists of EMC GDDR Control Systems (C-Systems), the z/os and open systems hosts, and EMC Symmetrix storage systems which support an organization's mission-critical workload. Each GDDR Complex can manage one consistency group. A consistency group is a named group of source (R1) volumes managed by the EMC Consistency Group (ConGroup) application as a unit. The volumes can be on multiple Symmetrix units. Figure 2 depicts a logical view of a typical EMC GDDR Complex. Primary site Secondary site DC1 Sysplex #n Sysplex #n DC2 C1 Sysplex #2 Sysplex #1 Sysplex #2 Sysplex #1 C2 Master C Local BCV R1 SRDF/S BCV R2 Local SYM Figure 2 EMC GDDR Complex The following are brief descriptions of the components that comprise a GDDR Complex: BCVs BCVs (Business Continuance Volumes) can be supported at each of the sites. They may be established at the DC2 site and split at the DC1 site. Note: The EMC TimeFinder/Mirror for z/os Product Guide provides more information about BCVs and about the establish and split operations. C1 and C2 C1 and C2 are the EMC GDDR Control LPARs (or C-Systems) at each of the sites. Primary site The primary site is the site where the production workload is located. Primary DASD site The primary DASD (direct access storage device) site is the site where the source (R1) DASD is located. The primary DASD site is the same as the primary site. Secondary site The secondary site is the site where the contingency or stand by systems are located. Secondary DASD site The secondary DASD site is the site where the target (R2) DASD is located. The secondary DASD site is the same as the secondary site. Sites DC1 and DC2 Sites DC1 and DC2 are the primary and secondary data centers of critical production applications and data. DC1 is the primary site, with SRDF/S data replication to the secondary site, DC2. These sites are considered fully equivalent for strategic production applications, connected with highly redundant direct network links. At all times, all production data is replicated synchronously between the two sites. Sysplexes A system processing complex. A single GDDR complex supports multiple sysplexes. The EMC GDDR Complex 23

24 Product Overview EMC GDDR fundamentals This section discusses: Control systems Workload location Managed workloads EMC GDDR processes Control systems The EMC GDDR control systems are more commonly referred to as EMC GDDR C-Systems. One EMC GDDR C-System is located at each site (DC1 and DC2). C-Systems can be configured in monoplex mode or as standalone systems. Each EMC GDDR C-System runs in monoplex mode from local DASD. EMC suggests that you locate the C-System DASD on separate controllers from the production DASD. Because the EMC software applications run from local C-System volumes, this separation ensures that the C-Systems are not affected by any events that may impact the availability of the managed systems. The main functions of a EMC GDDR C-System are to: Control the recovery after an outage Control a planned site swap EMC GDDR C-Systems do not run any production workload. One of the C-Systems is the Master C-System. During normal operations, the Master C-System is the central control point for all EMC GDDR activities. The Master C-System is located at the primary DASD site. In the event of the loss of the primary DASD site, EMC GDDR transfers the Master C-System to the secondary site, for completion of the restart coordination. Some EMC GDDR functions can only be carried out by the Master C-System, for example: Running planned processes Updating EMC GDDR parameters All EMC GDDR C-Systems are potential candidates to takeover as the Master C-System. Workload location Production system In an EMC GDDR Complex, the business or production workload runs at a single site; that is, one side of the sysplex. This is the same location as the primary DASD site. A production system is a system that normally runs the site s production workload and updates the primary DASD. Production systems are located at the same location as the primary DASD. 24 EMC GDDR Product Guide

25 Product Overview Contingency or standby system A contingency or standby system is a system that normally provides a hot backup to a production system. A contingency system: Is in the same sysplex as its production system partner Is IPLed, but runs no business workload Contingency or standby systems are typically located at the same location as the secondary DASD. Note: Subsequent references to the term production system in this document refer to both production systems and contingency systems. Managed systems Any production or contingency/standby system defined to EMC GDDR is known as an EMC GDDR managed system. Managed workloads EMC GDDR can trigger the stop and restart of production workloads on: z/os systems Distributed systems External workloads External workloads run in mainframe systems which do not have their DASD in the managed Symmetrix units. EMC GDDR can coordinate Stop and Start of the workload on these "non-managed" mainframe systems with the workload Stop and Start for managed systems. Excluded systems EMC GDDR can be configured to exclude certain systems from workload management, although these systems have their DASD in the managed Symmetrix units. HMC-only systems EMC GDDR can be configured to limit IPL and CBU actions for certain systems to the online interface. No other actions or automation are performed for these systems. EMC GDDR fundamentals 25

26 Product Overview EMC GDDR processes An EMC GDDR process is a predetermined sequence of function calls. Generally one function call corresponds to one action. An EMC GDDR process is started by calling EMC GDDR provided routines, either from a batch job or as a result of specific messages being issued. There are two types of EMC GDDR processes: Planned process An EMC GDDR planned process is initiated through the EMC GDDR interface to perform a planned task. Unplanned process/takeover process The EMC GDDR unplanned process or takeover process can only be initiated following an error that results in a possible takeover situation. Takeover processes are initiated as a result of certain messages being issued or specific events occurring. The messages or events that trigger an unplanned or takeover process can originate on any system, either a C-System or a production system. They only take place on the current Master C-System. They are invoked following operator confirmation of any of the following types of failure or loss: Sites DASD Systems Loss of SRDF link Loss of host channels Process restart The return codes from the function calls that make up an EMC GDDR process are saved in CA-OPS/MVS global variables. For functions that issue EMC SRDF Host Component commands, the return code of the commands are also saved. If multiple commands are issued from one function, the return codes from each command are saved in CA-OPS/MVS global variables. After the cause of the original failure has been identified and resolved, the EMC GDDR process can be rerun. EMC GDDR uses the saved return codes to establish the point of restart; that is, the point of the previous failure. This ensures that no modifications to the supplied EMC GDDR process jobs are required in order to rerun after a failure. 26 EMC GDDR Product Guide

27 Product Overview EMC GDDR components EMC GDDR is comprised of a number of components: Parameters User interface Events Monitors Message rules Parameters EMC GDDR parameters define the environment and configuration that it manages. The parameters can modify the sequence of function calls that is an EMC GDDR process. User interface The EMC GDDR user interface is an ISPF application. It is available only on EMC GDDR C-Systems. Events An EMC GDDR event is a change in state of a component part of the environment that EMC GDDR is actively monitoring. Examples of EMC GDDR events include: CGT ConGroup trip has occurred/state change CGD ConGroup group is disabled/state change MHB missing C-System heartbeat The event can have a state of either TRUE or FALSE. If the event has a state of TRUE, it has occurred or is currently occurring. If the event has a state of FALSE, it is no longer occurring. An event that is TRUE is considered an exception. EMC GDDR events are used by the GDDR event monitor and GDDR processes to determine environment state. A change in state can then: Request operator confirmation of the event and present the relevant actions Prevent a planned process from running Monitors There are three monitors on each EMC GDDR C-System: The EMC GDDR event monitor The EMC GDDR heartbeat monitor The optional EMC GDDR audit monitor EMC GDDR components 27

28 Product Overview Event monitor Heartbeat monitor Audit monitor The EMC GDDR event monitor runs on each C-System and is used to analyze event state changes in which EMC GDDR is interested. On detecting the occurrence of selected events, the event monitor determines what action to take and prompts operators with the appropriate choices. For example: EMC GDDR detects that a production system has failed and prompts the operators with the following options: IPL:ssss EMC GDDR to restart ssss at current location DCn. SYSSITEn EMC GDDR to start business applications at site DCn. SYSRESET EMC GDDR to system reset ssss at site DCn only. Ignore EMC GDDR to do nothing. The EMC GDDR heartbeat monitor aids the event monitor in determining the status of the EMC GDDR managed environment. The lack of a heartbeat from a particular C-System is used to determine the state of a C-System and the site. The EMC GDDR audit monitor captures and externalizes data that EMC GDDR automation uses for business continuance processing (BCP) decision criteria and operations. This data includes dates and times of parameter load and backup jobs, global variable updates, override usage, and state changes relevant to EMC GDDR managed systems and storage. Message rules EMC GDDR is supplied with AOF (Automated Operations Facility) message rules to be installed only on the GDDR C-Systems. These rules are disabled as shipped by EMC to allow for operations training and familiarity before the message handling is automated. EMC GDDR does not operate properly if all these message rules are not enabled on all C-systems. Note: The EMC GDDR Message and Code Guide contains more information about EMC GDDR messages. The AOF message rules have two primary functions: To detect events that EMC GDDR is interested in and set the appropriate EMC GDDR event TRUE or FALSE. To detect events that EMC GDDR processes have to wait for (WTOR), and reply as to the success or failure of the waited for event. This will determine if an EMC GDDR process proceeds or terminates. EMC GDDR uses the EMC ResourcePak Base Cross System Communication (CSC) facility to route message traffic to production systems. You or EMC service personnel can use the arrival of a message at the target production system to trigger an automation rule (for example in IBM Tivoli NetView or BMC Control-M). Such rules can be used to start or shut down workloads on the appropriate systems, even though they do not run CA-OPS/MVS. DYNAPI interface The EMC GDDR interface to EMC DYNAPI allows EMC GDDR to run dynamic SRDF commands in parallel. 28 EMC GDDR Product Guide

29 Product Overview EMC GDDR supported scripts EMC GDDR provides a number of scripts that allow you to perform any of the following actions: Planned event management Unplanned event management Resumption after planned or unplanned outages Planned event management Operations personnel can handle planned event management scenarios by running any of the following scripts. Note: DC1 and DC2 represent the current primary DASD site or current secondary DASD site. When these representations are shown in italic type in script titles, this indicates the values are interchangeable. The descriptions assume that DC1 is the Primary DASD site and Primary site at the beginning of the script. Abandon Site DC1 (site swap) Stops the business workload at the primary DASD site Waits for the stop of all business applications Resets clear all production systems managed by EMC GDDR Restart production at DC2 after site swap This script performs the following actions after the loss of the primary site: Attempts reset clear of all systems at the primary DASD site Activates CBU (if required) Activates all needed LPARs, including CFs at the secondary DASD site Creates a consistency point at the secondary DASD site Prepares the SRDF environment IPLs all needed production systems Perform test IPL from BCVs at DC2 Splits BCVs, makes them R/W Activates test LPARs using BCV volumes Starts test business workload, if applicable Perform test IPL from R2s at DC2 Confirms that SRDF/S has been stopped normally via a Congroup trip Activates LPARs using R2 volumes Starts test business workload, if applicable EMC GDDR supported scripts 29

30 Product Overview Unplanned event management Operations personnel can manage unplanned events in one of two ways: The EMC GDDR Event Monitor prompts the operator for management confirmation of trigger events which indicate a site or DASD outage. The operator replies to the prompt in the affirmative and the GDDR recovery script is started. The operator may start the appropriate unplanned script and respond to prompts. The script initiates and validates that the state of the current host and storage environments matches the script prerequisites before proceeding. Recover after loss of DC1 (LDR) Confirms that a ConGroup trip occurred Confirms that SRDF links failed Confirms that a local disaster (LDR) event occurred Shuts down applications at the primary site, if applicable Splits BCVs and conditions R2s at secondary site for restart Activates contingency systems Restarts applications Resume replication after loss of DC1 Confirms SRDF/S links are down Splits BCVs at the secondary site, if applicable Issues ConGroup cleanup and SRDF/S restart commands Reestablishes BCVs at the secondary site Resumption after planned or unplanned outages Operations personnel can resume operations after planned or unplanned outages by running any of the following scripts. Resume after test IPL from BCVs at DC2 Stops test business workload, if applicable Reset clears test LPARs Reestablishes the BCVs Resume after test IPL from R2s at DC2 Stops test business workload, if applicable Reset clears test LPARs Restarts SRDF/S to DC2 Resume replication after link failure Confirms SRDF/S links are down Stops ConGroup on all systems Splits BCVs at the secondary site, if applicable Issues ConGroup cleanup and restart commands Reestablishes BCVs at the secondary site 30 EMC GDDR Product Guide

31 Product Overview Parameter wizard: Telling EMC GDDR what to manage The environment that EMC GDDR manages is described to EMC GDDR through a collection of common variables. The EMC GDDR parameter management wizard groups these variables in a series of ISPF panels, each backed by a member in a PDS. The variable groups include the following: Configuration-defining variables These variables define the type of managed configuration, the C-systems, the initial role for each site, the consistency group names and the MSC group names. Storage object variables These variables define the actual SRDF and TimeFinder devices, SRDF groups, GNS groups, and gatekeeper devices that form the configuration that EMC GDDR will manage. Host Object variables These variables define the managed, external and HMC-only systems, and their LPARs, IPL-parameters and CPCs. Host object variables also define HMC-consoles, Sysplex objects and EMC Mainframe Enablers started tasks. GDDR option variables These variables define user-selectable values for a variety of actions taken in the course of GDDR automation sequences. GDDR option variables also define site defaults for JCL and utilities used by GDDR, messaging and audit related options, and tuning values. Parameter wizard: Telling EMC GDDR what to manage 31

32 Product Overview 32 EMC GDDR Product Guide

33 Invisible Body Tag 2 Installing EMC GDDR This chapter describes the EMC GDDR installation procedure. Preinstallation tasks Installation procedure Post-installation tasks Installing EMC GDDR 33

34 Installing EMC GDDR Preinstallation tasks Before you begin installing EMC GDDR, review the hardware and software requirements listed below.! CAUTION EMC GDDR is only to be installed on designated EMC GDDR Control Systems (C-Systems). Mainframe environment requirements The basic infrastructure must support SRDF/S with Congroup. In addition to this, EMC GDDR has the following specific infrastructure requirements: There must be network connectivity between all C-Systems. An HMC (Hardware Management Console) must be available at each site that can be accessed from each C-System (access to these HMCs can be protected by means of a private VLAN). EMC GDDR has the mainframe environment requirements listed in Table 1. Before you install EMC GDDR, make sure your environment meets these requirements. Table 1 Mainframe environment requirements Item Processor hardware configuration DASD hardware configuration Software Requirements Any system that supports current IBM mainframe operating systems Any supported Symmetrix DASD model at an Enginuity microcode level specified in the EMC GDDR Release Notes Any currently supported IBM operating system Minimum software requirements The minimum software prerequisites needed to run EMC GDDR 3.1 are as follows: z/os IBM Hardware Management Console (HMC) API CA-OPS/MVS SRDF/Host Component ResourcePak Base Consistency Group Note: The EMC GDDR Release Notes provide information regarding supported software release levels for the above items. You can find installation procedures for the EMC software products in the EMC Mainframe Enablers Installation and Customization Guide. 34 EMC GDDR Product Guide

35 Installing EMC GDDR Additional configuration requirements SRDF/S with ConGroup Please refer to the EMC SRDF Host Component for z/os Product Guide for information on configuring an SRDF/S environment. Minimum hardware requirements Table 2 describes the recommended minimum processor and I/O configuration for an EMC GDDR C-System. Table 2 Minimum hardware requirements Item Logical processors MSU Storage Requirements 1 (2 are recommended) 15 on a IBM (or equivalent) 512 MB Logical paths to own local DASD devices 4 Logical paths to managed DASD devices 4 DASD support EMC GDDR supports and can manage the following combinations of DASD in a single Enterprise Consistency Group: Single EMC Symmetrix controllers configured with any of the following: All CKD devices All FBA and FBA-META devices Any combination of CKD, FBA and FBA-META devices Multiple EMC Symmetrix controllers configured with any of the following: All CKD devices All FBA and FBA-META devices Any combination of CKD, FBA and FBA-META devices Management and monitoring of both CKD and FBA/FBA-META devices is performed from the z/os platform where the EMC GDDR application resides. From the EMC GDDR point of view, CKD and FBA/FBA-META Symmetrix devices are the same; that is, each is treated no differently than the other. They are all command targets of SRDF Host Component configuration commands using local, remote or GNS syntax. EMC GDDR requires that if even only one device in an RDF group is defined to GDDR, then all devices in that group must be defined to GDDR. Most GDDR actions are directed at the RDF group level (although in some cases, GDDR will act on device ranges if that is appropriate). EMC GDDR has no limitations on the number of EMC Symmetrix controllers/devices that can be managed. Any limitations are subject to restrictions in EMC hardware and software. Preinstallation tasks 35

36 Installing EMC GDDR Installation procedure This section describes how to install EMC GDDR. The EMC GDDR installation kit is provided in two forms: As an electronic download from Powerlink As a CD! CAUTION Keep in mind that EMC GDDR is only to be installed on designated EMC GDDR Control Systems (C-Systems). Before you begin EMC GDDR is a user application under CA-OPS/MVS; therefore, CA-OPS/MVS and its prerequisite CA-Common Services must be installed before you can start the EMC GDDR installation process. The procedure for the EMC GDDR installation is as follows for each EMC GDDR C-System: Table 3 Installation tasks Task Reference 1. Review pre-installation information Preinstallation tasks on page 34 and Gather EMC GDDR installation information on page Install CA-Common Services Unicenter CA-OPS/MVS Event Management and Automation Getting Started a 3. Install CA/OPS/MVS Unicenter CA-OPS/MVS Event Management and Automation Getting Started 4. Install EMC GDDR Install EMC GDDR on page Customize CA-OPS/MVS Customize CA-OPS/MVS for EMC GDDR on page 64 b a. This document is available on the Unicenter CA-OPS/MVS for EMC Geographically Dispersed Disaster Restart Documentation CD. b. Refer to Using OPSVIEW facilities for EMC GDDR administration on page 159 for assistance with administration tasks that are performed using the CA-OPS/MVS - OPSVIEW Primary Options panel shown in Figure 75 on page EMC GDDR Product Guide

37 Installing EMC GDDR Gather EMC GDDR installation information Before beginning the EMC GDDR installation, you need to gather information in preparation for the installation. Identify or decide upon the following items: CLIST library and EDIT macro Determine a name for the edit macro created by the installation dialog. You also need to determine the name of a CLIST library where you can store the edit macro. Product dataset name prefix Choose the dataset prefix you will use to install EMC GDDR. Names for the product datasets consist of a final qualifier, such as LINKLIB, and a dataset prefix. For example, if you choose a dataset prefix of EMC.GDDRvrm, the LINKLIB dataset will be named EMC.GDDRvrm.LINKLIB. Ensure that you have RACF ALTER authority (or the equivalent from another security manager) for the datasets created with this dataset prefix. Note: Throughout this guide, datasets created using this dataset prefix are referred to as if they had been created with the suggested value.the actual fmid for your installation may be different. ResourcePak Base dataset name prefix Specify the dataset name prefix you used when you install ResourcePak Base. EMC recommends that you use EMC.fmid if it agrees with your site standards. SMP/E dataset name prefix Choose the name prefix for the SMP/E datasets into which you installed EMC GDDR. If you have installed another EMC product using SMP/E, you should install EMC GDDR into the same CSI. If you are installing an EMC SMP/E maintained product for the first time, EMC recommends using EMC.SMPE. SMP/E datasets volser Choose the disk volume onto which you will install the distribution libraries required by SMP/E. This may be the same volume you use for the product libraries. However, many customer sites prefer to keep SMP/E-related datasets on separate volumes from product libraries. An amount of space similar to that needed for the product libraries is required. Install-to-disk volser Determine the disk volume onto which you will install the target (that is, runtime) datasets. The space required is nominal. EMC suggests that you use EMC.fmid if it agrees with your site standards. Disk unit name Decide upon a disk unit name for the above volumes. For many users, SYSDA will suffice. However, use whatever generic or esoteric name your local standards require. Installation procedure 37

38 Installing EMC GDDR Install EMC GDDR The EMC GDDR kit consists of a PDS containing TSO TRANSMIT images of files needed to perform an SMP/E indirect-library installation. This PDS is packaged on CD or as an electronic download from EMC Powerlink. To install EMC GDDR on an EMC GDDR control system, take the following steps: 1. Load the TSO TRANSMIT file, GDDRvrm.XMITLIB, to the mainframe disk. 2. Run GDDRvrm.XMITLIB(#EXTRACT) to extract ds-prefix.rimlib and the SMP/E indirect libraries. 3. Customize the RIMLIB JCL. 4. Run the installation jobs. 5. Perform cleanup. 6. Apply maintenance updates. The following sections describes these steps in more detail. Load GDDRvrm.XMITFILE to disk 1. Take one of the following steps: If you are installing EMC GDDR from a CD, complete the following steps: a. Mount the CD on an open system host. b. Allocate a working directory on the open system for the installation. c. Copy the contents of the CD to the working directory. If you are installing EMC GDDR from an EMC Powerlink download, complete the following steps: a. Log in to a privileged account on an open systems host (root on UNIX or administrator on Windows). b. Allocate a working directory on the open system for the installation. c. Log on to: d. Navigate to Support>Software Downloads and Licensing> Downloads E-I>Geographically Dispersed Disaster Restart (GDDR). Note: If you are not able to access this location, you may not have registered your software or registered it incorrectly. Follow the prompts to register your software, correct your registration, or contact EMC in the event of a problem. e. Click the product version you want to download. The product version consists of a zip file that contains the installation kit and the installation instructions. f. Download the installation kit into the working directory. 2. If your current host is a Windows system, unzip the file in the working directory. If your current host is a UNIX system, unzip and untar the file into the working directory. 3. Locate GDDRvrm.XMITFILE. This file is in TSO TRANSMIT format and contains a flattened copy of GDDRvrm.XMITLIB, a PDS that holds other TRANSMIT images, the JCL to extract them, and necessary SMP/E installation files. 38 EMC GDDR Product Guide