Implementing EMC SRDF/A DSE at a Midwestern U.S. Financial Services Company

Transcription

1 Implementing EMC SRDF/A DSE at a Midwestern U.S. Financial Services Company Applied Technology Abstract This white paper discusses how EMC SRDF /A Delta Set Extension technology was applied at a medium-size, midwestern U.S. financial services company that was selected as a suitable candidate for the SRDF/A Delta Set Extension Early Customer Implementation program. September 2008

2 Copyright 2008 EMC Corporation. All rights reserved. EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. THE INFORMATION IN THIS PUBLICATION IS PROVIDED AS IS. EMC CORPORATION MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com All other trademarks used herein are the property of their respective owners. Part Number h5587 Applied Technology 2

3 Table of Contents Executive summary...4 Introduction...4 Audience... 5 Terminology... 5 Customer environment...5 Pre-implementation planning...6 SRDF/A DSE implementation and testing process...7 Testing results and observations...8 Conclusions and recommendations...11 References...11 Appendix: Sample JCL for managing SRDF/A DSE...12 Create the DSE pools Add devices to the DSE pool...13 Enable DSE Display the DSE pools Drain a device in the DSE pool...16 Disable DSE Applied Technology 3

4 Executive summary In late 2007, the Symmetrix Engineering organization selected a medium-size, midwestern U.S. financial services company as a suitable candidate for the SRDF /A Delta Set Extension Early Customer Implementation program. SRDF/A Delta Set Extension solves the problem of accurately predicting cache and bandwidth requirements for production workloads as customer environments grow and change over time. Previously, SRDF/A stored all of its cycle data in cache. Then, if conditions changed in an environment that caused Symmetrix cache to reach capacity, data would drop. With SRDF/A Delta Set Extension, sessions are prevented from dropping as the system offloads data that normally resides in cache to preconfigured DSE volumes. After reviewing the benefits of SRDF/A Delta Set Extension, which is also useful in mainframe environments where DFSMS is configured, the financial services company agreed to participate in the program. To set expectations and mitigate risk, several pre-implementation and test plan discussions were held during the weeks prior to the actual test. Five days after the test, the customer adopted SRDF/A Delta Set Extension as a permanent component of their storage replication infrastructure having assurances that SRDF/A stays active during an all SRDF links down situation for a specified timeframe so long as a balanced configuration is maintained. Introduction In 2007, the Symmetrix Engineering organization established an SRDF/A Delta Set Extension Early Customer Implementation program to facilitate the introduction of SRDF/A Delta Set Extension into the customer base. After evaluating many companies, a medium-size, midwestern U.S. financial services company was selected as a suitable candidate for the program. This white paper 1 describes the customer s environment into which SRDF/A Delta Set Extension was implemented and highlights salient planning and test steps. It includes test results, observations, and sample JCL similar to the JCL used to build the SRDF/A Delta Set Extension environment, and it concludes with best practice recommendations. SRDF/A Delta Set Extension became generally available with the initial release of Enginuity 5772 in March, Delta Set Extension is part of the SRDF/A Reserve Capacity enhancements that also include SRDF/A Transmit Idle. Specifically, Delta Set Extension is targeted at assisting customers with issues associated with: Having to accurately predict actual cache and bandwidth requirements for production workloads. Maintaining the ongoing balance across the SRDF/A environment over time, particularly in mainframe environments where DFSMS is in use. Dealing with real-world networks that do not always behave in an ideal manner, and where temporary interruptions in link availability or bandwidth degradation are common. SRDF/A Delta Set Extension helps customers avoid these issues by temporarily extending the available amount of cache space for SRDF/A by offloading some or all of the SRDF/A cycle data to preconfigured disk storage within the Symmetrix system. More importantly, Delta Set Extension will not help prevent unintended SRDF/A replication drops when the unintended drop is an unbalanced configuration due to factors such as: An undersized SRDF/A Symmetrix cache configuration Insufficient available bandwidth for the SRDF/A links A slower back end at the secondary (target or R2) side of the SRDF/A configuration than at the primary (source or R1) side An ever-increasing steady rise in the host write IO profile to SRDF/A devices Steadily decreased (or highly extended) network bandwidth reductions, or extended outages of some of the configured links By virtue of the complementary nature of SRDF/A Reserve Capacity features, using SRDF/A Delta Set Extension in combination with SRDF/A Transmit Idle provides the maximum level of resilience to unplanned replication outages. Transmit Idle allows SRDF/A to continue processing during periods of SRDF link disruption; Delta Set Extension allows data to be replicated to be paged out to preconfigured disks. Together the components provide users with a replication environment that is significantly more tolerant of network disruptions and bursts in host workload when compared to environments where the combined features are not in use. 1 With regards to the customer s motivation on whose Delta Set Extension implementation this white paper is based, the primary motivation was to enhance resilience of the replication environment to accommodate unexpected periods of network disruption. Applied Technology 4

5 Audience This white paper is intended for technology professionals, system architects, and IT administrators or technical staff interested in SRDF/A Delta Set Extension and its implementation, recommended best practices, and JCL examples. The reader should be familiar with EMC Symmetrix and IBM mainframe host environments. Terminology The following terms are used in this white paper: Term Customer Information Control System (CICS) Data Facility Storage Management Subsystem (DFSMS) Job Control Language (JCL) System Display and Search Facility SDSF Virtual storage access method (VSAM) Description A family of application servers and connectors that provides online transaction management and connectivity. CICS runs under z/os. System-managed storage that enables z/os to perform storage management tasks that previously were done manually. An IBM mainframe scripting language that instructs the system how to run a batch job or start a subsystem. A secure way to monitor, manage, and control a z/os sysplex. An IBM disk file storage access methodology. Customer environment As Figure 1 illustrates, the customer s primary site processing environment consists of a Symmetrix DMX storage processor that is attached to an IBM mainframe running z/os version 1.7. Data on the Symmetrix DMX is replicated to another DMX of roughly comparable configuration using SRDF over Fibre Channel through Ciena CN2000 extenders over an OC-3 optical link. The round-trip circuit distance, based on ping times measured in the environment, is approximately 3,400 km. About 2,500 host-accessible logical volumes are being replicated and used to house 14.1 TB of raw data from applications built on top of CICS, DB2, and VSAM. Figure 1. Customer environment identified for an SRDF/A DSE implementation Applied Technology 5

6 Pre-implementation planning During the weeks prior to the SRDF/A Delta Set Extension implementation, several discussions were held with the customer. These discussions primarily covered topics such as: Customer workload profile, including: o Current overall Symmetrix DMX and component utilizations o Network bandwidth availability and current utilization o Host capacity availability and the potential for workload bursts DMX configuration and capacity required to support SRDF/A Delta Set Extension, including: o Delta Set Extension Spillover device types and protection o DMX configuration and component throughput capabilities required to support the customer s workload Implementation of Delta Set Extension including: o Enginuity and host software code levels required o Pre-implementation day Enginuity, host software, and Symmetrix DMX hardware deployment dates, activities, and cross-site coordination strategies o Enabling, activating, deactivating, and disabling Delta Set Extension o Selection of appropriate testing and implementation dates o Implementation day activities and scheduling o Customer and Engineering expectations Note: The key feature of all these discussions was the identification of risks to the project and the development of strategies to eradicate or contain them. The physical devices to be used for Delta Set Extension (DSE) were installed into the customer s Symmetrix DMX units in late January, along with an appropriate Enginuity release ( ). It should be noted that in the case of this particular implementation, while the DSE devices were spread across all the DA pairs in the Symmetrix DMX units involved, the DSE pool devices were completely contained within the new physical disks that were added to the frames. From the perspective of best practices for the implementation of Delta Set Extension, this was considered suboptimal, as the ideal configuration has each physical disk in the configuration containing a small amount of the DSE pool, thus facilitating the spreading of SRDF/A data that is being paged out or into cache across the entire back end of the Symmetrix DMX, which maximizes Delta Set Extension bandwidth capabilities. However, since the customer had fully populated the existing devices with user data, there was effectively no way around having to constrain DSE activity to only the new devices other than a substantial data unload and reload project that would need to be completed prior to Delta Set Extension deployment. After appropriate analysis of the customer s workload and storage environment, it was recognized that this constraint would only add minimal risk to the possibility for success of the project, and the project moved ahead on schedule. The ability to enable, activate, deactivate and disable Delta Set Extension was validated using a small number of devices during a quiet period in the customer s workload a week prior to the first test date. Appendix: Sample JCL for managing SRDF/A DSE on page 12 contains sample JCL comparable to the code that was used to configure the DSE pool, add devices to the pool, enable Delta Set Extension, display DSE pool usage, drain devices in the DSE pool, and disable Delta Set Extension. Applied Technology 6

7 SRDF/A DSE implementation and testing process Table 1 lists the identified tasks and timetable in the overall implementation and testing plan. Table 1. Implementation and testing plan Task Responsibility Timetable Activate, collect, and analyze Symmetrix STP performance data used for sizing the Delta Set Extension requirements. Install additional physical disks to be used as the DSE pool. Upgrade to an appropriate Enginuity release level ( ). Add DSE SAV devices to the running DMX configurations at both source and target site via changes to impl.bin files. Upgrade host software (SRDF Host Component 5.5 and ResourcePak Base 5.7.0). Install accumulated maintenance. Provide definition and configuration of DSE pool. Add DSE devices to the pool. Validate the DSE and host software configuration by: Enabling DSE for a couple of devices. 2 Activating DSE for a few minutes via SRDF Host Component command. 3 Deactivating DSE. 4 Disabling the enabled DSE devices. 5 Activate STP to provide measurement data from test period. Enable DSE through batch job submission. 6 EMC Customer Engineer and Symmetrix Engineering resources assigned EMC Customer Engineers EMC Customer Engineers Customer EMC Customer Engineer Activity commenced six weeks prior to initial test date, and took approx. two weeks Two weeks prior to initial test date Two weeks prior to initial test date One week prior to initial test date 10May2008:09:00 (60 minutes before test start) Activate DSE using SRDF Host Component command. Customer 10May2008:10:00 Remove physical SRDF links on primary Symmetrix DMX EMC Customer Engineer 10May2008:10:12 system. (This simulates a network failure.) Monitor the storage environment. Customer, EMC Customer Engineer, and Symmetrix 10May2008:10:12 to 10May2008:11:15 Engineering resource Reinsert the physical SRDF links back into the primary Symmetrix DMX system. EMC Customer Engineer 10May2008:11:15 Monitor the storage environment until normal SRDF/A cycle switching resumed. Customer, EMC Customer Engineer, and Symmetrix Engineering resource 10May2008:11:15 to 10May2008:11:55 Deactivate DSE using SRDF Host Component command. 7 Customer 10May2008:11:55 Collect hosts syslog, internal Symmetrix log, STP and Customer and 10May2008:12:00 and after. CMF (host performance) data. EMC Customer Engineer 2 Refer to the Appendix for sample JCL to enable Delta Set Extension. 3 Example SRDF Host Component command used to activate Delta Set Extension: #SC SRDFA_DSE,LCL(ucb,rdfgrp#),ACT 4 Example SRDF Host Component command used to deactivate Delta Set Extension: #SC SRDFA_DSE,LCL(ucb,rdfgrp#),DEACT 5 Refer to the Appendix for sample JCL to disable Delta Set Extension. 6 Refer to the Appendix for sample JCL to enable Delta Set Extension. 7 Permanent activation of SRDF-DSE was scheduled for a few days later following a successful test. This was done so as to adhere to the customer s standard production processing change-control practices. Applied Technology 7

8 With regard to the monitoring of the storage environment during the test, this task was accomplished using standard SRDF Host Component, SRDF/A, and SRDFA_DSE queries that were issued repeatedly during the period that the SRDF link was down and during the period between when the SRDF links were re-connected and normal SRDF/A cycle switching was re-established. Additionally, during these times, overall system performance was monitored using the host SDSF displays. During the test period, the customer s production workload primarily consisted of large DB2 table reorg jobs that were moderately write intensive. Testing results and observations Overall, this testing of SRDF/A Delta Set Extension can be considered to have successfully demonstrated that when properly configured, Delta Set Extension can significantly increase the robustness of an SRDF/A-based replication environment, with only minimal impact to host application response times and Symmetrix component utilization during the periods for which Delta Set Extension is invoked. The rationale for this statement is based on the analysis of data captured during the period of the test, a summary of which follows. The chart in Figure 2 details host IO rates and response times as measured by CMF, during the period for which the SRDF links were removed from the primary Symmetrix DMX system and SRDF/A Transmit Idle was active, overall host IO rates and response time were essentially flat at around 1100 IOs/sec and 1.0 ms average, respectively, across all volumes configured within the Symmetrix DMX. Only during the period immediately after the SRDF links were reconnected and where the propagation of accumulated SRDF/A writes destined for the secondary Symmetrix DMX system was taking place was there any impact to host activity, and only for a short duration (approximately 15 minutes). This impact, while measurable, was not of significant magnitude to cause any major delays to production jobs. Figure 2. SSCH count and average response time (ms) during the SRDF/A DSE test Applied Technology 8

9 With regard to the effect of SRDF/DSE on the Symmetrix units involved, the chart in Figure 3 indicates how the use of DSE impacted the use of cache by SRDF/A, and how Delta Set Extension balanced the SRDF/A workload between cache and the DSE SAV devices. As can be seen in the chart, DSE SAV device space was not required until roughly 160,000 cache slots had been consumed by SRDF/A. From that point onward until the SRDF links were re-inserted, Delta Set Extension maintained the amount of cache in use by SRDF/A at about 160,000 slots by paging out SRDF/A writes as necessary to the pre-configured DSE SAV devices. At the point in time when the cables were plugged back in (cycle switch #34889), all SRDF/A data that had been accumulated into the active (capture) cycle transitioned automatically into the transmit (or inactive) cycle, and commenced synchronizing across the SRDF link. An interesting observation here is what happened to the balance between data in cache and data on DSE SAV devices during this synchronization. As can be seen in the chart, as soon as the cycle switch occurred, the data in the transmit cycle rapidly commenced propagation across the links, with the data held in cache being the first data to propagate. Only after most of the data in cache associated with the transmit cycle had propagated across the link was data paged back in from the DSE pool to cache as part of the propagation process, and only at a rate that was able to keep the SRDF links busy. This paging design was purposely adopted so as to ensure that cache remains as free as possible so as to be able to accept in coming host writes. Figure 3. SRDF/A capture and transmit delta set sizes during the SRDF/A DSE test There are several other points to note in the chart in Figure 3: The rounded saw-tooth pattern of growth in the capture cycle that occurs after each cycle switch is indicative of the amount of cache slot re-use associated with the locality of reference of the host workload (also known as write folding ), as well as of the amount by which the configured SRDF bandwidth exceeds the level of host write activity. There is a need to ensure that DSE SAV devices are configured on both sides of the SRDF/A link. If DSE SAV devices had only been present in the primary Symmetrix DMX during this testing, there is no way that SRDF/A would have been able to remain active, as when cycle switch occurred it would have been Applied Technology 9

10 necessary for the secondary Symmetrix to have been able to accommodate approximately 465,000 tracks of data. This is approximately 1.5 times the amount of cache available to SRDF/A in the secondary frame. Lastly, an examination of the primary Symmetrix component utilizations as detailed in the chart in Figure 4 indicates that throughout the period when DSE was active no component came close to hitting a utilization threshold. Perhaps the most interesting correlation here is actually between the disk utilization (shown in blue on this chart), and average IO response time as shown in crimson on the host performance chart in Figure 3. Of particular interest is the period between roughly 11:15 and 11:30, when both disk utilization and IO response time spike. It is suggested that the use of DMX dynamic cache partitioning to limit the amount of cache able to be used by the DSE pool devices, and as such the rate at which data can be propagated from the DSE SAV devices into cache, could be beneficial in any attempt to maintain a more consistent IO response time during periods when DSE is paging data back in. + Figure 4. Maximum Symmetrix component utilization Applied Technology 10

11 Conclusions and recommendations Based on the success of this testing, the customer adopted SRDF/A Delta Set Extension as a permanent component of their storage replication infrastructure five days after the conclusion of this project through a planned production change. The testing demonstrated that through the use of Delta Set Extension, they would now be able to rely on SRDF/A staying active during an all SRDF links down situation for well in excess of an hour, which was the design point they felt it was necessary to achieve in order to cater for unplanned network outages. From the perspective of recommendations that can be derived from this project, the value of conducting a comprehensive planning phase involving all necessary participants, and that undertakes an appropriate level of investigation into the workload to be supported and the characteristics of the storage environment, cannot be underestimated. Specific attention should be given to: Ensuring that the objectives and requirements for the implementation of SRDF/A Delta Set Extension are clearly understood by all participants. Ensuring that the amount of cache configured in the primary and secondary Symmetrix DMX systems is adequate for the normal day-to-day operation of SRDF/A. SRDF/A Delta Set Extension should not be considered a substitute for cache. Ensuring that the amount of SRDF link bandwidth configured is sufficient to accommodate the normal dayto-day operation of SRDF/A, and that there is additional bandwidth available for resuming normal cycle switching within a reasonable timeframe at the conclusion of a period where the invocation of SRDF/A Delta Set Extension has been necessary due to network outage or workload burst. Ensuring that the levels of both the Enginuity and host software involved with the replication environment are appropriately maintained. The number, size, and placement of the DSE SAV devices within the Symmetrix DMX systems. As has been demonstrated herein, the adoption of SRDF/A Delta Set Extension as part of a balanced SRDF/A configuration can add significant resilience to the replication solution and should be considered favorably as a valuable means of providing additional business value through the prevention of unnecessary replication outages. References The following documents, which are available on EMC Powerlink, provide additional information on SRDF/A Delta Set Extension. Technical note SRDF/A Reserve Capacity Enhancements: Transmit Idle and Delta Set Extension for Mainframe Environments Product documentation EMC SRDF Host Component for z/os Product Guide EMC ResourcePak Base for z/os Product Guide EMC Solutions Enabler Symmetrix SRDF Family CLI Product Guide TechBooks EMC SRDF/A and SRDF/A Multi-Session Consistency on z/os Solutions Guide EMC SRDF/A and SRDF/A Multi-Session Consistency on UNIX and Windows Solutions Guide Applied Technology 11

12 Appendix: Sample JCL for managing SRDF/A DSE The following JCL examples are aimed at providing appropriate guidance as to how to control SRDF/A Delta Set Extension only and can be tailored as necessary to meet local site standards and conventions prior to use. Create the DSE pools //jobname JOB (account info),pgmmrname,class=a,msgclass=x, // MSGLEVEL=(1,1),NOTIFY=&SYSUID THIS JOB CREATES THE POOLS CREATE LOCAL DSE POOL CONFIGPOOL CREATE(LOCAL(UNIT(gatekeeper ucb)) TYPE(DSEPOOL) - POOL(LCLDSEPOOL) ) NOW CREATE REMOTE DSEPOOL CONFIGPOOL CREATE(REMOTE(UNIT(gatekeeper ucb) RAGROUP(01) - CONTROLLER(DMX serial number)) TYPE(DSEPOOL) POOL(RMTDSEPOOL)) // Applied Technology 12

13 Add devices to the DSE pool //jobname JOB (account info),pgmmrname,class=a,msgclass=x, // MSGLEVEL=(1,1),NOTIFY=&SYSUID THIS JOB ADDS DEVICES TO THE POOLS ADD DEVICES TO LOCAL DSE POOL CONFIGPOOL ADD(LOCAL(UNIT(gatekeeper ucb)) TYPE(DSEPOOL) - POOL(LCLDSEPOOL) DEV(F ) MEMBERSTATE(DISABLE)) NOW ADD DEVICE TO REMOTE DSEPOOL CONFIGPOOL ADD(REMOTE(UNIT(gatekeeper ucb) RAGROUP(01) - CONTROLLER(DMX serial number)) TYPE(DSEPOOL) POOL(RMTDSEPOOL) - DEV(114B-11DA) MEMBERSTATE(DISABLE)) // Applied Technology 13

14 Enable DSE //jobname JOB (account info),pgmmrname,class=a,msgclass=x, // MSGLEVEL=(1,1),NOTIFY=&SYSUID THIS JOB ENABLES THE DEVICES IN THE POOLS NOTE - THE REMOTE POOL IS ENABLED FIRST SO AS TO ENSURE THERE IS SPACE AVAILABLE IN THE TARGET TO ACCOMODATE A LARGE DELTA SET THAT IS SENT FROM THE SOURCE ENABLE 16 DEVICES IN THE REMOTE DSEPOOL CONFIGPOOL ENABLE(REMOTE(UNIT(gatekeeper ucb) RAGROUP(01) - CONTROLLER(DMX serial number)) TYPE(DSEPOOL) POOL(RMTDSEPOOL) - DEV(114B-115A)) ENABLE 16 DEVICES IN THE LOCAL DSE POOL CONFIGPOOL ENABLE(LOCAL(UNIT(gatekeeper ucb)) TYPE(DSEPOOL) - POOL(LCLDSEPOOL) DEV(F85-F94)) // Applied Technology 14

15 Display the DSE pools //jobname JOB (account info),pgmmrname,class=a,msgclass=x, // MSGLEVEL=(1,1),NOTIFY=&SYSUID THIS JOB DISPLAYS THE POOLS DISPLAY THE LOCAL DSE POOL CONFIGPOOL DISPLAY(LOCAL(UNIT(gatekeeper ucb)) TYPE(DSEPOOL) - POOL(LCLDSEPOOL)) NOW DISPLAY THE REMOTE DSEPOOL CONFIGPOOL DISPLAY(REMOTE(UNIT(gatekeeper ucb) RAGROUP(01) - CONTROLLER(DMX serial number)) TYPE(DSEPOOL) POOL(RMTDSEPOOL)) // Applied Technology 15

16 Drain a device in the DSE pool //jobname JOB (account info),pgmmrname,class=a,msgclass=x, // MSGLEVEL=(1,1),NOTIFY=&SYSUID THIS JOB DRAINS A DEVICE IN THE POOL NOTE - THE LOCAL POOL DEVICE IS DRAINED FIRST SO AS TO ENSURE THERE IS SPACE AVAILABLE IN THE TARGET TO ACCOMODATE A LARGE DELTA SET THAT IS SENT FROM THE SOURCE DRAIN A DEVICE IN THE LOCAL DSE POOL CONFIGPOOL DRAIN(LOCAL(UNIT(gatekeeper ucb)) TYPE(DSEPOOL) - POOL(LCLDSEPOOL) DEV(F85)) DRAIN A DEVICE IN THE REMOTE DSEPOOL CONFIGPOOL DRAIN(REMOTE(UNIT(gatekeeper ucb) RAGROUP(01) - CONTROLLER(DMX serial number)) TYPE(DSEPOOL) POOL(RMTDSEPOOL) - DEV(114B)) // Applied Technology 16

17 Disable DSE //jobname JOB (account info),pgmmrname,class=a,msgclass=x, // MSGLEVEL=(1,1),NOTIFY=&SYSUID THIS JOB DISABLES DSE NOTE - THE LOCAL POOL IS DISABLED FIRST SO AS TO ENSURE THERE IS SPACE AVAILABLE IN THE TARGET TO ACCOMODATE A LARGE DELTA SET THAT IS SENT FROM THE SOURCE DISABLE DEVICES IN THE LOCAL DSE POOL CONFIGPOOL DISABLE(LOCAL(UNIT(gatekeeper ucb)) TYPE(DSEPOOL) - POOL(LCLDSEPOOL) DEV(ALL)) DISABLE DEVICES IN THE REMOTE DSEPOOL CONFIGPOOL DISABLE(REMOTE(UNIT(gatekeeper ucb) RAGROUP(01) - CONTROLLER(DMX serial number)) TYPE(DSEPOOL) POOL(RMTDSEPOOL) - DEV(ALL)) // Applied Technology 17