IBM Virtualization Engine TS7720 and TS7740 Release 3.0 Performance White Paper - Version 2

Transcription

1 IBM System Storage March 27, 213 IBM Virtualization Engine TS772 and TS774 Release 3. Performance White Paper - Version 2 By Khanh Ly and Luis Fernando Lopez Gonzalez Tape Performance IBM Tucson

2 Page 2 Table of Contents Table of Contents...2 Introduction...3 TS77 Performance Evolution...4 Hardware Configuration...6 TS77 Performance Overview...6 TS77 Basic Performance...1 Additional Performance Metrics...23 Performance at Distance...27 Performance Tools...29 Conclusions...3 Acknowledgements...31

3 Page 3 Product Release Enhancements Introduction This paper provides performance information for the IBM Virtualization Engine TS772 and TS774, which are the two current products in the TS77 Virtualization Engine family of products. This paper is intended for use by IBM field personnel and their customers in designing virtual tape solutions for their applications. This is an update to the previous TS77 paper dated January 3, 213 and reflects changes for release 3., which introduces the new disk cache models 3956-CS9 for TS772 and 3956-CC9 for TS774. Cache throughput for the CS9/CC9 has improved over the CS8/CC8 and thus significantly increases performance runs which are limited by disk cache throughput. This paper is published in two different R 3. minor versions. Contents for each version are: Version 1: Standalone TS772 with VEA or VEB server and CS8 or CS9 cache. Standalone TS774 with V6 or V7 server and CC8 or CC9 cache Grid configurations with 4x1Gb links. Some grid configuration with 2x1Gb links. Version 2: More grid configurations with 2x1Gb links. Performance vs. premigration drives. Performance vs. data compression. Copy Performance over distance. The following are performance related changes in release R 3.: Disk Cache Refresh with larger capacity and better cache throughput 3956 CC9/CX9 with 6 GB DDM One, two, or 3 drawers configurations Up to 28+ TB capacity Improved base write data rate by over 6% as compared to 3956 CC8/CX CS9/XS9 with 3 TB DDM Up to 1 drawers (about 24 TB) in the base frame Up to 16 drawers (about 384 TB) in the expansion frame Total capacity up to 623 TB Base write data rate for 3956 CS9/XS9 with base frame is about the same as 3956 CS8/XS7 base plus an expansion frame. Encryption for data at rest.

4 Page 4 TS77 Performance Evolution The TS77 architecture continues to provide a base for product growth in both performance and functionality. Figures 1 and 2 shows the write and read performance improvement histories. Standalone VTS/TS77 Performance History Standalone Performance Evolution B1 VTS (4xFICON) B2 VTS (8xFICON) W/FPA TS774 V6/CC6/CX6 R 1.3 (4DRs/4x2Gb/z9) TS774 V6/CC6/CX6 R 1.5 (4DRs/4x2Gb/z9) TS774 V6/CC6/CX6 R 1.5 (4DRs/4x2Gb/z99) TS772 VEA/CS7/XS7 R 1.5 (7DRs/4x4Gb/z1) TS774 V6/CC7/CX7 R 1.6 (4DRs/4x4Gb/z1) TS772 VEA/CS7/XS7 R 1.6 (7DRs/4x4Gb/z1) TS774 V6/CC8/CX7 R 1.7 (4DRs/4x4Gb/z1) TS772 VEA/CS8/XS7 R 1.7 (7DRs/4x4Gb/z1) TS772 VEA/CS8/XS7 R 1.7 (19DRs/4x4Gb/z1) TS774 V7/CC8/CX7 R 2. (4DRs/4x4Gb/z1) TS772 VEB/CS8/XS7 R 2. (7DRs/4x4Gb/z1) TS772 VEB/CS8/XS7 R 2. (19DRs/4x4Gb/z1) TS774 V7/CC8/CX7 R 2.1 (4DRs/4x4Gb/z1) TS772 VEB/CS8/XS7 R 2.1 (7DRs/4x4Gb/z1) TS772 VEB/CS9/XS9 R 2.1PGA2 (5DRs/4x4Gb/z1) TS772 VEB/CS9/XS9 R 2.1PGA2 TS772 VEB/CS9/XS9 R 3. (1-DRs/4x4Gb/z1) TS774 V7/CC9/CX9 R 3. (3DRs/4x4Gb/z1) Host MB/s (uncompressed) Sustained Peak Figure 1. VTS/TS77 Standalone Bandwidth. All runs were made with 128 concurrent jobs, each job writing 8 MiB (3 MiB compression) using 32KiB blocks, and QSAM BUFNO = 2. Notes: (ndrs/ 4x4Gb/ z1) refers to (number of cache drawers/four 4Gb FICON channels/z1 host)

5 Page 5 Standalone VTS/TS77 Read Hit Performance History B1 VTS (4xFICON) B2 VTS (8xFICON) W/FPA TS774 V6/CC6/CX6 R 1.3 (4DRs/4x2Gb/z9) TS774 V6/CC6/CX6 R 1.5 (4DRs/4x2Gb/z9) TS774 V6/CC6/CX6 R 1.5 (4DRs/4x2Gb/z99) TS772 VEA/CS7/XS7 R 1.5 (7DRs/4x4Gb/z1) TS774 V6/CC7/CX7 R 1.6 (4DRs/4x4Gb/z1) TS772 VEA/CS7/XS7 R 1.6 (7DRs/4x4Gb/z1) TS774 V6/CC8/CX7 R 1.7 (4DRs/4x4Gb/z1) TS772 VEA/CS8/XS7 R 1.7 (7DRs/4x4Gb/z1) TS772 VEA/CS8/XS7 R 1.7 (19DRs/4x4Gb/z1) TS774 V7/CC8/CX7 R 2. (4DRs/4x4Gb/z1) TS772 VEB/CS8/XS7 R 2. (7DRs/4x4Gb/z1) TS772 VEB/CS8/XS7 R 2. (19DRs/4x4Gb/z1) TS774 V7/CC8/CX7 R 2.1 (4DRs/4x4Gb/z1) TS772 VEB/CS8/XS7 R 2.1 (7DRs/4x4Gb/z1) TS772 VEB/CS9/XS9 R 2.1PGA2 (5DRs/4x4Gb/z1) TS772 VEB/CS9/XS9 R 2.1PGA2 TS772 VEB/CS9/XS9 R 3. (1-DRs/4x4Gb/z1) TS774 V7/CC9/CX9 R 3. (3DRs/4x4Gb/z1) Host MB/s (uncompressed) Read Hit Figure2. VTS/TS77 Standalone Read Bandwidth. All runs were made with 128 concurrent jobs, each job writing 8 MiB (3 MiB compression) using 32KiB blocks, and QSAM BUFNO = 2. See definition for Read Hit in the section TS77 Performance Overview. Notes: (n-drs - 4x4Gb z1) refers to (number of cache drawers four 4Gb FICON channels z1 host) Notes: The following convention is used in this paper: Binary Decimal Name Symbol Values in Bytes Symbol Name Values in Bytes kibibyte KiB 2 1 KB kilobyte 1 3 mebibyte MiB 2 2 MB megabyte 1 6 gibibyte GiB 2 3 GB gigabyte 1 9 tebibyte TiB 2 4 TB terabyte 1 12 pebibyte PiB 2 5 PB petabyte 1 15

6 Page 6 Hardware Configuration The following hardware was used in performance measurements. Performance workloads are driven from IBM System z1 host with four 4Gb FICON channels. Standalone Hardware Setup Hardware Configuration TS77 Model Drawer count Cache size Tape Drives TS772 VEA 3956 CS TB N/A TS772 VEA 3956 CS8/XS TB N/A TS772 VEB 3956 CS TB N/A TS772 VEB 3956 CS8/XS TB N/A TS772 VEB 3956 CS8/XS TB N/A TS772 VEB 3956 CS TB N/A TS772 VEB 3956 CS9/XS TB N/A TS772 VEB 3956 CS9/XS TB N/A TS772 VEB 3956 CS9/XS TB N/A TS772 VEB 3956 CS9/XS TB N/A TS774 V CC8 1 7 TB 12 TS114 TS774 V CC8/CX TB 12 TS114 TS774 V CC8 1 7 TB 12 TS114 TS774 V CC8/CX TB 12 TS114 TS774 V CC TB 12 TS114 TS774 V CC9/CX TB 12 TS114 TS774 V CC9/CX TB 12 TS114 Grid Hardware Setup TS77 Model Draw er count Cache size Tape Drives Grid links TS772 VEB 3956 CS9/XS TB N/A 4x1Gb or 2x1Gb TS772 VEB 3956 CS9/XS TB N/A 4x1Gb or 2x1Gb TS774 V CC9/CX TB 12 TS114 4x1Gb or 2x1Gb TS77 Performance Overview Performance Workloads and Metrics

7 Page 7 Metrics and Workloads Performance shown in this paper has been derived from measurements that generally attempt to simulate common user environments, namely a large number of jobs writing and/or reading multiple tape volumes simultaneously. Unless otherwise noted, all of the measurements were made with 128 simultaneously active virtual tape jobs per active cluster. Each tape job was writing or reading 8 MiB of uncompressed data using 32 KiB blocks and QSAM BUFNO=2, using data that compresses at the TS77 at. Measurements were made with four 4-gigabit (Gb) FICON channels on a z1 host. All runs are begun with the virtual tape subsystem inactive. Unless otherwise stated, all runs were made with default tuning values (DCOPYT=125, DCTAVGTD=1, PMPIOR=16, PMTHLVL=2, ICOPYT=ENABLED, Reclaim disabled, Number of premigration drives per pool=1). Refer to the IBM Virtualization Engine TS77 Series Best Practices - Understanding, Monitoring and Tuning the TS77 Performance white paper for detailed description of different tuning settings Types of Throughput Because the TS772 is a disk-cache only cluster. The read and write data rates have been found to be fairly consistent throughout a given workload. Because the TS774 contains physical tapes to which the cache data will be periodically written and read, the TS774 has been found to exhibit four basic throughput rates: peak write, sustained write, read-hit, and recall. These four rates are described below. Peak and Sustained Throughput. For all TS774 measurements, any previous workloads have been allowed to quiesce with respect to pre-migration to back tape and replication to other clusters in the grid. In other words, the test is started with the grid in an idle state. Starting with this initial idle state, data from the host is first written into the TS774 disk cache with little if any premigration activity taking place. This allows for a higher initial data rate, and is termed the peak data rate. Once a pre-established threshold is reached of non-premigrated data, the amount of premigration is increased, which can reduce the host write data rate. This threshold is called the premigration priority threshold, and has default value of 16 gigabytes (GB). When a further threshold of non-premigrated data is reached, the incoming host activity is actively throttled to allow for increased premigration activity. This throttling mechanism operates to achieve a balance between the amount of data coming in from the host and the amount of data being copied to physical tape. The resulting data rate for this mode of behavior is called the sustained data rate, and could theoretically continue on forever, given a constant supply of logical and physical scratch tapes. This second threshold is called the premigration throttling threshold, and has a default value of 2 gigabytes (GB). These two thresholds can be used in conjunction with the peak data rate to project the duration of the peak period. Note that both the priority and throttling thresholds can be increased or decreased via a host command line request.

8 Page 8 Read-hit and Recall Throughput Similar to write activity, there are two types of TS774 read performance: readhit (also referred to as peak ) and recall (also referred to as read-miss ). A read hit occurs when the data requested by the host is currently in the local disk cache. A recall occurs when the data requested is no longer in the disk cache and must be first read in from physical tape. Read-hit data rates are typically higher than recall data rates. These two read performance metrics, along with peak and sustained write performance are sometimes referred to as the four corners of virtual tape performance. The following charts in the paper show three of these corners: 1. peak write 2. sustained write 3. read hit Recall performance is dependent on several factors that can vary greatly from installation to installation, such as number of physical tape drives, spread of requested logical volumes over physical volumes, location of the logical volumes on the physical volumes, length of the physical media, and the logical volume size. Because these factors are hard to control in the laboratory environment, recall is not part of lab measurement.

9 Page 9 Grid Considerations Grid Considerations Up to four TS77 clusters can be linked together to form a Grid configuration. Five- or six-way grid configuration is available via irpq. The connection between these clusters is provided by two 1-Gb TCP/IP links (default). Four 1- Gb links or two 1-Gb links options are also available. Data written to one TS77 cluster can be optionally copied to the other cluster(s). Data can be copied between the clusters in either deferred, RUN (also known as Immediate ), or sync mode copy. In RUN copy mode the rewind-unload response at job end is held up until the received data is copied to the other cluster(s). In deferred copy mode data is queued for copying, but the copy does not have to occur prior to job end. Deferred copy mode allows for a temporarily higher host data rate than RUN copy mode because copies to the other cluster(s) can be delayed, which can be useful for meeting peak workload demands. Care must be taken, however, to be certain that there is sufficient recovery time for deferred copy mode so that the deferred copies can be completed prior to the next peak demand. Whether delay occurs and by how much is configurable through the Library Request command. In sync mode copy, data synchronization is up to SYNC level granularity across two clusters within a grid configuration. In order to provide a redundant copy of these items with a zero recovery point objectives (RPO), the sync mode copy function will fork the host record writes to two clusters simultaneously.

10 Page 1 TS77 Basic Performance The following sets of graphs show basic TS77 bandwidths. The graphs in Figures 3 and 4 show single cluster, or standalone. The performance metric shown in these and all other data rate charts in this paper is host-view (uncompressed) MB/sec. TS772 Standalone Performance Standalone TS772 R 3. Performance TS772 Standalone Bandwidth H os t M B /s (U nc om pres s ed) :1 Read 1:1 Mixed 1:1 Read Mixed VEA/CS8/1 DR VEA/CS8/7 DRs VEB/CS8/1 DR VEB/CS8/4 DRs VEB/CS8/7 DRs VEB/CS9/1 DR VEB/CS9/2 DRs VEB/CS9/3 DRs VEB/CS9/6 DRs VEB/CS9/1 DRs VEB/CS9/26 DRs Figure 3. TS772 Standalone and/or Read Bandwidth. All runs were made with 128 concurrent jobs, each job writing 8 MiB (uncompressed) using 32KiB blocks, QSAM BUFNO = 2, using four 4Gb FICON channels from a z1 LPAR. Notes: Mixed workload refers to 5% write plus 5% read hit workload driven from the host. The resulting data rate seen from TS772 is not 5-5 read/write mixed; normally it is more read than write.

11 Page 11 TS774 Standalone Performance Standalone TS774 R 3. Performance Comparison 12 TS774 Standalone Bandwidth Host MB/s (Uncompressed) Pk Wr 1:1 St. Wr 1:1 Read 1:1 Pk Mx 1:1 St. Mx 1:1 Pk Wr St. Wr Read Pk Mx St. Mx V6/CC8/1 DR V6/CC8/4DRs V7/CC8/1DR V7/CC8/4DRs V7/CC9/1DR V7/CC9/2DRs V7/CC9/3DRs Figure 4. TS774 Standalone and/or Read Bandwidth. All runs were made with 128 concurrent jobs, each job writing 8 MiB (uncompressed) using 32KiB blocks, QSAM BUFNO = 2, using four 4Gb FICON channels from a z1 LPAR. Notes: Mixed workload refers to 5% write plus 5% read hit workload driven from the host. The resulting data rate seen from TS774 is not 5-5 read/write mixed; normally it is more read than write.

12 Page 12 TS77 Grid Performance TS77 Grid Bandwidth Figures 5, 6, 8 through 1, 12, 14, 16, 18, 2 and 22 show Grid bandwidth. For these charts D stands for deferred copy mode, S stands for sync mode copy and R stands for RUN (immediate) copy mode. For example, in Figure 5, RR represents RUN for cluster, and RUN for cluster 1. SS refers to synchronous copies for both clusters. All measurements for these graphs were made at zero or near-zero distance between clusters. Two-way TS77 Grid with Single Active Cluster Performance 12 Two-way TS772 R 3. Performance (Single Active Cluster) Two-way TS77 Grid with Single Active Cluster Bandwidth Host MB/s (Uncompressed) DD SS RR Read Hit VEB/CS9/1 DRs/4x1Gb VEB/CS9/26 DRs/4x1Gb VEB/CS9/1 DRs/2x1Gb Figure 5. Two-way TS772 Grid Maximum Bandwidth for Single Active Cluster. Unless otherwise stated, all runs were made with 128 concurrent jobs. Each job writing or reading 8 MiB (3MiB compression) using 32 KiB block size, QSAM BUFFNO = 2, using four 4Gb FICON channels from a z1 LPAR. Clusters are located at zero or near zero distance to each other in laboratory setup. DCT=125. Tuning Tips: The CS9 allows concurrent copy during batch without much impact to the host throughput, so it is recommended to set the DCT to when using a deferred copy mode in a single active cluster.

13 Page 13 Two-way TS774 R 3. Performance (Single Active Cluster) 12 Host MB/s (Uncom pressed) DD Peak DD Sust. SS Peak SS Sust. RR Peak RR Sust. Read Hit - V7/CC9/3 DRs/4x1Gb V7/CC9/3 DRs/2x1Gb Figure 6. Two-way TS774 Grid Maximum Bandwidth for Single Active Cluster. All runs were made with 128 concurrent jobs. Each job writing or reading 8 MiB (3MiB compression) using 32 KiB block size, QSAM BUFFNO = 2, using four 4Gb FICON channels from a z1 LPAR. Clusters are located at zero or near zero distance to each other in laboratory setup. DCT=125.

14 Page 14 Figure 7. Two-way TS77 Hybrid Grid H1 Two-way TS77 R 3. Hybrid H1 Performance (Single Active Cluster) Host MB/s (Uncompressed) DD Read Hit VEB/CS9/1 DRs - V7/CC9/3 DRs - 4x1Gb Figure 8. Two-way TS77 Hybrid Grid H1 Maximum Bandwidth for Single Active Cluster. All runs were made with 128 concurrent jobs. Each job writing or reading 8 MiB (3 MiB compression) using 32 KiB block size, QSAM BUFFNO = 2, using four 4Gb FICON channels from a z1 LPAR. Clusters are located at zero or near zero distance to each other in laboratory setup. DCT=125. Tuning Tips: The powerful CS9 allows concurrent copy during batch without much impact to the host throughput, so it is recommended to set the DCT to with DD copy mode for single active cluster.

15 Page 15 Two-way TS77 Grid with Dual Active Cluster Performance 25 Two-way TS772 R 3. Performance (Dual Active Clusters) Host MB/s (Uncompressed) Two-way TS77 Grid with Dual Active Clusters Bandwidth DD SS RR Read Hit VEB/CS9/1 DR/4x1G VEB/CS9/1 DR/4x1Gb/64 jobs VEB/CS9/1 DR/2x1Gb VEB/CS9/1 DR/2x1Gb/64 jobs VEB/CS9/26 DR/4x1Gb * 64 jobs 128 jobs (64 jobs per active cluster) Figure 9. Two-way TS772 Grid Maximum Bandwidth for Dual Active Clusters. Unless otherwise stated, all runs were made with 256 concurrent jobs (128 jobs per active cluster). Each job writing or reading 8 MiB (3 MiB compression) using 32 KiB block size, QSAM BUFFNO = 2, using eight 4Gb FICON channels from a z1 LPAR (four 4Gb channels per active cluster). Clusters are located at zero or near zero distance to each other in laboratory setup. DCT=125. Tuning Tips: Lower the number of concurrent jobs improves host data rate for SS copy mode.

16 Page 16 Two-way TS774 R 3. Performance (Dual Active Clusters) Host M B/s (Uncom pressed) DD Peak DD Sust. SS Peak SS Sust. RR Peak RR Sust. Read Hit - R V7/CC9/3 DRs/4x1Gb R V7/CC9/3 DRs/2x1Gb R V7/CC9/3 DRs/4x1Gb/64 jobs R V7/CC9/3 DRs/2x1Gb/64jobs * 64 jobs 128 jobs (64 jobs per active cluster) Figure 1. Two-way TS774 Grid Maximum Bandwidth for Dual Active Clusters. Unless otherwise stated, all runs were made with 256 concurrent jobs (128 jobs per active cluster). Each job writing or reading 8 MiB (3 MiB compression) using 32 KiB block size, QSAM BUFFNO = 2, using eight 4Gb FICON channels from a z1 LPAR (four 4Gb channels per active cluster). Clusters are located at zero or near zero distance to each other in laboratory setup. DCT=125. Tuning Tips: Lower the number of concurrent jobs improves host data rate for SS copy mode.

17 Page 17 Figure 11. Two-way TS77 Hybrid Grid H2 Two-way TS77 R 3. Hybrid H2 Performance (Dual Active Clusters) Host MB/s (Uncompressed) DD Peak DD Sust. Read Hit VEB/CS9/1 DRs - V7/CC9/3 DRs - 4x1Gb * 64 jobs 128 jobs (64 jobs per active cluster) Figure 12. Two-way TS77 Hybrid Grid H2 Maximum Bandwidth for Dual Active Clusters. Unless otherwise stated, all runs were made with 256 concurrent jobs (128 jobs per active cluster). Each job writing or reading 8 MiB (3 MiB compression) using 32 KiB block size, QSAM BUFFNO = 2, using eight 4Gb FICON channels from a z1 LPAR (four 4Gb channels per active cluster). Clusters are located at zero or near zero distance to each other in laboratory setup. DCT=125.

18 Page 18 Three-way TS77 Grid with Dual Active Cluster Performance Three-way TS77 Grid with Dual Active Clusters Bandwidth Figure 13. Three-way TS77 Hybrid Grid H3 Host MB/s (Uncompressed) Three-way TS77 R 3. Hybrid H3 Performance (Dual Active Clusters) RND/NRD SSD Peak SSD Sust. RRD Read Hit R VEB/CS9/1 DRs - V7/CC9/3 DRs - 4x1G R VEB/CS9/1 DRs - V7/CC9/3 DRs - 4x1G/64 jobs * 64 jobs 128 jobs (64 jobs per active cluster) Figure 14. Three-way TS77 Hybrid Grid H3 Maximum Bandwidth for Dual Active Clusters. Unless otherwise stated, all runs were made with 256 concurrent jobs (128 jobs per active cluster). Each job writing or reading 8 MiB (3 MiB compression) using 32 KiB block size, QSAM BUFFNO = 2, using eight 4Gb FICON channels from a z1 LPAR (four 4Gb channels per active cluster). Clusters are located at zero or near zero distance to each other in laboratory setup. DCT=125.

19 Page 19. Figure 15. Three-way TS77 Hybrid Grid H4 Three-way TS77 R 3. Hybrid H4 Performance (Dual Active Clusters) Host MB/s (Uncompressed) DDD Peak DDD Sust. SSD Peak SSD Sust. RRD Peak RRD Sust. Read Hit VEB/CS9/1 DRs - V7/CC9/3 DRs - 4x1Gb VEB/CS9/26 DRs - V7/CC9/3 DRs - 4x1Gb VEB/CS9/1 DRs - V7/CC9/3 DRs - 2x1Gb/64jobs VEB/CS9/1 DRs - V7/CC9/3 DRs - 4x1Gb/64jobs VEB/CS9/1 DRs - V7/CC9/3 DRs - 2x1Gb * 64 jobs 128 jobs (64 jobs per active cluster) Figure 16. Three-way TS77 Hybrid Grid H4 Maximum Bandwidth for Dual Active Clusters. Unless otherwise stated, all runs were made with 256 concurrent jobs (128 jobs per active cluster). Each job writing or reading 8 MiB (3 MiB compression) using 32 KiB block size, QSAM BUFFNO = 2, using eight 4Gb FICON channels from a z1 LPAR (four 4Gb channels per active cluster). Clusters are located at zero or near zero distance to each other in laboratory setup. DCT=125. Tuning Tips: Lower the number of concurrent jobs improves host data rate for SS copy mode.

20 Page 2 Four-way TS77 Grid with Dual Active Cluster Performance Four-way TS77 Hybrid Grid with Dual Active Clusters Bandwidth Figure 17. Four-way TS77 Hybrid Grid H5 Four-way TS77 R 3. Hybrid H5 Performance (Dual Active Clusters) Host MB/s (Uncompressed) RNDD/NRDD Peak Read Hit R VEB/CS9/1 DRs - V7/CC9/3 DRs - 4x1Gb/DCT=125 R VEB/CS9/1 DRs - V7/CC9/3 DRs - 4x1Gb/DCT=1 R VEB/CS9/1 DRs - V7/CC9/3 DRs - 4x1Gb/DCT= R VEB/CS9/1 DRs - V7/CC9/3 DRs - 2x1Gb/DCT=1 R VEB/CS9/1 DRs - V7/CC9/3 DRs - 2x1Gb/DCT= Figure 18. Four-way TS77 Hybrid Grid H5 Maximum Bandwidth for Dual Active Clusters. All runs were made with 256 concurrent jobs (128 jobs per active cluster). Each job writing or reading 8 MiB (3 MiB compression) using 32 KiB block size, QSAM BUFFNO = 2, using eight 4Gb FICON channels from a z1 LPAR (four 4Gb channels per active cluster). Clusters are located at zero or near zero distance to each other in laboratory setup. DCT=125.

21 Page 21 Figure 19. Four-way TS77 Hybrid Grid H6 Four-way TS77 R 3. Hybrid H6 Performance (Dual Active Clusters) Host MB/s (Uncompressed) DDDD Peak DDDD Sust. SSDD Peak SSDD Sust. RRDD Peak RRDD Sust. Read Hit VEB/CS9/1 DRs - V7/CC9/3 DRs - 4x1Gb VEB/CS9/1 DRs - V7/CC9/3 DRs - 2x1Gb VEB/CS9/1 DRs - V7/CC9/3 DRs - 4x1Gb/64jobs VEB/CS9/1 DRs - V7/CC9/3 DRs - 2x1Gb/64jobs * 64 jobs 128 jobs (64 jobs per active cluster). Figure 2. Four-way TS77 Hybrid Grid H6 Maximum Bandwidth for Dual Active Clusters. Unless otherwise stated, all runs were made with 256 concurrent jobs (128 jobs per active cluster). Each job writing or reading 8 MiB (3 MiB compression) using 32 KiB block size, QSAM BUFFNO = 2, using eight 4Gb FICON channels from a z1 LPAR (four 4Gb channels per active cluster). Clusters are located at zero or near zero distance to each other in laboratory setup. DCT=125. Tuning Tips: Lower the number of concurrent jobs improves host data rate for SS copy mode.

22 Page 22 Four-way TS77 Hybrid Grid with Quadruple Active Clusters Bandwidth Figure 21. Four-way TS77 Hybrid Grid H7 Four-way TS77 R 3. Hybrid H7 Performance (Quadruple Active Clusters) Host MB/s (Uncompressed) SSDD-DDSS Peak SSDD-DDSS Sust. RRDD-DDRR Peak RRDD-DDRR Sust. Read Hit VEB/CS9/1 DRs - V7/CC9/3 DRs - 4x1Gb VEB/CS9/1 DRs - V7/CC9/3 DRs - 2x1Gb VEB/CS9/1 DRs - V7/CC9/3 DRs - 4x1Gb/64jobs VEB/CS9/1 DRs - V7/CC9/3 DRs - 2x1Gb/64jobs * 64 jobs 256 jobs (64 jobs per active cluster) Figure 22. Four-way TS77 Hybrid Grid H7 Maximum Bandwidth for Quadruple Active Clusters. Unless otherwise stated, all runs were made with 256 concurrent jobs per z1 LPAR (each LPAR drives two clusters). Each job writing or reading 8 MiB (3 MiB compression) using 32 KiB block size, QSAM BUFFNO = 2, using eight 4Gb FICON channels from a z1 LPAR (four 4Gb channels per active cluster). Clusters are located at zero or near zero distance to each other in laboratory setup. DCT=125. Tuning Tips: Lower the number of concurrent jobs improves host data rate for SS copy mode.

23 Page 23 Additional Performance Metrics Premigration Rate vs. Premigration Drives The TS774 premigration rates, i.e. the rates at which cache-resident data is copied to physical tapes, depend upon the number of TS114 drives reserved for premigration. By default, the number of drives reserved for premigration is ten per storage pool. The TS774 sustained write rate is the rate at which the host write rate is balanced with the premigration rate to tape. The sustained host write rate depends on the number of premigration drives. The data for figure 23 was measured with no TS774 activity other than the sustained writes and premigration (host write balanced with premigration to tape). The figures 23 and 24 show how the number of premigration drives affects the premigration and sustained write rates: Sustained Data Rate vs. Premigration Drives S u stain e d M B /s (U n c o m p res s ed ) Standalone TS774 (V7/CC9/CX9/3 DRs) Sustained Performance TS114 Drives for Premigration Figure 23. Standalone TS774 sustained write rate vs. the number of TS114 drives reserved for premigration. All runs were made with 128 concurrent jobs. Each job wrote 8 MB (3MB compression) using 32KB blocks, QSAM BUFNO = 2, and four 4Gb FICON channels from a z1 LPAR.

24 Page 24 Standalone TS774 (V7/CC9/CX9/3 DRs) Premigration Performance 12 Premigration Rate vs. Premigration Drives P rem ig a tio n M B /s (C o m p re s s e d ) TS114 Drives for Premigration Figure 24. Standalone TS774 premigration performance vs. the number TS114 drives used for premigration.

25 Page 25 Performance vs. Data Compression Incoming host data is compressed by the TS772/TS774 channel adapters prior to being written to the disk cache. The TS772/TS774 data compression can be turned off, but it should be left on except in those exceptional situations where the data has already been compressed prior to sending it to the TS77. Note that data compression does not reduce host throughput. For two clusters are at zero distance (no grid latency) for these measurements. The following charts show data rates on a two cluster TS77 grid with a single active cluster using RR or SS copy mode and different compression ratios. Two-way TS772 VEB/CS9/1 Drawers/2x1Gb (Single Active Cluster) Performance vs Data Compression Two-way TS772 Single-active RR/SS Data Rates vs. Data Compression H o s t M B / s (U n c o m p re s s e d ) SS RR 1 to to 1 2 to to 1 Data Compression Figure 25. TS772 VEB 3956 CS9/XS9/1 drawers/2x1gb link (single active cluster) RR and SS performance vs. data compression. All runs were made with 128 concurrent jobs, each job writing 8 MB (3MB compression) using 32KB blocks, QSAM BUFNO = 2, and four 4Gb FICON channels from a z1 LPAR.

26 Page 26 Two-way TS774 V7/CC9/3 Drawers/12 E7s/2x1Gb (Single Active Cluster) PERFORMANCE VS. DATA COMPRESSION 12 Two-way TS774 Single-active RR/SS Data Rates vs. Data Compression H o s t M B / s ( U n c o m p r e s s e d ) SS Peak SS Sustained RR Peak RR Sustained 1 to to 1 2 to to 1 Data Compression Figure 26. TS774 V CC9/CX9/3 drawers/2x1gb link (single active cluster) RR and SS performance vs. data compression. All runs were made with 128 concurrent jobs, each job writing 8 MB (3MB compression) using 32KB blocks, QSAM BUFNO = 2, and four 4Gb FICON channels from a z1 LPAR.

27 Page 27 Performance at Distance Grid link performance varies based on several factors: 1. distance between clusters, 2. quality of the grid links, 3. speed of links (1Gb or 1 Gb), 4. number of utilized TCP/IP links (2 or 4 1Gb links or 2 1Gb links), 5. percentage of link bandwidth available, 6. Type of I/O activity (write, read, or copy). The following setup was used to assess the deferred copy performance vs. different inter-cluster distances and packet loss percentages in the laboratory: Two TS772 were connected via an Anue Ethernet Network Emulator. Delay times were used to simulate various distances. Different percentages of packet loss were applied. Total link bandwidth (four 1Gb TCP/IP links) was available for these tests. Sync Mode Copy Figure 27 shows the single-direction sync mode copy rate between two VEB based TS772s each with a CS9/XS9 1 drawer disk cache. Four 1-Gb grid links were used. Different one-way time delay values and packet loss percentages were applied to both directions. TS772 (VEB/CS9/1-DR/4x1G Links) R 3. Sync Mode Copy Performance vs. One-way Time Delay and Packet Loss TS772 VEB/CS9/XS9 (1-Drawers/ 4x1G links) Single-direction Sync Mode Copy Copy MB/s (Compressed) % Packet Loss.3% Packet Loss.1% Packet Loss.3% Packet Loss One-way Latency (ms) Figure 27. Two-wayTS772 VEB 3956 CS9/XS9/1 drawers/4x1gb link (single active cluster) sync mode copy performance vs latency and packet loss. All runs were made with 128 concurrent jobs, each job writing 67 MB (2518 MB compression) using 32KB blocks, QSAM BUFNO = 2, and four 4Gb FICON channels from a z1 LPAR.

28 Page 28 Remote Copy Figure 28 shows the single-direction deferred copy rate between two 3956-VEB based TS772s each with a CS9/XS9 1 drawer disk cache. Four 1-Gb grid links were used. The number of deferred copy tasks was set to 4 (default). This provides an example of how latency and packet loss can affect replication rates. A configuration with 2x1Gb links would provide improved results. During the run, host I/O was driven to one TS772 (active cluster) using deferred copy mode for the second copy. After tens of terabytes (TB) of uncopied data were queued at the active cluster, the workload was stopped. The only activity on the active cluster was the copying to the second TS772 cluster. Different oneway time delay values (latency) and packet loss percentages were applied to both directions. TS772 VEB/CS9/XS9 (1-Drawers/ 4x1G links) Single-direction Copy (4 deferred copy tasks) C o p y M B / s (C o m p r e s s e d ) Two-way TS772 (VEB/CS9/1-DR/4x1G) 1-way Deferred Copy Performance vs. 1-way Latency and Packet Loss (Default: 4 Deferred Copy Tasks) One-way Latency (ms).% Packet Loss.1% Packet Loss.3% Packet Loss.1% Packet Loss.3% Packet Loss Figure 28. Two-wayTS772 VEB 3956 CS9/XS9/1 drawers/4x1gb link (single active cluster) copy performance vs latency and packet loss. All runs were made with 128 concurrent jobs, each job writing 67 MB (2518 MB compression) using 32KB blocks, QSAM BUFNO = 2, and four 4Gb FICON channels from a z1 LPAR.

29 Page 29 Performance Tools Batch Magic This tool is available to IBM representatives and Business Partners to analyze SMF data for an existing configuration and workload, and project a suitable TS77 configuration. Performance Aids BVIRHIST plus VEHSTATS BVIRHIST requests historical statistics from a TS77, and VEHSTATS produces the reports. The TS77 keeps the last 9 days of statistics. BVIRHIST provides a means for users to save statistics for periods longer than 9 days. Performance Analysis Tools A set of performance analysis tools is available on Techdocs that utilizes the data generated by VEHSTAT. Provided are spreadsheets, data collection requirements, and a 9 day trending evaluation guide to assist in the evaluation of the TS77 performance. Spreadsheets for a 9 day, one week, and a 24 hour evaluation are provided. Also, on the Techdocs site is a webinar replay that teaches you how to use the performance analysis tools. BVIRPIT plus VEPSTATS BVIRPIT requests point-in-time statistics from a TS77, and VEPSTATS produces the reports. Point-in-time statistics cover the last 15 seconds of activity and give a snapshot of the current status of drives and volumes. The above tools are available at one of the following web sites: External users: Internal users: ftp://submit.boulder.ibm.com/download/tapetool/ Business Partners tape tools:

30 Page 3 Conclusions Conclusions The TS77 Virtualization Engine provides significant performance and increased capacity. Release 3. provides for larger cache storage capacity plus data encryption at rest. The TS77 architecture will continue to provide a base for product growth in both performance and functionality.

31 Page 31 Acknowledgements Acknowledgements The authors would like to thank Joseph Swingler, Jim Fisher, Ralph Beeston, Gary Anna, Eileen Maroney, and Luis Fernando Lopez Gonzalez for their review comments and insight, and also to Eileen Maroney and Jim Fisher for distributing the paper. Finally, the authors would like to thank Dennis Martinez, James F. Tucker, Vanessa Earle, Harold Koeppel, Douglas Clem, Michael Frick, Jim Lewis, and William Hengen for hardware/zos/network support.

32 Page 32 International Business Machines Corporation 213 IBM Systems Group 56 Cottle Road, San Jose, CA Printed in the United States of America 3-13 All Rights Reserved IBM, the IBM logo, System Storage, System z, zos, TotalStorage, Virtualization Engine, DFSMSdss, DFSMShsm, ESCON and FICON are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. Other company, product and service names may be trademarks or service marks of others. Product data has been reviewed for accuracy as of the date of initial publication. Product data is subject to change without notice. This information could include technical inaccuracies or typographical errors. IBM may make improvements and/or changes in the product(s) and/or programs(s) at any time without notice. Performance data for IBM and non-ibm products and services contained in this document were derived under specific operating and environmental conditions. The actual results obtained by any party implementing such products or services will depend on a large number of factors specific to such party s operating environment and may vary significantly. IBM makes no representation that these results can be expected or obtained in any implementation of any such products or services. References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services available in all countries in which IBM operates or does business. THE INFORMATION PROVIDED IN THIS DOCUMENT IS DISTRIBUTED AS IS WITHOUT ANY WARRANTY, EITHER EXPRESS OR IMPLIED. IBM EXPRESSLY DISCLAIMS ANY WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR INFRINGEMENT. IBM shall have no responsibility to update this information. IBM products are warranted according to the terms and conditions of the agreements (e.g., IBM Customer Agreement, Statement of Limited Warranty, International Program License Agreement, etc.) under which they are provided.