RECORD LEVEL CACHING: THEORY AND PRACTICE 1

Transcription

1 RECORD LEVEL CACHING: THEORY AND PRACTICE 1 Dr. H. Pat Artis Performance Associates, Inc Spyglass Lane Palm Desert, CA (760) drpat@perfassoc.com Abstract: In this paper, we will examine experimental data collected for a responding to an I/O workload comprised of random read/write pairs. Read/write pair workloads are typical of business critical online transaction processing (OLTP) workloads such as credit card or ATM applications. The experimental data will be used as a basis to examine past theoretical explanations as well as to comment on the efficacy of IBM s new record level cache one (RLC I) facility. 1 Introduction Since the introduction of the cache control unit in 1981, users have enjoyed the benefits of substantially reduced device service times for read oriented workloads which demonstrate a high locality of reference. Briefly, the term locality of reference is used to describe the tendency of a sequence of I/O operations to make multiple references to one or just a few tracks over a brief period of time. To exploit this behavior, cache control units incorporate a memory resource to store recently referenced tracks. When a read-miss occurs, the cache control unit loads more than just the record which caused the read-miss in an effort to reduce the number of physical I/O operations associated with future read requests. Typically, the requested record and the remainder of the records on the track are staged into cache. However, a variety of other algorithms may also be employed to optimize the quantity of data staged in the event of a cache-miss or prestaged in anticipation of future I/O requests. These algorithms are discussed in Section 3. Read requests which are serviced from the cache, i.e. avoid a reference to the disk, are called read-hits. While the increased the number of workloads for which read caching was effective, caching writes presented data integrity issues. Specifically, when a control unit caches a write operation, it must accept the data integrity responsibility for the write until the data is actually written to disk. While the performance benefits of write caching were seductive, the microcode of the 3880-x3 control units only supported read caching. To ensure data integrity and address write performance, the incorporated nonvolatile storage (NVS) as well as traditional cache. When a write-hit or format write is cached, data is stored in both NVS and normal cache until it is written to disk. Hence, a failure of either resource during this brief interval (prior to destage) cannot result in the loss of data. To protect the data from environmental factors (i.e., power failures), a battery backup system was incorporated in the to maintain the data in NVS for 48 hours. Similar control unit implementations have been introduced by Amdahl, Hitachi, and StorageTek. Unfortunately, some of the most critical workload types were still resistant to performance improvement by caching. Specifically, online transaction processing (OLTP) workloads that have very low read-hit fractions and high write percentages often run worse when cached even with traditional /6 class control units. In March of 1994, IBM introduced record level caching one (RLC I) to extend the benefits of caching to OLTP workloads. In this paper we will examine the performance characteristics of record level caching in comparison with normal and bypass cache modes of cache operation. 2 Caching Issues for OLTP Workloads To understand the requirement for record level caching, it is useful to create a mental image of the type of I/O workload that present the greatest problems to traditional caching schemes. Consider the case of a bank with an online system that supports thousands of ATMs. Customers randomly arrive at the different ATM terminals and enter their account number by swiping their cash card through a reader and then entering their PIN number to validate the transaction. If you think about it for a minute, it is clear that the read for your account number should be a read-miss since your cash card has been in your pocket (i.e., not in use for a long time) and there are only a few other customer accounts on the same disk track as yours out of the hundreds of thousands of accounts the bank services Performance Associates, Inc., all rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior written consent of the copyright owner. Specific permission is granted to the Computer Measurement Group to publish this paper in their 1995 Winter Transactions.

2 Normal DEF EXT NOR READ /6 Stage to End of Track Front End Sequential DEF EXT SEQ READ n Pass Record to Channel /6 Full Cylinder Stage Read Record 4 Pass Record to Channel stage 3 to 15 tracks RLC I DEF EXT RLC READ Potentially Stage the Record Pass Record to Channel Read Record 5 Figure 1. Define Extent Options On a read-miss, the traditional mode of operation (normal caching) is to stage the record and the remainder of the track into memory. If we assume a 3390 track architecture and a 4K record (12 records per track), on average you will request the 6th or 7th record on a track and then stage the requested record along with the remaining records on the track in anticipation of future I/O operations. Unfortunately, in a random environment like the ATM workload we have discussed, there is little or no probability that the other records on the track ever be referenced in the near term. Hence, the control unit stages six and one-half times more data into cache than actually is required to service your request. As a result of this excess staging, the back-end data paths of the control unit can become saturated, resulting in degraded performance. However, the read provides some benefit when the write half of your transaction occurs. Specifically, when the transaction debits your account at the end of the transaction to reflect your new balance. On a class controller, this write will be a write-hit only if the underlying track is still in cache when the write occurs so that the track format can be verified without reading the track layout again from the disk. Unfortunately from the perspective of traditional cached controllers, some data bases employ large buffers and deferred write back schemes to improve performance. In the case of deferred write back, if the delay before the write exceeds the residency time [1] of the cache control unit, the write half of the read/write pair will also be a write-miss since the underlying track will have aged out of the cache by the time the write occurs. [2] Since a class control unit normally stages the remainder of the track on a write-miss too, the back-end data paths become even more saturated with data traffic that is unlikely to be of benefit to any future I/O operations.

3 3 Define Extent Options When an I/O request is issued by the operating system, the channel program is preceded by a define extent and locate record. [3] The define extent provides the control unit with advanced information about the probable access pattern of future I/Os in the stream. Figure 1 provides an overview of normal, sequential, and record level cache one (RLC I) define extents. For most operations, the define extent specifies normal cache replacement. In the top of the figure, a read is issued for record 4 of a track that contains 12 records. In the case shown in the figure, record 4 is passed to the channel and records 4 through 12 are staged into the cache. In the event that one or more of these records are later accessed (i.e., before the track leaves on a least recently used (LRU) basis), then the resulting read-hits will improve performance. In the same fashion, if any of the records are written, the writes will also be write-hits contributing to a decrease in overall response time. As we have already noted, if none of the other records on the track are ever accessed, then the back-end control unit resources employed to stage the track are wasted. The second example in the figure denotes the sequential mode of operation. Clearly, normal cache replacement would be unsuitable for sequential access patterns since most sequential files are blocked half track. That is, a read-miss would occur for the first record on every track followed by a read-hit for the second record, yielding a 50% read hit ratio which is far from ideal. Moreover, reading a large file sequentially would tend to flush all of the other data from the cache since normal cache replacement tracks age from the cache on an LRU basis. Hence, when define extent specifies sequential, the control unit pre-stages from 3 to 15 tracks in anticipation of future read requests. Unlike LRU schemes where previously used tracks age out of cache, the track images in cache are immediately freed (and reused) after they are read. In the ideal case, a job might be able to read an entire file from cache after the first read-miss started the prestage process. The third example in the figure denotes the record level cache one (RLC I) mode of operation. To qualify for IBM s record level cache one, a data set must meet one of two criteria. They are: 1) the track must have a regular data format to be eligible for write caching without verification of the underlying track format, or 2) the RLC I function must be enabled in the define extent by the DCME function of DFSMS/MVS for record level caching of reads. [4] The term regular data format is used to define data sets that have a fixed block size and a constant number of records per track. This condition is imperative since a RLC I data set need not verify the track format via a read operation before accepting a write. To meet the second criteria, the algorithms in DCME maintain a dynamic access history by job and data set to identify record level cache candidates. With RLC I, the control unit responds to a read-miss by staging at most, only the requested record. 2 As shown in the lower third of the figure, the read request for record 5 does not result in the staging of records 6 through 12 on the track as would have been the case with normal cache replacement. In addition, if record 5 is later rewritten, the write will be a write-hit regardless of whether or not the underlying track has aged out of the cache (or was ever present) since the track has a regular data format. Simply stated, when RLC I is enabled for a write operation, a write-miss cannot occur since the track layout need not be verified before the write is accepted by the cache control unit. Hence, RLC I conceptually addresses the problems of excess control unit back-end utilization as well as deferred write backs for OLTP workloads. 4 Theoretical Explanations Historically, there have been theoretical explanations of the performance problems encountered by OLTP workloads employing normal cache replacement. They are subsequent I/O delay and excess back-end utilization. The subsequent I/O delay explanation is similar to the concept of stepping on your own feet. That is, the application issues a request for another record on the same track while the control unit is still staging the remainder of the track for the prior I/O request. Under this explanation, the subsequent I/O request is PENDed by the control unit since the logical device is still busy staging data. For this explanation to be true, the interarrival time of I/Os to the device would have to be less than one-half of the revolution time (an average stage) plus the service time for the device. The second explanation is back-end utilization. That is, the excess utilization of the control unit s back-end data paths staging records which will never be used results in delays for all of the other I/O requests being processed by the control unit. Unlike the subsequent I/O delay explanation, delays are a function of the aggregate arrival rate to the control unit rather than the interarrival rate of any specific device. Both of these potential theoretical explanations will be investigated in the experiment described in the following section. 2 When DCME detects that the read-hit ratio for a data set being accessed by a job is lower than 10%, not even the record is staged into cache. Note, not even the record need to be in cache for writes to be treated as write-hits for a RLC I managed data set.

4 5 Experimental Design A simple experiment was designed to test the effectiveness of RLC I as well as to investigate which of the two theoretical explanations best describe the behavior observed during the experiment. The experiment was conducted using a tool developed by the author called the PAI/O Driver which was designed to evaluate the performance of DASD subsystems. The experiments were conducted on an enhanced at engineering change level C The control unit was configured with one gigabyte of cache, 32 megabytes of NVS, and sixty-four devices. A 401 cylinder data set, formatted with 4K blocks, was allocated on each device. Hence, the total data active behind the control unit was approximately 20 gigabytes. The same reference pattern was employed for each data set. Specifically, a read operation was issued to randomly selected block on a randomly selected cylinder, i.e., a read-miss. The program then issued a write to the same block, i.e., a write-hit. After the write half of the read/write pair was completed, the PAI/O Driver selected another random cylinder for the subsequent read/write pair. The average seek distance between the read/write pairs was five cylinders. Using the timing facilities of the PAI/O Driver, the arrival rate of each device in the DASD subsystem was ramped from 4 to 16 I/Os per second in steps of two. For sixty-four devices, this corresponds to a target subsystem rate ranging from 256 to 1,024 I/Os per second. Each of the experimental rates was synchronized with the system s RMF measurement interval so that the experiment results could easily be evaluated using data from RMF and the Cache RMF Reporter. The test series was repeated three times with the define extents set to bypass cache, normal cache replacement, and record level caching to evaluate the relative performance of each option. To ensure that the outcomes were not a result of some other variable, initial cache conditions, or random chance, the entire series of tests were repeated several times. 6 Results Table 1 provides an overview of the test results comparing the relative benefits of normal cache replacement, record level caching, and bypass cache for read/write pairs where 100% of the reads are probable misses. 3 The first column of the table provides the subsystem I/O rate and the remaining three columns of the table provide the average subsystem response time in milliseconds for the define extent options. Blanks in the table indicate that the subsystem could not sustain the desired I/O rate with the specified define extent option. Subsystem Normal Cache Record Level Bypass I/O Rate Replacement Cache Cache Table 1. Average Subsystem Response Time (Milliseconds) The second column of the table provides the results for normal cache replacement. While the results at low subsystem I/O rates (i.e., less than 250 I/Os per second) were superior to the bypass cache results shown in the fourth column, the average cache response time was actually worse than bypass cache at higher arrival rates. Moreover, the subsystem began to saturate at approximately 495 I/Os per second as can be seen in Figure 2. Although the subsystem was able to sustain slightly higher I/O rates, the response time curve for normal cache replacement demonstrates a classic saturation profile. The third column of the table provides the results for record level caching. As can be seen in the table, record level caching is superior to both normal cache replacement and bypass cache for I/O rates of up to 618 4K blocks a second. It is important for the reader to note that this I/O rate is comprised of 314 reads and 314 writes. Since the back-end path utilization was significantly lower because only 4K blocks rather than half-tracks were being transferred, back-end utilization obviously was not the cause of the subsystem saturation. Investigation of the RMF Cache Reporter data revealed that the number of fast write delays substantially increased when the aggregate NVS destage rate exceeded 1 megabyte per second. At the 618 I/Os per second saturation point, the aggregate NVS destage rate was 1.2 megabytes per second. Repeated experiments revealed this as a maximum limit for the configuration being tested. In fact, after this limit was reached, the aggregate sustainable I/O rate for the controller significantly decreased as is depicted in Figure 2. 3 The reader may wish to note that it is physically impossible to achieve a zero read-hit percentage due to the residual data content of the cache controller. In our experiments, the data sets were roughly twenty times larger than the control unit s one gigabyte cache. Hence, it is reasonable to expect approximately a 5% gratuitous read-hit rate simply based on the ratio of the size of the cache to the aggregate size of the data sets.

5 Response Msec IBM Record Level Cache 4K Blocks - 100% R/W - 100% Read Miss SSCH Rate Normal Cache Replacement Record Level Cache Bypass Cache IBM GB/32MB CYLs 4 17MB ESCON Channels - LV C s 25 June 1994 Figure 2. Average Subsystem Response Time (Milliseconds) The final column of the table provides the results for bypass cache. While the bypass cache results were inferior to both normal cache replacement and record level caching at rates less than 250 I/Os per second, bypass is superior to normal cache replacement at higher rates. Bypass cache provides the highest aggregate throughput for the due to the NVS destage bottleneck discussed in the prior paragraph. 7 Observations Based on the experimental data collected during this study, it is clear that back-end utilization is the proper explanation of the poor response time which was characteristic of OLTP workloads that employed normal cache replacement. This conclusion is based on the fact that saturation occurred at a rate of approximately 8 I/Os per device per second. Since this corresponds to an interarrival time of approximately 125 milliseconds between subsequent I/Os at a response time of 39.2 milliseconds, we must reject the assertion that the delay is a result of a subsequent I/O encountering the completion of the stage event for a prior I/O. The reader should also note that the results presented in this paper are ideal results for the DASD subsystem tested since the I/Os were uniformly distributed. If the I/Os had not been uniformly distributed, the maximum results would have likely been defined by the saturation of a few devices at a far lower aggregate arrival rate. In addition, microcode updates which have been made since the tests were conducted in late June of 1994 or changes which may be made in the future could result in performance improvements for the same engineering test series. Hence, these results should be viewed as a performance observation at a point in time rather than as an absolute prediction of any future characteristics of RLC I or the From a broader perspective, one fact is abundantly clear. IBM s record level cache one (RLC I) works as advertised and extends the benefits of caching to OLTP workloads that have been historically resistant to caching with IBM s prior cache control unit offerings.

6 References [1] McNutt, B. & J.W. Murray, A Multiple Workload Approach to Cache Planning, Proceedings of CMG 87. [2] Houtekamer, G.E. & Artis, H.P., MVS I/O Subsystems: Configuration Management and Performance Analysis, McGraw Hill, [3] Fairchild, W., The Anatomy of a Thoroughly Modern I/O, Proceedings of CMG 90, [4] Berger, J. A., DFSMS: Dynamic Cache Management Enhancement, CMG Spring Transactions, [5] Artis, H.P., DASD Subsystems: Evaluating the Performance Envelope, CMG Spring Transactions, 1994.