Characterizing Data-Intensive Workloads on Modern Disk Arrays

Transcription

1 Characterizing Data-Intensive Workloads on Modern Disk Arrays Guillermo Alvarez, Kimberly Keeton, Erik Riedel, and Mustafa Uysal Labs Storage Systems Program 4 th Workshop on Computer Architecture Evaluation using Commercial Workloads January 21, 2001 CAECW-01, 0 Motivation New features for midrange and high-end disk arrays have significant impact on performance: Large caches (256MBx2 in HP FC-60, 64GB in HP XP512) LUN-level prefetching Stream detection (for prefetching and caching) Degraded operation and on-line reconstruction Our understanding of how these array features perform for complex workloads is insufficient Most workload characterizations don t have attributes to capture the effectiveness of these features Goals: Quantify performance effects of array features Identify workload behaviors that leverage these features CAECW-01, 1 1

2 Potential benefits Understanding how a real array works Quantifying the (relative) influence of each array feature on overall performance Array architecture: Which configuration knobs and monitoring points are the right ones? Improving the fidelity of our workload specifications Better recreation of workloads (see Kurmas, et al.) Better prediction of workload performance Laying the groundwork for more realistic array models Behavioral models require detailed understanding Most work has focused on very simple (stochastic) workloads CAECW-01, 2 Outline Motivation Description of our approach Experimental infrastructure Prefetching investigation Conclusions and ongoing/future work CAECW-01, 3 2

3 Approach: find and quantify sweet spots Identify array features ( sweet spots ) that impact performance Identify workload behaviors for which the features may work especially well/badly Run synthetic experiments that exhibit one such behavior at a time Establish limitations of each sweet spot Quantify performance (latency) impact of each sweet spot Propose workload metrics that capture each behavior Replay traces from real applications Measure new workload metrics on each trace Explain trace s performance as a function of metric CAECW-01, 4 Experimental infrastructure: HP FC-60 disk array Characteristic Cache size Disk enclosures Disks per enclosure Disk capacity Total capacity Our configuration 256 MB x GB 0.5 TB (raw) Max configuration 512 MB x GB 4.4 TB (raw) Two controllers One fibre channel port/controller Each connected to all enclosures Six ultra-wide SCSI interfaces to controllers (40 MB/s each) Cache: distributed across controllers Write-back policy, with writes mirrored across controllers Top observed performance per controller: 84 MB/s, 5400 IO/s CAECW-01, 5 3

4 Identify array sweet spots Prefetching Behaviors How much useless read-ahead work? Holes in sequential runs? Interference between streams? Array control knobs Minimum prefetch Prefetch multiplier Disable length Caching Burstiness Stream detection Degraded mode CAECW-01, 6 Identify workload behaviors Issue: Prefetching may still be effective in face of non-strictly sequential workload behavior Additional workload behaviors to consider: Jumps/holes in sequential runs Jump distance = gap between next address and expected next address if accesses are sequential Interleaved accesses from different streams Interference distance = number of requests from other streams that occur between successive requests from this (sequential) stream CAECW-01, 7 4

5 Synthetic experiments to focus on behaviors Synthetic workloads to isolate behaviors: Holes in sequential runs Interleaved streams Evaluation methodology: Compare workload performance with no forced array prefetching vs. forced minimum array prefetching MinPrefetch = 0, 64K (also 16K, 32K, 48K) Performance = array service time no device driver queueing CAECW-01, 8 Workload metrics and real applications Workload metrics Jump distance and interference distance Replay traces from real workloads: 300-GB TPC-D: table, index, temp, log, summary Open Mail server Cello file server traces: root, news, home, ssp Start with single-lun experiments Pick a representative array, and a representative LUN CAECW-01, 9 5

6 Prefetching investigation Exploration of jump distance How does it impact array service time? How big are the jumps in these workloads? Exploration of interference distance How does stream interference impact service time? How much stream interference occurs in workloads? Proposal for new sequentiality workload metric CAECW-01, 10 Impact of jumps on service time? Average Service Tim e (s) R5_4 disk, 4 KB, 50 req/sec, minprefetch=0 1.E+00 1.E+01 1.E+02 1.E+03 1.E+04 1.E+05 1.E+06 1.E+07 Jum p Distance (KB) RunLength=2 x 4K RunLength= 4 x 4K RunLength=16 x 4K RunLength=64 x 4K RunLength=256 x 4K Large runs have consistently low service time Short runs with small jumps (< 64KB) have low service time Disk track buffers? CAECW-01, 11 6

7 Prefetching success for spatial jumps? Average Service Time (s) E+00 1.E+01 1.E+02 1.E+03 1.E+04 1.E+05 1.E+06 1.E+07 Jump Distance (KB) RunLength= 4 x 4K, RunLength=256 x 4K, RunLength=4 x 4K, RunLength=256 x 4K, Prefetching successful for: both long and short runs with small jumps (< ~1MB) CAECW-01, 12 Fraction of accesses How big are workload spatial jumps? E+00 1.E+06 2.E+06 3.E+06 4.E+06 5.E+06 Jump Distance (KB) cello-home cello-news cello-root cello-ssp open-mail tpc-d Cello-ssp and cello-root are mostly sequential Other workloads have short and long jumps forwards (and backwards) CAECW-01, 13 7

8 Impact of interfering streams on service time? Request Rate = 5 req/sec, Average Service Time (s) Number of Streams For small number of streams (<= 20), run length impacts service time Run Length=4 x 4K Run Length=256 x 4K CAECW-01, 14 Prefetching success for interfering streams? Average Service Time (s) Request Rate=5 req/sec, R5_4disk Interference Distance RunLength=4 x 4K, RunLength=256 x 4K, RunLength=4 x 4K, RunLength=256 x 4K, Workload metric to approx. no. of streams: interference distance Prefetching helps for low interference distance (< 50) & large runs Prefetching hurts for all other cases CAECW-01, 15 8

9 Prefetching success for interfering streams (2)? Average Service Time (s) Prefetching helps for low interference distance and large runs true for TPC-D, cello-ssp (sequential accesses) Prefetching hurts for all other cases true for OpenMail, cello-home; not for cello-{root,news} (bursty) CAECW-01, 16 Request Rate=5 req/sec, R5_4disk Interference Distance RunLength=4 x 4K, RunLength=256 x 4K, RunLength=4 x 4K, RunLength=256 x 4K, cello-root, cello-root, cello-home, cello-home, cello-new s, cello-new s, open-mail, tpc-d, cello-ssp, cello-ssp, open-mail, tpc-d, Average Service Time (s) Prefetching success for interfering streams (3)? Interference Distance Focus: constant aggregate request rate Service time increases as # streams increases Prefetching effective for long runs & low aggregate req rates, fewer streams at higher aggregate req rates CAECW-01, 17 R5_4disk ReqRate=500, RunLength=4 x 4K, ReqRate=100, RunLength=4 x 4K, ReqRate=500, RunLength=256 x 4K, ReqRate=100, RunLength=256 x 4K, ReqRate=500, RunLength=4 x 4K, ReqRate=500, RunLength=256 x 4K, ReqRate=100, RunLength=4 x 4K, ReqRate=100, RunLength=256 x 4K, 9

10 Summary of results Array prefetching effective for: Longer runs Small number of streams (low interference distance) For shorter sequential runs, smaller jumps between runs Simple run count spatial locality metrics too restrictive to capture prefetch-friendly behavior New workload characteristics to be captured: Short jumps within a stream Interleaved accesses from other streams Proposal for new run count metric to incorporate these characteristics CAECW-01, 18 Preliminary proposal for sequentiality metric Average Service Time (s) Sequential Run Count (KB) cello-home, cello-home, cello-news, cello-news, cello-root, cello-root, cello-ssp, cello-ssp, open-mail, open-mail, tpc-d, tpc-d, New-and-improved run count: size of run (in KB) where: Short forward jumps permitted Interleaved requests from other streams permitted Does new metric predict performance? Prefetching beneficial for large run counts (TPC-D, cello-ssp) Prefetching ineffective for small run counts (OpenMail) Inconclusive for bursty workloads (cello-{home,news,root}) CAECW-01, 19 10

11 Related work How to present requests to devices, given a workload prefetching and caching (main software differentiator) informed [Tomkins 97], [Kimbrel 96] uninformed [Chervenak 99] Studies of array performance characteristics Few on the real thing: [Chen 90],[ChervenakKatz91] Conseqs of data distribution: layouts [LeeKatz93],[Alvarez 98] How arrays can take advantage of application characteristics Autoraid [Wilkes 96] Setting storage system params:[chenlee95],[chenpatterson90],[shenoyvin97],[jacob96] Commercial products: [EMC s SRDF], [XP s moral eqv] Degraded mode performance studies [HollandGibson93], [ThomasianMenon94] Analytical modeling approaches [Shriver 98],[MuntzLui90] CAECW-01, 20 Conclusions and future work Array features impact performance Developed methodology for evaluating sweet spots Identified several potential sweet spots and behaviors Used synthetic workloads to isolate workload behaviors Measured proposed metrics on traces replayed from real workloads Preliminary results for prefetching behavior for complex workloads New sequentiality metric proposed to capture workload behaviors amenable to prefetching Ongoing/future work Quantify effects of cache capacity and interference Understand consequences of burstiness from synthetic experiments CAECW-01, 21 11