On the Accuracy and Scalability of Intensive I/O Workload Replay

Transcription

1 On the Accuracy and Scalability of Intensive I/O Workload Replay Alireza Haghdoost, Weiping He, Jerry Fredin *, David H.C. Du University of Minnesota, * NetApp Inc. Center for Research in Intelligent Storage

2 2 What is IO Workload Replay? Storage performance is heavily dependent on the workload Replay & Performance evaluation with real workloads o Does not require a real infrastructure to produce real workload o Does not expose a production systems to potential downtime risk I/O Trace Production IO Workload Storage Array I/O Trace Replay Tool Test Storage Array

3 3 What are the Challenges? Replay Intensive IO workload with High Accuracy is challenging o Intensive Workload? (Give me example please.) More than 100K IOPS Less than 2ms Response Time IO Workload for Enterprise Block Storage Arrays (SAN)

4 4 What are the Challenges? Replay Intensive IO workload with High Accuracy is challenging o Intensive Workload? o What Does High Accuracy Mean? 1. Replay on Similar Storage 2. Replay on Faster Storage

5 5 What Does High Accuracy Mean? Replay on Similar storage: Reproduce the workload with the same Throughput Response time IO Request Ordering

6 6 What Does High Accuracy Mean? Replay on Faster storage: Emulate original application behavior Replay with the same throughput and response-time that original app would do on the faster storage Throughput Response time

7 7 Existing Replay Approaches 1. Timestamp Based o Issue IO request in order based on their relative timestamp o Used to Replay on Similar Storage o Not capable to scale IO workload 2. As Fast As Possible o Issue IO request as fast as possible o Used to Replay on Faster storage o Not accurate, overlook computation and wait time o Ignore dependencies between IO requests 3. FileSystem or Application Trace Replay o IO dependency information is available o Does not scale well because of filesystem and application layers overhead

8 Intensity 8 Existing Replay Approaches 1. Timestamp Based (TS) 2. As Fast As Possible (AFAP) 3. FileSystem or Application Trace Replay (FS) AFAP TS Accuracy hfplayer FSTS

9 9 Agenda Motivation Replay on Similar Storage Replay on Faster Storage Evaluation

10 10 Unscaled Replay TS Replay Ideal IO Stack User-Space Kernel-Space TS Replay Linux IO Stack Storage Out-of-Order Unexpected Response Time

11 11 Sources of Workload Replay Uncertainty Forced preemption of system call threads User space to Kernel space context switch Limited queuing in the IO stack Lack of inflight IO control ftrace therapy

12 12 Sources of Workload Replay Uncertainty Forced preemption of system call threads User space to Kernel space context switch Limited queuing in the IO stack Lack of inflight IO control

13 13 Sources of Workload Replay Uncertainty Forced preemption of system call threads User space to Kernel space context switch Limited queuing in the IO stack Lack of inflight IO control

14 14 Sources of Workload Replay Uncertainty Forced preemption of system call threads User space to Kernel space context switch Limited queuing in the IO stack Lack of inflight IO control

15 15 Limited queuing in the Linux IO stack libaio Block Layer SCSI Layer io_getevents(2) 128 head 32 io_submit(2) tail To/From SAN Controller

16 16 Limited queuing in the Linux IO stack I am Full now libaio Block Layer SCSI Layer io_getevents(2) head io_submit(2) tail To/From SAN Controller Scheduling a Kernel Worker thread is expensive & take unpredictable time & Changes replayed IO ordering

17 17 Any Remedies? Block Layer queue size = SCSI Layer queue size = Max Queue Depth that HBA support

18 Response-Time 18 Any Side-effects? The more in-flight requests sitting in the queue: o Exponential increase of response time In-Flight IO Requests

19 In-Flight Requests 19 Any Remedies for Side-effects? Dynamic In-Flight I/O Control implemented in the replayer Calculate desired number of in-flight requests Throttle the replay speed by bounding the number of in-flight request at run time Maximum Desired Time

20 20 Agenda Motivation Replay on Similar Storage Replay on Faster Storage Evaluation

21 21 Inferring Potential Dependencies 1. Assume all consecutive IO requests are dependent 2. Identify independent requests using their timing relationship o Requests with overlapped timing (e.g. A and B) o Requests with short think time (e.g. B and C) 3. Remove redundant dependencies

22 22 Replay on Scaled Storage Example: o IO request B is dependent to A A B t 0 t 1 t 2 t 3 Production Workload Think time A Response time A Think time B Response time B Workload on Faster Storage Think time A Response time A Think time B Response time B

23 23 Replay on Scaled Storage Emulates production workload on scaled storage preserve think time and dynamically adjust issue time t 0 t 1 t 2 t 3 Production Workload Think time A Response time A Think time B Response time B Workload on Faster Storage Think time A Response time A Think time B Response time B Dependency Replay Think time A Response time A Think time B Response time B t c t c + (t 3 -t 2 )

24 24 Agenda Motivation Replay on Similar Storage Replay on Faster Storage Evaluation

25 25 Evaluation Methodology on Unscaled Storage I/O Trace IO Intensive Workload I/O Trace hfplayer Fast Storage Array SSD SSD SSD SSD Fast Storage Array SSD SSD SSD SSD

26 26 Evaluation Methodology on Unscaled Storage I/O Trace IO Intensive Workload I/O Trace hfplayer btreplay Fast Storage Array SSD SSD SSD SSD Fast Storage Array SSD SSD SSD SSD

27 27 Evaluation Methodology on Unscaled Storage I/O Trace IO Intensive Workload I/O Trace hfplayer btreplay blkreplay Fast Storage Array SSD SSD SSD SSD Fast Storage Array SSD SSD SSD SSD

28 Relative Exec. Time Error (%) Relative Avg. Latency Error (%) Reordered IO Ratio (%) 1.9% 0.2% 0.1% 1.2% 1.0% 0.1% % 0.7% 0.7% 1.3% 0.6% 0.5% 0.5% 0.6% 1.0% hfplayer btreplay blkreplay 20% Execution Time Error (%) 15% 10% 5% 0% 800% Avg Latency Error (%) 600% 400% 200% 0% Reorder IO (%) 40% 20% 0% 40K-SeqW 68K-SeqW 92K-SeqW 99K-SeqW 104K-SeqW 40K 68K 92K 99K 104K Sequential Write IOPS

29 29 Evaluation Methodology on Scaled Storage Benchmark I/O Trace Source Storage Array HDD HDD HDD HDD Target hfplayer Benchmark Fast Storage Array SSD SSD SSD SSD Compare Fast Storage Array SSD SSD SSD SSD

30 ms ms ms Application Running on SSD Array (Target) AFAP Replay Source Trace on Target hfplayer Replay Source Trace on Target Normalized IOPS Normalized Avg Response Time CopyFiles CreateFiles mkfs CopyFiles CreateFiles mkfs

31 31 Conclusions Provide detailed analysis of replay uncertainty sources Introduced new methods to infer IO decencies from block IO trace Introduced new replay methods with more accuracy on both scaled and unscaled storage hfplayer is open source and available in GitHub

32 Questions Alireza Haghdoost, Center for Research in Intelligent Storage

33 33 IO Workload Trace o Sample Block IO Trace 5104, , ,READ,30, , ,6,1,0,264,8, ,7,6, , , ,READ,31, , ,6,1,0,272,8, ,7,6, , , ,READ,32, , ,6,1,0,280,8, ,7,6, , , ,READ,33, , ,6,1,0,288,8, ,7,6, , , ,WRITE,0, , ,7,1,0,296,8, ,7,6, , , ,WRITE,1, , ,7,1,0,304,8, ,7,6, , , ,WRITE,2, , ,7,1,0,312,8, ,7,6, , , ,WRTIE,3, , ,7,1,0,320,8, ,7,6,2.91 Dependent or Independent? That is the question. (Fake) Hamlet

34 34 hfplayer Internal Architecture Dependency analyzer module Worker threads issue IO requests o o According to their scheduled timestamp in unscaled replay mode After all of its parents plus their corresponding think time in scaled replay mode Dependency Analyzer Harvester Thread hfplayer Prepare I/O and Monitoring Timer Thread Worker Threads