Recovering Disk Storage Metrics from low level Trace events

Transcription

1 Recovering Disk Storage Metrics from low level Trace events Progress Report Meeting May 05, 2016 Houssem Daoud Michel Dagenais École Polytechnique de Montréal Laboratoire DORSAL

2 Agenda Introduction and objectives Architecture Required Instrumentation Data Model Storage Performance Metrics Visualization Cost of the Analysis Use Cases

3 Introduction Storage performance investigation : Benchmarking Tracing Benchmarking limitations: Workloads are not representative Results can be affected by many factors (caches, etc...) It doesn't help in finding the origin of the problem

4 Introduction Tracing offers a more accurate insight into the internals of the storage subsystem. It can be used to analyze the behavior of real workloads and to detect performance bottlenecks. But Tracing overhead can affect the normal behavior of the system. The amount of data generated by tracing is huge and needs to be post processed.

5 Introduction Objectives Tracing the storage subsystem with a minimal overhead Extract performance metrics from the collected low level events Create a visualization system to show the metrics

6 Architecture

7 Required Instrumentation Tracepoint Description block_getrq Create the request data structure block_rq_insert An I/O request is inserted into the disk waiting queue block_rq_issue The request goes from the waiting queue to the dispatch queue, where it starts to be processed block_rq_complete The disk drive finished the processing of the request block_bio_backmerge block_bio_frontmerge A new BIO is added to an existing request block_sleeprq The request is not inserted because the waiting queue is full addons_elv_merge_requests Two I/O requests in the waiting queue are merged together lttng_statedump_block_device This event is generated at the beginning of the tracing session and provides a mapping between block devices' names and IDs

8 Data Model I/O Request Life Cycle

9 Data Model Stateful Analysis: the state of the system is kept in a historical database built incrementally in a single pass over the trace Required properties: the data structure should be able to store the data as an association between a state and a time range The data structure should be a tree like data structure The Modeled State System

10 Storage Performance Metrics Disk Utilization Utilization is the fraction of time the disk was busy during an observation time. B U= T where U is the utilization ratio, B is the amount of time the disk was busy during T, the total observation time. A disk is considered busy if at least one IO request is being processed in the dispatch queue.

11 Storage Performance Metrics Disk Utilization (t 1 t 0 )+(t 3 t 2)+(t 5 t 4 ) U= T end T start The general formula is : U = (t e t s ) i i i T end T start t s : time of the transition Idle Busy t e : time of the transition Busy Idle

12 Storage Performance Metrics Latency Latency = Preparation Time +Waiting Time + Service Time

13 Storage Performance Metrics Seeks per second Seek time is the time that the disk drive takes to locate the target sector. sector i+1 sector i Seeks second= i T end T start

14 Storage Performance Metrics Throughput size(ri ) Throughput= i T end T start

15 Storage Performance Metrics Queue length /sys/block/sdx/queue/nr_requests : waiting queue size /sys/block/sdx/device/queue_depth : dispatch queue size When the waiting queue is full and a process inserts a new request, the event block_sleeprq is generated

16 Visualization

17 Visualization It is difficult to show the timeline of each request and the length of the queue one view GPUView We decided to create two separate views

18 Cost of the Analysis Test setup : Benchmarking is done using Sysbench To reduce the effect of cache, we created a RamFS that uses most of the available RAM. Only 500MB are left to the system. Workload size is 12GB ( 24 times bigger than the free memory) We ran the tests on an Intel i with 32 GB of memory and an SSD drive Workloads are: (1) sequential read, (2) sequential write, (3) random read, (4) random write, and (5) random read/write Each workload is run with: (a) no tracing, (b) tracing to the same disk, and (c) tracing to an external disk

19 Cost of the Analysis Sequential Read Overhead NoTracing External Internal Overhead (%) Throughput (MB/s) Sequential Sequential ReadRead 4 External Internal Buffer size (KB) Buffer size (KB) Sequential Write Sequential Write Sequential Write Overhead NoTracing External Internal Overhead (%) Throughput (MB/s) External Internal Buffer size (KB) Buffer size (KB) 2048

20 Cost of the Analysis Random Read NoTracing External Internal Overhead (%) Throughput (MB/s) External Internal Buffer Size (KB) Buffer Size (KB) NoTracing External Internal Overhead (%) Throughput (MB/s) Random Write 4 External Internal Buffer size (KB) Buffer size (KB) 2048

21 Cost of the Analysis Random Read/Write NoTracing External Internal Overhead (%) Throughput (MB/S) External Internal Buffer size (KB) Buffer Size (KB) Observations The overhead is bigger for random workloads, compared to sequential workloads The maximum overhead is 6% for internal tracing, and 1% for external tracing. The overhead decreases when the buffer size increases

22 Use Cases Synchronous/Asynchronous file copying Asynchronous File Copying followed by a sync Synchronous File Copying

23 Use Cases Disk Write back Cache Writing data to a persistent disk storage device is a slow operation. To speed up disk writes, almost all modern disk drives offer a fast volatile memory that behaves as a write back cache.

24 Use Cases Request Priority ionice In this experiment, Thread 1 and Thread 2 are competing for the same disk. We can easily see the impact of ionice on scheduling behavior. Requests inserted by the thread with the highest priority are processed before the other requests. Thread 1 Thread 2

25 Thank You!