Cut Me Some Slack : Latency-Aware Live Migration for Databases. Sean Barker, Yun Chi, Hyun Jin Moon, Hakan Hacigumus, and Prashant Shenoy

Transcription

1 Cut Me Some Slack : Latency-Aware Live Migration for s Sean Barker, Yun Chi, Hyun Jin Moon, Hakan Hacigumus, and Prashant Shenoy University of Massachusetts Amherst NEC Laboratories America Department of Computer Science

2 Workload 1 Cloud Systems! Multitenant database servers Multiple resident DBs per physical server Independent workloads! Level Agreement (SLA) Minimum agreed performance E.g., latency, throughput Penalty for violations Tenant 1 Tenant 3 Workload 2 Tenant 2 Tenant 4! Tenant packing (tenants per server) Maximize utilization (more packing) Minimize SLA violations (less packing) Tradeoff! Workload 3 Workload 4 2

3 Dynamic Workloads! Tenants and workloads are dynamic Tenants added or deleted Time of day variations Flash crowds! Potential for overload Request Rate Time SLA ok SLA ok SLA Violations resource usage Idle Tenant 3 Tenant 2 Idle Tenant 3 Tenant 2 Tenant 3 Tenant 2 Tenant 1 Tenant 1 Tenant 1 (a) stable (b) workload change (c) overload 3

4 Workload 5 Workload 1 Migration! Respond to potential overload E.g., new tenant, increasing load! Migrate tenants to other servers Alleviate source hotspot, increase target utilization Workload 6 Workload 2 Server A Tenant 1 Tenant 3 Tenant 2 Tenant 5 Tenant 4 Tenant 6 Server B Workload 4 Workload 3 4

5 Workload 5 Workload 1 Problem 1: Downtime! Tenant downtime during migration Workload 6 Workload 2 Server A Tenant 1 Tenant 3 Tenant 2 Tenant 5 Tenant 4 Tenant 6 Server B Workload 4 Workload 3! Solution: Live migration! Virtual machine level [Clark-NSDI 5]! DB level [Elmore-SIGMOD 11, Das-PVLDB 11] 5

6 Problem 2: Interference! Migration itself is a load! Source server may be heavily loaded already Migration may make situation temporarily worse Almost overloaded, migrate! resource usage SLA ok Idle Tenant 3 Tenant 2 Tenant 1 (a) before migration SLA Violations Tenant 3 Migration Tenant 3 Tenant 2 Tenant 1 (b) during migration Violations due to migration!! Goal: Migrate with minimal tenant interference 6

7 Outline! Motivation! Study of migration interference! Interference-aware live migration! Prototype evaluation! Conclusions 7

8 Case Study: Migration Interference! Does migration interference matter?! Scenario: heavily loaded source Tenant 1 servicing workload Tenant 2 migrating to target! Impact on non-migrating tenant Unfair interference! Migration workload measure Migration rate (MB/sec)! Interference measure Transaction latency (ms) Workload Server A DB 1 DB 2 DB 2 Server B 8

9 Impact on Neighboring Tenant Txn Latency (ms) Baseline Time Series Baseline Average (79 ms) No migration, base workload M Throttle Time Series 4M Throttle Average (153 ms) 4 MB/s migration, modest impact Txn Latency (ms) Time (18 seconds in total) 12M Throttle Time Series 12M Throttle Average (72 ms) 12 MB/s migration, major impact Time (281 seconds in total) 16M Throttle Time Series 16M Throttle Ave (2254 ms) 16 MB/s migration, over capacity Time (13 seconds in total) Time (95 seconds in total)! Limited capacity to be used for migration! 9

10 Migration Slack! Migration slack Resources usable towards migration Migration rate as resource currency! Slack determined by SLA Migration speed vs. latency Acceptable latency, variance Txn Latency (ms) Average Latency Migration Duration Duration (s) Migration Speed (MB/sec) 1

11 Slack-Aware Live Migration! Consume exactly all slack for migration Too fast poor performance or server overload Too slow migration takes too long! But workloads are dynamic So slack is dynamic too! Slow down migration!! Approach: adaptive dynamic throttle Heavy workload slow down migration Lighter workload speed up migration Request Rate Time Slack 11

12 Migration Throttle Control (1)! Proportional-integral-derivative (PID) controller Well-known feedback loop in control systems Current error PID action new error PID... Past error P K p e(t) Current error Setpoint Σ Error I K i t e(τ)dτ Σ Output Future error D K d de(t) dt Process Variable 12

13 Migration Throttle Control (1)! Proportional-integral-derivative (PID) controller Well-known feedback loop in control systems Current error PID action new error PID... P K p e(t) Setpoint Σ Error PID Controller I K i t e(τ)dτ Σ Output D K d de(t) dt Process Variable Migration speed 12

14 Migration Throttle Control (2)! Process variable: What should we try to control? Measure of interference, SLA-based: recent query latency! Setpoint: Where do we want latency to be? Goals: low latency and high slack utilization Transaction Latency Setpoint Latency Under Slack Over Slack At Slack Server Load / Migration Speed! Modestly elevated target latency But still acceptable to SLA 13

15 Migration Throttle Control (3) PID Controller P K p e(t) Tenant Workloads Setpoint Target Latency + Σ Error I K i t e(τ)dτ Σ Output Migration Workload Σ D K d de(t) dt Multitenant Server current transaction latency (feedback) 14

16 Prototype Implementation! Slacker: our framework for database migrations Middleware system above database itself Server 3! Live migration via hot backup Copy/delta/handover MySQL and xtrabackup Other DBs possible mysqld mysqld migration controller! Latency-aware migration Throttled migration rate PID-controlled throttle migration controller mysqld mysqld migration controller mysqld mysqld mysqld mysqld mysqld Server 1 Server 2 15

17 Slacker Evaluation! Yahoo Cloud Serving Benchmark (YCSB) Multi-op transactions, read/write mix, I/O intensive YCSB Instance 1 Workload 1 migration controller migration controller mysqld mysqld transaction latencies mysqld mysqld Workload 2 Source Server Workload 3 Target Server YCSB Instance 2 YCSB Instance 3! Static (fixed) throttle vs. dynamic (PID) throttle 16

18 Evaluation: Controlling Latency! Accuracy of setpoint latency Actual Latency (ms) Setpoint Slacker Setpoint Latency (ms)! Successfully achieves setpoint during migration 17

19 Evaluation: Dynamic vs. Static Throttle! Migration performance at given latency 1 fixed throttle Slacker Txn Latency (ms) Average Throttle Speed (MB/sec) Same latency, faster migration 18

20 Evaluation: Dynamic vs. Static Throttle! Migration performance at given latency 1 fixed throttle Slacker Txn Latency (ms) Same speed, lower latency Average Throttle Speed (MB/sec) 18

21 Evaluation: Dynamic vs. Static Throttle! Migration performance at given latency Txn Latency (ms) fixed throttle Slacker Average Throttle Speed (MB/sec) Steady throttle near max slack! Outperforms fixed throttle at same speed or latency 18

22 Evaluation: Slack Sensitivity! Responsiveness to micro -level workload changes Transfer Rate (MB/sec) throttling speed transaction latency setpoint latency Latency (ms) Time (143 seconds in total)! Exploits normal latency variation to speed migration 19

23 Evaluation: Dynamic Workload! Responsiveness to macro -level workload changes 4% workload increase after 6s 8 fixed throttle latency 7 slacker latency 6 slacker throttle Txn Latency (ms) Time (second) ! Slacker quickly adjusts to dynamic workloads 1 Throttle (MB/sec) 2

24 Conclusions! Managing interference from database migrations Impact on neighboring tenants! Slacker: Latency-aware live migration -independent middleware PID controller to manage migration speed Reacts in real-time to maintain SLA-acceptable performance! Ongoing work Automatic setpoint management Other migration questions e.g. Which tenant? Which server? 21

25 Questions? Sean Barker Department of Computer Science

26 PID Controller Parameters! P, I, D parameters determine speed and degree of controller response Bad: Too cautious Bad: Too optimistic Good: Near-optimal 1% 1% 1% migration speed 5% Slack migration speed 5% Slack migration speed 5% Slack % time % time % time! Currently manually tuned! Many other well-known approaches possible 23

27 Evaluation: Multitenant Workload! Average latency across five tenant workloads Ave Txn Latency(ms) fixed throttle slacker (5 tenants) slacker setpoint Time (second)! Maintains latency in multi-tenant environment 24