DOLLY: Virtualization-Driven Database Provisioning for the Cloud

Size: px

Start display at page:

Download "DOLLY: Virtualization-Driven Database Provisioning for the Cloud"

Stuart Walker
5 years ago
Views:

1 DOLLY: Virtualization-Driven Database Provisioning for the Cloud Emmanuel Cecchet Joint work with Rahul Singh, Upendra Sharma and Prashant Shenoy

2 THE CLOUD Virtualization Pay as you go Elasticity Internet Frontend/ Load balancer App. Servers Databases 2

3 PROVISIONING IN THE CLOUD Based on request volume and resource usage Reactions based on thresholds Works for stateless tiers Internet Frontend/ Load balancer App. Servers Provisioning logic Databases 3

4 WHY IS IT HARD TO ADD A DB REPLICA? 5pm MySQL backup snapshot MySQL restore New replica Replay updates Replica ready 4

5 WHY IS IT HARD TO ADD A DB REPLICA? 2pm 2pm snapshot 5pm Replay updates MySQL restore New replica Replica ready 5

6 WHY IS IT HARD TO ADD A DB REPLICA? RUBiS x users RUBiS x users with indices MySQL backup snapshot MySQL restore RUBiS New replica 14min Queries are slow Let s improve this! CREATE INDEX i1 on Table1 ; CREATE INDEX i2 on Table 2 MySQL backup snapshot MySQL restore RUBiS New replica 1H36min 6

7 WHY IS IT HARD TO ADD A DB REPLICA? Warehouse 1GB Warehouse 10GB PostgreSQL PostgresSQL backup snapshot restore PostgreSQL PostgresSQL backup snapshot restore Warehouse 1GB Warehouse 10GB apt-get update postgresql echo 1 > /proc/sys/magic_options CREATE USER x GRANT PRIVILEGES TO y 24min 1H30min 7

8 WHAT ARE THE MAIN PROBLEMS? When to start replica spawning? How to predict replica spawning time? How to make replica spawning platform independent? When to generate new snapshots? How can we minimize resource usage? Power/cooling in private cloud $ cost in public cloud 8

9 VM CLONING: BACKUP/RESTORE IN CONSTANT TIME Filesystem snapshot/copy is & DB agnostic Only depends on VM size Database DB size on disk DB Backup Restore Dolly 4GB VM cloning Dolly 16GB VM cloning RUBiS c i 1022MB 843s 281s 899s RUBiS +c+bi 1.4GB 5761s 282s 900s RUBiS +c+fi 1.5GB 6017s 280s 900s TPC-W 684MB 288s 275s 905s TPC-H 1GB 1.8GB 1477s 271s 918s TPC-H 10GB 12GB 5573s n/a 911s 9

10 Dolly Database replication in the Cloud Provisioning with Dolly Prototype & Evaluation

11 SPAWNING A REPLICA WITH CLONING Backup & Restore replace by VM cloning Client SQL requests Replication middleware Load balancer DB 1 DB 2 VM 1 VM 2 Transactional log 2 clone DB 2 VM 3 1 Management add console replica clone 3 DB 2 3 VM 3 4 resynchronize 11

12 SPAWNING IN A PRIVATE CLOUD Clone entire virtual machine for backup/restore Backup server is optional DB 1 VM 1 DB 2 VM 4 stop 1 start 3 DB 1 VM 1 resume DB 2 VM clone DB 2 VM 3 DB 2 VM 2 B clone start DB 2 VM

13 SPAWNING IN A PUBLIC CLOUD Storage decoupled from computing resource Starting a new instance clones the volume DB 1 Vol 1 stop DB 1 DB 1 Vol 1 restart snapshot DB 2 DB 2 Vol 2 start DB 2 register DB 2 Vol 2 Vol 1 Vol 4 Vol 3 13

14 Dolly Database replication in the Cloud Provisioning with Dolly Prototype & Evaluation

15 MODELING SPAWNING TIME Predictable backup and restore times are required Replay time can be estimated from write throughput w t : current workload write throughput w max : replay speed of the spawning replica replica spawning time backup restore replay time b i r i ( b + r ) i i w t w t wt... w w w max max max updates 15

16 WHEN TO SNAPSHOT? Time to spawn from a live replica s wmax = ( b i + r i ) w w max Time to spawn from an existing snapshot ( ) max s = r i + replay i w w max t t w Faster to take a new snapshot j to spawn a new replica than using old snapshot i if: backup j +restore j < restore i +replay i 16

17 DOLLY OVERVIEW Input capacity prediction write prediction Output schedule of snapshots schedule of replica spawning admission control if needed Predictors capacity predictions Capacity Provisioning HA adjuster Spawning options Write throttling write throttling/ read throttling Scheduler Admission Control write predictions start/stop clone/ snapshot Management API Dolly Snapshot scheduler Paused pool cleaner delete VM/ snapshot Monitoring Free pool Manager reclaim 17

18 PROVISIONING REPLICAS Dolly does not provide predictors Dolly can work with any predictor (see [Eurosys09]) Workload prediction Capacity prediction Write prediction 18

19 CLOUD CT FUNCTIONS Adapt the provisioning decisions to the cloud platform specifics Cost can be $ on public cloud or time on private cloud Cost function name pause_cost(vm, t) spawn_cost(s, t, d) spawn_cost(vm, t, d) running_cost(vm,t1,t2) pause_resume_cost(vm, t1, t2) Definition cost of pausing VM at time t cost to spawn a replica from snapshot s at time t to meet deadline d cost to spawn a replica from a paused VM at time t to meet deadline d cost to run a VM from time t1 to time t2 cost to pause a VM at time t1 and resume it at time t2 backup_paused_cost(vm) backup_live_cost(vm, t) cost to backup a paused VM cost to backup an active VM at time t 19

20 PROVISIONING REPLICAS Parse capacity provisioning predictions Decrease capacity by pausing VMs Increasing capacity Check if we can reuse a paused VM Check if we can spawn from an existing snapshot Choose cheapest options according to spawn_cost function Perform admission control if all replicas cannot be provisioned in time 20

21 SNAPSHOT SCHEDULING How to snapshot? Clone a paused VM Pause an active VM to clone it When to snapshot? At time j when backup j +restore j <restore i +replay i If new snapshot is scheduled, re-run capacity provisioning Prediction window must have minimum size 21

22 Dolly Database replication in the Cloud Provisioning with Dolly Prototype & Evaluation

23 IMPLEMENTATION TPC-W load injector admission control predictions Private Dolly EC2 C-JDBC/Sequoia replication middleware OpenNebula Cloud management middleware Cost functions private cloud: minimize resource utilization time Amazon EC2: minimize cost Scheduler DB 1 DB 2 DB 3 VM 1 Sequoia driver SQL requests Load balancer VM 2 Sequoia controller VM 3 write throttling Backupers Dolly OpenNebula start/stop/ clone/ add/remove replica snapshot/pause/ JMX Management API Recovery Log Dump table New replica VM 4 Log table New replica VM 5 OpenNebula clone clone Backup server or NAS DB 3 snapshot VM clone 23

24 IMPLEMENTATION CT FUNCTIONS Private cloud: minimize resource utilization Amazon EC2: minimize cost Cost function name Private Cloud EC2 pause_cost(vm, t) return 1/VM->machine->temp return 60-((t-VM->start)%60) spawn_cost(s, t, d) return d-t comp$=(d-t)/60*hour$ io$=ebs_storage$*s->size + EBS_io$* (s->restore_io+s->replay_io) return comp$+io$ spawn_cost(vm, t, d) return d-t comp$=(d-t)/60*hour$ io$= EBS_io$* (s->resume_io+s->replay_io) return comp$+io$ running_cost(vm,t1,t2) return 1 (t2-t1)/60*hour$ pause_resume_cost(vm, t1, t2) if (t2-t1 > VM->pause + VM->resume) return 0 else return 2 io$= EBS_io$* (VM->pause_io+VM->resume_io) comp$=(60-(vm->stop-vm->start) %60)/60*hour$ return io$+ comp$ backup_paused_cost(vm) return backup_time return S3_storage$*s->size backup_live_cost(vm, t) return VM->pause + backup_time + VM->resume return pause_cost(vm, t)$+ S3_storage$*s->size + (VM->stop_io+VM->start_io)* EBS_io$ 24

25 TPC-W EVALUATION Multi-tier online bookstore benchmark 4GB Xen VM for the database Large EC2 instances from EBS volumes with CloudWatch Operation Private Cloud Public Cloud (EC2) start VM 42s 220s pause VM 26s 30s resume VM 42s 30s backup (stop/clone) 150s 320s restore (clone/start) 165s 220s w max 149 writes/sec 197 writes/sec Avg IOs per write

26 WORKLOAD DESCRIPTION Snapshot s 0 available at t 0 26

27 Overprovisioning with 6 replicas 1h snapshot Cost MRM Private cloud 720m 0 Amazon EC2 $

28 Reactive provisioning Cost MRM Private cloud 410m 42.1 Amazon EC2 $ replica spawning triggered here replicas available 28

29 Reactive provisioning 15m snapshot Cost MRM Private cloud 381m42s 17.5 Amazon EC2 $

30 Dolly 30m Prediction Window Private Cloud Cost MRM Private cloud 352m 0 Amazon EC2 $ Amazon EC2 s 1 s 2 cheaper to leave instances online 30

CONCLUSION VM cloning Solves administration issues by blackboxing the database Constant time backup/restore needed to predict replica spawning

31 CONCLUSION VM cloning Solves administration issues by blackboxing the database Constant time backup/restore needed to predict replica spawning time New provisioning algorithm Decouples capacity provisioning from snapshot scheduling Cost functions to optimize for cloud platform specifics 31

32 BONUS SLIDES

33 Dolly 10m Prediction Window Private Cloud Cost MRM Private cloud 381m54s 0 Amazon EC2 $ Amazon EC2 s 1 s 2 s 1 s 2 33

34 Reactive provisioning 1h snapshot Cost MRM Private cloud 360m30s 25.8 Amazon EC2 $

35 BACKUP/RESTORE TECHNIQUES Database native tools Vendor specific or 3 rd party ETL Understand database semantics Filesystem copy Low-level data copy Need to know what to copy VM cloning Copies database content + configuration + Unused space can be compressed 35

36 DATABASE SIZES RUBiS Benchmark DB size Snapshot size VM size MyISAM no constraint MyISAM w/ constraints MyISAM w/ constraint & index InnoDB no constraint InnoDB w/ constraints InnoDB w/ constraint & index 836MB 1.1GB 1.2GB 1022MB 1.4GB 1.5GB TPC-W PostgreSQL binary dump 684MB PostgreSQL sql dump TPC-H scale 1(GB) TPC-H scale 10(GB) PostgreSQL binary dump PostgreSQL sql dump PostgreSQL binary dump PostgreSQL sql dump 1.8GB 12GB 844MB 210MB 314MB 307MB 1.2GB 2.0GB 7.3GB 4.1GB 2.1GB 1.1GB () + 2.1GB (data) 16GB 36

37 BACKUP/RESTORE PERFORMANCE (1/3) Performance depends on database content Time in seconds VM shutdown VM copy VM cloning VM boot MySQL backup MySQL restore VM cloning MySQL MyISAM MySQL InnoDB VM cloning MySQL MyISAM MySQL InnoDB VM cloning MySQL MyISAM MySQL InnoDB RUBiS no constraint RUBiS w/ constraints & basic index RUBiS w/ constraints & full text index 37

38 BACKUP/RESTORE PERFORMANCE (2/3) File copy is the most effective for small databases Time in seconds VM shutdown VM/File copy VM cloning VM boot PostgreSQL stop/start PG bin backup PG bin restore PG sql backup PG sql restore File copy VM cloning direct VM cloning & copy PG bin PG sql TPC-W 38

39 BACKUP/RESTORE PERFORMANCE (3/3) VM cloning most effective on large databases Time in seconds File copy VM shutdown VM/File copy VM cloning VM boot PostgreSQL stop/start PG bin backup PG bin restore PG sql backup PG sql restore VM cloning direct VM cloning & copy TPC-H 1GB PG bin PG sql File copy VM cloning direct 1215 VM cloning & copy TPC-H 10GB 5573 PG bin 5256 PG sql 39

40 BACKUP/RESTORE SUMMARY Feature Database specific knowledge DB Backup/ Restore Filesystem Copy VM Cloning Medium Very high None Performance Slow Fastest Fast Snapshot size Small DB size VM size Spawning time predictability Hard Moderate Easy Database installation Moderate Moderate None Database configuration Hard Hard None Missing data in transfer Possible Unlikely No Spawning atomicity No No Yes Resynchronization limitations Yes Yes Yes 40

41 DOLLY MAIN ALGORITHM Capacity provisioning depends on available snapshots Snapshots scheduled according to capacity demand Decouple capacity provisioning from snapshot scheduling if (predictor.capacity_changes predictor.write_workload_changes) { do { schedule = capacity_provisioning(predictions) snapshot_schedule = snapshot_scheduling(predictions) } while (snapshot_schedule schedules new snapshots) scheduler.schedule(snapshot_schedule) scheduler.schedule(capacity_schedule) } if (time since last operation > threshold) { paused_pool_cleaner.release_old_paused_vms(); paused_pool_cleaner.delete_old_snapshots(); } 41

42 RELEASING RESOURCES Paused VMs VM never re-used if cost to resume > cost to spawn from last snapshot Snapshots Old snapshots can be released based on cost to keep them around Free server pool Can reclaim servers with paused VMs when pool is empty 42

Dolly: Database Provisioning for the Cloud

University of Massachusetts, Technical Report UM-CS-2010-006 1 Dolly: Database Provisioning for the Cloud Emmanuel Cecchet, Rahul Singh, Upendra Sharma, Prashant Shenoy University of Massachusetts, Amherst