Single-pass restore after a media failure. Caetano Sauer, Goetz Graefe, Theo Härder

Transcription

1 Single-pass restore after a media failure Caetano Sauer, Goetz Graefe, Theo Härder

2 20% of drives fail after 4 years High failure rate on first year (factory defects) Expectation of 50% for 6 years 2

3 Bad batches among otherwise reliable products More reliability at a premium price tag 3

4 Disk is dead? Drawbacks of traditional recovery apply (in varying degrees) to all types of media Sep

5 2. Log replay Media restore Primary storage (latency-optimized) Secondary storage (bandwidth-optimized) Log Log archive DB Backups Full Incr. 1. Restore backups Replacement 5

6 How bad can it get? Scenario: Drive: 6 TB; 200 MB/s bandwidth; 4ms latency Backups: full every week; incremental every day Workload: 2% of data pages change daily Full Incr. Incr. Incr. Incr. Incr. Incr. Sun 11PM Mon 11PM Tue 11PM Wed 11PM Thu 11PM Fri 11PM Sat 11PM Sun 10PM Disk failure* * Subject to Murphy s law 6

7 How bad can it get? 1. Restore full backup: 6 TB at 200 MB/s = 9 hours 2. Restore 6 incremental backups: 6 * 16 mi pages * 1 ms = 26h (Assuming 1ms latency of jump-sequential access) 3. Log replay with 75% buffer hit ratio = 10h (Assuming 20 log records per modified page) TOTAL = 45 hours Depends on device Depends on workload (growth rate & skew) and luck Depends on amount of RAM and luck Single-pass restore: 7

8 Scope: physiological logging Fundamental assumption: page-based storage (i.e., virtually all database systems in use today) Every log record reflects changes to a delimited region of physical storage page = unit of recovery recovery independence among objects (C. Mohan) 8

9 Some recent work 9

10 Some recent work 10

11 Sorted log archive Log Log archive Sorted log archive DB Merge join Replacement Full Merge log and backup in a single pass! 11

12 So what's new? 12

13 Partially sorted log archive External sort has two phases: 1. Run generation 2. Merge during log archiving (normal processing) during restore Log DB In-memory sorting Log archive Run Run Run Run Run Run Run Run 13

14 Single-pass restore Run 1 Run 2 Run 3... Run 1000 Merge Full backup Sorted log No need for incrementals Merge Restored DB 15

15 Single-pass restore Run 1 Run 2 Run 3... Run 1000 RAM Buffer Buffer Buffer Buffer Merge Merge fan-in is limited by memory Full backup Merge But number of runs can be kept manageable Restored DB 16

16 Run management Back to example: 6 TB with 2% daily change; 20 log records per page Assume average log record size = 128 bytes Log volume: 2 GB per day; 14 GB per week Assume initial run size = 100 MB 20 runs per day Assume 1 MB merge buffers 140 MB to merge a whole week worth of log 1 MB of RAM to merge 100 MB of log Runs as history units: Jan 2014 Feb 2014 W W Mar MB to merge 1 year worth of log (11 monthly + 3 weekly + 6 daily + 20 current day) 17

17 Backups reconsidered How often? monthly, quarterly, annually? log replay bottleneck eliminated Virtual backups new full backup generated by merging old backup and log archive decoupled from DB activity may run on remote site Old backup Merge New backup Run Run Run Run Run Run Run Run Monthly run 19

18 Incremental backups Goal: alleviate cost of log replay without overhead of taking a full backup But does it still make sense? In single-pass restore: very fast log replay (faster than loading scattered pages) log replay hidable on load of full backup log records can be aggregated (net change) log archiving with zero interference in transaction processing logic full backups much cheaper to take 20

19 Experiments 21

20 Hypotheses 1. The cost of run generation during normal processing is negligible i.e., substantially faster media recovery practically for free 2. Single-pass restore uses very little memory, independent of DB size i.e., memory is not stolen from buffer pool 3. Single-pass restore hides log replay costs i.e., restoring up-to-date DB takes the same as loading an outdated full backup 22

21 Run generation overhead is negligible Scenarios: B T S = sorting (run generation) S+M = sorting with asynchronous merging = baseline (no log archiving no media recovery) = traditional log archiving (process and copy; no sorting) Setup: Shore-MT 24-core machine TPC-C benchmark All in memory 23

22 Run generation overhead is negligible Log in DRAM 1.5% Slightly higher CPU utilization (OLTP workloads rarely fully utilize CPU) B = baseline T = traditional log archiving S = sorting (run generation) M = async. merging 24

23 Run generation overhead is negligible Log in SSD No observable difference in throughput B = baseline T = traditional log archiving S = sorting (run generation) M = async. merging 25

24 Single-pass restore uses very little memory Small memory footprint, independent of database size 26

25 Single-pass restore hides log replay Outdated (days or weeks) Up-to-date (TPC-C databases) 27

26 Future work (teaser) 28

27 Instant restore Log DB In-memory sorting + index load Log archive Run Restore failed device: incrementally on-demand while transactions are running (on buffer pool and healthy/restored storage) Backup 30

28 Conclusion Traditional restore: Single-pass restore: R R Instant restore: New transactions running Thank you! 31