Pergamum Replacing Tape with Energy Efficient, Reliable, Disk- Based Archival Storage

Transcription

1 ergamum Replacing Tape with Energy Efficient, Reliable, Disk- Based Archival Storage Mark W. Storer Kevin M. Greenan Ethan L. Miller Kaladhar Voruganti* University of California, Santa Cruz *Network Appliance

2 ergamum: Historical Background Black Sea ergamum Turkey Mediterranean Sea Greek library built by Eumenes II (rule: BC) Credited with the transition from papyrus to parchment Took shelf placement s effect on preservation into account Ended when its contents were gifted to Cleopatra by Mark Antony 2

3 Archival Storage exabytes of data generated in 2006 rofessional: legislated requirements (preservation, retrieval, privacy, etc.) ersonal: histories being recorded digitally (photos, documents, etc.) Archival Storage Traditional, Enterprise Storage Workload write once, (read maybe) variable erformance adequate latency, decent throughput low latency, high throughput Cost cheaper is better commensurate to performance Reliability long-term short-term Scalability time, capacity, technology, vendors capacity M. Baker, K. Keeton, and S. Martin. Why traditional storage systems don t help us save stuff forever. In roceedings of the First IEEE Workshop on Hot Topics in System Dependability (HOTDE 2005), Yokohama, Japan, Jun

4 Existing Archival Approaches Tape Disk Array MAID ergamum Media Costs low medium - high medium low Random Access erformance Centralized Controller poor good medium medium yes usually yes no ower Usage low high medium low Media per reader many one one one 4

5 ergamumʼs Approach Evolvable, distributed network of intelligent, disk-based devices Tomes are smart enough to function independently Together they provide inter-disk redundancy Building blocks for more complex systems Handle errors at multiple levels Scale the response to the size of the problem Control static costs Commodity hardware Standardized interfaces Control operational costs Use low power hardware Keep disks spun down 5

6 ergamum Tome Tome D0 D1 D2 Region with 3 Segments Region with 5 Segments SATA Disk Segment arity Low-ower CU timestamp(d0) timestamp(d1) timestamp(d2) NVRAM sig(d0) sig(d1) sig(d2) Network Low power CU & DRAM: NAS functions, consistency checking, parity ops SATA hard drive: low-cost, persistent storage Data stored in segments with parity appended NVRAM: metadata storage (data signatures, time stamps, index, etc.) Ethernet controller: commodity interconnection network 6

7 Energy Consumption Component Active ower Usage Idle ower Usage SATA Hard Drive 7.5 W 0.3 W Low-ower CU board (w/ NIC) W < 0.3 W NVRAM < 0.6 W 0 W ower usage is proportional to the square of the voltage (½) Voltage (¼) power consumption Modern laptop processor consumes 31W Within the limits of ower over Ethernet Could allow for very simple physical interface 7

8 Two-Level Reliability Approach Redundancy Group arity Region Redundancy Group R R R R R R R R R R R R R R R R R R Redundancy Group R R R R R R R R R R R R R R R R R R Redundancy Group R R R R R R Redundancy Group R R R R R R R R R R R R Tome 0 Tome 1 Tome 2 Tome 3 Intra-disk - protection from latent sector errors Increased reliability reduces scrubbing needs Fix errors with as little spin-up as possible Inter-disk - protection from device loss 8

9 Reading and Writing Data Reads : 1.Spin up the Tome and serve the request - Signatures can confirm the integrity of the read Writes : 1.Spin up the disk in the target Tome 2.Write to a new data block 3.Update internal redundancy 4.Generate deltas for external redundancy 5.Update groups redundancy information 6.Update segment mapping - Old block is retained until all redundancy is updated 9

10 Reading and Writing Data rotocol is simple Read an object Write an object Currently: custom protocol Next version: HTT-based (use common standards!) Choosing a tome for new writes ick a particular tome: similar to a tape-based system Have ergamum pick a tome - Tome with available space? - Tome that s already spun up? 10

11 Consistency Checking: Scrubbing D D D D D D D D D D D D D D D D Tome 0 eriodically verify data stored on a tome (by segment) Read segment, including parity, and verify consistency Record signatures in NVRAM Need to scrub entire disk... but not all at once! Scrub a gigabyte or two each time the disk is powered on - Checks basic disk functioning & detects dead (or nearly so) disks Ensure that entire disk is scrubbed within a year - Intra-disk parity makes data loss highly unlikely 11

12 Consistency Checking: Algebraic Signatures d0 d1... dn-1 = p sig(d0) sig(d1)... sig(dn-1) = sig(p) Signature of parity is the same as parity of the signatures Works with codes calculated using XOR operations - e.g. EvenOdd, X-codes, Reed-Solomon over a Galois field Signatures calculated during writes and scrubs Stored in NVRAM to avoid extra disk spin ups Schwarz, S. J., T., and Miller, E. L. Store, forget, and check: Using algebraic signatures to check remotely administered storage. In roceedings of the 26th International Conference on Distributed Computing Systems (ICDCS 06) (Lisboa, ortugal, July 2006), IEEE. 12

13 Efficiently Finding Faults L0 sig sig sig sig L1 sig sig sig sig sig sig = sig sig L2 sig = sig sig sig sig sig sig sig sig sig sig sig sig sig sig sig Data Intra-Disk Redun. = Tome 0 Tome 1 Tome 2 Tome 3 Inter-disk Redun. Tomes calculate a hash tree of algebraic signatures Tree roots (L0) are exchanged Reduces amount of data that must be transmitted Inconsistency located by traversing down the tree Bad block identified by code or internal parity R R R R 13

14 Efficiently Finding Faults L0 sig sig sig sig L1 L2 Data Intra-Disk Redun. sig = sig sig sig sig sig sig sig sig = sig sig sig sig sig sig sig sig sig sig sig sig sig sig sig = R R R R Tome 0 Tome 1 Tome 2 Tome 3 Inter-disk Redun. Tomes calculate a hash tree of algebraic signatures Tree roots (L0) are exchanged Reduces amount of data that must be transmitted Inconsistency located by traversing down the tree Bad block identified by code or internal parity 14

15 Rebuilding Lost Data Naïve approach: spin up entire redundancy group Consumes far too much power! ergamum approach: rebuild lost data incrementally Target disk for lost information is always spun up Other disks are spun up one at a time, merging their data into the target disk Advantage: lower peak energy consumption Disadvantage: higher total energy consumption: target disk writes data multiple times as each update comes in Multiple disks can be spun up simultaneously if power is available... 15

16 Results: Reliability (hours) loss to data Ideal Intra Intra 62+2 Intra Intra time Mean Inter-disk reliability level for 16 total disks Discrete event simulation Each disk fully scrubbed each year Rebuilding at 3 MB/s Latent sector error rate: 1/13,245 hours Disk failure rate: 1/100,000 hours Disk power cycle costs 10 hours of lifetime Ideal system never encounters latent sector errors 16

17 Results: Cost Efficiency System Media ower Aware Static Cost Oper. Cost Redundancy EMC Centera SATA HD No $6,600 $1,800 arity ARAID SCSI HD Yes $37,800 $1,200 RAID Copan Revolution SATA HD Yes $19,000 $250 RAID-5 RAIL UDO2 No $57,000 $225 RAID-4 (5+1) ergamum SATA HD Yes $4,700 $50 2-Level Costs for 10B in thousands of dollars at $0.20 per kwh 17

18 Results: Cost Efficiency vs. Tape System Media Static Cost Oper. Cost Redundancy Media er Reader Avg. Access Time Sun StorageTek SL8500 T10000 Tape $4,250 $60 None Many 70+ secs ergamum SATA HD $4,700 $50 2-Level 1 <13 secs Costs for 10B in thousands of dollars at $0.20 per kwh ergamum s cost is comparable to tape Improved random access performance allows Inter-device reliability Consistency checking and auditing Standardized interfaces and access protocols facilitate evolvable storage 18

19 Results: erformance Raw Data Transfer Read Unsafe Write XOR Write (63+1) RS Write (63+1) RS Write (63+1, 3+2) Throughput (MB/s) Throughput was CU bound Implementation was affected by ython s buffer management Low-power processors benefit from optimized implementations Adequate for archival purposes 1000 tomes with spin-rate of 5% could ingest over 175 MB/s 19

20 Future Work Archival storage management What happens when disks are added or removed? - Due to failure - Due to long-term storage evolution? Standardized protocol rovides scalability through hardware and vendors Distributed indexing and searching Data migration Device refreshment for system reliability Optimized implementation Low power processors very sensitive to implementation Leveraging client CU power 20

21 Questions? Thanks to our sponsors: SSRC industrial sponsors etascale Data Storage Institute Thanks to ergamum team members Kevin M. Greenan Mark W. Storer Kaladhar Voruganti (Network Appliance) 21