CSE 502 Graduate Computer Architecture. Lec 22 Disk Storage

Transcription

1 CSE 52 Graduate Computer Architecture Lec 22 Disk Storage Larry Wittie Computer Science, StonyBrook University and ~lw Slides adapted from David Patterson, UC-Berkeley cs252-s6

2 Magnetic Disks Outline RAID Advanced Dependability/Reliability/Availability I/O Benchmarks, Performance and Dependability Conclusion 5/3/ CSE52-S, Lec 22 Disk Storage 2

3 Storage Hierarchy Third Level - I/O (Disks) 5/3/ CSE52-S, Lec 22 Disk Storage 3

4 I/O (Disk) Performance 5/3/ CSE52-S, Lec 22 Disk Storage 4

5 Disk Parameters 5/3/ CSE52-S, Lec 22 Disk Storage 5

6 Disk Performance 5/3/ CSE52-S, Lec 22 Disk Storage 6

7 Disk Performance Example 5/3/ CSE52-S, Lec 22 Disk Storage 7

8 Disk Performance: Queuing Theory 5/3/ CSE52-S, Lec 22 Disk Storage 8

9 More Extensions to Conventional Disks 5/3/ CSE52-S, Lec 22 Disk Storage 9

10 Disk Figure of Merit: Areal Density Bits recorded along a track Metric is Bits Per Inch (BPI) Number of tracks per surface Metric is Tracks Per Inch (TPI) Disk Designs Brag about bit density per unit area Metric is Bits Per Square Inch: Areal Density = BPI x TPI Year Areal Density ,9 2 7, 26 3,,, Areal Density,, B i t s / s q. i n., Year 5/3/ CSE52-S, Lec 22 Disk Storage

11 Historical Perspective 956 IBM Ramac IBM Dual CPU Disk - early 97s Winchester Developed for mainframe computers, proprietary interfaces, 63.25MB 5 in. platter Steady shrink in form factor: 27 in. to 4 in. Form factor and capacity drives market more than performance 97s developments 8 inch => 5.25 inch floppy disk form-factor (microcode into mainframe) Emergence of industry standard disk interfaces Early 98s: PCs and first generation workstations Mid 98s: Client/server computing Centralized storage on file server» accelerates hard disk downsizing: 8 inch to 5.25 inch Mass market disk drives become a reality» industry standards: SCSI, IPI (Intelligent Peripheral Interface), IDE» 5.25 inch to 3.5 inch drives for PCs, End of proprietary interfaces 99s: Laptops => 2.5 inch drives 2s: What new devices leading to new drives? (iphones) 5/3/ CSE52-S, Lec 22 Disk Storage

12 Use Arrays of Small Disks? Randy Katz and Dave Patterson in 987: Can smaller disks be used to close gap in performance between disks and CPUs? Conventional: 4 disk designs Low End 4 High End Disk Array: disk design 3.5 5/3/ CSE52-S, Lec 22 Disk Storage 2

13 Replace Small Number of Large Disks with Large Number of Small Disks! (988 Disks) Capacity Volume Power Data Rate I/O Rate MTTF Cost IBM 339K 2 GBytes 97 cu. ft. 3 KW 5 MB/s 6 I/Os/s 25 KHrs $25K IBM 3.5" 6 32 MBytes. cu. ft. W.5 MB/s 55 I/Os/s 5 KHrs $2K x7=>raid 23 GBytes cu. ft. KW 2 MB/s 39 IOs/s??? Hrs $5K Disk Arrays have potential for large data and I/ O rates, high MB per cu. ft., high MB per KW, but what about reliability? 5/3/ CSE52-S, Lec 22 Disk Storage 3 9X 3X 8X 6X.7X

14 Array Reliability Reliability of N disks = Reliability of Disk N 5, Hours 7 disks = 7 hours (~9K hrs/yr) Disk system MTTF: Drops from 6 years to month! Arrays (without redundancy) too unreliable to be useful! Hot spares support reconstruction in parallel with access: very high media availability can be achieved 5/3/ CSE52-S, Lec 22 Disk Storage 4

15 Redundant Arrays of Inexpensive Disks Newer RAID Slides F9 Files are "striped" across multiple disks Redundancy yields high data availability Availability: service still provided to user, even if some components have failed Disks will still fail Contents reconstructed from data redundantly stored in the array Capacity penalty to store redundant info Bandwidth penalty to update redundant info Next 2 pages (from cse32 lec25 pgs 5-26) [Adapted from Computer Organization and Design, 4 th Edition, Patterson & Hennessy, 28, MK, with many additions by Mary Jane Irwin, PennStateU] 5/3/ CSE52-S, Lec 22 Disk Storage 5

16 RAIDs: Disk Arrays Redundant Array of Inexpensive Disks Arrays of small and inexpensive disks Increase potential throughput by having many disk drives - Data is spread over multiple disk - Multiple accesses are made to several disks at a time Reliability is lower than a single disk But availability can be improved by adding redundant disks (RAID) Lost information can be reconstructed from redundant information MTTR: mean time to repair is in the order of hours MTTF: mean time to failure of disks is tens of years 5/3/ CSE52-S, Lec 22 Disk Storage 6

17 RAID: Level (No Redundancy; Striping) sec sec2 sec3 sec4 sec,b sec,b sec,b2 sec,b3 Multiple smaller disks as opposed to one big disk Spreading the sector over multiple disks striping means that multiple blocks can be accessed in parallel increasing the performance - A 4 disk system gives four times the throughput of a disk system Same cost as one big disk assuming 4 small disks cost the same as one big disk No redundancy, so what if one disk fails? Failure of one or more disks is more likely as the number of disks in the system increases 5/3/ CSE52-S, Lec 22 Disk Storage 7

18 RAID: Level (Redundancy via Mirroring) sec sec2 sec3 sec4 sec sec2 sec3 sec4 redundant (check) data Uses twice as many disks as RAID (e.g., 8 smaller disks with the second set of 4 duplicating the first set) so there are always two copies of the data # redundant disks = # of data disks so twice the cost of one big disk - writes have to be made to both sets of disks, so writes will be only /2 the performance of a RAID What if one disk fails? If a disk fails, the system just goes to the mirror for the data 5/3/ CSE52-S, Lec 22 Disk Storage 8

19 RAID: Level + (Striping with Mirroring) sec,b sec,b sec,b2 sec,b3 sec,b sec,b sec,b2 sec,b3 sec blk2 blk3 blk4 blk blk2 blk3 blk4 Combines the best of RAID and RAID, data is striped across four disks and mirrored to four disks, called a mirror of stripes or RAID, but may be marketed as RAID + Four times the throughput (due to striping) - # redundant disks = # of data disks, so 2X the cost of one big disk - writes have to be made to both sets of disks, so writes will be only /2 the performance of RAID What if one disk fails? redundant (check) data + If a disk fails, the system just goes to the mirror for the data 5/3/ CSE52-S, Lec 22 Disk Storage 9

20 RAID: Level 2 (Redundancy via Hamming ECC) sec,b sec,b sec,b2 sec,b3 Checks 4,5,6,7 Checks 2,3,6,7 Checks,3,5, error ECC disks ECC disks 4 and 2 point to either data disk 6 or 7 as being bad, but ECC disk says disk 7 is okay, so disk 6 must be in error ECC (Hamming Error Correcting Code) disks contain the even-parity of data on a set of distinct overlapping disks # ECC redundant disks = log 2 (total # of data + ECC disks) so almost twice the cost of one big disk if small # of data disks used. - writes require computing parity to write to half the ECC disks - reads require reading half the ECC disks and confirming parity Can tolerate limited disk failure, since the data can be reconstructed; first correction method; no longer used. Why it works ECC: 42 5/3/ CSE52-S, Lec 22 Disk Storage 2

21 RAID: Level 3 (Bit-Interleaved Parity) sec,b sec,b sec,b2 sec,b3 (odd) bit parity disk Cost of higher availability is reduced to /N where N is the number of disks in a protection group # redundant disks = # of protection groups - writes require writing the new data to the data disk as well as computing the parity, meaning reading the other disks, so that the parity disk can be updated Can tolerate limited (single) disk failure, since the data can be reconstructed - reads require reading all the operational data disks as well as the parity disk to calculate the missing data that was stored on the failed disk 5/3/ CSE52-S, Lec 22 Disk Storage 2

22 RAID: Level 3 (Bit-Interleaved Parity) sec,b sec,b sec,b2 sec,b3 (odd) bit parity disk disk fails Cost of higher availability is reduced to /N where N is the number of disks in a protection group # redundant disks = # of protection groups - writes require writing the new data to the data disk as well as computing the parity, meaning reading the other disks, so that the parity disk can be updated Can tolerate limited (single) disk failure, since the data can be reconstructed - reads require reading all the operational data disks as well as the parity disk to calculate the missing data that was stored on the failed disk 5/3/ CSE52-S, Lec 22 Disk Storage 22

23 RAID: Level 4 (Block-Interleaved Parity) sec sec2 sec3 sec4 Cost of higher availability still only /N but the parity is stored as blocks associated with sets of data blocks Four times the throughput (striping) block parity disk # redundant disks = # of protection groups Supports small reads and small writes (reads and writes that go to just one (or a few) data disk in a protection group) - by watching which bits change when writing new information, need only to change the corresponding bits on the parity disk - the parity disk must be updated on every write, so it is a bottleneck for back-to-back writes Can tolerate limited (single) disk failure, since the data can be reconstructed 5/3/ CSE52-S, Lec 22 Disk Storage 23

24 Small Writes RAID 3 writes New D data D D2 D3 D4 P 3 reads and 2 writes involving all the disks RAID 4 small writes New D data D D2 D3 D4 P D D2 D3 D4 P 2 reads and 2 writes involving just two disks D D2 D3 D4 P 5/3/ CSE52-S, Lec 22 Disk Storage 24

25 RAID: Level 5 (Distributed Block-Interleaved Parity) one of these assigned as the block parity disk Cost of higher availability still only /N but any single parity block may be located on any of the disks so there is no single bottleneck for writes Still four times the throughput (striping) # redundant disks = # of protection groups Supports small reads and small writes (reads and writes that go to just one (or a few) data disk in a protection group) Allows multiple simultaneous writes as long as the accompanying parity blocks are not located on the same disk Can tolerate limited (single) disk failure, since the data can be reconstructed 5/3/ CSE52-S, Lec 22 Disk Storage 25

26 Distributing Parity Blocks Bottle- RAID 4 RAID 5 neck Time P P 9 2 P P P P 8 9 P2 2 3 P Can be done in parallel By distributing parity blocks to all disks, some small writes can be performed in parallel using RAID 5 5/3/ CSE52-S, Lec 22 Disk Storage 26

27 Summary Four components of disk access time: Seek Time: advertised to be 3 to 4 ms but lower in real systems Rotational Latency: 5.6 ms at 54 RPM and 2. ms at 5 RPM Transfer Time: 3 to 8 MB/s => just.6 to.7 ms / 52B-sector Controller Time: typically less than.2 ms RAIDS can be used to improve availability RAID and RAID 5 widely used in servers, one estimate is that 8% of disks in servers are RAIDs RAID + (mirroring) EMC, Tandem, IBM RAID 3 Storage Concepts RAID 4 Network Appliance RAIDS have enough redundancy to allow continuous operation, but not hot swapping 5/3/ CSE52-S, Lec 22 Disk Storage 27

28 RAID 6: Recovering from 2 failures Why > failure recovery? operator accidentally replaces the wrong disk during a failure since disk bandwidth is growing more slowly than disk capacity, the MTTRepair a disk in a RAID system is increasing increases the chances of a 2nd failure during repair since takes longer reading much more data during reconstruction means increasing the chance of an uncorrectable media failure, which would result in data loss 5/3/ CSE52-S, Lec 22 Disk Storage 28

29 RAID 6: Recovering from 2 failures Network Appliance s row-diagonal parity or RAID-DP Like the standard RAID schemes, it uses redundant space based on parity calculation per stripe Since it is protecting against a double failure, it adds two check blocks per stripe of data. If p+ disks total, p- disks have data; assume p=5 Row parity disk is just like in RAID 4 Even parity across the other 4 data blocks in its stripe Each block of the diagonal parity disk contains the even parity of the blocks in the same diagonal 5/3/ CSE52-S, Lec 22 Disk Storage 29

30 Example p = 5 Row diagonal parity starts by recovering one of the 4 blocks on the failed disk using diagonal parity Since each diagonal misses one disk, and all diagonals miss a different disk, 2 diagonals are only missing block Once the data for those blocks is recovered, then the standard RAID recovery scheme can be used to recover two more blocks in the standard RAID 4 stripes Process continues until two failed disks are restored Data Disk Data Disk Data Disk 2 Data Disk 3 Row Parity Diagona l Parity /3/ CSE52-S, Lec 22 Disk Storage 3

31 Berkeley History: RAID-I RAID-I (989) Consisted of a Sun 4/28 workstation with 28 MB of DRAM, four dual-string SCSI controllers, inch SCSI disks and specialized disk striping software Today RAID is $24 billion dollar industry, 8% of non-pc disks are sold in RAIDs 5/3/ CSE52-S, Lec 22 Disk Storage 3

32 Summary: RAID Methods: Goal Was Performance. Highly Popular Since Reliable Storage Disk Mirroring, Shadowing (RAID ) Each disk is fully duplicated onto its "shadow" Logical write = two physical writes % capacity overhead Parity Data Bandwidth Array (RAID 3) Parity computed horizontally Logically a single high data bw disk High I/O Rate Parity Array (RAID 5) Interleaved parity blocks Independent reads and writes Logical write = 2 reads + 2 writes 5/3/ CSE52-S, Lec 22 Disk Storage 32

33 Definitions Examples on why precise definitions so important for reliability Is a programming mistake a fault, error, or failure? Are we talking about the time it was designed or the time the program is run? If the running program doesn t exercise the mistake, is it still a fault/error/failure? If an alpha particle hits a DRAM memory cell, is it a fault/error/failure if it does not change the value? Is it a fault/error/failure if the memory doesn t access the changed bit? Did a fault/error/failure still occur if the memory had error correction and delivered the corrected value to the CPU? 5/3/ CSE52-S, Lec 22 Disk Storage 33

34 IFIP Standard terminology Computer system dependability: quality of delivered service such that reliance can be placed on service Service is observed actual behavior as perceived by other system(s) interacting with this system s users Each module has ideal specified behavior, where service specification is agreed description of expected behavior A system failure occurs when the actual behavior deviates from the specified behavior failure occurred because an error, a defect in a module The cause of an error is a fault When a fault occurs it creates a latent error, which becomes effective when it is activated When error actually affects the delivered service, a failure occurs (time from error to failure is error latency) 5/3/ CSE52-S, Lec 22 Disk Storage 34

35 Fault v. (Latent) Error v. Failure An error is manifestation in the system of a fault, a failure is manifestation on the service of an error Is If an alpha particle hits a DRAM memory cell, is it a fault/error/failure if it doesn t change the value? Is it a fault/error/failure if the memory doesn t access the changed bit? Did a fault/error/failure still occur if the memory had error correction and delivered the corrected value to the CPU? An alpha particle hitting a DRAM can be a fault if it changes the memory, it creates an error error remains latent until effected memory word is read if the effected word error affects the delivered service, a failure occurs 5/3/ CSE52-S, Lec 22 Disk Storage 35

36 Fault Categories. Hardware faults: Devices that fail, such as an alpha particle hitting a memory cell 2. Design faults: Faults in software (usually) and hardware design (occasionally) 3. Operation faults: Mistakes by operations and maintenance personnel 4. Environmental faults: Fire, flood, earthquake, power failure, and sabotage Also by duration:. Transient faults exist for limited time and not recurring 2. Intermittent faults cause a system to oscillate between faulty and fault-free operation 3. Permanent faults do not correct themselves over time 5/3/ CSE52-S, Lec 22 Disk Storage 36

37 Fault Tolerance vs Disaster Tolerance Fault-Tolerance (or more properly, Error- Tolerance): mask local faults (prevent errors from becoming failures) RAID disks Uninterruptible Power Supplies Cluster Failover Disaster Tolerance: masks site errors (prevent site errors from causing service failures) Protects against fire, flood, sabotage,.. Redundant system and service at remote site. Use design diversity From Jim Gray s Talk at UC Berkeley on Fault Tolerance " /9/ 5/3/ CSE52-S, Lec 22 Disk Storage 37

38 Case Studies - Tandem Trends Reported MTTF by Component Mean Time to System Failure (years) by Cause environment total maintenance hardware operations software SOFTWARE Years HARDWARE Years MAINTENANCE Years OPERATIONS Years ENVIRONMENT Years SYSTEM Years Problem: Systematic Under-reporting From Jim Gray s Talk at UC Berkeley on Fault Tolerance " /9/ Minor Problems.... Major Problems 5/3/ CSE52-S, Lec 22 Disk Storage 38

39 % 8% 6% 4% 2% 5% 5% 2% Is Maintenance the Key? Rule of Thumb: Maintenance X HW Over 5 year product life, ~ 95% of cost is maintenance Cause of System Crashes 5% 8% 2% 53% 8% % % (est.) VAX crashes 85, 93 [Murp95]; extrap. to Sys. Man.: N crashes/problem, SysAdmin action Actions: set params bad, bad config, bad app install HW/OS 7% in 85 to 28% in 93. In, %? 5/3/ CSE52-S, Lec 22 Disk Storage 39 69% 5% 5% Other: app, power, network failure System management: actions + N/problem Operating System failure Hardware failure

40 HW Failures in Real Systems: Tertiary Disks A cluster of 2 PCs in seven 7-foot high, 9-inch wide racks with GB, 72 RPM, 3.5-inch IBM disks. The PCs are P6-2MHz with 96 MB of DRAM each. They run FreeBSD 3. and the hosts are connected via switched Mbit/second Ethernet Component Total in System Total Failed % Failed SCSI Controller % SCSI Cable % SCSI Disk % IDE Disk % Disk Enclosure -Backplane % Disk Enclosure - Power Supply % Ethernet Controller 2 5.% Ethernet Switch 2 5.% Ethernet Cable % CPU/Motherboard 2 % 5/3/ CSE52-S, Lec 22 Disk Storage 4

41 Unused Slides Fall 2 5/3/ CSE52-S, Lec 22 Disk Storage 4

42 Extensions to Conventional Disks 5/3/ CSE52-S, Lec 22 Disk Storage 42

43 Does Hardware Fail Fast? 4 of 384 Disks that Failed in Tertiary Disk Messages in system log for failed disk No. log msgs Duration (hours) Hardware Failure (Peripheral device write fault [for] Field Replaceable Unit) Not Ready (Diagnostic failure: ASCQ = Component ID [of] Field Replaceable Unit) Recovered Error (Failure Prediction Threshold Exceeded [for] Field Replaceable Unit) Recovered Error (Failure Prediction Threshold Exceeded [for] Field Replaceable Unit) /3/ CSE52-S, Lec 22 Disk Storage 43

44 High Availability System Classes Goal: Build Class 6 Systems System Type Unmanaged Managed Well Managed Fault Tolerant High-Availability Very-High-Availability Ultra-Availability Unavailable (min/year) 5, 5, Availability 9.% 99.% 99.9% 99.99% % % % Availability Class UnAvailability = MTTR/MTBF can cut it in ½ by cutting MTTR or MTBF From Jim Gray s Talk at UC Berkeley on Fault Tolerance " /9/ 5/3/ CSE52-S, Lec 22 Disk Storage 44

45 How Realistic is "5 Nines"? HP claims HP-9 server HW and HP-UX OS can deliver % availability guarantee in certain pre-defined, pre-tested customer environments Application faults? Operator faults? Environmental faults? Collocation sites (lots of computers in building on Internet) have network outage per year (~ day) power failure per year (~ day) Microsoft Network unavailable recently for a day due to problem in Domain Name Server: if only outage per year, 99.7% or 2 Nines 5/3/ CSE52-S, Lec 22 Disk Storage 45

46 Magnetic Disks Outline RAID Advanced Dependability/Reliability/Availability I/O Benchmarks, Performance and Dependability Conclusion 5/3/ CSE52-S, Lec 22 Disk Storage 46

47 I/O Performance Metrics: Response Time vs. Throughput 3 2 Response Time (ms) % Throughput (% total BW) % Queue Proc IOC Device Response time = Queue + Device Service time 5/3/ CSE52-S, Lec 22 Disk Storage 47

48 I/O Benchmarks For better or worse, benchmarks shape a field Processor benchmarks classically aimed at response time for fixed sized problem I/O benchmarks typically measure throughput, possibly with upper limit on response times (or 9% of response times) Transaction Processing (TP) (or On-line TP=OLTP) If bank computer fails when customer withdraw money, TP system guarantees account debited if customer gets $ & account unchanged if no $ Airline reservation systems & banks use TP Atomic transactions makes this work Classic metric is Transactions Per Second (TPS) 5/3/ CSE52-S, Lec 22 Disk Storage 48

49 I/O Benchmarks: Transaction Processing Early 98s great interest in OLTP Expecting demand for high TPS (e.g., ATM machines, credit cards) Tandem s success implied medium range OLTP expands Each vendor picked own conditions for TPS claims, report only CPU times with widely different I/O Conflicting claims led to disbelief of all benchmarks chaos 984 Jim Gray (Tandem) distributed paper to Tandem + 9 in other companies propose standard benchmark Published A measure of transaction processing power, Datamation, 985 by Anonymous et. al To indicate that this was effort of large group To avoid delays of legal department of each author s firm Still get mail at Tandem to author Anonymous Led to Transaction Processing Council in /3/ CSE52-S, Lec 22 Disk Storage 49

50 I/O Benchmarks: TP by Anon et. al DebitCredit Scalability: size of account, branch, teller, history function of throughput TPS Number of ATMs Account-file size,. GB,. GB,,. GB,,,. GB Each input TPS =>, account records, branches, ATMs Accounts must grow since a person is not likely to use the bank more frequently just because the bank has a faster computer! Response time: 95% transactions take second Report price (initial purchase price + 5 year maintenance = cost of ownership) Hire auditor to certify results 5/3/ CSE52-S, Lec 22 Disk Storage 5

51 Unusual Characteristics of TPC Price is included in the benchmarks cost of HW, SW, and 5-year maintenance agreements included price-performance as well as performance The data set generally must scale in size as the throughput increases trying to model real systems, demand on system and size of the data stored in it increase together The benchmark results are audited Must be approved by certified TPC auditor, who enforces TPC rules only fair results are submitted Throughput is the performance metric but response times are limited eg, TPC-C: 9% transaction response times < 5 seconds An independent organization maintains the benchmarks COO ballots on changes, meetings, to settle disputes... 5/3/ CSE52-S, Lec 22 Disk Storage 5

52 TPC Benchmark History/Status Benchmark Data Size (GB) Performance st Results Metric A: Debit Credit (retired). to transactions/ s Jul-9 B: Batch Debit Credit. to transactions/s Jul-9 (retired) C: Complex Query to 3 new order Sep-92 OLTP (min. 7 * tpm) trans/min (tpm) D: Decision Support, 3, queries/hour Dec-95 (retired) H: Ad hoc decision, 3, queries/hour Oct-99 support R: Business reporting queries/hour Aug-99 decision support (retired) W: Transactional web ~ 5, 5 web inter - actions/sec. Jul- App: app. server & web services Web Service Interactions/sec (SIPS) Jun-5 5/3/ CSE52-S, Lec 22 Disk Storage 52

53 I/O Benchmarks via SPEC SFS 3. Attempt by NFS companies to agree on standard benchmark Run on multiple clients & networks (to prevent bottlenecks) Same caching policy in all clients Reads: 85% full block & 5% partial blocks Writes: 5% full block & 5% partial blocks Average response time: 4 ms Scaling: for every NFS ops/sec, increase capacity GB Results: plot of server load (throughput) vs. response time & number of users Assumes: user => NFS ops/sec 3. for NSF 3. Added SPECMail (mailserver), SPECWeb (webserver) benchmarks 5/3/ CSE52-S, Lec 22 Disk Storage 53

54 Response time (ms) 25 Example SPEC SFS Result: NetApp FAS35c NFS servers 2.8 GHz Pentium Xeon microprocessors, 2 GB of DRAM per processor, GB of Non-volatile memory per system 4 FDDI networks; 32 NFS Daemons, 24 GB file size 68 fibre channel disks: 72 GB, 5 RPM, 2 or 4 FC controllers processors Operations/second 34,89 47,927 4 processors 5/3/ CSE52-S, Lec 22 Disk Storage 54

55 Availability benchmark methodology Goal: quantify variation in QoS metrics as events occur that affect system availability Leverage existing performance benchmarks to generate fair workloads to measure & trace quality of service metrics Use fault injection to compromise system hardware faults (disk, memory, network, power) software faults (corrupt input, driver error returns) maintenance events (repairs, SW/HW upgrades) Examine single-fault and multi-fault workloads the availability analogues of performance micro- and macrobenchmarks 5/3/ CSE52-S, Lec 22 Disk Storage 55

56 Example single-fault result Linux Solaris Compares Linux and Solaris reconstruction Linux: minimal performance impact but longer window of vulnerability to second fault Solaris: large perf. impact but restores redundancy fast 5/3/ CSE52-S, Lec 22 Disk Storage 56

57 Reconstruction policy (2) Linux: favors performance over data availability automatically-initiated reconstruction, idle bandwidth virtually no performance impact on application very long window of vulnerability (>hr for 3GB RAID) Solaris: favors data availability over app. perf. automatically-initiated reconstruction at high BW as much as 34% drop in application performance short window of vulnerability ( minutes for 3GB) Windows: favors neither! manually-initiated reconstruction at moderate BW as much as 8% app. performance drop somewhat short window of vulnerability (23 min/3gb) 5/3/ CSE52-S, Lec 22 Disk Storage 57

58 Unused Slides Fall 29 5/3/ CSE52-S, Lec 22 Disk Storage 58

59 Redundant Arrays of (Inexpensive) Disks RAIDs Files are "striped" across multiple disks Redundancy yields high data availability Availability: service still provided to user, even if some components have failed Disks will still fail Contents reconstructed from data redundantly stored in the array Capacity penalty to store redundant info Bandwidth penalty to update redundant info 5/3/ CSE52-S, Lec 22 Disk Storage 59

60 Redundant Arrays of Inexpensive Disks RAID : Disk Mirroring/Shadowing recovery group Each disk is fully duplicated onto its mirror Very high availability can be achieved Bandwidth sacrifice on write: Logical write = two physical writes Reads may be optimized Most expensive solution: % capacity overhead (RAID 2 - Bit-level striping with Hamming ECC - not interesting, skip.) 5/3/ CSE52-S, Lec 22 Disk Storage 6

61 Redundant Array of Inexpensive Disks RAID 3: Parity Disk... logical record Striped physical records 5/3/ CSE52-S, Lec 22 Disk Storage 6 P contains sum of other disks per stripe mod 2 ( parity ) If disk fails, subtract P from sum of other disks to find missing information P

62 RAID 3 Sum computed across recovery group to protect against hard disk failures, stored in P (parity) disk Logically, a single high capacity, high transfer-rate disk: good for large transfers Wider arrays reduce capacity costs, but decrease availability 33% capacity cost for parity if 3 data disks and parity disk 5/3/ CSE52-S, Lec 22 Disk Storage 62

63 Inspiration for RAID 4 RAID 3 relies on parity disk to discover errors on Read But every sector has an error detection field To catch errors on read, rely on error detection field vs. the parity disk Allows independent reads to different disks simultaneously 5/3/ CSE52-S, Lec 22 Disk Storage 63

64 Redundant Arrays of Inexpensive Disks RAID 4: High I/O Rate Parity Insides of 5 disks D D D2 D3 P D4 D5 D6 D7 P Increasing Logical Disk Address Example: small read D & D5, large write D2-D5 D8 D9 D D P D2 D3 D4 D5 D6 D7 D8 D9 5/3/ CSE52-S, Lec 22 Disk Storage 64 P P D2 D2 D22 D23 P Disk Columns... Stripe Of Sectors

65 Inspiration for RAID 5 RAID 4 works well for small reads Small writes (write to one disk): Option : read other data disks, create new sum and write to Parity Disk Option 2: since P has old sum, compare old data to new data, add the difference to P Small writes are limited by Parity Disk: Write to D, D5 both also write to P disk twice D D D2 D3 P D4 D5 D6 D7 P 5/3/ CSE52-S, Lec 22 Disk Storage 65

66 Redundant Arrays of Inexpensive Disks RAID 5: High I/O Rate Interleaved Parity Independent writes possible because of interleaved parity Example: write to D, D5 uses disks,, 3, 4 D D D2 D3 P D4 D5 D6 P D7 D8 D9 P D D D2 P D3 D4 D5 P D6 D7 D8 D9 D2 D2 D22 D23 P Disk Columns.... 5/3/ CSE52-S, Lec 22 Disk Storage Increasing Logical Disk Addresses

67 Problems of Disk Arrays: Small Writes RAID-5: Small Write Algorithm Logical Write = 2 Physical Reads + 2 Physical Writes D' D D D2 D3 P new data old data (. Read) old parity (2. Read) + XOR + XOR (3. Write) (4. Write) D' D D2 D3 P' 5/3/ CSE52-S, Lec 22 Disk Storage 67

68 Review Virtual Machine Revival Overcome security flaws of modern OSes Processor performance no longer highest priority Manage Software, Manage Hardware VMMs give OS developers another opportunity to develop functionality no longer practical in today s complex and ossified operating systems, where innovation moves at geologic pace. [Rosenblum and Garfinkel, 25] Virtualization challenges: processor, virtual memory, I/O Paravirtualization, ISA upgrades to cope with those difficulties Xen as example VMM using paravirtualization 25 performance on non-i/o bound, I/O intensive apps: 8% of native Linux without driver VM, 34% with driver VM Opteron memory hierarchy still critical to performance 5/3/ CSE52-S, Lec 22 Disk Storage 68

69 Future Disk Size and Performance Continued advance in capacity (6%/yr) and bandwidth (4%/yr) Slow improvement in seek, rotation (8%/yr) Time to read whole disk Year Sequentially Randomly ( sector/seek) 99 4 minutes x 5.5K 2 weeks (=22K min.) 2 2 minutes x 5K 2 months (!) (=.6M min.) minutes x 32K 6 months (SCSI) (=.8M min.) 26 7 minutes x 25K 4 months (SATA) (4.2M min.) SATA.2 GB/s 25: SATA2.5.3 GB/s => 22 GB/hr 5/3/ CSE52-S, Lec 22 Disk Storage 69

70 Advantages of Small Form-Factor Disk Drives Low cost/mb High MB/volume High MB/watt Low cost/actuator Cost and Environmental Efficiencies 5/3/ CSE52-S, Lec 22 Disk Storage 7