John Lazzaro ( Dave Patterson ( www-inst.eecs.berkeley.

Transcription

1 Review CS2 Computer Architecture and Engineering Lecture 8 ECC, RAID, Bandwidth vs. Latency 228 John Lazzaro ( Dave Patterson ( wwwinst.eecs.berkeley.edu/~cs2/ CS 2 L8 Disks, RAID, () Buses are an important technique for building largescale systems Their speed is critically dependent on factors such as length, number of devices, etc. Critically limited by capacitance Direct Memory Access (dma) allows fast, burst transfer into processor s memory: Processor s memory acts like a slave Probably requires some form of cachecoherence so that DMA ed memory can be invalidated from cache. Networks and switches popular for LAN, WAN Networks and switches starting to replace buses on desktop, even inside chips CS 2 L8 Disks, RAID, (2) Review: ATA cables Serial ATA, Rounded parallel ATA, Ribbon parallel ATA cables inches max vs. 8 inch Outline ECC RAID: Old School & Update Latency vs. Bandwidth (if time permits) Serial ATA cables are thin CS 2 L8 Disks, RAID, () CS 2 L8 Disks, RAID, () Computer memories can make errors occasionally To guard against errors, some memories use errordetecting codes or errorcorrecting codes (ECC) => extra bits are added to each memory word When a word is read out of memory, the extra bits are checked to see if an error has occurred and, if using ECC, correct them Data + extra bits called code words Given 2 code words, can determine how many corresponding bits differ. To determine how many bits differ, just compute the bitwise Boolean EXCLUSIVE OR of the two codewords, and count the number of bits in the result The number of bit positions in which two codewords differ is called the Hamming distance if two code words are a Hamming distance d apart, it will require d singlebit errors to convert one into the other CS 2 L8 Disks, RAID, () CS 2 L8 Disks, RAID, (6)

2 For example, the code words and are a Hamming distance apart because it takes singlebit errors to convert one into the other. Xor s = Hamming distance As a simple example of an errordetecting code, consider a code in which a single parity bit is appended to the data. The parity bit is chosen so that the number of bits in the codeword is even (or odd). E.g., if even parity, parity bit for is. Such a parity code has Hamming distance 2, since any singlebit error produces a codeword with the wrong parity It takes 2 singlebit errors to go from a valid codeword to another valid codeword => detect single bit errors. Whenever a word containing the wrong parity is read from memory, an error condition is signaled. The program cannot continue, but at least no incorrect results are computed. CS 2 L8 Disks, RAID, (7) CS 2 L8 Disks, RAID, (8) a Hamming distance of 2k + is required to be able to correct k errors in any data word As a simple example of an errorcorrecting code, consider a code with only four valid code words:,,, and This code has a distance, which means that it can correct double errors. If the codeword arrives, the receiver knows that the original must have been (if there was no more than a double error). If, however, a triple error changes into, the error cannot be corrected. CS 2 L8 Disks, RAID, (9) ErrorCorrecting Codes CS 2 L8 Disks, RAID, () Hamming Codes How many paritybits are needed? m paritybits can code 2 m m infobits Infobits Paritybits < <2 <27 <8 6 <2 7 How correct single error (SEC) and detect 2 errors (DED)? How many SEC/DED bits for 6 bits data? Administrivia HW, Lab Lab is next: Plan by Thur for TA, Meet with TA Friday, Final Monday ECC Hamming Code. Hamming Coding is a coding method for detecting and correcting errors. ECC Hamming distance between 2 coded words must be Number bits from right, starting with All bits whose bit number is a power of 2 are parity bits We use EVEN PARITY in this example This example shows a data bits Bit will check (parity) in all the bit positions that use a in their number D D D P D P P 7BIT CODEWORD Bit 2 will check all the bit positions that D D D P use a 2 in their number D D D P Bit will check all the bit positions that D D D P use a in their number Etc. CS 2 L8 Disks, RAID, () CS 2 L8 Disks, RAID, (2) 2

3 Example: Hamming Code. Example: The message would be sent as, since: BIT CODEWORD EVEN PARITY If number of s is even then Parity = Else Parity = Let us consider the case where an error caused by the channel transmitted message received message > BIT: BIT: Example: Hamming Code. transmitted message received message > BIT: BIT: The above error (in bit ) can be corrected by examining which of the three parity bits was affected by the bad bit: BIT CODEWORD NOT! OK! NOT! bad parity bits labeled point directly to the bad bit since binary equals CS 2 L8 Disks, RAID, () CS 2 L8 Disks, RAID, () Will Hamming Code detect and correct errors on parity bits? Yes! RAID Beginnings transmitted message received message > BIT: BIT: The above error in parity bit (bit ) can be corrected by examining as below: 7 6 CS 2 L8 Disks, RAID, () 2 7BIT CODEWORD NOT! the bad parity bits labeled point directly to the bad bit since binary equals. In this example error in parity bit is detected and can be corrected by flipping it to a OK! OK! We had worked on generations of Reduced Instruction Set Computer (RISC) processors Our expectation: I/O will become a performance bottleneck if doesn t get faster Randy Katz gets Macintosh with disk along side Use PC disks to build fast I/O to keep pace with RISC? CS 2 L8 Disks, RAID, (6) Redundant Array of Inexpensive Disks (9879) Hard to explain ideas, given past disk array efforts Paper to educate, differentiate? RAID paper spread like virus Products from Compaq, EMC, IBM, RAID I Sun /28, 28 MB of DRAM, dualstring SCSI controllers, 28.2 MB disks + SW RAID II Gbit/s net +. 2 MB disks st Network Attached Storage Ousterhout: Log Structured File Sys. widely used (NetAp) Today RAID ~ $2B industry; 8% of server disks in RAID 998 IEEE Storage Award Students: Peter Chen, Ann Chevernak, Garth Gibson, Ed Lee, Ethan Miller, Mary Baker, John Hartman, Kim Keeton, Mendel Rosenblum, Ken Sherriff, CS 2 L8 Disks, RAID, (7) CS 2 L8 Disks, RAID, (8) Latency Lags Bandwidth Over last 2 to 2 years, for network disk, DRAM, MPU, Latency Lags Bandwidth: Bandwidth d 2X to 22X But Latency d only X to 2X Look at examples, reasons for it Relative (Late ncy improve = Bandwidth improve) Relative Latency

4 Disks: Archaic(Nostalgic) v. Modern(Newfangled) Latency Lags Bandwidth (for last ~2 years) CDC Wren I, 98 6 RPM. GBytes capacity Tracks/Inch: 8 Bits/Inch: 9 Three.2 platters Bandwidth:.6 MBytes/sec Latency: 8. ms Cache: none CS 2 L8 Disks, RAID, (9) Seagate 7, 2 RPM (X) 7. GBytes (2X) Tracks/Inch: 6 (8X) Bits/Inch:, (6X) Four 2. platters (in. form factor) Bandwidth: 86 MBytes/sec (X) Latency:.7 ms (8X) Cache: 8 MBytes Relative (Late ncy improve = Bandwidth improve) Relative Latency CS 2 L8 Disks, RAID, (2) Disk: 6,, 72,, RPM (8x, x) = bestcase) Memory:Archaic(Nostalgic)v. Modern(Newfangled) 98 DRAM (asynchronous).6 Mbits/chip 6, xtors, mm 2 6bit data bus per module, 6 pins/chip Mbytes/sec Latency: 22 ns (no block transfer) CS 2 L8 Disks, RAID, (2) 2 Double Data Rate Synchr. (clocked) DRAM 26. Mbits/chip (X) 26,, xtors, 2 mm 2 6bit data bus per DIMM, 66 pins/chip (X) 6 Mbytes/sec (2X) Latency: 2 ns (X) Block transfers (page mode) Latency Lags Bandwidth (last ~2 years) Relative Memory Relative Latency CS 2 L8 Disks, RAID, (22) (Late ncy improve = Bandwidth improve) Memory Module: 6bit plain DRAM, Page Mode DRAM, 2b, 6b, SDRAM, DDR SDRAM (x,2x) Disk: 6,, 72,, RPM (8x, x) = bestcase) LANs: Archaic(Nostalgic)v. Modern(Newfangled) Ethernet 82. Year of Standard: 978 Mbits/s link speed Latency: µsec Shared media Coaxial cable Coaxial Cable: CS 2 L8 Disks, RAID, (2) Ethernet 82.ae Year of Standard: 2, Mbits/s (X) link speed Latency: 9 µsec (X) Switched media Category copper wire "Cat " is twisted pairs in bundle Plastic Covering Twisted Pair: Braided outer conductor Insulator Copper core Copper, mm thick, twisted to avoid antenna effect Latency Lags Bandwidth (last ~2 years) Relative Memory CS 2 L8 Disks, RAID, (2) Netw ork (Late ncy improve = Bandwidth improve) Relative Latency Ethernet: Mb, Mb, Mb, Mb/s (6x,x) Memory Module: 6bit plain DRAM, Page Mode DRAM, 2b, 6b, SDRAM, DDR SDRAM (x,2x) Disk: 6,, 72,, RPM (8x, x) = bestcase)

5 CPUs: Archaic(Nostalgic) v. Modern(Newfangled) 982 Intel MHz 2 MIPS (peak) Latency 2 ns, xtors, 7 mm 2 6bit data bus, 68 pins Microcode interpreter, separate FPU chip (no caches) CS 2 L8 Disks, RAID, (2) 2 Intel Pentium MHz (2X) MIPS (peak) (22X) Latency ns (2X) 2,, xtors, 27 mm 2 6bit data bus, 2 pins way superscalar, Dynamic translate to RISC, Superpipelined (22 stage), OutofOrder execution Onchip 8KB Data caches, 96KB Instr. Trace cache, Latency Lags Bandwidth (last ~2 years) Note: Proces s or Processor Biggest, Memory Smallest Relative Memory Netw ork Relative Latency Processor: 286, 86, 86, Pentium, Pentium Pro, Pentium (2x,22x) Ethernet: Mb, Mb, Mb, Mb/s (6x,x) Memory Module: 6bit plain DRAM, Page Mode DRAM, 2b, 6b, SDRAM, DDR SDRAM (x,2x) Disk : 6,, 72,, RPM (8x, x) 26KB L2 cache CS 2 L8 Disks, RAID, (26) = bestcase) (Late ncy improve = Bandwidth improve) Annual per Technology Annual Bandwidth (all milestones) Annual Latency (all milestones) Again, CPU fastest change, DRAM slowest But what about recent, Latency change? Annual Bandwidth (last milestones) Annual Latency (last milestones) How summarize vs. Latency change? CS 2 L8 Disks, RAID, (27) CPU DRAM LAN Disk Towards a Rule of Thumb How long for Bandwidth to Double? Time for Bandwidth to Double (Years, all milestones) How much does Latency in that time? Latency in Time for Bandwidth to Double..2.. (all milestones) But what about recently? Time for Bandwidth to Double (Years, last milestones) CS 2 L8 Disks, RAID, (28) Latency in Time for Bandwidth to Double (last milestones)..2.2 Despite faster LAN, all.2x to.x 2.8. Rule of Thumb for Latency Lagging In the time that bandwidth doubles, latency improves by no more than a factor of.2 to. Stated alternatively: Bandwidth improves by more than the square of the improve in Latency (and capacity improves faster than bandwidth) 6 Reasons Latency Lags Bandwidth. Moore s Law helps more than latency Faster transistors, more transistors, more pins help Bandwidth MPU Transistors:. vs. 2 M xtors (X) DRAM Transistors:.6 vs. 26 M xtors (X) MPU Pins: 68 vs. 2 pins (6X) DRAM Pins: 6 vs. 66 pins (X) Smaller, faster transistors but communicate over (relatively) longer lines: limits latency Feature size:. to vs..8 micron (8X,7X) MPU Die Size: vs. 2 mm 2 (ratio sqrt 2X) DRAM Die Size: 7 vs. 27 mm 2 (ratio sqrt 2X) CS 2 L8 Disks, RAID, (29) CS 2 L8 Disks, RAID, ()

6 6 Reasons Latency Lags Bandwidth (cont d) 2. Distance limits latency Size of DRAM block long bit and word lines most of DRAM access time Speed of light and computers on network. & 2. explains linear latency vs. square?. Bandwidth easier to sell ( bigger=better ) E.g., Gbits/s Ethernet ( Gig ) vs. µsec latency Ethernet MB/s DIMM ( PC ) vs. ns latency Even if just marketing, customers now trained Since bandwidth sells, more resources thrown at bandwidth, which further tips the balance CS 2 L8 Disks, RAID, () 6 Reasons Latency Lags Bandwidth (cont d). Latency helps, but not vice versa Spinning disk faster improves both bandwidth and rotational latency 6 RPM RPM =.2X Average rotational latency: 8. ms 2. ms Things being equal, also helps by.2x Lower DRAM latency More access/second (higher bandwidth) Higher linear density helps disk (and capacity), but not disk Latency 9, BPI, BPI 6X in CS 2 L8 Disks, RAID, (2) 6 Reasons Latency Lags Bandwidth (cont d). Bandwidth hurts latency Queues help Bandwidth, hurt Latency (Queuing Theory) Adding chips to widen a memory module increases Bandwidth but higher fanout on address lines may increase Latency 6. Operating System overhead hurts Latency more than Bandwidth Long messages amortize overhead; overhead bigger part of short messages Ways to Cope with Latency Lags Bandwidth If a problem has no solution, it may not be a problem, but a factnot to be solved, but to be coped with over time Shimon Peres ( Peres s Law ). Caching (Leveraging Capacity) Processor caches, file cache, disk cache 2. Replication (Leveraging Capacity) Read from nearest head in RAID, from nearest site in content distribution. Prediction (Leveraging Bandwidth) Branches + Prefetching: disk, caches CS 2 L8 Disks, RAID, () CS 2 L8 Disks, RAID, () HW Example: Micro Massively Parallel Processor (µmmp) Intel (97): bit processor, 22 transistors,. MHz, micron PMOS, mm 2 chip RISC II (98): 2bit, stage pipeline,,76 transistors, MHz, micron NMOS, 6 mm 2 chip shrinks to ~ mm 2 at micron 2 mm 2 chip,.9 micron CMOS = 22 RISC IIs + Icache + Dcache RISC II shrinks to ~. mm 2 at.9 mi. Caches via DRAM or transistor SRAM ( Proximity Communication via capacitive coupling at > TB/s (Ivan Sutherland@Sun)! " # $ % & ' ( ) & * +, +. / CS 2 L8 Disks, RAID, () µ Too Optimistic so Far (its even worse)? Optimistic: Cache, Replication, Prefetch get more popular to cope with imbalance Pessimistic: These already fully deployed, so must find next set of tricks to cope; hard! Its even worse: bandwidth gains multiplied by replicated components parallelism simultaneous communication in switched LAN multiple disks in a disk array multiple memory modules in a large memory multiple processors in a cluster or SMP CS 2 L8 Disks, RAID, (6) 6

7 Conclusion: Latency Lags Bandwidth For disk, LAN, memory, and MPU, in the time that bandwidth doubles, latency improves by no more than.2x to.x improves by square of latency improve Innovations may yield onetime latency reduction, but unrelenting improve If everything improves at the same rate, then nothing really changes When rates vary, require real innovation HW and SW developers should innovate assuming Latency Lags Bandwidth CS 2 L8 Disks, RAID, (7) 7