CS 152 Computer Architecture and Engineering

Transcription

1 CS 152 Computer Architecture and Engineering Lecture 23 Buses, Disks, and RAID John Lazzaro ( TAs: Ted Hong and David Marquardt www-inst.eecs.berkeley.edu/~cs152/

2 Last Time: Multithreading, Multiple Cores (1) Threads on two cores that use shared libraries conserve L2 memory. (2) Threads on two cores share memory via L2 cache operations. Much faster than 2 CPUs on 2 chips.

3 Today: Buses, Disks, and RAID Buses: Shared physical wires that act to communicate signals between several devices (often peripherals ) Buses let computers be expandable: add more memory, a better graphics card, a webcam, etc. Disks: Store bits as the orientation of miniature bar magnets on a rotating platter. A mechanical device: slow and prone to failure.

4 Properties of bus structures... Control lines: Controls transactions, signals what is on data lines T0 T1 T2 T3 T4 T5 T6 CLK COMMAND READ NOP NOP NOP READ NOP NOP ADDRESS BANK, COL n BANK, COL b X = 1 cycle DQ DOUT n DOUT n + 1 DOUT n + 2 DOUT n + 3 DOUT b CAS Latency = 2 Data lines: Carries information across the interface Buses are an abstraction for communication: helps designers compose large, complex systems.

5 Buses are defined in layers A1 A2 34 A4 35 A6 36 A8 37 A10 38 A12 39 VDD 40 VDD 41 CK0 42 VSS 43 NU 44 S2 45 DQMB2 46 DQMB3 47 NU 48 VDD 49 CB10 NC 50 CB11 NC 51 CB2 NC 52 CB3 NC 53 VSS 54 DQ16 55 DQ17 56 DQ18 57 DQ19 58 VDD 59 DQ20 60 NC, MWAIT 61 VREF, NC 62 CKE1 63 VSS 64 DQ21 65 DQ22 66 DQ23 67 VSS 68 DQ24 69 DQ25 70 DQ26 71 DQ27 72 VDD 73 DQ28 74 DQ29 75 DQ30 76 DQ31 77 VSS 78 CK2 79 NC 80 NC 81 SDA 82 SCL 83 VDD 84 X80 ECC MODE DIMM X72 ECC MODE DIMM X64 DIMM FRONT SIDE REAR SIDE 118 A3 119 A5 120 A7 121 A9 122 A A VDD 125 CK1 126 A VSS 128 CKE0 129 S3 130 DQMB6 131 DQMB7 132 A VDD 134 NC CB NC CB NC CB6 137 NC CB7 138 VSS 139 DQ DQ DQ DQ VDD 144 DQ NC, MIRQ 146 VREF, NC 147 NC 148 VSS 149 DQ DQ DQ VSS 153 DQ DQ DQ DQ VDD 158 DQ DQ DQ DQ VSS 163 CK3 164 NC 165 SA0 166 SA1 167 SA2 168 VDD Example: DIMM DRAM bus. The name of every wire is defined in a standards document. Transaction Protocols Signal Timing on Wires Wires Electrical Properties Mechanical Properties JEDEC: Joint Electron Device Engineering Council. Makes the DRAM bus standards.

6 Lower levels of DRAM bus specification SDRAM Pin L4 L3 L2 Loaded Clock Transaction Protocols SDRAM Pin L4 L3 L1 L0 SO-DIMM Connector Signal Timing on Wires SDRAM Pin SDRAM Pin L4 L4 L3 L3 L2 Unloaded Clock 10 Ohm SO-DIMM Connector Wires 10 pf Stuff for Unloaded Clock Table 9: Trace Length Table for Clock Topologies Segment L0 L1 L2 L3 L4 Total Min Total Max Length Tolerance ± 0.05 ± 0.02 ± 0.02 ± 0.02 ± 0.05 Layer Outer Inner Inner Inner Outer 1 All distances are given in inches and should be kept within tolerance, and routed on the indicated Electrical Properties Mechanical Properties 67.6mm (2.66") Ideally, DIMMs made by any manufacturer should fit into any compliant socket, and work. 54 pin or TSOP (II) 54 pin or TSOP (II) SP D 54 pin or TSOP (II) 54 pin or TSOP (II) (31.75mm 1.25"

7 ( ( Upper levels of DRAM bus specification ) ) Figure 13: Consecutive READ Bursts CLK COMMAND ADDRESS DQ tas T0 BANK tah T1 T2 DOUT n BANK T3 READ NOP NOP NOP READ NOP NOP BANK, COL n DOUT n + 1 T4 BANK, COL b X = 1 cycle DOUT n + 2 T5 DOUT n + 3 T6 DOUT b ( ) ( ) ( ( ) ) ( ( Collaboration ) ) between DRAM manufacturers (Samsung, Micron) and DRAM users (Intel, Cisco,... ). 3 CAS Latency = 2 Transaction Protocols TIM ING PARAM ETERS CLK T0 T1-7E -75 SYM BOL* M IN M AX M IN M AX UNITS t AC (3) ns COMMAND t AC (2) 5.4 NOP 6 READ ns t AH ns t AS ns t CH ns t BANK, BANK, ADDRESS CL ns t COL n COL b CK (3) ns t CK (2) ns t DOUT DOUT CKH DQ ns T2 T3 READ NOP NOP NOP NOP n T4 n + 1 T5 X = 2 cycles DOUT n + 2 T6 Signal Timing on Wires -7E -75 SYM BOL* M IN M AX M IN M AX t CKS Wires t CM H NOP t CM S t HZ (3) Electrical Properties t HZ (2) t LZ 1 1 t OH 3 3 Mechanical t RCD DOUT DOUT Properties n + 3 T7 b

8 Bus wires shared between many DIMMS Apple Xserve G5 - has 8 DIMM slots, to support 8GB. Processor module 64-bit PowerPC G5 microprocessor DIMMs respond to transaction requests. Since memory controller is only bus master, and there are a small number of DIMM slots, bus sharing is easy: use dedicated wires to each slot. Main logic board DIMM slots 400 MHz ECC DDR memory bus 16-bit 4.8 GBps Hyper Transport U3H memory controller and PCI bus bridge Pro bu pro Memory controller is the only bus master - it can start transactions on the bus, but the DIMMs cannot.

9 Buses: pros and cons Low cost. One set of wires from memory controller can support up to 8 DIMMs. Processor module 64-bit PowerPC G5 microprocessor --- Latency of bus increases with length of wires (needed to reach all 8 DIMM sockets), and the loading of 8 DIMMs. Must design for worst-case (8 DIMMs), even if only 1 DIMM is present. Main logic board DIMM slots 400 MHz ECC DDR memory bus 16-bit 4.8 GBps Hyper Transport U3H memory controller and PCI bus bridge Pro bu pro --- Shared wires limit maximum bandwidth from memory. If memory controller had 8 sets of dedicated wires, one per DIMM, memory bandwidth would be much better (but more expensive).

10 Buses turn a CPU into a product Case Study: Mac Mini

11 Constraints: Size, low price (499 USD) Size fixed by the form factor (physical size) of desktop DIMMS. Laptop DRAM is smaller, but too expensive for $499 price.

12 Users expansion via serial buses Serial: Data is sent bit by bit over one logical wire. Serial pros and cons: +++ Sending data over many wires introduces skew - signals travel on each wire at a slightly different speed. Skew limits speed and length of a bus. Serial buses have fewer skew issues, because they only use one logical wire. +++ Low cost: a small number of wires cost less. Also, cheap wires and connectors can be used, since skew is less a problem. USB, FireWire Ethernet. --- When only using one wire, there is a bandwidth limit. Thus, DIMMs uses many wires (a parallel bus, not serial ).

13 Many other buses hidden from user Processor bus. How the CPU talks to everything else. Not standardized. Bus controller. Just 1 for low cost. High-end products have two: fast North Bridge, slow South Bridge. DDR SDRAM DIMM slot Hard disk drive Optical drive PMU power controller Power button Bluetooth 167 MHz Memory bus Device 0 Ultra ATA/100 bus Device 1 Fan I2C 12 Mbps USB PowerPC G4 microprocessor (L2 cache: 512K 1:1) 167 MHz MaxBus Intrepid memory controller and I/O device controller AGP 4X bus I2S Built-in speaker PCI bus I2S Radeon 9200 graphics IC 32 MB DDR RAM FireWire PHY Ethernet PHY Audio codec AirPort Extreme Boot ROM PCI USB 2.0 controller Data pump and DAA Modem module CPU: PowerPC G4 (Freescale) DVI/VGA/composite/S-video output port FireWire 400 port Ethernet port 10/100 Mbps Headphone/audio line-out jack USB 2.0 port (480 Mbps) USB 2.0 port (480 Mbps) Modem port

14 Uses many standard parallel buses... AGP 4X bus. Graphics chip. ATA/100 bus. For hard disk, DVD/CD ROM. PCI, ATA, AGP devices can be bus master, for Direct Memory Access (DMA). Disk can write RAM directly. DDR SDRAM DIMM slot Hard disk drive Optical drive PMU power controller Power button Bluetooth 167 MHz Memory bus Device 0 Ultra ATA/100 bus Device 1 Fan I2C 12 Mbps USB PowerPC G4 microprocessor (L2 cache: 512K 1:1) 167 MHz MaxBus Intrepid memory controller and I/O device controller AGP 4X bus I2S Built-in speaker PCI bus I2S Radeon 9200 graphics IC 32 MB DDR RAM FireWire PHY Ethernet PHY Audio codec AirPort Extreme Boot ROM PCI USB 2.0 controller Data pump and DAA Modem module PCI bus: Boot ROM, USB 2. DVI/VGA/composite/S-video output port FireWire 400 port Ethernet port 10/100 Mbps Headphone/audio line-out jack USB 2.0 port (480 Mbps) USB 2.0 port (480 Mbps) Modem port

15

16

17

18 Reminder: Friday DRAM controller checkoff Run your test vector suite on the Calinx board, display results on LEDs. T e s t IM Bus D R A M Test ongoing transactions on both buses, with randomized start times. Load, store, and verify different data word patterns. V e c t o r s DM Bus C o n t r o l l e r DRAM

19 Disks

20 Trick: use bar magnets to code bits Symbol N S

21 Write and read bar direction on a disk Longitudinal Recording: Today s technology. Magnets tend to erase each other. Perpendicular Recording: Disk surface Coming soon. 01 0

22 Read head signal from spinning disk...

23 Data block written on a sector of a track Each ring is a track. A track is divided into sectors. A sector codes a fixed # of bytes (ex: 4K blocks). Outer tracks hold more sectors. Many more tracks and sectors than shown! 2005 desktop rotation speed: 7200 RPM

24 Read/write head mounted on arm A seek : When the arm is moved so that the heads are over the desired track. Seek time : Average time to move from one track to another track. Pessimistic estimate of real-world performance desktop drive typical seek time: 8.5 milliseconds.

25 Disk Latency Equation Latency of a disk block read = Queueing Time + Controller Time + Seek Time + Rotation Time + Transfer Time Zero if no other accesses pending. Usually short. 2005: about 8 ms RPM 1/2 full rotation time RPM

26 RAID Redundant Array of Inexpensive Disks A Case for Redundant Arrays of Inexpensive Disks (RAID) Davtd A Patterson, Garth Gibson, and Randy H Katz Computer Saence D~v~smn Department of Elecmcal Engmeermg and Computer Sclencea 571 Evans Hall Umversity of Cabforma Berkeley. CA (partrsl@wf -ky du)

27 Why disk arrays? Reliability + throughput host with duplicated paths, higher performance can be obtained when there are no failures I/O Controller Fully dual redundant host I/O Controller Goal: No Single Points of Failure Array Controller Array Controller Recovery Groups (columns) Disk(s) can fail in a group w/o losing access to the data stored in the group.

28 Raid Level: Recovery Group Organization D0 5-disk Recovery Group RAID 1: Disk Mirroring B0 B1 B2 Each disk holds a copy of each block. D1 D2 B0 B0 B1 B1 B2 B2 +++ High availability. D3 B0 B1 B2 +++ High read bandwidth. D4 B0 B1 B2 Logical Blocks Bn on Array --- High disk capacity cost.

29 Recall from ECC lecture: Parity Codes Simple case: Two 1KB blocks of data (A and B) Create a parity block, P: Parity codes P = A xor B (do xor on each bit of block) Read all three blocks. If A or B is not available but P is, regenerate A or B: A = P xor B B = P xor A The math is easy: the trick is system design! Examples: RAID, voice-over-ip parity FEC.

30 For lower capacity cost: parity codes 5-disk Recovery Group RAID 3: Parity Disk 80% of disks hold unique data D0 B0 B4 B8 +++ Low cost. D1 D2 B1 B2 B5 B6 B9 B Only one disk may fail. D3 B3 B7 B Pdisk limits write bandwidth Parity P0 P1 P2 Logical Blocks Bn on Array --- No read bandwidth gain.

31 Improve write performance: Interleave P D0 D1 5-disk Recovery Group RAID 5: Interleaved Parity Disks B0 B1 B4 B5 B8 B9 +++ Writes of parity blocks distributed across 5 disks D2 D3 Parity B2 B6 B3 P1 P0 B7 P2 B10 B11 Logical Blocks Bn on Array COD/e3 Page for RAID details. Will be responsible for level of detail in book for Mid-term II.

32 Conclusions: Buses, Disks, and RAID Buses make the machine. Value and high-end machines may have similar CPUs, different chip-sets. Disks are different: As mechanical devices, they move on a millisecond time frame, and suffer mechanical failure. System design must cope with this reality, not ignore it.