Department of Electr rical Eng ineering, Chapter 6 Storage and Other I/O Topics 王振傑 (Chen-Chieh Wang) ccwang@mail.ee.ncku.edu.tw ncku edu Feng-Chia Unive ersity Outline 6.1 Introduction 6.2 Dependability, Reliability, and Availability 6.3 Disk Storage 6.4 Flash Storage 6.5 Connecting Processors, Memory, and I/O Devices 6.6 Interfacing I/O Devices to the Processor, Memory, and Operating System 6.9 Parallelism and I/O: RAID 2
Introduction I/O Design affected by many factors (expandability, resilience) Performance : access latency throughput connection between devices and the system the memory hierarchy the operating system A variety of different users (e.g., banks, supercomputers, engineers) Processor Cache Main memory Interrupts I/O controller Disk Memory I/O bus Disk I/O controller Graphics output I/O controller Network 3 Depar rtment of Electr rical Engineering, Feng-Chia Unive ersity I/O Devices Very diverse devices behavior ( i.e., input vs. output t ) partner ( who is at the other end? ) data rate 4
Impact of I/O on System Performance Suppose we have a benchmark that executes in 100 seconds of elapsed time, where 90 seconds is CPU time and the rest is I/O time. If CPU time improves by 50% per year for the next five years but I/O time doesn t improve, how much faster will our program run at the end of five years? The improvement in elapsed time is only 4.58 The I/O time has increased form 10% to 45% of the elapsed time. 5 Feng-Chia Unive ersity Outline 6.1 Introduction 6.2 Dependability, Reliability, and Availability 6.3 Disk Storage 6.4 Flash Storage 6.5 Connecting Processors, Memory, and I/O Devices 6.6 Interfacing I/O Devices to the Processor, Memory, and Operating System 6.9 Parallelism and I/O: RAID 6
Fault Cause and Result A representation of a defect. A fault may create a latent error which may become effective when it is activated, and the result is a failure.. Error A wrong output produced by a defective system. The cause of an error is called a fault. Error may be propagated from one unit to another unit. Failure Failure occurs because of an error. 7 Metrics for Measurement Reliability A measure of continuous service (of the time to failure) Mean time to failure (MTTF) Failure rate = 1/MTTF Overall failure rate = Sum of failure rates of the modules. Availability Module availability = MTTF/(MTTF + MTTR) MTTR: mean time to repair Dependability Quality of delivered service Fault-tolerance Using time or space redundancy to provide service in spite of the occurrences of fault 8
Feng-Chia Unive ersity Outline 6.1 Introduction 6.2 Dependability, Reliability, and Availability 6.3 Disk Storage 6.4 Flash Storage 6.5 Connecting Processors, Memory, and I/O Devices 6.6 Interfacing I/O Devices to the Processor, Memory, and Operating System 6.9 Parallelism and I/O: RAID 9 Magnetic Disk Platters Tracks Sectors Cylinder Disk access time Seek time Rotational delay (Rotational latency) Transfer time Controller time Magnetic Disks Platter Track Platters Tracks Sectors 10
Disk Read Time What is the average time to read or write a 512-byte sector for a typical disk rotating at 5400RPM? The advertised average seek time is 12ms, the transfer rate is 5MB/sec, and the controller overhead is 2ms. Assume that the disk is idle so that there is no waiting time. Average disk access time = (average seek time) + (average rotational delay) + (transfer time) + (controller overhead) 0.5R 512Byte 12 ms 2 ms 5400RPM 5MBps 12ms 5.56ms 0.1ms 2ms 19.66ms 11 Feng-Chia Unive ersity Outline 6.1 Introduction 6.2 Dependability, Reliability, and Availability 6.3 Disk Storage 64 6.4 Flash Storage 6.5 Connecting Processors, Memory, and I/O Devices 6.6 Interfacing I/O Devices to the Processor, Memory, and Operating System 6.9 Parallelism and I/O: RAID 12
Feng-Chia Unive ersity Flash Storage Nonvolatile semiconductor storage 100 1000 faster than disk Smaller, lower power, more robust But more $/GB (between disk and DRAM) 13 Flash Types NOR flash: bit cell like a NOR gate Random read/write access Used for instruction memory in embedded systems NAND flash: bit cell like a NAND gate Denser (bits/area), but block-at-a-time a time access Cheaper per GB Used for USB keys, media storage, Flash bits wears out after 10,000 s of accesses Not suitable for direct RAM or disk replacement Wear leveling: remap data to less used blocks 14
Feng-Chia Unive ersity Outline 6.1 Introduction 6.2 Dependability, Reliability, and Availability 6.3 Disk Storage 6.4 Flash Storage 6.5 Connecting Processors, Memory, and I/O Devices 6.6 Interfacing I/O Devices to the Processor, Memory, and Operating System 6.9 Parallelism and I/O: RAID 15 Depar rtment of Electrical Engineering, Feng-Chia Unive ersity Desktop PC Source : http://en.wikipedia.org/wiki/front_side_bus 16
Buses Shared communication link (one or more wires) Difficult design: may be bottleneck length of the bus number of devices tradeoffs (buffers for higher bandwidth increases latency) support for many different devices cost Types of buses: processor-memory (short high speed, custom design) backplane (high h speed, often standardized, di d e.g., PCI) I/O (lengthy, different devices, e.g., USB, Firewire) Synchronous vs. Asynchronous use a clock and a synchronous protocol, fast and small but every device must operate at same rate and clock skew requires the bus to be short don t use a clock and instead use handshaking 17 Depar rtment of Electrical Engineering, Feng-Chia Unive ersity PCI Express System Source : PCI Express Base Specification Revision 1.0a 18
I/O Bus Examples Firewire USB 2.0 PCI Express Serial ATA Serial Attached SCSI Intended use External External Internal Internal External Devices per 63 127 1 1 4 channel Data width 4 2 2/lane 4 4 Peak bandwidth Hot pluggable 50MB/s or 100MB/s 0.2MB/s, 1.5MB/s, or 60MB/s 250MB/s/lan e 1, 2, 4, 8, 16, 32 300MB/s Yes Yes Depends Yes Yes 300MB/s Max length 4.5m 5m 0.5m 1m 8m Standard IEEE 1394 USB Implementers PCI-SIG SATA-IO INCITS TC T10 Forum 19 System on a Chip (SoC) Processing Elements Microcontroller, Microprocessor, DSP Memory Elements ROM, RAM, Flash I/O & Peripherals Timer, Watchdog, UART, I 2 C, GPIO Interconnection AMBA, CoreConnect, NoC 20
Depar rtment of Electrical Engineering, Feng-Chia Unive ersity TI OMAP 1510 Platform Architecture Source : http://www.ti.com 21 Depar rtment of Electrical Engineering, Feng-Chia Unive ersity ITRI PAC Platform Architecture Source : http://int.stc.itri.org.tw 22
AMBA (1/3) The Advanced Microcontroller Bus Architecture (AMBA) was introduced in 1996 and is widely used as the on-chip bus for ARM processors. AMBA is designed for use in System-on-a-chip (SoC) designs. The important aspect of a SoC is not only which components or blocks it houses, but also how they are interconnected. AMBA is a solution for the blocks to interface with each other. 23 AMBA (2/3) Four buses/interfaces defined: ASB ( Advanced System Bus ) APB ( Advanced Peripheral Bus ) AHB ( Advanced High-performance Bus ) AXI ( Advanced extensible Interface ) The objective of the AMBA specification: be technology independent, enhance design reusability using IP cores, encourage modular system design to improve processor independence, minimize the silicon infrastructure required. 24
Feng-Chia Unive ersity AMBA (3/3) The first AMBA buses were ASB and APB. In its 2nd version, ARM introduced AHB that is a single clock-edge protocol. This protocol is today a de-facto standard for 32-bit embedded processors because it is well documented and can be used without royalties. In 2003, ARM introduced the 3rd generation of AMBA including AXI high-performance interconnect. 25 Depar rtment of Electrical Engineering, Feng-Chia Unive ersity AMBA 2.0 26
Depar rtment of Electrical Engineering, AXI Features : AMBA3 AXI separate address/control and data phases support for unaligned data transfers using byte strobes separate read and write data channels to enable low-cost DMA ability to issue multiple outstanding addresses out-of-order transaction completion 27 Depar rtment of Electrical Engineering, Advanced High-performance Bus (AHB) AHB Features Multiple master ( up to 16 masters ) Burst transfers Split transactions Single cycle bus master handover Single clock edge operation Non-tristate implementation Wider data bus configurations. 28
Depar rtment of Electrical Engineering, Feng-Chia Unive ersity Advanced High-performance Bus (AHB) Source : AMBA Specification (Rev 2.0), 1999. http://www.arm.com 29 AMBA AHB-based system CPU Core IntMem (64KB SRAM) MuxM2S MuxS2M CPU Master AHB Slave Wrapper Wrapper Default Slave APBif External Memory Controller Arbiter Reset controller Decoder AHB Master Wrapper IP AHB Slave Wrapper 30
ReadReq 1 Data Ack DataRdy Handshaking protocol 2 2 3 4 4 5 6 7 ReadReq Memory Data Ack I/O Device 1. When memory sees the ReadReq line, it reads the address from the Data bus and raises Ack to indicate it has been seen. 2. I/O device sees the Ack line high and releases the ReadReq and Data lines. 3. Memory sees that ReadReq is low and drops the Ack line to acknowledge the ReadReq signal. 4. This step starts when the memory has the data ready. It places the data from the read request on the Data lines and raises DataRdy. 5. The I/O device sees DataRdy, reads the data from the bus, and signals that it has the data by raising Ack. 6. The memory sees the Ack signal, drops DataRdy, and releases the data lines. 7. Finally, the I/O device, seeing DataRdy go low, drops the Ack line, which indicates that the transmission is completed. DataRdy 31 Department of Electrical Engineering, Feng-Chia Unive ersity 32
Feng-Chia Unive ersity Outline 6.1 Introduction 6.2 Dependability, Reliability, and Availability 6.3 Disk Storage 6.4 Flash Storage 6.5 Connecting Processors, Memory, and I/O Devices 6.6 Interfacing I/O Devices to the Processor, Memory, and Operating System 6.9 Parallelism and I/O: RAID 33 I/O Interface Giving commands to I/O devices Memory-mapped I/O (MMIO) Special I/O instructions Communicating with the processor Polling Interrupt-driven I/O Transferring the data between a device and memory CPU Direct Memory Access (DMA) 34
Depar rtment of Electrical Engineering, Feng-Chia Unive ersity System on a Chip (SoC) 35 Feng-Chia Unive ersity Outline 6.1 Introduction 6.2 Dependability, Reliability, and Availability 6.3 Disk Storage 6.4 Flash Storage 6.5 Connecting Processors, Memory, and I/O Devices 6.6 Interfacing I/O Devices to the Processor, Memory, and Operating System 6.9 Parallelism and I/O: RAID 36
RAID Redundant Arrays of Inexpensive Disks (RAID) RAID 0 : No Redundancy RAID 1 : Mirroring RAID 2 : Error Detecting and Correcting Code RAID 3 : Bit-Interleaved Parity RAID 4 : Block-Interleaved Parity RAID 5 : Distributed Block-Interleaved Parity References : http://en.wikipedia.org/wiki/raid http://www.acnc.com/04_00.html com/04 00 html 37 RAID 0 : No Redundancy Striping RAID 1 : Mirroring ( shadowing ) RAID 0 & RAID 1 Striping Strip 0 Strip 1 Strip 2 Strip 3 Strip 4 Strip 5 Strip 6 Strip 7 Strip 8 Strip 9 Strip 10 Strip 11 Strip 12 Strip 13 Strip 14 Strip 15 38
Depar rtment of Electr rical Engineering, RAID 01 (mirrored stripes) RAID 10 (striped mirrors) RAID 0 + RAID 1 Striping Mirroring Mirroring Mirroring Mirroring Strip 0 Strip 0 Strip 1 Strip 1 Strip 2 Strip 2 Strip 3 Strip 3 Strip 4 Strip 4 Strip 5 Strip 5 Strip 6 Strip 6 Strip 7 Strip 7 Strip 8 Strip 8 Strip 9 Strip 9 Strip 10 Strip 10 Strip 11 Strip 11 Strip 12 Strip 12 Strip 13 Strip 13 Strip 14 Strip 14 Strip 15 Strip 15 39 RAID 2 & RAID 3 RAID 2 : ECC ( ex: Hamming Code ) b 0 b 1 b 2 b 3 F 0(b) F 1(b) F 2(b) RAID 3 : Bit-Interleaved Parity 40
RAID 3 D0 (i) D0(i) P (i) = D0 (i) D1(i) D2(i) D3(i) P (i) P(i) (N-1) Reads, 2 Writes RAID 4 / RAID 5 Update a Disk D0 (i) D0(i) P (i) = D0 (i) D1(i) D2(i) D3(i) = D0 (i) D1(i) D2(i) D3(i) 0 = D0 (i) D1(i) D2(i) D3(i) D0(i) D0(i) = D0 (i) P(i) D0(i) P (i) P(i) 2R Reads, 2W Writes 41 RAID 4 & RAID 5 RAID 4 : Block-Interleaved Parity Block 0 Block 1 Block 2 Block 3 P(0-3) Block 4 Block 5 Block 6 Block 7 P(4-7) Block 8 Block 9 Block 10 Block 11 P(8-11) Block 12 Block 13 Block 14 Block 15 P(12-15) Block 16 Block 17 Block 18 Block 19 P(16-19) RAID 5 : Distributed Block-Interleaved Parity Block 0 Block 1 Block 2 Block 3 P(0-3) Block 4 Block 5 Block 6 P(4-7) Block 7 Block 8 Block 9 P(8-11) Block 10 Block 11 Block 12 P(12-15) Block 13 Block 14 Block 15 P(16-19) 19) Block 16 Block 17 Block 18 Block 19 42
RAID 6: P + Q Redundancy N + 2 disks Like RAID 5, but two lots of parity Greater fault tolerance through more redundancy Multiple RAID More advanced systems give similar fault tolerance with better performance 43 RAID Summary RAID can improve performance and availability High availability requires hot swapping Assumes independent disk failures Too bad if the building burns down! See Hard Disk Performance, Quality and Reliability http://www.pcguide.com/ref/hdd/perf/index.htm 44