Input/Output. q Principles of I/O hardware & software q Secondary storage q Next: File systems

Transcription

1 Input/Output q Principles of I/O hardware & software q Secondary storage q Next: File systems

2 Operating systems and I/O Two key OS goals with respect to I/O Control I/O devices Provide a simple, easy-to-use, interface to them Problem a wide range of devices Data rates From 10B/sec (keyboard) to 12,800MB/sec (100 Gigabit Ethernet) Complexity of control A printer (simple) or a disk Units of transfer Character or block devices Error condition nature of errors, reporting, consequences, Difficult to get a uniform & consistent approach 2

3 I/O devices I/O devices components Device itself mechanical component Device controller or adapter electronic component Device controller Converts serial bit stream to block of bytes Performs error correction as necessary Makes data available in main memory Maybe more than one device per controller Some standard interface between controller and devices: IDE, ATA, SATA, SCSI, FireWire, Thunderbolt, 3

4 I/O controller & CPU communication Device controllers have A few registers for communication with CPU Data-in, data-out, status, control, command, A buffer that OS can read/write (e.g. video RAM) Registers Status Command Data Interface Micro-controller (CPU) Memory (DRAM or SRAM or both) Other hardware specific chips Internals A canonical device 4

5 The OS/device interaction A typical interaction While (STATUS == BUSY) ; // wait until devices is free Write data to DATA register Write command to COMMAND register // this starts the device and executes the command While (STATUS == BUSY) ; // wait until is done with your request This model of waiting until ready is call polling When the main CPU is involved in this transfer of data, we call this programmed I/O Simple but polling is very inefficient wasting CPU cycles 5

6 Interrupts to the rescue Instead of waiting, the OS issues a requests, puts the process to sleep and context switches When device is done, raises an interrupt causing CPU to jump to an interrupt handler to finish the request Much better but not always the best, with a fast device it may be best polling Or perhaps a hybrid model polling for a bit and then use interrupts 6

7 Moving data with DMA Another issue with our canonical protocol transferring large chunk of data also wastes CPU as it waits for the copies Direct Memory Access DMA) Another device to take over this OS tells DMA where the data is, much to copy and where to send it to DMA does the work; raises an interrupt when done 7

8 Direct Memory Access OS can use it only if HW has DMA controller Either on the devices (controller) or the parentboard DMA controller has access to the system bus, independent of CPU Various registers accessible by the CPU CPU program the DMA controller DMA controller CPU issues command to disk read, to internal buffer and check DMA operation Address Buffer CPU Count Control + ACK Disk controller Main memory Interrupt when done DMA requests xfer to mem. Data xfer 8

9 The OS/device interaction Issue read command to I/O module Read status of I/O module CPU I/O I/O CPU Issue read command to I/O module Read status of I/O module CPU I/O Do something else I/O CPU Interrupt Issue read command to I/O module Read status of I/O module CPU DMA Do something else DMA CPU Interrupt Not ready Check status Error Check status Error Ready Ready Read word from I/O module I/O CPU Read word from I/O module I/O CPU Write word into memory Write word into memory CPU Mem No Done? No Done? Yes Programmed I/O Yes Interrupt-driven I/O Direct Memory Access 9

10 How does the OS talks with the device a. Separate I/O and memory space, each control register assigned an I/O port and special I/O instructions IBM 360 (IN REG,PORT) Memory Two addresses spaces I/O Ports b. Memory-mapped I/O Mapped all control registers into the memory space; first in PDP-11 One address space Single address spaces c. Hybrid Pentium (e.g., graphic controller) Two addresses spaces 10

11 Memory-mapped I/O pros and cons No special instructions or protection mechanism needed Instruction that can reference memory can reference control registers Write the software that interact with devices entirely in C Caching? Disable it on a per-page basis One AS, so all memory modules and all I/O devices must check all references Easy with single bus, harder with dual-bus arch High-bandwidth memory bus CPU Mem I/O 11

12 Drivers for abstraction How to fit different devices into a general OS? Device drivers abstracts the specifics E.g.., File system and apps oblivious of disk details Application POSIX API [open, close, read, write, ] File system Generic block i/f [block read/write] Generic block layer Specific block i/f [protocol specific read/write] Device driver [SCSI, ATA, etc] User Kernel mode But may hide some useful capabilities Because every device needs a driver >70% of OS code Not written by kernel hackers, so more bugs per line 12

13 An IDE disk driver A simple interface to the system Four type of registers: control, command block, status, error Available for r/w to specific I/O addresses (e.g., 0x3F6) with in and out instructions Control register: Address 0x3F6 = 0x08 (0000 1RE0): R=reset, E=0 means enable interrupt Command block registers: Address 0x1F0 = Data Port Address 0x1F1 = Error Address 0x1F2 = Sector Count Address 0x1F3 = LBA low byte Address 0x1F4 = LBA mid byte Address 0x1F5 = LBA hi byte Address 0x1F6 = 1B1D TOP4LBA: B=LBA D=drive Status Register (Address 0x1F7: BUSY READY FAULT SEEK DRQ CORR IDDEX ERROR Error register (Address 0x1F1): (check when Status ERROR = 1) BBK UNC MC IDNF MCR ABRT TONF AMNF 13

14 Basic protocol Assuming it has been initialized What for drive to be ready Read status register until drive is READY and not BUSY Write parameters to command registers Write the sector count, logical block address (LBA) of the sector to be accessed, and drive number to command registers Start the I/O by issuing r/w to command register Data transfer (for writes) Wait until drive status is READY and DRQ (drive request for data); write data to data port Handle interrupts Simplest case, handle interrupt for each sector transferred,.. Error handling After each operation, check status register for ERROR 14

15 Look at the xv6 IDE driver Add the read/write request to the queue If this is the only request then start it else it will be started in time Wait for the request to complete void ide_rw(struct buf *b) { aqcuire(&ide_lock); for (struct buf **p = &ide_queue; *pp; pp=&(*pp)->qnext) ; // walk queue *pp = b; // add request to end if (ide_queue == b) // if q is empty ide_start_request(b); // send req to disk while ((b->flags & (B_VALID B_DIRTY))! B_VALID) sleep(b, &ide_lock); // wait for completion release(&ide_lock); } 15

16 Look at the xv6 IDE driver static int ide_wait_ready() { while (((int r = inb(0x1f7)) & IDE_BSY)!(r & IDE_DRDY)) ; // loop until drive isn t busy } static void ide_start_request(struct buf *b) { ide_wait_ready(); outb(0x3f6, 0); // generate interrupt outb(0x1f2, 1); // how many sectors? outb(0x1f3, b->sector & 0xff); // Logical Block Address here outb(0x1f4, (b->sector >> 8) & 0xff); //.. and here outb(0x1f5, (b->sector >> 16) & 0xff); //.. and here outb(0x1f6, 0xe0 ((b->dev&1)<<4) ((b->sector>>24)&0x0f)); if(b->flags & B_DIRTY) { outb(0x1f7, IDE_CMD_WRITE); // this is a WRITE outs1(0x1f0, b->data, 512/4); // transfer data too! } else { outb(0x1f7, IDE_CMD_READ); // this is a READ (no data) } } 16

17 Look at the xv6 IDE driver Eventually, disk will finish its operation and trigger an interrupt. trap will call ideintr to handle it. If buffer was being read and disk controller has data waiting, reads data into buffer with insl void ide_intr() { struct buf *b; acquire (&ide_lock); if (!(b->flags & B_DIRTY) && ide_wait_ready() > 0) insl(0x1f0, b->data, 512/4); // if READ: get data b->flags!= B_VALID; b->flags &= ~B_DIRTY; wakup(b); // wake waiting process if ((ide_queue = b->qnext)!= 0) // start next request ide_start_request(ide_queue); // (if one exists) release(&ide_lock); } Now buffer is ready, sets B_VALID, clears B_DIRTY, and wakes up any process sleeping on buffer 17

18 Now a short break 18

19 Disk A concrete I/O device The disk surface spins at a fixed rotational rate ( RPM) Platters with two surfaces sector tracks spindle spindle spindle spindle The read/write head is attached to the end of the arm and flies over the disk surface on a thin cushion of air. By moving radially, the arm can position the read/write head over any track. arm spindle read/write heads move in unison from cylinder to cylinder 19

20 Where the time goes Move the head to the write track seek time (T seek ) On a Cheetah 15K.5 T seek = 4 ms And wait for the sector to go under rotational delay (T rotation ) T rotation = T max rotation /2 = 1/RPMs x 60 sec/1 min x 1/2 15,00 RPMs or 2 ms Now transfer the data (T transfer ) 30 microsecs Total I/O time T I/O = T seek + T rotation +T transfer 6 ms Rate of I/O - size of the transfer / total I/O time 4KB reads / 6ms ~ 0.66 MB/s Random Sequential (a bunch of sectors at once) ~ 125 MB/s 20

21 Disk arm scheduling With disk scheduling easier to estimate time Time to read/write a disk block dominated by seek If request come one at a time, little you can do - FCFS Starting at 53 Requests: 98,183,37,122, 14,124,65,67 21

22 SSTF Given a queue of request for blocks scheduling to reduce head movement Starting at 53 Requests: 98,183,37,122, 14,124,65,67 As SJF, possible starvation 22

23 SCAN, C-SCAN and C-LOOK SCAN C-SCAN Nice, but no need to be blind Starting at 53 Requests: 98,183,37,122, 14,124,65,67 C-LOOK Assuming a uniform distribution of requests, where s the highest density when head is on the left? 23

24 Some scheduling issues Where is scheduling done, OS or disk controller? Originally the OS In modern systems, the controller has all the information and can do it OS just issue a bunch of requests at once How long should the system wait before issuing a new I/O request to the disk? Don t let the device idle work-conserving Wait a bit for a better request non-work-conserving Couldn t the scheduler merge requests? Go over a set of request and merge consecutive ones if possible 24

25 Redundant Array of Inexpensive Disks Independent Disk transfer rates are improving, but slower than CPU performance Use multiple disks to improve performance Strip content across multiple disks Use parallel I/O to improve performance To the FS, just another disk an array of blocks Only bigger, potentially faster and more reliable Evaluating RAID designs Capacity with N disks of size B blocks, what is the available capacity? Reliability how many disk faults can we tolerate? Performance 25

26 RAID 0: Striping Files (range of sectors) are striped across; not really a RAID level, no redundancy But upper bound on performance and capacity (N*B) High read throughput Best write throughput (nothing extra to write) Chunk size is key Small more files striped across many disks, but worst access time But striping reduces reliability (n*mtbf) Add redundancy for reliability Disk 0 Disk 1 Disk 2 Disk

27 RAID 1: Mirroring Files are striped some of the disks and data is written in 2+ places Read from either copy On failure, use the surviving one Two models with four disks RAID 1-0: Mirror then stripes RAID 0-1: Stripes then mirror Of course you need at least 2x space (N*B/2) Read latencies of one disk; writes need two writes to complete and so a bit slower Disk 0 Disk 1 Disk 2 Disk 3 Consistency? Atomicity with write-ahead log

28 RAID 4: Redundancy with parity Like RAID 0 with a strip-for-strip parity written onto an extra drive Parity with a simple XOR Across all bits of the set of blocks Tolerates one disk failure With small capacity cost, (N 1)*B Reads Disk 0 Disk 1 Disk 2 Disk Disk 4 P0 P1 P2 28

29 RAID 5: Distributed parity Block interleaved distributed parity Distribute parity info over all disks Similar capacity and fault tolerance to RAID 4 Much better performance (no hot spot), particularly random So, typically used instead of RAID 4 Disk 0 Disk 1 Disk 2 Disk P1 Disk 4 P P

30 RAIDs other issues RAID 2, 3 work on word (or byte) basis Extra EEC bits added to parts of a word and distributed over all disks RAID 2 A single parity bit computed per word and written to the parity disk RAID 3 Software instead of hardware RAID Tradeoffs Granularity (multiple blocks or words) and redundancy (across all disks or just a few) 30

31 Next time File systems Interface Implementation And a number of good examples 31

32 Look at the xv6 IDE driver (v9) // Sync buf with disk. // If B_DIRTY is set, write buf to disk, clear B_DIRTY, set B_VALID. // Else if B_VALID is not set, read buf from disk, set B_VALID. void iderw(struct buf *b) { struct buf **pp; if(!(b >flags & B_BUSY)) panic("iderw: buf not busy"); if((b >flags & (B_VALID B_DIRTY)) == B_VALID) panic("iderw: nothing to do"); if(b >dev!= 0 &&!havedisk1) panic("iderw: ide disk 1 not present"); acquire(&idelock); // Append b to idequeue. b >qnext = 0; for(pp=&idequeue; *pp; pp=&(*pp) >qnext) ; *pp = b; // Start disk if necessary. if(idequeue == b) idestart(b); First request? Start it } // Wait for request to finish. while((b >flags & (B_VALID B_DIRTY))!= B_VALID){ sleep(b, &idelock); } Instead of polling, go to sleep, waiting for the interrupt handler release(&idelock); to record in buffer s flags that operation is done 32

33 Look at the xv6 IDE driver (v9) // Start the request for b. Caller must hold idelock. static void idestart(struct buf *b) { if(b == 0) panic("idestart"); if(b >blockno >= FSSIZE) panic("incorrect blockno"); int sector_per_block = BSIZE/SECTOR_SIZE; int sector = b >blockno * sector_per_block; int read_cmd = (sector_per_block == 1)? IDE_CMD_READ : IDE_CMD_RDMUL; int write_cmd = (sector_per_block == 1)? IDE_CMD_WRITE : IDE_CMD_WRMUL; if (sector_per_block > 7) panic("idestart"); } idewait(0); outb(0x3f6, 0); // generate interrupt outb(0x1f2, sector_per_block); // number of sectors outb(0x1f3, sector & 0xff); outb(0x1f4, (sector >> 8) & 0xff); outb(0x1f5, (sector >> 16) & 0xff); outb(0x1f6, 0xe0 ((b >dev&1)<>24)&0x0f)); if(b >flags & B_DIRTY){ outb(0x1f7, write_cmd); outsl(0x1f0, b >data, BSIZE/4); } else { outb(0x1f7, read_cmd); } Getting these details right is one reason why writing device drivers is challenging 33

34 Look at the xv6 IDE driver (v9) // Interrupt handler. void ideintr(void) { struct buf *b; // First queued buffer is the active request. acquire(&idelock); if((b = idequeue) == 0){ release(&idelock); // cprintf("spurious IDE interrupt\n"); return; } idequeue = b >qnext; Eventually, disk will finish its operation and trigger an interrupt. trap will call ideintr to handle it. Look at first buffer in queue to see which operation was happening // Read data if needed. if(!(b >flags & B_DIRTY) && idewait(1) >= 0) insl(0x1f0, b >data, BSIZE/4); // Wake process waiting for this buf. b >flags = B_VALID; b >flags &= ~B_DIRTY; wakeup(b); // Start disk on next buf in queue. if(idequeue!= 0) idestart(idequeue); If buffer was being read and disk controller has data waiting, reads data into buffer with insl Now buffer is ready, sets B_VALID, clears B_DIRTY, and wakes up any process sleeping on buffer } release(&idelock); Finally, pass next waiting buffer to disk 34