Compsci 104. Duke University. Input/Output. Instructors: Alvin R. Lebeck. Some Slides provided by Randy Bryant and Dave O Hallaron and G.

Transcription

1 Input/Output Instructors: Alvin R. Lebeck Some Slides provided by Randy Bryant and Dave O Hallaron and G. Kedem 1

2 Administrivia HW #7 up, Due Monday Dec 5 (no late submissions) Work on processor project, Due Dec 7 (no late submissions) Final Exam: Sunday Dec 18, 2pm- 5pm Alex will have a review session This is where you should ask about anything from the sample final I will post 2

3 Overview Core i7 Virtual Memory I/O devices device controller Device drivers Memory Mapped I/O Programmed I/O Direct Memory Access (DMA) RotaRonal media (disks) 3

4 Review: Virtual Memory Virtual Address (VA) Physical Address (PA) Process = virtual address space + thread of control TranslaRon VA - > PA What physical address does virtual address A map to Is VA in physical memory? ProtecRon (access control) Do you have permission to access it? CPS 104 4

5 Review: Page Tables A page table is an array of page table entries (PTEs) that maps virtual pages to physical pages. Per- process kernel data structure in DRAM Valid PTE PTE 7 1 Physical page number or disk address null null Memory resident page table (DRAM) Physical memory (DRAM) VP 1 VP 2 VP 7 VP 4 Virtual memory (disk) VP 1 VP 2 VP 3 VP 4 VP 6 VP 7 PP 0 PP 3 5

6 Review: IntegraRng VM and Cache PTE CPU Chip CPU VA MMU PTEA PA PTEA hit PTEA miss PA miss PTE PTEA PA Memory PA hit Data Data L1 cache VA: virtual address, PA: physical address, PTE: page table entry, PTEA = PTE address 6

7 Review: Speeding up TranslaRon with a TLB Page table entries (PTEs) are cached in L1 like any other memory word PTEs may be evicted by other data references PTE hit sqll requires a small L1 delay SoluRon: Transla@on Lookaside Buffer (TLB) Small hardware cache in MMU Maps virtual page numbers to physical page numbers Contains complete page table entries for small number of pages 7

8 Intel Core i7 Memory System Processor package Core x4 Registers InstrucRon fetch MMU (addr translaron) L1 d- cache 32 KB, 8- way L1 i- cache 32 KB, 8- way L1 d- TLB 64 entries, 4- way L1 i- TLB 128 entries, 4- way L2 unified cache 256 KB, 8- way L2 unified TLB 512 entries, 4- way QuickPath interconnect GB/s each To other cores To I/O bridge L3 unified cache 8 MB, 16- way (shared by all cores) DDR3 Memory controller 3 x GB/s 32 GB/s total (shared by all cores) Main memory 8

9 Review of Symbols Basic Parameters N = 2 n : Number of addresses in virtual address space M = 2 m : Number of addresses in physical address space P = 2 p : Page size (bytes) Components of the virtual address (VA) TLBI: TLB index TLBT: TLB tag VPO: Virtual page offset VPN: Virtual page number Components of the physical address (PA) PPO: Physical page offset (same as VPO) PPN: Physical page number CO: Byte offset within cache line CI: Cache index CT: Cache tag 9

10 End- to- end Core i7 Address TranslaRon VPN TLB miss CPU VPO 32 4 TLBT TLBI Virtual address (VA)... TLB hit 32/64 Result L1 hit L1 d- cache (64 sets, 8 lines/set)... L2, L3, and main memory L1 miss L1 TLB (16 sets, 4 entries/set) CR3 9 9 VPN1 VPN2 PTE 9 9 VPN3 VPN4 PTE PTE PTE PPN PPO Physical address (PA) CT CI CO Page tables 10

11 Core i7 Level 1-3 Page Table Entries 63 XD 62 Unused Page table physical base address Unused G PS A CD WT U/S R/W P=1 Available for OS (page table locaron on disk) P=0 Each entry references a 4K child page table P: Child page table present in physical memory (1) or not (0). R/W: Read- only or read- write access access permission for all reachable pages. U/S: user or supervisor (kernel) mode access permission for all reachable pages. WT: Write- through or write- back cache policy for the child page table. CD: Caching disabled or enabled for the child page table. A: Reference bit (set by MMU on reads and writes, cleared by so^ware). PS: Page size either 4 KB or 4 MB (defined for Level 1 PTEs only). G: Global page (don t evict from TLB on task switch) Page table physical base address: 40 most significant bits of physical page table address (forces page tables to be 4KB aligned) 11

12 Core i7 Level 4 Page Table Entries 63 XD 62 Unused Page physical base address Unused G D A CD WT U/S R/W P=1 Available for OS (page locaron on disk) P=0 Each entry references a 4K child page P: Child page is present in memory (1) or not (0) R/W: Read- only or read- write access permission for child page U/S: User or supervisor mode access WT: Write- through or write- back cache policy for this page CD: Cache disabled (1) or enabled (0) A: Reference bit (set by MMU on reads and writes, cleared by so^ware) D: Dirty bit (set by MMU on writes, cleared by so^ware) G: Global page (don t evict from TLB on task switch) Page physical base address: 40 most significant bits of physical page address (forces pages to be 4KB aligned) 12

13 Core i7 Page Table TranslaRon 9 VPN 1 9 VPN 2 9 VPN 3 VPN Virtual VPO address CR3 Physical address of L1 PT 40 / L1 PT Page global directory L1 PTE 40 / L2 PT Page upper directory L2 PTE 40 / L3 PT Page middle directory L3 PTE 40 / L4 PT Page table L4 PTE / 12 Offset into physical and virtual page 512 GB region per entry 1 GB region per entry 2 MB region per entry 4 KB region per entry Physical address of page 40 / Physical PPO address PPN 13

14 Cute Trick for Speeding Up L1 Cache Access CT Tag Check Physical address (PA) CT CI CO PPN PPO Virtual address (VA) Address TranslaRon VPN VPO No Change ObservaRon Bits that determine CI idenqcal in virtual and physical address Can index into cache while address translaqon taking place Generally we hit in TLB, so PPN bits (CT bits) available next Virtually indexed, physically tagged Cache carefully sized to make this possible CI L1 Cache 14

15 Virtual Memory of a Linux Process Different for each process Iden@cal for each process Process- specific data structs (ptables, task and mm structs, kernel stack) Physical memory Kernel code and data Kernel virtual memory %esp User stack brk Memory mapped region for shared libraries RunRme heap (malloc) Process virtual memory 0x (32) 0x (64) 0 UniniRalized data (.bss) IniRalized data (.data) Program text (.text) 15

16 Linux Organizes VM as CollecRon of Areas task_struct mm mm_struct pgd mmap vm_area_struct vm_end vm_start vm_prot vm_flags vm_next Process virtual memory Shared libraries pgd: Page global directory address Points to L1 page table vm_prot: Read/write permissions for this area vm_flags Pages shared with other processes or private to this process vm_end vm_start vm_prot vm_flags vm_next vm_end vm_start vm_prot vm_flags vm_next Data Text 0 16

17 Linux Page Fault Handling vm_area_struct Process virtual memory vm_end vm_start vm_prot vm_flags vm_next vm_end vm_start vm_prot vm_flags shared libraries data 1 read 3 read Segmentation fault: accessing a non- exisrng page Normal page fault vm_next vm_end vm_start vm_prot vm_flags vm_next text 2 write ProtecRon excepron: e.g., violarng permission by wrirng to a read- only page (Linux reports as SegmentaRon fault) 17

18 VirtualizaRon (VMware) Boot and run mulrple operarng systems Windows, MacOS, Linux all at once Not dual booqng! Requires special support for efficiency New privilege mode for hypervisor 18

19 Overview I/O devices device controller Device drivers Memory Mapped I/O Programmed I/O Direct Memory Access (DMA) RotaRonal media (disks) 19

20 Why I/O? InteracRve ApplicaRons (keyboard, mouse, screen) Long term storage (files, data repository) Swap for VM Many different devices character vs. block Networks are everywhere! 10 6 difference CPU (10-9 ) & I/O (10-3 ) Response Time vs. Throughput Not always another process to execute OS hides (some) differences in devices same (similar) interface to many devices Permits many apps to share one device 20

21 I/O Device Examples Device Behavior Partner Data Rate Keyboard Input Human 0.01 (KB/s) Mouse Input Human 0.02 (KB/s) Laser Printer Output Human (KB/s) OpQcal Disk Storage Machine (KB/s) MagneQc Disk Storage Machine (MB/s) USB 1.1 Input/Output Machine 1.50 (MB/s) USB 2.0 Input/Output Machine (MB/s) Flash drive I/O device Machine (MB/s) Network- LAN Input/Output Machine 1.00 (Gb/s) Graphics Display Output Human 8.00 (GB/s) L2 cache bandwidth ~50.00 (GB/s) 21

22 I/O Systems Processor interrupts Cache Memory Bus Core Chip Set I/O Bridge Main Memory Disk Controller I/O Bus Graphics Controller Network Interface I/O Devices Disk Disk Graphics Network Time(workload) = Time(CPU) + Time(I/O) - Time(Overlap) 22

23 Device Drivers Sosware that provides access to devices (mp3, camera, phone, FLASH drive) top- half ApplicaQon Programming Interface (API): e.g., open, close, read, write, ioctl I/O Control: e.g., IOCTL, device specific arguments botom- half interrupt handler communicates with device resumes process Must have access to user address space and device control registers => runs in kernel mode. 23

24 Review: Handling an Interrupt/ExcepRon User Program movl add movl mul jne movl Sub jne Interrupt Handler iret Service Routines Invoke specific kernel rourne based on type of interrupt interrupt/excepqon handler Must determine what caused interrupt Clear the interrupt Return from interrupt (iret) 24

25 Processor <- > Device Interface Issues InterconnecRons Busses Processor interface I/O InstrucQons Memory mapped I/O I/O Control Structures Device Controllers Polling/Interrupts Data movement Programmed I/O / DMA Capacity, Access Time, Bandwidth 25

26 Device Controllers Interrupt? Busy Done Error Bus Command Status Data 0 Device Controller Data 1 Data n-1 Controller deals with mundane control (e.g., position head, error detection/correction) Processor communicates with Controller Device 26

27 I/O InstrucRons Separate instrucrons (in, out): Like the processor in your project Independent I/O Bus CPU memory bus Memory Controller Controller Device Device 27

28 Memory Mapped I/O Issue command through store instrucron Check status with load instrucron Caches? Physical Address ROM RAM CPU $ Device I/O L2 $ Controller Memory Bus I/O bus Memory Bus Adapter Bridge 28

29 CommunicaRng with the processor Polling can waste Qme waiqng for slow I/O device busy wait can interleave with useful work Interrupts interrupt overhead interrupt could happen anyqme - asynchronous no busy wait 29

30 Data Movement Programmed I/O processor has to touch all the data too much processor overhead for high bandwidth devices (disk, network) DMA processor sets up transfer(s) DMA controller transfers data complicates memory system 30

31 Programmed I/O & Polling Advantage: CPU totally in control Disadvantage: Overhead of polling Program must perform check of device, thus can t do useful work yes Is the data ready? load data no store data done? no yes 31

32 Programmed I/O & Interrupt Driven Data Transfer CPU $ L2 $ Device Controller (1) I/O interrupt add sub and or nop user program Memory Bus (2) save PC I/O bus Memory Bus Adapter (3) interrupt service addr User program progress halted only during actual transfer (4) read store... rq memory interrupt service rouqne Interrupt overhead can dominate transfer Rme Processor must touch all data too slow for some devices 32

33 Direct Memory Access (DMA) CPU sends a starqng address, direcqon, and length count to DMAC. Then issues "start". CPU delegates responsibility for data transfer to a special controller CPU $ 0 ROM L2 $ Memory Bus I/O bus Memory Mapped I/O RAM Memory Bus Adapter DMA CNTRL DMAC provides handshake signals for device controller, and memory addresses and handshake signals for memory. n Peripherals DMAC 33

34 I/O and Virtual Caches Virtual Cache Processor interrupts Physical Addresses Cache Memory Bus I/O Bridge I/O is accomplished with physical addresses DMA flush pages from cache need pa- >va reverse translaron coherent DMA Main Memory Disk Controller Disk Disk I/O Bus Graphics Controller Graphics Network Interface Network 34

35 Other I/O Issues MulRple devices want to use the same bus (wires)? ArbitraRon Daisy chain Centralized Distributed 35

36 Obtaining Access to the Bus Control: Master inirates requests Bus Master Data can go either way Bus Slave One of the most important issues in bus design: How is the bus reserved by a devices that wishes to use it? Chaos is avoided by a master- slave arrangement: Only the bus master can control access to the bus: It iniqates and controls all bus requests A slave responds to read and write requests The simplest system: Processor is the only bus master All bus requests must be controlled by the processor Major drawback: the processor is involved in every transacqon 36

37 MulRple PotenRal Bus Masters: the Need for ArbitraRon Bus arbitraron scheme: A bus master wanqng to use the bus asserts the bus request A bus master cannot use the bus unql its request is granted A bus master must signal to the arbiter a^er finish using the bus Bus arbitraron schemes usually try to balance two factors: Bus priority: the highest priority device should be serviced first Fairness: Even the lowest priority device should never be completely locked out from the bus Bus arbitraron schemes can be divided into four broad classes: Distributed arbitraqon by self- selecqon: each device wanqng the bus places a code indicaqng its idenqty on the bus. Distributed arbitraqon by collision detecqon: Ethernet uses this. Daisy chain arbitraqon: single device with all request lines. Centralized, parallel arbitraqon: see next- next slide 37

38 Centralized Bus ArbitraRon ArbitraRon Circuit Request- B Grant- B Request- A Grant- A Release Request- C Grant- C Master A Master B Master C 38

39 The Daisy Chain Bus ArbitraRon Scheme Device 1 Highest Priority Device 2 Device N Lowest Priority Grant Grant Grant Bus Arbiter Release Request Advantage: simple Disadvantages: Cannot assure fairness: A low- priority device may be locked out indefinitely The use of the daisy chain grant signal also limits the bus speed 39

40 Types of Storage Devices MagneRc Disks FLASH Drive (USB SRck, SSD) MagneRc Tapes CD/DVD Juke Box (automated tape library, robots) 40

41 What s Inside A Disk Drive? Arm Spindle Platters Actuator SCSI connector Electronics (including a processor and memory!) Image courtesy of Seagate Technology 41

42 Disk Geometry Disks consist of platers, each with two surfaces. Each surface consists of concentric rings called tracks. Each track consists of sectors separated by gaps. Tracks Surface Track k Gaps Spindle Sectors 42

43 Disk Geometry (Muliple- Plater View) Aligned tracks form a cylinder. Cylinder k Surface 0 Surface 1 Surface 2 Surface 3 Surface 4 Surface 5 Platter 0 Platter 1 Platter 2 Spindle 43

44 Disk Capacity Capacity: maximum number of bits that can be stored. Vendors express capacity in units of gigabytes (GB), where 1 GB = 10 9 Bytes ( < 2 20 Bytes, be careful) Capacity is determined by these technology factors: Recording density (bits/in): number of bits that can be squeezed into a 1 inch segment of a track. Track density (tracks/in): number of tracks that can be squeezed into a 1 inch radial segment. Areal density (bits/in 2 ): product of recording and track density. Modern disks parrron tracks into disjoint subsets called recording zones Each track in a zone has the same number of sectors, determined by the circumference of innermost track. Each zone has a different number of sectors/track 44

45 CompuRng Disk Capacity Capacity = (# bytes/sector) x (avg. # sectors/track) x (# tracks/surface) x (# surfaces/plater) x (# platers/disk) Example: 512 bytes/sector 300 sectors/track (on average) 20,000 tracks/surface 2 surfaces/plarer 5 plarers/disk Capacity = 512 x 300 x x 2 x 5 = 30,720,000,000 = GB 45

46 Disk OperaRon (Single- Plater View) The disk surface spins at a fixed rotational rate The read/write head is attached to the end of the arm and flies over the disk surface on a thin cushion of air. spindle spindle spindle spindle By moving radially, the arm can position the read/write head over any track. 46

47 Disk OperaRon (MulR- Plater View) Read/write heads move in unison from cylinder to cylinder Arm Spindle 47

48 Disk Structure - top view of single plater Surface organized into tracks Tracks divided into sectors 48

49 Disk Access Head in position above a track 49

50 Disk Access Rotation is counter-clockwise 50

51 Disk Access Read About to read blue sector 51

52 Disk Access Read After BLUE read After reading blue sector 52

53 Disk Access Read After BLUE read Red request scheduled next 53

54 Disk Access Seek After BLUE read Seek for RED Seek to red s track 54

55 Disk Access RotaRonal Latency After BLUE read Seek for RED Rotational latency Wait for red sector to rotate around 55

56 Disk Access Read After BLUE read Seek for RED Rotational latency After RED read Complete read of red 56

57 Disk Access Service Time Components After BLUE read Seek for RED Rotational latency After RED read Data transfer Seek RotaRonal latency Data transfer 57

58 Disk Access Time Average Rme to access some target sector approximated by : Taccess = Tavg seek + Tavg rotaqon + Tavg transfer Seek Rme (Tavg seek) Time to posiqon heads over cylinder containing target sector. Typical Tavg seek is 3 9 ms RotaRonal latency (Tavg rotaron) Time waiqng for first bit of target sector to pass under r/w head. Tavg rotaqon = 1/2 x 1/RPMs x 60 sec/1 min Typical Tavg rotaqon = 7200 RPMs Transfer Rme (Tavg transfer) Time to read the bits in the target sector. Tavg transfer = 1/RPM x 1/(avg # sectors/track) x 60 secs/1 min. 58

59 Disk Access Time Example Given: RotaQonal rate = 7,200 RPM Average seek Qme = 9 ms. Avg # sectors/track = 400. Derived: Tavg rotaqon = 1/2 x (60 secs/7200 RPM) x 1000 ms/sec = 4 ms. Tavg transfer = 60/7200 RPM x 1/400 secs/track x 1000 ms/sec = 0.02 ms Taccess = 9 ms + 4 ms ms Important points: Access Qme dominated by seek Qme and rotaqonal latency. First bit in a sector is the most expensive, the rest are free. SRAM access Qme is about 4 ns/doubleword, DRAM about 60 ns Disk is about 40,000 Qmes slower than SRAM, 2,500 Qmes slower then DRAM. 59

60 Logical Disk Blocks Modern disks present a simpler abstract view of the complex sector geometry: The set of available sectors is modeled as a sequence of b- sized logical blocks (0, 1, 2,...) Mapping between logical blocks and actual (physical) sectors Maintained by hardware/firmware device called disk controller. Converts requests for logical blocks into (surface,track,sector) triples. Allows controller to set aside spare cylinders for each zone. Accounts for the difference in formared capacity and maximum capacity. 60

61 I/O Bus CPU chip Register file ALU System bus Memory bus Bus interface I/O bridge Main memory USB controller Graphics adapter I/O bus Disk controller Expansion slots for other devices such as network adapters. Mouse Keyboard Monitor Disk 61

62 Reading a Disk Sector (1) CPU chip Register file ALU CPU initiates a disk read by writing a command, logical block number, and destination memory address to a port (address) associated with disk controller. Bus interface Main memory I/O bus USB controller Graphics adapter Disk controller mouse keyboard Monitor Disk 62

63 Reading a Disk Sector (2) CPU chip Register file ALU Disk controller reads the sector and performs a direct memory access (DMA) transfer into main memory. Bus interface Main memory I/O bus USB controller Graphics adapter Disk controller Mouse Keyboard Monitor Disk 63

64 Reading a Disk Sector (3) CPU chip Register file ALU When the DMA transfer completes, the disk controller notifies the CPU with an interrupt (i.e., asserts a special interrupt pin on the CPU) Bus interface Main memory I/O bus USB controller Graphics adapter Disk controller Mouse Keyboard Monitor Disk 64

65 Solid State Disks (SSDs) Solid State Disk (SSD) Flash memory Block 0 I/O bus Page 0 Page 1 Page P-1 Flash translation layer Requests to read and write logical disk blocks Block B-1 Page 0 Page 1 Page P-1 Pages: 512KB to 4KB, Blocks: 32 to 128 pages Data read/writen in units of pages. Page can be writen only aser its block has been erased A block wears out aser 100,000 repeated writes. 65

66 SSD Performance CharacterisRcs SequenRal read tput 250 MB/s SequenRal write tput 170 MB/s Random read tput 140 MB/s Random write tput 14 MB/s Rand read access 30 us Random write access 300 us Why are random writes so slow? Erasing a block is slow (around 1 ms) Write to a page triggers a copy of all useful pages in the block Find an used block (new block) and erase it Write the page into the new block Copy other pages from old block to the new block 66

67 SSD Tradeoffs vs RotaRng Disks Advantages No moving parts à faster, less power, more rugged Disadvantages Have the potenqal to wear out MiQgated by wear leveling logic in flash translaqon layer E.g. Intel X25 guarantees 1 petabyte (10 15 bytes) of random writes before they wear out In 2010, about 100 Qmes more expensive per byte ApplicaRons MP3 players, smart phones, laptops Beginning to appear in desktops and servers 67

68 SRAM Storage Trends Metric :1980 $/MB 19,200 2, access (ns) DRAM Metric :1980 $/MB 8, ,000 access (ns) typical size (MB) ,000 8, ,000 Disk Metric :1980 $/MB ,600,000 access (ms) typical size (MB) ,000 20, ,000 1,500,000 1,500,000 68

69 CPU Clock Rates InflecRon point in computer history when procesors (chips) hit the Power Wall :1980 CPU Pentium P-III P-4 Core 2 Core i7 --- Clock rate (MHz) Cycle time (ns) Cores Effective cycle ,000 time (ns) 69

70 The CPU- Memory Gap The gap 100,000, ,000, ,000, ,000.0 between DRAM, disk, and CPU speeds. Disk SSD ns 10, , DRAM Disk seek time Flash SSD access time DRAM access time SRAM access time CPU cycle time Effective CPU cycle time CPU Year 70