CSE A215 Assembly Language Programming for Engineers

Size: px

Start display at page:

Download "CSE A215 Assembly Language Programming for Engineers"

Barry Dawson
5 years ago
Views:

1 CSE A215 Assembly Language Programming for Engineers Lecture 13 Storage and I/O (MMIO, Devices, Reliability/Availability, Performance) 20 November 2012 Sam Siewert

2 Hardware/Software Interface for I/O Basics and Driver Concept Sam Siewert 2

3 PCI (Peripheral Component Interconnect) System CPU FSB Graphics Adapter AGP North Bridge SDRAM/ DDR PCI 2.x Bus ISA Bus Audio IDE South Bridge Ethernet Expansion Slots Super IO Sam Siewert 3 COM-A COM-B

Hardware View of Device Interfaces Analog I/O DAC analog output: servos, motors, heaters,... ADC analog input: photodiodes, thermistors,... Digital I/O Direct TTL I/O or GPIO Digital Serial (I2C, SPI,.

4 Hardware View of Device Interfaces Analog I/O DAC analog output: servos, motors, heaters,... ADC analog input: photodiodes, thermistors,... Digital I/O Direct TTL I/O or GPIO Digital Serial (I2C, SPI,... - Chip-to-Chip) Bus Interfaces Parallel PCI 2.x, PCI-X, SCSI, etc (32-bit, 64-bit, synchronous parallel transfer) Differential Serial USB Infiniband gige / 10GE Ethernet Fiber Channel SAS/SATA Sam Siewert 4

Software View of Drivers MMIO Device Buffers Decode Memory Bus Addresses (Outside of RAM address space) Character Register Control/Config, Status, Data Typical of Low-Rate I/O Interfaces (RS232)

5 Software View of Drivers MMIO Device Buffers Decode Memory Bus Addresses (Outside of RAM address space) Character Register Control/Config, Status, Data Typical of Low-Rate I/O Interfaces (RS232) Linux User Space Buffer Drivers (Direct IO) e.g. SCSI Generic Block FIFOs, Dual-Port RAM and DMA Typical of High-Rate I/O Interfaces (Network, Storage) Only Interface for 512 Byte LBA/Sector HDDs Network Driver Stacks OSI 7 Layer Model (Phy, Link, Network, Transport, Session, Presentation, Application) TCP/IP/Ethernet/Cat-6e Sam Siewert 5

6 open/close, read/write, creat, ioctl If Output Ring-Buffer Full then {SemTake or EAGAIN} else {Process and Return} Output Ring-Buffer Linux Char Driver Design Application(s) SemGive ISR Hardware Device EAGAIN, Block, Data, Status If Input Ring-Buffer Empty then {SemTake or EAGAIN} else {Processand Return} Input Ring-Buffer Application Interface Application Policy Blocking/Non-Blocking Multi-thread access Abstraction Device Interface SW/HW Interface Immediate Buffering Interrupt Service Routine App/Device Interface Sam Siewert 6

7 Cached Memory and DMA Cache Coherency Making sure that cached data and memory are in sync Can become out of sync due to DMAs and Multi-Processor Caches Push Caches Allow for DMA into and out of Cache Directly Cache Snooping by HW may Obviate Need for Invalidate Drivers Must Ensure Cache Coherency Invalidate Memory Locations on DMA Read Completion Flush Cache Prior to DMA Write Initiation IO Data Cache Line Alignment Ensure that IO Data is Aligned on Cache Line Boundaries Other Data That Shares Cache Line with IO Data Could Otherwise Be Errantly Invalidated Sam Siewert 7

8 How Reliable and Available are Data Center Systems? Availability vs. Reliability Sam Siewert 8

9 Reliability and Recovery Redundancy Dual String Side A, Side B Pilot, Co-Pilot Fail-Over Fault Detection, Protection, Recovery Backup System Independent Design e.g. Backup Flight System Cross Strapping of Sides Dual String A & B 3 Components C1, C2, C3 8 Possible Configurations 4 Component Switches A B Select Switch A B C1 C1 SW1 SW2 C2 C2 SW3 SW4 C3 C3 Configurations C1 C2 C3 1 A A A 2 A A B 3 A B A 4 A B B 5 B A A 6 B A B 7 B B A 8 B B B Sam Siewert 9

10 High Availability Service Up-time is Figure of Merit Number of Times down? How long down? Quick recovery is key Hot or Warm Spare Equipment Protection Fault Detection and Fail Over Without Service Outage Excess Capacity E.g. Diverse Routing in a Network Overlapping Coverage in Cell Phone Systems On-orbit spare satellites Sam Siewert 10

11 Availability vs. Reliability Are They the Same? Are all Reliable Systems Highly Available? Are all Highly Available Systems Reliable? Reliability = Long MTTF Mean Time To Failure Mean Time Between Failures, MTBF=MTTF+MTTR FDIR When Failures Do Occur Fault Detection, Isolation and Recovery Safing MTTR (Mean Time to Recover) Availability = MTTF / (MTTF + MTTR) = % Uptime MTTF = 8,766 hours (525,960 minutes) MTTR = 5 minutes Availability = 525,960 / (525, ) = % Uptime Sam Siewert 11

12 Storage I/O Storing Data Long Term Sam Siewert 12

13 A Single Disk Drive Read and Write 512-byte Sectors at LBA (Logical Block Address) A 2TB 3.5 SATA Disk Drive has 4 billion 512-byte sectors to manage The Operating System SATA/SCSI Driver and Filesystem Layered on the Block Driver Provide Use of a Disk Drive The Operating System Caches Pages (Typically 4K), that are Written Back (like CPU cache) from RAM to Disk When Needed (See slabtop) Filesystem Manages Access to Sectors Block I/O Can Be Done Directly as Well Sam Siewert 13

14 RAID-10 A1,A2,A3, A12 RAID-0 Striping Over RAID-1 Mirrors RAID-1 Mirror RAID-1 Mirror RAID-1 Mirror A1 A4 A7 A10 A1 A4 A7 A10 A2 A2 A3 A3 A5 A5 A6 A6 A8 A8 A9 A9 A11 A11 A12 A12 Sam Siewert 14

15 RAID5,6 XOR Parity Encoding MDS Encoding, Can Achieve High Storage Efficiency with N+1: N/(N+1) and N+2: N/(N+2) 100.0% 90.0% 80.0% 70.0% Storage Efficiency 60.0% 50.0% 40.0% 30.0% RAID6 RAID5 20.0% 10.0% 0.0% Number of Data Devices for 1 XOR or 2 P,Q Encoded Devices Sam Siewert 15

16 RAID-50 A1,B1,C1,D1,A2,B2,C2,D2,E1,F1,G1,H1,, Q2,R2,S2,T2 RAID-0 Striping Over RAID-5 Sets RAID-5 Set RAID-5 Set A1 B1 C1 D1 P(ABCD) E1 F1 G1 P(EFGH) H1 I1 J1 P(IJKL) K1 L1 M1 P(MNOP) N1 O1 P1 P(QRST) Q1 R1 S1 T1 A2 B2 C2 D2 P(ABCD) E2 F2 G2 P(EFGH) H2 I2 J2 P(IJKL) K2 L2 M2 P(MNOP) N2 O2 P2 P(QRST) Q2 R2 S2 T2 Sam Siewert 16

RAID-60 (Reed-Solomon Encoding) A1,B1,C1,D1,A2,B2,C2,D2,E1,F1,G1,H1,, Q2,R2,S2,T2 RAID-0 Striping Over RAID-6 Sets RAID-6 Set RAID-6 Set A1 B1 C1 D1 P(ABCD) E1 F1 G1 P(EFGH) Q(EFGH) I1 J1 P(IJKL)

17 RAID-60 (Reed-Solomon Encoding) A1,B1,C1,D1,A2,B2,C2,D2,E1,F1,G1,H1,, Q2,R2,S2,T2 RAID-0 Striping Over RAID-6 Sets RAID-6 Set RAID-6 Set A1 B1 C1 D1 P(ABCD) E1 F1 G1 P(EFGH) Q(EFGH) I1 J1 P(IJKL) Q(IJKL) K1 M1 P(MNOP) Q(MNOP) N1 O1 P(QRST) Q(QRST) Q1 R1 S1 QABCD) H1 L1 P1 T1 A2 B2 C2 D2 P(ABCD) E2 F2 G2 P(EFGH) Q(EFGH) I2 J2 P(IJKL) Q(IJKL) K2 M2 P(MNOP) Q(MNOP) N2 O2 P(QRST) Q(QRST) Q2 R2 S2 QABCD) H2 L2 P2 T2 Disk1 Disk2 Disk3 Disk4 Disk5 Disk6 Disk1 Disk2 Disk3 Disk4 Disk5 Disk6

18 I/O Performance Some Methods to Improve I/O on Linux Sam Siewert 18

19 Hiding IO Latency Overlapping with Processing Simple Design Each Thread has READ, PROCESS, WRITE-BACK Execution READ F(1) Process F(1) Write-back F(1) READ F(2) Frame rate is READ+PROCESS+WRITE latency e.g. 10 fps for 100 milliseconds If READ is 70 msec, PROCESS is 10 msec, and WRITE-BACK 20 msec, predominate time is IO time, not processing Disk drive with 100 MB/sec READ rate can only read 16 fps, 62.5 msec READ latency Sam Siewert 19

20 Hiding IO Latency Schedule Multiple Overlapping Threads? READ F 1 Process F 1 Write-back F 1 READ F 4 Process F 4 Write-back F 4 READ F 2 Process F 2 Write-back F 2 READ F 5 Process F 5 READ F 3 Process F 3 Write-back F 3 Read F 6 Start-up Core #1 Continuous Processing Core #1 Continuous Processing READ F 1 Process F 1 Write-back F 1 READ F 4 Process F 4 Write-back F 4 READ F 2 Process F 2 Write-back F 2 READ F 5 Process F 5 READ F 3 Process F 3 Write-back F 3 Read F 6 Start-up Core #2 Continuous Processing Core #2 Continuous Processing Requires N threads = N stages x N cores 1.5 to 2x Number of Threads for SMT (Hyper-threading) For IO Stage Duration Similar to Processing Time More Threads if IO Time (Read+WB+Read) >> 3 x Processing Time Sam Siewert 20

21 Hiding Latency Dedicated IO Schedule Reads Ahead of Processing Read F 1 Read F 2 Read F 3 Read F 4 Read F 5 Read F 6 Read F 7 Read F 8 Wait Process F 1 Process F 3 Process F 5 Wait Process F 2 Process F 4 Process F 6 Wait WB F 1 WB F 2 WB F 3 WB F 4 WB F 5 WB F 6 Start-up Dual-Core Concurrent Processing Completion Requires N threads = 2 + N cores Synchronize Frame Ready/Write-backs Balance Stage Read/Write-Back Latency to Processing 1.5 to 2x Threads for SMT (Hyper-threading) Sam Siewert 21

22 Processing Latency Alone Write Code with Memory Resident Frames Load Frames in Advance Process In-Memory Frames Over and Over Do No IO During Processing Provides Baseline Measurement of Processing Latency per Frame Alone Provides Method of Optimizing Processing Without IO Latency Sam Siewert 22

23 IO Latency Alone Comment Out Frame Transformation Code or Call Stubbed NULL Function Provides Measurement of IO Frame Rate Alone Essentially Zero Latency Transform No Change Between Input Frames and Output Frames Allows for Tuning of IO Scheduler and Threading Sam Siewert 23

24 Tips for IO Scheduling blockdev --getra /dev/sda Should return 256 Means that reads read-ahead up to 128K Function calls read, fread should request as much as possible Check actual bytes read, re-read as needed in a loop blockdev --setra /dev/sda (8MB) Switch CFQ to Deadline Use lsscsi to verify your disk is /dev/sda substitue block driver interface used for file system if not sda cat /sys/block/sda/queue/scheduler echo deadline > /sys/block/sda/queue/scheduler Options are noop, cfq, deadline Sam Siewert 24

CS A490 Digital Media and Interactive Systems

CS A490 Digital Media and Interactive Systems Lecture 11 Thread Scaling and I/O Threading and Async I/O on Linux October 30, 2013 Sam Siewert Parallel Processing Speed-up Grid Data Processing Speed-up