Boot Interrupt Quirks and (RealTime) Interrupt Handling on x86. Olaf Dabrunz, Stefan Assmann

Transcription

1 Boot Interrupt Quirks and (RealTime) Interrupt Handling on x86 Olaf Dabrunz, Stefan Assmann

2 Overview 1. Interrupt Handling 2. RT Interrupt Handling is a bit Different 3. So all is well, no? 4. x86 Interrupt Routing Overview 5. Boot Interrupts 6. Fixing it in Software 7. What's in it for you 2

3 Interrupt Handling Device sends an Interrupt Request (IRQ) to the CPU Ethernet card receives a packet Mouse is moved IRQ handler services the device Normal processing continues We will show IRQ handling using IRQ 10 as example 3

4 Interrupt Handling (non-shared) Hardware Software Network Card IRQ IRQ 10 eth IRQ Handler 4

5 Interrupt Handling (shared) Hardware Software Network Card Sound Card IRQ IRQ 10 eth IRQ Handler sound IRQ Handler 5

6 Overview 1. Interrupt Handling 2. RT Interrupt Handling is a bit Different 3. So all is well, no? 4. x86 Interrupt Routing Overview 5. Boot Interrupts 6. Fixing it in Software 7. What's in it for you 6

7 IRQ Handling - Vanilla vs. RT Vanilla IRQ 10 IRQ 10 Handler 7 Blocks CPU (Interrupts deactivated) Other programs can run (Interrupts activated)

8 IRQ Handling - Vanilla vs. RT Vanilla IRQ 10 IRQ 10 Handler IRQ 10 Thread 10 waker wake up thread Threaded IRQ 10 Handler RealTime 8 Blocks CPU (Interrupts deactivated) Other programs can run (Interrupts activated)

9 Threaded IRQ Handling Low latency The CPU is only blocked for a short time High-priority programs are delayed for shorter periods of time Realtime programs need this 9

10 RT IRQ Handling Really cool! :) IRQ 10 Thread 10 waker wake up thread Threaded IRQ 10 Handler RealTime 10 Blocks CPU (Interrupts deactivated) Other programs can run (Interrupts activated)

11 Threaded IRQ Handler Problem After waking up the thread, IRQ is delivered again IRQ 10 IRQ 10 Thread 10 waker wake up thread Threaded IRQ 10 Handler 11 Blocks CPU (Interrupts deactivated) Other programs can run (Interrupts activated)

12 Threaded IRQ Handler: Solution Mask IRQ until IRQ thread handled it Prevent device from delivering same IRQ again IRQ 10 pending on device IRQ 10 Thread 10 waker wake up thread Threaded IRQ 10 Handler IRQ 10 is masked 12 Blocks CPU (Interrupts deactivated) Other programs can run (Interrupts activated)

13 Overview 1. Interrupt Handling 2. RT Interrupt Handling is a bit Different 3. So all is well, no? 4. x86 Interrupt Routing Overview 5. Boot Interrupts 6. Fixing it in Software 7. What's in it for you 13

14 Nobody cared!? irq 16: nobody cared (try booting with the "irqpoll" option) Call Trace: [<ffffffff80278dae>] report_bad_irq+0x1e/0x80 [<ffffffff802790be>] note_interrupt+0x2ae/0x2e0 [<ffffffff d>] thread_simple_irq+0x7d/0xa0 [<ffffffff80278bf3>] do_irqd+0x233/0x3a0 [<ffffffff802789c0>] do_irqd+0x0/0x3a0 [<ffffffff802789c0>] do_irqd+0x0/0x3a0 [<ffffffff b>] kthread+0x4b/0x80 [<ffffffff8020ae98>] child_rip+0xa/0x12 [<ffffffff8021ceb0>] lapic_next_event+0x0/0x10 [<ffffffff >] kthread+0x0/0x80 [<ffffffff8020ae8e>] child_rip+0x0/0x12 14

15 Problem: no Handler for this Device Hardware Software Network Card Sound Card IRQ IRQ 16 eth IRQ Handler sound IRQ Handler 15

16 Nobody cared! Kernel tries to find origin of interrupt Loop through handlers on IRQ Line No handler is found that could handle the interrupt Interrupt is counted as unhandled Unhandled interrupts are called Spurious Interrupts Kernel tracks Spurious Interrupts IRQ line disabled when too many Spurious Interrupts IRQ nobody cared message 16

17 Consequences Devices stop working System may hang System breaks 17

18 Overview 1. Interrupt Handling 2. RT Interrupt Handling is a bit Different 3. So all is well, no? 4. x86 Interrupt Routing Overview 5. Boot Interrupts 6. Fixing it in Software 7. What's in it for you 18

19 IRQ Routing PIC Southbridge IO-APIC Interrupts can be processed by the PIC and/or IO- APIC 19

20 IRQ Routing with PIC only PIC Southbridge IO-APIC Interrupts are processed by the PIC IO-APIC lines are masked 20

21 IRQ Routing with APIC only PIC Southbridge IO-APIC Interrupts are processed by the IO-APIC PIC lines are masked 21

22 Overview 1. Interrupt Handling 2. RT Interrupt Handling is a bit Different 3. So all is well, no? 4. x86 Interrupt Routing Overview 5. Boot Interrupts 6. Fixing it in Software 7. What's in it for you 22

23 What s a Boot Interrupt anyway? Boot Interrupts are generated by (PCI) Bus Bridges Deliver IRQs from other buses to the PIC during boot Boot Interrupt PIC Southbridge IO-APIC 0 IRQ Device X IRQ IO-APIC 1 PCI Bridge

24 Boot Interrupts and RT 1. IRQ on non-primary bus is delivered to CPU Delivered by non-primary IO-APIC PIC Southbridge IO-APIC 0 Device X IRQ IO-APIC 1 PCI Bridge IRQ 42 24

25 Boot Interrupts and RT 2. Handler masks IRQ, Boot Interrupt is generated Delivered through primary IO-APIC PIC Boot Interrupt Southbridge IO-APIC 0 IRQ Device X IRQ IO-APIC 1 PCI Bridge

26 Boot Interrupts and RT 3. Spurious IRQ 16 (Device X Handler not installed) IRQ 16 might get disabled by kernel!!! PIC Boot Interrupt Southbridge IO-APIC 0 IRQ Device X IRQ IO-APIC 1 PCI Bridge

27 Boot Interrupts and RT 4. What we see in the system Multiple Interrupts delivered based on single IRQ PIC Southbridge Boot Interrupt IO-APIC 0 IRQ 16 Device X IRQ IO-APIC 1 PCI Bridge IRQ 42 27

28 Overview 1. Interrupt Handling 2. RT Interrupt Handling is a bit Different 3. So all is well, no? 4. x86 Interrupt Routing Overview 5. Boot Interrupts 6. Fixing it in Software 7. What's in it for you 28

29 Disable Delivery of Boot IRQs " Disabling Boot Interrupts is chipset specific " Not possible with every chipset PIC Boot Interrupt Southbridge IO-APIC 0 Device X IRQ IO-APIC 1 PCI Bridge IRQ 42 29

30 Reroute to Boot Interrupt " Move IRQ Handler to primary IO-APIC " Very deep chipset knowledge necessary PIC Boot Interrupt Southbridge IO-APIC 0 IRQ 16 Device X IRQ IO-APIC 1 PCI Bridge IRQ 42 30

31 Overview 1. Interrupt Handling 2. RT Interrupt Handling is a bit Different 3. So all is well, no? 4. x86 Interrupt Routing Overview 5. Boot Interrupts 6. Fixing it in Software 7. What's in it for you 31

32 What's in it for you? " Threaded IRQ Handler works with today's hardware " Faster Response Times during IRQ Processing " Buggy IRQ Handlers do not bring the system down that easily " Threaded IRQ Handlers are easier to debug 32

33 What's next Get Threaded IRQ Handler upstream Talk to vendors to make disabling of Boot Interrupts possible put the disable bit in a common place 33

34 More details and Acknowledgements Git repository git://git.kernel.org/pub/scm/linux/kernel/git/x86/linux-2.6-tip.git Branch pci-ioapic-boot-irq-quirks Additional slides on the conference web page Acknowledgements for many discussions go to Alexander Graf, Hannes Reinecke, Torsten Duwe, Ihno Krumreich, Daniel Gollub, Sven-Thorsten Dietrich, Thomas Gleixner and Maciej W. Rozycki 34

35 More details and Acknowledgements Git repository git://git.kernel.org/pub/scm/linux/kernel/git/x86/linux-2.6-tip.git Branch pci-ioapic-boot-irq-quirks Additional slides on the conference web page Acknowledgements for many discussions go to Alexander Graf, Hannes Reinecke, Torsten Duwe, Ihno Krumreich, Daniel Gollub, Sven-Thorsten Dietrich, Thomas Gleixner and Maciej W. Rozycki Questions? 35

36 License This work is licensed under the Creative Commons Attribution-Noncommercial-Share Alike 3.0 Germany License. To view a copy of this license, visit or send a letter to Creative Commons, 171 Second Street, Suite 300, San Francisco, California, 94105, USA. General Disclaimer This document is not to be construed as a promise by any participating company to develop, deliver, or market a product. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. Novell, Inc. makes no representations or warranties with respect to the contents of this document, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. The development, release, and timing of features or functionality described for Novell products remains at the sole discretion of Novell. Further, Novell, Inc. reserves the right to revise this document and to make changes to its content, at any time, without obligation to notify any person or entity of such revisions or changes. All Novell marks referenced in this presentation are trademarks or registered trademarks of Novell, Inc. in the United States and other countries. All third-party trademarks are the property of their respective owners.

37 Details

38 Details 1. ACPI Interrupt Routing Setup 2. Edge- and Level-Triggered Interrupts 3. Possible and Impossible ;) Solutions 38

39 ACPI Interrupt Routing Setup " IRQ routing information is stored in the ACPI tables " Device driver gets loaded " Kernel queries ACPI for IRQ line of corresponding device 39

40 Details 1. ACPI Interrupt Routing Setup 2. Edge- and Level-Triggered Interrupts 3. Possible and Impossible ;) Solutions 40

41 Edge-triggered Interrupts Edge-triggered interrupts are active only during a very short time Pros: Need no interrupt acknowledge on the device Need not be masked Cons: Cannot share an interrupt line Are more vulnerable to electrical disturbances on wiring 41

42 Level-triggered Interrupts Level triggered interrupts remain asserted until acknowledged on the causing device Pros: Can share interrupt lines Are less vulnerable to transmission disturbances on wiring Cons: Need interrupt acknowledge on the device Need masking for threaded interrupt handling 42

43 Details 1. ACPI Interrupt Routing Setup 2. Edge- and Level-Triggered Interrupts 3. Possible and Impossible ;) Solutions 43

44 Possible and Impossible ;) Solutions 1 A) Avoid Generating the Problem Use IRQ delivery method w/o Boot IRQs (not everywhere) Use Alternatives For Masking > Disable Local APIC temporarily (thwarts threaded IRQ benefits) > Delayed End of Interrupt signaling (does not work for some reason) > Level-edge-level patch (works unreliably, difficult to predict) Acknowledge IRQ early on device (future, may often not work) B) Avoid Symptoms Never disable lines with spurious IRQs (system may hang) Subtract real IRQs from spurious IRQ count (cannot count) 44 Solution does not work, works unreliably or other severe problems Solution works on several platforms Proposed solution, may or may not have problems

45 Possible and Impossible ;) Solutions 2 C) Prevent Problem Propagation Hide Boot IRQs (does not work) Disable Delivery of Boot IRQs (unavailable where needed, contradictory specs) Disable Boot IRQs *) (not possible everywhere) Reroute to Boot IRQ *) (increases IRQ sharing) Mask Boot IRQ (code is more spread out, increases IRQ latencies as if sharing was increased) 45 *) Solution implemented by us Solution does not work, works unreliably or other severe problems Solution works on several platforms Proposed solution, may or may not have problems

46 Use IRQ delivery method w/o Boot IRQs (1/2) " Modern IRQ delivery methods do not trigger Boot IRQs " PCI has MSI and MSI-X " During system boot, devices supporting MSI or MSI-X use legacy IRQ delivery methods equivalent to boot IRQs (virtual wire INTx) or methods that can generate boot IRQs (INTx) so during boot or when for other reasons in PIC mode IRQ delivery still works " Solution: in APIC mode use MSI or MSI-X IRQ delivery 46

47 Use IRQ delivery method w/o Boot IRQs (2/2) " Pros: Simple to implement Available on all modern systems and many older systems MSI / MSI-X are fast and reliable " Cons: MSI / MSI-X is buggy on several chipsets, software disables it 47

48 Use Alternatives for Masking " Masking causes Boot IRQ generation " Solution: Use other methods to temporarily stop IRQ delivery " Methods: Disable Local APIC temporarily Delayed End of Interrupt signaling Level-edge-level patch 48

49 Disable Local APIC temporarily (1/2) " Final IRQ delivery to CPU is done by Local APIC " Local APIC can temporarily be disabled " Solution: Disable Local APIC temporarily 49

50 Disable Local APIC temporarily (2/2) " Pros: Simple to implement Covers most or all modern processors " Cons: Local APIC cannot be disabled on old processors All IRQs are affected: very dependent on re-enabling LAPIC Advantage of threaded IRQ handling is lost: CPU is blocked until IRQ handler thread finishes 50

51 Delayed End of Interrupt signaling " Current RT kernels signal End of Interrupt to APICs after waking up IRQ handler thread " APICs can then interrupt the CPU again " Solution: Instead of masking, delay End of Interrupt until IRQ handler thread is finished " Pros: Simple to implement Should cover all hardware " Cons: Does not work: system hangs during boot (reasons unknown) 51

52 Level-edge-level patch (1/2) " Only level-triggered IRQs need masking " When IO-APIC is told to recognize edge-triggered IRQs on a line, a level-triggered IRQ that is pending on that line will not be recognized " Solution: set IO-APIC line to recognize edge-triggered IRQs instead of masking 52

53 Level-edge-level patch (2/2) " Pros: Simple to implement In principle should work on all hardware " Cons: Triggers IO-APIC lockups on the switched line on several types of hardware Result: devices do not work anymore, system may hang Attempts to fix this in software have been unsuccessful 53

54 Never disable lines with spurious IRQs (1/2) " Current kernels disable IRQ lines when the current rate of spurious IRQs reaches a certain level ( screaming IRQs ) " Solution: never disable lines with spurious IRQs 54

55 Never disable lines with spurious IRQs (2/2) " Pros: Very simple to implement Covers all hardware " Cons: Hardware without drivers can cause CPUs to hang > Hardware generates IRQ for some reason > No driver available to disable the IRQ > CPU will re-enter the IRQ handler immediately after leaving it > No way to stop this endless loop (disabling the line was the solution) Drivers bugs can cause system hangs Broken hardware can cause system hangs (Temporary) electrical problems can cause system hangs 55

56 Substract real IRQs from spurious IRQ count " Every real IRQ generates at most one Boot IRQ " Solution: Substracting real IRQ count from spurious IRQ count should make spurious IRQ count accurate " Pros: Not dependent on hardware " Cons: Need to find Boot IRQ line corresponding to real IRQ Masking on the Boot IRQ line (for other devices on that line) makes real IRQ count diverge easily from corresponding spurious IRQ count this is unreliable 56

57 Disable Delivery of Boot IRQs (1/2) " Boot IRQs are generated on PCI bus bridges " They use legacy INTx signals to make sure Boot IRQs are delivered to primary IO-APIC " Solution: disable legacy INTx signal generation on PCI bridge 57

58 Disable Delivery of Boot IRQs (2/2) " Pros: Simple to implement Little maintenance required (new bridges should support this, as it is required by the later PCI(e?) standards) " Cons: Some older (but widely used) hardware does not support disabling INTx Are passed-through INTx disabled by this bit as well? > Documentation is unclear or contradictory INTx is used for other purposes as well (error reporting) 58

59 Hide Boot IRQs " Use Same Vector for IRQ and corresp. Boot IRQ " Pros: Simple to implement Works on all hardware " Cons: Still two IRQs are received for each real IRQ No way to decide which IRQs are Boot IRQs, which are real IRQs and which are spurious IRQs 59

60 Disable Boot IRQ " Modern PCI bridges can disable Boot IRQ generation " Solution: disable Boot IRQ generation " Pros: Works reliably Simple to implement and maintain, when bridge known " Cons: Several older (and often still in use) bridges cannot disable Boot IRQ generation Implementation very specific to PCI bridge Not possible when device documentation unavailable 60

61 Reroute to Boot IRQ (1/2) " Keeping bridge IRQs masked at all times always causes the generation of boot IRQs, and the original IRQs are never recognized " Solution: keep bridge IRQs masked and tell driver to always use the corresponding boot IRQ line instead 61

62 Reroute to Boot IRQ (2/2) " Pros: Moderately simple to implement Works with devices that cannot disable boot IRQs Works with devices without (public) documentation " Cons: Need to find Boot IRQ line corresponding to real IRQ Needs lots of testing with different hardware configurations IRQ sharing between devices is increased on Boot IRQ lines 62

63 Mask Boot IRQ (1/3) " Delivery of Boot IRQ may be prevented by masking / manipulating the IO-APIC or LAPIC " Solution: mask primary IO-APIC before masking secondary IO-APIC When bridge IRQ comes in, in the IRQ handler first mask the corresponding (boot IRQ) line on the primary IO-APIC Then mask the IRQ line on the bridge IO-APIC Handle bridge IRQ (this will acknowledge the IRQ on the device) Unmask IRQ line on the primary IO-APIC 63

64 Mask Boot IRQ (2/3) " Pros: Moderately simple to implement Works with devices that cannot disable boot IRQs Works with devices without public documentation Should work with all IO-APICs > Level-asserts or negates occurring on a masked level-sensitive pin are [...] ignored and have no side effects. Intel 82093AA IOAPIC datasheet > Pending level-asserts from other devices on the IRQ line will be recognized as soon as the line is unmasked again No additional interrupt sharing on primary IO-APIC 64

65 Mask Boot IRQ (3/3) " Cons: Code is more spread out Need to find Boot IRQ line corresponding to real IRQ Needs lots of testing with different hardware configurations Masks other IRQs that share the line until real IRQ handled > introducing additional delay, as if sharing was increased Side-Notes: EOI before unmasking was an idea to clear pending Boot IRQs, so that we can unmask much earlier > Is not expected to work, as the IO-APIC is not in the correct state yet for EOI IRQ needs to be delivered fist, leading to the known problems 65

66 Acknowledge IRQ early on device (1/2) " If IRQ can be acknowledged early and quickly on the causing device, masking is not needed " Solution: require drivers to implement early IRQ ack method and use that instead of masking 66

67 Acknowledge IRQ early on device (2/2) " Pros: " Cons: > Infrastructure moderately simple to implement (and is now in the kernel) > Makes one IRQ thread per device possible (as opposed to one IRQ thread per IRQ line ) -> better latency > Puts burden of early ack implementation on driver developer > Early ack may take moderately long on some devices > Device may be unable to early ack -- this problem may persist, we need more experience here > Device may not offer implementation alternatives, such as masking (and disabling IRQs may cause lost IRQs) > Little experience with this approach: BSD has one driver > Impact on development, maintainability and performance difficult to predict 67

68 Mask / Reroute to Boot IRQ (revisited) " Need to find Boot IRQ line corresponding to real IRQ " How to do this? " ACPI tables cannot be used In APIC mode, they only give information about the real IRQ's IO-APIC line In PIC mode, they only give information about the PIC IRQ line (which is the boot IRQ line) The PIC / APIC mode can only be selected once mode selection can irreversibly reconfigure IRQ hardware 68

69 Mask / Reroute to Boot IRQ (revisited) " Need to trace the IRQ routing through the hardware: PCI standards apply to most components PCI standards do not apply to Northbridge internals and Northbridge/Southbridge links Routing is moderately dependent on chipsets 69 > Intel has the longest list of chipset generations, but most older generations behave identical wrt IRQ routing > ATM some newer Intel Northbridges are special, and come with special IRQ routing capabilities Rerouting needs to be revised for new Northbridges Need to detect unknown Northbridges and to default to opt out of rerouting in these cases Not a real problem: rerouting is unlikely to be needed with newer chipsets, as newer PCI bridges can disable boot IRQs

70