PCI Interrupt Controller for Industry Standard PCI-ISA Bus Architecture using PCI-to-PCI Bridge Technology. Ross L. Armstrong

Transcription

1 nterrupt ontroller for ndustry tandard - Bus rchitecture using -to- Bridge echnology oss L. rmstrong igital quipment orporation (cotland) Ltd., osshill ndustrial state, yr, cotland. K6 6B. he demand for more Peripheral omputer nterconnect () device configurations beyond the limit set in the local bus specification has prompted the development of several - bridge solutions. his paper describes a new ndustrial omputer anufacturers Group (PG) - bus architecture implementation using igital quipment orporation - bridge technology. Layered bus architectures, interrupt latency implications and performance optimizations for - bridge designs are discussed. eference will be made to igital s family of 6-bit lpha ingle Board omputers (Bs) and - backplanes which have been specifically designed to address multiple, low cost, high performance requirements associated with high speed communications and graphics in embedded applications.

2 Overview igital's mbedded and eal-ime line of business has developed a series of modular computing products supporting an open systems environment based upon the PG - B standard. wo goals of the igital odular omputing omponent () program were to: Figure PG ingle Bus nterrupt Binding B LO LO LO LO B develop a number of based backplane products that would enable customers to procure and configure industry standard and /O option cards L0967 provide extensive /O option card slots and maintain optimum bandwidth/ performance for bridged slots lthough the former requirement was a simple undertaking, the latter provided a greater challenge to the platform designers. eduction in bandwidth, however moderate, could be encountered due to software or hardware inefficiencies i.e. degraded interrupt servicing (latency) or propagation/timing delay due to the bridge implementation. he choice of igital - bridge chip adequately meets the required timing specification, however interrupt lines derived from secondary bus devices are not routed through the igital - bridge chip. his allowed the designers to thoroughly review and improve upon the standard interrupt binding strategy for buses, where multiple devices might have to share interrupt lines. he PG single board computer connector has only four interrupt lines assigned to it: #, B#, # and #, as does each slot connector. routing or binding strategy is required to connect between the option /O and the B X# line it uses when requesting an interrupt. n hardware terms, Figure, PG ingle Bus nterrupt Binding, shows how this structure is intended to be provided. he line per slot is assigned to [:8] as per the PG pecification. hese are used to identify device numbers as given in the configuration address. he system firmware (or BO) code must assume an interrupt binding architecture for its environment. he PG specified binding for the primary bus ( Bus 0) is shown in Figure, i.e. it is hard coded. Because only the firmware (or BO) knows how the X# lines are routed to the system controller, a mechanism is required to inform the operating system device driver of an interrupt occurrence. his mechanism typically requires a chained 'software' search of each device using a specific hardware interrupt to identify the source. his can be achieved by having the firmware/bo poll all devices to determine which originated the interrupt request and then initiate the correct interrupt service routine, or alternatively, have the operating system kernel interrupt dispatch routine sequentially call each individual device interrupt service routine until the correct source has been identified and serviced. (he latter example is implemented by icrosoft Windows ). he binding structure becomes even more congested when additional (bridged) buses are implemented in accordance with the -to- Bridge rchitecture pecification evision.o. heir bus interrupts must be connected as per Figure, econdary Bus nterrupt Binding. igital quipment orporation

3 Figure econdary Bus nterrupt Binding LO LO LO LO n alternative solution was investigated in order to improve upon this industry standard binding architecture to avoid compromising interrupt latency performance in large systems. (ny proposal would take cognisance of, and retain support for, the traditional wire-o scheme.) he design goal was to provide improved performance while maintaining an open system architecture capable of supporting both existing and alternative modes. B nterrupt ontroller ssumptions/limitations 0 _9 his secondary binding architecture must be overlaid upon the respective primary slot binding that the bridge now occupies. he net effect being illustrated in the example for one -to- bridge in Figure, -to- Bridge implementation. Figure -to- Bridge mplementation B LO LO LO _8 L0968 he interrupt controller must be able to support up to primary devices, a primary device being either a bridge or a physical connector. ach bridge can have up to secondary devices implemented behind the bridge. he largest configuration would mean a maximum of 6 individual connectors, as demonstrated in Figure, aximum llowable onfiguration. Figure aximum llowable - onfiguration B B ingle Board omputer Bus 0 - Bridge evice evice evice evice Bus LO LO LO LO - Bridge - Bridge evice evice evice evice evice evice evice Bus evice Bridge hip - Bridge evice evice evice Bus evice Bus 0 _9 _8 L0969 he use of the wire-o (sometimes also known as shared) binding design, means that the polling and decode of an interrupt request can be significantly less than optimal. he latency for this operation is also unpredictable, i.e. with slots, determining the originator of the interrupt request could take a minimum of, up to a maximum of (-) bus read cycles. his implies a maximum of 6 interrupt sources. controller that can service all of these product scenarios, or some subset thereof, must do the following: support up to primary devices L0970 be able to identify whether a primary device is either igital quipment orporation

4 an on-board - bridge (with up to 'bridged' secondary connectors behind it) O physical connector be able to uniquely identify each of the 6 potential interrupts that can be generated from a bridged device (i.e. four interrupts from each of the secondary devices) be able to uniquely identify each of the potential interrupts that come from a physical connector his implies that the controller will have the following: a register (or similar feature) to detail whether a primary device is a physical connector or an on-board - bridge a register (or similar feature) for each primary device (i.e. in total) to provide status for each interrupt supported by that primary device. ote that the requirements for a bridged device are very different from those for a non-bridged device and the format of the register will be different for each case. a register (or similar feature) to identify which primary device caused an interrupt (i.e. to prevent having to read all interrupt registers to determine the interrupt source). he backplanes developed as part of the program are intended for use with many operating systems and non-lpha single board computers. herefore, they must also be compliant with the shared interrupt scheme as defined in the PG - ard dge onnector Proposal for ingle Board omputer (B) pecification, evision.0 and Bridge Board dge onnector for ingle Board omputer pecification. o meet this two fold requirement the proposed controller supports two unique modes of operation with some means of switching between them. For convenience this was determined to be software selectable. he default mode, at power-up, makes the backplane compliant to the PG specification. his will be known as PG ode. Operating systems (O) needing to make use of the interrupt controller must explicitly switch, via software, to the desired mode of operation. bonus of this design is that the hardware is (in simple terms) a form of hardware interrupt accelerator usable by multiple operating systems and hardware platforms, if their corresponding BO or firmware code is appropriately configured. his mode is known as the ccelerator ode. Generic rchitecture he basic form of the nterrupt controller is shown in Figure 5, nterrupt ontroller Block iagram. he interrupt controller is split into multiple functional blocks, each section's usage being dependent on the desired mode of operation; either PG ode or ccelerator ode. hese modes are mutually exclusive and are discussed in the following sections. Figure 5 nterrupt ontroller Block iagram evice (6 nterrupts) evice (6 nterrupts) evice (6 nterrupts) evice (6 nterrupts) thru nterrupts nterrupt ontroller nterrupt egister nterrupt egister nterrupt egister nterrupt egister onfiguration egister aster nterrupt egister nterrupt outing Logic ystem /O O ntel 878 L097 he example and illustrations used throughout this paper refer to a specific lpha 06 PG - single board computer implementation. he concepts are generic and can be fully utilised by alternative platforms. n Figure 5, the interrupt controller functionality is shown within the shaded area and is physically located on the backplane; the ystem /O and PU are resident on the actual B. nt nt nt nt - nterrupt nterrupt 06 PU Q<> Q<> igital quipment orporation

5 evice (6 nterrupts) evice (6 nterrupts) evice (6 nterrupts) evice (6 nterrupts) nterrupt ontroller nterrupt egister nterrupt egister nterrupt egister nterrupt egister onfiguration egister aster nterrupt egister nterrupt outing Logic n PG mode, device interrupts are taken directly to the nterrupt outing Logic where they are simply wire- Oed, to provide, B, and, as specified in the PG - ard dge onnector Proposal for ingle Board omputer (B) pecification, evision.0 and PG - Bridge Board dge onnector Proposal for ingle Board omputer (B), evision. hese are routed to the ystem /O Q lines in the demonstration example provided. Figure 6 nterrupt ontroller Block iagram PG ode nterrupts thru ystem /O O ntel nterrupt 06 PU Q<> Q<> L097 PG ode his is the default mode for the nterrupt ontroller on power-up. he egister Logic blocks are disabled and all inputs are fed into the nterrupt outing Logic block. t implements the necessary binding to be compliant with the appropriate PG specification (as per figures & ) and addresses the routing for 6 individual interrupt request lines to outputs, i.e. the four B X#. When in this mode, the nterrupt ontroller appears as shown in Figure 6, nterrupt ontroller Block iagram - PG ode. nterrupt egisters he aster nterrupt egister and nterrupt egisters through are not available and have no meaning when accessed (i.e. writes are not stored and reads give indeterminable results). onfiguration egister ccelerator ode he interrupt controller or ccelerator ode can be enabled via software only. o enable this mode, the ontrol egister must be written to, prior to enabling interrupts. When in this mode the nterrupt ontroller looks as shown in Figure 7, nterrupt ontroller Block iagram - ccelerator ode. Figure 7 nterrupt ontroller Block iagram ccelerator ode evice (6 nterrupts) evice (6 nterrupts) evice (6 nterrupts) evice (6 nterrupts) nterrupt ontroller nterrupt egister nterrupt egister nterrupt egister nterrupt egister onfiguration egister nt nt nt nt he onfiguration egister again has no real meaning in this mode, however it is always active since it is the means to switch to ccelerator mode. ee later for details on how this is achieved. nterrupt outing nterrupts aster nterrupt egister nterrupt outing Logic ystem /O O ntel nterrupt nterrupt 06 PU Q<> Q<> L097 igital quipment orporation 5

6 he software sequence for enabling ccelerator ode is shown in Figure 8, ccelerator ode - oftware nabling Bit Funct 5 0 equence Figure 8 ccelerator ode - oftware nabling equence 9 F G onfiguration egister 8 F G 7 F G 6 F G 5 P O K B 6 5 aster nterrupt egister B 0 L099 he backplane configuration details are stored in FG[:], (bits [9:6] of the onfiguration egister), defining which primary slots are connectors and which are bridge chips i.e. which have four versus sixteen potentially active interrupt lines. Power O in PG ode Write to onfiguration ddress et ode bit ow in ccelerator ode L097 able onfiguration and aster nterrupt egister n PG ode, the bit defines whether the four X# interrupts are routed to the ystem /O or whether the one interrupt line is routed directly to the PU. n typical PG applications interrupts are heavily used and the P bit can free up to four interrupts. t is always set in ccelerator ode. K is used to support interrupt polling. When enabled, the interrupt status bits in the four interrupt registers are dependent upon their corresponding K bits. When disabled, they match the status of the interrupt source. ll registers are implemented as bit registers addressable in space. (he interrupt controller could as easily have been implemented as a device, however it would then be counted as a full device load and could have had an adverse impact on the total 0-load limit.) onfiguration and aster nterrupt egister able, onfiguration and aster nterrupt egister, defines the register bit allocation. he onfiguration egister is always active and is the only means of controlling the nterrupt ontroller's behaviour. he onfiguration and aster nterrupt egister is located at /O address 0500h - 050h. part from having the mode enable bit (O), it also stores the high order /O address bits for nterrupt egisters through ([5:]). he low order address bits ([:0]) are fixed at 0000, 000, 000, 00 respectively. he aster nterrupt egister is only enabled when in accelerator mode. his register gets its input from the nterrupt egisters and is used to determine which nterrupt egister should be read to find the source of the interrupt. [:] reflects the status of the corresponding nterrupt egister[:], i.e. status is the logical Oing of the sixteen nterrupt tatus bits stored in nterrupt egister. [:] can be correspondingly masked by [:]. n this way the interrupt source can be determined in two read cycles; one to the aster nterrupt egister and one to the specific nterrupt egister. nterrupt egisters[:] ach of the nterrupt egisters represents a primary device. he following table maps the primary device to it's associated nterrupt registers. he address of these interrupt registers is defined by the contents of the bits in the onfiguration egister. igital quipment orporation 6

7 able, nterrupt egister apping, defines the onfiguration pace ddress for each of the primary devices and gives an example of possible /O addresses for each nterrupt register. able nterrupt egister apping he exact format of each interrupt register is dependent on whether a primary device is a physical connector or a - bridge. he format of nterrupt egister through is defined by the FG bits within the onfiguration egister. his can be used to determine the exact configuration of the backplane, and hence the number of potentially active interrupt lines. able, nterrupt egisters[:], defines the register bit allocation. able nterrupt egisters[: ]. f the primary device is a connector, the nterrupt egister stores only the status and K bits for four interrupt lines, i.e. only bits [:0] and [9:6] have any meaning. Figure 9 nterrupt/ ask Operation nterrupt egister ask Bits. 5 6 nterrupt tatus Bits. 5 6 [6 off]. O o ingle aster nterrupt x# L0975 he interrupt U bits [5:0] are ed with their corresponding 6 nterrupt egister K bits. he results of each operation are then Oed together to form a single X# signal that is routed to the aster nterrupt egister, as illustrated in Figure 9 nterrupt / ask Operation. Primary evice ssociated nterrupt egister Primary evice onfig. pace ddress Bit Funct ote : f a multifunction option card (i.e. an option with a bridge) is plugged into a physical connector, there is support for the primary interrupts from behind its onboard bridge. f more than four interrupts are used (i.e. via sharing), they are not supported. ccelerator nterrupt ecode Hardware nterrupt rchitecture n ccelerator ode, [:] in the aster nterrupt egister reflects the status of the corresponding nterrupt egister[:] (i.e. [:] status is the logical Oing of the sixteen nterrupt tatus bits stored in each nterrupt egister [: ]). his two stage interrupt register strategy allows rapid decoding of the interrupt source without expanding any individual register set beyond bits. Figure 0 nterrupt ecode chematics nterrupt egister /O ddress 050h-05h 05h-057h 058h-05Bh 05h-05Fh B B B B B B B L099 0 B L099 When the primary device is a - bridge, the corresponding nterrupt egister must store the status of up to 6 interrupt lines for secondary connectors implemented behind the - bridge and also the K bits for each individual interrupt line (i.e. all [:0] bits are valid). igital quipment orporation 7

8 he [:] interrupt status bits are ed with their corresponding aster nterrupt egister K bits. he results of each operation are then Oed together to form the interrupt request signal that is routed to a single Q on the PU, as illustrated in Figure 0 nterrupt ecode chematics. aster nterrupt egister ask Bits O o ingle PU nterrupt he final logical routing of the aster nterrupt register is not limited by the K bit status. outing of the [:] interrupts is determined only by the value of the interrupt status bit and it's corresponding mask bit. Firmware/BO nterrupt ecode he interrupt accelerator decode architecture influences the host PU firmware/bo, and is usually transparent to the target operating system. he firmware/bo must implement a decode routine as per Figure oftware ecode teps. nterrupt egister # nterrupt egister # nterrupt egister # nterrupt egister # nterrupt tatus Bits of an interrupt request, is two bus read cycles. he decode operation logically occurs in parallel with the read cycle. nterrupt Latency L0976 he interrupt dispatch latency is the elapsed time from receipt of an interrupt request t-o dispatch to the interrupt service routine. he interrupt service latency is the elapsed time from entry of the interrupt service routine to its completion. Figure oftware ecode teps - ccelerator ode nterrupt egister nterrupt equest cknowledge aster nterrupt egister O which nterrupt egister nterrupt egister O which nterrupt Line Find evice ype from onfiguration eg. WO BU YL nitiate nterrupt ervice outine L0977 he predictable and repeatable time to dispatch the appropriate interrupt vector (service routine), after receipt xact interrupt latency, for a given system configuration, will be operating system dependent (i.e. the interrupt service latency may vary significantly between operating systems even when the dispatch latency in firmware/bo is identical). Operating systems vary in interrupt service routine efficiency and can be equally dissimilar across hardware platforms. he interrupt accelerator optimises the hardware aspect of this process. PG ode he wire-oed binding strategy is not optimal in large slot configurations and most - bridge implementations. t directly impacts the achievable interrupt dispatch latency, and some interrupt service methodologies (e.g. round robin, etc.) can further reduce efficiency in these types of environments. igital quipment orporation 8

9 he dispatch latency is also unpredictable i.e. with slots, determining the originator of the interrupt request could take a minimum of, up to a maximum of (-) bus read cycles. Figure oftware ecode teps - PG ode can be significantly less than optimal. he interrupt latency is also unpredictable. he proposed interrupt accelerator design described in this paper results in the predictable, repeatable (consistent) and improved interrupt dispatch latency, key for real-time applications. he interrupt dispatch latency in very large systems can result in severe degradation of the system performance. his is caused by /O devices 'stalling' because they cannot get serviced efficiently. n extreme configurations, a particular device may 'never' get its interrupt serviced, resulting in failure of that functionality (e.g. a network card may 'drop' in-coming packets or a serial line may 'drop' received characters). nterrupt egister nterrupt equest cknowledge POLL First lot to dentify Originator s lot Originator? Y Poll ext lot to dentify Originator ccelerator ode Find evice ype From onfiguration eg. nitiate nterrupt ervice outine he accelerator architecture offers predictable and consistent interrupt dispatch latency resulting in higher performance for large configurations. Predictability is key in most real-time applications. he corresponding accelerator interrupt dispatch latency is always two Bus read cycles. Physical mplementation his paper is not intended to imply any particular physical implementation. he generic functionality for a PG application can be implemented either as an or based device, however it could also be supported in alternative bus architectures. ummary tandard motherboard implementations provide interrupt binding (for the firmware/bo decode in the physical etch routing). his binding structure becomes congested when additional (bridged) buses are implemented in the system. his means that the polling and decode of an interrupt request cknowledgements he program was the cooperative effort of a large number of engineers whose initiatives have contributed to this paper. hanks in particular to lan ilne, ean cgrane, obin lexander, Vikas ontakke and John Lenthall, all of whom have worked within the &t engineering design team to ensure the successful product implementation of the concepts detailed in this paper. eferences. Local Bus pecification, evision.0. PG, - ard dge onnector Proposal for ingle Board omputers, evision.0. PG, - Bridge Board dge onnector Proposal for ingle Board omputers, evision.0. to Bridge rchitecture pecification, evision.0 uthor aximum (n-) BU YL L0978 oss rmstrong, the Project Leader for the GL odular omputing omponents () program, is a igital quipment orporation 9

10 principal hardware design engineer with the GL mbedded and eal-ime (&t) engineering Organization. oss holds a B.c. (ng.) Hon's from berdeen University and a joint aster of echnology anagement (...) from trathclyde and Heriot Watt Universities. GL believes the information in this publication is accurate as of its publication date; such information is subject to change without notice. GL is not responsible for any inadvertent errors. GL conducts its business in a manner that conserves the environment and protects the safety and health of its employees, customers, and the community. GL and the GL logo are trademarks of igital quipment orporation. icrosoft is a registered trademark and Windows is a trademark of icrosoft orporation. opyright 996 igital quipment orporation ll ights eserved. igital quipment orporation 0