Leveraging the PCI Support in Windows 2000 for StarFabric-based Systems

Size: px

Start display at page:

Download "Leveraging the PCI Support in Windows 2000 for StarFabric-based Systems"

Holly Fleming
6 years ago
Views:

1 Leveraging the PCI Support in Windows 2000 for StarFabric-based Systems Mark Overgaard President, Pigeon Point Systems

2 Agenda Background StarFabric Bus Driver for Windows 2000 Using StarFabric s Inherent PCI- and CompactPCI-Compatibility Advanced Uses of StarFabric Facilities Conclusion

3 Background Pigeon Point Systems CompactPCI Support in Windows 2000 (Win2K) Note: References to Microsoft Windows are primarily to Win2K, but should be applicable to successor releases, including Windows XP/2002 (formerly code-named Whistler)

4 Pigeon Point Systems Founded in 1997 to focus on high availability system s/w for CompactPCI Authored software aspects of CompactPCI Hot Swap Specification (PICMG 2.1 R1.0) Led software aspects of Revision 2.0 just adopted by PICMG in January, 2001 Chair of PICMG Software Interoperability Subcommittee (for PICMG 2.12) First demo and product shipments for General Use Hot Swap software infrastructure products

5 PPS Background (Cont.) Licensed Hot Swap Kit to Microsoft for Windows XP/2002 Embedded Also to leading CompactPCI platform & board vendors Licensed Open MultiComputing Kit (OMCK backplane messaging) to MCG, other vendors Member of Advisory Board for StarGen, Inc. Founding member of StarFabric Working Group

6 CompactPCI Support with Win2K Standard Win2K operates on CompactPCI platform as on any PCI-based systems But, no built-in knowledge of CompactPCI-speciific features such as hot swap Win2K s Plug and Play (PnP) has foundation for dynamic arrive/depart of PCI devices Additional layer can interact with CompactPCI-specific hardware and invoke PnP facilities Key signal ENUM# indicates board arrival or imminent departure ENUM# detection mechanism is platform-dependent Pigeon Point Hot Swap Kit (HSK) is one example of such a layer Works as is with StarFabric s inherent CompactPCI compatibility Can be extended for advanced StarFabric configurations

7 Base Win2K

8 PnP-based Hot Swap w/ Win2K

9 Agenda Background StarFabric Bus Driver for Windows 2000 Using StarFabric s Inherent PCI- and CompactPCI-Compatibility Advanced Uses of StarFabric Facilities Conclusion

10 StarFabric Bus Driver for Win2K Layers above Fabric Primitives Library (FPL) Enables visibility in Win2K via StarFabric of: Native StarFabric devices, for example: SG1010 StarFabric switches Agere Systems T8150 H.1x0-to-StarFabric bridges PCI buses accessed via StarFabric end nodes and the PCI devices on those buses Initial StarFabric end node: SG2010 PCI-to-StarFabric bridge Development sponsored by StarGen at PPS

11 StarFabric Bus Driver (Cont.) Useful for systems with either address-routed or path-routed access to PCI segments Address-routed ( PCI mode ) access still benefits from event handler for fabric exceptions Path routed ( fabric mode ) access requires considerable additional support Even in this usage, however, goal is to maximize reuse of existing PCI software

12 Agenda Background StarFabric Bus Driver for Windows 2000 Using StarFabric s Inherent PCI and CompactPCI Compatibility Advanced Uses of StarFabric Facilities Conclusion

13 PCI/CompactPCI Compatibility SG2010 provides fully transparent bridging to extension chassis or extension bus segments Excellent flexibility in subsystem architecture StarFabric Working Group s Proof of Concept (PoC) chassis shows example uses for SG2010s: Spawning CompactPCI bus segments in segment controller boards linked to StarFabric backplane CompactPCI bus segment slots use existing CompactPCI boards Uses SG2010 in leaf mode Spawning PMC sites with direct links to StarFabric backplane Existing PMCs can benefit from a dedicated PCI bus (64bit/66MHz) Sites available on segment controller boards Uses SG2010 in leaf mode Enabling existing Single Board Computers (SBCs) to access/manage StarFabric Uses SG2010 in root mode in peripheral slot driven by SBC

14 PCI/CompactPCI Compatibility (Cont.) All PCI buses spawned by SG2010s in PoC chassis have redundant StarFabric connections PoC chassis also supports dual redundant SBCs

15 21-Slot Proof of Concept Backplane Links on J5 H110 0 H110 1 H Links on J3 PCI 0 PCI 1 PCI 2 PCI 3 PCI 4 System slot: Existing Single Board Computer (SBC) board Fabric-native slot: Bus/fabric (#2/5) or fabric-native (#3/6) board Bus segment controller slot: Spawns PCI/H.110 bus segments StarFabric full-duplex link = 4 CPCI connector rows IO slot: Existing CompactPCI peripheral boards

16 Proof of Concept Backplane Interconnections Slot 2 Slot 5 Slot 3 Slot 6 Slot 7 Slot 12 Slot 17 Link with etch interconnect between slots Optional front-panel links with RJ45 s for inter-chassis connections

17 Using Fabric-Native Slots Fabric-native boards or PMC-onfabric site PMC site P C I SG 2010 PCI 0 PCI 1 Existing SBCs Bus/Fabric boards RJ45 RJ45 RJ45 SG 1010 SG 1010 SG 1010 SG 2010 PCI CompactPCI bus connection not used in these slots RJ45 SG 2010 PCI

18 Dual Hosted, Multi-Segment Setup Existing SBC cards as dual redundant system hosts PCI 0 PCI 1 PCI 2 PCI 3 PCI 4 RJ45 SG 1010 RJ45 RJ45 RJ45 SG 1010 SG 1010 SG 2010 PCI Bus/Fabric boards: SG1010 switches & SG2010 in root mode for SBC access/control Segment controller boards: SG2010s in leaf mode for CompactPCI bus and PMC site PMC site P C I SG 2010 SG 2010 PCI

19 CompactPCI Hot Swap Support in StarFabric Existing Pigeon Point HSK transparently supports hot swapping in CompactPCI extension chassis/bus segments Support for configuring sparse address windows on P2P bridges to CompactPCI segments Applies to StarFabric-emulated P2P bridges Ensures resource availability for hot insertions ENUM# signals forwarded by StarFabric to main CompactPCI chassis/bus segments Existing ENUM# detection h/w and s/w works as is in that main chassis

20 CompactPCI Compatibility (Cont.) SG2010 provides excellent hot swap support as a CompactPCI bus interface First available device compliant with PICMG 2.1R2.0 s advanced programming interface Disappears from config space when extraction authorized Avoids mid-transaction extractions Also disappears when BD_SEL# breaks or is deasserted BD_SEL#: Signal on CompactPCI backplane that controls power up/down of board Short connector pin: breaks first on extraction, makes last on insertion Extends mid-transaction extraction safety

21 H.1x0 to StarFabric Bridge Allows Extended TDM Bus, Too T8150 allows extension of H.100/H.110 segments in addition to PCI Applies to both variants of the Time Division Multiplexed (TDM) CT Bus H.100 for PCI systems H.110 for CompactPCI systems Single fabric connection to extension chassis supports both PCI and TDM traffic See later session: What Does StarFabric Mean for Telephony? for more details

22 Agenda Background StarFabric Bus Driver for Windows 2000 Using StarFabric s Inherent PCI- and CompactPCI-Compatibility Advanced Uses of StarFabric Facilities Conclusion

23 Advanced Uses of StarFabric Facilities with Win2K Handling Fabric Events Using Native Fabric Connections Potential ENUM# Handling Optimizations Supporting Path-Accessed PCI Buses

24 Handling Fabric Events Useful for PCI devices on address-routed leaf buses PCI devices on path-routed leaf buses Native StarFabric devices Need to distinguish: Signaled events: resulting from PCI bus signals (e.g., INTx, ENUM#) on leaf buses Chip events: resulting directly from a StarFabric device

25 Handling Fabric Events (Cont.) For PCI devices on address-routed leaf buses Signaled events are Reproduced on root bus by default Handled by normal h/w and s/w measures for PCI (e.g., INTx lines and handling) Chip events Would primarily result from exceptions or explicitly designated events inside the fabric Are important to handle in robust systems, even if StarFabric PCI hierarchy emulation is used For PCI devices on path-routed leaf buses StarFabric h/w provides much flexibility, potentially constrained by O/S

26 Win2K Event Handling Interface Exported by StarFabric bus driver for kernel mode and user mode clients Kernel mode client uses device interface mechanism to get function table, including functions to: Get Channel 255 CSR Address, Fabric ID, etc. Register/unregister Interrupt Service Routines (ISRs) Subscribe/unsubscribe for events ISRs are source interchangeable with normal Win2K ISRs; called at interrupt time Event subscription notifications happen on less urgent basis

27 Win2K Event Handling (Cont.) User mode interface provides IOCTL-based support for: Subscribing and unsubscribing for events Subscribing application provides handle to event object and size of event queue Retrieving an event from the queue Loss of events due to queue overflow can be detected via sequence numbers

28 Using Native Fabric Connections Based on explicitly arranged mappings in gateway devices at each end of connection Can leverage Multicast Class of service Bandwidth reservations Useful for PCI devices in either address-routed or path-routed contexts Enables focused use of advanced features, even in address-routed contexts

29 Demo Shows Native Fabric Connections in Action Communicating nodes are PCI Mezzanine Cards (PMCs) in Proof of Concept chassis Three PMCs in segment controller boards that spawn CompactPCI segments Each of these PMCs has a dedicated SG2010-based interface to the fabric One PMC in non-intelligent CompactPCI carrier in an SG2010-spawned CompactPCI bus One PMC in a SG2010-based carrier linked by Cat 5 cable to the two redundant fabric boards Dual redundant fabric boards provide fabric connections among chassis segments

30 Connections Among Fabric- Linked PMCs Simulate high bandwidth data plane communication among peers Established and monitored via address-routed PCI control plane Use source and destination channel features of StarFabric Provide convenient address translations among peers Destination channel on each PMC establishes h/w protected region for interaction w/ all its peers Source channel on each PMC establishes h/w-protected buffer area within those regions, for use by that specific PMC

31 Key StarFabric Properties Demonstrated Redundant routes: one through each fabric board Failover can be accomplished: In hardware on primary route failure, or In software for managed switch-over to secondary route Class of service with competing data streams into a single PMC Multicast distribution of a single stream to multiple PMCs

32 Elements of a PMC-based Fabric Node

33 Fabric Connections in Action

34 Potential ENUM# Handling Optimizations Useful for both address-routed and path-routed CompactPCI leaf buses Can be configured independently of operational mode Standard ENUM# detection assumes that s/w handler polls all devices on all buses for ENUM# activator StarFabric-extended system could include 10s or 100s of buses Potential approach: configure ENUM# to be reported via event record, versus wire assertion on root bus Host registers for ENUM# event notification Notification designates the leaf bus where ENUM# was activated, so only devices on that bus need to be polled

35 ENUM# Handling Optimizations (Cont.) Other implications of ENUM# detection via event record ENUM# detection for StarFabric-accessed buses becomes platform-independent Traditional ENUM# detection is platform-dependent Host processor need not be a CompactPCI board Example: racked mounted conventional server w/ SG2010 as fabric root could manage many CompactPCI chassis

36 Supporting Path-Accessed PCI Buses Address-accessed [Compact]PCI buses fully emulate P2P bridges Enables complete reuse of existing devices and device drivers Path-routed access could potentially add valuable advanced options, such as Redundant routes to leaf buses for route resilience Multicast and class of service treatment for selected PCI transfers Challenge: Enable path-accessed configurations, While maximizing use of existing software, In context of rich PCI infrastructure of Win2K.

37 Supporting Path-Accessed PCI in Win2K: Overall Approach Simulate in software a two-level PCI hierarchy Root PCI-to-PCI bridge corresponds to root SG2010 PCI-to- StarFabric bridge Leaf PCI-to-PCI bridges correspond to leaf StarFabric-to-PCI bridges Simulated bridges provide familiar means for Win2K to manage devices and drivers associated with leaf buses Address windows in leaf bridges establish address space partitioning among leaf buses Set up connections in SG2010 gateways to route PCI transactions consistently w/ PCI hierarchy From the root bus to the leaf buses From each leaf bus to its peers, where appropriate

38 Path-Accessed PCI in Win2K: Overall Approach (Cont.) Hardware routing accomplished via gateway mimics routing that would have been done for address-accessed segments Key difference: all controls inherent in StarFabric path-routed configurations are available

39 Limitations on Path-Accessed PCI in Win2K Preliminary; will be revisited as implementation proceeds IO addressing of devices in path-accessed leaf buses not supported by StarFabric hardware Leaf devices & drivers must support memory accesses PCI interrupts for existing PCI device drivers must be handled by standard means Standard swizzling of INT lines between leaf and root buses Standard means to poll ISRs to identify interrupt ownership

40 Agenda Background StarFabric Bus Driver for Windows 2000 Using StarFabric s Inherent PCI- and CompactPCI-Compatibility Advanced Uses of StarFabric Facilities Conclusion

41 Conclusions StarFabric s PCI- and CompactPCI-compatibility under Win2K allow: Using existing devices and device drivers as is Architectural flexibility for subsystems in many dimensions Transparent hot swap for CompactPCI bus segments using Pigeon Point Hot Swap Kit StarFabric bus driver for Win2K enables: Basic fabric event handling for legacy mode Incremental use of advanced StarFabric facilities, including Native fabric connections Path-accessed PCI subsystems

42 Conclusions (Cont.) Native fabric connections between StarFabric-linked PCI subsystems under Win2K Support accesses between subsystems with hardwareenforced/enabled Address-range checks Class-of-service-based preferential flow rates Redundant routes Allow data plane traffic to share one fabric with control plane traffic, even in legacy mode Path-accessed subsystems under Win2K Maximize accessibility of advanced StarFabric facilities Still support high compatibility with existing PCI devices and device drivers Overall, StarFabric and Win2K are an excellent fit for each other!

43 Questions? Mark Overgaard Pigeon Point Systems Related session: Wednesday, May 16: 1:45-2:45pm Robust Hot Swap Infrastructure Software: The Key Ingredients

Management building blocks speed AdvancedTCA product development

TELECOM S P E C I A L F E A T U R E Management building blocks speed AdvancedTCA product development By Mark Overgaard The IPM Sentry Intelligent Platform Management products provide off-the-shelf building