PCI Express ExpressModule Electromechanical Specification Revision 1.0

Transcription

1 PCI Express ExpressModule Electromechanical Specification Revision 1.0 February 14, 2005

2 Revision Revision History Date 1.0 Initial release. 02/14/05 PCI-SIG disclaims all warranties and liability for the use of this document and the information contained herein and assumes no responsibility for any errors that may appear in this document, nor does PCI-SIG make a commitment to update the information contained herein. Contact the PCI-SIG office to obtain the latest revision of this specification. Questions regarding the PCI Express ExpressModule Electromechanical Specification or membership in PCI-SIG may be forwarded to: Membership Services administration@pcisig.com Phone: Fax: Technical Support techsupp@pcisig.com DISCLAIMER This PCI Express ExpressModule Electromechancial Specification is provided "as is" with no warranties whatsoever, including any warranty of merchantability, noninfringement, fitness for any particular purpose, or any warranty otherwise arising out of any proposal, specification, or sample. PCI-SIG disclaims all liability for infringement of proprietary rights, relating to use of information in this specification. No license, express or implied, by estoppel or otherwise, to any intellectual property rights is granted herein. ExpressModule, PCI, PCI Express, PCIe, and PCI-SIG are trademarks or registered trademarks of PCI-SIG. All other product names are trademarks, registered trademarks, or service marks of their respective owners. Copyright 2005 PCI-SIG 2

3 Contents 1. INTRODUCTION TERMS AND DEFINITIONS REFERENCE DOCUMENTS SPECIFICATION CONTENTS OBJECTIVES OVERVIEW EXPRESSMODULE AUXILIARY INTERFACES REFERENCE CLOCK Low Voltage Swing, Differential Clocks Spread Spectrum Clocking (SSC) REFCLK Specifications REFCLK Phase Jitter Specification MRST# SIGNAL WAKE SIGNAL INTERNAL STORAGE INTERFACE Storage Interface Signal Definitions POWER MANAGEMENT Initial Power-Up (G3 to L0) Power Management States (S0 to S3/S4 to S0) Power Down MANAGEMENT Capacitive Load of High-power SMBus Lines Minimum Current Sinking Requirements for SMBus Devices SMBus Back Powering Considerations Power-on Reset SMBus Termination and Power SMBAlert Management Bus Topology Data Integrity Basic Management Status Register Satellite Management Controller Remote Management Card Access VPD (FRU) Information Format Using an SMBus Multiplexer with a Host Controller Implementing ExpressModule Management in an IPMI Environment Sending Data to a Management Controller on a Module Receiving Data From a Management Controller on a Module SMBAlert Control and Status Message Class Values for Management Controller Messaging Retention of Output Data SMBAlert Signal Handling for Message Transfers Polling for Output Data

4 SMBus NACKs and Error Recovery PEC Handling Summary of SMBus Commands Values for Management Controller Messaging SMBus Timeout and Hang Handling Management Controller Messaging Timing IPMI Management Controller Message Formats Pre-assigned/Reserved Slave Addresses AUXILIARY SIGNAL PARAMETRIC SPECIFICATIONS DC Specifications AC Specifications HOT INSERTION/REMOVAL SCOPE HOT-PLUG SUB-SYSTEM ARCHITECTURE Power Enable Power Fault Wake Module Power Good Module Reset Present Detection System Management Bus System Management Bus Alert Power LED Attention LED Manual Retention Latch Electromechanical Interlock Electromechanical Interlock Status Attention Switch MODULE POWER INTERFACE POWER Module Primary Power Supply Module Auxiliary Power Supply EXPRESSMODULE POWER SUPPLY REQUIREMENTS POWER CONSUMPTION POWER SUPPLY SEQUENCING POWER SUPPLY DECOUPLING MODULE PCI EXPRESS INTERFACE PCI EXPRESS LINK SIGNALS PCI EXPRESS ELECTRICAL TOPOLOGIES AND LINK DEFINITIONS Topologies Link Definitions PCI EXPRESS ELECTRICAL BUDGETS AC Coupling Capacitors Insertion Loss Values (Voltage Transfer Function) Jitter Values

5 Crosstalk Lane-to-Lane Skew Equalization Skew Within the Differential Pair EYE DIAGRAMS AT THE EXPRESSMODULE INTERFACE ExpressModule Transmitter Path Compliance Eye Diagram ExpressModule Minimum Receiver Path Sensitivity Requirements System Board Transmitter Path Compliance Eye Diagram System Board Minimum Receiver Path Sensitivity Requirements EXPRESSMODULE CONNECTOR CONNECTOR PIN COUNTS CONNECTOR PIN ASSIGNMENTS CONNECTOR INTERFACE DEFINITIONS CONNECTOR SIGNAL INTEGRITY REQUIREMENTS AND TEST PROCEDURES CONNECTOR ENVIRONMENTAL AND OTHER REQUIREMENTS Environmental Requirements Mechanical Requirements Current Rating Requirement Additional Considerations MODULE MECHANICAL SPECIFICATION MECHANICAL OVERVIEW DIMENSIONS AND TOLERANCES SYSTEM DATUM PLANE DEFINITION MODULE DESCRIPTION Module Materials Singlewide and Doublewide Module Form Factor Module Raw Card Module Ejector and Latch Details I/O Plate Details Module Air Vent Design for EMI FILLER COMPONENT OR MODULE CHASSIS SLOT DESCRIPTION Backplane and Chassis Slot Details DESIGN CONSIDERATIONS COOLING/THERMAL ENVIRONMENT Longitudinal Cooling Default Mode Lateral Cooling Alternate Mode Cooling Consideration for Storage Extension Slot Module EMI Design MODULE ESD DESIGN MODULE INTEROPERABILITY SLOT/MODULE COLOR CODING AND LABELING Module Hot Remove and Add Capability Modules That May Require a System Power Down

6 Internal Storage Modules Optional Module Labeling Slot Labeling on the System Optional x16 Doublewide Slot Labeling Slot Numbering and Labeling

7 Figures FIGURE 2-1: DIFFERENTIAL REFCLK WAVEFORM FIGURE 2-2: EXAMPLE REFERENCE CLOCK SOURCE TERMINATION FIGURE 2-3: SINGLE-ENDED MEASUREMENT POINTS FOR ABSOLUTE CROSS POINT AND SWING. 19 FIGURE 2-4: SINGLE-ENDED MEASUREMENT POINTS FOR DELTA CROSS POINT FIGURE 2-5: SINGLE-ENDED MEASUREMENT POINTS FOR RISE AND FALL TIME MATCHING FIGURE 2-6: DIFFERENTIAL MEASUREMENT POINTS FOR DUTY CYCLE AND PERIOD FIGURE 2-7: DIFFERENTIAL MEASUREMENT POINTS FOR RISE AND FALL TIME FIGURE 2-8: DIFFERENTIAL MEASUREMENT POINTS FOR RINGBACK FIGURE 2-9: REFERENCE CLOCK SYSTEM MEASUREMENT POINT AND LOADING FIGURE 2-10: POWER UP FIGURE 2-11: POWER MANAGEMENT STATES FIGURE 2-12: POWER DOWN FIGURE 2-13: EXAMPLE SMBUS TOPOLOGY FIGURE 2-14: BASIC MANAGEMENT STATUS REGISTER ACCESS FIGURE 2-15: BMC SINGLE-PART WRITE FIGURE 2-16: MULTI-PART WRITE START FIGURE 2-17: MULTI-PART WRITE MIDDLE FIGURE 2-18: MULTI-PART WRITE END FIGURE 2-19: SINGLE-PART READ FIGURE 2-20: MULTI-PART READ START FIGURE 2-21: MULTI-PART READ MIDDLE FIGURE 2-22: BMC MULTI-PART READ END FIGURE 2-23: MULTI-PART READ RETRY FIGURE 2-24: GET MESSAGE INTERFACE STATUS FIGURE 2-25: SET MESSAGE INTERFACE CONTROL FIGURE 2-26: BMC TO MODULE IPMI STANDARD REQUEST FIGURE 2-27: BMC TO MODULE IPMI STANDARD RESPONSE FIGURE 2-28: MODULE TO BMC IPMI STANDARD REQUEST FIGURE 2-29: MODULE TO BMC IPMI STANDARD RESPONSE FIGURE 2-30: WAKE RISE AND FALL TIME MEASUREMENT POINTS FIGURE 3-1: TYPICAL HOT-PLUG INTERFACE IMPLEMENTATION FIGURE 5-1: EXAMPLE INTERCONNECT TERMINATED AT THE CONNECTOR INTERFACE FIGURE 5-2: INSERTION LOSS BUDGETS FIGURE 5-3: JITTER BUDGET FIGURE 5-4: EXPRESSMODULE TRANSMITTER PATH COMPLIANCE EYE DIAGRAM FIGURE 5-5: REPRESENTATIVE COMPOSITE EYE DIAGRAM FOR EXPRESSMODULE RECEIVER PATH COMPLIANCE FIGURE 5-6: SYSTEM BOARD TRANSMITTER PATH COMPOSITE COMPLIANCE EYE DIAGRAM FIGURE 5-7: TWO PORT MEASUREMENT MODEL FIGURE 5-8: REPRESENTATIVE COMPOSITE EYE DIAGRAM FOR SYSTEM BOARD RECEIVER PATH COMPLIANCE FIGURE 6-1: EXPRESSMODULE CONNECTOR FORM FACTOR

8 FIGURE 6-2: EXPRESSMODULE CONNECTOR FORM FACTOR WITH STORAGE EXTENSION FIGURE 6-3: RECOMMENDED FOOTPRINTS FIGURE 6-4: ADD-IN MODULE EDGE-FINGER DIMENSIONS FIGURE 6-5: ADD-IN MODULE WITH STORAGE EDGE-FINGER DIMENSIONS FIGURE 6-6: ILLUSTRATION OF ADJACENT PAIRS FIGURE 6-7: CONTACT RESISTANCE MEASUREMENT POINTS FIGURE 7-1: CHASSIS ASSEMBLY FIGURE 7-2: EXPLODED VIEW OF SINGLEWIDE MODULE FIGURE 7-3: SIMPLE CIRCUIT REPRESENTATION OF AFP STEEL USING FOUR-POINT RESISTANCE METHOD FIGURE 7-4: CRITICAL DIMENSIONS FIGURE 7-5: SINGLEWIDE CROSS SECTION FIGURE 7-6: DOUBLEWIDE CROSS SECTION FIGURE 7-7: MODULE RAW CARD REFERENCE DRAWING FIGURE 7-8: EJECTOR ASSEMBLY FIGURE 7-9: MODULE I/O PLATE FIGURE 7-10: MODULE VENT DESIGN FIGURE 7-11: CHASSIS REQUIREMENTS FIGURE 8-1: LONGITUDINAL SYSTEM AIRFLOW FIGURE 8-2: SINGLEWIDE MODULE: REQUIRED CFM VS. MODULE INLET TEMPERATURE, DEFAULT MODE FIGURE 8-3: SINGLEWIDE MODULE: REQUIRED PRESSURE DROP RANGE VS. FLOW RATE, DEFAULT MODE FIGURE 8-4: UNINTENDED RECIRCULATION FIGURE 8-5: SINGLEWIDE MODULE: REQUIRED CFM VS. MODULE INLET TEMPERATURE, ALTERNATE MODE FIGURE 8-6: SINGLEWIDE MODULE: REQUIRED PRESSURE DROP RANGE VS. FLOW RATE, ALTERNATE MODE FIGURE 8-7: STORAGE SLOT VENTING FIGURE 8-8: EMI GASKET PROFILE FIGURE 8-9: REQUIRED MODULE I/O PLATE ATTENUATION FIGURE 8-10: MODULE LATCH LABELING LOCATIONS FIGURE 8-11: REQUIRED MODULE LABEL WITH EXAMPLES FIGURE 8-12: REQUIRED SLOT LABELING FIGURE 8-13: OPEN VENT SYMBOL FIGURE 8-14: CLOSED VENT SYMBOL

9 Tables TABLE 2-1: REFCLCK DC SPECIFICATION AND AC TIMING REQUIREMENTS TABLE 2-2: MAXIMUM ALLOWED PHASE JITTER WHEN APPLIED TO FIXED FILTER CHARACTERISTIC TABLE 2-3: STORAGE SIDEBAND SIGNALS: I2C/SMBUS TABLE 2-4: STORAGE SIDEBAND SIGNALS: SGPIO TABLE 2-5: BASIC MANAGEMENT STATUS REGISTER TABLE 2-6: VPD (FRU) COMMON HEADER TABLE 2-7: OEM RESERVED SPACE HEADER TABLE 2-8: VPD (FRU) RECORD FORMAT TABLE 2-9: VPD (FRU) RECORD TYPES TABLE 2-10: BOARD INFO DATA FORMAT TABLE 2-11: PRODUCT INFO DATA FORMAT TABLE 2-12: PRODUCT GUID DATA FORMAT TABLE 2-13: EXPRESSMODULE MODULE INFO DATA FORMAT TABLE 2-14: PCI EXPRESS MODULE INFO DATA FORMAT TABLE 2-15: EXPRESSMODULE MULTIPLEXER RECORD DATA FORMAT TABLE 2-16: OEM RECORD DATA FORMAT TABLE 2-17: LANGUAGE CODES TABLE 2-18: INTERFACE STATUS FIELD DEFINITION TABLE 2-19: INTERFACE CONTROL FIELD DEFINITION TABLE 2-20: MESSAGE CLASS VALUES TABLE 2-21: SUMMARY OF SMBUS COMMANDS FOR MESSAGE PROTOCOLS TABLE 2-22: SMBUS MESSAGING TIMING SPECIFICATIONS TABLE 2-23: MANAGEMENT I/O ADDRESSES TABLE 2-24: AUXILIARY SIGNAL DC SPECIFICATIONS TABLE 2-25: POWER SEQUENCING AND RESET SIGNAL TIMINGS TABLE 3-1: POWER SIGNALS TABLE 3-2: MANAGEMENT INTERFACE TABLE 3-3: USER INTERFACE TABLE 3-4: EMLS SIGNAL TABLE 4-1: POWER SUPPLY RAIL REQUIREMENTS TABLE 5-1: ALLOCATION OF INTERCONNECT PATH INSERTION LOSS BUDGET TABLE 5-2: TOTAL SYSTEM JITTER BUDGET TABLE 5-3: ALLOCATION OF INTERCONNECT JITTER BUDGET TABLE 5-4: ALLOWABLE INTERCONNECT LANE-TO-LANE SKEW TABLE 5-5: EXPRESSMODULE TRANSMITTER PATH COMPLIANCE EYE REQUIREMENTS TABLE 5-6: EXPRESSMODULE MINIMUM RECEIVER PATH SENSITIVITY REQUIREMENTS TABLE 5-7: SYSTEM BOARD TRANSMITTER PATH COMPLIANCE EYE REQUIREMENTS TABLE 5-8: SYSTEM BOARD MINIMUM RECEIVER PATH SENSITIVITY REQUIREMENTS TABLE 6-1: EXPRESSMODULE CONNECTOR PIN TYPE/COUNT TABLE 6-2: OPTIONAL STORAGE PIN COUNT/TYPE TABLE 6-3: EXPRESSMODULE CONNECTOR PIN ASSIGNMENTS TABLE 6-4: SIGNAL INTEGRITY REQUIREMENTS AND TEST PROCEDURES

10 TABLE 6-5: TEST DURATIONS TABLE 6-6: MECHANICAL TEST PROCEDURES AND REQUIREMENTS TABLE 6-7: END-OF-LIFE CURRENT RATING TEST SEQUENCE TABLE 6-8: ADDITIONAL REQUIREMENTS TABLE 7-1: DATUM TABLE TABLE 7-2: LED COLOR TABLE 8-1: X8 SLOT CONNECTOR INTEROPERABILITY

11 1 1. Introduction This document is a companion specification to the PCI Express Base Specification. Its primary focus is the implementation of a modular I/O form factor that is focused on the needs of workstations and servers from mechanicals and electrical requirements. The discussions are confined to the modules and their chassis slots requirements. Other form factors are covered in other separate specifications Terms and Definitions x1, x2, x4, x8, x16 x1 refers to one PCI Express lane of basic bandwidth; x4 to a collection of four PCI Express lanes; etc. Auxiliary signals Signals not required by the PCI Express architecture but necessary for certain desired functions or system implementation; for example, the SMBus signals Basic bandwidth Contains one PCI Express lane BMC Base Management Controller Down-plugging Plugging a module with larger number of Links into a connector with a smaller number of Links wired; for example, plugging a module with x8 lanes wired into a slot with x4 lanes wired on the connector Down-shifting Plugging a PCI Express card into a connector that is not fully routed for all of the PCI Express lanes; for example, plugging a x4 card into a x8 capable connector with only four lanes being routed ExpressModule A PCI Express I/O module with the two form factors defined in this specification FRU Field Replaceable Unit Hot plug Insertion and/or removal of an ExpressModule into an active backplane or system board as defined in Revision 1.0 of the PCI Standard Hot Plug Controller and Subsystem Specification. This specification defines the special module requirements to support hot-plug. Hot Swap Insertion and/or removal of an ExpressModule into a passive backplane. The card must satisfy specific requirements to support Hot Swap. Interoperability Ability to plug a PCI Express module into different Link connectors and the system works; for example, plugging a x8 PCI Express I/O module into a x16 module slot Link A collection of one or more PCI Express lanes Pro-class graphics Graphics hardware, such as AGP PRO, that requires more than 40 watts of power Up-plugging Plug a module with a smaller number of Links wired into a slot with a larger number of Links wired in the connector; for example, plugging a module wired for x1 into a slot connector wired for x4 11

12 PCI Express lane VPD One PCI Express lane contains two differential lines for transmitter and two differential lines for receiver. A by-n Link is composed of N lanes. Vital Product Data. Information about the product Reference Documents This specification references the following documents: PCI Express Base Specification, Revision 1.1 PCI Express Card Electromechanical Specification, Revision 1.1 PCI Express Jitter Modeling PCI Express Jitter and BER System Management Bus (SMBus) Specification, Version 2.0 EIA : Environmental Test Methodology for Assessing the Performance of Electrical Connectors and Sockets Used in Business Office Applications EIA-364: Electrical Connector/Socket Test Procedures Including Environmental Classifications 1.3. Specification Contents This specification contains the following information: Adapter Module system interface signals Adapter Module hot insertion and removal Management interface Power delivery Adapter Module/backplane electrical budget Adapter Module form factor and implementation System slot requirements Bandwidth and interoperability matrix 12

13 1.4. Objectives The objectives of this specification are: Support PCI Express bit rate of 2.5 Gb/s with consideration for future bit rates up to 6.25 Gb/s Enable Hot-plug and hot-swap as a native function Leverage server/workstation commonality (I/O connectors and other components) Maximize adapter module interoperability for user flexibility Provide a low cost solution 1.5. Overview PCI Express is the third generation of multi-purpose I/O interface that can be used across the computing industry from mobile through high-end servers and even communication equipment. The broad usage of this technology yields the need to package the I/O adapters in form factors that meet the needs for the environment that they will be used in. The focus of this specification is on a form factor that fulfills the needs of servers and workstations. Servers have always been required to be highly reliable. The demands on server reliability are steadily increasing while the I/O structure has not effectively changed in 20+ years. While the industry has been able to work around many issues with this, its shows up as increased cost to the server. A good example of this was adding hot plug or hot swap to PCI cards. This was done at a large expense to the system and required opening the chassis while operating (risking interruptions of service caused by moving the systems or by moving internal or external cables). Servers need an I/O adapter form factor that addresses items like closed chassis adapter removal/insertion, integrated hot-plug, adapter protection from ESD and handling damage, standardized management interfaces and features, adequate cooling, single bulk power supply, and reduced I/O footprint. Workstations have many of the same needs as servers with one exception, the graphics adapter. Workstation high-end graphics adapters require high wattages and require more adapter board space than the typical server I/O application. Workstation high-end graphics adapter requirements are not within the scope of this specification. This specification defines two PCI Express modular form factors for I/O adapters, otherwise defined as ExpressModules. These modular I/O adapters are designed to provide closed chassis installation and removal. They are also designed for native hot-plug, meaning that no damage will occur to the module or chassis from installing or removing the modules while the chassis is powered. The two form factors are both the same height and depth and vary only by the width or slot space they consume in the chassis. The first module is considered a singlewide and will fit in any slot that supports ExpressModules. The second module is the doublewide module and requires two adjacent slots in a chassis supporting ExpressModule slots. 13

14 A singlewide module supports up to a x8 PCI Express connection, and doublewide modules can support up to x16 connection. Chassis support for x16 doublewide slots is optional, and these slots shall be marked as defined in this specification. Chassis that support hot-plug x16 doublewide slots shall also support dynamic reconfiguration so that two modules may be exchanged for a single x16 module or vice versa during a hot-removal/hot-install events. A minimum management interface is standard on all modules. This consists of an EEPROM on each module that provides management data about the adapter to the host system. An optional internal storage interface is supported for up to four 1x ports of SAS/SATA and a sideband interface for controlling drive LEDs. ExpressModules and chassis that support them share requirements for cooling. Chassis that support ExpressModules are required to provide a minimum airflow to each slot as defined by this specification. ExpressModules are required to provide a minimum and maximum airflow resistance. This results in value to both ExpressModules and supporting chassis. ExpressModules will have a defined airflow for I/O cooling, and the I/O thermal load is not added to the chassis. 14

15 2 2. ExpressModule Auxiliary Interfaces The auxiliary signals are provided on the connector to assist with certain system level functionality or implementation. These signals are not required by the PCI Express architecture. The high-speed signal voltage levels are compatible with advanced silicon processes. The low speed signals are defined to be compatible with +3.3V signaling, as this is the lowest common signaling available and are referenced to common logic ground. ExpressModule slots only provide +12V and a +3.3Vaux supplies to the module. A DC to DC converter on the module is required to provide a supply for the 3.3 V compatible signals that are not aux powered. The use of 3.3 V signaling allows the non- PCI Express signals to be used with existing control bus structures avoiding a buffered set of signals and bridges between the buses. The PCI Express connector and ExpressModule interfaces support the following auxiliary signals: REFCLK-/REFCLK+ (Must be supplied by system): low voltage differential signals. MRST# (required): indicates when the applied main power is within the specified tolerance and stable. MRST# goes inactive after a delay of T PVPERL time from the power rails achieving specified tolerance on power up. WAKE (optional): an active high signal that is driven high by a PCI Express function to reactivate the PCI Express Link hierarchy s main power rails and reference clocks. WAKE is required on any ExpressModule or system board that supports wakeup functionality compliant with this specification. Internal Storage interface (optional): Four ports of SAS/SATA plus sideband interface. SMBCLK (required): the SMBus interface clock signal. It is an open-drain signal. SMBDAT (required): the SMBus interface address/data signal. It is an open-drain signal. SMBAlert (optional Module only): the SMBus alert is an open-drain signal. Hot-plug signals (required): ATNSW, ATNLED, PWRLED, PWREN#, MPWRGD#, PWRFLT# PRSNT#, EMIL, and EMILS. See Chapter 3 for details Reference Clock Low Voltage Swing, Differential Clocks To reduce jitter and allow for future silicon fabrication process changes, low voltage swing, differential clocks are being used, as illustrated in Figure 2-1. The nominal single-ended swing for each clock is 0 to 0.7 V and a nominal frequency of 100 MHz ±300 PPM. The clock is defined in Chapter 4 of the PCI Express Base Specification, Revision

16 REFCLK REFCLK+ T period A-0444 Figure 2-1: Differential REFCLK Waveform The reference clock pair is routed point-to-point to each connector from the system board according to best-known clock routing rules. The reference clock distribution to all devices must be matched to within 15 inches on the system board. The phase delay between the transmitter and receiver clock is assumed to be less than 10 ns. The combination of the maximum reference clock mismatch and the maximum channel length will contribute approximately 7-8 ns and the remaining time is allocated to the difference in the insertion delays of the Tx and Rx devices. The routing of each signal in any given clock pair between the clock source and the connector must be well matched in length (< inch) and appropriately spaced away from other non-clock signals to avoid excessive crosstalk. The PCI Express silicon in an ExpressModule is not required to use the reference clock on the connector but is required to maintain the 600-ppm data rate matching specified in Section of the PCI Express Base Specification, Revision 1.1. Any terminations required by the clock are to be on the system board. An example termination topology for a current-mode clock generator is shown in Figure 2-2. Electromagnetic Interference (EMI) emissions will be reduced if clocks to open sockets are shut down at the clock source. The method for detecting the presence of a module in a slot and controlling the clock gating is platform specific. Rs REFCLK+ Add-In Card Reference Clock REFCLK- PCI Express Rs Z C-DC Z C-DC A-0445 Figure 2-2: Example Reference Clock Source Termination Termination on the add-in card is allowed, but is not covered by the specifications in Section 2.7. While the same measurement techniques can be used as specified in that section, receiver termination will reduce the nominal swing and rise and fall times by half. The low input swing and low slew rates need to be validated against the clock receiver requirements as they can cause excessive jitter in some clock input buffer designs. 16

17 The reference clock timings are based on nominal 100 Ω, differential pair routing with approximately 5-mil trace widths. This timing budget allows for a maximum add-in card trace length of 4.0 inches. No specific trace geometry, however, is explicitly defined in this specification Spread Spectrum Clocking (SSC) The reference clocks may support spread spectrum clocking. Any given system design may or may not use this feature due to platform-level timing issues. The minimum clock period cannot be violated. The preferred method is to adjust the spread technique to not allow for modulation above the nominal frequency. This technique is often called down-spreading. The requirements for spread spectrum modulation rate and magnitude are given in the PCI Express Base Specification, Revision REFCLK Specifications All specifications in Table 2-1 are to be measured using a test configuration as described in Note 11 with a circuit as shown in Figure

18 Table 2-1: REFCLCK DC Specification and AC Timing Requirements Symbol Parameter 100 MHz Input Unit Note Min Max Rise Edge Rate Rising Edge Rate V/ns 2, 3 Fall Edge Rate Fall Edge Rate V/ns 2, 3 V IH Differential Input High Voltage +150 mv 2 V IL Differential Input Low Voltage -150 mv 2 V CROSS Absolute crossing point voltage mv 1, 4, 5 V CROSS DELTA Variation of Vcross over all rising clock edges +140 mv 1, 4, 9 V RB Ring-Back Voltage Margin mv 2, 12 T STABLE Time before VRB is allowed 500 ps 2, 12 T PERIOD AVG Average Clock Period Accuracy Ppm 2, 10, 13 T PERIOD ABS Absolute Period (including jitter and Spread Spectrum) ns 2, 6 T CCJITTER Cycle to Cycle jitter 150 ps 2 V MAX Absolute Max input voltage V 1, 7 V MIN Absolute Min input voltage -0.3 V 1, 8 Duty Cycle Duty Cycle % 2 Rise-Fall Matching Rising edge rate (REFCLK+) to falling edge rate (REFCLK-) matching 20 % 1, 14 ZC-DC Clock source DC impedance Ω 1, 11 Notes: 1. Measurement taken from single ended waveform. 2. Measurement taken from differential waveform. 3. Measured from -150 mv to +150 mv on the differential waveform (derived from REFCLK+ minus REFCLK-). The signal must be monotonic through the measurement region for rise and fall time. The 300 mv measurement window is centered on the differential zero crossing. See Figure Measured at crossing point where the instantaneous voltage value of the rising edge of REFCLK+ equals the falling edge of REFCLK-. See Figure Refers to the total variation from the lowest crossing point to the highest, regardless of which edge is crossing. Refers to all crossing points for this measurement. See Figure Defined as the absolute minimum or maximum instantaneous period. This includes cycle to cycle jitter, relative PPM tolerance, and spread spectrum modulation. See Figure Defined as the maximum instantaneous voltage including overshoot. See Figure Defined as the minimum instantaneous voltage including undershoot. See Figure Defined as the total variation of all crossing voltages of Rising REFCLK+ and Falling REFCLK-. This is the maximum allowed variance in VCROSS for any particular system. See Figure

19 10. Refer to Section of the PCI Express Base Specification, Revision 1.1 for information regarding PPM considerations. 11. System board compliance measured at the connector using the circuit of Figure 2-9. REFCLK+ and REFCLK- are to be measured at the load capacitors CL. Single ended probes must be used for measurements requiring single ended measurements. Either single ended probes with math or differential probe can be used for differential measurements. Test load CL = 2 pf. 12. T STABLE is the time the differential clock must maintain a minimum ±150 mv differential voltage after rising/falling edges before it is allowed to droop back into the VRB ±100 mv differential range. See Figure PPM refers to parts per million and is a DC absolute period accuracy specification. 1 PPM is 1/1,000,000th of MHz exactly or 100 Hz. For 300 PPM, then, we have an error budget of 100 Hz/PPM * 300 PPM = 30 khz. The period is to be measured with a frequency counter with measurement window set to 100 ms or greater. The ±300 PPM applies to systems that do not employ Spread Spectrum or that use common clock source. For systems employing Spread Spectrum, there is an additional 2500 PPM nominal shift in maximum period resulting from the 0.5% down spread resulting in a maximum average period specification of PPM. 14. Matching applies to rising edge rate for REFCLK+ and falling edge rate for REFCLK-. It is measured using a ±75 mv window centered on the median cross point where REFCLK+ rising meets REFCLKfalling. The median cross point is used to calculate the voltage thresholds the oscilloscope is to use for the edge rate calculations. The Rise Edge Rate of REFCLK+ should be compared to the Fall Edge Rate of REFCLK-. The maximum allowed difference should not exceed 20% of the slowest edge rate. See Figure 2-5. V MAX = 1.15 V REFCLK V CROSS MAX = 550 mv V CROSS MIN = 250 mv REFCLK+ V MIN = V A-0446 Figure 2-3: Single-Ended Measurement Points for Absolute Cross Point and Swing REFCLK V CROSS DELTA = 140 mv REFCLK+ A-0447 Figure 2-4: Single-Ended Measurement Points for Delta Cross Point 19

20 REFCLK- REFCLK- T FALL T RISE V CROSS MEDIAN +75 mv V CROSS MEDIAN V CROSS MEDIAN V CROSS MEDIAN -75 mv REFCLK+ REFCLK+ A-0448 Figure 2-5: Single-Ended Measurement Points for Rise and Fall Time Matching Positive Duty Cycle (Differential) Clock Period (Differential) Negative Duty Cycle (Differential) 0.0 V REFCLK+ minus Figure 2-6: Differential Measurement Points for Duty Cycle and Period A-0449 Rise Edge Rate Fall Edge Rate V IH = +150 mv 0.0 V V IL = -150 mv REFCLK+ minus A-0450 Figure 2-7: Differential Measurement Points for Rise and Fall Time 20

21 T STABLE T STABLE V RB V IH = +150 mv V RB = +100 mv 0.0 V V RB = -100 mv V IL = -150 mv REFCLK+ minus V RB A-0451 Figure 2-8: Differential Measurement Points for Ringback Reference Clock Rs PCI Express Connector PCI Express Add-In Module Test REFCLK+ REFCLK- Rs Z C-DC Motherboard Trace Z C-DC CL = 2 pf CL = 2 pf A-0452 Figure 2-9: Reference Clock System Measurement Point and Loading REFCLK Phase Jitter Specification The phase jitter of the reference clock is to be measured using the following clock recovery function where: s* t _ delay [ H ( s) * e H ( s) ] H ( ) H ( s) = s 2 2sζω1 + ω1 H 1( s) = 2 2 s + 2sζω1 + ω1, 2 2sζω 2 + ω2 H 2 ( s) = 2 2 s + 2sζω 2 + ω2, s H 3 ( s) =, s + ω 3 21

22 ζ = * π * ω1 = ζ * π * ω 2 = ζ + 6 ω 3 = 2 * π * Rad / s 9 t _ delay = s 2 ( 1 + 2ζ ) 2 2 ( 1 + 2ζ ) 2 Rad / s + 1 Rad / s + 1 The maximum allowed magnitude of the peak-peak reference clock jitter is given in Table 2-2. For information about the maximum peak-peak phase jitter value, see PCI Express Jitter Modeling. Multiple methods can be used to measure the maximum allowed peak-peak phase jitter value. Real time sampling scopes must use a sampling rate of 20 giga-samples per second or better and take enough data to guarantee the proper bit error rate (BER). Reference clock measurements for cards should be taken with a differential, high-impedance probe using the circuit of Figure 2-9 at the load capacitors CL. Measurements for devices on the same board should be made using a differential, high-impedance probe as close to the REFCLK+ and REFCLK- input pins as possible. Table 2-2: Maximum Allowed Phase Jitter When Applied to Fixed Filter Characteristic BER Maximum Peak Peak Phase Jitter Value (ps) MRST# Signal The MRST# signal is used to indicate when the power supply is within its specified voltage tolerance and is stable. It also initializes a component s state machines and other logic once power supplies stabilize. On power up, the de-assertion of MRST# is delayed (T PVPERL ) from the power rails achieving specified operating limits. Also, within this time, the reference clocks (REFCLK+, REFCLK-) also become stable, at least T MRST-CLK before MRST# is de-asserted. MRST# is asserted in advance of the power being switched off in a power-managed state like S WAKE Signal The WAKE signal is an active high signal that is driven high by a PCI Express component to reactivate the PCI Express slot s main power rails and reference clocks. Only ExpressModules that support the wake process connect to this pin. If the ExpressModule has wakeup capabilities, it must support the WAKE function. Likewise, only systems that support the wakeup function need to connect to this pin; but if they do, they must fully support the wake functionality. Such systems are not required to support Beacon as a wakeup mechanism, but are encouraged to support it. If the wakeup process is used, the +3.3Vaux supply is used for this function. The assertion and de- 22

23 assertion of WAKE are asynchronous to any system clock. (See Chapter 5 of the PCI Express Base Specification, Revision 1.1, for more details on PCI-compatible power management.) The WAKE signal directly from the ExpressModule may not be bused to other PCI Express ExpressModule connectors. Platforms that choose to wire OR each slot s wake function to a single input to the power management (PM) controller can connect the WAKE signal through an open collector or open drain inverting buffer to a common WAKE# input to the PM. Hot-Plug requires that WAKE be controlled and driven inactive during the Hot-Plug/Hot Removal events. Refer to Section 6.2 for the connector pin assignment for the WAKE signal. Auxiliary power (+3.3Vaux) must be used by the asserting and receiving ends of WAKE in order to revive the hierarchy. The system vendor must also provide a pull-down on WAKE to prevent activation when the slot is empty. Note that all potential drivers of the WAKE signal must be 3.3 V tolerant. Note: The inversion of WAKE (WAKE#) is not PME# and should not be attached to the PCI- PME# interrupt signals. WAKE causes power to be restored but must not directly cause an interrupt. Additionally, the ExpressModule must ensure that it does not pull WAKE high unless WAKE is being intentionally asserted in all cases, including when the related function is in D3 cold. Other requirements on the system board/expressmodule designer include: If WAKE is supported on one ExpressModule connector in a chassis, it must be supported on all ExpressModule connectors in that chassis. ExpressModules are permitted to generate the Beacon wakeup mechanism in addition to using the wake mechanism; although, the system is not required to provide support for Beacon. Note: If the ExpressModule uses the Beacon mechanism in addition to the wake mechanism, the Beacon may be ignored by the system. PCI Express ExpressModule designers must be aware of the special requirements that constrain WAKE and ensure that their ExpressModules do not interfere with the proper operation of WAKE. The WAKE input into the system may deassert as late as 100 ns after the WAKE output from the ExpressModule deasserts (i.e., the WAKE pin must be considered indeterminate for a number of cycles after it has been deasserted). The maximum of the pull-down resistor for WAKE on the system board shall be sufficient to hold the WAKE input inactive when a module is not present and the minimum value shall be 2.2 kω ± 5%. The module WAKE driver shall ensure that WAKE charges up to a logic high voltage level in no more than 100 ns Internal Storage Interface This is an optional interface that enables an ExpressModule storage initiator to control a modest number of internal (to server chassis) SAS and/or SATA drives. Prior to ExpressModule, there was no need to define a storage interface as part of the adapter system interface specification, since standard adapters cable to internal storage via a connector(s) on the (undefined) top edge of the adapter PCB. ExpressModule has no such free PCB edge. The only way back into the server 23

24 chassis is through the ExpressModule backplane connector, hence the need to include the storage interface as part of this specification. Once the storage signals are brought through the connector onto the ExpressModule backplane, they can be cabled to internal storage via a connector(s) on the edge of the ExpressModule backplane PCB. The intent is that the same internal storage infrastructure (connectors, cables, drive backplanes) be usable regardless of whether the signals come from the top edge of a standard formfactor adapter, or the edge of an ExpressModule backplane. Issue: The electrical path from the initiator IC on board an adapter or ExpressModule to the connector that cables to internal storage is more complex (extra connector) and potentially longer in the ExpressModule case than in the standard adapter case. Key features of the internal storage interface include: Up to four SAS/SATA channels. This enables direct support for up to four drives. Larger drive counts are supported via SAS Expanders or SATA II Port Multipliers, which typically reside on the internal drive backplane. In SAS mode, the four channels can act as independent singlewide pipes, or be aggregated into two doublewide or a single fourwide. Initial target is 3G SAS and 3G SATA II. The goal is to support 6G SAS in tandem with gen2 PCI Express. The storage interface is implemented via an optional connector extension. Dual-initiator support: A system can optionally provide two storage-enabled slots, which can be cabled to a dual-port drive backplane that supports dual-port SAS drives (or dual-port SATA drives by virtue of SATA II Port Multiplexors). Note: Any required inter-expressmodule communications path is outside the scope of this specification. (The needed path is typically provided on the drive backplane.) A storage-enabled slot can accept a nonstorage enabled ExpressModule (assuming of course something else is driving internal storage). A storage enabled ExpressModule can be plugged into nonstorage enabled slot. The storage interface supports an optional sideband interface for miscellaneous storage management functions like driving LED status indicators or reporting drive presence. Currently two different 4-wire sideband standards exist in the market for this function, referred in this specification as I2C/SMBus and Serial GPIO (SGPIO). Since only one or the other will be in use in any given system, this specification maps these two standards on the same four ExpressModule connector pins. If a storage-enabled ExpressModule is to work in either type system, it must implement both interfaces and multiplex them onto the same physical signals. Before it drives one or the other, it must first determine which type system it is plugged into and configure the multiplexer (and select appropriate management firmware) accordingly. The general philosophy of the ExpressModule is to enable mistake-proof closed chassis addition or replacement of I/O modules without having to take a server off-line. It is important to realize and account for the fact that when an ExpressModule is used to support internal storage, some of these objectives are unavoidably compromised. For example, you have to make sure the storage-enabled module gets into the storage-enabled slot. If a storage-enabled module is offered as an after-market upgrade (the system may initially ship with internal drives managed by a system board chip), then the 24

25 chassis may have to be opened to rearrange internal cables. Finally, it is unlikely a system can be built that will tolerate a hot-swap of the adapter that provides access to its boot volume. To mitigate these issues, an OEM (on select products) may prefer a non-expressmodule strategy like RAID-on-Motherboard (ROMB) to manage internal drives Storage Interface Signal Definitions The storage interface signals identified below and are listed in Table 6-2 with the connector pins locations identified in Table High Speed SAS/SATA Channels SAS/SATA_TX+[3:0], SAS/SATA_TX-[3:0]: SAS/SATA transmit pairs (eight signals total) driven by ExpressModule. SAS/SATA_RX+[3:0], SAS/SATA_RX-[3:0]: SAS/SATA receive pairs (eight signals total) sensed by ExpressModule Storage Sideband Signals I2C/SMBus or SGPIO As stated before, two orthogonal sideband interfaces I2C/SMBus and SPGIO are supported over the same physical signal pins. The STOR prefix in front of the I2C/SMBus signal names distinguishes them from the ExpressModule management interface, which also uses the I2C/SMBus standard. Note in the storage case, the module rather than system is the master. (The drive backplane is the slave.) Table 2-3: Storage Sideband Signals: I2C/SMBus Connector Pin Name Signal Name Function Module I/O STOR_SB-1 STOR_SMBCLK I2C/SMB CLOCK O STOR_SB-2 STOR_SMBDAT I2C/SMB DATA I/O STOR_SB-3 STOR_SMBINT I2C/SMB INTERRUPT I STOR_SB-4 STOR_SMBRST I2C/SMB RESET O 25

26 Table 2-4: Storage Sideband Signals: SGPIO Connector Pin Name Signal Name Function Module I/O STOR_SB-1 SCLOCK Clock O STOR_SB-2 SDATAOUT Output data O STOR_SB-3 SDATAIN Input data I STOR_SB-4 SLOAD Load data signal O 2.5. Power Management Power management signals are the signals that control module power, indicate power good, and wake the chassis. If the platform with the PCI Express interface supports the wake function, then the WAKE signal must be accepted from all PCI Express ExpressModule connectors. Table 4-1 lists the required specifications for the power supply rails available at the PCI Express connectors. The system designer is responsible for ensuring that the power delivered to the PCI Express connectors meets the specifications called out in Table Initial Power-Up (G3 to L0) As long as PWREN# is inactive, the primary power converter on the adapter is off and all PCI Express functions are powered off. When the system is powered on, the adapter s bulk power ramps up to its specified levels. The power enable (PWREN#) signal can be active during or after this system power stabilization time. In ExpressModule slots that do not support hot plug, the power enable pin can be connected to ground. This allows the module to power up at the same time as the rest of the system. In ExpressModule slots that do support hot-plug, the power enable pin is controlled by the hot-plug logic in the PCI Express port under software control and happens after the system power has stabilized in most cases. Once the ExpressModule primary power converter has been enabled (PWREN# active) and producing stable adapter primary power, the ExpressModule power good signal (MPWRGD#) goes active. The chassis combines ExpressModule present detection (PRSNT#) and module power good active (MPWRGD#) on the chassis to enable the REFCLK signals to the module slot. Once the adapter MPWRGD# signal is active for T PVPERL and the system reset input signal is inactive, the module slot reset (MRST#) signal goes inactive allowing the module to exit the reset state and the PCI Express Link to become active. The module initial power up order requires that 3.3V aux power be applied first or at the same time as the bulk 12V. 26

27 +12V PWREN# 2 MPWRGD# System Reset input MRST# 3 REFCLK PCI Express link 4 Active SMBus Active +3.3Vaux 1 1. Aux stable to SMBus driven. 2. Minimum time from +12V stable to MRST# inactive T PVPERL. 3. Minimum time from reference clock (REFCLK) valid to MRST# inactive T MRST-clk. 4. Minimum time from reference clock to PCI Express link active. A-0453 Figure 2-10: Power Up Power Management States (S0 to S3/S4 to S0) If the system wants to enter S3/S4, devices are placed into D3hot states with Links in L2 prior to any power transitions at the slot. The main power and reference clock supplied to the PCI Express slot will go inactive and stay inactive until a wakeup event. As a result of the removal of main power, devices enter the D3 cold state. During the D3 cold state, +3.3Vaux remains at 3.3 V. On the wakeup event (5 in Figure 2-11), the power manager restores the main power and reference clocks. As in the last section, MRST# de-asserts T PVPERL after the clocks and power are stable. 27

28 +3.3Vaux +12V MRST# MPWRGD# WAKE 6 PWREN# REFCLK PCI Express link Active Active SMBus Active PCI Express link goes to idle prior to MRST# going active. 2. REFCLK goes inactive after MRST# goes active. 3. PWREN# goes inactive after REFCLK goes inactive. 4. Wakeup event or WAKE goes active causing power to return then PWREN# goes active, restarts REFCLK and MRST# goes inactive as in power up sequences. 5. Minimum active time for MRST# is T MRST. 6. Maximum hold time for WAKE from MRST# active is T WAKE. A-0454 Figure 2-11: Power Management States 28

29 Power Down Figure 2-12 shows the power down sequences for an orderly power down. That is, the PCI Express bus goes idle then module reset (MRST#) goes active to the module. This is followed by the reference clock (REFCLK) going inactive and, finally, by power enable (PWREN#) going inactive which, in turn, deactivates the on-module power converter. During a surprise removal, present detection and power enable will both go inactive at the same time (short pins on the module connector) forcing the power converter and reference clock off. +3.3Vaux +12V PWREN# MPWRGD# MRST# REFCLK PCI Express link Active Inactive SMBus Active PCI Express link goes inactive prior to MRST# (Device in D3 hot ). 2. REFCLK goes inactive after MRST# goes active. 3. MRST# goes active before +12V goes unstable. 4. In the case of a surprise power down, MRST# goes active T FAIL after +12V goes unstable. A-0455 Figure 2-12: Power Down 2.6. Management The System Management Bus (SMBus) is a two-wire interface and a management alert through which various system component chips can communicate with each other and with the rest of the system. It is based on the principles of operation of I 2 C. This out-of-band management (not through the PCI Express interface) interface is used to provide information to the host chassis and for out-of-band control of some features of the adapter. SMBus provides a control bus for system and power management related tasks. A system may use SMBus to pass messages to and from devices instead of tripping individual control lines. Removing the individual control lines reduces pin count. Accepting messages ensures future extensibility. 29

30 The SMBus is the host s interface to the adapter s Vital Product Data (VPD) EEPROM. The VPD devices are required of all ExpressModules and provide manufacturer and service information (see Table 2-13 for required records). The VPD EEPROM shall be powered from the Vaux power. The VPD EEPROM can also provide additional information through optional record types; these records can enhance the serviceability of the module or provide information about special functions it supports. SMBus is described in the System Management Bus (SMBus) Specification, Version 2.0. Refer to this specification for DC characteristics and all AC timings. The system board and ExpressModule shall support SMBus topology as defined in Section Each ExpressModule slot in a chassis shall be provided with its own SMBus segment, and each slot shall have its own address space (the SMBuses of ExpressModule slots are not bused together). This allows all modules to have VPD and other components at the same bus addresses without address conflicts. Host chassis connect to each slot s management interface individually for module management; empty slots management interface will be weakly pulled down on the host side of the interface to minimize hot-plugging issues. The host choice of management multiplexer/switch needs to consider this because it could be locked to an empty slot Capacitive Load of High-power SMBus Lines The maximum capacitive load for a physical SMBus segment is 400 pf. Each ExpressModule SMBus and its host interface is a physical segment and thus can support a maximum capacitive load of 400 pf. The ExpressModule portion of the segment is limited to 200 pf per signal pin to provide system board layout flexibility. There is no limitation on the number of SMBus devices on the ExpressModule provided that meet the requirements specified within this specification. Capacitive load for each bus line includes all pin, wire, and connector capacitances. The maximum capacitive load affects the selection of the pull-up resistor or the current source in order to meet the rise time specifications of SMBus. Normally, pin capacitance is defined as the total capacitive load of one SMBus device as seen in a typical manufacturer s data sheet. The value in the DC specifications (C OUT in Table 2-24) is a recommended guideline so that two SMBus devices may, for example, be populated on an add-in card Minimum Current Sinking Requirements for SMBus Devices The ExpressModule uses the high-power DC electrical specification as defined in the System Management Bus (SMBus) Specification, Version 2.0. ExpressModule SMBus devices are required to sink a minimum current of 4 ma while maintaining the V OL (max) of 0.4 V. The requirement for 4 ma sink current determines the minimum value of the pull-up resistor R P that can be used in SMBus systems. 30

31 SMBus Back Powering Considerations Unpowered devices connected to either a low-power or high-power SMBus segment must provide, either within the device or through the interface circuitry, protection against back powering the SMBus. Unpowered devices connected to high-power segments must meet leakage specifications in Section of the System Management Bus (SMBus) Specification, Version Power-on Reset SMBus devices detect a power-on event in one of three ways: By detecting that power is being applied to the device By MRST# being asserted For self-powered or always powered devices, by detecting that the SMBus is active (clock and data lines have gone high after being low for more than 2.5 s) SMBus Termination and Power The ExpressModule SMBus is terminated on the module by resistors to the Vaux input. Any SMBus devices on the module that are not powered by Vaux power must be isolated from the SMBus during low power states. This allows the SMBus to operate in the absence of primary power providing VPD data and management access before powering the card or while in a reduced power state. Components attached to these signals need to have a 3.3V signaling tolerance SMBAlert The ExpressModule management bus alert is used to make the host aware of a need for attention from the management system. An example of this would be an optional temperature-monitoring device on a ExpressModule that is set to alert the management system when a set temperature is reached on the adapter. This is an optional output from ExpressModule but a required input for the system. If not supported by the ExpressModule, it shall be grounded. The host side of this signal is pulled low with a weak pull-down Management Bus Topology Figure 2-13 illustrates the general connections between the modules and a host system that incorporates a management microcontroller. In this example, the individual SMBAlert signals are routed to an intermediate I/O port device in order to speed the ability for the management controller to detect which given module generated the alert. The I/O port device provides both a mechanism for OR-ing the SMBAlert signals as well as status bits that the management microcontroller can use to rapidly determine which bus was the source of the alert. The approach also has the advantage of providing a way for the management controller to disable or isolate different segments in case there was a problem with one of the segments. Otherwise, if something 31

32 like a simple OR of the SMBAlert signals was done, a module that erroneously forced the SMBAlert signal into the asserted state could lock out alerts from the other modules. Note that an alternative implementation could also use GPIO pins on the management controller, external parallel latches, or other approaches to accomplish the same goals. However, this approach shown also works equally well if the device is a generic chipset SMBus controller. SMBus 1 Module 1 SMBus SMBus Multiplexer SMBus 2 Module 2 Host Management Controller (e.g. IPMI BMC) or SMBus Host Controller SMBus 3 Module 3 SMB Alert 1 SMB Alert SMBus I/O Port with Interrupt SMB Alert 2 SMB Alert 3 A-0456 Figure 2-13: Example SMBus Topology Data Integrity Data integrity for VPD devices is handled by checksums that are included in the data format. Data integrity for management controller messaging is provided using the SMBus Packet Error Code (PEC) byte (an 8-bit CRC defined in the System Management Bus (SMBus) Specification, Version 2.0). The Basic Management Status register is not protected by data integrity. However, because the values are read only, software can issue multiple reads as a means to verify that a change in data has occurred. 32

33 Basic Management Status Register The optional Basic Management Status register is a 16-bit register that is accessed using an SMBus Read Word transaction. Figure 2-14 illustrates the basic format of this transaction. Refer to the System Management Bus (SMBus) Specification, Version 2.0 for details of operation. S 0100_000x 1 A Data 0 A Data1 A=1 P Slave Addr R/W LSB MSB Where "S" = SMBus START condition, "P" = SMBus STOP condition, A = SMBus ACK bit. Shaded portions indicate data from module to host. Unshaded portions indicate data from host to module. Figure 2-14: Basic Management Status Register Access The host can detect the presence of the register by noting whether the device ACKs the device address. If the device is not present, the slave address will be NACK d (a 1 received for the first ACK bit). The Basic Management Status bits are self clearing. That is, the bits are not latched and will clear themselves when the condition clears. Table 2-5 specifies the individual bits of this status. Table 2-5: Basic Management Status Register Bit Function/Description Bit Function/Description 15 reserved 7 Fatal error 14 reserved 6 Predictive failure 13 reserved 5 Power subsystem initiated shutdown 12 reserved 4 Monitored power out-of-range 11 reserved 3 Monitored power warning 10 reserved 2 Over temperature shutdown 9 reserved 1 Critical over temperature 8 reserved 0 Over temperature warning 33

34 Satellite Management Controller If more sophisticated monitoring is desired, such as returning various voltage and temperature readings, delivering asynchronous events to a BMC, a module can optionally implement an IPMI Satellite Controller (an autonomous microcontroller that provides IPMI-based, discoverable management functions). If implemented, the controller must meet the following requirements: Has its interface at address D4h (1101_000x) on SMBus and is accessed using the PCI SMBus protocol per [IPMI]. Meets the mandatory requirements for a Satellite controller per General Mgmt. Controller Requirements in [IPMI]. Provides Device SDRs (Sensor Data Records) for the sensors it provides per [IPMI]. Remains operational on standby power (i.e., can be communicated with via IPMI commands on the SMBus). Some sensors or functions may become unavailable on standby power but must meet IPMI requirements; for example, implementing the reading/state unavailable bit for sensors that become unavailable due to power state. Is not reset with PCI resets. Only resets and initializes when standby power is first applied or by IPMI command. (There is no requirement to implement command-based resets.) Has its primary FRU data provided by a VPD (FRU) device directly accessed on the SMBus of the module. This FRU device shall follow the VPD (FRU) Information Format given in this specification. The module may also provide a primary FRU device that is accessible via the satellite controller via IPMI Read/Write FRU commands. This FRU information, if present, must be formatted per [IPMI] Remote Management Card Access A module may provide a Remote Management Card function for the system. Remote Management Cards typically operate on standby power, provide their own communications interfaces (e.g., LAN), and have on-board microcontrollers that can be used to provide system access, recovery operations, and health status under conditions where the system is inaccessible via software running on the host processor(s). If the motherboard contains an IPMI BMC (baseboard management controller) that is accessible via the SMBus connection, this connection can be used by the Remote Management Card to access motherboard sensor, event log, control (power on/off, reset, etc.), and other functions offered via IPMI. Because the BMC is required to remain available on standby power, functions such as event log and power/reset control are available even if the system is powered down. The presence of an IPMI BMC can be detected by sending an IPMI Get Device ID command to SMBus address 20h (0010_000x) using the PCI SMBus protocol specified in [IPMI]. 34

35 VPD (FRU) Information Format The VPD information is formatted as a series of variable length records following a Common Header. The Common Header is located at offset 00h in the VPD device and specifies which version of the specification defined the VPD Format and the offset to the first record, as shown in Table 2-6. Table 2-6: VPD (FRU) Common Header Byte Field 1 [7] - 1b = OEM Reserved Space present [6:4] - reserved [3:0] VPD Version - 01h for this specification 2 VPD Record Offset LSB - Offset to first VPD record, least significant byte, 0- based 3 VPD Record Offset MSB - Offset to first VPD record, most significant byte, 0- based 4 53h S The four ExpressModule bytes are to improve the robustness of discovery of an ExpressModule VPD device. 5 49h I 6 4Fh O 7 4Dh M 8 Common Header Checksum Note that typically the first record data will immediately follow the common header. However, a manufacturer can reserve a section of the VPD device for private use by setting an offset and leaving space between the end of the Common Header and the first VPD Record and indicating that this space is being reserved for OEM use by setting the OEM Reserved Space bit in the Common Header. If OEM Reserved Space is used, the first bytes of data following the Common Header shall be an OEM Reserved Space Header, as shown in Table

36 Table 2-7: OEM Reserved Space Header Field Length Field 3 Manufacturer ID. IANA Enterprise ID number for the manufacturer or organization that has defined the format and content of the space between the Common Header and first VPD Record. 1 Header Checksum (zero checksum) Utilities that are specific to the module manufacturer may change the Common Header as part of field updates of the module. Generic utilities are not allowed to alter the values of the Common Header VPD (FRU) Record Format Each VPD Record starts with a Record Header. The Record Header for each record defines the type of information contained in the record. Any data bytes associated with the record immediately follow the last field of the header. Unless otherwise specified, multi-byte numeric data is stored least-significant byte first. The Record Header also contains offset information needed to access any additional following records, if any, in a sequential list manner. The Record Header format is shown in Table 2-8. Table 2-8: VPD (FRU) Record Format Field Length Field 1 Record Type ID 1 7:7 End of list 6:4 Reserved, write as 000b 3:0 Record Format version (=1h unless otherwise specified) 1 Record Length 1 Header Checksum (zero checksum) N Record Data (according to Record Type) 1 Record Checksum (zero checksum). The record checksum covers all bytes in the Record Data field. 36

37 Record Header Fields The following provides additional information about the individual fields in the Record Header: Record Type ID This unsigned byte is used to identify the information contained in the record. Table 2-9 shows the currently defined record types. Table 2-9: VPD (FRU) Record Types ID Record Type Required* 0x00 reserved - 0x01 Board Info Record O 0x02 Product Info Record M 0x03 Product GUID O 0x04 Board GUID O 0x05 ExpressModule Module Info M 0x06 PCI Express Device Info M 0x07 ExpressModule Multiplexer O** 0x08-0xBF reserved - 0xC0-0xFF OEM Record Types O * Indicates whether record is required for the VPD device on a module ** The ExpressModule Multiplexer record is mandatory to be provided by a VPD device on the base board when the SMBuses are accessed via an SMBus host controller. O = Optional M = Mandatory End of List. This bit indicates if this record is the last record in the Multi-Record area. If this bit is zero, it indicates that one or more records follow. Record Format Version. The area version format is stored in the lower nibble of the second byte. This field is used to identify the revision level of information stored in this area. This number will start at zero for each new area. If changes need to be made to the record, e.g., fields added/removed, the version number will be increased to reflect the change. Unless otherwise specified, the record format version for all record types shall be 01h for this specification (including OEM Record Types). The latter provision is to provide for standardized data fields that precede the OEM specific data within the OEM Record. Record Length. This unsigned byte indicates the number of bytes of data in the record. This byte can also be used to find the next area in the list. If the End of List bit is zero, the length can be added to the starting offset of the current Record Data to get the offset of the next Record Header. This field allows for 0 to 255 bytes of data for each record. Header Checksum. This unsigned byte is used to give the generic header a zero checksum; i.e., the modulo 256 sum of the preceding bytes (starting with the first byte of the header) plus the checksum byte equals zero. 37

38 Record Checksum. This unsigned byte can be used to calculate a zero checksum for the Record Data following the Record Header; i.e., the modulo 256 sum of the record data bytes plus the checksum byte equals zero Board Info Record This optional record provides Serial Number, Part Number, and other information about a board or other FRU that is a sub-component of the module. Thus, the name Board Info Area is somewhat a misnomer, because the usage is not restricted to just circuit boards. This area can also be used to provide FRU information for any replaceable entities, boards, and or sub-assemblies within the module. Field Length Field Table 2-10: Board Info Data Format 1 Board Info Record Format Version 7:4 - reserved, write as 0000b 3:0 - format version number = 1h for this specification 1 Language Code (see Section ) 1 Board Number A number assigned by the module manufacturer to identify different types of sub-component within the module FFh = unspecified 3 Mfg. Date / Time Number of minutes from 0:00 hrs 1/1/96 LS byte first (little endian) 00_00_00h = unspecified 1 Board Manufacturer type/length byte P Board Manufacturer bytes 1 Board Product Name type/length byte Q Board Product Name bytes 1 Board Serial Number type/length byte* N Board Serial Number bytes* 1 Board Part Number type/length byte M Board Part Number bytes 1 FRU File ID type/length byte* R FRU File ID bytes* The FRU File version field is a predefined field provided as a manufacturing aid for verifying the file that was used during manufacture or field update to load the FRU information. The content is manufacturer-specific. This field is also provided in the Product Info area. Either or both fields may be null. 38

39 Field Length Field 1 Board EC Number type/length byte. Engineering change level (revision) of this board. X Board EC Number bytes xx Additional custom Mfg. Info fields. Defined by manufacturer. Each field must be preceded by a type/length byte. 1 C1h (type/length byte encoded to indicate no more info fields) Y 00h - any remaining unused space * These areas are always encoded as if the Language Code were English Product Info Record This record data is present if the FRU itself is obtainable as a separate product. This is typically seen when the FRU is an add-in card, sub-assembly, or a power supply from a separate vendor. This record will be present in all PCI Express modules. Field Length Field Table 2-11: Product Info Data Format 1 Product Info Record Format Version 7:4 - reserved, write as 0000b 3:0 - format version number = 1h for this specification 1 Language Code (See Section ) 1 Manufacturer Name type/length byte N Manufacturer Name bytes 1 Product Name type/length byte M Product Name bytes 1 Product Part/Model Number type/length byte O Product Part/Model Number bytes 1 Product Version type/length byte R Product Version bytes 1 Product Serial Number type/length byte* P Product Serial Number bytes* 1 Asset Tag type/length byte Q Asset Tag 1 FRU File ID type/length byte 39

40 Field Length Field R FRU File ID bytes The FRU File version field is a predefined field provided as a manufacturing aid for verifying the file that was used during manufacture or field update to load the FRU information. The content is manufacturer-specific. This field is also provided in the Board Info area. Either or both fields may be null. xx Custom product info area fields, if any (must be preceded with type/length byte) 1 C1h - (type/length byte encoded to indicate no more info fields) Y 00h - any remaining unused space * These fields are always encoded as if the Language Code were English; i.e., if the type/length code bits 7:6 = 11b the serial number will always be interpreted as ASCII+Latin 1, not UNICODE Product GUID Record This record provides a readable copy of the product (module) GUID. Field Length Table 2-12: Product GUID Data Format Field 1 Product Area Format Version 7:4 - reserved, write as 0000b 3:0 - format version number = 1h for this specification 16 GUID - least significant byte first 40

41 ExpressModule Module Info Record This record data provides information specific to ExpressModules. Field Length Table 2-13: ExpressModule Module Info Data Format Field 1 Product Info Record Format Version 7:4 - reserved, write as 0000b 3:0 - format version number = 1h for this specification 1 [7:5] - reserved [4] - 1b = management controller present [3] - 1b = basic status register present [2:0] - Module Size: 000b = Singlewide module 001b = Doublewide module all others = reserved 1 7:3 - reserved 2:0 - Link Width: 000b = x1 001b = x2 010b = x4 011b = x8 100b = x16 all others = reserved 2 Basic Status Support [15:0] - 1b = corresponding bit in Basic Status register is supported. 1 IPMI Satellite Controller Slave Address 00h = unspecified 1 Device Max Power Consumption in watts (1-based) 2 Module Peak 12 V Power consumption in ma 2 Module Typical (average) 12 V Power consumption in ma 41

42 PCI Express Device Info Record This record data provides a readable copy of the PCI Express device s class and function information. For multi-function devices, the record provides information about which PCI Device Functions are potentially available on a module. It does not, however, provide information on whether a given function is presently enabled for use or not. The record contains a Device Info Sub-record for each possible, separately accessible, function in the device. Field Length Table 2-14: PCI Express Module Info Data Format Field 1 PCI Express Device Info Record Format Version 7:4 - reserved, write as 0000b 3:0 - format version number = 1h for this specification 1 Number of device info sub-records that follow (1-based) 9 Device Info Sub-record 1 7:5 - reserved, write as 000b 4-0b = Real Function Number. The following function number corresponds to the PCI Function number used to select the function on the device. 1b = Pseudo Function Number. In some cases, the function number assignments may vary according to how the device is configured, and the VPD data may not be able to be kept in synch with the real function number. In this case, the functions in the device info subrecord can be given pseudo function numbers for the purpose of indicating a particular instance of a function. 3:0 - Function Number 2 Vendor ID*. LS byte first. 2 Device ID*. LS byte first. 1 Base Class Code* 1 Sub-class Code* 1 Interface Code* 1 Revision ID* * Refer to the PCI Express Base Specification, Revision 1.1 for the definition. 42

43 ExpressModule Multiplexer Record This record provides a signature that can be used to discover and confirm the presence of an ExpressModule-formatted baseboard VPD device on the motherboard. Table 2-15: ExpressModule Multiplexer Record Data Format Field Length Field 1 ExpressModule Multiplexer Record Format Version 7:4 - reserved, write as 0000b 3:0 - format version number = 1h for this specification 1 Multiplexer type: 00h = reserved 01h = Philips Semiconductors 9541 or compatible 02h = Philips Semiconductors 9542 or compatible 03h = Philips Semiconductors 9543 or compatible 04h = Philips Semiconductors 9544 or compatible 05h = Philips Semiconductors 9545 or compatible 1 Multiplexer level. 7:2 - reserved 1:0-00b = level 0. Connects directly to SMBus host controller. 01b = level 1. Multiplexer is directly connected behind a level 0 multiplexer. 10b = level 2. Multiplexer is directly connected behind a level 1 multiplexer. 11b = reserved 1 Multiplexer slave address. 7-bits, left justified. Least significant bit always is 0b. For example, a slave address of 1010_000x results in a value of A0h in this byte. 1 Slave address of multiplexer of upstream multiplexer (closer to SMBus host) that this multiplexer connects to. The address is 00h if this multiplexer is a level 0 multiplexer (connects directly to host). 43

44 Field Length Field 1 Starting bus/connector ID. The ID is 0-based. This is the lowest ID for the ExpressModule SMBus segments connected to the multiplexer. This does not include the segment to the host controller. 1 Ending bus/connector ID. 0-based. This is the highest ID for the ExpressModule SMBus segments connected to the multiplexer. This does not include the segment to the host controller. Segments must be connected to multiplexers sequentially and with no gaps in the sequence. In addition, the lowest connector ID must be connected to the lowest numbered (or lettered) bus identified on the multiplexer OEM Records OEM Records have one predefined data field: a 3-byte Manufacturer ID field, as specified in the Get Device ID command in the IPMI v1.0 specification. This field occupies the first 3 bytes of the data portion of the record. Subsequent fields within the record are defined by the given OEM. Note: OEM Records are also followed by a Record Checksum Byte, per the VPD (FRU) Record Format. Table 2-16: OEM Record Data Format Offset Length Description 0 3 Manufacturer ID. LS Byte first. 3:N N OEM data, as specified by the party identified by the Manufacturer ID field 44

45 Language Codes The following table should be used when interpreting the Language Code fields. The index number (1-136) is stored in the Language Code field. Any language code other than English indicates that the string data is encoded as Unicode when bits 7:6 of the Type/Length code = 11b. Table 2-17: Language Codes 1. aa Afar 51. it Italian 101. si Singhalese 2. ab Abkhazian 52. iw Hebrew 102. sk Slovak 3. af Afrikaans 53. ja Japanese 103. sl Slovenian 4. am Amharic 54. ji Yiddish 104. sm Samoan 5. ar Arabic 55. jw Javanese 105. sn Shona 6. as Assamese 56. ka Georgian 106. so Somali 7. ay Aymara 57. kk Kazakh 107. sq Albanian 8. az Azerbaijani 58. kl Greenlandic 108. sr Serbian 9. ba Bashkir 59. km Cambodian 109. ss Siswati 10. be Byelorussian 60. kn Kannada 110. st Sesotho 11. bg Bulgarian 61. ko Korean 111. su Sudanese 12. bh Bihari 62. ks Kashmiri 112. sv Swedish 13. bi Bislama 63. ku Kurdish 113. sw Swahili 14. bn Bengali; Bangla 64. ky Kirghiz 114. ta Tamil 15. bo Tibetan 65. la Latin 115. te Tegulu 16. br Breton 66. ln Lingala 116. tg Tajik 17. ca Catalan 67. lo Laothian 117. th Thai 18. co Corsican 68. lt Lithuanian 118. ti Tigrinya 19. cs Czech 69. lv Latvian, Lettish 119. tk Turkmen 20. cy Welsh 70. mg Malagasy 120. tl Tagalog 21. da danish 71. mi Maori 121. tn Setswana 22. de german 72. mk Macedonian 122. to Tonga 23. dz Bhutani 73. ml Malayalam 123. tr Turkish 24. el Greek 74. mn Mongolian 124. ts Tsonga 25. en English 75. mo Moldavian 125. tt Tatar 26. eo Esperanto 76. mr Marathi 126. tw Twi 27. es Spanish 77. ms Malay 127. uk Ukrainian 28. et Estonian 78. mt Maltese 128. ur Urdu 29. eu Basque 79. my Burmese 129. uz Uzbek 30. fa Persian 80. na Nauru 130. vi Vietnamese 31. fi Finnish 81. ne Nepali 131. vo Volapuk 32. fj Fiji 82. nl Dutch 132. wo Wolof 33. fo Faeroese 83. no Norwegian 133. xh Xhosa 34. fr French 84. oc Occitan 134. yo Yoruba 35. fy Frisian 85. om (Afan) Oromo 135. zh Chinese 36. ga Irish 86. or Oriya 136. zu Zulu 37. gd Scots Gaelic 87. pa Punjabi 38. gl Galician 88. pl Polish 39. gn Guarani 89. ps Pashto, Pushto 40. gu Gujarati 90. pt Portuguese 41. ha Hausa 91. qu Quechua 42. hi Hindi 92. rm Rhaeto-Romance 43. hr Croatian 93. rn Kirundi 44. hu Hungarian 94. ro Romanian 45. hy Armenian 95. ru Russian 46. ia Interlingua 96. rw Kinyarwanda 47. ie Interlingue 97. sa Sanskrit 48. ik Inupiak 98. sd Sindhi 49. in Indonesian 99. sg Sangro 50. is Icelandic 100. sh Serbo-Croatian 45

46 Type/Length Byte Format The following presents the specification of the type/length byte. Type/Length encoded field data is always stored most-significant digit/character first. 7:6 - type code 00 - binary or unspecified 01 - BCD plus (see below) 10-6-bit ASCII, packed (overrides Language Codes) 11 - Interpretation depends on Language Codes. 11b indicates 8-bit ASCII + Latin 1 if the Language Code is English for the area or record containing the field or 2-byte UNICODE (least significant byte first) if the Language Code is not English. At least two bytes of data must be present when this type is used. Therefore, the length (number of data bytes) will always be >1 if data is present; 0 if data is not present. 5:0 - number of data bytes indicates that the field is empty. When the type code is 11b, a length of indicates end of fields; i.e., Type/Length = C1h indicates end of fields. ASCII+LATIN1 is derived from the first 256 characters of Unicode 2.0. The first 256 codes of Unicode follow ISO 646 (ASCII) and ISO 8859/1 (Latin 1). The Unicode C0 Controls and Basic Latin set defines the first bit characters (00h-7Fh), and the C1 Controls and Latin-1 Supplement defines the second 128 (80h-FFh). 6-bit ASCII is the 64 characters starting from character 20h (space) from the ASCII+LATIN1 set. So 6-bit ASCII value b maps to 20h (space), b maps to 21h (!), etc. Packed 6-bit ASCII takes the 6-bit characters and packs them four characters to every 3 bytes, with the first character in the least significant 6 bits of the first byte. A table of 6-bit ASCII codes and an example of packed 6-bit ASCII characters are included later in this section. 46

47 BCD Plus Definition 0h - 9h = digits 0 through 9 Ah = space Bh = dash - Ch = period. Dh = reserved Eh = reserved Fh = reserved bit ASCII Definition 0 sp P 1! A 31 Q 2 " B 32 R 3 # C 33 S 4 $ D 34 T 5 % E 35 U 6 & F 36 V 7 ' G 37 W 8 ( H 38 X 9 ) I 39 Y A * 1A : 2A J 3A Z B + 1B ; 2B K 3B [ C, 1C < 2C L 3C \ D - 1D = 2D M 3D ] E. 1E > 2E N 3E ^ F / 1F? 2F O 3F _ bit ASCII Packing Example "IPMI" encoded in 6-bit ASCII is: I = 29h (101001b) P = 30h (110000b) M = 2Dh (101101b) I = 29h (101001b) Which gets packed into bytes as follows: bit hex byte h byte DCh byte A6h 47

48 Using an SMBus Multiplexer with a Host Controller If the modules are connected to the host via an SMBus host controller rather than a BMC or other management controller, the motherboard must provide an ExpressModule baseboard VPD Device located at address A6h (1010_011x) on the non-multiplexed SMBus segment straight out of the host controller. This device must be a validly formatted ExpressModule VPD (FRU) Device containing an ExpressModule Multiplexer Record. It can optionally contain other VPD records for the baseboard. Software can scan the addresses A6h to look for the VPD device and confirm it is an ExpressModule VPD Device by verifying the common header and checksums and by locating and verifying the presence of one or more ExpressModule Multiplexer records. If present, this record tells the software what type of multiplexer device is being used, the corresponding slave address, and what ExpressModule module buses are connected to the multiplexer. More than one multiplexer device can be used; in which case, there will be more than one ExpressModule Multiplexer record. Note that multiplexers can be cascaded up to two levels deep behind a multiplexer that is directly connected to SMBus segment to the host controller Implementing ExpressModule Management in an IPMI Environment The use of the SMBus for management access facilitates the use of IPMI Management Controllers. This can be accomplished by providing a mechanism whereby the SMBus can be used for sending and receiving IPMI management messages to the modules. IPMI presently defines communication protocols for an SMBus-based management bus used for PCI, and an I 2 C-based bus called the IPMB. These protocols define how to use those bussed for sending and receiving IPMI management messages between management microcontrollers on the bus. There are significant differences between those busses and the ExpressModule management bus. The PCI SMBus and the IPMB are implemented as single-segment busses where all management controllers share the same bus segment. This enables communication between any two devices as peers using the multi-master arbitration capabilities of SMBus and I 2 C. The protocols use a request/response approach where one party arbitrates for and masters the bus, and then transmits a request to another party on the bus. The receiver of the message later arbitrates for and masters the bus and sends a response back to the originator of the request. This request/response protocol allows peer-to-peer communication and also provides a way to asynchronously generate traffic to the host management controller (system board management controller, or BMC, in IPMI terminology) or any other controller on the bus. Thus, the PCI SMBus and the IPMB do not utilize an SMBAlert lines. The ExpressModule Management Bus, however, is implemented as a segment per connector, where there is electrical isolation between the segments. This eliminates issues with address assignment and contention and the complexities of multi-master arbitration. But because the segments are isolated, direct peer-to-peer communication is not possible. Furthermore, in order to facilitate using multiplexers that are controlled by the BMC, the devices on the modules are always accessed as slaves. 48

49 Therefore, a modified approach is required to enable an equivalent level of management communications Slave Attention-based Protocol Instead of using a request/response protocol where the BMC masters the bus to send a request to the module, and the module in turn masters the bus to send a response back to the BMC, the request/response protocol is modified so that the BMC can pull messages from management controllers on modules by accessing them as slaves. It is straightforward for the BMC to select the connection to a given bus segment and transmit a message to a management controller on any module. In order for a management controller on the module to generate a message for the BMC however, it first places its message in a buffer and then raises an SMBAlert that signals the BMC to retrieve it. The BMC then retrieves the message from the module by reading the module s management controller as a slave. To facilitate this messaging, the slave must support a new Retrieve Message command that enables the BMC to retrieve message data from the slave. Standard IPMI messages can then be received from modules using this slave transaction mechanism Peer-to-Peer Proxy Messaging The PCI SMBus and IPMB also have the characteristic that different management controllers can directly talk to each other as peers. In order to allow the ExpressModule management bus segments to be isolated by simple multiplexers, the BMC must provide a forwarding service where the BMC routes requests and responses between peers. The BMC therefore acts as a proxy or router for messaging between management controllers on ExpressModules. In order to support this, there must be additional addressing information that enables the BMC to forward messages to the appropriate management bus segment. The management controller slave addresses are not sufficient for this purpose, because different management controllers on different segments can have the same slave address. Thus, the messaging format must include a bus ID that can be used for routing purposes Sending Data to a Management Controller on a Module Writing to a management controller on a module is accomplished using protocols that are based on top of the SMBus Write-Block protocols. The SMBus Write-Block protocol is limited to transferring 32 bytes of data payload plus a CMD (command) byte. This specification defines two transfer protocols that use the SMBus Write-Block protocol to deliver IPMI message data to a management controller on a module. The first protocol is called the Single-part Write protocol. This protocol is used when the message can fit within the 32-byte data payload of the SMBus Write-Block. The second protocol is called the Multi-part Write protocol and is used when more than 32 bytes are required to transfer the IPMI message. Both IPMI Requests and Response messages can be carried over these protocols. 49

50 Single-part Write Protocol The Single-part Write requires a single SMBus Write-Block transaction. The following table shows the format of this transaction. The values in parentheses indicate the number of bits for the particular field when the given field is not 8 bits. Only the address and data portions of the SMBus transactions are shown. SMBus START and STOP conditions and ACK/NACK bits are left out for simplification. The length field provides the count of data bytes of IPMI message content, up to 32 bytes. [PEC] indicates the presence of SMBus PEC (packet error code) byte. Note that this byte is not included in the byte count provided in the length field. Slave Address (7) R/W=0 (1) SMBus CMD = 02h Length Message Class Figure 2-15: BMC Single-part Write Message Data (1 to 31 bytes) PEC Multi-part Write Protocol A multi-part write is used when more than 32-bytes of message data need to be written to the BMC. This requires two or more SMBus Write-Block transactions. Three possible types of Write-Block transactions can be used in a multi-part write: A Start transaction, a Middle transaction, and an End transaction. A given message transfer consists of a Start transaction followed by zero or more Middle transactions, and then an End transaction. The first part of the message is written using the Start transaction. Since Multi-part writes are for the purpose of transferring message data that exceeds 32 bytes, the Start transaction is required to always move 32 bytes of data and the value of the length byte for the Start transaction is always 20h. The combination of a Start transaction followed by an End transaction can transfer up to 63 bytes of IPMI message. The Middle transaction is available when there is a need to transfer an IPMI message of greater than 63 bytes. As of this writing, there are no standard IPMI messages to the BMC that are longer than 63 bytes. Therefore, the Middle transaction is defined solely as needed by any OEM/group network functions (network function codes 2Ch:3Fh) in the particular BMC implementation. It is required that all multi-part write transfers end with an End transaction. Middle transactions must always move 31 bytes of data, plus a 1-byte block number, therefore the value of the length byte for Middle transactions is always 20h. The End transaction is used for the last portion of message data that is written to the BMC. It indicates to the BMC that the message data transfer has completed and the BMC can process the message. The number of message data bytes in the End transaction can range from 1 to 31 bytes. The following figures show the format of the SMBus transactions for multi-part writes. The block number is a value that starts from 0 and is incremented for each Middle transaction;. i.e., the first Middle transaction has a block number of 1, the second 2, and so on. 50

51 The last block number is either 0 (if no middle transactions are used) or matches the block number that was used in the last middle transaction. Slave Address (7) R/W=0 (1) SMBus CMD = 06h Length =20h message class Message Data (31 bytes) PEC Figure 2-16: Multi-part Write Start Slave Address (7) R/W=0 (1) SMBus CMD = 07h Length =20h block number Figure 2-17: Multi-part Write Middle Message Data (31 bytes) PEC Slave Address (7) R/W=0 (1) SMBus CMD = 08h Length= 01h to 20h last block number Message Data (1 to 31bytes) PEC Figure 2-18: Multi-part Write End Error Conditions for Multi-part Writes It is possible that out-of-order operations may occur in the course of restarting systems or loading and unloading software. For example, the controller could have just received a Middle transaction when a system restart causes the next operation to be a Start transaction, or data could get corrupted or hot-plug could cause a transaction to be lost. The management controller shall discard any multi-part write data it has received if a Start transaction is received prior to receiving a complete End transaction. If the management controller receives an incorrect length (not = 20h) in a Start or Middle transaction, it shall discard any message data that has been received for the Multi-part Write until the next Start transaction is received. If the management controller detects missing block numbers, it shall discard any message data that has been received until the next Start transaction is received Receiving Data From a Management Controller on a Module Data is read from a management controller on a module using protocols built on top of the SMBus Read-Block protocol. Like the SMBus Write-Block protocol, the Read-Block is limited to 32 data bytes of payload. The Read-Block protocol is driven from the host (typically the BMC) accessing the management controller on the module as a slave. An SMBAlert from the module indicates when the management controller on the module has data to be retrieved. Similar to the write operations, there are two types of protocol for reads: a Single-part Read protocol that can get up to 31 bytes of data from a management controller on a module and a Multi-part Read protocol for retrieving more than 31 bytes. 51

52 Single-part Read Protocol The following figure illustrates the format of a SMBus Read Block protocol for a Single-part Read Protocol. The shading indicates the direction of data on the bus. Unshaded fields represent data written from the master (typically a BMC) to the slave (management controller on the module). Shaded fields represent data being read from the slave by the master. Tag is a fixed value field that is used to differentiate between a Single-part Read and a Multi-part Read Start transaction. The Tag is included in the length count; e.g., if there is one byte of message data, the value of the Length field would be 2. Slave Address (7) R/W = 0 (1) SMBus CMD = 03h Slave Address (7) R/W=1 (1) Length (1 to 32) Tag = 0x00 message class Msg. Data (1 to 31 bytes) PEC Figure 2-19: Single-part Read Multi-part Read Protocol The following table illustrates the format of a SMBus Read Block transaction for the Multi-part Read protocol. There are four different transactions that can be used: Multi-part Read Start, Multipart Read Middle, Multi-part Read Retry, and Multi-part Read End. The Read Start and Read End transactions are used together when more than 30 bytes of message data must be read from the management controller. The Read Start transfers the first 30 bytes of message data and the Read End transfers the next 31 bytes. If more than 61 bytes must be read, one or more Read Middle transactions are also used. The Read Middle includes a 1-byte Block Number field as the first byte of SMBus data in the Read Block. Thus, each Read Middle transaction can carry up to 31 bytes of message data. The Multi-part Read Retry is used to retry Middle blocks of data in a multi-part read transaction that uses Read Middle transactions. This is described in more detail below. The following figures illustrate the SMBus protocol formats for the multi-part read transactions. All multi-part read transactions follow the SMBus Read Block protocol with the exception of the Multipart Read Retry transaction which uses the Write Block protocol. Slave Address (7) R/W = 0 (1) Tag = 0x01 SMBus CMD = 03h message Class Slave Address (7) Message Data (30 bytes) PEC Figure 2-20: Multi-part Read Start R/W = 1 (1) Length =20h 52

53 Slave Address (7) R/W = 0 (1) SMBus CMD = 09h Slave Address (7) Message Data (31 bytes) PEC R/W = 1 (1) Length =20h Block number Figure 2-21: Multi-part Read Middle Where: Tag is a fixed value field that is used to differentiate between a Single-part Read and a Multipart Read Start transaction. Block number is a number starting with 1 for the first block of message data returned using the Middle transaction and then incremented by 1 for each following Middle Transaction. Block number values 00h and FFh are reserved for identifying the Multi-part Read End transaction. The multi-part read End transaction is a Read-Block transaction that concludes a multi-part read operation. Slave Address (7) R/W = 0 (1) SMBus CMD = 0Ah Slave Address (7) Message Data (1 to 31 bytes) [PEC] R/W=1 (1) Length (1 to 32) [Block Number] Figure 2-22: BMC Multi-part Read End Where: The Block Number field contains the block number from the last middle transaction or 00h if no middle transactions were used. The Multi-part Read Retry transaction is an SMBus Write-Block transaction that tells the management controller to return the given middle Block Number on the next multi-part read Middle transaction. This transaction is only required to be supported when multi-part read Middle transactions are implemented by the controller. Slave Address (7) R/W = 0 (1) SMBus CMD = 0Bh IPMI Data (1 to 32 bytes) [PEC] Length =1 Figure 2-23: Multi-part Read Retry Block number 53

54 SMBAlert Control and Status The following transactions retrieve status information for the messaging interface of a management controller on a module Get Message Interface Status This operation uses a single SMBus Read Block transaction. Slave Address (7) R/W = 0 (1) SMBus CMD = 04h Interface Status Slave Address (7) PEC Figure 2-24: Get Message Interface Status R/W = 1 (1) Length =01h Where: Field Length Table 2-18: Interface Status Field Definition Field 1 Interface Status 7-1b = OEM non-message SMBAlerts enabled. Indicates that SMBAlert is enabled to be generated for sources other than messages that can be retrieved with the Single- and Multi-part read protocols. Note that such sources may require explicit OEM commands or operations to clear the SMBAlert. 6-1b = IPMI non-message SMBAlerts enabled. Indicates that SMBAlert is enabled to be generated for IPMI non-communication interrupts. 5-1b = Message SMBAlerts enabled. Indicates that SMBAlert is enabled to be generated for messages that are retrieved via Single- and Multi-part read protocols. 4:3 - reserved 2-1b = OEM non-message notification asserted. (Indicates a non-message source that would assert SMBAlert is active, and would be asserting SMBAlert if enabled to do so.) 1-1b = IPMI non-message notification asserted. This is a source that reflects IPMI non-communication interrupt status that is read and cleared using the IPMI Get and Set Interrupt Flags commands, respectively. 0-1b = Message available for retrieval. A message is available for retrieval via the Single- or Multi-Part Read commands. 54

55 Set Message Interface Control This operation uses a single SMBus Write Block transaction. Where: Field Length Slave Address R/W = 0 (7) (1) Interface Control SMBus CMD = 05h PEC Length =01h Figure 2-25: Set Message Interface Control Table 2-19: Interface Control Field Definition Field 1 Interface Control Note that disabling a given SMBAlert source removes it from generating SMBAlert even if the root source of the notification is still asserted. 7-1b = Enable OEM non-message SMBAlerts 6-1b = Enable IPMI non-message SMBAlerts 5-1b = Enable message SMBAlerts 4:0 - reserved Message Class Values for Management Controller Messaging Table 2-20: Message Class Values Message Class 0x00 Definition IPMI Messages Retention of Output Data A BMC that implements PEC must retain previous output message data until the occurrence of a valid Write Start transaction, at which time the output message data is cleared. This behavior is needed to better support PEC. If the BMC automatically discarded data as it was read out, there might be no way to recover the message data if the PEC indicated the data was corrupted. However, with this provision, system software can retry the read transaction that had the error. The BMC will return the retained data if the Single- and Multi-part Read Start transactions are retried prior to the next valid write Start transaction. To re-read a given block of Middle data in a multi-part read, the block number must first be written to the BMC using the Multi-part Read Retry transaction. The next Multi-part Read Middle transaction will then return that block number, and 55

56 any subsequent Multi-part Read Middle transactions will increment the block number and return following blocks SMBAlert Signal Handling for Message Transfers The management controller on the module asserts SMBAlert to signal that it has data to be retrieved or a status change to be queried. The SMBAlert signal is automatically cleared by the management controller the first time a Single-part Read Start or Multi-part Read Start is used to read a given set of data. SMBAlert will not be asserted by the management controller again until a new instance of data or status is available. A management controller is allowed to implement OEM functions that can assert SMBAlert. However, since such functions could interfere with the generic operation of the bus, special knowledge may be required to clear such sources. Therefore, such sources of SMBAlert must be disabled by default and require explicit transactions with the management module to be enabled Polling for Output Data If SMBAlert is disabled, a management controller can be polled for output by Read Start transactions until data is returned. If there is no data available, the management controller will NACK the read portion of the SMBus transfer SMBus NACKs and Error Recovery The management controller can NACK the SMBus host controller if it is not ready to accept a new transaction. For example, this could occur if write transactions follow too closely together. (See Table 2-22.) Typically, this will be exhibited by the management controller NACK ing its slave address. In some cases, the management controller may NACK a SMBus data byte that is being written to it. This can occur if software attempts to write more data bytes to the management controller than it can handle (for example, in a multi-part write), or if some internal state change caused the management controller to need to reset its internal operation. If the management controller NACKs a single part transaction, software can simply retry it. If a Middle or End transaction is NACK d, software should not retry the particular but should restart the multi-part read or write from the beginning Start transaction for the transfer PEC Handling The System Management Bus (SMBus) Specification, Version 2.0 allows a slave that receives a PEC to check the PEC and NACK the byte that carried the PEC value if the PEC is incorrect. Accomplishing this may require special hardware in order to generate the NACK without significant SMBus clock stretching. In order to avoid this requirement, a BMC implementation is allowed to always ACK the PEC. A BMC that receives an invalid PEC shall drop the data for the write transaction and any further transactions (read or write) until the next valid read or write Start transaction is received. 56

57 A management controller that implements PEC will automatically start using PEC in read transactions after it receives any SSIF single-part write or multi-part write Start transaction that includes a valid PEC. The management controller shall cease using PEC in read transactions after it receives any single-part write or multi-part write Start transaction that does not include a PEC byte. (A management controller detects PEC by noting that it has received one more byte in the SMBus Write-Block transaction than was indicated by the length byte. If this occurs, it assumes that the additional byte was the PEC byte and then checks it for validity.) Summary of SMBus Commands Values for Management Controller Messaging The following table summarizes the allocation of SMBus commands for module management controller messaging. Note that there are command values that are reserved for future definition by the specification. Table 2-21: Summary of SMBus Commands for Message Protocols Operation SMBus CMD SMBus Protocol Single-part Write 02h Write Block Multi-part Write Start - first part 06h Write Block Middle part(s) if any 07h Write Block, first data byte after length = block number End - last part 08h Write Block, first data byte after length = block number Single-part Read 03h Read Block, first data byte after length = Tag = 0x00 Multi-part Read Start - first part 03h Read Block, first data byte after length = Tag = 0x01 Middle part(s) if any 09h Read Block, first data byte after length = block number End - last part 0Ah Read Block, first data byte after length = block number Retry 0Bh Write Block, first data byte after length = block number Get Message Interface Status Set Message Interface Control 04h 05h Read Block Write Block Reserved 0Ch-17h Reserved for future definition for module management functions OEM 00h, 01h Available for OEM use All other 57

58 SMBus Timeout and Hang Handling The System Management Bus (SMBus) Specification, Version 2.0 provides an option for devices to drop off the bus and go back to waiting for the SMBus START condition if the SMBus clock is held low for greater than 25 ms. There is no requirement for management controllers to implement this option. Per the System Management Bus (SMBus) Specification, Version 2.0, the management controller must synch up to a SMBus START or STOP condition, regardless of which SMBus clock the condition occurs on. In order for a master to place a START or STOP condition on the bus, there must be no other party driving the SMBus data line low during the SMBus clock. It is possible on SMBus that a missed clock or incorrectly terminated transfer could leave a slave device that is being read in the state where it is outputting a 0 data bit on the bus waiting for the master to continue to clock the bus for the next bit(s). In this condition, the bus is in a state where START and STOP conditions cannot be generated by the master because the slave is holding the data line low Management Controller Messaging Timing The management controller must allow the bus master to resynchronize the bus by allowing the master to clock the management controller until the master can issue a START or STOP condition. This means that a management controller should drop off the bus and let its data and clock lines go high (not driven) if it gets clocked for returning more data than it has available. Table 2-22 lists the recommended timing specifications on SMBus for a management controller implementing protocols for SMBus messaging. Note that this timing can be dependent on the clock rate at which the master drives the bus. Specifications are given for a SMBus driven at 100 kbps. Table 2-22: SMBus Messaging Timing Specifications Internal Timing Specifications Min Max Comments Overall Message Duration T1-20 ms Time-out waiting for bus free T2 60 ms - Time-out waiting for a response, internal T3 60 ms [1] T6max [1] Time between Event Message Requests T4 5 ms - Request-to-Response time T5 - T6max- T1max- 3 ms [1] Number of Request retries C1 5 [2] - Recommended Time between Request retries T6 60 ms 250 ms Number of Event Message Request retries C This interval is measured from the end of the request transmission through the end of response transmission (SMBus STOP to SMBus STOP). 58

59 Internal Timing Specifications Min Max Comments Overall Message Byte Duration T7 per the SMBus spec. SM Bus Clock Low hold T8 per the SMBus spec. 3 ms Recommended 3 ms Recommended The BMC should avoid stretching the clock more than 3 ms at a time. The BMC must tolerate the clock being stretched up to the maximum value specified by the System Management Bus (SMBus) Specification, Version 2.0. Notes: 1. Unless otherwise specified, this timing applies to the mandatory and optional commands specified in the Intelligent Platform Management Interface Specification. For controller-specific Application and Firmware commands, the Responder should attempt to meet this specification. In cases that it cannot, the interface specification for the Responder must clearly specify the Request to Response time that was implemented. Because timing can vary according to command and controller, communication routines should be designed to support response timeouts and retry counts accordingly. 2. This is a recommended value only. The protocol does not require that non-event Message requests be retried. The implementation of retries and the number used is based on the application s requirements for message delivery IPMI Management Controller Message Formats The following tables show examples of IPMI messages carried as the message data using the Singlepart Write and Single-part Read protocols. IPMI messages use message class 0x00. If multi-part transfers are required, the IPMI CMD data will span into the message data of the transactions following the Start transaction BMC to Module IPMI Message Format Examples Figure 2-26 shows the example of a standard IPMI Request message delivered to a module using the Single-part Write protocol. This is typical of a transaction that a BMC would use to send a request to a module. The format for sending a response is similar, with the main difference of the inclusion of a Completion Code in the IPMI message data and the swapping of addressing fields between the request and response. 1 Responder s Slave Address (7) R/W=0 (1) SMBus CMD = 02h Length Message Class (0x00 = IPMI) NetFn [even] (6), Responder s LUN (2) Requester s Slave Address Requester s Seq (6), Requester s LUN (2) IPMI CMD IPMI CMD data [PEC] Figure 2-26: BMC to Module IPMI Standard Request 1 Note that some OEM IPMI NetFn values specify different fields following the CMD byte, but the presence of a CMD field and all fields preceding the CMD byte are common for all IPMI messaging, regardless of the NetFn value. 59

60 Requester s Slave Address R/W=0 (1) SMBus CMD = 02h Length Message Class (0x00 = IPMI) NetFn [odd] (6), Requester s LUN (2) Responder s Slave Address (7) Requester s Seq (6), Responder s LUN (2) IPMI CMD completion code IPMI CMD data [PEC] Figure 2-27: BMC to Module IPMI Standard Response Where: Responder s Slave Address = Slave address of the target management controller on the module. SMBus CMD = Identifies the particular Write transaction operation to perform on the bus. This field is used to differentiate between operations at the transaction level. For a request, this field is 02h indicating a Single-part Write transaction. Length = SMBus length for an SMBus Write Block transaction. This value represents the number of following data bytes less PEC. Per the System Management Bus (SMBus) Specification, Version 2.0, the LEN must be the third byte of an SMBus Block Write protocol transaction. The Bus Op 1 field occupies the position of the SMBus v2.0 CMD byte for a Block Write protocol transaction. NetFn = Network Function Code. This 6-bit field identifies the type of message data that follows. Besides identifying IPMI message classes, it also provides values that support namespaces for messages from other defining bodies and OEMs. The IPMI convention is that even NetFn values are used for Requests and odd values for Reponses. LUN = Logical Unit Number. This 2-bit field is auxiliary addressing information for IPMI. A management controller can accept messages up to four different logical units. Typically, a management controller implements just one LUN, 00b. For non-ipmi namespaces, this value can be defined by the respective OEM or defining body. Seq = Sequence Number. This 6-bit field is used to match responses up with requests. This enables the responder the option of not immediately responding to a request and allows management controller messaging to be multi-threaded. The originator of the request (the Requester) picks the sequence number for the request; the responder returns that same sequence number in the response. IPMI CMD = The command value for IPMI messages as defined in the IPMI specification. IPMI data = The accompanying request data bytes for the given command, if any. PEC = Packet Error Code. The PEC is an 8-bit CRC calculated per the System Management Bus (SMBus) Specification, Version 2.0. PEC is MANDATORY for IPMI messages to/from PCI Express Modules. 60

61 Module to BMC IPMI Message Format Examples The following figures show the format of Single-part Read protocol accesses to retrieve IPMI requests and responses from a management controller on a module. In these examples, the Request and Response are being addressed to the BMC (address 0x20, 0010_000x). See Section for field definitions. Slave Address (7) Responder s Slave Address (e.g. 0010_000) R/W = 0 (1) 0 SMBus CMD = 03h NetFn [even] (6), Responder s LUN (2) Slave Address (7) Requester s Slave Address R/W=1 (1) Requester s Seq (6), Requester s LUN (2) Length IPMI CMD Figure 2-28: Module to BMC IPMI Standard Request message class = 0x00 (IPMI) IPMI CMD data Tag = 0x00 [PEC] Slave Address (7) Requester s Slave Address (e.g. 0010_000) 0 R/W = 0 (1) SMBus CMD = 03h NetFn [odd] (6), Requester s LUN (2) Slave Address (7) Responder s Slave Address (7) Requester s Seq (6), Requester s LUN (2) R/W=1 (1) IPMI CMD Length completion code Figure 2-29: Module to BMC IPMI Standard Response message class = 0x00 (IPMI) IPMI CMD data Tag = 0x00 [PEC] Pre-assigned/Reserved Slave Addresses The following table lists the SMBus address assignments and ranges available for on-module devices. The address ranges are split into four ranges: b for board-set. These are defined or reserved for definition by the board-set manufacturer. M for Module. These are defined or reserved for use by the module manufacturer. c for chassis. These are reserved for use by a chassis that a board-set goes into. A vendor that is a system integrator that provides both the board and chassis as a packaged product can combine the board and chassis ranges and use them for either purpose. - for reserved. These are addresses that are reserved under this specification. Unshaded rows indicate pre-assigned and reserved addresses. Shaded rows indicate addresses that are available for use by the designated party. This specification provides the addresses for an individual SMBus segment connected to a module slot. Note that the incorporation of multiplexing can provide isolation that enables overlapping address assignments on SMBus segments that are separate from the module SMBus segments. This specification does not cover the assignment of addresses to non-expressmodule segments, except for the location of a baseboard VPD device that is required if a multiplexer is used for accessing modules via an SMBus host controller. 61

62 Table 2-23: Management I/O Addresses Address Range Function/Device Number of Addresses 0000_000x 00h - Reserved (I 2 C/IPMI Broadcast) _0001:0000_111x 01h:0Eh - Reserved (SMBus/I 2 C) _000x 10h b SMBus Host _001x 12h b Board OEM device _010x 14h M Module OEM device _011x 16h c Chassis OEM device _100x 18h b SMBus ARA _000x 20h b BMC _001x:0010_011x 22h:26h b Board OEM devices _100x:0010_101x 28h:2Ah M Module OEM devices _110x:0010_111x 2Ch:2Eh c Chassis OEM devices _000x:0011_011x 30h:36h b Board OEM devices _100x:0011_101x 38h:3Ah M Module OEM devices _110x:0010_111x 3Ch:3Eh c Chassis OEM devices _000x:0100_010x 40h:44h b Board OEM devices _011x 46h M Module OEM device _100x 48 M Basic Management Status Register _100x:0100_111x 4Ah - Reserved _110x:0100_111x 4Ch:4Eh c Chassis OEM devices _000x:0101_011x 50h:56h b Board OEM devices _100x:0101:101x 58h:5Ah M Module OEM devices _100x:0101:101x 5Ch:5Eh c Chassis OEM devices _000x:0111_011x 70h:76h b Board OEM devices _100x:0111_101x 78h:7Ah M Module OEM devices _110x:0111_111x 7Ch:7Eh c Chassis OEM devices _000x:1000_011x 80h:86h b Board OEM devices _100x:1000_101x 88h:8Ah M Module OEM devices _110x:1000_111x 8Ch:8Eh b Chassis OEM devices _000x:1001:011x 90h:96h b Board OEM devices _100x:1001:101x 98h:9Ah M Module OEM devices _110x:1001:111x 9Ch:9Eh c Chassis OEM devices _000x:1010_011x A0h:A6h M Module FRU/VPD Devices _100x:1010_111x A8h:AEh b Board OEM devices _001x C2h b,m,c SMBus 2.0 Device Default Address _010x:1101_011x D0h:D2h b Board OEM devices 2 62

63 Address Range Function/Device Number of Addresses 1101_010x:1101_011x D4h:D6h M Module Satellite Management Controller _100x:1101_101x D8h:DAh M Module OEM devices _100x:1101_111x DCh:DEh c Chassis OEM devices _000x:1110_011x E0h:E6h B Board OEM devices _100x:1110_101x E8h:EAh M Module OEM devices _110x:1110_111x ECh:EEh c Chassis OEM devices _000x:1111:111x F0h:FEh - Reserved (SMBus/I 2 C) Auxiliary Signal Parametric Specifications DC Specifications Table 2-24: Auxiliary Signal DC Specifications Symbol Parameter Conditions Min Max Unit Notes V IL1 Input Low Voltage V 2, 7, 9 V IH1 Input High Voltage V 2, 7 V IL2 Input Low Voltage V 4 V IH2 Input High Voltage 2.1 Vaux V 4 V IH3 Input High Voltage Min voltage at 20 ma sink V 9 V OL Output Low Voltage 3.0 ma 0.4 V 1, 3, 4, 6, 8 V HMAX Max High Voltage 3.6 V 3, 8 I IH Input High Current ma 9 I in Input Leakage Current 0 to 3.3 V µa 2, 4 I lkg Output Leakage Current 0 to 3.3 V µa 3, 5, 6, 8 C in Input Pin Capacitance 7 pf 2 C out Output (I/O) Pin Capacitance 30 pf 3, 4, 6, 8 Notes: 1. These are open-drain outputs that require pull-ups on the system board. There is no V OH specification for these signals. The voltage listed is the maximum voltage that can be applied to the pin. 2. Applies to MRST#. 63

64 3. Applies to WAKE. 4. Applies to SMBus signals SMBDATA and SMBCLK. 5. Leakage at the pin when the output is not active (high impedance). 6. Applies to SMBAlert. 7. Applies to PWREN#. 8. Applies to MPWRGD# and PWRFLT#. 9. Applies to ATTLED and PWRLED module inputs. These are input drive LEDs with series current limiting resistors on the module AC Specifications Table 2-25: Power Sequencing and Reset Signal Timings Symbol Parameter Min Max Units Notes Figure T PVPERL Power stable to MRST# inactive 100 ms 1 Figure 2-10 T MRST-CLK REFCLK stable before MRST# inactive 100 µs 2 Figure 2-10 T MRST MRST# active time 100 µs Figure 2-12 T fail Power level invalid to MRST# active 500 ns 3 Figure 2-12 T wkrf WAKE rise fall time 100 ns 4 Figure 2-30 T WAKE WAKE hold time from MRST# active 1 ms Figure 2-11 Notes: 1. Any supplied power is stable when it meets the requirements specified for that power supply. 2. A supplied reference clock is stable when it meets the requirements specified for the reference clock. The MRST# signal is asserted and deasserted asynchronously with respect to the supplied reference clock. 3. The MRST# signal must be asserted within T fail of any supplied power going out of specification. 4. Measured from WAKE assertion/deassertion to valid input level at the system PM controller. Since WAKE is an open-source signal, the rise time is dependent on the total capacitance on the platform and the system board pull-up resistor. It is the responsibility of the system designer to meet the rise time specification. 64

65 WAKE V IH_PMC 1 V IL_PMC 1 WAKE T WRKF Note 1: Power Management Controller input switching levels are platform dependent and are not set by this specification. A-0457 Figure 2-30: WAKE Rise and Fall Time Measurement Points 65

66 66 PCI EXPRESS EXPRESSMODULE ELECTROMECHANICAL SPECIFICATION, REV. 1.0

67 3 3. Hot Insertion/Removal The following text references to mechanical elements should be interpreted in the context of the PCI Express ExpressModule form factor definitions Scope This ExpressModule form factor intrinsically supports Hot Insertion and Removal (hot-plug) of add-in modules. This modular form factor is physically different than that of a PCI card. This changes some aspects of the hot-plug usage model but, in general, it is based on the standard model defined in the PCI Standard Hot Plug Controller and Subsystem Specification, Rev The following sections describe the add-in module native hot-plug features. For a detailed explanation of the register requirements and standard usage model, see Section 7.7 of the PCI Express Base Specification, Revision Hot-Plug Sub-System Architecture The module hot-plug model is very similar to that of the PCI card hot-plug model with most of the same steps required. The primary differences are mechanical along with a few electrical changes; that is, the module is an enclosed adapter with on-board power regulation. This changes the host adapter interface from the standard PCI and PCI Express cards model. That model requires a number of electrical switches to control power and interface signals on the host side of the interface to prevent damage to the card or host system during hot-plug operations. This interface reduces the number of signals that must be controlled to that of the reference clock. This is accomplished by two basic changes to the adapter/host interface. The first is to provide bulk power to the module, thus eliminating the need for power control devices on the system side of the interface. This allows the module to better meet the voltage needs of the logic on the adapter over time. New signals are also added to the adapter interface: power enable (PWREN#) and module power good (MPWRGD#). The power enable signal (PWREN#) controls the bulk power DC to DC converter on the module. This signal is connected to the standard hot-plug controller s power enable output signal on slots that fully support hot-plug and to ground on slots that do not fully support hot-plug. The adapter contact for PWREN# is shorter than the power and ground contacts thus ensuring that the module s DC to DC converters cannot be activated until the module has a good connection in the slot connector. The shorter contact also ensures that power is disabled before the module is fully removed from the slot connector. This is to prevent arcing during plugging or unplugging. The module power good signal (MPWRGD#) indicates to the system that the modules onboard power converter is stable. 67

68 The second basic change to the adapter/host interface is that the rest of the active interfaces are driven by open collector/open drain or tri-state drivers. Some of these signals are terminated on the adapter from Aux power. This protects the adapter during the hot-plug event and eliminates the need for switching these signals on the host side of the interface. This leaves only the reference clock (REFCLK) signals that must be disabled during a hot-plug event, which is enabled by MPWRGD# being active (logic 0) and the PRSNT# signal being active. This forces the reference clock signals inactive during a hot-plug or removal event and eliminates any EMI that could be produced by driving these signals into an empty slot connector. Note: Modules and slots designed to this specification will not be physically damaged by a hot insertion or removal event. However, slots that do not implement hot-plug controllers and drivers may experience unpredictable behavior during hot insertion or removal of module. All modules are required to support the hot-plug architecture. System slots, however, are encouraged to support the full hot-plug architecture but are only required to support the features that prevent damage to the adapter or slot, i.e., gating of the reference clock. Doublewide modules have the possibility to connect to two backplane connectors. The primary connector for a doublewide module is the one next to the module carrier or frame and the one closest to the cover of the module is the secondary interface connector. Doublewide modules with only a single interface connection are allowed to use either the primary or secondary connector in the module. There are no differences in the interface from a singlewide module. Doublewide modules that connect to both slot connectors have special interface requirements related to the secondary connector. The secondary connector in a doublewide module may only be used to support additional power (maximum of 50 watts) and or the additional x8 PCI Express lanes to form a x16 interface for the module. If power is used from the secondary connector, it must be controlled by the primary connectors PWREN# signal. The following interface signals are no connects for the secondary connector: ATNLED, PWRLED, ATNSW, WAKE, PRSNT#, PWREN#, PWRFLT#, MPWRGD#, MRST#, REFCLK, SMBCLK, SMBDAT, and SMBAlert. Figure 3-1 shows the elements of the PCI Express ExpressModule hot-plug interface. 68

69 Electro-mechanical Lock AND PWRLED ATNLED EMILS EMIL ATNSW# PWRLED ATNLED AUX PWR Bulk PWR Long Pin Short Pin Aux Power Host Interface PCI Express Bridge PCI Express Hot-Plug Controller PWREN# PRSNT# PERST# Delay Module Backplane Connector PWRFLT# MPWRGD# MRST# Short Pin Short Pin Bulk Power Converter WAKE# WAKE Short Pin PCI Express Ref Clock PCI Express Link PCI Express I/O Applications System Management Bus Controller System Slot Interface SMBus SMBAlert System Management Interface Module Adapter Interface A-0458 Figure 3-1: Typical Hot-Plug Interface Implementation As the module engages with the slot connector, most of the interface pins make contact at the same time. The exceptions are the presence detection (PRSNT#), power enable (PWREN#), power fault (PWRFLT#), module power good (MPWRGD#), Vaux (Vaux), and wake (WAKE) pins. These pins are shorter than the other pin on the module connector such that they make contact last during insertion and break contact first during removal. Some of the signals shown in Figure 3-1 are described in the PCI Express Base Specification, Revision 1.1 and the PCI Express Card Electromechanical Specification, Revision 1.1, but are listed in the following tables for clarity. This is important because some have different polarity than what is identified in the other specifications. 69

70 Table 3-1: Power Signals Signal Name I/O Definition as Viewed From the Host Side PWREN# O Power Enable: Required A slot Control Logic output that controls the power state of the slot. If this signal is low, power is enabled to the slot. PWRFLT# I Power Fault: Required This pin function is Module Power fault, indicating that the module s power converter has a failure. The Module driver is open drain. See the PCI Standard Hot Plug Controller and Subsystem Specification for more details. WAKE I Adapter Power Request: Required Requests the system to return to a full power state. Adapters that do not support this function must connect to ground. See the PCI Express Base Specification, Revision 1.1, for more details. MPWRGD# I Module Power Good: Required Indicates to the system that all module power supplies are good. The output of the on module DC to DC converter is stable. Module driver is open drain. MRST# O Module Reset: Required This system output provides two functions to the adapter module. First, that power and REFCLK are stable (within specification) during power up. Second, as an early warning that a power down is imminent. Note when the system s primary power is off and Aux power on, this output must be driven to a logic 0 (less than 0.4 V DC). Table 3-2: Management Interface Signal Name I/O Definition as Viewed From the Host Side PRSNT# I Present Detect Input: Required Indicates that a module is in the slot. Used to detect insertion or removal of a module as well as part of the reference clock enabling function. SMBus I/O Systems Management Bus: Required A two line interface compatible with most I2C components. See the System Management Bus (SMBus) Specification, Version 2.0, for a detailed description. Note: These inputs must have a weak pull-down on the host side to condition signals during hot-plug event. 70

71 Signal Name I/O Definition as Viewed From the Host Side SMBAlert I Systems Management Bus Alert: Optional for module only Input to the system management subsystem that an adapter needs attention. If this function is not implemented on the module, it must be connected to ground. Note: This input must have a weak pull-down on the host side to condition signals during hot-plug event. Table 3-3: User Interface Signal Name I/O Definition as Viewed From the Host Side PWRLED O Power LED: Required A slot control logic output that is used to drive the Power Indicator. This output is asserted to illuminate the Indicator. See the PCI Express Base Specification, Revision 1.1, for more details. Note: This interface signal sources the LED current for the ExpressModule; this is to minimize any dependencies on the module. See Table 2-24 for signal voltage and current. ATNLED O Attention LED: Required A slot control logic output that is used to drive the Attention Indicator. This output is asserted to illuminate the Indicator. See the PCI Express Base Specification, Revision 1.1, for more details. Note: This interface signal sources the LED current for the ExpressModule; this is to minimize any dependences on the module. See Table 2-24 for signal voltage and current. MRL# I Manual Retention Latch Sensor: Not supported The MRL is not supported for ExpressModule slots. EMIL O Electromechanical Interlock: Optional The ExpressModule is designed to support an optional locking mechanism that is controlled by a bit in the hotplug controller. See the PCI Express Base Specification, Revision 1.1, for more details on programming. This output is pulsed when a logic 1 is written to the control bit. The function of this pulse is to set a latching interlock mechanism. Each output pulse is 100 ms long. EMILS I Electromechanical Interlock State: Optional This input is required if an electromechanical Lock is implemented. This input is used to determine the current state of electromechanical lock that holds a module in the chassis. See the PCI Express Base Specification, Revision 1.1, for more details. 71

72 Signal Name I/O Definition as Viewed From the Host Side ATNSW# I Attention Switch: Required A low true (low logic level when the switch is pressed) input that is connected directly to the Attention switch. If the platform does not support Attention switch, this input must be wired to a high logic level. See the PCI Express Base Specification, Revision 1.1, for more details Power Enable The power enable (PWREN#) input is used to control the adapter module s primary power and associated supplies. In slots that do not support hot-plug, this interface signal is connected to logic ground. In slots that support hot-plug, the hot-plug controller power enable output is connected to this signal to manage the adapter s primary power. The PWREN# signal pad on the add-in module connector is shorter than most of the pads on the connector. This is to ensure that it is one of the last signals to make contact in the connector and the first to break contact. Note: Slots that do not support hot-plug controllers must enable power by grounding the PWREN# signal to the card slot and gate REFCLK to the module with the MPWRGD# and PRSNT# signals. See the PCI Express Base Specification, Revision 1.1, for more details on the REFCLK function Power Fault The power fault signal (PWRFLT#) is an open drain output from the module and provides an indicator that the module has a power fault; i.e., module power supplies cannot maintain power regulation. This signal is intended to warn the Hot-Plug Controller of power problems on the module. The power fault signal is a negative active signal in the ExpressModule interface to minimize the requirements of the module power supplies and hot-plug support. The slot input has a strong pull-up to prevent generating a false power fault during a hot-plug event or when a module is not present. Once the power fault condition has occurred, the PWRFLT# signal is latched by the adapter. It will remain latched in the fault state until the PWREN# signal is toggled, after which it is reset. The PWRFLT# signal pad on the add-in module connector is shorter than most of the pads on the connector. This is to ensure that it is one of the last signals to make contact in the connector and the first to break contact. See the PCI Standard Hot Plug Controller and Subsystems Specification, Revision 1.0, for more details on the power fault function Wake The wake (WAKE) signal is an output from the module and provides an indicator that the module request that the system power on. The wake connector pad on the module is short to give the module more time to ensure that WAKE event is not falsely triggered during a hot-plug event. The Wake signal is a positive signal in the ExpressModule interface to minimize the requirements of hotplug support. The slot input has a weak pull-down to prevent generating a wake event when a 72

73 module is not present. See the PCI Express Base Specification, Revision 1.1, or Section 3.2 for more details on the function of the Wake signal Module Power Good The module power good (MPWRGD#) signal is an open drain output from the module that indicates the on module power supplies are within operational limits for the application. This signal is used to generate the MRST# signal to reset the module application when supplies are not within operational limits. The MPWRGD# signal pad on the add-in module connector is shorter than most of the pads on the connector. This is to ensure that it is one of the last signals to make contact in the connector and the first to break contact. The slot input requires a strong pull to prevent a false indication during hot-plug events Module Reset The Module Reset (MRST#) signal is an input to the module and indicates that the system power supply is stable and the on-module power converters are stable. This signal is made from system reset and a delayed MPWRGD#. The system reset goes active as soon as system power is unstable driving MRST# active. The delayed MPWRGD# provides a delay after module power is stable to allow the reference clock to be enabled and stable before MRST# goes inactive. See Chapter 2 for more details on the function of the Module Reset signal Present Detection The PCI Express hot-plug controller detects the presence of an add-in module using the presence detection signal (PRSNT# ). It is the responsibility of the Root Complex or the Switch to determine the presence of the add-in module and set the present bits in the appropriate register as described in PCI Express Base Specification, Revision 1.1. The presence detection signal is required on all PCI Express ExpressModule connectors and add-in modules. Figure 6-4 shows the short pins on the card edge connector. These pins are used as the presence detection mechanism. The add-in module must be fully inserted in to the connector to ensure that the module is detected by the system. The PRSNT# signal pad on the add-in module is shorter than the rest of the pads on the connector. This is to ensure that it is one of the last signals to make contact in the connector and the first to break contact. This provides the system with a valid indicator that the module is installed in the slot. See the PCI Express Base Specification, Revision 1.1, for more details on the function of the presence detection signal System Management Bus See Section 2.5 and the PCI Express Base Specification, Revision 1.1, for details on the function of the System Management Bus. 73

74 System Management Bus Alert See Section for details on the System Management Bus Alert Power LED The Power LED (PWRLED) signal is an output from the PCI Express port s hot-plug controller. This signal controls the Power LED. Each module is required to support the Power LED. The slot Power LED signal provides the voltage and current to light the Power LED (see Table 2-24). The module provides the LED and current limiting resistor. The Power LED is defined in this fashion to increase its reliability by reducing its dependence on the module. See Figure 3-1 for an example of how this function is wired. See the PCI Express Base Specification, Revision 1.1, for more details on the Power LED s function Attention LED The Attention LED (ATNLED) signal is an output from the PCI Express port s hot-plug controller. This signal controls the Attention LED. Each module is required to support the Attention LED. The slot Attention LED signal provides the voltage and current to light the Attention LED (see Table 2-24). The module provides the LED and current limiting resistor. The Attention LED is defined in this fashion to increase its reliability by reducing its dependence on the module. The chassis can optionally have an Attention LED per slot to identify an empty slot. When a slot has a module installed (PRSNT# active), the chassis Attention LED must be disabled. See Figure 3-1 for an example of how this function is wired. See the PCI Express Base Specification, Revision 1.1, for more details on the Attention LED s function Manual Retention Latch The Manual Retention Latch (MRL) is not supported for ExpressModule slots. See the PCI Express Base Specification, Revision 1.1, for more details on the MRL s function Electromechanical Interlock The electromechanical interlock (EMIL) signal is an optional output from the PCI Express port s hot-plug controller. All slots supporting the EMIL function shall implement the EMIL as defined by this specification and the PCI Express Base Specification, Revision 1.1. This signal controls an electromechanical interlock for a single slot in the chassis. Silicon supporting this function shall drive this signal when activated for a pulse length greater than 100 ms and less than 150 ms duration. This signal causes the locking mechanism to change position (locked/unlocked). The EMIL mechanism must change states (locked/unlocked) in less than 500 ms from EMIL signal active. Software that controls this function shall provide a minimum of 1 second between pulses of the EMIL. The optional electromechanical interlock feature provides physical security for the module while also protecting the service personnel from accidentally removing the wrong module. The use of the 74

75 electromechanical interlock on doublewide module requires that the secondary slot control the electromechanical interlock. Note that the SMBus VPD may be the only means to determine that the module is doublewide. Systems that implement the electromechanical interlock must also support SMBus management. See the PCI Express Base Specification, Revision 1.1, for more details on the EMIL signal s function Electromechanical Interlock Status The electromechanical interlock status (EMLS) signal is an input to the PCI Express port s hot-plug controller. It is required only if the electromechanical interlock is implemented. The function of this signal is to provide the current state of the electromechanical interlock (locked/unlocked). The electromechanical interlock sensor status is read from the slot control register and has the definitions listed in Table 3-4. Table 3-4: EMLS Signal Value 0b 1b Status Disengaged Engaged See the PCI Express Base Specification, Revision 1.1, for more details on the EMLS signal s function Attention Switch The Attention switch is a required feature of all modules. The system is responsible for debouncing of the Attention switch signal if it is desired. See the PCI Express Base Specification, Revision 1.1, for more details on the Attention switch s function. 75

76 76 PCI EXPRESS EXPRESSMODULE ELECTROMECHANICAL SPECIFICATION, REV. 1.0

77 4 4. Module Power Interface 4.1. Power The I/O adapter module is supplied two power sources, one for Bulk or Primary power and one for Aux power. All of the adapter module s primary power needs are supplied from the Bulk power supply. The Aux supply provides limited power to a slot for management interface and wake features. The power delivery requirements defined in this section define the environment that the ExpressModule will be required to operate within Module Primary Power Supply The PCI Express modules defined in this specification require a primary supply of +12V. Systems that provide slots for ExpressModules are required to provide +12V supply to each slot connector with sufficient power and regulation to meet minimum requirements. See Table 4-1 for power level and tolerances Module Auxiliary Power Supply The PCI Express modules defined in this specification require an auxiliary supply of +3.3V. Systems that provide slots for ExpressModules are required to provide +3.3Vaux supply to each slot connector. See Table 4-1 for power level and tolerances. 77

78 4.2. ExpressModule Power Supply Requirements Table 4-1 identifies the power supply requirements for each module type. Table 4-1: Power Supply Rail Requirements Power Rail Singlewide Doublewide +12V Bulk Voltage Tolerance Continuous Current Initial Hot-plug Capacitance Input Capacitance ±15% (max) 2.08 A (max) 5000 pf (max) 500 µf (max) ±15% (max) 4.17 A (max) 5000 pf (max) 500 µf (max) +3.3V Aux Voltage Tolerance Continuous Current Peak Pre-charge Current Input Capacitance Pre-charge Pin Timing ±10% (max) 475 ma (max) 475 ma (peak) 150 µf (max) 3 ms (max) ±10% (max) 950 ma (max) 950 ma (peak) 300 µf (max) 3 ms (max) Notes: 1. The +12V power hot swap circuits shall be located on the module. 2. Currents during hot insertion shall not exceed the module maximum continuous current. 3. The module and connector shall not be damaged during hot removal or insertion. 4. The +3.3Vaux power shall pre-charge the module s 3.3Vaux input capacitors during hot insertion via mate first pre-charge pin. 5. Peak pre-charge current during hot insertion is determined by the value of the pre-charge resistor. Single wide example: Pre-charge resistor = +3.3V/475 ma = 7 Ω Vaux Pre-charge Pin Timing is the maximum time guaranteed during hot insertion from the +3.3Vaux pre-charge pin mating to the main power pins mating. The time constant with the maximum input capacitance and pre-charge resistor shall not exceed 1/3 of the Pre-charge Pin Timing. Example: 150 µf x 7 Ω = 1 ms (which is 1/3 of the maximum pre-charge pin timing of 3 ms). 7. The maximum current slew rate for each add-in module shall be no more than 0.1 A/µs. 8. Each add-in module shall limit its capacitance on each power rail at the backplane connector to that listed in the above table. 9. Continuous current = the highest averaged current value over any 1-second period. 78

79 4.3. Power Consumption There are two power configurations defined for the modules: Singlewide Module = Up to 25 watts Doublewide Module = Up to 50 watts This breaks down from a connector perspective to the following: The singlewide module connector supports the doublewide maximum power (50 watts) to allow a doublewide module to draw all its power from a single connector. The doublewide module can use both connectors available for power, but the maximum power that can be drawn is 50 watts. This allows a doublewide module to draw its power from either connector or both to the maximum of 50 watts. The power drawn from the connector has no relation to the width of the PCI Express Link supported in the connector; that is, a doublewide module that only uses a x4 Link can draw up to 50 watts from a single connector. Systems that do not support the maximum power on all slots must support active power management Power Supply Sequencing The Aux power supply must be applied to the module before or at the same time as the primary power. The module slot connector has both Aux and Primary power applied when the system is power on. During hot-plug events, the system does not remove power from the module slot connector. The system must drive the module reset (MRST#) signal active (logic 0) any time the Primary or Aux power goes out of tolerance Power Supply Decoupling Due to the low level signaling of the PCI Express interface, it is strongly recommended that sufficient decoupling of all power supplies be provided. This is recommended to ensure that power supply noise does not interfere with the recovery of data from a remote upstream PCI Express device. Some basic guidelines to help ensure a quiet power supply are provided below. Note: The following are guidelines only. It is the responsibility of the ExpressModule designer to properly test the design to ensure that ExpressModule circuitry does not create excessive noise on power supply or ground signals at the ExpressModule edge fingers. The add-in adapter module device decouple value should average 0.01 µf per device Vcc pin (for all devices on the add-in card). The trace length between a decoupling capacitor and the power supply or ground via should be less then 0.2 inches (5.08 mm) and be a minimum of 0.02 inches (0.508 mm) in width. 79

80 80 PCI EXPRESS EXPRESSMODULE ELECTROMECHANICAL SPECIFICATION, REV. 1.0

81 5 5. Module PCI Express Interface 5.1. PCI Express Link Signals See the PCI Express Base Specification, Revision 1.1, for details about Link signals. The singlewide ExpressModule supports Link widths of x1, x2, x4, and x8. The doublewide ExpressModule supports Link widths of x1, x2, x4, x8, and x PCI Express Electrical Topologies and Link Definitions This section describes the electrical characteristics of PCI Express add-in ExpressModule. The electrical characteristic at the adapter module interface is defined in terms of electrical budgets. This budget allocation decouples the electrical specification for system designer and the adapter module vendor and ensures successful communication between the PCI Express signal inputs and outputs Links on the system and adapter module interface. Unless otherwise noted, the specifications contained herein apply to all high-speed signals of each interface width definitions. The signaling rate for encoded data is 2.5 G transfers/s and the signaling is point-to-point Topologies There are four different electrical topologies for PCI Express: PCI Express devices on the same system board PCI Express devices across one connector, on systems with a system board and adapter card PCI Express devices across two connectors, on systems with a system board, riser card, and adapter card PCI Express devices across three connectors, on systems with a cable connecting to a secandary backplane with an adapter card plugged into it The first topology is defined for devices on the same board and not related to add-in adapter modules. Refer to the PCI Express Base Specification, Revision 1.1, for implementation of this topology. The second topology is defined for devices across one connector. This allows a plug-in adapter module to be plugged into a backplane that has a PCI Express bridge on it or directly onto a system board. 81

82 The third topology is defined for devices across two connectors. This allows plug-in adapter modules to be plugged into a backplane (sometimes referred to as a riser card) and the backplane to be plugged into the system board. The system board to backplane connector is not defined by this specification but is required to meet the topology budget defined in this specification. The fourth topology is defined for devices across three connectors. This allows plug-in adapter modules to be plugged into a backplane and the backplane to be connected to the system board via a cable. The system board to backplane cable is not defined by this specification but is required to meet the topology budget defined in this specification Link Definitions Typical ExpressModule PCI Express Links consist of the following: Transmitters/receivers on an ASIC on a system board Package fan-in-out trace topologies PCB coupled microstrip and/or striplines Vias for layer changes Optional system board to riser card connector and interface Optional riser card with microstrip and/or stripline trace Standard baseline riser card connector and card interface Coupled microstrip line and/or stripline traces on the ExpressModule Decoupling capacitors Transmitter/receivers on an ASIC on the ExpressModule The electrical parameters for the Link are subdivided into three components: Adapter plug-in card (ExpressModule) Riser card/backplane connector (ExpressModule connector) System board (and riser card with associated connector if exists) The electrical impact of discontinuities on the Link, such as via, bend, and test points, should be included in the respective components. 82

83 5.3. PCI Express Electrical Budgets A budget is defined for each of the following electrical parameters associated with the Link: AC coupling capacitors Insertion Loss (Voltage Transfer Function) Jitter Lane-to-lane skew Crosstalk Equalization Skew within a differential pair The electrical budgets are different for each of the two Link components: ExpressModule budget System board and PCI Express connector budgets The interconnect Link budget allocations associated with the Transmitters and Receivers differ. This is to account for any electrical characteristics the AC coupling capacitors may contribute to the Link AC Coupling Capacitors The PCI Express ExpressModule and system board shall incorporate AC coupling capacitors on the Transmitter differential pair. This is to ensure blocking of the DC path between the PCI Express ExpressModule and the system board. The suggested maximum size is a 603-type capacitor with a value as specified in the PCI Express Base Specification, Revision 1.1. Any additional attenuation or jitter caused by the coupling capacitors (larger than 603-type) must be accounted for as part of the budget allocation for the physical interconnect component s path on which the capacitors are mounted. The electrical budgets allocated for the AC coupling capacitors are defined in the following subsections. The allocated budget includes the electrical parasitic effects associated with the component s placement as mounted on the printed circuit board Insertion Loss Values (Voltage Transfer Function) The maximum loss values in decibels (db) are specified for the system board and the add-in card. The insertion loss values are defined as the ratio of the voltage at the ASIC package pin (Transmitter/Receiver) and the voltage at the PCI Express connector interface, terminated by 100 Ω differential termination, realized as two 50 Ω resistances. These resistances are referenced to ground at the interface (see Figure 5-1). 83

84 System Board Measurement reference point at the top of the add-in card edge fingers which have been mated with the connector Transmitter and Package AC Coupling Capacitor System Board Interconnect PCI Express Connector R = 50 Ω R = 50 Ω Measurement reference point at the top of the add-in card edge finger pads. (The edge finger pads are considered part of the connector/ system board interconnect.) Add-in Card R = 50 Ω R = 50 Ω Add-in Card Interconnect Transmitter and Package A-0459 Figure 5-1: Example Interconnect Terminated at the Connector Interface All PCI Express differential trace pairs are required to be referenced to the ground plane. The loss values associated with any riser card interface and adjoining connector implementation must collectively meet the system board loss budget allocations and associated eye diagrams. 84

85 System Board Add-in Card L ST L AR L SR PCI Express Connector L AT A-0460 Figure 5-2: Insertion Loss Budgets Table 5-1: Allocation of Interconnect Path Insertion Loss Budget Loss Parameter PCI Express ExpressModule System Board and Connector Loss Budget Value at 1.25 GHz (db) Loss Budget Value at 625 MHz (db) Comments L AR < 2.65 L AT < 3.84 L AR < 1.95 L AT < 2.94 Note 1, 2 L ST < 9.30 L SR < 8.11 L ST < 6.00 L SR < 5.01 Notes 1, 3 Guard Band Note 1 Total Loss L T < 13.2 L T < 9.2 Notes: 1. All values are referenced to 100 Ω, realized as two 50 Ω resistance to ground. The loss budget values include all possible crosstalk impacts (near-end and far-end) and potential mismatch of the actual interconnect with respect to the 100 Ω reference load. The PCI Express Base Specification. Revision 1.1, allows an interconnect loss of 13.2 db for 1.25 GHz (non de-emphasized) signals and 9.2 db for 625 MHz (de-emphasized) signals. From this, a total of 1.25 db is held in reserve as guard band to allow for any additional attenuation that might occur when the ExpressModule and system board are actually mated. The allocated loss budget values in the table directly correlate to the eye diagram voltages in Section 5.4. Tradeoffs in terms of attenuation, crosstalk, and mismatch can be made within the budget allocations specified. 2. As a guide for design and simulation, the following derivation of the budgets may be assumed for 1.25 GHz signals: 5.2 db is subtracted from 13.2 db to account for near-end crosstalk and impedance mismatches. Out of this, the 1.25 db is reserved as guard band. The following loss allocations are then assumed per differential pair: LAR = 1.4 db; LAT = 1.8 db; LSR = 6.2 db; LST = 6.6 db. These allocation assumptions must also include any effects of far-end crosstalk. 625 MHz values may be derived in a similar manner. The ExpressModule budget does not include the ExpressModule edge finger or connector. However, it does include potential AC coupling capacitor attenuation on the Transmitter (TX) interconnect on add-in card. Note that the budget allocations generally allow for a maximum of 4-inch trace lengths for differential pairs having an approximate 5-mil trace width. No specific trace geometry, however, is explicitly defined in this specification. The subscripts of the Symbol designators, T and R, represent the Transmitter and Receiver respectively. 85

86 3. The system board budget includes the PCI Express connector and assumes it is mated with the card edge finger. Refer to Section 5.3 for specifics on the stand alone connector budget. The system board budget includes potential AC coupling capacitor attenuation on the Transmitter (TX) interconnect on the system board. The subscripts of the symbol designators, T and R, represent the Transmitter and Receiver respectively. Note: The insertion loss budget distributions above are used to derive the eye diagram heights as described in Section 5.4. However, they are provided here only as a design guideline. Compliance measurements must actually be verified against the eye diagrams themselves as defined in Section Jitter Values The maximum jitter values in terms of percentage of Unit Interval (UI = 400 ps for 2.5 G transfers/s) are specified for the system board and the add-in card. The jitter associated with the riser card and associated proprietary connector will be part of the system board jitter budget. The jitter values are defined with respect to 100 Ω differential termination, realized as two 50 Ω resistances. These resistances are referenced to ground at the interface (see Figure 5-1). System Board Add-in Card J ST J AR J SR PCI Express Connector J AT A-0461 Figure 5-3: Jitter Budget The total system jitter budget is derived with the assumption of a minimum Rj for each of the 4 budget items. This minimum Rj component is used to determine the overall system budget. The probability distribution of the Rj component is at the Bit Error Rate (BER) indicated and is Gaussian. For any jitter distribution, the total Tj must always be met at the BER. Tradeoffs of Rj and Dj are allowed, provided the total Tj is always met. More information on the calculation of the system budget can be found in PCI Express Jitter and BER. 86

87 Jitter Contribution Table 5-2: Total System Jitter Budget Min Rj (ps) One Sigma Max Dj (ps) P-P Tj at BER (ps) Tj at BER 10-6 (ps) Tx Ref Clock Media Rx Linear Total Tj: Root Sum Square (RSS) Total Tj: Notes: 1. RSS equation for BER Tj = * 2 Rj Dj n n 2. RSS equation for BER 10-6 Tj = Dj n n * 2 Rj Table 5-3: Allocation of Interconnect Jitter Budget Jitter Parameter Peak Peak Jitter Budget Value (UI) Comments PCI Express ExpressModule System Board and Connector J AR < J AT < Notes 1, 2 J ST < J SR < Notes 1, 3 Total Jitter J T < Note 1 Notes: 1. All values are referenced to 100 Ω, realized as two 50 Ω resistances to ground. The jitter budget values include all possible crosstalk impacts (near-end and farend) and potential mismatch of the actual interconnect with respect to the 100 Ω reference load. The PCI Express Base Specification, Revision 1.1 allows an interconnect jitter budget of UI (equivalent to 90ps for a 400 ps Unit-Interval). The allocated jitter budget values in the Table 5-2 and Table 5-3 directly correlate to the eye diagram widths in Section 5.4. Tradeoffs in terms of attenuation, crosstalk, and mismatch can be made within the budget allocations specified. No additional guard band is specifically allocated. The jitter allocations are then assumed per differential pair according to the table. These allocation assumptions must also include any effects of far-end crosstalk. 2. All values are referenced to 100 Ω. The add-in card budget does not include the add-in card edge finger or connector. However, it does include potential jitter from the AC coupling capacitors on the Transmitter (TX) interconnect of the add-in card. The budget allocations generally allow for a maximum of 4-inch trace lengths for differential pairs having an approximate 5-mil trace width. No specific trace geometry, however, is explicitly defined in this specification. The subscripts of the symbol designators, T and R, represent the Transmitter and Receiver respectively. 87

88 3. All values are referenced to 100 Ω. The system board budget includes the PCI Express connector and assumes it is mated with the card edge finger. Refer to Section 6.4 for specifics on the stand alone connector budget. The system board budget includes potential jitter from the AC coupling capacitors on the Transmitter (TX) interconnect on the system board. The subscripts of the symbol designators, T and R, represent the Transmitter and Receiver respectively. Note: The jitter budget distributions above are used to derive the eye diagram widths as described in Chapter 4. However, they are provided here only as a design guideline. Compliance measurements must actually be verified against the eye diagrams themselves as defined in Section Crosstalk All ExpressModule designs must properly account for any crosstalk that may exist among the various pairs of differential signals. Crosstalk may be either near-end (NEXT) or far-end (FEXT). Each component can have potential impact on a design and must be planned for accordingly. Note that the total maximum crosstalk that a Receiver component in Electrical Idle is required to tolerate is less than the minimum value dictated by the Electrical Idle Detect Threshold (V RX-IDLE-DET-DIFFp-p ) in Table 4-6 of the PCI Express Base Specification, Revision 1.1. Additionally, crosstalk between differential pairs on the ExpressModule will influence and impact the data signals and any subsequent loss and jitter budgets as noted in Sections and Note that all eye diagrams in Section 5.4 must account for any and all crosstalk present. In order to limit crosstalk impacts and implications, it is recommended that the ExpressModule limit the total amount of NEXT to a maximum of 50 mv. All system boards interfacing with an ExpressModule must also properly account for crosstalk. The system board must also account for potential crosstalk that can occur on the printed circuit board as well as within the connector itself (see Section 6.4) Lane-to-Lane Skew The skew at any point is measured using zero crossings of differential voltage of the compliance pattern, while simultaneously transmitting on all physical lanes. The compliance pattern is defined in the PCI Express Base Specification, Revision

89 Table 5-4: Allowable Interconnect Lane-to-Lane Skew Skew Parameter Symbol Skew Values Comments Total Interconnect Skew PCI Express Add-in Card System Board Equalization S 1.6 ns This does not include T Transmitter output skew, L TX-SKEW (specified in the PCI Express Base Specification, Revision 1.1). The total skew at the Receiver (S T + L TX-SKEW ) smaller than L RX-SKEW (specified in the PCI Express Base Specification, Revision 1.1) to minimize latency for this ExpressModule topology. S 0.35 ns Estimates about a 2-inch trace A length delta on FR4 boards S 1.25 ns Estimates about a 7-inch trace s length delta on FR4 boards To reduce ISI, 3.5 db (+/-0.5 db) below the first bit de-emphasis in the Transmitter is required for the ExpressModule and the system board. For implementation details, refer to Chapter 4 in the PCI Express Base Specification, Revision Skew Within the Differential Pair The skew within the differential pair gives rise to a common-mode signal component, which can, in turn, increase EMI. The differential pair should be routed such that the skew within differential pairs is less than 5 mils for the ExpressModule and 10 mils for the system board Eye Diagrams at the ExpressModule Interface The eye diagrams defined in this section represent the compliance eye diagrams that must be met for both the ExpressModule and a system board interfacing with such an add-in card. The specific measurement requirements (probe test points, calibrated system board specifics, etc.) for compliance of physical components are to be specified in a separate PHY Electrical Test Considerations for PCI Express Architecture document. A BER of 10-6 is assumed for the eye diagram measurements. These measurements are to be made using the clock recovery function as defined in the PCI Express Base Specification, Revision 1.1. These compliance eye diagram with BER of can also be used for simulation by following the guidelines explained in Section

90 ExpressModule Transmitter Path Compliance Eye Diagram The eye diagram for the add-in card s Transmitter path compliance is defined in Table 5-5 and Figure 5-4. Table 5-5: ExpressModule Transmitter Path Compliance Eye Requirements Parameter Value Comments V TXA >= 514 mv Notes 1, 2, 5 V TXA_d T TXA >= 360 mv >= 287 ps Notes 1, 2, 5 Notes 1, 3, 5 J TXA-MEDIAN-to-MAX-JITTER <= 56.5 ps Notes 1, 4, 5 Notes: 1. An ideal reference clock without jitter is assumed for this specification. All Links are assumed active while generating this eye diagram. 2. Transition and non-transition bits must be distinguished in order to measure compliance against the de-emphasized voltage level (V TXA_d ). V TXA and V TXA_d are minimum differential peak-peak output voltages. 3. T TXA is the minimum eye width. The sample size for this measurement is 10 6 UI. This value can be reduced to 274 ps for simulation purpose at BER J TXA-MEDIAN-to-MAX-JITTER is the maximum median-to-max jitter outlier as defined in the PCI Express Base Specification, Revision 1.1. The sample size for this measurement is 10 6 UI. This value can be increased to 63 ps for simulation purpose at BER The values in Table 5-5 are referenced to an ideal 100 Ω differential load at the end of the interconnect path at the edge-finger boundary on the add-in card (see Figure 5-1). The eye diagram is defined and centered with respect to the jitter median. Exact conditions required for verifying compliance while generating this eye diagram are given in the PHY Electrical Test Considerations for PCI Express Architecture document. V tx A T tx A V tx A_d A-0462 Figure 5-4: ExpressModule Transmitter Path Compliance Eye Diagram 90

91 ExpressModule Minimum Receiver Path Sensitivity Requirements The minimum sensitivity values for the add-in card s Receiver path compliance are defined in Table 5-6, and a representative eye diagram is shown in Figure 5-5. Table 5-6: ExpressModule Minimum Receiver Path Sensitivity Requirements Parameter Value Comments V RXA 238 mv Notes 1, 2, 5 V RXA_d T RXA 219 mv 246 ps Notes 1, 2, 5 Notes 1, 3, 5 J RXA-MEDIAN-to-MAX-JITTER 77 ps Notes 1, 4, 5 Notes: 1. An ideal reference clock without jitter is assumed for this specification. All Links are assumed active while generating this eye diagram. 2. Transition and non-transition bits must be distinguished in order to measure compliance against the de-emphasized voltage level (V RXA_d ). V RXA and V RXA_d are differential peak-peak output voltages. 3. T RXA is the eye width. The sample size for this measurement is 10 6 UI. This value can be reduced to 233 ps for simulation purpose at BER J RXA-MEDIAN-to-MAX-JITTER is the maximum median-to-peak jitter outlier as defined in the PCI Express Base Specification, Revision 1.1. The sample size for this measurement is 10 6 UI. This value can be increased to 83.5 ps for simulation purpose at BER The values in Table 5-6 are initially referenced to an ideal 100 Ω differential load. The resultant values, when provided to the Receiver interconnect path of the add-in card, allow for a demonstration of compliance of the overall add-in card Receiver path. The sensitivity requirements are defined and centered with respect to the jitter median. Exact conditions required for verifying compliance against these values are given in the PHY Electrical Test Considerations for PCI Express Architecture document. V rx A T rx A V rx A_d Figure 5-5: Representative Composite Eye Diagram for ExpressModule Receiver Path Compliance A

92 System Board Transmitter Path Compliance Eye Diagram The eye-diagram for the system board s Transmitter compliance is defined in Table 5-7 and Figure 5-6. Table 5-7: System Board Transmitter Path Compliance Eye Requirements Parameter Value Comments V TXS >= 274 mv Notes 1, 2, 5 V TXS_d T TXS >= 253 mv >= 246 ps Notes 1, 2, 5 Notes 1, 3, 5 J TXS-MEDIAN-to-MAX-JITTER <= 77 ps Notes 1, 4, 5 Notes: 1. An ideal reference clock without jitter is assumed for this specification. All Links are assumed active while generating this eye diagram. 2. Transition and non-transition bits must be distinguished in order to measure compliance against the de-emphasized voltage level (V TXS_d ). V TXS and V TXS_d are minimum differential peak-peak output voltages. 3. T TXS is the minimum eye width. The sample size for this measurement is 10 6 UI. This value can be reduced to 233 ps for simulation purpose at BER J TXS-MEDIAN-to-MAX-JITTER is the maximum median-to-max jitter outlier as defined in the PCI Express Base Specification, Revision 1.1. The sample size for this measurement is 10 6 UI. This value can be increased to 83.5 ps for simulation purpose at BER The values in Table 5-7 are referenced to an ideal 100 Ω differential load at the end of the interconnect path at the edge-finger boundary on the add-in card when mated with a connector (see Figure 5-1). The eye diagram is defined and centered with respect to the jitter median. Exact conditions required for verifying compliance while generating this eye diagram are given in the PHY Electrical Test Considerations for PCI Express Architecture document. V tx S T tx S V tx S_d Figure 5-6: System Board Transmitter Path Composite Compliance Eye Diagram A

93 It is not always possible to measure the System Board Transmitter Path Eye Diagrams with an ideal reference clock. In this case a two port measurement can adjust the measurement for the non-ideal reference clock. The notation in this section assumes the discrete time domain form of x n. Equivalent notation sometimes used in other literature is x(n) and x[n]. The convolution operator is denoted as. Referring to Figure 5-7, x n is the sampled phase jitter on the reference clock at the connector. A band-pass function is used to measure the reference clock as described in the PCI Express Card Electromechanical Specification. h n is the impulse response of the transmitter PLL on the system board. n n is the intrinsic transmitter jitter. Using an ideal reference clock with 0 phase jitter, this would be the measured transmitter jitter. The peak-peak value of the total jitter must meet the eye requirements as specified in the PCI Express Base Specification. y n = [x n h n ] + n n is the total jitter of the reference clock, transmitter and system board interconnect at the connector, where is the discrete convolution operator. In the case of non-zero reference clock noise, the intrinsic jitter of the transmitter can be calculated in the discrete time domain by the following: n n = y n [x n h n ] where x n is convolved with h n. The peak-peak value of n n is then calculated. The equation above has two unknowns, h n and n n. If h n is known, it can be used directly. Otherwise, h n must be assumed to be the lowest limit of the allowed PLL bandwidth with no peaking (see the PCI Express Base Specification for the specification limits). A summary of this procedure is: 1. Take the simultaneous measurements of y n and x n for the system board at the connector. 2. Find x n h n. 3. Calculate n n = y n [x n h n ] 4. Calculate the peak-peak value of n n. 93

94 n n From Reference Clock x n x n h n h n + y n = x n h n + n n Total Jitter at Connector x n Reference Clock Jitter at Connector A-0465 Figure 5-7: Two Port Measurement Model System Board Minimum Receiver Path Sensitivity Requirements The minimum sensitivity values for the system board s Receiver path compliance are defined in Table 5-8, and a representative eye diagram is shown in Figure 5-8. Table 5-8: System Board Minimum Receiver Path Sensitivity Requirements Parameter Value Comments V RXS 445 mv Notes 1, 2, 5 V RXS_d T RXS 312 mv 287 ps Notes 1, 2, 5 Notes 1, 3, 5 J RXS-MEDIAN-to-MAX-JITTER 56.5 ps Notes 1, 4, 5 Notes: 1. An ideal reference clock without jitter is assumed for this specification. All Links are assumed active while generating this eye diagram. 2. Transition and non-transition bits must be distinguished in order to measure compliance against the de-emphasized voltage level (V RXS_d ). V RXS and V RXS_d are differential peak-peak output voltages. 3. T RXS is the eye width. The sample size for this measurement is 10 6 UI. This value can be reduced to 274 ps for simulation purpose at BER J RXS-MEDIAN-to-MAX-JITTER is the maximum median-to-peak jitter outlier as defined in the PCI Express Base Specification, Revision 1.1. The sample size for this measurement is 10 6 UI. This value can be increased to 63 ps for simulation purpose at BER The values in Table 5-8 are referenced to an ideal 100 Ω differential load. The resultant values, when provided to the Receiver interconnect path of the system board, allow for a demonstration of compliance of the overall system board Receiver path. The sensitivity requirements are defined and centered with respect to the jitter median. Exact conditions required for verifying compliance while generating this eye diagram are given in the PHY Electrical Test Considerations for PCI Express Architecture document. 94

95 V rx S T rx S V rx S_d Figure 5-8: Representative Composite Eye Diagram for System Board Receiver Path Compliance A

96 96 PCI EXPRESS EXPRESSMODULE ELECTROMECHANICAL SPECIFICATION, REV. 1.0

97 6 6. ExpressModule Connector The PCI Express ExpressModule card edge connectors support x1, x4, x8, and x16 Link widths to suit the different bandwidth requirements. These connectors support the PCI Express signal and power requirements, as well as auxiliary signals used to facilitate the interface between system board and add-in ExpressModule. This chapter defines the connector mating interfaces and footprints, as well as the electrical, mechanical, and environmental requirements. The primary ExpressModule interface connector is physically the same for all Link widths. The ExpressModule connector is limited to a single PCI Express Link. Multiple x1 or x4 Links are not allowed and unused PCI Express Link signal pins may not be used for any other purpose. This supports the maximum interchangeability between ExpressModule and system slots. The primary ExpressModule interface connector supports up to x8 lanes. x16 lanes are only supported in a doublewide module utilizing two connectors of x8 lanes each Connector Pin Counts Table 6-1 identifies the type and quantity of pins for a x8 backplane connector. Table 6-1: ExpressModule Connector Pin Type/Count Pin Qty Type PERp(0-7) 8 Differential Plus Input PERn(0-7) 8 Differential Minus Input PETp(0-7) 8 Differential Plus output PETn(0-7) 8 Differential Minus output Refclk (-/+) 2 Differential Inputs Gnd 40 Ground Bulk Power 6 12VDC power input VauxP 1 Resistive path to Aux input Vaux 1 3.3VDC power input SMB-Clk 1 System management clock (I/O) SMB-data 1 System management data (I/O) SMBAlert 1 SMbus Alert output MRST# 1 Module Reset input MPWRGD# 1 Module Power Good PWRFLT# 1 Power Fault output 97

98 Pin Qty Type PWREN# 1 Power Enable input PRSNT# 1 Present Detection output WAKE 1 Wake system output ATNSW 1 Attention switch output ATNLED 1 Attention LED input PWRLED 1 Power LED input RSVD 4 Reserved pins Total pins 98 Table 6-2: Optional Storage Pin Count/Type Pin Qty Type SRp(0-3) 4 Storage Differential Plus Input SRn(0-3) 4 Storage Differential Minus Input STp(0-3) 4 Storage Differential Plus output STn(0-3) 4 Storage Differential Minus output Gnd 18 Ground STOR_SB-1 1 See Table 2-3 or Table 2-4 for details STOR_SB-2 1 See Table 2-3 or Table 2-4 for details STOR_SB-3 1 See Table 2-3 or Table 2-4 for details STOR_SB-4 1 See Table 2-3 or Table 2-4 for details RSVD 2 Reserved Total Pins 40 98

99 6.2. Connector Pin Assignments Pin # Table 6-3: ExpressModule Connector Pin Assignments Side B Side A Name Description Name Description 1 +12V Bulk Supply GND Ground 2 +12V Bulk Supply GND Ground 3 +12V Bulk Supply GND Ground 4 RSVD Reserved RSVD Reserved 5 ATNSW Attention Switch PWRLED Power LED 6 MRST# Module Reset ATNLED Attention LED 7 SMDAT SM Data SMBAlert SMbus alert 8 SMCLK SM Clock PWRFLT# Power Fault * 9 +VauxP Pre AUX Power MPWRGD# Module Power Good * 10 GND Ground WAKE Signal to Wakeup * 11 +Vaux AUX Power * PRSNT# Present Detection * Key 12 PWR_EN# Power Enable * GND Ground 13 RSVD Reserved REFCLK+ Reference Clk + 14 GND Ground REFCLK- Reference Clk - 15 PETp0 Transmitter GND Ground 16 PETn0 Differential pair lane 0 GND Ground 17 GND Ground PERp0 18 GND Ground PERn0 Receive Differential pair lane 0 19 PETp1 Transmitter GND Ground 20 PETn1 Differential pair lane 1 GND Ground 21 GND Ground PERp1 22 GND Ground PERn1 Receive Differential pair lane 1 23 PETp2 Transmitter GND Ground 24 PETn2 Differential pair lane 2 GND Ground 25 GND Ground PERp2 26 GND Ground PERn2 Receive Differential pair lane 2 99

100 Pin # Side B Side A Name Description Name Description 27 PETp3 Transmitter GND Ground 28 PETn3 Differential pair lane 3 GND Ground 29 GND Ground PERp3 30 GND Ground PERn3 Receive Differential pair lane 3 31 PETp4 Transmitter GND Ground 32 PETn4 Differential pair lane 4 GND Ground 33 GND Ground PERp4 34 GND Ground PERn4 Receive Differential pair lane 4 35 PETp5 Transmitter GND Ground 36 PETn5 Differential pair lane 5 GND Ground 37 GND Ground PERp5 38 GND Ground PERn5 Receive Differential pair lane 5 39 PETp6 Transmitter GND Ground 40 PETn6 Differential pair lane 6 GND Ground 41 GND Ground PERp6 42 GND Ground PERn6 Receive Differential pair lane 6 43 PETp7 Transmitter GND Ground 44 PETn7 Differential pair lane 7 GND Ground 45 GND Ground PERp7 46 RSVD Reserved PERn V Bulk Supply GND Ground V Bulk Supply GND Ground V Bulk Supply GND Ground End of the x8 connector and Key for x8 with storage Receive Differential pair lane 7 50 GND Ground RSVD Reserved 51 SRp0 Receive Differential GND Ground 52 SRn0 pair, storage lane 0 GND Ground 53 GND Ground STp0 54 GND Ground STn0 Transmitter Differential pair storage lane 0 55 SRp1 Receive Differential GND Ground 56 SRn1 pair, storage lane 1 GND Ground 100

101 Pin # Side B Side A Name Description Name Description 57 GND Ground STp1 58 STOR_SB-1 SideBand 1 STn1 59 STOR_SB-2 SideBand 2 GND Ground Transmitter Differential pair storage lane 1 60 GND Ground STOR_SB-3 SideBand 3 61 SRp2 Receive STOR_SB-4 SideBand 4 62 SRn2 Differential pair storage lane 2 GND Ground 63 GND Ground STp1 64 GND Ground STn1 Transmitter Differential pair storage lane 2 65 SRp3 Receive GND Ground 66 SRn3 Differential pair storage lane 3 GND Ground 67 GND Ground STp3 68 GND Ground STn3 69 RSVD Reserved GND Ground Transmitter Differential pair storage lane 3 The following points should be noted: The short pins on the card connector are WAKE, PWRFLT#, VAUX, PRSNT#, MPWRGD# and PWR_EN#. These pins are denoted by an * after the name in Table 6-3. The pins are numbered as shown in Figure 6-3 in ascending order from the left to the right with side A on the top of the centerline and side B on the bottom of the centerline. The PCI Express interface pins PETp(x), PETn(x), PERp(x), and PERn(x) are named with the following convention: PE stands for PCI Express high speed, T for transmitter, R for receiver, p for positive (+), and n for negative. By default, PETp(x) and PETn(x) pins (the transmitter differential pair of the connector) shall be connected to the PCI Express transmitter differential pair on the system board and to the PCI Express receiver differential pair on the add-in card. By default, PERp(x) and PERn(x) pins (the receiver differential pair of the connector) shall be connected to the PCI Express receiver differential pair on the system board and to the PCI Express transmitter differential pair on the add-in card. However, the p and n connections may be reversed to simplify PCB trace routing and minimize vias if needed. All PCI Express receivers incorporate automatic Lane Polarity Inversion as part of the Link Initialization and Training and will correct the polarity independently on each lane. Refer to Section of the PCI Express Base Specification, Revision 1.1. If the component on the system board or ExpressModule does not support the optional PCI Express Lane Reversal functions, they must connect each transmitter and receiver lane to the ExpressModule connector lanes as shown in Table 6-3. For example, a x4 component must connect lane 0 to 0, lane 1 to 1, lane 2 to 2, and lane 3 to

102 If the component on the system board or ExpressModule supports the optional PCI Express Lane Reversal function, it may connect each transmitter and receiver lane to the ExpressModule connector lanes as shown in Table 6-3 or it may connect the transmitter and receiver lanes using a reversed lane ordering. Either lane ordering may be used to simplify PCB trace routing and minimize vias. However, the transmitting and receiving lanes must be connected with the same lane ordering. For example, a x4 component may connect lane 0 to 0, lane 1 to 1, lane 2 to 2, and lane 3 to 3, or it may connect lane 0 to 3, lane 1 to 2, lane 2 to 1, and lane 3 to 0. The system connectors and the ExpressModule are the same for all configurations. This allows full up-plugging and down-plugging between connectors and ExpressModule for all lane widths. Adjacent differential pairs are separated by two ground pins to manage the connector crosstalk. See Chapter 2 for auxiliary signals description and implementation, except the +3.3Vaux and PRSNT# pins. The requirements for +3.3Vaux are discussed in Chapter 4 and presence detect is discussed in Chapter 3. Detailed requirements on hot plug are covered in Chapter 3. Power pins (+3.3Vaux, and +12V) are defined based on the PCI Express power delivery requirements specified in Chapter 4 with the connector contact carrying capability being 1.1 A per pin. The power that goes through the connector shall not exceed the maximum power specified for a given ExpressModule size, as defined in Table 4-1. ExpressModule add-in cards shall utilize 30 microinches of gold plating over a 50-microinch Nickel underplate on all edge card contact positions. This is to protect the backplane connector from excessive wear while supporting full up- or down-plugging. ExpressModule connectors should provide a minmum 30 microinches of gold or equivalent (e.g., 30-microinch PdNi with gold flash) plating over a minimum of 50-microinch Nickel underplate in the mating area of all contacts Connector Interface Definitions The PCI Express ExpressModule connector is defined as a card edge connector similar to that defined in the PCI Express Card Electromechanical Specification, Revision 1.1, but with enhancements for lead-in and reliability. The PCI Express ExpressModule connector outline footprint and corresponding add-in edge-finger dimensions are shown in the following figures. 102

103 A NOTES: 1. BY DESIGN 2. SIZE AND LOCATION FOR DEFINED FOOTPRINT AND MEET THE FOLLOWING REQUIREMENT: 0.70 [.028] DIM "B" SEE NOTE [.394] (10 SPACES AT 1.00[.039]) REF REF B REF B [1.55] REF CONTACT INTERFACE SIZE AND LOCATION FOR DEFINED ADD-IN CARD SEE FIGURE [0.031] REF SEE NOTE 9 C [1.457] (37 SPACES AT 1.00[.039]) REF THE DIFFERENCE BETWEEN THE NOMINAL HOLE DIAMETER 0.70 [.028] AND THE NOMINAL DIAGONAL DIMENSION "B" OF THE SOLDER TAIL CROSS SECTION B.25. [.028-B.010] 3. POST DIAMETER SHOULD BE SIZED SO THAT THE MAJOR DIAMETER OF THE POST IS ENGAGED WITH THE PCB HOLE PRIOR TO THE LEADS ENTERING THE PLATED THRU HOLES. 4. PIN #A1 IDENTIFIED. 5. FREQUENCY AND LOCATION AT SUPPLIER DISCRETION. RIDGE MAY BE CONTINUOUS WITH NO BREAKS. 6. SLOT MUST ACCEPT STANDARD 1.57mm (.062)±10% THICK ADD-IN CARD mm OFFSET BETWEEN THE CENTERLINE OF THE PEG AND THE CENTERLINE OF THE CONNECTOR KEY. 8. DATUM STRUCTURE SHOWN IS IN REFERENCE TO THE SYSTEM ASSEMBLY DEFINED IN THIS SPECIFICATION. SEE SECTION DIMENSION SHOWN IS FROM CENTERLINE OF THE SLOT TO THE EDGE OF NOMINAL MODULE BOARD WHEN INSERTED INTO CONNECTOR. D 56.00±0.23 [2.205±.009] 45 TYP B D D SEE NOTE 3 LEAD-IN REQUIRED TO ASSIST IN PLACEMENT OF CONNECTOR TO PCB REF REF SECTION D-D 4.40±.20 [.173±.008] SEE NOTE ±.15 [.134±.006] D SEE NOTE 5 MAX [1.581] REF ORIENTATION POSTS SIZE & LOCATION FOR DEFINED FOOTPRINT. FULL DIAMETER NOT REQUIRED EX: A SEE NOTE 7 CENTERLINE OF KEY PCB THK. (REF) 1.57 [.062] 2.36 [.093] DIM "A" /-0.13 [ /-.005] /-0.13 [ /-.005] X8 CONNECTOR REF REF CENTERLINE OF PEG SECTION B-B Figure 6-1: ExpressModule Connector Form Factor 103

104 SEE NOTE [.394] (10 SPACES AT 1.00[.039]) REF A B REF B REF REF REF [1.457] (37 SPACES AT 1.00[.039]) REF [1.55] [.787] (20 SPACES AT 1.00[.039]) REF REF CONTACT INTERFACE SIZE AND LOCATION FOR DEFINED ADD-IN CARD SEE FIGURE 6-4 & 6-5 REF [0.031] SEE NOTE 9 C NOTES: 1. BY DESIGN 2. SIZE AND LOCATION FOR DEFINED FOOTPRINT AND MEET THE FOLLOWING REQUIREMENT: THE DIFFERENCE BETWEEN THE NOMINAL HOLE DIAMETER 0.70 [.028] AND THE NOMINAL DIAGONAL DIMENSION "B" OF THE SOLDER TAIL CROSS SECTION B.25. [.028-B.010] 0.70 [.028] DIM "B" 3. POST DIAMETER SHOULD BE SIZED SO THAT THE MAJOR DIAMETER OF THE POST IS ENGAGED WITH THE PCB HOLE PRIOR TO THE LEADS ENTERING THE PLATED THRU HOLES. 4. PIN #A1 IDENTIFIED. 5. FREQUENCY AND LOCATION AT SUPPLIER DISCRETION. RIDGE MAY BE CONTINUOUS WITH NO BREAKS. 6. SLOT MUST ACCEPT STANDARD 1.57mm (.062)±10% THICK ADD-IN CARD mm OFFSET BETWEEN THE CENTERLINE OF THE PEG AND THE CENTERLINE OF THE CONNECTOR KEY. 8. DATUM STRUCTURE SHOWN IS IN REFERENCE TO THE SYSTEM ASSEMBLY DEFINED IN THIS SPECIFICATION. SEE SECTION DIMENSION SHOWN IS FROM CENTERLINE OF SLOT TO EDGE OF NOMINAL MODULE BOARD WHEN INSERTED INTO CONNECTOR [.295] MAX REF REF D ±0.23 [3.19] [±.009] 45 TYP MAX B D D REF SEE NOTE 3 REF DIM "A" LEAD-IN REQUIRED TO ASSIST IN PLACEMENT OF CONNECTOR TO PCB REF REF SECTION D-D 4.40±.20 [.173±.008] SEE NOTE ±.15 [.134±.006] D SEE NOTE 5 MAX REF [1.581] ORIENTATION POSTS SIZE & LOCATION FOR DEFINED FOOTPRINT. FULL DIAMETER NOT REQUIRED EX: SEE NOTE 7 A CENTERLINE OF KEY PCB THK. (REF) 1.57 [.062] 2.36 [.093] DIM "A" /-0.13 [ /-.005] /-0.13 [ /-.005] X8E CONNECTOR REF CENTERLINE OF PEG REF SECTION B-B Figure 6-2: ExpressModule Connector Form Factor with Storage Extension 104

105 A SEE NOTE 1 REF REF A4 37X Ø0.70±0.08 [Ø.028±.003] Ø0.10[.004 M D Z M ZZ M A2 PIN #A1 IDENTIFIED BY SQUARE ANNULAR RING 10X 2X 2.35 REF REF A3 B3 B1 B2 B4 Ø2.35±.05 [Ø.093±.002] Z -D- PCB SURFACE 40.15±0.10 [1.581±.004]] 1.25 [0.049] 2X Ø2.35±0.05 [.093±.002] ZZ C SEE NOTE 2 RECOMMENDED FOOTPRINT FOR X8 98 POSITIONS NOTES: 1. DATUM A REFERENCES THE CONNECTOR KEY CENTERLINE LOCATION SEE SECTION DATUM C REFERENCES THE COMPONENT SURFACE OF THE NOMINAL MODULE CARD SEE SECTION 7.3. SEE NOTE 1 A REF REF 2.35 REF REF Ø0.70±0.08 [Ø.028±.003] Ø0.10[.004] M D Z M ZZ M A4 A2 PIN #A1 IDENTIFIED BY SQUARE ANNULAR RING A3 B3 10X 37X 20X 2X REF B1 B2 B4 Ø2.35±.05 [Ø.093±.002] Z -D- PCB SURFACE Ø2.35±0.05 [.093±.002] ZZ 1.25 [0.049] 2X C SEE NOTE [1.70] RECOMMENDED FOOTPRINT FOR X8E 140 POSITIONS NOTES: 1. DATUM A REFERENCES THE CONNECTOR KEY CENTERLINE LOCATION SEE SECTION DATUM C REFERENCES THE COMPONENT SURFACE OF THE NOMINAL MODULE CARD SEE SECTION 7.3. Figure 6-3: Recommended Footprints 105

106 B ±.05 [.075±.002] A B PIN B1 REF 10 SPACES AT 1.00[.039] PIN B12 TYP.50[.020]x45 TYP 37 SPACES AT 1.00[.039].10[.004] L C B A L.05[.002] L PIN "C" SEE TABLE SEE NOTE 1 C 20 ±5 SEE NOTE ± [.062 ± 0.006] ACROSS PADS 0.20 [.008] SEE NOTE 5 PRIMARY (COMPONENT SIDE) CONNECTOR LINK WIDTH # POS REF X8 98 "C" NOTE 1 A8, A9, A10, A11 B11, B12 REF PIN A11 PIN A8 NOTES: 1. NO TIE BAR PERMITTED FROM CARD EDGE TO LEADING EDGE OF PAD FOR PINS "C". 2. CHAMFER EDGES MUST BE FREE OF CUTTING BURRS. 3. DATUMS A,B,C REFERENCE THIS PART AS WELL AS THE SYSTEM ASSEMBLY DEFINED BY THIS SPECIFICATION IN SECTION TOLERANCE:.XX ±0.13 [0.005] UNLESS OTHERWISE NOTED 5. FLATNESS REQUIREMENTS APPLIES FROM EDGE OF CARD TO TOP OF KEY A1 SECONDARY (SOLDER SIDE) SEE TABLE SEE NOTE 1 Figure 6-4: Add-in Module Edge-Finger Dimensions 106

107 B [.126] REF 1.90±.05 [.075±.002] A DETAIL "A" PIN "C" 2.40 [0.94] SEE TABLE SEE NOTE 1.10[.004] L C B A L.05[.002] L 10 SPACES AT 37 SPACES AT 20 SPACES AT 1.00[.039] 1.00[.039] 1.00[.039] 0.20 [.008] SEE NOTE 5 B11 B12 B REF PIN B1 DETAIL A TYP.50[.020]X45 TYP. C 1.57 ± [.062 ± 0.006] ACROSS PADS PRIMARY (COMPONENT) SIDE A11 NOTES: A8 1. NO TIE BAR PERMITTED FROM CARD EDGE TO LEADING EDGE OF PAD FOR PINS "C". 2. CHAMFER EDGES MUST BE FREE OF CUTTING BURRS. 3. DATUMS A,B,C, REFERENCE THIS PART AS WELL AS THE SYSTEM ASSEMBLY DEFINED BY THIS SPECIFICATION IN SECTION TOLERANCE:.XX ±0.13 [0.005] UNLESS OTHERWISE NOTED 5. FLATNESS REQUIREMENTS APPLIES FROM EDGE OF CARD TO TOP OF KEY CONNECTOR LINK WIDTH # POS REF X8E 140 "C" NOTE 1 A8, A9, A10, A11 B11, B12 SECONDARY (SOLDER SIDE) A1 SEE TABLE SEE NOTE 1 Figure 6-5: Add-in Module with Storage Edge-Finger Dimensions The following points should be noted: The connector has a 1.00-mm contact pitch. The contact shall be pre-loaded, similar to the PCI connector. The connector shall have greater than 1.0 mm of wipe length. 107

108 The connector footprint (Figure 6-1 and Figure 6-2) requires two 2.35-mm diameter location holes, working with either plastic pegs/posts or metal board locks. Metal board locks are allowed, although Figure 6-1 and Figure 6-2 shows only the plastic pegs on the connector housing. Figure 6-5 defines only the mating interface related dimensions. Other ExpressModule dimensions are defined in Chapter 7. The short pins defined in the pin list and shown in Figure 6-4 are 1 mm shorter than the other fingers. No plating tie bar is allowed between the pad and leading edge of card for the short pins because these pins are meant to be last-mate and first-break. Detailed connector contact and housing designs are up to each connector vendor, as long as the requirements of form, fit, and function are met Connector Signal Integrity Requirements and Test Procedures A common electrical test fixture is specified and used for evaluating connector signal integrity. The test fixture will have 6-mil wide 50 Ω single ended traces that must be uncoupled. The impedance variation of those traces shall be controlled within ±5%. Refer to the PCI Express Connector High Speed Electrical Test Procedure for detailed discussions on the test fixture. Detailed testing procedures, such as the vector network analyzer settings, operation, and calibration, are specified in the PCI Express Connector High Speed Electrical Test Procedure. This document should be used in conjunction with the standard test fixture. For the insertion loss and return loss tests, the measurement shall include 1.2-inch long PCB traces (0.6 inches on the system board and 0.6 inches on the add-in card). Note that the edge finger pad is not counted as the ExpressModule PCB trace. It is considered to be part of the connector interface. The 1.2-inch PCB trace included in the connector measurement is a part of the trace length allowed on the system board. See Section 5.3 for a discussion of the electrical budget. Either single ended measurements that are processed to extract the differential characteristics or true differential measurements are allowed. The detailed definition and description of the test fixture and the measurement procedures are provided in the PCI Express Connector High Speed Electrical Test Procedure. An additional consideration for the connector electrical performance is the connector-to-system board and the connector-to-add-in-card launches. The connector through hole pad and anti-pad sizes shall follow good electrical design practices to minimize impedance discontinuity. On the addin card, the ground and power planes underneath the PCI Express high-speed signals (edge fingers) shall be removed. Otherwise, the edge fingers will have too much capacitance and greatly degrade connector performance. A more detailed discussion on the ExpressModule electrical design can be found in the PCI Express Connector High Speed Electrical Test Procedure. 108

109 Table 6-4 lists the electrical signal integrity parameters, requirements, and test procedures. Table 6-4: Signal Integrity Requirements and Test Procedures Parameter Procedure Requirements Insertion Loss (IL) EIA The EIA standard must be used with the following considerations: 1. The step-by-step measurement procedure is outlined in the PCI Express Connector High Speed Electrical Test Procedure document (Note 1). 2. A common test fixture for connector characterization shall be used. 3. This is a differential insertion loss requirement. Either true differential measurements must be made or post processing of the single ended measurements must be done to extract the differential characteristics of the connector. The methodology of doing so is covered in the PCI Express Connector High Speed Electrical Test Procedure document (Note 1). Return Loss (RL) EIA The EIA standard must be used with the following considerations: 1. The step-by-step measurement procedure is outlined in the PCI Express Connector High Speed Electrical Test Procedure document (Note 1). 2. A common test fixture for connector characterization shall be used. 3. This is a differential return loss requirement. Either true differential measurements must be made or post processing of the single ended measurements must be done to extract the differential characteristics of the connector. The methodology of doing so is covered in the PCI Express Connector High Speed Electrical Test Procedure document (Note 1). Intra-pair Skew Intra-pair skew must be achieved by design; measurement not required. 1 db max up to 1.25 GHz; [1.6* (F )+1] db for 1.25 GHz < F 3.75 GHz (for example, 5 db at F = 3.75 GHz) -12 db up to 1.3 GHz; -7 db up to 2 GHz; -4 db up to 3.75 GHz 5 ps maximum 109

110 Parameter Procedure Requirements Crosstalk: NEXT EIA db max up to The EIA standard must be used with the 1.25 GHz; -[32 - following considerations: 2.4*(F-1.25)] db for 1.25 GHz < F 1. The crosstalk requirement is with 3.75 GHz (for respect to all the adjacent differential example, pairs including the crosstalk from -26 db at opposite sides of the connector, as F = 3.75 GHz) illustrated in Figure 6-6. This is reflected in the measurement procedure. 2. The step-by-step measurement procedure is outlined in the PCI Express Connector High Speed Electrical Test Procedure document. 3. A common test fixture for connector characterization shall be used. 4. This is a differential crosstalk requirement between a victim differential signal pair and all of its adjacent differential signal pairs. Either true differential measurements must be made or post processing of the single ended measurements must be done to extract the differential crosstalk of the connector. The methodology of doing so is covered in the PCI Express Connector High Speed Electrical Test Procedure document (Note 1). Jitter By design; measurement not required. 10 ps maximum Notes: 1. The PCI Express Connector High Speed Electrical Test Procedure is available separately. 2. A network analyzer is required. Differential measurements require the use of a two port (or a four port) network analyzer to measure the connector. The differential parameters may be measured directly if the equipment supports True differential excitation. ( True differential excitation is the simultaneous application of a signal to one line of the pair and a 180 degree phase shifted version of the signal to the second line of the pair). If single ended measurements are made, the differential connector parameters must be derived from the single ended measurements as defined in the PCI Express Connector High Speed Electrical Test Procedure. 3. The connector is recommended to be 100 Ω differential impedance. 110

111 In Figure 6-6, pairs marked as 11-9, 5-7, 15-13, and are the adjacent pairs with respect to the victim pair Need to Terminate All Ports Victim Pair 5 7 A-0467 Figure 6-6: Illustration of Adjacent Pairs 6.5. Connector Environmental and Other Requirements Environmental Requirements Connector environmental tests shall follow EIA , Environmental Test Methodology for Assessing the Performance of Electrical Connectors and Sockets Used in Business Office Applications. The test groups/sequences and durations shall be derived from the following application requirements: Durability (mating/unmating) rating of 50 cycles Field temperature: 65 C Field life: seven years Since the connector defined in Section 6.3 has far more than mm wipe length, Test Group 6 in EIA is not required. Test Group 7 in EIA is optional since the durability cycles is 50. The temperature life test duration and the mixed flowing gas test duration values are derived from EIA based on the field temperature using simple linear interpolation. Table 6-5 lists these values. 111

112 Test Durability (preconditioning) Durability Table 6-5: Test Durations Duration/Temperature 20 cycles 50 cycles Temperature Life 168 hours at 105 C Temperature Life (preconditioning) 92 hours at 105 C Mixed Flowing Gas 10 days, unmated for 2/3 of the test duration, then mated for the remaining 1/3 of the test The low level contact resistance (LLCR) is required to be 30 mω or less, initially. Note that the contact resistance measurement points shall include the solder joint of the solder tail and the top of the contact-mating pad on the add-in card as illustrated in Figure 6-7. The resistance change, which is defined as the change in LLCR between the reading after stress and the initial reading, shall not exceed 25 mω, and the resistance change of no more than 1 percent of the contacts shall exceed 10 mω. Add-in Card PCI Express Connector Resistance Measurement Points PCB A-0468 Figure 6-7: Contact Resistance Measurement Points To be sure that the environmental tests measure the stability of the connector, it is highly desirable that the add-in cards used for testing be at the lower and upper limit of the card thickness requirement to give an indication of the stability of the connector. In any case, both the edge tab plating thickness and the card thickness shall be recorded in the environmental test report. 112

113 Mechanical Requirements Table 6-6 lists the mechanical parameters and requirements. Note that the sample size shall follow Section of EIA Table 6-6: Mechanical Test Procedures and Requirements Test Description Procedure Requirement Visual and dimensional inspections EIA Visual, dimensional, and functional per applicable quality inspection plan Insertion force EIA Measure the force necessary to mate the connector assemblies at a maximum rate of 12.5 mm (0.492 inches) per minute, using a steel gauge 1.70 mm thick with a tolerance , -.01 mm Removal force EIA Measure the force necessary to un-mate the connector assemblies at maximum rate of 12.5 mm (0.492 inches) per minute, using a steel gauge 1.44 mm thick with a tolerance +.01, mm Meets product drawing requirements 1.15 N maximum per contact pair 0.15 N minimum per contact pair 113

114 Current Rating Requirement Table 6-7 lists the contact current rating requirement and test procedure. Test Order 1 Contact current rating Table 6-7: End-of-Life Current Rating Test Sequence Test Procedure Condition Requirement EIA method 2 The sample size is a minimum of three mated connectors. The sample shall be soldered on a PC board with the appropriate footprint. Wire the six power pins (B1, B2, B3, B47, B48, and B49) and the six nearest ground pins (A1, A2, A3, A47, A48, A49, and B16) in a series circuit. The mated ExpressModule is included in this circuit. The ExpressModule shall have 1 oz. copper traces and its mating geometry shall conform to the applicable PCI Express drawings. A thermocouple of 30 AWG or less shall be placed on the card edge finger pad (pins B2 and B48) as close to the mating contact as possible. Conduct a temperature rise vs. current test. Mated 1.1 A per pin minimum. The temperature rise above ambient shall not exceed 30 C. The ambient condition is still air at 25 C Additional Considerations Table 6-8 lists the additional requirements. Table 6-8: Additional Requirements Parameter Procedure Requirement Flammability UL94V-1 minimum Material certification or certificate of compliance is required with each lot to satisfy the Underwriters Laboratories follow-up service requirements. Lead-free soldering Connector must be compatible with lead-free soldering process. Connector Color Color of the connector should be black. Exceptions will be made for color coding schemes This specification does not attempt to define the connector requirements that are considered application-specific. It is up to the users and their connector suppliers to determine if additional requirements shall be added to satisfy the application needs. The system level shock and vibration 114

115 tests are considered application-specific because results will depend on card weight and size, chassis stiffness, and retention mechanisms, as well as the connector. Therefore, those tests are not specified in the connector specification. It will be up to each system OEM to decide how the shock and vibration tests shall be done. 115

116 116 PCI EXPRESS EXPRESSMODULE ELECTROMECHANICAL SPECIFICATION, REV. 1.0

117 7 7. Module Mechanical Specification 7.1. Mechanical Overview The module mechanical specification defines the module and chassis dimensional parameters that are required for module and systems manufacturers to be able to design their components such that mechanical interoperability is achieved (see Figure 7-1). Only critical design parameters are described. This allows for the manufacturer to optimize their design for their functional needs and manufacturing processes. The module form factors supported are singlewide and doublewide. The singlewide module provides 26.4 square inches of primary surface for component placement. The doublewide module card area is dependent on how the design is implemented. Daughter cards are permissible within the limits of module volume. The modules are installed into the chassis system slots without requiring covers to be removed. The system slots are designed to allow the insertion of single or doublewide modules. Note that doublewide modules are not supported in 1U systems. Critical system slot requirements for thermal design, EMI, and module retention are included in this chapter. Internal system storage is supported by a dedicated slot. 117

118 Figure 7-1: Chassis Assembly 7.2. Dimensions and Tolerances All dimensions included in this chapter are in millimeters and shall have a general tolerance of ±0.25 mm unless otherwise specified by the drawings. 118

119 7.3. System Datum Plane Definition The sections of the mechanical specification are based on detailed dimensioned drawings which all use a common set of references (datum planes). Refer to Figure 7-4 and Figure These datum planes are defined in the following table in reference to a vertical module orientation. Table 7-1: Datum Table Datum Name A B C D Definition Horizontal plane, perpendicular to both the board and backplane, which passes through the center of the key on the backplane connector Vertical plane, coincident with the end of the card edge connector and parallel with the system backplane, that represents the mated surface with the backplane connector Vertical plane, coincident with the primary component surface of the module circuit board Vertical plane, coincident with the primary component surface of the system backplane in which the module plugs 7.4. Module Description The module consists of a base housing to which the electronics card is mounted. Refer to Figure 7-2. At one extreme is the card edge that plugs into the system s connector. At the opposite end are the connectors that allow the attachment of cables from peripherals outside of the system. The area where the I/O connectors escape is called the I/O plate. Attached to the side of the I/O plate is a lever that is used for ejection, extraction, and module retention. Adjacent to the lever is an EMI gasket. See Section for gasket details. The material adjacent to the I/O plate provides the surfaces for EMI containment. A cover protects the internal components of the module. If there are replaceable components within the module, the cover must be removable without the use of tools. Examples are batteries, memory upgrades, and daughter cards. Described in the following sections are the critical parameters that define the singlewide and the doublewide modules. 119

120 Cover Card-Edge Connector EMI Gasket Module Board Module Carrier Button Auxiliary Venting Ejection Lever Primary Venting A-0469 Figure 7-2: Exploded View of Singlewide Module Module Materials The module materials are not defined by this specification with the exception of the surfaces that contact the chassis and EMI and ESD contact surfaces (guide rails, ejector, EMI gasket). These surfaces shall be corrosion free and galvanically compatible to zinc finishes including any EMI gasket material chosen. The conductivity of the contact surfaces shall not exceed 1 mω using the test equipment and methodology defined in Section All module parts shall be free of burrs and sharp edges. It is strongly recommended that the materials selected for construction of the module comply with applicable hazardous materials restrictions as defined by the countries in which the product is marketed. There are a number of materials that modules could be formed from that will meet this specification s requirements. One common material in the industry today is precoated steel. Precoated steel can provide EMI containment, grounding, and structural support for the electronics card within the module. Systems are typically produced from precoated zinc steels or are post plated with nickel plate. The module can be formed out of sheet steel that is either preplated with electro galvanic zinc or hot dipped zinc. If zinc coating is chosen, it should be 20 to 25 g/m 2 on each side and should be finished with an antifinger print treatment, AFP, for corrosion resistance. Due to the industry s switch away from hex-chrome based AFP treatments, the corrosion resistance and the 120

121 surface conductivity of some of these sheet metals may not have sufficient performance to effectively meet this specification s requirements. Therefore, a minimum corrosion and electrical conductance are defined. The tests for corrosion and electrical conductivity are detailed below. The same test panel may be used for both tests provided that the conductivity test is performed before the corrosion test. If the sample fails the conductivity test, there is no point in running the corrosion test Conductivity Measurement Procedure The surface conductivity of the base sheet metal needs to be measured using a four point Loresta- EP CP-T360 meter equipped with type BSP probes. The maximum allowable impedance of the surface shall not exceed 1 mω. The Loresta BSP probes are gold plated for low resistance and spring loaded to give approximately 210 grams of force when fully depressed. The tips are rounded with a 0.37 mm radius. A mechanical fixture can be designed to hold the two probes at a constant separation with full deflection of the probe tips to achieve maximum force. A separation between probes of 2.5 cm minimum was found to be satisfactory for the purpose of evaluating the AFP coating. Current Source Voltage Resistance Surface Resistance Internal Resistance Figure 7-3: Simple Circuit Representation of AFP Steel Using Four-point Resistance Method The following procedure is used to measure AFP material conductivity: 1. Draw a 5 x 5 cm grid on the metal surfaces, front and back. 2. Gently wipe the surfaces using a soft dry towel or cloth to remove surface contaminants. DO NOT USE ABRASIVES, CHEMICAL CLEANERS, or SOLVENTS. 3. Place the probe tips in the approximate center of each grid cell with the probe tips separated by a minimum of 2.5 cm and fully depressed. 4. Allow the meter to stabilize. 5. Record the resistance in each cell. A

122 Measurements may be repeated up to three times if there are inconsistencies or unexplained variations observed. The sample passes when all of the cells have attained a reading of no more than 1 mω Corrosion Measurement Procedure The following defines an acceptable test for zinc precoated steel; other surface coatings must be tested to an equivalent corrosion level. If zinc precoated sheet steel is used for the module to chassis contact surfaces, it shall be able to withstand a minimum of 72 hours of salt spray testing per specification ASTM B117. After a rinse of the sample, white rust should not cover more than 5% of the total area. White rust is the corrosion of the zinc coating and will precede the onset of red rust (corrosion of base steel). The ASTM specification details the specific control parameters for chamber operation and sample orientation so that the corrosion rate is controlled. When testing precoated steel samples, the edges are sealed (either with a plastic tape or some other type of barrier coating such as epoxy or silicone) so that there is no exposed base material which would rust prematurely Singlewide and Doublewide Module Form Factor Figure 7-4 provides the critical dimensions for both single and doublewide modules. Figure 7-5 and Figure 7-6 show section views with dimensional relationships and component heights for the singlewide and doublewide modules. There are instances where a double wide module exists that contains two cards that are expected to mate with adjacent ExpressModule back plane connectors. In order to engage both of the connector s contacts at very nearly the same moment, the individual card s datum B positions, card edges, must be located within a window of 0.4 mm from each other. Refer to Figure 7-6. Doublewide modules that engage two adjacent backplane connectors from datum C to CC, must implement a board mounting scheme that accommodates the associated tolerances of mounting, module boards, and backplane connectors. While tolerances are dependent on the individual design, card mounting must be flexible enough to accommodate a window of 0.5 mm in order to provide adequate mating. 122

123 Figure 7-4: Critical Dimensions 123

124 B TOP SIDE CLEARANCE BOARD LENGTH C ( 21 ) ( 1.6 ) 2.7 BOTTOM SIDE CLEARANCE LEAD-IN RECOMMENDED ( 1 ) GAP 5.1 Figure 7-5: Singlewide Cross Section B TOP SIDE CLEARANCE BOARD LENGTH CC 22 1 C ( 43 ) ( ) ( 1 ) 2.7 LEAD-IN RECOMMENDED GAP BOTTOM SIDE CLEARANCE MATING TWO ADJACENT BACKPLANE CONNECTORS REQUIRES THAT THE BOARD-TO-BOARD MOUNTING ACCOMODATE ASSOCIATED TOLERANCES Figure 7-6: Doublewide Cross Section Module Raw Card The raw card drawing in Figure 7-7 shows the maximum dimensions for the ExpressModule adapter card. The dotted lines indicates that portion of the card that extends into the slot guide feature of the module enclosure. Refer to the section AA view of Figure 7-4 for an example of a card that extends into the slot guide feature. Note that cards that do extend into the slot guide feature must observe a component height restriction in those areas inside the feature. Cards that do not extend into the slot guide feature do not have this restriction and are subject only to the enclosure component height limit. 124

125 Figure 7-7: Module Raw Card Reference Drawing Module Ejector and Latch Details For the purpose of describing the module ejector function, this description assumes that the module is vertical in the chassis. The ejector functions the same for singlewide and doublewide modules. The ejection lever is attached to the module base housing by a pressed in pivot pin. Refer to Figure 7-4 and Figure 7-8. The lever rotates from an open position where it is stopped by a protrusion from the module base housing. In the open position, the lever is aligned such that the module can be slid into the system. A tab (1) on the lever stops the module at the proper displacement to start rotating the lever and start the engagement of the card edge into the system s connector. A second tab (2) on the lever is aligned with a slot in the bottom of the system. When the lever is at 45 degrees, the second tab on the lever engages the edge of the slot and starts to apply sufficient advantage to overcome the insertion forces of the system connector. When the lever is in the vertical orientation, the module is fully inserted into the system. A catch is positioned on the top of the module to retain the module by holding the lever at the vertical orientation. A spring loaded blue plastic button engages the catch. Recesses are provided on the button for labeling purposes. These are detailed in Section 8.4. When the module is not in the system, the lever assembly may be held in place by the button and the catch. To release the module from the system, one presses down on the blue button and rotates the lever down. 125

126 Figure 7-8: Ejector Assembly I/O Plate Details Refer to Figure 7-9 for details of the I/O plate. The I/O plate provides openings for ventilation, a connector area, a Power LED, an Attention LED, and an Attention switch. The Attention switch shall not extend through the I/O plate; this is to prevent accidental activation. The LEDs and the switch must be positioned at the locations shown. Round, square, or rectangular LEDs are acceptable. The Attention LED is yellow and the Power LED is green. Refer to Table 7-2 for details. Sufficient ventilation opening must be supplied so that the module pressure drop and flow requirements are met as detailed in the thermal section of this specification. The remainder of the area of the I/O plate is allowed for connector population. 126

127 Table 7-2: LED Color LED Color Wavelength Power Green (Green/Blue) nm Attention Yellow (Amber) nm The LEDs shall have a minimum luminescence of 80 candelas per square meter over an area of at least 1.5 mm and no more than 3.0 mm. Figure 7-9: Module I/O Plate Module Air Vent Design for EMI The module s airflow venting through the I/O plate causes special attention be required of the design of the vents so that the minimum EMI performance is supported by the I/O plate. Figure 7-10 shows an example of ventilation perforation pattern. Larger openings are possible but not recommended because waveguide technology will be required to meet minimum EMI shielding. All modules and fillers are required to meet minimum EMI shielding defined in Section

128 Figure 7-10: Module Vent Design 7.5. Filler Component or Module It is recognized that it may be necessary to fill an empty slot to maintain a system s EMI or thermal management intent. A specific design for a filler entity is not contained in this specification. This responsibility is left to the system manufacturer Chassis Slot Description The chassis slot for the PCI Express ExpressModule is defined by the mechanical interfaces that are required in order for a module to function in a system. Dimensions and details not defined here are left to be defined by the system developer. Figure 7-11 is based on the reference design. The slot consists of the following key features: Chassis guides that contain lead-ins that ensure smooth module insertion. A slot which functions with the ejection lever described in the module section. This slot is outside of the module EMI gasket contact area; proper EMI sealing techniques must be used. The ExpressModule system connector. EMI/ESD gasketing at the top, bottom, and leftmost side of the chassis slot to complete the four-sided EMI seal. An EMI gasket on the adjacent module completes the seal on a chassis with multiple slots. Chassis module guides may be formed out of plastic but must provide for a direct or resistive connection between the chassis and the module to complete the EMI management solution module grounding. An optional opening of the module guide to provide for airflow in a transverse direction. See the thermal section for greater detail. 128

129 Backplane and Chassis Slot Details Refer to Figure 7-11 for critical details for the design of the ExpressModule system slots. The slots may be oriented either horizontally or vertically at the system developer s discretion. For reference, all design details are shown in the vertical orientation. Slot connectors are on 22-mm centers (see datum CC to C in Figure 7-11). CC C 22 MODULE PITCH 9 REF CHASSIS EMI SEALS A 114 REF A A 11.5 PIN A1 REF PIN A1 REF 4.2 REF D CC C B REF REF REF EMI GASKET 10 MIN REF REF SECTION A-A Figure 7-11: Chassis Requirements 129

130 Chassis Slot EMI Design The same EMI gasket that is used on the module will be applied to the chassis in the locations shown. The gaskets must be attached to the system in a manner that would create a continuous seal around a module when the slot is filled. The gasket should be recessed in a manner that would prevent it from being sheared off when a module is being installed. See Figure 8-8 for details. 130

131 8 8. Design Considerations This specification defines a consistent and repeatable method for ensuring a minimum level of integrity and environmental performance. This section defines these details and standard test processes that all chassis, modules, and backplanes specified in this standard can use to be compliant. It is the system designer s responsibility to evaluate the relationship between the modules, chassis, and the system environment. The module s external surface plays a role in the enclosure s Cooling and EMI Containment strategy. In addition to meeting the pressure drop requirements of this section, the vent holes must also be sized to meet the EMI containment requirements of Section Cooling/Thermal Environment The design objective is to supply an adequate combination of airflow and temperature to ensure that module component temperature specifications are met. The module is designed to allow two basic types of cooling: (1) longitudinal flow, considered to be the default mode, which consists of air flowing through the length of the module passing through the external vent surface and (2) the alternate mode, lateral cooling, which consists of air flowing across the module and may or may not allow air to pass through the external vent surface. The system manufacturer will decide whether or not it is appropriate to allow air to pass through the external vent surface. If it is not acceptable, the system manufacturer shall specify the slot, slots, or entire chassis as closed vent operation and label according to Section To maintain the most flexibility between the cooling modes, module designs should incorporate multidirectional heat-sinks (like pin fin) rather than unidirectional (plate fin). Filler modules for empty slots will be left up to system manufactures and will not be detailed in this specification. In order to provide detail for the specification, a thermal test vehicle was fabricated. This test vehicle incorporated a discrete chip with a heatsink that was not fully ducted. The external module surface contained the maximum suggested connector blockage area of 975 mm. This subsequently results in the minimum suggested external venting surface area. All temperature and airflow requirements were generated from the test results from this test vehicle. 131

132 Longitudinal Cooling Default Mode When the modules are located in an enclosure in series with the enclosure s air movers such that air is pushed or pulled through the module, it is being cooled in the default mode and will use open vent operation. The module s external surface is an exhaust or intake vent. ExpressModule ExpressModule Figure 8-1: Longitudinal System Airflow A-0471A 132

133 Airflow Supplied to Module Default Mode The requirement for airflow supplied to each module is dependent upon the air temperature supplied. Each module shall receive a minimum flow rate as specified in Figure Module CFM C, Module Inlet Figure 8-2: Singlewide Module: Required CFM vs. Module Inlet Temperature, Default Mode A Module Pressure Drop Default Mode The pressure drop through the module must not be so low that it allows bypass nor so high that it does not allow the minimal flow rate. The module pressure drop shall fall in the shaded area within the curves specified in Figure 8-3. The module shall meet this pressure drop requirement with all side vent holes blocked so that all the airflow passes through the external vent surface. Doublewide modules require twice the flow rate as singlewide modules and would be represented in Figure 8-2 and Figure 8-3 by doubling the flow rates. 133

134 Figure 8-3: Singlewide Module: Required Pressure Drop Range vs. Flow Rate, Default Mode Lateral Cooling Alternate Mode Lateral cooling exists when the air is drawn through one side of the module and passes predominantly or entirely through the other side of the module. This can be accomplished in several ways and has an impact on the intended configuration of the module. When the modules are located in an enclosure in parallel with the enclosure s air movers such that drawing air through the module would cause an unintended short circuit due to the enclosure being under negative pressure (see Figure 8-4), the slot may need closed vent operation. It is the system manufacturer s responsibility to label the slot, slots, or entire chassis as in need of closed vent operation per Section No labeling is required for default longitudinal cooling or any configuration where open vent operation is designed by the system manufacturer. With a module operating in closed vent mode, external module venting must be blocked to avoid a short circuit of cooling air and introduction of re-circulated exhaust. The uses of a module in operating in closed vent mode must modify the module by adding an air re-circulation barrier (refer to Section ). 134

135 ExpressModule ExpressModule A-0474A Figure 8-4: Unintended Recirculation Air Re-circulation Barrier Closed Vent Operation In order to use a module in closed vent mode, the module should be configured to block external module vents with an air re-circulation barrier. The air re-circulation barrier may be as simple as an adhesive backed label that prohibits air flow. An air re-circulation barrier shall block a minimum of 90% of the external module vent holes. The ExpressModule supplier shall provide a re-circulator barrier with each module Airflow Supplied to Module Alternate Mode The requirement for airflow supplied to each module is dependent upon the supplied air temperature. Whether it is operating in open or closed vent mode, if the slot has a lateral cooling direction, each module shall receive a minimum flow rate as specified in Figure

136 Module CFM C, Module Inlet Figure 8-5: Singlewide Module: Required CFM vs. Module Inlet Temperature, Alternate Mode A Series Modules In a lateral cooling orientation, it is possible that modules may be installed such that they are cooled serially. In this case, the same airflow passes through the first, is preheated, and then passes through one or more subsequent modules. The flow rate through these modules shall meet the requirements for the worst case module inlet temperature, presumably the last module in series Module Pressure Drop Alternate Mode The pressure drop through the module must not be so low that it allows bypass nor so high that it does not allow the minimal flow rate. The module pressure drop shall fall in the shaded area within the curves specified in Figure 8-6. The module shall meet this pressure drop requirement with the external vent surface and the connector end of the module blocked so that all the airflow passes through the side module vent surfaces. 136

137 Inches H2O Figure 8-6: Singlewide Module: Required Pressure Drop Range vs. Flow rate, Alternate Mode Doublewide modules require twice the flow rate as singlewide modules and would be represented in Figure 8-5 and Figure 8-6 by doubling the flow rates. CFM A Cooling Consideration for Storage Extension Slot Slots that support the optional storage extension are a special case for cooling. The storage extension blocks most of the area reserved for airflow in the default mode. To make the description easer to understand, the airflow will be defined as entering the module from the backplane and exiting from the modules I/O plate with the module in the vertical orientation. The storage extension slot is limited to one per backplane segment in this configuration. The storage extension slot must be placed at the end of the backplane segment such that backplane does not extend beyond the backplane connector with the storage extension; see Figure 8-7. This creates an air-inlet to the module of about 10 mm wide from bottom to the top of the module. The long air-inlet provides the necessary cooling for the storage extension slot. 137

138 STORAGE EXTENSION CONNECTOR Figure 8-7: Storage Slot Venting Module EMI Design The module design must consider three areas of EMI containment. The first is the module to the system chassis, the second is the thermal air vents, and the third is any connector in the I/O plate. The EMI management of modules and fillers shall meet the EMI curve below independent of any agency requirements for the product s market. Figure 8-8: EMI Gasket Profile 138

139 Due to the nature of how the module and chassis slot fit together, the module is an integral portion of the chassis EMI containment strategy. The five-sided box that forms the modules I/O plate is also part of the chassis EMI containment and, therefore, must be defined such that it can contain system levels of EMI (not just card generated EMI). The module s EMI contacts mate with the chassis on at least two sides and to other modules or the chassis on the other two sides forming a 360-degree EMI contact around each module to contain the chassis and module s EMI. The chassis shall provide an EMI gasket on the top and bottom surfaces of the slot that mate with the module per Figure The module shall make a near continuous (no gap greater than 4 mm) EMI seal with the chassis and surrounding modules around its perimeter. The module shall provide an EMI gasket located per Figure 7-4. The gasket material shall be used with a nominal compression of up to 30% assuming it is recessed 1 mm into the chassis and compressed in a nominal gap of 1 mm. The gasket material shall be such that sliding contact causes no degradation. The gasket material shall have a finish that is galvanically compatible with zinc. Because the module also contains system EMI, the five-sided box that composes the module s I/O plate and EMI interface must provide a known level of EMI containment. The attenuation levels of Figure 8-9 shall be met by all modules regardless of the relative level of internal module emissions. 45 Attenuation (db) ,000 2, , ,000 10,000 Frequency (MHz) A-0477 Figure 8-9: Required Module I/O Plate Attenuation The implementation of the connection between signal ground and chassis ground is the responsibility of the system integrator. That connection should be made at a point that minimizes EMI. 139

140 Although not recommended, it is permissible to connect signal ground and chassis ground together on an ExpressModule. If such connection(s) must be made, and because a logic to frame ground connection has a greater potential for causing EMI problems if done improperly, modules making this connection shall be EMI tested with all supported I/O cables attached while exercising all I/O interfaces using typical customer data and the module shall continue to meet the ESD specifications Module ESD Design The ESD performance of the module shall meet the requirements of EN55024 following the procedures specified in IEC This standard requires that the module provide sufficient shielding and grounding to meet ESD discharges of the appropriate category. For general purpose computing equipment, a level 2 test of 4 kv contact and 8 kv indirect of both polarities is required. For network equipment building systems (NEBS), a level 4 test of 8 kv contact and 15 kv indirect of both polarities is required. The module guide surface shall be conductive to provide an ESD discharge path to the slot during insertion Module Interoperability PCI Express module and slots can exist with a variety of Link widths. The interoperability of module and slots is summarized in the following table: Table 8-1: x8 Slot Connector Interoperability Slot Module x8 Connector with x1 Wired x8 Connector with x2 Wired x8 Connector with x4 Wired x8 Connector with x8 Wired x1 Yes Yes Yes Yes x2 At x1 only Yes Optional Optional x4 At x1 only x1 Yes Yes x8 At x1 only x1 x4 and x1 Yes Note: x16 slots are composed of two adjacent singlewide x8 slots that have the ability to be combined into a single x16 slot. This is an optional configuration for both modules and chassis. x16 slots will be identified by markings defined in Section A x16 module must support operation at x8 width and be clearly labeled as x16 width Slot/Module Color Coding and Labeling Since the ExpressModule is designed to be installed from the outside of a system, the usual indicators that tell the user the system attributes are not visible. The user is literally blind to what s inside the system. For example in a PCI system you can tell a 32-bit slot from a 64-bit slot by the size of the system s connector and match the appropriate bandwidth of the card by the size of its 140

141 card edge connector. The goal of this labeling section is to ensure that enough information is contained on the module and at the system s slot to allow a user to easily match the module to an appropriate slot. A sans serif font is recommended for use on the labels because its characters are most easily recognizable. Required Module Label Optional Module Label Figure 8-10: Module Latch Labeling Locations A mm 17.5mm A B C D Figure 8-11: Required Module Label with Examples Required: A label on the top surface of the latch which has four fields: A, B, C, and D. Note that the total label space available is 16.5 mm x 17.5 mm. The image is purely illustrative and not to scale. The size of the fields may be determined by the needs of the manufacturer. Refer to Figure Field A should show the highest bandwidth supported by the PCI Express module; for example, x8. It is recommended that the text be 14 point and have a 10:1 contrast ratio with the background. It is recommended that the background be black with white text. Field B should list the interoperable bandwidths that are supported in descending order; for example, x4, x2, x1. It is recommended that text be 9 point and have a 10:1 contrast ratio with the background. It is recognized that, on occasion, a smaller font may be needed to fit in the required information. The / are shown at 10 point. It is recommended that the background be black with white text. 141