Supplement to InfiniBand TM Architecture Specification Volume Release.. Annex A: Pluggable Interfaces: CATx, Copper and Optical March, 0 Copyright 0, 0 by the InfiniBand SM Trade Association All rights reserved. All trademarks and brands are the property of their respective owners. This document contains information proprietary to the InfiniBand SM Trade Association. Use or disclosure without written permission by an officer of the InfiniBand SM Trade Association is prohibited.
InfiniBand TM Architecture Release.. Pluggable Interfaces: CATx, Copper and Optical March, 0 Volume - Physical Specifications Revision LEGAL DISCLAIMER Date Table 0 Revision History 0. 0//0 Draft document for general EWG member review 0. 0//0 Revised document for EWG member review 0. 0//0 Revised document for EWG member review 0. 0/0/0 Revised document for EWG member review 0. 0//0 Revised document for EWG member review - including full compliance statements and summary 0. //0 Release candidate - added interoperability table (Table ).0 //0 release This specification provided AS IS and without any warranty of any kind, including, without limitation, any express or implied warranty of non-infringement, merchantability or fitness for a particular purpose. In no event shall IBTA or any member of IBTA be liable for any direct, indirect, special, exemplary, punitive, or consequential damages, including, without limitation, lost profits, even if advised of the possibility of such damages. 0 0 0 InfiniBand SM Trade Association Page Proprietary and Confidential
InfiniBand TM Architecture Release.. Pluggable Interfaces: CATx, Copper and Optical March, 0 Volume - Physical Specifications TABLE OF CONTENTS Annex A: Pluggable Interfaces: CATx copper and Optical... A. Introduction... A.. Glossary... A. Overview...0 A. Pluggable Interfaces... A.. x Pluggable... A.. x Pluggable (Legacy)... A.. x Pluggable-QSFP... A. Use of x Pluggable or Pluggable-QSFP for x and x interfaces... A.. Pin Definitions and Lane Reversal Restrictions... A. Summary of Compliance Requirements... 0 0 0 InfiniBand SM Trade Association Page Proprietary and Confidential
InfiniBand TM Architecture Release.. Pluggable Interfaces: CATx, Copper and Optical March, 0 Volume - Physical Specifications 0 0 0 InfiniBand SM Trade Association Page Proprietary and Confidential
InfiniBand TM Architecture Release.. Pluggable Interfaces: CATx, Copper and Optical March, 0 Volume - Physical Specifications LIST OF FIGURES Figure x Pluggable-QSFP Module... Figure x Pluggable-QSFP Cage Assembly... Figure x port using twh x Pluggable devices... Figure x port using three x Pluggable devices... Figure Example of incorrect cabling of an x port using two x Pluggable devices.. 0 0 0 InfiniBand SM Trade Association Page Proprietary and Confidential
InfiniBand TM Architecture Release.. Pluggable Interfaces: CATx, Copper and Optical March, 0 Volume - Physical Specifications 0 0 0 InfiniBand SM Trade Association Page Proprietary and Confidential
InfiniBand TM Architecture Release.. Pluggable Interfaces: CATx, Copper and Optical March, 0 Volume - Physical Specifications LIST OF TABLES Table 0 Revision History... Table Pin Assignment for x Pluggable-QSFP Interface... Table Signal connections between x-pluggable-qsfp and x board connector... 0 0 0 InfiniBand SM Trade Association Page Proprietary and Confidential
InfiniBand TM Architecture Release.. Pluggable Interfaces: CATx, Copper and Optical March, 0 Volume - Physical Specifications 0 0 0 InfiniBand SM Trade Association Page Proprietary and Confidential
InfiniBand TM Architecture Release.. Pluggable Interfaces: CATx, Copper and Optical March, 0 Volume - Physical Specifications ANNEX A: PLUGGABLE INTERFACES: CATX, COPPER AND OPTICAL A. INTRODUCTION A.. GLOSSARY Loss of Signal (LOS) QSFP Squelch This annex extends the definition for pluggable interfaces, which were defined in Volume, Section., along with definition for port type in Section.. Motivations for extension of the pluggable interfaces include better support for InfiniBand links over twisted-pair cabling (e.g., CAT S/FTP, CAT UTP or enhanced CAT), better configurability between copper cabling and optical cabling, denser connector density than is provided with standard copper connectors, and incorporation of definitions for more modern pluggable devices, and more types of pluggable devices, than were included in Volume, Release... A pluggable interface port is designed to be used with either copper or optical devices. Pluggable devices are designed to support transmission over cable. A pluggable device may either be part of the cable assembly (i.e., pigtailed), or may be detachable from the cable assembly. Pluggable devices will typically incorporate active transceiver components in the pluggable device, but may also operate purely passively. To allow this annex to be clear as a stand-alone document, some material is incorporated from Volume. To further clarify potential uses of the material included in this document, including options for providing InfiniBand links over twisted pair cabling and active optical or copper cables, a separate whitepaper on pluggable interfaces is published, available at http://www.infinibandta.org. The following terms are supplied in addition to the terms in Volume Chapter : Glossary. Lack of an input signal of strength sufficient for unambiguous detection Quad Small Form-factor Pluggable. A pluggable device in a small formfactor that provides high-speed differential signal pairs per direction A circuit function that suppresses the circuit s output during input LOS 0 0 0 InfiniBand SM Trade Association Page Proprietary and Confidential
InfiniBand TM Architecture Release.. Pluggable Interfaces: CATx, Copper and Optical March, 0 Volume - Physical Specifications A. OVERVIEW Due to the need to account for the jitter introduced in the assembly between the board and the cable, a particular port type, referred to as port type, is defined for these devices. The unique electrical requirements for port type are listed in Section.. In the case of the copper pluggable devices, the transceiver may be part of the cable assembly that is mated with the port connector on the board. The active devices may contain circuitry to compensate for the jitter introduced in the cable assembly. The active optical device external interface is defined in Section.0. The passive copper device is permanently attached to the cable, so the port type interface to the board is the only separable interface. This Annex describes the following pluggable interfaces: x interfaces: "x Pluggable" - The interface connector used on InfiniBand boards for x pluggable copper and optical ports at SDR and DDR speeds is the same one defined for Small Formfactor Pluggables, as specified in SFF-0 for Fibre Channel applications. This interface supports x InfiniBand electrical interfaces at SDR and DDR speeds. "x Pluggable-QDR" - For operation at QDR speeds, there is no currently defined pluggable interface. However, if a QDR pluggable interface is required, it is expected that the "SFP+" formfactor will be used. SFP+ is defined as an SFP-compatible formfactor with advanced electronic interfaces for faster speeds. This pluggable interface is specified in SFF-, with reference to SFF- (Improved Pluggable Formfactor (IPF)) for mechanical interfaces, as well as to other specifications as described in the specification (ftp://ftp.seagate.com/sff/sff-.pdf). x interfaces "x Pluggable" - The x Pluggable interface is based in part on the XPAK MSA. The mechanical specifications are identical to XPAK, while the High Speed electrical and optical interface specifications are based on the InfiniBand specifications listed in the InfiniBand Architecture Specification, Volume, Chapters and. This interface supports x InfiniBand electrical interfaces at SDR speeds. The physical size of the module is mm wide x mm deep. "x Pluggable-QSFP" - The x Pluggable-QSFP interface is based on the QSFP MSA, as described at http://www.qsfpmsa.org. This interface supports the x InfiniBand electrical interfaces at SDR 0 0 0 InfiniBand SM Trade Association Page 0 Proprietary and Confidential
InfiniBand TM Architecture Release.. Pluggable Interfaces: CATx, Copper and Optical March, 0 Volume - Physical Specifications A. PLUGGABLE INTERFACES A.. X PLUGGABLE A... DESCRIPTION speeds, and may support operation at DDR and QDR speeds. The physical size of the module is.mm wide x.mm high x mm deep. x and x interfaces There is currently no definition for x or x pluggable interfaces. These may be defined in the future. In the meantime, x and x links are supported using multiple x pluggable interface modules, as described in Section A., Use of x Pluggable or Pluggable-QSFP for x and x interfaces, on page. More detailed descriptions for each of these formfactors are shown below. The interface connector used on InfiniBand boards for x Pluggable copper and optical ports is the same one defined for Small Formfactor Pluggables ("SFP"), as specified in SFF-0 for Fibre Channel applications. A... PHYSICAL, MECHANICAL, AND ELECTRICAL REQUIREMENTS A.. X PLUGGABLE (LEGACY) Implementation Note The x Pluggable-QSFP interface definition has several significant advantages over the (XPAK-based) x Pluggable interface definition included in Volume, Release.. and earlier releases. It is physically much smaller, requires fewer signal pins, provides better electrical performance and provides enhanced cooling capabilities relative to the original x Pluggable interface. Therefore, implementation of the x Pluggable-QSFP is highly encouraged over the x Pluggable interface. Detailed description of the x Pluggable interface is included in Volume, Release.., Section.. x Pluggable interface. The x Pluggable interface is designed to allow a copper cable or optical transceiver to be attached to an InfiniBand HCA, TCA or Switch. The x Pluggable interface supports Active Copper, Passive Copper, and Optical (x-sx and x-lx) type devices. The x Pluggable is based in part on the XPAK MSA. The mechanical specifications are identical to XPAK, while the high speed electrical and optical interface specifications are based on the InfiniBand specifications listed in the InfiniBand Architecture Specification, Volume, Chapter and. 0 0 0 InfiniBand SM Trade Association Page Proprietary and Confidential
InfiniBand TM Architecture Release.. Pluggable Interfaces: CATx, Copper and Optical March, 0 Volume - Physical Specifications A.. X PLUGGABLE-QSFP Detailed description of the x Pluggable interface is included in Volume, Release.., Section.. x Pluggable interface and Volume, Release.., Section.. x Optical Pluggable Devices. The x Pluggable-QSFP interface provides similar capabilities as the original x Pluggable interface, but with a smaller formfactor and improved electrical performance. The x Pluggable-QSFP is based in part on the QSFP MSA which is described at http://www.qsfpmsa.org/. The mechanical specifications are identical to the QSFP MSA, while the electrical interface specifications are based on the InfiniBand specifications listed in the InfiniBand Architecture Specification, Volume, Chapter and. A... MECHANICAL REQUIREMENTS The diagram in Figure on page shows the mechanical outline of the device, and Figure on page shows the cage assembly. CA-.: x Pluggable-QSFP interface shall meet or exceed the physical and mechanical performance specifications of the official QSFP MSA documentation. Figure x Pluggable-QSFP Module 0 0 0 InfiniBand SM Trade Association Page Proprietary and Confidential
InfiniBand TM Architecture Release.. Pluggable Interfaces: CATx, Copper and Optical March, 0 Volume - Physical Specifications A... PIN ASSIGNMENTS Figure x Pluggable-QSFP Cage Assembly Specific module dimensions and host PCB layout are described in the QSFP documentation. CA-.: The pin assignment shown in Table shall be used for the for the x Pluggable-QSFP interface. CA-.: The pin assignment shown in Table shall be used for links connecting the x Pluggable-QSFP interface with previously-specified x board connectors and x active board connectors. 0 0 0 InfiniBand SM Trade Association Page Proprietary and Confidential
InfiniBand TM Architecture Release.. Pluggable Interfaces: CATx, Copper and Optical March, 0 Volume - Physical Specifications Table Pin Assignment for x Pluggable-QSFP Interface Top side Bottom Side Pin No Name Pin Length Pin Length Name GND Card Edge GND TXn TXn TXp TXp GND GND TXn Txn TXp Txp GND GND Reserved ModSelL 0 Vcc LPMode_Reset VccTx VccRx 0 IntL SCL ModPrsL SDA GND GND RXp Rxp RXn Rxn GND GND RXp Rxp RXn RXn GND GND. The operation of the low-speed lines (ModSelL, LPMode_Reset_ModPrsL, IntL, SCL, and SDA) for control and status is described in the QSFP MSA documentation.. Variable contact lengths on the card edge allow for hot plug capability and ensure that power and ground connections are not made or broken while high-speed differential connections are active. Actual contact lengths are described in the QSFP specification. Pin No 0 0 0 InfiniBand SM Trade Association Page Proprietary and Confidential
InfiniBand TM Architecture Release.. Pluggable Interfaces: CATx, Copper and Optical March, 0 Volume - Physical Specifications A... REGISTER SET Table Signal connections between x-pluggable-qsfp and x board connector Pin Number G-G x InfiniBand Signal Signal Ground QSFP Signal Name The register set is as described in the QSFP MSA documentation. A... HIGH SPEED SIGNALING CA-.: The high speed signaling interface to the x Pluggable-QSFP shall comply with the requirements listed in Section A... QSFP QSFP Pin Number <--> GND,,,,,,,,,,, S IBtxIp(0) <-- p TXp S IBtxIn(0) <-- n TXn S IBtxIp() <-- p TXp S IBtxIn() <-- n TXn S IBtxIp() <-- p TXp S IBtxIn() <-- n TXn S IBtxIp() <-- p TXp S IBtxIn() <-- n TXn S IBtxOn() --> n RXn S0 IBtxOp() --> p RXp S IBtxOn() --> n RXn S IBtxOp() --> p RXp S IBtxOn() --> n RXn S IBtxOp() --> p RXp S IBtxOn(0) --> n RXn S IBtxOp(0) --> p RXp Housing Chassis Ground Housing Chassis ground (case common). For the x active board connector, G=Sense-V, G=Sense-.V, and G=Vcc. These signals are not carried through to the other end of the link, since each end of the cable is powered separately.. Chassis ground is isolated from Signal Ground (GND) in the cable and connector, on both x board connector and QSFP ends, to allow equipment designer flexibility regarding connections between EMI shields and circuit ground. 0 0 0 InfiniBand SM Trade Association Page Proprietary and Confidential
InfiniBand TM Architecture Release.. Pluggable Interfaces: CATx, Copper and Optical March, 0 Volume - Physical Specifications With the exception of squelch behavior, the high-speed signaling for Tx and Rx (pluggable module data inputs from and outputs to the InfiniBand port, respectively) are as described in Volume, Release.., Section. Differential Driver Outputs and Volume, Release.., Section. Differential Receiver Inputs. For QSFP modules, Rx Squelch, for loss of input signal from the link (Rx LOS), is required. In the event that the signal on any channel becomes equal to or less than the level required to assert Rx LOS (Receiver Loss of Signal), the receiver data output for that channel shall be squelched or disabled. In the squelched or disabled state, output impedance levels shall be maintained while the differential voltage swing shall be less than 0 mv-pp. Rx squelch may optionally be disabled. For QSFP modules, Tx Squelch, for loss of input signal from the Infini- Band port, is an optional function. Where implemented it shall function as follows. In the event of the differential, peak-to-peak electrical signal on any channel becomes equal to or less than 0 mv-pp, then the transmitter optical output for that channel shall be squelched or disabled and the associated Tx LOS flag set. In InfiniBand links, the use of Rx Squelch and Tx Squelch may interfere with the correct operation of Beacon signaling. As described in Volume, Release.., Section... Polling States and Section... Sleeping States, the beaconing sequence is a periodic repeating pattern, with an active TS transmission period of ms and a quiescent period of 00 ms. Since Tx and Rx Squelch, if enabled, will activate during the 00 ms quiescent period, the time from resumption of Rx input and Tx output signals until normal signaling condition is reached must be well under ms. To meet this condition, the Rx Squelch Deassert Time (Time from resumption of Rx input signals until normal Rx output condition is reached) shall be less than 0 μs (as specified in the QSFP MSA specification). The Tx Squelch Deassert time, if Tx Squelch is enabled, shall be less than 00 μs. (Note that the QSFP MSA specification specifies a much longer Tx Squelch Deassert time, 00 ms, so Tx Squelch may generally be disabled). A... OPTICAL X PLUGGABLE-QSFP DEVICES CA-.: x Optical Pluggable-QSFP devices that have de-mateable optical connectors shall comply with requirements in their respective x optical distance classification (SX or LX) in the InfiniBand Architecture 0 0 0 InfiniBand SM Trade Association Page Proprietary and Confidential
InfiniBand TM Architecture Release.. Pluggable Interfaces: CATx, Copper and Optical March, 0 Volume - Physical Specifications Specification, Volume, Chapter : Fiber Attachment -. Gb/s,.0 Gb/s, & 0 Gb/s at the de-mating points. Implementation Note Note that while optical pluggable devices with de-mateable interfaces must comply with standard optical link requirements as described in the architecture, there is no such requirement on pluggable electrical devices or on optical active cables that interface with the pluggable device interface, which may use any appropriate signaling format. A. USE OF X PLUGGABLE OR PLUGGABLE-QSFP FOR X AND X INTERFACES CA-.: The use of x Pluggable-QSFP for x or x links shall comply with the requirements listed in Section A.. As described in Volume, Release.., Section..0 x to -x Copper Cables, devices may have the capability to configure two or three separate x ports to operate as an x or x port. Similarly, it is possible for two or three separate x pluggable devices to be configured to operate as the equivalent of an x or x device. This capability to combine multiple x pluggable devices to operate as an x or x link is subject to the constraints and requirements listed in this section. A.. PIN DEFINITIONS AND LANE REVERSAL RESTRICTIONS As shown in Figure on page, two x Pluggable-QSFP devices operating as a single x link shall be oriented such that lanes 0- on each end are interconnected by one pair of pluggable devices, and lanes - on each end are interconnected by a different pair of pluggable devices. 0 0 0 InfiniBand SM Trade Association Page Proprietary and Confidential
InfiniBand TM Architecture Release.. Pluggable Interfaces: CATx, Copper and Optical March, 0 Volume - Physical Specifications x Port 0 x Mappings 0 0 0 0 x Mappings Figure x port using twh x Pluggable devices As shown in Figure on page three x Pluggable-QSFP devices operating as a single x link shall be oriented such that lanes 0- on each end are interconnected by one pluggable device pair, lanes - on each end are interconnected by a different pluggable device pair, and lanes - on each end are interconnected by the third pluggable device pair. 0 x Port 0 0 0 InfiniBand SM Trade Association Page Proprietary and Confidential
InfiniBand TM Architecture Release.. Pluggable Interfaces: CATx, Copper and Optical March, 0 Volume - Physical Specifications x Port 0 0 x Mappings 0 0 0 When multiple x devices are used to construct an x or x link, it is important that the devices be plugged with lanes correctly connected, both within each x link and across x links, as shown in Figure on page and Figure on page. There are several incorrect ways of connecting multiple x devices that result in the reversal of subsets of lanes. One example of an incorrect configuration for an x link is shown in Figure on page. Particular care must be taken with x links constructed from three x device pairs because of the potential for five improper configurations of pairs. The lane reversal function described in Volume, Release.., Section... Lane Reversal Correction (Optional) can not correct for these situations since it operates across the full width of the link. 0 0 0 x Mappings Figure x port using three x Pluggable devices 0 0 x Port 0 0 0 InfiniBand SM Trade Association Page Proprietary and Confidential
InfiniBand TM Architecture Release.. Pluggable Interfaces: CATx, Copper and Optical March, 0 Volume - Physical Specifications x Port 0 A... SKEW BUDGET x Mappings 0 0 INCORRECT Figure Example of incorrect cabling of an x port using two x Pluggable devices Combinations of x Pluggable device pairs operating as x or x links shall have matched transmission times such that the aggregate bit-to-bit skew requirements, as defined in Volume, Release.., Section. Bit to Bit Skew x Optical Pluggable Devices, are met. Retiming repeaters may be incorporated into the pluggable devices, but no retiming repeaters shall be inserted between the transmitter or receiver and the pluggable interfaces. Implementation Note 0 0 x Mappings The full amount of bit-to-bit skew allowed at the receiver is 0 UI (,, or ns at SDR, DDR, and QDR, respectively). Of this,. ns of skew must be reserved to allow for 00 ps of bit-to-bit skew in the Driving Transmitter (S DBtB ) and 00 ps of Adapter board skew (S ABtB ) at each end, so the maximum bit-to-bit skew between sets of pluggable devices is limited to. ns, 0. ns, or. ns at SDR, DDR, and QDR rates, respectively. This budget must be allocated, in a vendor-dependent manner, between skew in the pluggable devices, potentially including embedded retiming repeaters, and skew in signal propagation through the cable itself. 0 x Port 0 0 0 InfiniBand SM Trade Association Page Proprietary and Confidential
InfiniBand TM Architecture Release.. Pluggable Interfaces: CATx, Copper and Optical March, 0 Volume - Physical Specifications A... LINK TRAINING As described in Volume, Release.., Section Link Initialization and Training, a link may operate with only a subset of the lanes connected. For example, if some lanes of a x port are disconnected, the port may come up as a x or x link, with the other lanes disabled. This feature has implications for the case of multiple x Pluggable devices configured to construct an x or x link. If the ports on both ends of an x or x link are active (i.e., their Link Training State Machines are in the Polling state) while the x pluggable devices are being plugged in, the ports will generally traverse the full initialization and training sequence after the first x pluggable device pair is connected, coming up as a x link. Plugging in the second or third device pairs may not force retraining, and the link may continue to operate with only the first pair active as a x link. In order to ensure that all lanes of an x or x link constructed using x pluggable devices are used, the link must be trained or retrained after all devices are connected and operational. Since aggregation of multiple x links into a wider link is unspecified, the methods for accomplishing this retraining are implemented in a vendor-dependent manner. A. SUMMARY OF COMPLIANCE REQUIREMENTS In order to claim compliance for the support of Pluggable Interfaces as described in this annex, a product shall meet all requirements specified in this section. CA-.:x Pluggable-QSFP, Physical Requirements......... Page CA-.:x Pluggable-QSFP, Pin Assignments.............. Page CA-.:QSFP-to-x Conversion Link, Signal Connections..... Page CA-.:x Pluggable-QSFP, General Signaling Interface...... Page CA-.:Optical x Pluggable-QSFP Devices, general......... Page CA-.:x Pluggable-QSFP, Use for x & x links.......... Page End of Document 0 0 0 InfiniBand SM Trade Association Page Proprietary and Confidential
InfiniBand TM Architecture Release.. Pluggable Interfaces: CATx, Copper and Optical March, 0 Volume - Physical Specifications 0 0 0 InfiniBand SM Trade Association Page Proprietary and Confidential