HCAL DCC Technical Reference E. Hazen - Revised March 27, 2007 Note: Latest version of this document should be available at:

Transcription

1 HCAL DCC Technical Reference E. Hazen - Revised March 27, 2007 Note: Latest version of this document should be available at: Table of Contents Overview...1 VME Motherboard...3 VME Interface...3 PCI Buses... 4 Motherboard JTAG Programming... 5 Motherboard Connectors...5 PC MIP Sites... 5 PMC Sites... 5 Other Custom Connectors...6 Power Supplies and Fuses... 6 Local Control and Clocks...7 Link Receiver Boards...7 DCC Logic Board... 8 Power Supplies...8 Programmable Logic Devices... 9 TTCrx, SLink, TTS and LEDs... 9 JTAG... 9 TTC/TTS Jumper Board LEDs References...13 Overview This document describes the hardware of the HCAL DCC for CMS. It is intended as a reference for troubleshooting and repair. For a functional description of the DCC, see the firmware specification [1]. Complete schematics and PCB layout files for all components are available at the counting house repository [2]. The DCC is built of a VME motherboard with several daughter-boards installed. The VME motherboard provides a VME to PCI bridge, and several daughter-board sites. The daughter-board sites are interconnected with 3 PCI buses, which are bridged to the VME bus so that any daughterboard may be accessed directly from VME. In addition power, clocks and JTAG signals are distributed to the daughter-board sites. The photo below illustrates the components installed on a typical DCC.

2 SLink64 Board DCC Logic Board TTCrx Board TTC/TTS Jumper Board LRB (5 typ.) Illustration 1: Photograph of DCC Assembly LED Cable The block diagram below shows a schematic of the same elements

3 TTS/TTC Jumper LRB 0 (MIP1) LRB 1 (MIP2) SLink64 PCI1 TTCrx Universe 2 VME/PCI Bridge TTL Buffers VME LRB 2 (MIP1) PCI Bus 1 Xilinx Local Control JTAG Flash LRB 3 (MIP4) LRB 4 (MIP5) LRB 5 (MIP6) Bridge 1 PCI2 RAM DCC Logic Board PMC Site (spare) Bridge 0 PCI3 PCI PCI Bus 3 64 bit 33 MHz PCI Bus 2 32 bit 33 MHz Illustration 2: DCC Block Diagram VME Motherboard The VME motherboard [3] provides a platform on which the DCC is built. It has sites to support a variety of daughter-boards, with power, clock and PCI bus access provided. The VME motherboard was originally developed for the Silicon Track Trigger (STT) project of the D0 experiment, and so contains various features which are not used for CMS. These will be mentioned only briefly in this document. VME Interface The VME interface is provided by a Tundra Universe 2 VME/PCI bridge [4]. This is a commercial chip which translates VME bus transactions into PCI bus transactions (or vice-versa). The VME interface is strictly compatible only with the base VME64 standard, though some features of the VME64x standard (slot-based geographical addressing, dynamic base address setting) are provided. The operation of the Universe 2 is controlled by a set of registers which are accessible as a VME slave device. These registers occupy 4 Kbytes of space and respond to D32 VME cycles only. At power-up

4 the Universe 2 VMEbus configuration and status registers are mapped to an A24 address related to the VMEbus slot number as follows: 0x (slot-2) * 0x80000 Software will then configure the Universe 2 to provide four ranges of VME address space which provide windows mapping VME cycles to PCI bus cycles. The four ranges provide access to the four types of PCI accesses supported: configuration, single-word memory, block-transfer memory, and I/O. PCI Buses Three PCI buses are provided on the motherboard to interconnect the daughter-boards. Three buses are used to make efficient use of bandwidth and to overcome trace length limitations for a single bus. The buses are bridged together as shown in the block diagram, so that any device on any PCI bus may be accessed directly from VME. The PCI bus device number assignments are given in the table below. These bus/device numbers are used only during initial software configuration. Note that the external names for the buses differ from the software bus numbers for historical reasons. Bus No Dev. No Logical Name Notes PCI br3 Bridge between buses 0 and log3 DCC logic board LOG3 PCI target 0 2 uv2 Universe 2 PCI configuration 0 3 bc Unused PMC site (future Ethernet?) 0 4 lc Motherboard local control PCI target PCI br2 Bridge between buses 1 and log2 DCC logic board LOG3 PCI target 1 2 mip4 LRB#3 1 3 mip5 LRB#4 1 4 mip6 LRB#5 2 0 not used PCI log1 DCC logic board LOG1 PCI target 2 2 mip1 not used 2 3 mip2 LRB#1 2 4 mip3 LRB#2

5 Motherboard JTAG Programming The motherboard programmable logic devices may be reprogrammed using a JTAG cable connected to P4. The pin-out is the standard one for an Altera Byte-Blaster cable: 1 TCK 2 GND 3 TDO 4 Vcc 5 TMS 6 n.c. 7 n.c. 8 n.c. 9 TDI 10 GND The JTAG chain and file names for default files is shown below: Due to the fact that the chain contains a mix of Xilinx, Altera and other parts, the procedure to program the motherboard from scratch is quite complex. Please see the on-line documentation for JTAG reprogramming instructions, and the necessary files. Motherboard Connectors The location of connectors on the motherboard are shown in the figure below. The logical names for the PCI devices are shown in blue. The black triangles mark the location of pin 1 on each connector. PC MIP Sites The sites MIP1..MIP6 comply with the PC MIP standard [ANSI/VITA ][5]. The connector name suffixes (P1, P2, P3) correspond to the connector names in the standard. These sites support 32 bit PCI operation at up to 33MHz. The P1 and P2 connectors carry the PCI bus signals and power, while the P3 connector is user-defined. For the CMS DCC, three pins (16, 17, 19) are used to provide an out-of-band data available signal to LOG1/2. PMC Sites The LOG1, LOG2, LOG3 and spare BC sites all comply with the PCI Mezzanine Card (PMC) standard [IEEE ][6]. In principle four PMC boards could be mounted on the motherboard. In the CMS DCC, one large logic board occupies the LOG1, LOG2, LOG3 sites. LOG1 and LOG2 implement 32 bit 33MHz PCI, while LOG3 and BC implement 64 bit 33MHz PCI. The connector name suffixes (J1,

6 J2, J3, J4) correspond to the connector names in the PMC standard. Other Custom Connectors The LOGIC_J5, LOGIC_J6, LOGIC_J7 and LOGIC_J8 connectors are not used for CMS. Illustration 3: Motherboard Connector Locations Power Supplies and Fuses The motherboard requires 3.3V and 5.0V power from VME. In addition, 2.5V is generated by a voltage regulator (U27) on the board. +12V and -12V are provided on the motherboard but are not used by the CMS DCC. The following fuses are installed next to J0 on the motherboard: Reference Rating Voltage F1 20A +5V

7 Reference Rating Voltage F2 2A +12V F3 2A -12V F4 10A +3.3v An additional fuse (F5) protects a 5V output to a transition module not used for CMS. The 2.5V power generated on the motherboard powers only U24 (local control FPGA). It can be tested only by removing the DCC logic board and measuring voltage across C169 or C170 in the right-center of the motherboard. The 3.3V and 5.0V are best observed by the front panel LEDs. 3.3V can be tested across C1 on the logic board (upper left corner). 5.0V is not easy to access on an assembled DCC, but can be measured across D6 on the rear of the motherboard below P2. Local Control and Clocks A dedicated PCI target device known as the local control is implemented in an Altera EP1K30FC256 FPGA (U24) on the VME motherboard. It's only function on the CMS DCC is to provide JTAG access to the MIP sites where the LRBs are installed, thus permitting the in-situ downloading of new firmware to the LRBs. A Xilinx XC9536 CPLD (U26) configures U24 at power-up from a parallel flash memory. Reprogramming this flash memory can be accomplished through the local control PCI device, permitting in-situ updating of the local control firmware. This does present a bootstrap problem; when the flash memory does not contain valid firmware the local control will not function. In this situation (as with a new motherboard) a JTAG programming cable must be used to configure U24 with initial firmware before the flash memory can be programmed. An additional Xilinx XC9536 CPLD (U17) provides clock distribution to various devices on the motherboard. The fundamental clock source for the motherboard is a MHz crystal oscillator. Most devices on the motherboard are provided with a MHz clock derived from this oscillator. All 3 PCI buses operate from this MHz clock. Link Receiver Boards Each Link Receiver Board (LRB) [7] receives 3 LVDS serial data streams using National Semiconductor DS90CR286. Buffering is provided on a 1 MByte SSRAM. All logic is contained on an Altera FPGA. The three input links each include a clock; data is buffered on the FPGA in a FIFO which also serves to cross to the domain of the system clock from the PCI bus (32 MHz). Additional buffering of 256 MByte/channel is provided by an external Micron MT55L256L36PT RAM. The RAM is operated at a cycle rate of 128 MHz with a clock derived from the PCI clock by a Cypress W133-3 zero delay clock buffer/multiplier. A voltage regulator (U9) provides 2.5V power for the FPGA. The 2.5V may be tested on pin 5 of U9.

8 Illustration 4: Link Receiver Board Block Diagram The FPGA configuration is stored in Altera EPC2 flash memory device. The EPC2 may be programmed using JTAG from VME via the motherboard local control device (using the LRBProg utility), or alternatively by using the 10 pin (2mm) connector J6. This connector follows the pinout of the Altera Byte Blaster programming cable (though the 2mm connector may require an adapter). Note: on some LRBs the programming connector is mounted on the wrong side of the PCB (the side away from the motherboard). This causes the pinout of J6 to be reversed, so a special adapter which swaps even and odd pins must be use for programming. DCC Logic Board The logic board is a large PCB which contains the main DCC operating logic. It contains 3 PCI bus interfaces, flash memory to support the configuration of its FPGAs and interfaces for the TTC, TTS and SLink I/O required by CMS. Power Supplies The DCC logic board uses 3.3V power from the motherboard, and contains three voltage regulators to generate 1.5V (U15) and 2.5V (U16) for the logic, plus 2.5V for SLink (U7). Convenient test points for the power supplies are as follows: 1.5V C229 (upper right corner) 2.5V (main) C219 (lower right corner) 2.5V (Slink) C205 (upper left corner) 3.3V C202 (upper left corner) - input power from Motherboard. The voltages used by the various devices are as follows: U1 1.5V, 2.5V, 3.3V U2-U4 2.5V, 3.3V Flash 3.3V SDRAM 2.5V Misc. logic 3.3V

9 Programmable Logic Devices The logic board contains 5 programmable logic devices which function as follows: CPLD (U18) This device configures the 4 FPGAs from a 2 MByte flash memory (U17). The flash memory is divided into 5 sectors, one for each of the FPGAs plus an alternate sector for PCI3 which can be used to hold a known good configuration. In addition, the CPLD contains logic to allow the flash memory to be reprogrammed from VME via PCI3. PCI3 (U4) This device implements a 64-bit PCI target interface. It provides the main interface between the DCC logic contained in U1 and the VME bus. In addition, it provides logic to permit reprogramming of the flash memory via the CPLD. PCI 1/2 (U2, U3) These devices implement 32-bit PCI master/target interfaces. During normal data-taking, they function as PCI masters. When data is available in the LRBs, a PCI burst transfer read is initiated by PCI1 or PCI2 and the data is transferred over an on-board 16-bit bus to U1. There is a simple PCI target interface also for configuration purposes. Xilinx (U1) This device contains the main operating logic of the DCC. It contains many registers which are accessed from VME via PCI3. It receives event fragments from PCI1 and PCI2, buffers them in the SDRAM (U14) and builds events. If SLink output is enabled, data is transmitted to the SLink daughter-board which is plugged into the logic board. Triggers are received from the TTCrx daughterboard and processed in U1. An I2C interface for configuration of the TTCrx is provided. TTS outputs are driven directly from U1 via an LVDS buffer chip (U13). TTCrx, SLink, TTS and LEDs A TTCrx daughter-board (produced by University of Maryland) mounts directly on the logic board (J1 and J2). Most TTCrx signals are connected directly to U1. The TTS output signals are driven from U1 through U13. The TTCrx and TTS signals are carried to the front panel by means of a jumper PCB connected to J4. The LEDs are driven by a simple serial interface connected to U1 via P2. JTAG The logic board has it's own JTAG chain, used only for initial programming of the board, or in case the flash memory or CPLD become corrupted or erased. JTAG access is provided via P1, a 10-pin header which is compatible with an Altera Byte-Blaster programming cable. For reference, the pin assigments are: 1-TCK, 3-TDO, 5-TMS, 9-TDI, 2-GND, 4-3.3V, 10-Sense. The 'Sense' input must be grounded on the programming cable. It operates a multiplexer, which switches control of the JTAG chain from the motherboard to the cable. In order from TDI to TDO, the devices in the chain are: 1. U4 EP1K30F256 PCI 3 2. U18 XC9572XL-TQ100 Flash memory control 3. U1 XC2V2000-BG525 Main DCC logic 4. U3 EP1K30F256 PCI 2 5. U2 EP1K30F256 PCI 1

10 Programming the board from scratch is a rather complex procedure described in detail in a separate online document [REF??]. Briefly, one must follow this procedure: Program CPLD on logic board using Xilinx Impact Program PCI1, PCI2, PCI3 with temporary firmware using Altera MAX+plus II Program flash sectors for PCI1, PCI2, PCI3, Xilinx with latest firmware via VMEbus TTS output (LVDS) TTC encoded signal in TTCrx SLink SLink out via transmitter board PCI1 PCI1 Altera EP1K30 FPGA (U2) Xilinx XC2V2000 FPGA (U1) PCI3 Altera EP1K30 (U4) PCI3 PCI2 PCI2 Altera EP1K30 FPGA (U3) DDR SDRAM 2 Mbyte Flash Memory 2 Mbyte Xilinx XC9572 CPLD (U18) Configure all FPGAs Illustration 5: Logic Board Block Diagram TTC/TTS Jumper Board A simple PCB [8] carries the TTC and TTS signals from the front panel to the logic board. The PCB layout is shown below, along with a table enumerating the pinout of the RJ-45 connectors and connector to the daughterboard.

11 Illustration 6: TTC/TTS Jumper PCB Layout CMC Pin RJ-45 Pin Description CMC Pin RJ-45 Pin Description 1 A-8 80MHz clock in A-7 80MHz clock in GND 6 - GND 7 A-6 40MHz clock in A-3 40MHz clock in GND 12 - GND 13 A-4 BC0 in A-5 BC0 in GND 18 - GND 19 A-2 Encoded TTC in A-1 Encoded TTC in GND 24 - GND GND B-8 T1 (OUT_OF_SYNC) out B-7 T1 (OUT_OF_SYNC) out GND 44 - GND 45 B-6 T3 (READY) out B-3 T3 (READY) out GND 50 - GND 51 B-4 T0 (OVERFLOW) out B-5 T0 (OVERFLOW) out

12 55 - GND 56 - GND 57 B-2 T2 (BUSY) out B-1 T2 (BUSY) out GND 62 - GND LEDs The DCC has many LEDs visible on the front panel. At the extreme top of the panel are four LEDs driven by motherboard logic. The right two should blink slowly (~1Hz) in a chase pattern whenever the motherboard is powered. If they do not blink, it indicates that the CPLD or local control FPGA are not configured, or the clock circuit is not operating correctly. These LEDs are mounted on the motherboard. The remainder of the DCC LEDs are mounted on a dedicated PCB [9] mounted behind the front panel and connected to the DCC logic board by a 10 pin ribbon cable. At the bottom of the panel are two LEDs which indicate the presence of 3.3V and 5V input from VME. If these LEDs are not illuminated it indicates that VME power is absent, or a motherboard fuse is blown. The remaining 24 LEDs are controlled using a two-wire serial interface driven from the DCC logic board. At power-up the 3.3V and 5V LEDs should illuminate. The 24 status LEDs power up in an undefined state (often all illuminated) and then the DCC logic firmware generates a 'chase' test where each LED illuminates in turn for about 500ms. After this test completes (about 10 seconds) the LEDs display the current DCC status as described in the firmware specification [1].

13 References Bibliography [1] CMS Data Concentrator Card Operational Specification, E. Hazen, 2006 [2] CERN Counting House Document Repository, [3] VME 9Ux400mm Motherboard Documentation Page, [4] Universe II User Manual, [5] VITA 29 - PC-MIP Specification, MIP_20Spec_20r92b.pdf [6] Draft Standard Physical and EnvironmentalLayers for PCI Mezzanine Cards:PMC, [7] LRB and LTB V2 Documentation, [8] Documentation for TTC/aTTS Jumper Board, [9] DCC LED Status Display PCB,