UniBoard V1.0 Board Description

Transcription

1 V1.0 Board Description Auteur(s) / Author(s): Organisatie / Organization Datum / Date Sjouke Zwier Gijs Schoonderbeek ASTRON Controle / Checked: ASTRON Goedkeuring / Approval: ASTRON Autorisatie / Authorisation: Handtekening / Signature ASTRON ASTRON-FO / 16 ASTRON 2010 All rights are reserved. Reproduction in whole or in part is prohibited without written consent of the copyright owner.

2 Distribution list: Group: Eric Kooistra Jonathan Hargreaves Daniël van der Schuur Sjouke Zwier Gijs Schoonderbeek Others: Andre Gunst Document history: Revision Date Chapter / Page Modification / Change Creation all Update Section 3.1 / page 7 Function of SFP+ light pipes added 2 / 16

3 Table of contents: 1 Introduction Document Scope Applicable documents (AD) Reference documents (RD) Abbreviations overview Connectors GbE SFP+ interfaces Backplane interfaces Clock and Control Test IO JTAG Connector Connector Overview Jumpers JTAG Jumpers Clock Jumpers LEDs Monitoring and control FPGA and board ID Memory Ethernet Switch G interface Power Power consumption Thermal management Reset and Power-up PCB Top Components Bottom Components List of figures: Figure 1 block diagram... 6 Figure 2 SFP+ Cage... 7 Figure 3 Function of SFP+ Light Pipes... 7 Figure 4 Ethernet control connector... 8 Figure 5 JTAG jumper locations Figure 6 Location of clock selection jumpers / 16

4 Figure 7PCB build up Figure 8 top side Figure 9 bottom side List of tables: Table 1 SFP+ Cage... 7 Table 2 Backplane connector 1 (top)... 7 Table 3 Backplane connector Table 4 Test IO... 9 Table 5 Boundary Scan connector pinning... 9 Table 6 Connector overview... 9 Table 7 JTAG jumper functions Table 8 FPGA ID Table 9 Ethernet Switch Used Ports Table 10 Power Connector Table 11 PCB characteristics / 16

5 1 Introduction 1.1 Document Scope This document describes the hardware of the rev 1.0. The document starts with a functional overview of the and the systems where it can be used for. In further sections of this document the is described for a user/programmer. 1.2 Applicable documents (AD) Ref.nr. Document number Title AD.1 ASTRON-RP-303 Quinten Architectural Design AD.2 ASTRON-RP-316 Hardware Design AD.3 ASTRON-RP-381 Firmware Platform Design Document 1.3 Reference documents (RD) Ref.nr. Document number Title RD-1 BOM Partlist RD-2 SCH Schematic RD-3 PCB Rev 1.1 PCB RD-4 VMDS Datasheet VSC Gbps XAUI to XFI Transceiver. Rev 4.0 September Abbreviations ADC Analog to Digital converter AD-n n th document in the list of Applicable Documents CLK Clock CPLD Complex Programmable Logic Device FPGA Field programmable gate array GND Ground LVDS Low Voltage differential signal PPS 1 pulse per second RD-n n th document in the list of Reference Documents SFP+ Small form-factor pluggable transceiver for 10GbE XAUI 10G Attachment Unit Interface (10G Media independent Interface) 2 overview The, as the name suggests, is a universal processing platform which will be used on multiple processing application like future EVN correlator and aperitif beamformer. The board exists of 8 processing FPGAs configured in two columns, see block diagram of Figure 1. The front column contains the front nodes, these FPGA are each connected to four 10GbE SFP+ inputs making copper and optical interfacing possible. Via mesh on the board a front node is connected to each back node with a 10Gbps link. The backside or 5 / 16

6 back node FPGA are connected to a backplane. Besides four 10GbE links a back node has four 8 bits ADC inputs as well. The control of the board is done via an onboard 1GbE switch with four copper interfaces on the front panel and eight 1GbE connection one for each processing FPGA. Figure 1 block diagram 3 Connectors GbE SFP+ interfaces For data IO on the front side of the board four 2x2 SFP+ cages are placed. In Figure 2 a SFP+ cages is shown. 6 / 16

7 Figure 2 SFP+ Cage In the caged SFP+ direct attachment copper cable like the Tyco can be used but optical modules like the Finisar FTLX8571D3BCL can be used as well. In Table 1 the interfaces are shown and in Figure 3 the function of the light pipes. Table 1 SFP+ Cage Bottom Left IO nr 0 Top left IO nr 1 Top Right IO nr 2 Bottom Right IO nr Rx Tx Tx Rx Rx Tx Tx Rx Figure 3 Function of SFP+ Light Pipes 3.2 Backplane interfaces On the backplane differential connectors are place for four XAUI and four 8 bit LVDS interfaces per FPGA. In Table 2 and Table 3 the pinning of the backplane connectors is shown. Table 2 Backplane connector 1 (top) Row Column 1 (AB) Column 2 (CD) Column 3 (EF) Column 1 (GH) 1 BN_BI_2_TX_0 BN_BI_2_RX_0 ADC_A_0 ADC_A_1 2 BN_BI_2_TX_1 BN_BI_2_RX_1 ADC_A_2 ADC_A_3 3 BN_BI_3_TX_0 BN_BI_3_RX_0 ADC_A_4 ADC_A_5 4 BN_BI_3_TX_1 BN_BI_3_RX_1 ADC_A_6 ADC_A_7 5 CLK input* PPS input* Control 0 Control 1 6 BN_BI_2_TX_2 BN_BI_2_RX_2 ADC_B_0 ADC_B_1 7 / 16

8 7 BN_BI_2_TX_3 BN_BI_2_RX_3 ADC_B_2 ADC_B_3 8 BN_BI_1_TX_0 BN_BI_1_RX_0 ADC_B_4 ADC_B_5 9 BN_BI_1_TX_1 BN_BI_1_RX_1 ADC_B_6 ADC_B_7 10 ID0/ID1 * ID2/3* ADC_CLK A ADC_CLK B * only for bottom row of FPGAs/connectors Table 3 Backplane connector 2 Row Column 1 (AB) Column 2 (CD) Column 3 (EF) Column 1 (GH) 1 BN_BI_1_TX_2 BN_BI_1_RX_2 ADC_CLK C ADC_CLK D 2 BN_BI_1_TX_3 BN_BI_1_RX_3 ADC_C_0 ADC_C_1 3 BN_BI_0_TX_0 BN_BI_0_RX_0 ADC_C_2 ADC_C_3 4 BN_BI_0_TX_1 BN_BI_0_RX_1 ADC_C_4 ADC_C_5 5 ID4 / TRST* int A / int B* ADC_C_6 ADC_C_7 6 BN_BI_3_TX_2 BN_BI_3_RX_2 Control 2 Control 3 7 BN_BI_3_TX_3 BN_BI_3_RX_3 ADC_D_0 ADC_D_1 8 BN_BI_0_TX_2 BN_BI_0_RX_2 ADC_D_2 ADC_D_3 9 BN_BI_0_TX_3 BN_BI_0_RX_3 ADC_D_4 ADC_D_5 10 TMS / TCK * TDI / TDO* ADC_D_6 ADC_D_7 * only for bottom row of FPGAs/connectors 3.3 Clock and Control On the lower backplane connectors, the differential clock and PPS inputs are placed. For stand alone us, SMA connectors are placed for the clock (P12) and PPS (P11) as well. These inputs are 2.5 V compliant (min > GND, max < 2.5V) with 50 Ω termination. The clock is AC coupled but the PPS needs DC offset round 1.125V. On the front panel a 2x2 RJ45 connector has been placed. All four inputs are connected to an onboard gigabit Ethernet switch. In Figure 4 the connector is shown. More detail about the Ethernet switch can be found in section 8. Figure 4 Ethernet control connector 8 / 16

9 3.4 Test IO Every processing FPGA has 2 jumpers, four test points and one multi color LED on the front panel. In Table 4 the Test IO is summarized Table 4 Test IO FN 0 FN 1 FN 2 FN 3 BN 0 BN 1 BN 2 BN 3 Test_IO 0 J1_F0 3-4 J1_F1 1-2 J1_F2 1-2 J1_F3 3-4 J1_B0 1-2 J1_B1 1-2 J1_B2 1-2 J1_B3 1-2 Test_IO 1 J1_F0 1-2 J1_F1 3-4 J1_F2 3-4 J1_F3 1-2 J1_B0 3-4 J1_B1 3-4 J1_B2 3-4 J1_B3 3-4 Test_IO 2 D1_F0 Red D1_F1 Red D1_F2 Red D1_F3 Red D2_B0 Red D2_B1 Red D2_B2 Red D2_B3 Red Test_IO 3 D1_F0 Green D1_F1 Green D1_F2 Green D1_F3 Green D2_B0 Green D2_B1 Green D2_B2 Green D2_B3 Green Test_IO 4 T1_F0-3 T1_F1-2 T1_F2-2 T1_F3-2 T1_B0-2 T1_B1-2 T1_B2-2 T1_B3-2 Test_IO 5 T1_F0-2 T1_F1-4 T1_F2-4 T1_F3-4 T1_B0-4 T1_B1-4 T1_B2-4 T1_B3-4 Test_IO 6 T1_F0-1 T1_F1-1 T1_F2-1 T1_F3-1 T1_B0-1 T1_B1-1 T1_B2-1 T1_B3-1 Test_IO 7 T1_F0-4 T1_F1-3 T1_F2-3 T1_F3-3 T1_B0-3 T1_B1-3 T1_B2-3 T1_B JTAG Connector For Boundary scan and programming of the FPGA a connector is placed on. The pinning of this connector is compliant with Altera USB blaster. In Table 5 the pinning is shown Table 5 Boundary Scan connector pinning Pin 1 TCK Pin 2 GND Pin 3 TDO Pin 4 2V5 Pin 5 TMS Pin 6 Not Connected Pin 7 Not Connected Pin 8 Not Connected Pin 9 TDI Pin 10 GND 3.6 Connector Overview In Table 6 an overview of the connectors used on is shown. Table 6 Connector overview P1,P2,P3,P4,P5,P6,P7,P8 Backplane connector for ADC and High-Speed data P9 RJ45 2x2 Gigabit Ethernet connection P10 Power input P11 PPS input P12 Clock input P14 Test output for serial clock A P15 Test output for serial clock B P16 JTAG connector P1_B1,P1_B0,P1_F1,P1_F0,P1_B2,P1_F2, Memory sockets P1_F3,P1_B3,P2_B1,P2_B0,P2_F1,P2_F0, P2_B2,P2_F2,P2_F3,P2_B3 P1_S1,P1_S0,P1_S2,P1_S3 SFP+ cages. 9 / 16

10 4 Jumpers 4.1 JTAG Jumpers In the firmware of the Lattice CPLD (U31) the JTAG jumpers are programmed as shown in Table 7. Table 7 JTAG jumper functions J4 J6 J7 J8 J9 Function open open open open open Front node FPGAs in single loop open open open open placed Back Node FPGAs in single loop open open open placed open Top 8 10G transceivers in single loop open open open placed placed Lower 8 10G transceivers in single loop open open placed open open Ethernet switch in single loop open open placed open placed Front and Back nodes in single loop open open placed placed placed Outputs floating (used to program the CPLD) other Linker mode. All loops can be accessed with boundary scan tools J12 and J11 must be placed to pin 1-2 for normal operation (to the front of the board), by placing the jumpers to 3-4 the CPLD (U31) can be programmed. J5 and J10 are not used by the firmware in the Lattice device (U31). In Figure 5 the locations of the jumpers are shown. 4.2 Clock Jumpers Figure 5 JTAG jumper locations With J2 and J3 the clock respectively the PPS input can be selected. By placing the jumpers the inputs of P12 and P11 for clock respectively the PPS are used. In Figure 6 the location of the jumpers are shown (J2 34,118mm and J3 34,136mm) 10 / 16

11 Figure 6 Location of clock selection jumpers 5 LEDs Every FPGA has a multicolor LED on the front panel, these LEDs are controlled by the firmware (see RD-.. and RD-1), the green color is connected to Test_IO2 and the red color to Test_IO3. The bottom light pipes (closed to the PCB) the front nodes and the top light pipes for the back node, accept for the first row where it is swapped. Beside a LED on the front panel, every FPGA has a green LED nearby; this LED can be used to show that the FPGA is configured. 6 Monitoring and control Every FPGA has an independent temperature monitoring device. This device MAX1618 has an I2C interface at address b Depending on the firmware the temperature of the FPGA die can be read out. It is possible with to set the MAX1618 in a Thermostat mode. By setting a high and low threshold a temperature controlled fan can be connected to the FPGA. 6.1 FPGA and board ID The board ID is given to the board through the backplane connector. On the backplane the ID bits must be pulled low to make a 0, this means that when a board is tested on the lab outside a rack the FPGA IDs are 0xFFh till 0xF8h. Together with the FPGA an 8 bit address is given to each FPGA. Table 8 FPGA ID ID 7 ID 6 ID 5 ID 4 ID 3 ID 2 ID 1 ID 0 Board Address FPGA address The IDs are started by front node 0, than front node 1 and end with back node 3. 7 Memory On the bottom side of DDR3 SODIMM memory sockets are place. In these 204-pin sockets standard 1.5V DDR3 SDRAM SODIMM modules can be placed, like the MT16JSF51264HZ-1G4 and the 11 / 16

12 MT8JSF12864HZ-1G4. The DDR3 modules can be dual and single rank. The design with 16 address lines and 3 bank address lines is made for modules of up-to 4GByte. To read out the module information every module has its own I2C interface. The address of the module for this interface is set to 0x0h. 8 Ethernet Switch On a gigabit Ethernet switch is placed for control of the FPGA. The program of the switch is stored in a SPI EEPROM on board. This EEPROM is a pre programmed device. In Table 9 the connections to the Ethernet switch are shown, where port 0 till 7 are MII interfaces (single line 1Gbps) and port 8 till 11 are twisted pair interfaces (1GBASE-T) to the front panel 2x2 connector. Table 9 Ethernet Switch Used Ports Port 0 BN-3 Port 8 RJ45 top left Port 1 FN-3 Port 9 RJ45 top left Port 2 BN-2 Port 10 RJ45 top left Port 3 FN-2 Port 11 RJ45 bottom right Port 4 BN-1 Port 12 Not Used Port 5 FN-1 Port 13 Not Used Port 6 BN-0 Port 14 Not Used Port 7 FN-0 Port 15 Not Used With light pipes next to the connector the status of the Ethernet link is shown. 9 10G interface Between the FPGA and the SFP+ cage (the physical 10GbE interface) a Vitesse Semiconductor VSC8486 is used. Every front node has 4 identical interfaces. From the FPGA a XAUI interface (4 full duplex lanes of 3.125Gbps) is made to the transceiver. From the transceiver a 10Gbps SFI interface is made to the SFP+ cage. To control each transceiver, a MDIO serial interface is implemented. The floating port address lines have internal pull down resistors, given a base address of 0x0h. Between the transceiver and the SFP+ module an I2C interface is implemented. This can be used to read out an EEPROM inside the SFP+ module. The SFP+ has for each interface two light pipes, one connected to Tx-alarm and on to Rx-alarm. The function of these lines can be modified by the MDIO interface. More information about the VSC8486 can be found in RD-4 10 Power The power to the board is delivered through P10. Due to the isolated input stage two options can be used, plus 48V and the telecom standard minus 48 V. In Table 10 the pinning is shown. Table 10 Power Connector Pin Option 1 Option 2 1 GND -48V 2 +48V GND 3 GND -48 At the input of the board a hot swap controller is placed. This limits the inrush current during boot-up, and protects against short circuit. With the I2C interface to the hot swap controller, depended on the firmware a power cycle can be given to the board. 12 / 16

13 10.1 Power consumption The estimated power consumption of a working board is 250W. When the board is not configured the power consumption is approximately 50W Thermal management For thermal management active or passive heatsinks can be placed on the FPGAs. For active heatsinks (a heatsink with an integrated fan) the temperature monitor of every FPGA can be used to control the fan. A 0.1 connector is placed for every FPGA (J4_B1,J4_B0,J4_F1,J4_F0,J4_B2,J4_F2,J4_F3,J4_B3). The power on this connector is 5V (the 9V6 is reduced to 5V with a 4.7 V zener diode). For the passive heat control heatsinks with a thermal resistance of 2.5 1m/s airflow are placed on the FPGA. With a separate fan the airflow of 1m/s must be achieved. With this heat sink the expected temperature rise of the FPGA is 50 degrees. When the airflow is increased this temperature will decrease Reset and Power-up Every FPGA has its own reset and power up device. These devices monitor the VCC_A / VCC_PRM (power supply for the configuration of the FPGA). After the power supply has reached V the device wait for ms and releases the reset line. With a single reset button on the front panel a general reset line is pulled low, this will reset the FPGA and the gigabit Ethernet switch. The 10G transceivers (XAUI-SFI transceivers) are reset by the firmware of the FPGA. Because these lines have a pull down resistors, a reset of the FPGA will normally generate a reset of the transceiver (when the line is high in the normal operating FPGA). The monitoring device of the FPGA have a watchdog capability as well the. By toggling the WDI line a reset is prevented. The timeout period of the watchdog is 1.6 s. 11 PCB For a 14 layer stack-up has been chosen. In Figure 7 the build up is shown. 13 / 16

14 H.S GND H.S GND Signal PWR PWR Signal GND H.S GND H.S Figure 7PCB build up A B C In Table 11 information about the PCB is shown. Table 11 PCB characteristics Board size 340x366.66mm (13.38x14.44 inch) Number of components 6243 (2359 on the top and 2228 on the bottom) Number of connections Number of differential pairs 967 Number of via s Total length of traces 270m (10, inch) Board use % is used by components 14 / 16

15 11.1 Top Components Figure 8 top side 15 / 16

16 11.2 Bottom Components Figure 9 bottom side 16 / 16