. CMS/DAQ overview. Environment. FRL-Slink (Front-End Readout Link) - Boards - Features - Protocol with NIC & results - Production.FMM (Fast Monitoring Module) -Requirements -Implementation -Features -Production.Conclusions
Custom Hardware SLINK64: FED -> DAQ FRL (incl x4 extension): FEDBuilder input LVDS cables FMM : stts network, atts hardware interface GTPe (GTP emulator): for comissioning and testing Trigger Distributor: Control module in Compact PCI crate
CompactPCI backplane -3 Busses 32bit@33MHz -One slot system -up 20 users cards -Geographical address CompactPCI backplane up to 21 cards PC-CPCI link up to 13 meters I/O I/O LVDS LVDS I/O LVDS Memory Memory Internal bus PCI Interface PCI Interface PC (linux) One PC with a PC-CPCI link -No address translation -card is seen as it was inside the PC -CPCI/PC throughput up to 132 MB/s (measure 70 MB/s with controls) Memory PCI Interface
Protocol compatibility SLINK64 is a protocol: Handshake to write data Rules for backpressure LinkDown protocol Reset Protocol Test Link SLINK data format integrity the SLINK header and trailer follow the rules of the game. Relevant for the DAQ are : SLINK header Marker field 0x5 First event is No.1 2^24 events wrap to 0 SLINK trailer Marker field 0xA Event length in 64bit words including header and trailer 16 bit CRC to detect transmission problems Unused bits are left at 0
Three frequency to send data over SLINK cable (40, 50 and 60 MHz) Auto-check link JTAG FED writes data to SLINK using a FIFO protocol SLink64 protocol Altera ACEX LVDS LVDS Flow Data Backpresure Commands LVDS cables: 11.5 meters 10 meters 8 meters 6.5 meters - FED can write data up to 100 MHz - Data is sent to LVDS cable @ 50 MHz (400 MB/s) ; data multiplex on cable - The CRC Event is computed on board and compared with the CRC provided by FED. A bit on the Event trailer is set in case of error. - The CMC measures the frequency used by FED to send data and accumulates the backpressure time. These informations are sent between events regularly. - Commands received from FRL : >Test-LINK: a pattern generator sends data on LVDS link; at the FRL side, data are checked. The version is sent during the test. >Link_DOWN: FRL informs FED that the DAQ is not ready to receive data >DC-Balance & DESKEW mode: the LVDS components configuration can be setup by FRL
NIC Myrinet IN_1 Commercial card 64kB PCI connector 64-bit 64b @ 66 or 100MHz JTAG connector IN_2 64kB FRL Function PCI 64b @ 66 (528 MB/s) IN_3 64kB Memory 4Mbytes Bridge Compact PCI 32-bit 33MHz IN_4 64kB 64b@100MHz Compact PCI Back-plane -2 LVDS links @ 400MB/s -64 Kbytes buffer for each link -Independent internal PCI bus (64b@66MHz) -A slot PCI for NIC card -4MB 64@100MHz to spy, histogram -One to internal logic ; one to interface with CPCI
NIC Myrinet IN_1 Commercial card 64kB PCI connector 64-bit 64b @ 66 or 100MHz JTAG connector IN_2 64kB FRL Function PCI 64b @ 66 (528 MB/s) 64kB Bridge IN_3 Memory 4Mbytes Compact PCI 32-bit 33MHz 64b@100MHz Compact PCI Back-plane 64kB IN_4-4 LVDS links @ 400MB/s -64 Kbytes buffer for each link -Independent internal PCI bus (64b@66MHz) -A slot PCI for NIC card -4MB 64b@100MHz shared access to spy, histogram -One to internal logic ; one to interface with CPCI
FEATURES 1 to 4 Slink64 Input(s) Memory Flash to keep up to 4 designs -Normal functionality -Fragments generated inside FRL -.. Auto-check link IN_1 IN_2 IN_3 IN_4 64kB 64kB 64kB 64kB NIC PCI connector 64-bit Memory -Send events to NIC card memory blocks (with FRL header).each link can be disabled.one event fragment of each link are merged in one -Spies events (All events, on CRC error, on Evt#, out_of_sync..) -Histogram :.WC distribution of event size (256 points with programmable resolution).bunch crossing -Up to four designs uploaded inside the flash memory -The user can reprogram design to EEPROM and switch between designs remotely Spy fragments : -all fragments -list of up to 1024 fragments numbers -on CRC error.. WC histogram for each links (256 pitches of programmable step:8 to 1K bytes) Bunch crossing histogram
FED# Trigger# Packet# 0 Payload size Reserved Data Page size PCI A B C D E F G H I J K Memory Page Header A,B,C K FRL header Payload Myrinet trailer Myrinet header FRL (Internal ) Myrinet (NIC) -Protocol used is push-mode (write has less overhead) -NIC board has a memory divided in memory pages (FRL_Header + data) -FRL header indicate the memory pages structure for each event -Memory page size can be adapted to optimize the throughput of the optical link through switches
Average Event fragment size -Blocs size 4KB
LVDS signals @ 350 MHz over 10 meters cable 2.8 ns -Long test duration was done without problem -One problem was discovered and solved with our preserie installation
720 Slink cards and 560 FRL cards were produced last year. Four test benched were developed to validate the production and the installation before the commissioning. - Test bench 1 (card per card) : Electrical and board configuration (EEPROM, jumpers ) - Test bench 2 (4 cards at the time): using all functionalities, the test board was done (Component soldering, short-circuit, component dead.). - Test bench 3 (16 cards at the time with Myrinet board). The real functionality. - Test bench 4 (card per card) each card+cable was tested installed underground with a USB efed module. - All results are inserted in CMS database (see next slide) Card tested success FRL 557 557 (100%) SLINK sender 717 715 (99.7%) LVDS cable 800 799 (99.8%) - All hardware is already installed at USC55 CMS. Test bench4 started
designs Test bench 1&2 Test bench 4
FRLversion Number of triggers FRL Serial number CMC version SLINK Status (backpressure, link down ) CRC error
Process (merges) the partition device states to form the detector partition status in a fast way (~100 ns) Monitors the dead time introduced by the partition devices Identification of (potential) pathologic FEDs Keeps a history memory of the state changes Allows to monitor the device states or playback for detailed analysis Generates input patterns for Trigger Control System Is also the output card for the atts
Compact PCI 6U double width form factor TTS connector allows standard RJ45 network cables At 40 MHz transition rate, LVDS drivers allows hundreds meter of cable length PCI control interface re-used from FRL design update Can be done remotly Through a JTAG connection
20 x RJ45 from lvds receivers JTAG connector FMM Function 32 bit @40 MHz Private bus Bridge 2 X RJ45 to lvds drivers Memory 2Mbytes 64b@80MHz Compact PCI 32-bit 33MHz Compact PCI Back-plane Switches (Reset-reprog )
Standard RJ45 connector is used Low cost, reliable, small footprint, high-density front panel Socket with light-guides for bi-color LEDs Pin 1 -busy Pin 2 +busy Pin 3 -ready Pin 4 +overflow warning Pin 5 -overflow warning Pin 6 + ready Pin 7 -out of synch Pin 8 +out of synch
LV1A rate Fine Ready LV1A rate too high Warning Overflow LV1A rate reduced LV1A rate still too high LV1A rate reduced Busy Killing LV1A Out of sync Resynch Error Repair Reconnect Discon- nected
States are provided on 4 bits: max transition rate = 40 MHz but we expect ~100 Hz! 6 states defined for FEDs using 7 values 9 values reserved If a FED is in any reserved state, the FMM propagates a new state: illegal FEDs linked to an FMM can be in a different state: state priorities (decreasing order) are as follows: Disconnect Error Out_of_sync Busy Overflow Illegal Ready
24 connectors with LEDs, configurable as input or output at soldering time Allows to deal with 1 or 2 partitions and enable the card to be atts output Mask register a pathologic FED will not disturb the system once detected and identified Hardware dead-time monitors early detection of potential problem Cyclic history memory: only state transitions are recorded with time tag 2 MB/128 k transitions (16 bytes/transitions) Time tag resolution/range: 25 ns/40 bit (~7.6 hours) System clock at 80 MHz, Inputs sampled at 80 MHz but processed at 40 MHz History data can be pushed directly to host PC ( ala FEDKIT) (eeprom) can be reconfigured through PCI and JTAG connector
FMM tests : 2 stage test system Two FMM was mounted/configured as tester module (stts sequencer) :20 ports configured as output Test software with direct connection to database was developed. During the implementation of the test system a lot of firmware improvements have been added. 55 out of 60 (90%) modules where successfully tested. 5 need to be repaired (small problems). (46 are needed in CMS during operation)
-Production and tests are done -Production is installed -Slink cables installation is done, FMM cables installation is on going -Test on site has started