The Global Trigger Emulator System for the CMS experiment K. Zachariadou a1, T. Geralis 1, S. Kyriazopoulou 1, C. Markou 1, I. Michailakis 1 Abstract--We present the development of the Global Trigger Emulator System (egtp) of the CMS Data Acquisition System (DAQ). The egtp generates Level-1 triggers and exchanges information with other DAQ components. The egtp is based on a multi-purpose PCI board (Generic PCI Platform III) built around an FPGA, hosted in a PC. The board is programmed to function as a trigger summary pseudo-data source with partitioning capabilities. Data transmission is achieved via the SLINK-64 protocol. The egtp aims at decoupling the Leve1-1 trigger system from the readout system, for testing, installation and maintenance purposes. I I. INTRODUCTION n the CMS Data Acquisition System, for every beam crossing (25ns), the Global Trigger Processor (GTP) calculates up to 128 different trigger conditions and combines them to a Level-1 Accept (L1A) signal. The L1A signal is sent via the Timing, Trigger and Control (TTC) optical network to all the Front-End Drivers of the sub-detectors and to the Data Acquisition Event Manager (EVM) (Fig. 1). Furthermore, it controls the delivery of L1A signals according to feedback signals through the Trigger Throttling System (atts, stts). At the same time it guarantees that the sequence of L1A complies with a set of trigger rules of the general form No more than a certain number of L1A triggers within a given time interval, in order to minimize the sub-detector s buffers overflow probability. Detailed description of the Global Trigger can be found in [1]. The egtp (Global Trigger Processor emulator) system emulates most of the GTP s functions. Its implementation is driven by the necessity of testing the performance of the DAQ components during development (Preseries) as well as testing the performance of the DAQ system in run conditions. (The egtp will be part of the final system running in parallel with the GTP system). Moreover, during the DAQ system installation in the surface, the GTP emulator will be used to decouple the Level1 system (underground area) from the readout system (surface) (Fig. 1). a Corresponding author, e-mail: Katerina.Zachariadou@cern.ch, (telephone: +30-210-6503536) 1 K. Zachariadou is with the Institute of Nuclear Physics, NCSR Demokritos, GR-15310 Ag. Paraskevi Attiki, Greece and the University of Aegean, Fostini 31, GR-82100- Chios, Greece 1 T. Geralis is with the Institute of Nuclear Physics, NCSR Demokritos, GR- 15310 Ag. Paraskevi Attiki, Greece (telephone: +30-210-6503536, e- mail: Theodoros.Geralis@cern.ch) 1 S. Kyriazopoulou is with the Institute of Nuclear Physics, NCSR Demokritos, GR-15310 Ag. Paraskevi Attiki, Greece (telephone: +30-210- 6503526, e-mail: Sophia.Kyriazopoulou@cern.ch) 1 C. Markou is with the Institute of Nuclear Physics, NCSR Demokritos, GR- 15310 Ag. Paraskevi Attiki, Greece (telephone: +30-210-6503509, e- mail: christos.markou@inp.demokritos.gr) 1 I. Michailakis is with the Institute of Nuclear Physics, NCSR Demokritos, GR-15310 Ag. Paraskevi Attiki, Greece (telephone: +30-210-9622947, e- mail: imich@otenet.gr) Fig. 1. The Global Trigger Emulator System( egtp) in the CMS DAQ. II. THE GTP EMULATOR SYSTEM A. Overview The egtp system performs the following functions: a) random generation of L1A triggers for each partition. These are generated at non-empty beam crossings and user defined frequency in the range of 10Hz to 123kHz. (Moreover, the delivery of L1A triggers complies with a set of trigger rules imposed by the sub-detectors of each partition.) L1A triggers
are sent via a distribution system used to broadcast signals to the FRLs, b) partitioning (there are eight DAQ partitions and the number of sub-detector partitions in each DAQ partition is limited to eight [2]), c) emulation of the LHC proton beam structure, d) generation of trigger summary pseudo-data, encapsulated in the FED Common Data Format [2], to be sent to the FED Builder and e) receipt of feedback signals from DAQ partitions and from detector partitions (stts-fmm). 3564 [( 72 b 3 + 30 e] 2 + [( 72 b 4 + 31 e] + bunches = [( 72 b 3 + 30 e] 3 + 81 where b denotes full bunch crossings and e, empty ones. 3 + () 1 B. The egtp hardware and configuration code The egtp system is based on a generic, multi-purpose PCI card, (Generic PCI Platform III-GIII) [3] developed within the CMS DAQ group. The GIII board is a PCI 64b@66MHz board featuring a single FPGA (APEX with 400k gates, from Altera), 1MB flash RAM, a 32MB SDRAM, a set of user s connectors plus connectors compliant with the S-LINK64 (64b@66MHz)[4] pin-out. The GIII board is plugged into a PC (64b@66MHz) running Linux OS. The egtp is controlled via dedicated user s interface processes (LabView virtual instruments, or C-programs). For development purposes, data are transferred, via the SLINK-64 protocol, into a second GIII board (receiver) and are retrieved by dedicated readout software. The general FPGA configuration is shown in Fig. 2. The code for the PCI controller, the SLINK-64 controller and the JTAG has been developed using Quartus 2.2 (Altera) design software [5], whereas the code for the GTP emulation has been developed using DK1 1.1 (Celoxica) design software [6]. Fig. 2. FPGA configuration schematic. C. The egtp design The main functional blocks of the egtp design are shown in Fig. 3. To emulate the LHC bunch crossing frequency a quartz (40MHz) on the GIII board is used. The BX_gen module emulates the LHC proton beam structure. The total length of each orbit is 3564 bunch crossing intervals (89µs). The proton bunches are grouped in 39 trains, 72 bunches each. At the end of the orbit there is a period of 119 missing bunches (3µs long). This structure is represented by the formula: Fig. 3. The egtp functional block diagram. The beam structure signal (BX) is fed to the Level-1 generator module (L1_Gen). For each partition the L1_Gen module generates eight random LlA signals by implementing 22-bits long random number generators. L1A signals are generated only at non-empty bunch crossings and at frequencies defined by the user (10Hz to 123kHz). The partition selection and the associated rates for each partition are set via the PCI. The L1_Gen module associates the following data, according to the trigger summary block [2]: a) a bunch crossing number (12b) counting the number of bunch crossings within each LHC orbit, b) an event number (24b) counting the number of L1A signals per DAQ partition, c) a trigger number (24b) counting all the L1A signals generated since the last egtp reset, d) a DAQ partition number (3b), indicating the DAQ partition that gives the L1A signal, e) an orbit number (32b) counting the number of orbits since the last egtp reset. The trigger summary block data are formatted with the common FED Data encapsulation format [2] as seven 65-bits words. Given the fact that the assertion of Level-1 triggers must follow a set of trigger rules in order to minimize the subdetector s buffers overflow probability, different trigger rules are implemented in the L1_Gen module code for each subdetector of the DAQ partitions. The different trigger rules for the sub-detectors are listed in Table I.
TABLE I TRIGGER RULES [6] Detectors Trigger rules All sub-detectors At least one empty BX between subsequent triggers Pixel, Tracker, Muons At least 2 empty BXs between CSC subsequent triggers HCAL No more than 22 triggers per orbit Pixel Empty orbit after a period of one second Preshower No more than 1+13 triggers over the service time of 5.4 s The L1_Gen module receives feedback signals (interpreted as inhibit of all Level-1 triggers) from: a) atts_busy (x8) (DAQ-partitions, FRL, etc) b) stts_busy (x8) (sub-detector partitions) c) S-LINK64 controller (LFF). The LFF signal is LOW (backpressure) when the receiver is not running or cannot accept more data and d) the event buffer (see below), in case that the local FIFO has less than 16 empty words. The Write_Evm module receives the data fragments (seven 65-bit words in parallel) and temporarily stores them in a cyclic event buffer of two events length. The buffering is necessary to serialize the output data stream and to avoid dead time when successive triggers occur. If the local FIFO (128 65- bit words length) has at least sixteen empty words then data fragments are fed to it and then flow out to the receiver via the S-LINK64 CMC cards upon receipt of a read_fifo command from the S-LINK64 controller (section II.E.). Furthermore, event losses and dead time statistics are kept in counters that can be read by the PCI. D. PCI controller The PCI controller provides PCI communication plus registers for control, status, error and reset operations. In total there are 13 registers that can be accessed by the PCI. E. S-LINK64 controller To transfer the GTP Emulator pseudo-data, the S-LINK64 (Simple Link Interface in a Common Mezzanine Card) protocol is used. Detailed description of the S-LINK64 can be found in [4]. Upon the reception of the UWEN command (UWEN=low) from the PCI controller, the S-LINK64 controller initiates the reading of the local FIFO, so that data start to flow to the receiver side. Backpressure signal (LFF=low) is sent to the L1_gen module when the receiver cannot accept more data or LDOWN signal is low, indicating that the receiver does not run. FRL that broadcasts the data fragments to the FED Builder. Interface to the Trigger distribution system and the Throttling (atts, stts) system: a) The egtp system distributes eight L1A signals to the eight sub-detector partitions (FRLs). b) Each of the eight sub detector partitions sends to the egtp system, via FRL-sTTS 1, four LVDS signals (READY, BUSY, OUT_OF_SYNC and WARNING). If a sub-detector sends a not READY signal the egtp system inhibits L1A for the partition in which the subdetector belongs. c) Each of the eight DAQ partition controllers (atts) receives from the egtp system its status and sends to the egtp system four LVDS signals (READY, BUSY, OUT_OF_SYNC and WARNING). If a partition sends a not READY signal, the egtp system inhibits L1A signal for this partition. The signals a), b), and c) are transmitted to the external systems via a specially designed module (egtp-io), with an Altera-ACEX 30k, which performs the necessary encoding/decoding from/to LVDS/TTL signals. The connection of the egtp-io to the egtp is done via a Mezzanine card (egtp CMC) plugged on the fast GIII connector (Fig.4, Fig.5) Fig. 4.The egtp interfaces schematic. III. INTERFACES The egtp system has the following hardware interfaces to external systems (Fig.4): Interface to the Event Manager: The egtp system sends (via SLINK-64) the L1A data fragments (encapsulated in the FED Common Data Format) to an 1 The equivalent of stts of the real system
Trigger number, partition event number and trigger rules have been verified for a large number of accumulated events (up to 2x10 9 ) for trigger rates 10kHz up to totally 750kHz and for different DAQ partition and sub-detectors partition schemes (up to eight partitions). The synchronization of the L1A signal with the corresponding event fragment has been verified up to 2x10 8 events, using a fast CAEN counter timer, at a rate of 120kHz. Moreover, the event timing has been tested, for different trigger rates, to follow Poisson statistics (Fig. 6). Fig. 5. a) The egtp-io module (left), the egtp system (middle) and the EVM receiver board (right) are shown, together with the connection of egtp to/from egtp-io (flat cable) and the connection to EVM receiver (LVDS S- LINK64 cable), b) close-up view of the egtp system board. IV. PERFORMANCE TESTS An egtp test-bench has been set up in order to test the performance of the system (Fig.4). It is implemented by a PC running Linux RedHat7.3.3 with 64b@66MHz PCI bus. The egtp is implemented by a GIII card and sends event fragments, via the S-LINK64 CMC cards, into a second GIII card (receiver) plugged into the PC. To control the egtp a set of drivers have been developed based on LabView (National Instruments [7]) whereas the FedKit 1_30 software [8] has been used to read and analyze data in the receiver side. Two different sets of tests have been performed using this test-bench. In the first test (slow test; limited to rates up to 20KHz) data arriving via the S-LINK64 are dumped into the PC s hard disk and are analyzed offline in order to verify that the system responds adequately to the S-LINK64 backpressure and that the LHC beam structure is correctly reproduced. In the second test, fast readout measurements have been taken and data are read and analyzed online at full speed. Fig. 6. Probability of observing a subsequent trigger as a function of time for events generated at 100kHz. V. CONCLUSIONS A Global Trigger Processor Emulator system (egtp) has been developed aiming at decoupling the L1 trigger system (underground) from the readout system (surface DAQ) for testing, installation and maintenance purposes. The egtp system emulates the Global Trigger Processor by generating level-1 triggers, applying partitioning, emulating the LHC proton beam structure, generating trigger summary pseudodata and receiving feedback signals from DAQ and detector partitions. The egtp system is based on the multi-purpose PCI card Generic III hosted in a Personal Computer and data transmission to the receiver board (FRL) is achieved via the SLINK-64 protocol. Extensive tests have verified that the system responds satisfactorily to the S-LINK64 backpressure and that the LHC beam structure is correctly reproduced. Furthermore, the trigger number, partition event number, trigger rules and event timing have been verified for a large number of accumulated events, for different trigger rates and for different DAQ partition and sub-detectors partition schemes.
VI. ACKNOWLEDGMENT We wish to thank Eric Cano and Dominique Gigi for their constant support during the development and testing of the egtp system. Fruitful discussions with Christoph Schwick and Joao Varela are gratefully acknowledged. VII. REFERENCES [1] CMS Collaboration, "The Trigger and Data Acquisition project, Volume I, TDR", CERN/LHCC 2000/038. [2] CMS Collaboration, "The Trigger and Data Acquisition project, Volume II, TDR", CERN/LHCC 2002/026. [3] "GIII FedKit", http://cmsdoc.cern.ch/cms/tridas/html/documents.html [4] A. Racz, R. McLaren, E. van der Bij, "The S-LINK 64 bit extension", http://hsi.web.cern.ch/hsi/s-link/spec. [5] Altera, http://www.altera.com. [6] Celoxica, http://www.celoxica.com. [7] National Instruments, http://www.ni.com. [8] V. Brigljevic, G. Bruno, E. Cano, S. Cittolin, S. Erhan, D. Gigi, F. Glege, R. Gomez-Reino Garrido, M. Gulmini, J. Gutleber, C. Jacobs, M. Kozlovszky, H. Larsen, I. Magrans de Abril, F. Meijers, E. Meschi, S. Murray, A. Oh,L. Orsini, L. Pollet, A. Racz, D. Samyn, P. Scharff- Hansen, C. Schwick, P. Sphicas, J. Varela, "FEDkit: a design reference for CMS data acquisition inputs", CMS CR 2003/036