CMS Calorimeter Trigger Phase I upgrade

Transcription

1 Journal of Instrumentation OPEN ACCESS CMS Calorimeter Trigger Phase I upgrade To cite this article: P Klabbers et al View the article online for updates and enhancements. Related content - CMS level-1 upgrade calorimeter trigger prototype development P Klabbers, M Bachtis, J Brooke et al. - A demonstration of a Time Multiplexed Trigger for the CMS experiment R Frazier, S Fayer, G Hall et al. - The AMC13XG: a new generation clock/timing/daq module for CMS MicroTCA E Hazen, A Heister, C Hill et al. Recent citations - The upgrade of the CMS trigger system M Jeitler and for the CMS collaboration - CMS level-1 upgrade calorimeter trigger prototype development P Klabbers et al - Upgrade of the trigger system of CMS Manfred Jeitler This content was downloaded from IP address on 28/11/2018 at 23:32

2 PUBLISHED BY IOP PUBLISHING FOR SISSA TOPICAL WORKSHOP ON ELECTRONICS FOR PARTICLE PHYSICS 2011, SEPTEMBER 2011, VIENNA, AUSTRIA RECEIVED: November 17, 2011 ACCEPTED: December 14, 2011 PUBLISHED: January 10, 2012 CMS Calorimeter Trigger Phase I upgrade P. Klabbers, a,1 T. Gorski, a M. Bachtis, a K. Compton, b S. Dasu, a A. Farmahini-Farahani, b R. Fobes, a A. Gregerson, b M. Grothe, a I. Ross, a D. Seemuth, b M. Schulte c and W.H. Smith, a a Physics Department, University of Wisconsin Madison, WI, U.S.A. b Electrical and Computer Engineering, University of Wisconsin Madison, WI, U.S.A. c AMD Research, Austin, TX, U.S.A. pamc@hep.wisc.edu ABSTRACT: We present a design for the Phase-1 upgrade of the Compact Muon Solenoid (CMS) calorimeter trigger system composed of FPGAs and Multi-GBit/sec links that adhere to the µtca crate Telecom standard. The upgrade calorimeter trigger will implement algorithms that create collections of isolated and non-isolated electromagnetic objects, isolated and non-isolated tau objects and jet objects. The algorithms are organized in several steps with progressive data reduction. These include a particle cluster finder that reconstructs overlapping clusters of 2x2 calorimeter towers and applies electron identification, a cluster overlap filter, particle isolation determination, jet reconstruction, particle separation and sorting. KEYWORDS: Trigger concepts and systems (hardware and software); Trigger algorithms 1 Corresponding author. c 2012 IOP Publishing Ltd and SISSA doi: / /7/01/c01046

3 Contents 1 Introduction The CMS Calorimeter Trigger Upgrade Calorimeter Trigger algorithms Simulated rates and efficiencies 2 2 Technology upgrades for the Compact Calorimeter Trigger Base-X Ethernet demonstrator Module Management Controller project Flash-over-LAN Unified system alignment 4 3 Compact Calorimeter Trigger design Calorimeter Trigger Processor card 6 4 Installation 8 5 Conclusions 8 1 Introduction The current Compact Muon Solenoid (CMS) Level-1 Calorimeter Trigger has been designed to operate up to the Large Hadron Collider (LHC) design luminosity of cm 1 s 2. It is expected that in the future the LHC instantaneous luminosity will increase over the original design. Because of this, the number of interactions in an LHC collision (pileup) will increase up to 5 times. To fullfil the physics needs of the experiment, Level-1 trigger thresholds need to remain about the same. The present Level-1 Calorimeter Trigger algorithms will become inadequate above or with 50 ns bunch spacing. Pileup will degrade the electron, photon, and tau isolation, requiring more sophisticated clustering and isolation for these busier events. The use of modern FPGAs and the full granularity of the calorimeter trigger information allows more complexity, flexibility, and tuning of algorithm isolation and energy cuts. Initial L1 Trigger simulations show a significant rate reduction with an upgraded calorimeter trigger. 1.1 The CMS Calorimeter Trigger This upgrade will replace the current CMS Calorimeter Trigger. This consists of two subsystems, the Regional Calorimeter Trigger (RCT) [1], and the Global Calorimeter Trigger (GCT) [2]. The RCT receives Trigger Primitives (TPs) in the form of an 8-bit rank (E T ) and quality bit from the over 8000 towers in the Hadron, Forward, and Electromagnetic Calorimeters for η < 5. It processes this information in parallel and sends 144 e/γ candidates and 36 energy sums per 4 4 1

4 Figure 1. A pictorial representation of the new trigger algorithms. HCAL and ECAL trigger towers are the same size and the information from both is used for the algorithms for e/γ, τ, and jet triggers. tower region and bits for object classification. The GCT sorts the e/γ candidates further and finds jets (Central, Forward, and Tau) and calculates global quantities like Missing E T. It then sends eight e/γ candidates, four each of Central, Tau, and Forward Jets, Jet counts, and several global quantities to the CMS Global Trigger (GT). 1.2 Upgrade Calorimeter Trigger algorithms In the calorimeter the trigger tower sizes will remain the same. ECAL will still send 9 bits of data, and the HCAL is studying the possibility to send more information including a longitudinal profile. The upgraded calorimeter trigger begins to find objects by applying tower thresholds to the calorimeter and creating overlapped 2 2 clusters of trigger towers. A cluster overlap filter follows this and removes any overlap between clusters, identifies local maxima, and prunes the low energy clusters. Cluster isolation and particle identification are applied to the local maxima and isolation deposits are calculated around 2 2 and 2 3 clusters. This identifies particles as e/γ candidates (2 2) and taus (2 3). The jet reconstruction is applied to the filtered clusters and groups them to find jets. Finally a sort is applied to find the most energetic candidates of electrons, taus, and jets. These and the E T Sums and Missing E T are calculated from the clusters and sent to the CMS global trigger for final decision-making. Diagrams of the object algorithms are in figure Simulated rates and efficiencies In order to evaluate the performance of proposed new algorithms with respect to the current algorithms, simulated QCD and physics data with 25 pileup events per crossing was used with the software emulators of both versions of the calorimeter trigger. The main benefit of the calorimeter trigger upgrade will be the improved performance of e/g and τ triggers [3]. In figure 2 the results for the rates and efficiencies for isolated electrons and taus are shown. A rate reduction of up to 4 times is possible with the new calorimeter trigger algorithms. Efficiencies for electrons are kept about the same and the tau efficiency improves significantly. 2 Technology upgrades for the Compact Calorimeter Trigger Upgrading the calorimeter trigger involves updating the hardware. For the Compact Calorimeter Trigger (CCT), the µtca standard is being adopted [4]. Some of the advantages of the standard 2

5 Figure 2. The rates and efficiencies for electrons and taus for the current and upgraded calorimeter trigger. are compact, hot-swappable boards, and a smaller size that allows placement of the entire new system in an available single spare rack next to the existing system, permitting parallel operation. The CCT will also use optical links instead of copper, significantly reducing the cable volume of the system. The system has advanced monitoring and configuration tools accessible over a network with a commercial µtca Controller Hub (MCH), 1000Base-X Ethernet over the backplane, and IPMI (Intelligent Platform Management Interface) for initialization and monitoring. Also, modern FPGAs will be used which allow a great deal of flexibility in the algorithms and integrated links for data transmission. Currently design is proceeding with the Xilinx R Virtex R -6 [5], but the recently available Virtex R -7 is being considered Base-X Ethernet demonstrator One of the first steps in assessing the viability of the system is to understand if the 1000Base-X Ethernet over the backplane would provide sufficient bandwidth for uploading firmware and other operations for configuration and monitoring of the system. For this a Virtex R ML506 Evaluation board was set up so that it was running a lightweight IP (lwip) stack under the Xilkernel. A MCH was connected to the University of Wisconsin Physics Department s network via its Ethernet connection. Communication was routed to the ML506 via the backplane, a test card, and a SATA cable connected between the test card and the ML506. The program iperf was used to check the receive and transmit rates, which measured 14 Mbps and Mbps respectively. As a second test an echo server was setup between two ML506 boards connected via the backplane. In this mode, both boards were each running client and server applications. Both tests revealed that the bandwidth achieved was sufficient. 3

6 Figure 3. The MMC use block diagram and a protoype MMC mezzanine (left side of card) on a test card. 2.2 Module Management Controller project The Module Management Controller (MMC) is the IPMI endpoint for managing cards in µtca crates. The UW project includes a full implementation of an MMC based on the Atmel AVR 32- bit microcontroller. It supports the standard IPMI commands dictated by the specifications plus additional commands for operations outside the scope of the MMC specification. It communicates with the µtca module prior to and after FPGA initialization via a network connection to the MCH in the µtca crate. It is used for power control and monitoring (e.g. over-voltage and temperature protection), FPGA boot image selection and load control, and post-boot FPGA configuration (e.g. geographical card IP address). It is fully functional and already in use on several CMS upgrade designs. The block diagram and a photo of the MCC on a test card are shown in figure Flash-over-LAN The objective of the Flash-over-LAN (FoL) subproject is to support the remote update of the FPGA flash over the µtca Gigabit Ethernet (GbE) connection. For this, the FPGA acts as a server using the Xilinx R Microblaze TM processor with a TCP/IP stack (lwip) running under the Xilkernel. A PC-based client then connects to the server on the AMC cards to deliver new image. Advantages of the FoL architecture are its implementation in c/c++ as opposed to HDL for a more productive development environment. A common driver API can support additional file types as necessary (e.g. SREC or binary) and the common server can support different device types via a device-specific driver interface. In this way multiple boot images and FPGAs can be managed on a single AMC card. The method is already working and a block diagram is shown in figure Unified system alignment Unified system alignment is absolutely necessary for tower-level data sharing across calorimeter regional boundaries. In the future system, there exist several sources of possible misalignment: the length of the connection, the phase and latency variation in the Serial-Deserializer (SerDes) links, and the phase variation between the clocks and the cards. In order to study this and come up with a solution, a custom 2 2 test fabric with 4 test cards was used as a link synchronization test bed (see a photo and block diagram of the test fabric in figure 5). Each test card had a Xilinx R Virtex R -5 FPGA with Rocket I/O GTP links. An LHC 4

7 Figure 4. Flash-over-LAN block diagram. Figure 5. The test fabric and cards for the system alignment study. The clock for the system comes in via the orange fiber in the upper right side of the picture. A block diagram of the test bed is on the right. style clock distribution was used, with a clock generated by a local Trigger Timing and Control (TTC) system. This simulated 2 separate crates of 2 cards each. The goal was to demonstrate the alignment of all 56 separate channels operating on the same time base. A latency target and scheduled launch and arrival times were identified for data at all SerDes endpoints and based on a common global reference, such as the Bunch Crossing Zero (BC0) from the TTC system. The actual latencies were measured using special test characters (8b/10b K char) from the transmit links at scheduled times and measuring their arrival times. Arrival times were automatically compensated for by adding delay at the receive end. Fractions of an LHC clock could be added depending on the link rate. In a healthy system, no change is expected in the settings through the course of the period of running and short elastic buffers on the receive side would compensate for any clock jitter. The alignment state can be checked during periods like the LHC abort gap as part of the monitoring of the health of the system. Measurements with the Rocket I/O suggest that the optical links will need nine 25 ns LHC clock cycles of latency to serialize, transmit, and deserialize. In the current system with copper cables this is 6.6 LHC clock cycles. There is sufficient contingency to allow for this slight increase in latency. 5

8 Figure 6. A possible configuration for one of the µtca crates in the CCT design. In the end, four test cards with 56 links were synchronized at 1.6 Gbps. Proper alignment was verified by using test pipelines to compare data expected from the links to the actual data generated in the local pipeline. 3 Compact Calorimeter Trigger design Two different designs for the upgraded calorimeter trigger are being proposed. The Time Multiplexed Trigger (TMT) that is described in a separate paper [6], and the Compact Calorimeter Trigger (CCT). The CCT will use up to 6 µtca crates to cover the entire η φ space of the CMS barrel, endcap, and forward calorimeters. Each crate will house a number of boards covering the full η range and 12 ECAL and HCAL trigger towers in φ. A MCH with Ethernet uplinks will be used for control and configuration. Boston University s AMC13 [7] will be used for the experimental clock and control as well as the readout of data by the CMS data acquisition. A possible configuration has the input ECAL and HCAL Trigger Primitives arriving via fiber into 6 Calorimeter Trigger Processor cards. Four data sharing I/O cards will be needed to move edge tower information among the crates for the algorithms. All input and output of the crate will be via optical fibers, with the exception of the Ethernet uplink. A custom backplane will be needed to share data between cards in the crate, and will be made by a µtca crate vendor, VadaTech [8]. A diagram can be seen in figure 6, and the proposed backplane connections in figure Calorimeter Trigger Processor card Instead of designing different cards for each purpose (e.g. processing, sharing) a single Calorimeter Trigger Processor (CTP) card will be designed that can be programmed with operation-specific firmware. For its external data connections it will have four 12-channel optical receivers and a 12-channel transmitter. Sharing will be done with a combination of external connections and connections to the custom backplane. Connections to the AMC13 for clocks, global control, and 6

9 Figure 7. The custom backplane configuration needed for the crate layout in figure 6. The sharing of tower information along η for isolation algorithms will be done on the backplane, φ will be done on fibers between crates. Figure 8. A block diagram of the Calorimeter Trigger Processor card, backplane connections on the left side of the diagram and front-panel connections on the right. data acquisition will be over the backplane. Each card will have two Virtex R -6 FPGAs or one Virtex R -7 FPGA, an MMC for configuration, local control, and monitoring, SDRAM, and FPGA image flash. A block diagram appears in figure 8, with the backplane connection on the left and the front-panel connections on the right. 7

10 4 Installation The plan for installation depends on staging the new system alongside the present system. Starting in 2013, during the first LHC long shutdown (LS1), the replacement of the ECAL-RCT copper connections with optical fibers will be performed. As part of this process two new mezzanine cards will be built. On the ECAL side, the oslb (optical Synchronization and Link Board) will replace the existing SLB. On the RCT side an ORM (Optical Receiver Mezzanine) similar in style to the oslb, will replace the existing receiver mezzanine card. In between a patch panel with optical splitters will duplicate the ECAL outputs, one for the existing RCT and a copy for the new CCT. In total more than 500 link boards for each side will have to be built. At the same time, the µtca crates will be installed and be partially populated so that a fraction of the new CCT can operate in parallel with the existing RCT and be read out for validation. As boards are built and the system validated, more boards will be installed so that the entire system will be commissioned while the existing RCT continues to operate in parallel. During LS2 (around 2016) the final commissioning and switch over will occur. 5 Conclusions A new CMS Compact Calorimeter Trigger will meet and exceed the needs of the experiment as the LHC luminosity and pileup increase. It is a flexible design using the µtca standard to ensure ease of operation and maintenance. Its compact size allows installation in parallel with the current RCT and validation using existing algorithms. A suite of new tools and techniques to operate the system are mature: Flash over LAN, an MMC with IPMI, and a robust data synchronization technique. We expect to have a demonstrator system during In we expect to install a partial system in parallel by replacing some of the existing system with optical fibers and links and the full system to be installed by References [1] P. Klabbers, Performance of the CMS Regional Calorimeter Trigger, TWEPP 2009, Paris, France, and references therein. [2] J. Brooke, Performance of the CMS Global Calorimeter Trigger, TIPP 2009, Tsukuba City, Japan. [3] CMS collaboration, Technical Proposal for the Upgrade of the CMS Detector through 2020, CERN-LHCC (2011). [4] for details on the specification. [5] [6] G. Iles, A Demonstration of a Time Multiplexed Trigger for CMS, TWEPP 2011, Vienna, Austria. [7] E. Hazen, Simplified MCH providing clock/controls/daq functions, TWEPP 2011, Vienna, Austria. [8] 8