The CERN CMS Tracker Control System F. Drouhin, L. Gross, D. Vintache, A. Marchioro, C. Paillard, C. Ljuslin, L. Mirabito, P.G. Verdini Abstract Due to the high integration level of the experiments planned for the Large Hadron Collider (LHC) of the European Organization for Nuclear Research (CERN), the data acquisition and the control systems need complex developments both in hardware and software. The purpose of this paper is to describe the control system of a sub-detector of one of the CERN experiments, the Tracker of the Compact Muon Solenoid (CMS). The CMS Tracker control system is based on dedicated hardware and software. The hardware is based on the Front End Controller (), an interface board that hosts token rings for the communication with the Control and Communication Unit () modules. These in turn contain dedicated I/O channels for the front end readout and control chips. The software is built in layers: one device driver, a C++ dedicated application program interface (API) plus a database for the storage of all the information needed for the front end electronics. This system will also be adopted in some other CMS sub-detectors. Index Terms Control System, Software, Oracle Database, ASIC I. INTRODUCTION THE CMS apparatus [1] is a multi-purpose detector designed for operation at the CERN LHC [2] and optimized for the identification of the Higgs boson and the measurement of its properties over a wide mass range, as well as for the search of new physics phenomena. A high-precision muon detection and tracking system is installed around a four Tesla superconduction solenoid. Inside the magnet are located the hadronic and electromagnetic calorimeters, and the Central Tracker. The CMS Tracker [3] consists of an inner Pixel system and of an outer Silicon Strip Tracker (SST); the control system of the latter is described in this paper. The SST pseudorapidity coverage ranges up to η 2.5. It will provide at least ten points per track, resulting in a transverse momentum resolution for isolated tracks in the central region better than δp (15 p +0.5)%, where p is measured in TeV/c. This in turn will result in a reconstruction efficiency better than 98 % for muon tracks over the full F. Drouhin is with Université de Haute-Alsace, Mulhouse, France. Corresponding author: Frederic.Drouhin@cern.ch, Institut de Recherches Subatomiques de Strasbourg, Batiment 21, 23 rue du Loess, BP28, F67037 Strasbourg Cedex L. Gross is with Institut de Recherches Subatomiques de Strasbourg, France D. Vintache is with Institut de Recherches Subatomiques de Strasbourg, France A. Marchioro is with European Organization for Nuclear Research, Geneva, C. Paillard is with European Organization for Nuclear Research, Geneva, C. Ljuslin is with European Organization for Nuclear Research, Geneva, L. Mirabito is with Institut de Physique Nucléaire, Lyon, France P.G. Verdini is with Istituto Nazionale di Fisica Nucleare, Pisa, Italy pseudorapidity range, and better than 90 % for high energy electrons, while keeping the occupancy and the amount of ghost tracks at the percent level. These features require a single-hit resolution of 20µm, well within the possibilities of the technology chosen. The SST has been designed to guarantee stable operating conditions for ten years of LHC running, nominally. One of the critical issues for the viability of the system is the longterm survivability after heavy irradiation. The CMS SST will operate in a hostile environment, where the integrated particle fluence, expressed in 1 MeV neutron equivalent, is expected to reach 1.6 10 14 cm 2. In order to minimize the negative effects of annealing of radiation-induced damage to the Silicon bulk, and to prevent thermal runaway in the later stages of operation, the ambient temperature in the SST will be kept around 20 o C, when the saturation vapour pressure of water over ice corresponds to 3000 ppm. The need to compensate for accrued radiation damage in the sensors and in the front-end electronics has led to the development of complex, programmable readout electronics; the extreme environmental conditions have prompted the development of a radiation-hard ASIC which can be deployed on every individual module and which can be used to monitor the temperature of the sensors and of the front-end hybrid, the low voltage levels and the bias currents drawn by the sensors. The control system for the CMS SST must be capable of monitoring the temperature, humidity, voltage and current measurements coming from 16,000 modules, and at the same time ensure the download of correct parameters to the front-end electronics while checking for the occurrence of Single Event Upset phenomena in the chip registers. A dedicated control system, both in hardware and in software, has been developed to face this challenge. With increasing radiation exposure, the response of the silicon sensors as well as that the readout electronics is bound to change. Therefore, for the CMS Tracker special electronics has been developed whose mode of operation can be modified by programming specific registers. The Single Event Upset (SEU) phenomenon can change register values (bit changes). There are five kinds of front end chips shown in figure 1: - The APV [4]: front end chip for the multiplexed readout of 128 strips; - The APV Multiplexer (APV Mux): chip that multiplexes the outputs of two APV on a single optical link; - The PLL: programmable delay and clock rephaser; - The Laserdriver: used for the transfer of data between the front end electronics and the Front End Driver (analogue to digital converter with memory) over an optical fiber;
Fig. 1. A Tracker module with two silicon sensors and the electronic hybrid that hosts the front end electronics - The DCU (Detector Control Unit): this chip is very important in the CMS Tracker control system because it gives condition parameters for voltage, current and temperature. It contains an eight channel analog to digital converter, two constant current sources and a temperature sensor. It will monitor two sets of thermistors, one on the sensor and one on the hybrid, its own internal temperature, the silicon detector leakage current and the two (1.25 V and 2.5 V) low voltages. So there are three temperatures, two voltages and one current measurement for each hybrid. Moreover, each DCU has a unique hardware identification number that can be read by software. This identification number will be used in order to identify in the final Tracker construction the position of each module and to act as a link between the construction database that stores the module information (pedestal, calibration,...), the condition database and the online parameter database. The APV chips (4 or 6 per module), APV Mux, PLL and DCU (in the following text referenced to generically as devices ) each have several registers to configure. All registers are accessed over an Inter Integrated Circuit (I 2 C) bus with the standard modes described in the specification [5] and a specific addressing mode for the APV with a non standard extension of the I 2 C extended mode, the RAL mode. Before starting data taking through the data acquisition system [6], the front end electronics must be configured properly and each device register must be set through the I 2 Cbus. For the Silicon Strip Tracker, about 16,000 modules will be produced and installed. Thus the potential problems of the system (both in hardware and software) come from its complexity and from its dimensions. In fact, the parameter set and therefore the number of registers to access is about 1,680,000. Most of the device registers must be tuned in order to find the best conditions for taking physics data. In parallel to this electronic design, a control system was developed to set and retrieve all the device parameters. The next sections describe the hardware and software architecture. Note that this hardware and software control architecture is also used in other parts of the CMS experiment with the same / rings but different front end electronic chips. II. HARDWARE ARCHITECTURE A. Hardware description Token Ring Fig. 2. rings. 16 i2c channels 1 Memory channel 1 JTAG channel : Control and Communication Unit : Front End Controller 4 PIO channel i2c channels Hybrids / Devices 1 Trigger channel Hardware architecture. Note that one can manage up to 8 token Figure 2 shows the hardware architecture of the control system. This architecture is based on a specific board, the Front End Controller () [7], shown in figure 3, which hosts a token ring with modules, Communication and Control Units () [8], shown in figure 4. Each manages several kinds of channels:
- 16 I2 C channels for the access to the front end chips. Each channel can be configured for different speeds (100, 200, 400 khz and 1 MHz). - 4 Input/Output-like parallel bus controllers (Parallel input output (PIO)) such as the ones used in the Motorola PIA, etc. These channels are used in the Tracker in order to reset the front end readout electronics; - 1 memory channel, memory-like bus controller to access devices such as static memories, analog digital converters, etc. - 1 JTAG1 master controller ; - 1 trigger distribution controller ; - 1 node controller (the control itself is seen as a special channel capable, for instance, to report the status of the other channels). Fig. 3. Current prototype of the PMC Front End Controller Board for PCI support The memory, JTAG and the trigger channels are not used in the Tracker control system, but some other parts of the CMS experiment use the memory channel and check the possibility to use the trigger channel. The is built to be radiation resistant but SEU can cause bit changes, so a Cyclic Redundancy Checksum (CRC) and an auto-correction algorithm are implemented in case of errors. For SEU and noise reasons, the connexion between the and the ring is optical with laserdrivers. In the Tracker, the is completed with a DCU for the voltage and temperature measurements which must also be read. In order to prevent any problem due to the loss of a that would break the ring and consequently lead to lose all the on the ring, an additional set of connection, are used 1 IEEE Standard 1149.1 (JTAG), IEEE Standard Test Access Port and Boundary-Scan Architecture Fig. 4. Current prototype of the Control and Command Unit module with 6 I2 C channels and the two rings A/B for reliability to implement a second, redundant ring. This redundant ring makes it possible to bypass one by configuring the input and output of each and. Once the ring is built, for each access, whatever the channel is, a frame is built with a number, a address, a channel, a transaction and a command number. The channel number specifies one of the channels. The transaction number allows to recover the action which initiates the frame, for example, a read command. The command number gives the command to be executed (read, write, read modify write). For the I2 C accesses, the frame must also contain an address and possibly some data for write or read-modify-write operations. Note that the ring number is not given; it is configured via a specific register into the and. For research and development reasons, two versions of the have been developed, one with an electrical and one with an optical connection to the ring. Both are developed under the Peripheral Component Interconnect (PCI) bus but the final version of the will be on the VME bus with eight optical rings. An intermediate version will be available in mid-2004. For the same reasons two versions of exist (6 and 25). III. S OFTWARE ARCHITECTURE The software architecture presented in figure III is built upon the hardware architecture presented in the previous section. The software is split up into several parts: - Run Control [9]: controls the data acquisition and control system with the help of the Detector Control System. - Information Message service (IMS) integrated in the run control: dedicated task that receives all the messages (information, error,...) belonging to the run control. - device driver: a Linux device driver for the. - supervisor (, the FecSoftware API): central part of the control, it manages several s in order to read and write values into the front end electronics.
Supervisor PCI bus i2c access Oracle OCCI SOAP (Finite State Machine) Database Supervisor CMS General Run Control Supervisor IMS subscribe DCU Analyser Diagnosis system Database access through Oracle Call C++ Interface (OCCI) Finite state machine: SOAP message Hardware access Device driver access (software: open, read, write, ioctl, close) Fig. 5. FecSoftware Architecture - Database: Oracle database used to store all the values needed for the hardware. It manages all the parameters to set the hardware with the help of a versioning system. - DCU analyzer: dedicated process to analyze all the values provided by the DCU. - diagnostic system: system that diagnoses the source of errors and possibly takes a decision and signals it to the run control system. The software control is built around the device driver, database, (FecSoftware API), DCU analyzer and the diagnostic system. The control software issue is divided in two actions: - Download and upload parameters in the front end electronics: these two operations require that all / / channels (PIO and I 2 C channels) needed for the access, are enabled and configured. Once the control hardware is configured, the software sends the value needed to each register of the front end electronics (download). At the end of the download operation, the parameters set are read back from each device and compared to the values downloaded. If a difference appears, the faulty device register is stored in the database and an error is reported to the run control through the IMS. A similar operation can be undertaken directly between the event builder and the control processes (FecSupervisor) to make calibration or a delay curve. During this kind of operation, the event builder asks the control system to set PLL or APV values. The download/upload system must be done in a relatively short time (about 2 minutes for the current measurements) - Control the environmental parameters: this operation consists in reading all DCU on the hybrid and and to send all this information to the IMS. By a mechanism of web subscription, the information is sent to the DCU analyzer to check the correctness of the parameters (voltages and temperatures). These readings are performed every thirty seconds but this rate can be changed dynamically depending on the needs of the system. This system must detect a failure in any system in terms of temperatures and power. A second system exists but it is controlled through hardware locks to switch off all the Tracker to prevent, burning for example. Note that the DCU system is important to avoid the switching off of the complete system. If a problem occurs in temperature or humidity, the DCU analyzer must detect the divergence of the system and must send a message or take the decision to switch off the system correctly one part at a time. Each part of the software control system will be described with its requirements, the mode of operation and performances. The components are: device driver, Database, C++ Fec- Software API, DCU analyzer and finally the diagnostic system. IV. CONCLUSION A. FecSoftware summary Throughout this paper, all the interfaces needed to make system accesses and configurations will be presented: - /Ring/ hardware: physical layer; - device driver: low level software layer; - FecDevice class: interface to the device driver; - FecAccess class: concurrent access management; - DeviceAccess class and its sub-classes: specific device access; - FecSupervisor and FecAccessManager classes: software application.
All the tools and the API are completely documented with the Doxygen tool [11] and this documentation is available on the web page [10]. It gives developers the ability to develop and use the software for new front end electronic chip accesses and new channels. So the tools developed (device driver, C++ API) are used in some other parts of the CMS experiment: the Preshower, the µ chambers and the Electromagnetic Calorimeter for the time being. All software and tools described here are used in several Tracker production centers (Aachen, Karlsruhe, Louvain la Neuve, IPN Lyon, INFN Pisa, IReS Strasbourg, etc.) for the tests and the production of the detectors (silicon sensors for the Tracker). B. Performance This system has been fully tested in beam tests (2002, 2003) and has proved to fulfill all the requirements needed by the final experiment in terms of performance and functionalities for the Tracker. The download performance is good, for the whole Tracker in the worst case (only one supervisor and one database machine) the complete Tracker parameters settings are downloaded in 205 s. Concerning the upload performance, the time needed is not so good: about 2000 s is required (always in the worst case) due essentially to the database upload time. But this time is only valid if the complete Tracker front end electronics is in a bad state, in case of a power cut for example. In the best case (no errors), the upload time will only cost about 5 s to retrieve the parameters from the hardware and do a comparison, knowing that only the devices with errors in settings will be stored in the database. Note that these numbers come from the different curves that will be shown in the paper. Note that the performances described in this paper are given for hardware currently used. Due to choices of the Tracker and CMS collaboration, the Tracker will use the VME bus and personal computers for crate controllers with a PCI/VME links (see next paragraph on the evolution). We are obviously planning to repeat the measurements for the final hardware. is to manage the system through the same API, and the interfaces must be the same when using PCI bus or VME bus. Another upgrade is to develop the diagnostic system for the Tracker control. The use of an expert system is justified by the complex interconnections of the system. For example if the detector s data are not correct (detected by the data acquisition system), the diagnostic system must ask the control system if all the parameters are correct (power, values correctness in the front end chips) and an expert system can take the decision to recover the system by switching one of the or to the redundancy channel, resetting the or simply to re-downloading the values. The current plans are to develop this system and see how far we can automatize the solution of problems, i.e. whether this will be just and aid to the expert on chift or an automatic recovery tool. REFERENCES [1] CMS Collaboration, Technical Proposal, CERN/LHCC 94-38, December 1994 [2] http://lhc-new-homepage.web.cern.ch/lhc-new-homepage/ The Large Hadron Collider (LHC) project of CERN, [3] CMS Collaboration, The Tracker Projet - Technical Design Report, CERN/LHCC 98-6, April 1998 [4] P. Placidi, A. Marchioro, P. Moreira, K. Kloukinas, A 40 MHz Clock and Trigger Recovery Circuit for the CMS Tracker Fabricated in a 0.25µm CMOS Technology and Using a Self calibration Technique, presented at the 5th Workshop on Electronics for LHC Experiments, Snowmass, CO, USA, September 1999. [5] Philips Semiconductors, The I 2 C-Bus Specification, Version 2.1, document order number: 9398 39340011, January 2000 [6] L. Mirabito et al., Tracker data acquisition for beamtest and integration, Internal note, CMS IN 2003-021, May 2003 [7] A. Marchioro, C. LJustin, C. Paillard, Optical, Front End Control Unit for Embedded Slow Control, Draft 0.16, not pulished, February 2003 [8] A. Marchioro, C. LJustin, C. Paillard, 25, Communication and Control Unit for Embedded Slow Control, Draft 2.0, not pulished, May 2002 [9] M. Bellato et al. Run Control and Monitor System for the CMS Experiment, Submitted to Computing in High-Energy and Nuclear Physics, La Jolla CA, USA, March 24-28, 2003, [10] F. Drouhin, L. Gross, D. Vintache The FecSoftware documentation, http://cmsdoc.cern.ch/cms/cmt/system aspects/fe [11] Doxygen web site, http://www.doxygen.org/index.html C. Evolution The next upgrade of the system will be of the hardware. In fact, the final format of the board will be the VME and the software must be changed in order to take care of this bus. The CMS collaboration has chosen to use PCI/VME link, essentially for two reasons: - A Personal Computer with standard PCI interface is cheaper than a VME-board computer and has a more modern, powerfull processor - All CMS online software is developed on Linux PC that takes advantage of recent Linux kernel developments and all available Linux tools. The tests will be performed on a PCI/VME link, but the final link has not been chosen yet. The goal of this upgrade