High level trigger online calibration framework in ALICE

Transcription

1 Journal of Physics: Conference Series High level trigger online calibration framework in ALICE To cite this article: S R Bablok et al 2008 J. Phys.: Conf. Ser View the article online for updates and enhancements. Related content - The ALICE High Level Trigger: status and plans Mikolaj Krzewicki, David Rohr, Sergey Gorbunov et al. - ALICE HLT Cluster operation during ALICE Run 2 J Lehrbach, M Krzewicki, D Rohr et al. - CMS - HLT Configuration Management System Vincenzo Daponte and Andrea Bocci Recent citations - Results from the first p+p runs of the ALICE High Level Trigger at LHC Kalliopi Kanaki (for the ALICE HLT collaboration) This content was downloaded from IP address on 26/08/2018 at 06:19

2 High Level Trigger Online Calibration framework in ALICE Sebastian Robert Bablok 1, Øystein Djuvsland 1, Kalliopi Kanaki 1, Joakim Nystrand 1, Matthias Richter 1, Dieter Röhrich 1, Kyrre Skjerdal 1, Kjetil Ullaland 1, Gaute Øvrebekk 1, Dag Larsen 1, Johan Alme 1, Torsten Alt 2, Volker Lindenstruth 2, Timm M. Steinbeck 2, Jochen Thäder 2, Udo Kebschull 2, Stefan Böttger 2, Sebastian Kalcher 2, Camilo Lara 2, Ralf Panse 2, Harald Appelshäuser 3, Mateusz Ploskon 3, Håvard Helstrup 4, Kristin F. Hetland 4, Øystein Haaland 4, Ketil Roed 4, Torstein Thingnæs 4, Kenneth Aamodt 5, Per Thomas Hille 5, Gunnar Lovhoiden 5, Bernhard Skaali 5, Trine Tveter 5, Indranil Das 6, Sukalyan Chattopadhyay 6, Bruce Becker 7, Corrado Cicalo 7, Davide Marras 7, Sabyasachi Siddhanta 7, Jean Cleymans 8, Artur Szostak 8,7, Roger Fearick 8, Gareth de Vaux 8, Zeblon Vilakazi 8 1 Department of Physics and Technology, University of Bergen, Norway 2 Kirchhoff Institute of Physics, Ruprecht-Karls-University Heidelberg, Germany 3 Institute for Nuclear Physics, University of Frankfurt, Germany 4 Faculty of Engineering, Bergen University College, Norway 5 Department of Physics, University of Oslo, Norway 6 Saha Institute of Nuclear Physics, Kolkata, India 7 I.N.F.N. Sezione di Cagliari, Cittadella Universitaria, Cagliari, Italy 8 UCT-CERN, Department of Physics, University of Cape Town, South Africa Sebastian.Bablok@uib.no Abstract. The ALICE High Level Trigger (HLT ) is designed to perform event analysis of heavy ion and proton-proton collisions as well as calibration calculations online. A large PC farm, currently under installation, enables analysis algorithms to process these computationally intensive tasks. The HLT receives event data from all major detectors in ALICE. Interfaces to the various other systems provide the analysis software with required additional information. Processed results are sent back to the corresponding systems. To allow online performance monitoring of the detectors an interface for visualizing these results has been developed. 1. Introduction of the ALICE High Level Trigger ALICE is designed to study heavy ion (Pb-Pb) and proton-proton (pp) collisions at an event rate of up to 1kHz. In the Time Projection Chamber (TPC ), the main tracking detector in ALICE with the largest data volume, the size of a single event is around 75 MByte. After a hierarchical selection of Level 0, 1 and 2 triggers and combination with the data of the other relevant detectors this sums up to a data rate of 25 GByte/s. To match this amount with the Data Acquisition (DAQ) archiving rate of about 1.25 GByte/s, the HLT performs online event c 2008 IOP Publishing Ltd 1

3 Figure 1. ALICE systems structure and event data flow. analysis and data reduction. For this purpose, the HLT receives raw event data of the Front- End-Electronics (FEE), which are sent as direct copies of the event data by the DAQ-ReadOut Receiver Cards (D-RORC ) during the run. In return, the HLT provides a Level 3 trigger (event selection) and efficient event data compression (e.g. entropy coding). Additionally the HLT allows for selection of Regions-of-Interest (RoI) within an event, performance monitoring of the ALICE detectors and calculation of new calibration data online. As shown in figure 1, the whole setup is steered and synchronized via the Experiment Control system (ECS) [1] [2]. To cope with the large processing requirements involved in these tasks, the HLT consists of a large computing farm with several hundred off-the-shelf PCs. These computers contain a dual board equipped with AMD dual core Opteron 2 GHz CPUs, 8 GByte of RAM, two Gigabit Ethernet connections and an Infiniband backbone for high throughput communications. An upgrade to quad core CPUs is foreseen. HLT-ReadOut Receiver Cards (H-RORC ) inside dedicated Front-End-Processor (FEP) nodes accept the raw event data and perform a first reconstruction step. Detector Algorithms (DA) on additional cluster nodes take over and accomplish the above mentioned tasks. Detector- Data-Links (DDL), which transfer data over optical fibers, cover the transportation of results back to DAQ. There the results are stored together with the event data. The layout of the cluster nodes, presented in figure 2, is matching the structure of the ALICE detectors and the different event analysis steps involved. Dedicated infrastructure nodes are reserved for services and cluster maintenance (e. g. an 8 TByte AFS (Andrew File System) file server and two gateway machines). Portal nodes take care of the exchange with the other ALICE systems in the online and offline world. These portal nodes and their specialized software are the main focus of this article. The latest server version of Ubuntu Linux, currently 6.6 LTS (Long Term Support), serves as operating system inside the cluster. An interweaved system of TaskManagers organizes the cluster and steers the tasks on each node internally. A dynamic data transport framework, designed after the publish/subscriber principle, takes care of the data flow [3] [4] [5]. Detector Algorithms (DA) of the AliRoot package, the analysis framework of ALICE, analyze the incoming raw event data and calculate new calibration settings [6]. The analysis software works independent from the transportation framework, which allows the DAs to run in Offline as well 2

4 Figure 2. HLT architecture and data flow inside the HLT cluster. The cluster organization matches the structure of the ALICE detectors (TPC, Transition Radiation Detector (TRD), Inner Tracking System (ITS),... ) and their analysis steps from cluster finding to trigger decisions and data compression. without any changes. This enables result comparison later on [7]. The cluster itself is monitored by the SysMES framework (System Management for Networked Embedded Systems and Clusters) and Lemon (LHC Era Monitoring) [8]. Fail safety and the avoidance of single points of failure have been major issues in the design of the cluster. 2. HLT Interfaces 2.1. Overview The HLT has redundant interfaces to the various other systems in ALICE. These include the ALICE online systems like the ECS, FEE, DAQ and the Detector Control System (DCS), as well as the ALICE Offline system and the Alice Event Monitoring (AliEve) framework. The latter one will allow for monitoring ALICE online in the ALICE Control Room (ACR). For receiving raw event data, 365 DDLs from the different detectors in ALICE are connected to the H-RORCs in the FEPs. The data are analyzed inside the HLT cluster. These tasks are in detail: provision of Trigger decisions, selection of RoIs (only the data of the relevant parts are streamed out), lossless data compression (like entropy coding, vector quantization in the TPC data model) [9] [10]. These data are sent back to the DAQ-LDCs (DAQ Local Data Concentrator) for permanent storage via 12 DDLs. The interfaces to ECS, DCS, Offline and AliEve are described in the following subsections. In case of a failure of one portal the backup node takes over the corresponding task. In the DCS and Offline case the tasks for communication in the two exchange directions (receiving data and sending data) are separated in different applications with own names. DCS data are fetched via the so called Pendolino, while HLT data are sent to the DCS over the Front-End-Device (FED) API [11]. Offline can fetch data from HLT using the Offline Shuttle mechanism and data from Offline are retrieved over the HLT Taxi. A sketch of these interfaces is presented in figure ECS interface The HLT, like all other ALICE systems, is controlled by the ECS. An ECS-proxy, consisting of a finite state machine, contacts the ECS and informs about its current state. Transition commands issued by ECS trigger state changes and provide the initial settings of the upcoming run. This information includes the upcoming run number, the experiment type (Pb-Pb or p-p), operating mode, trigger classes, DDL lists, etc. The ECS-proxy accepts the current state from the Master TaskManagers, which control the HLT cluster internally. All state transition commands, issued by ECS, are referred to the Master TaskManagers as well [12]. The proxy is implemented in SMI++ (State Management Interface), which communicates with the ALICE ECS system using DIM (Distributed Information Management), a 3

5 Figure 3. Overview of the HLT interfaces (FEP = Front-End-Processor; DDL = Detector Data Link; HOMER = HLT Online Monitoring Environment including ROOT) communication framework developed at CERN [13]. The ECS is connected to all other ALICE systems. This allows for synchronizing the HLT with the other parts of ALICE DCS interface Pendolino The DCS controls and configures the FEE of all detectors in ALICE. In addition the current detector status is monitored; their temperature, voltage and current values are measured. These run conditions are received in the PVSS (Process Visualization and Steering System) panels of the corresponding detectors and then stored as datapoints to the DCS Archive DB during the run. The DAs running in the HLT cluster require a well defined subset of these values to calculate calibration settings and observables like the TPC drift velocity. Therefore a special HLT application, the Pendolino, contacts the DCS during the run and fetches the desired values. Since these values are constantly measured and can vary during the run, the Pendolino requests these values frequently from an Amanda server (Alice MANager for Dcs Archives), which sits on top of the DCS Archive DB [14]. It is foreseen to have three different Pendolinos running, each with a different frequency and each requesting a different subset of datapoints. These datapoints are received as timestamp value pairs. To allow the DAs to read the data regardless of whether running online or offline, the pairs have to be preprocessed and enveloped into ROOT objects. Each detector providing DAs to the HLT has to implement its own preprocessing routine. This procedure is adapted from the Offline Shuttle mechanism, which is used to store DCS data into 4

6 Figure 4. Deployment of the Pendolino for fetching DCS values from the DCS Archive DB and providing them as ROOT files to the DAs in the HLT cluster. the Offline Condition DataBase (OCDB) [15]. Due to the fact that it can take up to two minutes until DCS data are shipped from the PVSS panels to the DCS Archive DB, the preprocessing routine for HLT has to encode some prediction calculation for the retrieved values in the produced ROOT objects. This might be required for certain values in order to cope with online processing. The prediction encoding gives the routine its name: PredictionProcessor. The implementation of the prediction calculation is up to the detectors requiring the data. The produced ROOT file is stored in a file catalogue called HLT Condition DataBase (HCDB). The file catalogue is distributed to the cluster nodes running the DAs and updated each time new data are available. Afterwards a notification about new content in the HCDB is percolated through the analysis chain. The Pendolino procedure is visualized in figure FED-portal To return data like the TPC drift velocity to the DCS system, the HLT uses the Front-End-Device (FED) API, which is common among all detectors integrated in the DCS. Therefore DCS related data inside the HLT cluster are collected by the FED-portal during the run. A DIM server implementing the FED API sends these data from the FED-portal to the corresponding PVSS panels on the DCS side. From there it is included automatically in the DCS Archive DB Offline interface Taxi Assumed or in former runs calculated calibration and condition settings are stored as ROOT files in the OCDB [15]. The DAs require them in order to analyze events and calculate new calibration objects. A special application, called Taxi, requests the OCDB for latest available calibrations settings in regular time intervals and synchronizes them with the local copy of the OCDB, the HCDB. To reduce traffic, the Taxi first checks if the data are already available in the HCDB before it is fetched from the OCDB. The whole procedure runs independently and asynchronously to any run. At the start of each run the current version of the HCDB is fixed to avoid updates of calibrations settings during the run. Then the HCDB is distributed to all cluster nodes running DAs. Access to the HCDB is granted through the AliCDB (AliRoot Conditions Database) Access classes, which are also used in Offline to request the OCDB. This guarantees transparent access for the DAs, independent from running online or offline. The AliCDB Access classes return automatically the latest version of calibration settings valid for a given run number. 5

7 Figure 5. Deployment of the Shuttle portal and the Offline Shuttle mechanism for retrieving new calculated calibration objects from the HLT cluster Shuttle portal After each run the Shuttle portal collects all newly calculated calibration objects from the DAs. Transportation of the data are realized via dedicated components of the PublisherSubscriber framework. The calibration objects are stored in a File EXchange Server (FXS), while additional meta data for each file (like run number, detector, file ID, file size, checksum and timestamps) are stored in a MySQL DB. When all new objects are saved, the Shuttle portal notifies the ECS-proxy, that the collection process has finished. Now the ECS can trigger the start of the Offline Shuttle. The Shuttle requests the meta data of the latest run for the new entries in the FXS from the MySQL DB. Then it fetches the according files from the Shuttle portal non-interactively, using an shh-key. All new files are preprocessed by detector specific ShuttlePreprocessors and enveloped in ROOT files, if not already done inside the HLT cluster [15]. Afterwards the new entries are stored in the OCDB, where the Taxi can fetch them for the next run. The detour over the OCDB has been chosen to guarantee coherent version control of the calibration objects. The whole mechanism is sketched in fig AliEve interface Since the HLT performs the task of event analysis and calculation of new calibration data online, observation of the results is also possible online. Therefore the HLT provides the HOMER (HLT Online Monitoring Environment including ROOT ) interface, which offers a connection to the Alice Event monitoring framework (AliEve). AliEve is part of the AliRoot package and includes 3D visualization, as well as displaying of ROOT structures and histograms [16]. HOMER can fetch produced results at any step of the HLT analysis chain and transport them to any AliEve application inside the CERN General Purpose Network (GPN). This enables the operators to display results directly in the ACR. 3. Time line sequence and synchronization Each of the presented interfaces have dedicated places in the usage sequence. As shown in figure 6, this sequence is divided in five different time periods: Independent from a run (asynchronous to the runs): Before a first run (and repeated in regular time intervals) the Taxi requests latest calibration 6

8 Figure 6. Sequence diagram displaying the interplay of the different interfaces and portals, participating in the calibration framework of HLT. SoR (Start-of-Run) and EoR (End-of-Run) are special events triggered by ECS to indicate the start and end of a run. settings from the OCDB and caches them locally in the HCDB. Actually, this task is accomplished completely asynchronous to any run and can be performed also during a run. Initialization period before a run: The ECS informs the HLT about an upcoming run with a first INITIALIZE command. In addition, several run settings (like run number, beam type, trigger classes, etc) are transmitted. During the following configuration steps, the HLT freezes the current version of the HCDB and distributes it to the cluster nodes running the DAs. In case the Taxi fetches new data from the OCDB during a run, the new settings are only stored to the HCDB version located on the Taxi portal node, but not updated on the DA nodes. This guarantees that the DAs use a coherent version during the complete run. The completion of the initialization procedure is signaled back to the ECS. During a run: Every run starts with a special event triggered by ECS: i. e. the Start-of-Run (SoR). After the SoR event, raw event data are received from the FEE on the FEP nodes. The data are processed and analyzed over several steps. New calibration settings are calculated. For additional input, the Pendolino fetches current environment and condition settings from the DCS Archive DB (like temperature, voltages, etc). After preprocessing and enveloping them, they are available for the DAs via the HCDB. Analyzed events and trigger decisions are streamed out to DAQ for permanent storage. Freshly calculated, DCS relevant data are sent through the FED-portal for monitoring and storage in DCS. Online visualization of events and calibration data is enabled via the HOMER interface and allows to monitor the performance of the detectors in the ACR. This is continuously repeated during the run, and a notification about new DCS data in the HCDB is percolated through the analysis chain after each update. End of a run: At the end of a run, ECS issues again a special event, called End-of-Run (EoR). The event is percolated through the analysis chain and notifies each component to terminate. This phase is called completing, because it can take some time until all events are worked off and until the HLT is ready for the next run. During this time the Shuttle portal collects all freshly produced calibration objects, fills them in the FXS and stores additional meta data in the MySQL DB. As soon as this is finished, the ECS-proxy signals to ECS that the Offline Shuttle can start collecting HLT data. 7

9 After the end: Finally, the Offline Shuttle can contact the MySQL DB and the FXS on the corresponding HLT portal node and fetch the new data for the OCDB. The HLT cluster can already be used for the next run, since the fetching does not require actions from the HLT side. 4. Status and Performance These interfaces are in different stage of development. Most of them have been implemented and are in the test phase, which leads to an ongoing optimization and fine tuning of the different interface. The ECS-proxy has been implemented over a year ago and its functionality has been tested widely in tests with ECS and during the TPC commissioning in June The Shuttle portal and the Taxi are written and now deployed for performance tests and enhancements. First measurements indicate that they will do their job according to the requirements. The Pendolino is implemented without the PredictionProcessor and executing currently performance tests as well. At the moment the Pendolino takes 9 seconds to fetch 250 different datapoints, using all 250 datapoint names in one request. A soon to come upgrade in the Amanda server, which avoids a detour in the request over PVSS will bring further speed enhancements. The PredictionProcessor interface is in the final discussion and a first prototype using the TPC PredictionProcessor is about to be implemented soon. The FED-API of the FED-portal is implemented and waiting to be tested in the PubSub (PublishSubscriber) framework. Inclusion in the corresponding PVSS panels is pending. The HOMER has been implemented a while ago and widely tested: Last time in the setup of the TPC commissioning. The HLT has been able to monitor the TPC online during the commissioning in the ACR, the results are very promising. A combined test with all interfaces is pending, but scheduled for the full dress rehearsal in beginning of November Summary The ALICE HLT consists of a large computing farm with approx 1000 computing units. Fast connections guarantee high performance throughput of data. The layout of the cluster matches the structure of the ALICE detectors and their analysis steps. Interfaces to other parts of ALICE allow for data exchange with online and offline systems. Current run conditions are read from DCS, calibration settings fetched from Offline. Connections in the vice versa direction allow for feeding back new data. An interface to AliEve allows to visualize processed events online. External cluster control and synchronization is achieved via the ECS-proxy. The framework, presented in this article, enables the HLT for detector performance measurements and physics monitoring, as well as calibration calculation online. The HLT will be able to provide all required data for the analysis software performing first physics in ALICE Acknowledgments The development of these HLT interfaces have been accompanied by very good and fruitful cooperations with the collaborations of the connected systems in ALICE. The ALICE HLT project has been supported by the Norwegian Research Council (NFR). References [1] ALICE Collaboration, ALICE Technical Proposal for A Large Ion Collider Experiment at the CERN LHC, CERN/LHCC (1998) [2] ALICE Collaboration, ALICE Technical Design Report of the Trigger, Data Acquisition, High-Level Trigger, and Control System, ALICE-TDR-10, CERN-LHCC (2003) pp [3] Steinbeck T M et al New experiences with the ALICE High Level Trigger Data Transport Framework, in Proc. Computing in High Energy Physics Conf (CHEP04) Interlaken, Switzerland, 8

10 [4] Steinbeck T M et al 2002 An object-oriented network-transparent data transportation framework, IEEE Trans. Nucl. Sci. 49 (2002) pp [5] Steinbeck T M et al 2002 A Framework for Building Distributed Data Flow Chains in Clusters, Proc. 6th International Conference PARA 2002 Espoo, Finland, June 2002, Lecture Notes in Computer Science LNCS 2367, pp , Springer-Verlag Heidelberg [6] ALICE Off-line project, [7] Richter M et al, High Level Trigger applications for the ALICE experiment, submitted to IEEE Trans. Nucl. Sci. [8] Lara C, The SysMes architecture: System management for networked embedded systems and clusters, Date 2007 PhD Forum, Nice, France (2007), [9] Röhrich D and Vestbø A, Efficient TPC data compression by track and cluster modeling, Nucl. Instrum. Meth. A566 (2006) pp [10] Lindenstruth V et al, Real time TPC analysis with the ALICE High Level Trigger, Nucl. Instrum. Meth. A534 (2004) pp [11] [12] Bablok S et al, ALICE HLT interfaces and data organisation, Proc. Computing in High Energy and Nuclear Physics Conf (CHEP 2006) Mumbai, India, ed Banerjee S vol 1 Macmillian India Ltd (2007) pp [13] Gaspar C et al, An architecture and a framework for the design and implementation of large control system, Proc. ICALEPS 1999, Trieste, Italy [14] ALICE DCS Amanda project, [15] Colla A and Grosse-Oetringhaus J F, Alice internal note describing the Offline Shuttle mechanism (about to be published) [16] Tadel M and Mrak-Tadel A, AliEVE ALICE Event Visualization Environment, Proc. Computing in High Energy and Nuclear Physics Conf (CHEP 2006) Mumbai, India, ed Banerjee S vol 1 Macmillian India Ltd (2007) pp