The CMS L1 Global Offline Software Vasile Mihai Ghete Institute for High Energy Physics, Vienna, Austria Seminar 08-09 June 2009, HEPHY Vienna
CMS experiment Tracker pixel detector: 3 barrel layers, 2 discs / endcap silicon strip detector: Barrel: layers: 4/inner, 6/outer Endcaps: discs: 3/inner, 9/outer inside the solenoid: 4 T magnetic field. Calorimetry: inside the solenoid CMS General purpose experiment LHC, CERN Intercation Point 5 (IP5) high resolution lead-tungstate crystal EM calorimeter: η < 3 endcap preshower hadron calorimeter: η < 5 Muon spectrometer: superconducting solenoid, 4 T. barrel: DTs and RPCs endcaps: CSCs and RPCs. 2
ing at CMS Level-1 (L1) Custom hardware trigger Reduce event rate from 40 MHz to 100 khz (complete) or 50 khz (DAQ staged). Total processing time: 3 μs. Selection: based on coarse information from calorimeter and muon detector High Level (HLT) tasks: reduce the event rate to O(100) Hz cover widest possible range of discovery physics Main characteristics: use the offline software framework Output event rate: 100 Hz 8 slices, each slice allow 12.5 khz Software trigger: Average processing time: ~ 40 ms. Has access to all event data, with full precision and granularity L1 seeded (conditional reconstruction) partial (regional) reconstruction 3
Introduction to CMSSW CMSSW CMS offline software built around a Framework, an Event Data Model (EDM) and Services Written in C++ language One executable cmsrun and many plug-in modules which run algorithms. The Full Framework uses a modular architecture concept Module: component to be plugged into cmsrun via shared object libraries It s a unit of clearly defined event-processing functionality Event Data Model (EDM) based on the event: Single entity in memory: edm::event C++ object container Modular content (EDM products) Applications are steered and defined via Python job configurations 4
Data Flow in CMSSW Software Bus Model Communication among modules only via the event Objects (products) in the event, once stored, are read-only 5
CMSSW Modules Source Reads in an Event from a ROOT file or create empty events EDProducer Reads in data from the Event in one format, produce something from the data, and output the product, in a different format, into the Event. EDFilter Reads data from the Event and returns a Boolean value that is used to determine if processing of that Event should continue for that path. EDAnalyzer Reads data from the Event but is neither allowed to add data to the Event nor affect the execution of the path. EDLooper Control 'multi-pass' looping over an input source's data. OutputModule Reads data from the Event, and once all paths have been executed, stores the output to external media. 6
Event Setup Event Setup Contains information about the detector environment and status. This information (non-event data) is not tied to a given event, but rather to the time period for which it is valid (interval of validity, IOV) 7
Calorimeter HF HCAL ECAL Regional Calorimeter Global Calorimeter MIP+ Muon CSC DT RPC Local CSC Local DT CSC Track Finder DT Track Finder Pattern Comparator 4+4 µ 4µ 4µ ISO bits Global Muon e, J, ET, HT, ETm, Htm,NJ, HF 4µ Supervisor 40 MHz Pipeline, Latency < 3.2 µs Level-1 System (with MIP/ISO bits) Global max. 100 khz L1 Accept 8
GCT GMT GT - L1Extra GCT unpacker GCT readout collections FED GCT candidates GCT Mip/Iso bits GMT PSB1-3 GT GMT candidates GMT Readout Collection bx -1 bx 0 GMT bx 1 GMT L1Extra collection HLT GTFE GT unpacker GMT readout readout readout record record record rr rrrr rrrr PSB5 rr rr PSB6rrrr L1Extra GT Readout Collection GTFE rr bx -1 bx 0 rr bx 1 FDL rr rr rr PSB1rrrr rr PSB2rrrr rr PSB3rrrr Ref GMTrc 9
GT Offline Software - Components DataFormats/L1Global Define the format of the EDM products produced by the GT Two records: GT DAQ record, GT EVM record Common between hardware and Monte Carlo simulation CondFormats/DataRecord, CondFormats/L1TObjects Define the event setup records for the GT L1GT Unpacker/Packer Translate from FEDRawDataCollection to GT DAQ and EVM record (unpacker: RawToDigi) and vice-versa (packer: DigiToRaw) L1 GT Emulator Emulate bit-by-bit the functionality of the GT hardware L1 GT Analyzer report, various analyzers L1 GT DQM (Data Quality Monitoring) 10
GT Unpacker FEDRawDataCollection Raw data - binary data produced by hardware Used also in MC Unpacker (RawToDigi): converts from raw data format to digitized data format Uses the data formats defined in DataFormats Modular, can unpack partially the records Packer (DigiToRaw): converts from digitized data to raw data Used together with unpacker to test the process DigiToRaw -> RawToDigi: final digis must be identical with initial digis Used in MC to convert the HLT required records in raw format, such that HLT starts in MC and in data from the same format. 11
L1 Emulation L1 emulation task Emulate bit by bit each L1 subsystem Requirements Modular design Use the same input and the same input format as the hardware Produce same output as hardware, with identical format If possible and required, produce additional information, not saved by the hardware Follow the modularity of the hardware subsystems Integrated in CMS modular offline software framework Intermediate information Information not made available in the hardware Configuration see separate slide Stringent timing requirements for all modules, especially those running online for each event. 12
Calorimeter HCAL TPG ECAL TPG HF HCAL ECAL Emulator Emulator Regional Calorimeter RCT Emulator Global Calorimeter GCT Emulator MIP+ Muon CSC DT RPC CSC TPG Local Emulator CSC DT TPG Local Emulator DT CSC CSCTrack TF Finder Emulator DT Track DT TF Finder Emulator RPC Pattern Comparator Emulator 4+4 µ 4µ 4µ ISO bits GMTMuon Emulator Global e, J, ET, HT, ETm, Htm,NJ, HF 4µ Supervisor 40 MHz Pipeline, Latency < 3.2 µs L1 Emulation - Components (with MIP/ISO bits) Global Emulator Global max. 100 khz L1 Accept 13
L1 Emulation Configuration Configuration requirements Use the same configuration as the hardware Enable to overwrite for emulators some hardware parameters L1 Hardware configured from OMDS database Not accessible offline Transfer of the configuration special O2O application to run in Run Control From OMDS to offline DB (ORCOF) From OMDS to DB used for CMSSW online (ORCON) Set also the Interval of Validity (IoV) - run dependent Associate configuration with data 14
L1 Global - Hardware L1 Global PSB PSB DT CSC RPC 4 muons GCT GMT EVM payload GTL GTFE S-LINK DAQ FDL L1A L1AOUT L1A Signal to 32 Detector Partitions 32 TTCci GT Emulator CMS & FED objects: GCT, GMT Technical triggers - no objects Logical conditions - no objects Task TIM TCS Hardware design - cards Input evaluates in parallel up to 128 algorithms (physics trigger requirements, given in the L1 Menu) based on input objects Algorithm = logical expression of conditions Condition= object template for physics requirement (up to 4 objects in a condition) Produce L1 Accept, GT DAQ and EVM records (Acronyms in backup slides) 15
Emulator execution logic Data format definition Configuration producers Configuration definition L1 Global Emulator 16
Use Cases for L1 Emulators Validation of trigger hardware Run emulator on same input data as hardware and compare results Run pattern tests in emulator and trigger hardware and compare results Monte Carlo simulation of L1 Can test thoroughly choosing adequate pattern tests Monitoring of trigger hardware Use validated emulators to monitor trigger hardware Implies running the emulators during data taking Probably not all emulators for each event due to timing constraints For existing hardware and configuration For new proposed objects menu development L1 seeding of HLT Use the GT emulator to find L1 objects which actually triggered Too complex task to be done on FPGA Start HLT algorithms from these L1 objects Determine expected rates Propose prescale factors Conditional and regional reconstruction Avoid volunteers 17
Validation and Performance Improvement Modules running online have stringent timing requirements Average HLT time per event 40 ms Streamline the code Tailored configuration: use only what is needed online Avoid string comparisons, string parsing Avoid useless copying Careful declaration of container size to avoid resizing Choose carefully standard containers Extensive caching of event setup IgProf (Ignominious Profiler) Valgrind Example: Global Methods Tools used for optimization Initial release: ~40 ms! Next optimization: ~14 ms Next optimization: ~7 ms Streamlined code Run only 1 bunch cross in event (L1A) instead of 3 Mix of changes (see link column) Next optimization: 1-3 ms Validation - iterative process Use data collected in cosmic Global Runs, MC samples Compare results 18
Example: Global Data FDL Data vs Emulator Emulator 20 0 C R U ZE T G ea rl y lr un No Bx number in Emulator Technical triggers in hardware FinalOR wrong key for emulator Local Bx in emulator lo ba 8 Discrepancies: all fixed 19
Online DQM Run 98532 20
Backup slides 21
Acronyms HCAL Hadronic Calorimeter HF Hadronic Calorimeter Forward ECAL Electromagnetic Calorimeter RPC Resistive Plate Chambers CSC Cathode Strip Chambers DT Drift Tubes Chambers TF Track Finder TP Primitives TPG Primitive Generator RCT Regional Calorimeter GCT Global Calorimeter GMT Global Muon GT Global 22
Acronyms (contd.) e, μ, Etm, HTm, HF, - trigger objects (electron, muon, missing transverse energy, missing calibrated transverse energy, HF objects) ISO, MIP isolation and minimum ionizing particle quantities Bx bunch crossing FED Front End Data DAQ Data Acquisition System EVM Event Manager: controls data flow of events in DAQ GTL Global Logic card FDL Final Decision Logic card PSB Pipeline Synchronize Buffer card GTFE Global Front End card TCS Central Control System TIM Timing card 23
Acronyms (contd.) L1A - signal to read all Front End buffers of CMS S-LINK64 Fast data Link extended to 64 bits FPGA Fully Programmable Gate Array OMDS Online Master Data Storage database ORCON Offline Reconstruction Online subset database ORCOF Offline Reconstruction Offline subset database CRUZET CMS Run at Zero Tesla global run CRAFT CMS Run at Four Tesla global run 24