LHCb Online System BEAUTY-2002

Transcription

1 BEAUTY th International Conference on B-Physics at Hadron machines June antiago de Compostela, Galicia (pain ) Niko Neufeld, CERN EP (for the LHCb Online Team) 1

2 Mission The LHCb Online system provides the acquisition and storage of the raw-data the processing power for the high level trigger algorithms the distribution of timing signals and trigger decisions to the front-end electronics the control and monitoring of the whole experimental facility The LHCb Online system strives for: uniformity, homogeneity, simplicity CERN, EP 2

3 LHCb trigger LHC produces > B-mesons / s, out of which < 10 are interesting (CP-violation) LHCb applies four level trigger scheme to efficiently select interesting B-events L0 (hardware) 40 MHz -> 1 MHz L1 (software) use Vertex Detector Data to fully reconstruct secondary vertices and select B- events: 1 MHz -> 40 khz DAQ takes over here L2 & L3 (software) use all event data with partial/full reconstruction 40 khz -> 200 Hz ee the presentation by H. Dijkstra tomorrow morning: The LHCb Trigger ystem CERN, EP 3

4 cope of the ystem The LHCb Online ystem has three main components Data Acquisition (DAQ) ystem Timing and Fast Controls (TFC) ubsystem Experiment Control ystem (EC) Experiment Control ystem BX 40 MHz L0 Trigger L0Y 1 MHz L1 Trigger L0Y 1 MHz L1Y 40 khz Timing and Fast Control L0Y 1 MHz L1Y 40 khz Throttle Throttle Front-End Electronics 40 khz 100 kb 500+ Links Data Multiplexing 40 khz 100 kb ~60 Links Event Building 40 khz 100 kb ~60 Links Event Filtering 200 Hz 200 kb 1 Link Data torage CERN, EP 4

5 Context Permanent torage elected Events Display Information Operator Commands/ ettings tatus Throttle Trigger ystem Accelerator Infrastructure ystems tatus Information tatus Information Commands/ settings Trigger Decisions LHCb Online ystem Physics Data tatus afety alarms & ettings Event related control s data afety ystem Conditions Database Front-end Electronics Offline Analysis CERN, EP 5

6 Design Goals and Principles (1) Architecture-centred Technology independence implicity imple functionalities and protocols eparation of controls and data path Reliability calability Upgrade path to higher rates without change of architecture and protocols Uniformity As small a number of component types and technologies as possible Lower cost and easier maintenance CERN, EP 6

7 Design Goals and Principles (2) Main-stream technologies where possible (commercial components) Lower cost and improved reliability Homogeneous control system A common approach in the design and common tools and components for the implementation of the control of the Detector and the DAQ: Facilitates inter-domain integration Makes it easier to use: tandard features throughout the system Uniform Look and Feel Needs less manpower Allows an easier upgrade and maintenance upport Partitioning Higher operational and diagnostics efficiency CERN, EP 7

8 Data Acquisition Performance Requirements Average Physics Event-ize Average Level-1 Trigger rate Average RAW Data Rate Total CPU power for Online Data Processing Average Event ize to torage (RAW+ED data) Average Rate to torage Average Data Rate to torage 100 kb 40 khz 4 GB/s I kb 200 Hz 40 MB/s CERN, EP 8

9 ystem Architecture Timing and Fast Control Dataflow system Front-End Multiplexers Readout Units Readout Network ub-farm Controllers Event Filter Farm Experiment Control ystem General Computing Infrastructure Level 0 Trigger Fixed Latency 4.0?s Level 1 Trigger Variable Latency <2 ms VELO TRACK ECAL HCAL MUON RICH 40 MHz 1 MHz Timing L0 Level-0 40 khz & Fast Front-End Electronics Control L1 (TFC) Level-1 1 MHz Throttle LHCb Detector LAN Data Rates 40 TB/s 1 TB/s Front-End Multiplexers (FEM) 4 GB/s Front-End Links RU RU RU Read-out Units (RU) Read-out Network (RN) 4 GB/s FC FC ub-farm Controllers (FC) Control & Trigger Level 2 & 3 CPU CPU Monitoring Event Filter (EC) CPU CPU Variable latency L2 ~10 ms 40 MB/s L3 ~200 ms torage CERN, EP 9

10 Data Links and Protocols Only Point-to-point links between modules No shared busses (VME bus, etc.) Improved reliability and diagnostics capabilities Push protocol ource sends data to destination as soon as available ( no further synchronisation) Assumes that always sufficient buffer available at destination ( Throttle to stop trigger) Custom light-weight protocol (no TCP) No central Event Manager (Gb) Ethernet as link technology throughout the system Large market and large range of speeds Low price CERN, EP 10

11 Reading out the Front-end Electronics Reduce Number of Links (n:1) by merging event fragments Assign merged fragments to a sub-farm of the EFF (based on the event number) end them via a large Gigabit Ethernet witching Network Input 4:1 Data Merging Output RU/FEM Application Level 0 Trigger Fixed latency 4.0??? ss Level 1 Trigger Variable latency <2 ms Header Trailer One module to do all this the Readout Unit (RU) 40 MHz 1 MHz 40 khz 1 MHz Timing L0 & Fast Control L1 (TFC) Throttle FC CPU CPU LHCb Detector VELO TRACK ECAL HCAL MUON RU RU RU FC ub - Farm Controllers (FC) CPU CPU Level Front- --End Electronics Level Front-End Multiplexers (FEM) Front-End Links Read-out Units (RU) Read - out Network (RN) Trigger Level 2 & 3 Event Filter Variable latency L2 ~10 ms L3 ~200 ms RICH LAN Control & Monitoring (EC) Data Rates 40 TB/s 1 TB/s 4 GB/s 4 GB/s 40 MB/s torage CERN, EP 11

12 The Readout Unit Two implementations have been pursued (Backup) solution using FPGAs Prototypes produced, functional (Baseline) solution based on Network Processors (IBM NP4G3 or NP4G4) oftware-based, programmable solution very flexible oftware written and benchmarked by simulation and measurement Performance sufficient Network Processor based RU is a fully programmable 8-port Gigabit Ethernet witch with many powerful extra features 5.1V 3.7V GND Power Island Reset EC Interface 5.0V 3.3V 2.5V 1.8V I2C JTAG PCI Clock Island 125 MHz, MHz Throttle 512kx18 512kx18 RAM RAM LU CH DRAM Control PARITY DATA DATA D4 D0 D1 DRAM Data DRAM Control D3 D2 D1 D0 DRAM Data 32Mx4 32Mx4 32Mx4 PARITY DATA DATA D6 D6 D6 NP4G3 DMUs A B C D PCI DAL A PCI DAL A NP4G3 DMUs DRAM Control D3 D2 D1 D0 DRAM Data 32Mx4 32Mx4 32Mx4 PARITY DATA DATA D6 D6 D6 512kx18 512kx18 RAM RAM LU CH DRAM Control PARITY DATA DATA D4 D0 D1 DRAM Data Phy Phy Phy Phy Throttle Phy Phy Phy Phy A B C D 10/100 Base-T Ethernet RJ45 CERN, EP 12

13 Topologies for the central readout network Classical Banyan Topology Fully Connected Topology DAQ architecture requires that every Readout Unit can send to any ubfarm Controller ources Basic element is a 8- port Gigabit witch D D Basic tructure D D D D D D 64 x 64 port configuration ources x 72Destinations Uses fewer ports while reducing the average load on a single link Destinations CERN, EP 13

14 Local trigger (optional) L0 L1 Readout upervisor L0 Throttle switch L1 trigger system Readout upervisor TFC switch TFC Architecture Designed around RD12- TTC system L1 Throttle switch Readout upervisor D1 TTCtx D2 TTCtx Dn TTCtx L0 TTCtx L1 TTCtx Optical couplers Optical couplers Optical couplers Optical couplers Front-End Electronics DAQ LHC clock Throttle OR TTCrx L0 Trigger L0 Trigger splitter Clock fanout BC and BCR TTCrx L1 Trigger L1 Trigger splitter Front-End Electronics DAQ GP receiver Throttle OR TTC system Functional Components Clock transmission TFC signal generation Readout upervisor TFC signal distribution witches RD12-TTC ystem (TTCtx s, Optical Couplers, TTCRx s) Buffer overflow control Central in R (synchronous Part) Throttle signals and infrastructure (ORs and witches) Partitioning upport CERN, EP 14

15 EC view of the Online ystem DC Devices (HV, LV, GA, Temperatures, etc.) Detector Channels Experiment Control ystem Trigger Front End Electronics Readout Network Processing/Filtering torage DAQ External ystems (LHC, Technical ervices, afety, etc) CERN, EP 15

16 Experiment Control ystem In charge of the Control and Monitoring of: Detector Operations (ex low Controls) Gas, HV, LV, temperatures... Data Acquisition and Trigger FE Electronics, Event building, EFF, etc. Experimental Infrastructures Cooling, ventilation, electricity distribution,... Interaction with the outside world Magnet, accelerator system, safety system, etc. Based on the Joint Controls Project (JCOP) of the four LHC experiments JCOP itself based on a commercial CADA ystem PVII: distributed, network-based, multi-o, extensible CERN, EP 16

17 Generic W Architecture EC T.. LHC DC DAQ D tatus & Alarms Commands GA ubys1 DetDcs1 ubys2... DetDcsN DetDaq1 ubysn Dev1 Dev2 Dev3 DevN To Devices (hardware or software) Each node is able to:... Configure, monitor and control its children equence & Automate operations Recover errors Handle Alarms Filter and display alarms Partitioning Exclude one or more of its children User Interfacing Present information and receive commands CERN, EP 17

18 Board level electronics Electronics in barracks Front-ends, Readout Units, Timing and Fast Control components, etc. Credit Card PC s 66? 85? 12 mm 3 Pentium Compatible CPU Linux/DIM (no PV) I2C JTAG Bus I2C JTAG Bus Ethernet I2C JTAG Bus (~500+) CERN, EP 18

19 Test Beam Run Control CERN, EP 19

20 ummary The LHCb Online system consists of 3 main components: DAQ, TFC and EC The DAQ system has a simple push-through protocol over point-to-point to links uses Ethernet as link technology, hence leverages on a large market with the advantage of a wide range of speeds applies Network Processors in a single (home-made) module for the data-flow The TFC system provides timing and trigger decisions at 1 MHz avoids buffer overflows throughout the system The EC is uniformly covering all aspects of the experiment s operation provides a control hierarchy which includes automatic error-recovery, partitioning CERN, EP 20

21 Classical low Control High Voltages, Low Voltages, etc. Try to use Commercial HW (CAEN,WIENER,IEG,etc) Interface: OPC ervers (provided by manufacturer) Analog & Digital IO (temperatures, etc.) Commercial (e.g. WAGO) ELMBs / OPC ervers Others: PLCs (iemens, chneider) / OPC ervers Home made / DIM or OPC ervers Node FieldBus Node Experimental equipment Node PLC FieldBus CERN, EP 21

22 Front-End Electronics In Radiation Areas PEC (evolved from ATLA PAC) erial Protocol 10Mb/s lave is radiation tolerant ELMB CAN protocol (0.5Mb/s) I2C JTAG I2C JTAG Controls PC M PEC CAN CERN, EP 22 I (~1000+) I2C JTAG