Architectural Design and Supervisor RoI Builder

Transcription

1 Architectural Design and Supervisor RoI Builder T/DAQ Organization The Pilot Project Supervisor & RoI Builder

2 DAQ and HLT Construction Completed Dec (reduced processing power) TLAS Integration of detectors with T/DAQ/DCS Dec completed Major T/DAQ Milestones LVL1 Technical Design Report June 1998 Technical Progress Report and Work Plan June 1998 for DAQ, EF, LVL2 and DCS Technical Proposal on DAQ and HLT Dec Technical Design Report on DAQ and HLT June 2001 LVL1 Trigger Construction Completed Dec (excluding on detector electronics) Stand-alone DCS System Completed Jan. 2003

3 T/DAQ IB Members and Responsibilities Institute Representative address L1 L2 DAQ/EF DCS Argonne National Lab Robert BLAIR 1 1 Bern Klaus PRETZL pretzl@lhep.unibe.ch NEW SINCE IMOU 1 Birmingham John GARVEY j.garvey@bham.ac.uk 1 Bucharest Mihai CAPRINI CAPRINI@CPPM.IN2P3.FR 1 CERN Livio MAPELLI MAPELLI@MAIL.CERN.CH Copenhagen, NBI John Renner HANSEN renner@nbi.dk 1 1 Cracow Zbigniew HAJDUK HAJDUK@VXCERN.CERN.CH 1 Edinburgh Owen Boyle owen.boyle@cern.ch NEW SINCE IMOU 1 1 Geneva Allan CLARK clarka@mail.cern.ch 1 Genova Paulo MORETTINI morettini@ge.infn.it 1 1 Heidelberg Eike-Erik KLUGE ekluge@mail.cern.ch 1 Innsbruck Dietmar KUHN DIETMAR.KUHN@UIBK.AC.AT 1 1 Istanbul Arif MAILOV amailov@afsmail.cern.ch NEW SINCE IMOU 1 Jena Andreas REINSCH REINSCH@UNI-JENA.DE JINR, Dubna Vladislav KOTOV VKOTOV@VXCERN.CERN.CH 1 KEK Hiroyuki IWASAKI iwasakih@kekvax.kek.jp 1 1 Kobe Hiroshi TAKEDA takeda@phys.kobe-u.ac.jp NEW SINCE IMOU 1 Kyoto University Hiroshi SAKAMOTO sakamoto@scphys.kyoto-u.ac.jp NEW SINCE IMOU 1 Lecce Ornella PALAMARA PALAMARA@LE.INFN.IT 1 Lisbon Amelia TEIXEIRA MAIO AMELIA@VAXLIP.LIP.PT 1 Liverpool Robert DOBINSON Bob.Dobinson@MACMAIL.cern.ch 1 1 London, QMW Eric EISENHANDLER e.eisenhandler@qmw.ac.uk 1 London, RHBNC John STRONG J.STRONG@RHBNC.AC.UK 1 1 London, UCL Peter CLARKE clarke@mail.cern.ch 1 1 Mainz Lutz KOPKE koepke@dipmza.physik.uni-mainz.de 1 1 Manchester Richard HUGHES JONES RICH@V1.PH.MAN.AC.UK 1 1 Mannheim Reinhard MAENNER MAENNER@MP-SUN1.INFORMATIK.UNI-MANNHEIM.DE 1 1 Marseille Francois ETIENNE ETIENNEF@VXCERN.CERN.CH 1 Michigan State Maris A. ABOLINS abolins@pa.msu.edu 1 1 Moscow State Serguei SIVOKLOKOV SIVOKL@mail.cern.ch 1 Nagasaki Yasushi NAGASAKA nagasaka@elc.nias.ac.jp NEW SINCE IMOU 1 Naples Sergio PATRICELLI patricelli@na.infn.it NEW SINCE IMOU 1 NIKHEF Joseph VERMEULEN i73@nikhef.nl Novosibirsk Iouri MERZLIAKOV YIM@VXCERN.CERN.CH 1 Pavia Valerio VERCESI VERCESI@COBRA1.PV.INFN.IT 1 Prague AS & CU Maria SMIZANSKA msmizans@mail.cern.ch 1 Protvino Serguei KOPIKOV KOPIKOV@VXCERN.CERN.CH 1 RAL Charles Norman GEE C.N.P.Gee@RL.AC.UK Rome I Emilio PETROLO PETROLO@ROMA1.INFN.IT 1 1 Rome II Rinaldo SANTONICO santonico@roma2.infn.it 1 Saclay Jiri BYSTRICKY BYSTRI@HEP.SACLAY.CEA.FR 1 1 Sheffield Chris BOOTH Christopher.Booth@cern.ch NEW SINCE IMOU? 1 Shinshu Tohru TAKESHITA tohru@azusa.shinshu-u.ac.jp NEW SINCE IMOU 1 St Petersburg Iouri RIABOV RYABOV@LNPI.SPB.SU 1 1 Stockholm Sten HELLMAN Sten.Hellman@physto.se 1 Technion Simon ROBINS simon.robins@cern.ch NEW SINCE IMOU 1 1 Tel Aviv Halina ABRAMOWICZ HALINA@POST.TAU.AC.IL 1 1 Tokyo ICEPP Tomio KOBAYASHI tomio@icepp.s.u-tokyo.ac.jp NEW SINCE IMOU 1 Tokyo MU Ryosuke HAMATSU hamatsu@comp.metro-u.ac.jp NEW SINCE IMOU 1 UCI Andrew James LANKFORD lankford@lankford.ps.uci.edu 1 1 Udine Fabrizio Scuri Fabrizio.Scuri@cern.ch NEW SINCE IMOU 1 Weizmann Lorne LEVINSON FHLEVINS@WICC.WEIZMANN.AC.IL 1 1 Wisconsin Sau Lan WU wu@wisconsin.cern.ch 1 1

4 National Commitments LVL 1 LVL 2 DAQ / DCS total trigger trigger evt. filter Armenia 0 Australia 0 Austria Azerbaijan 0 Belarus 0 Brazil 0 Canada 0 Czech Republic Denmark Finland 0 France IN2P3 0 France CEA Georgia 0 Germany BMBF Greece 0 Israel Italy Japan Morocco 0 Netherlands Norway 0 Poland Portugal Romania 0 Russia + JINR Slovak Republic 0 Slovenia 0 Spain 0 Sweden Switzerland Turkey United Kingdom US DoE + NSF CERN total

5 Level 1 Trigger Participating Institutes Germany Heidelberg Mainz Italy Naples Rome I Rome II Israel Haifa Tel Aviv Weizmann Japan KEK Kobe Kyoto UE Shinshu Tokyo ICEPP Tokyo MU Sweden Stockholm UK Birmingham London QMW RAL CERN

6 TLAS TTC Distribution Backbone (1310 kchf, Dec. 2002) CERN (100%) Commitments to LVL1 Calorimeter Trigger Processor (7640 kchf, Dec. 2002) Birmingham, London QMW, RAL (49%); Heidelberg, Mainz (38%); Stockholm (8%); not covered (5%) Muon Trigger Processor for RPC System (2605 kchf, Dec. 2002) Naples, Rome I & II (100%) Muon Trigger Processor for TGC System (3340 kchf, Dec. 2002) KEK, Kobe, Kyoto, Shinshu, Tokyo ICEPP & MU (91%); Haifa, Tel Aviv, Weizmann (9%) Muon Central Trigger Processor (675 kchf, Dec. 2002) CERN (100%) Central Trigger Processor (535 kchf, Dec. 2002) CERN (100%)

7 TLAS Level 2 Participating Institutes Austria Innsbruck Chech Republic Prague AS & CU Denmark Copenhagen France CEA/Saclay Germany Mannheim Italy Genova Lecce RomeI Israel Haifa Tel Aviv Weizmann Netherlands Poland NIKHEF Cracow

8 LVL2 Participating Institutes Cont. Russia Moscow SU UK Edinburgh London UCL Liverpool Manchester London RHBNC RAL US Argonne U of Wisconsin UC Irvine Michigan State U CERN

9 Commitments to LVL2 LVL2 Calorimeter Trigger (1870 kchf, Dec. 2003) Argonne, Michigan State (45%); Mannheim (11%); Saclay (45%) LVL2 Muon Trigger (1290 kchf, Dec. 2003) CERN (16%); Genova, Lecce, Rome I & II (53%); Mannheim (28%); Haifa, Tel Aviv, Weizmann (5%) LVL2 tracking trigger (4595 kchf, Dec. 2003) CERN (22%); Copenhagen (11%); Cracow (3%); Mannheim (10%); London RHBNC & UCL, Manchester, RAL (18%); NIKHEF (7%); Prague AS & CU (1%); UC Irvine, Wisconsin (28%) LVL2 global trigger (1590 kchf, Dec. 2003) Argonne, Michigan State (18%); CERN (19%); Genova, Lecce, Rome I (14%); Liverpool, Manchester, RAL (19%); Saclay (31%) LVL2 Supervisor/RoI Builder (845 kchf, Dec. 2003) Argonne, Michigan State (100%)

10 Data Acquisition and Event Filter TLAS Austria Innsbruck Denmark Copenhagen France CEA Saclay Germany Mainz Italy Pavia Udine Netherlands NIKHEF Portugal Romania Russia Bucharest Novosibirsk JINR Protvino St. Petersburg NPI

11 Data Acquisition and Event Filter cont. Switzerland Bern Geneva Turkey Ankara Istanbul UK US Edinburgh London UC Sheffield Argonne Wisconsin Liverpool Manchester UC Irvine London RHBNC RAL Michigan State CERN

12 Commitments to DAQ/EF DAQ Readout (including readout buffers, local DAQ and interfaces to other systems) (9275 kchf excluding Common Project items, Dec 2003) Argonne, Michigan State, UC Irvine, Wisconsin (7%); Bern, Geneva (18%); CERN (22%); Copenhagen (5%); Edinburg, Liverpool, London RHBNC & UCL, Manchester, RAL, Sheffield (12%); Pavia, Udine (12%); Ankara, Istanbul (1%); NIKHEF (3%); Saclay (20%) Event Builder (3710 kchf, Dec 2003) Bern, Geneva (32%); CERN (27%);KEK, Nagasaki (27%); Saclay (18%) Event Filter and Back-End DAQ (5675 kchf excluding Common Project Items, Dec. 2003) Bern, Geneva (19%); CERN (25%); Pavia, Udine (23%); Innsbruck (5%); Ankara, Istanbul (1%); JINR (1%); KEK, Nagasaki (9%); Lisbon (5%); Mainz (14%); NIKHEF (1%)

13 Detector Control System 1720 kchf, excluding Common Project Items, Jan Netherlands: NIKHEF 12% CERN 93% Russia St. Petersburg NPI manpower contribution only

14 Guidelines Pilot Project Definition month program starting in March 98 Coherent and complementary activities focused on main issues Integrate and coordinate present and future work of LVL2 group Inputs URD Lessons from Demo Program Goals Demonstrate full LVL2 functionality Select technology candidates Define architecture and prepare integration with DAQ/EF

15 Critical Issues for LVL2 Data Collection from ROB s RoI Mapping Preprocessing Data flow and control protocols Data Transfer: via Supervisor or Processor Separate or integrated control network Processor allocation strategy Architectural Considerations Boundary between LVL2 and EF Can LVL2 and EF use same network technology?

16 Pilot Project Timeline TPR TP TDR Architecture choice Technical choices Demonstrators Pilot project Technology candidates H L T Functional Components Software testbeds System integration DAQ-1 I `N T E G R A T I O N High LevelTrigger Prototype & Design phase Final spec.

17 Pilot Project Activities Matrix Pilot Project Functional Components Software Testbeds System Design ROB Complex Supervisor Reference Application Modelling Integration Inputs to ROB URD MultiROB Processors, i/f s & Networks Processors Co-processors Networks Technology watch ROI Builder ROI distributors Small Full slice Training samples Algorithm benchmarks Framework Larger commercial Network Menus & strategies Evaluation of: - SW options - Performance requirements Paper Computer URD Cost New ideas

18 ATM Testbed Workplan Organizer Maris Abolins, Bob Blair, Patrick Le Du Need A large ATM switch is needed to study system performance. Full functionality should be demonstrated, including the operation and performance of LVL2 trigger algorithms. Objectives Establish a 64-port ATM testbed for system performance studies Approach Construct a large ATM testbed with commercial components (PCs and VME SBCs). Install 16 ATM ports in a 64-port FORE switch at Saclay. Install a 16-port FORE switch at ANL/MSU. Import U.S. material to Saclay to complete a 32-port system. Transport to CERN to install a port testbed combining components from the ATM and Ethernet testbeds. Duration June 98 - Dec 99 Deliverables Saclay testbed with 16 ATM ports Jun 98 U.S. testbed with 16 ATM ports Jun 98 Milestones U.S. testbed operational Jun 98 Combined 32-port testbed at Saclay Dec 98 Combined port testbed at CERN Mar 99 Dependencies Availability of RD31 ATM modules and components. Availability of Ethernet testbed components. Agreement with DAQ/EF group for additional components. Partners ANL, CERN, MSU, RHBNC, Rome, Saclay Risks No risk. Commercial components. Status/date 27/4/98 Saclay testbed with 12 ports equipped. 27/4/98 U.S. testbed: FORE switch and compnonents ordered. 27/4/98 RD31 ATM switch available Jan-June 1999.

19 Workplan to Integrate RoI Builder and Multi-supervisor Organizer John Dawson Objectives Demonstrate event routing of Level 1 information and CPU scaling of supervisor Approach Construct routing hardware using S-link input and implementing hardware event assembly for ROI builder. Measure performance versus number of supervisor CPU's for scaling. Duration June - Dec 1998 Deliverables Hardware router for ROI builder Nov 98 Supervisor scaling document Dec 98 Milestones All orders placed for components Apr 98 LVL1/LVL2 Interface Fully Specified Apr 98 Preliminary Design of RoI Builder May 98 US testbed assembled at ANL Jun 98 US testbed fully operational Jul 98 ANL testbed meeting Jul 98 S-Link Performance Measurements Aug 98 RoI Builder Design Complete Aug 98 Scaling Tests of Supervisor Start Sep 98 RoI Builder Prototype Ready Oct 98 Supervisor Scaling tests complete Nov 98 RoI Builder to Saclay Dec 98 Partners ANL & MSU

20 Summary of Pilot Project Milestones LVL1/LVL2 Interface Fully Specified Preliminary Design of Pilot RoI Builder Complete Users Requirement Document (URD) form and content defined Supervisor Emulator Design Ready Start S-Link Performance Measurements U.S. Pilot Testbed Operational Final Design of Pilot RoI Builder Complete Start Scaling Tests of Pilot Supervisor Supervisor Emulator Written S-Link Performance Measurements Complete System Design Document (SDD) for 1st baseline design ready New Cost Estimate Ready Generic models for selected test setups ready in Simdaq Start Integration of Reference Software Pilot RoI Builder Complete Draft 0,1 of SDD Ready Draft 0.1 of Design Costing Document (DCD) Ready Supervisor Scaling Tests Finished WEB document on event & control records LVL1 to LVL2 & LVL2 to LVL3 first draft ready 15-Apr May Jun Jul Jul Jul Aug Sep Sep Sep Sep Sep-98 1-Oct Oct Oct Oct Oct-98 1-Nov Nov-98

21 Pilot Project Milestones - continued LVL2 TTC record and interface to LVL1 Trigger Throttle Defined Pilot RoI Builder & Switch shipped to Saclay Reference Software Integration Complete Pilot Ethernet SRB ships to CERN Details of LVL1 Throttle Specified SRB Interface to LVL2 and Event Filter Defined URD Updated Proposal for Common Supervisor for LVL2 and EF complete URD Version 1.0 Review Complete Final Baseline Design Complete Costing of Final Baseline Design Approved Comparison of performance of selected test setups with models complete Version 1.0 of SDD Ready Version 1.0 of DCD Ready Documentation of modeling of selected test setups and comparison with actual performance complete 30-Nov Nov Dec Jan-99 1-Feb-99 1-Mar-99 1-Apr-99 1-Jun Jun Jun Jun Jun-99 1-Jul Jul-99 1-Oct-99

22 U.S. ATM Testbed

23 U.S. ATM Switch and Hub

24 Supervisor and RoI Builder major contributions by Bob Blair, John Dawson, Yuri Ermoline and Jim Schlereth

25 LVL1/LVL2 Layout LVL1 MuonToCTP Interface LVL2 i/f CTP LVL2 i/f e/γ RoI Collector Calorimeter Trigger τ jets RoI Collector RoI Collector E T RoI Collector L1A LVL1 information: RoI generation algorithms Trigger tower mapping to modules/crates RoI read-out organisation in modules/crates RoI address encoding ROB Complex ROB Complex LVL2 - RoI Builder/Supervisor In FIFO ROI CPU NW i/f Input Router In FIFO In FIFO ROI ROI CPU CPU NW i/f NW i/f In FIFO ROI CPU NW i/f ROBIN ROBIN NW i/f ROBIN ROBIN Controller ROBIN ROBIN NW i/f ROBIN ROBIN Controller NetWork NW i/f LVL2 CPU NW i/f LVL2 CPU NW i/f LVL2 CPU LVL2 CPU Farm NW i/f LVL2 CPU NW i/f LVL2 CPU NW i/f LVL2 CPU

26 LVL1 RoI Readout LVL1 ROI Read-out e/γ, τ jets E T, E T, µ, CTP RoI Read-out ROC RoI Read-out ROC RoI Read-out ROC Assumptions One Readout Link per LVL1 Crate --> 12 Links Same links through whole system (calorimeter, muon, CTC) Zero-suppressed data is transferred Questions Link Technology Data Formats

27 LVL1 Calorimeter Data Formats * Error Word 2 32 TLAS * End of Event (Trailer) 32 Item: Length Comments: (bits): * Start of Event (Header) 32 * L1ID bits * BCID bits * RoI data RoI Type 4 e.g. 1 = e/g RoI eta-phi index x 0.1 RoI Threshold set 8 * Error Word 1 32 Word # Content 1 Header 2 L1ID 00 3 BCID N+4 RoI data Error Word 1 N+5 Error Word 2 N+6 Trailer N - number of RoIs

28 LVL1 Muon Data Format * Error Word 2 32 TLAS * End of Event (Trailer) 32 * Start of Event (Header) 32 * L1ID bits * BCID bits * RoI data (16 candidates maximum) Sector number sectors Subsector number 5 Pt value 3 * Error Word 1 32 Word # Content 1 Header 2 L1ID 00 3 BCID N+4 RoI data Error Word 1 N+5 Error Word 2 N+6 Trailer N - number of muon RoIs

29 Supervisor & RoI Builder FPGA card Event is assembled in card and fed to RoI Processors in turn. In 9U VME VME interface permits emulation of LVL1 input. 12 S-Links from LVL1 RoI Builder S-Link RoI Processor RoI Processor RoI Processor RoI Processor Add Processors as needed To Network

30 TLAS Important Questions Is S-Link suitable? Tests in progress. Are current processors adequate for 100 khz? Is the design robust and fault tolerant? Do we need an event size cutoff? Yes - 32 RoI s. Is a busy from LVL2 necessary and how will it be implemented? Yes... How will we test the device? Do we need a LVL1 emulator? Design in progress.