HLT Infrastructure Commissioning

Transcription

1 HLT Infrastructure Commissioning Steven Robertson Institute of Particle Physics ATLAS NSERC Review Vancouver, B.C. November 14th, 2007

2 Outline Overview of ATLAS trigger system HLT hardware installation and commissioning schedule Trigger commissioning and technical runs Canadian HLT Testbed and online integration 2

3 Trigger Concept Trigger design motivated by need to minimize data movement and processing time Regions of Interest (ROIs) access data only in detector regions flagged by previous trigger levels/algorithms LVL1 implemented in hardware based on reduced granularity detector information LVL2 access data from individual detector Read Out Buffers (s), but only in ROIs EF uses event fragments from event building with access to full detector granularity but limited by processing time 3

4 ATLAS trigger and dataflow 4

5 HLT Overview SV: Level-2 Supervisor - controls Level-2 trigger DFM: Data-Flow Manager - controls event building pros: pseudo-ros buffers results from Level-2 trigger 5

6 Trigger Slices Trigger organized as as series of vertical slices representing classes of distinct trigger signatures Trigger algorithms executed per ROI, controlled by trigger steering Feature extraction (FEX) algorithms Hypothesis algorithms apply selection T2CaloJet L2 time L1 JETROI TrigL2JetHypo Level 1: TrigCaloCellMaker Sliding 0.8x0.8 window with 0.4x0.4 central cluster Level 2: One FEX algorithm (calorimeter jets with RCone = 0.4) Hypothesis Algorithm: Cut on jet ET EF TrigJetRec Event Filter: TrigCaloTowerMaker FEX algorithm: unpack calorimeter data, construct calorimeter towers, perform jet reconstruction TrigJetHypo Hypothesis algorithm: Cut on jet ET (See talk by B. Vachon) 6

7 HLT farms Maximum HLT input rate fixed by L1 hardware,output rate by ATLAS data storage constraints (i.e. doesn't scale with physics rate) HLT farm capacity dictates sophistication of trigger algorithms (i.e. what data can be unpacked and how much processing can be performed) Full HLT ~500 L2 PCs, ~1800 EF PCs (all multi-core), ~800 of these will be EF2 PCs capable of being used either as L2 or EF nodes Consistent with TDR total HLT capacity, but with somewhat different hardware 7

8 HLT preseries hardware Currently 181 EF2 PCs (~5.2 racks) installed in SDX1 mostly 2xClovertown(quadcore) with 1GB/core RAM, but also some 2xWoodcrest (dualcore) with 2GB/core Preseries hardware specifications and configuration similar to those in the RTI request e.g. 2x1Gb/s network, coldswappable power supplies, remote management via IPMI etc studies to evaluate need for more memory on HLT nodes (i.e. more than 1GB/core?) 8

9 Hardware acquisition schedule Large fraction of dataflow and online hardware has already been purchased: ROS, DFM, SFO, monitoring and online essentially 100% So far, only HLT hardware which is specifically needed for commissioning (technical runs, detector commissioning and cosmic-ray runs) has been purchased: <10% of overall HLT system Plan to expand to full system in as luminosity requires and funding permits Initial purchase for summer 2008 to bring total to >=36 racks for first LHC beams, but delay purchase as late as possible and potentially stage into early and late phases Full size of system will depend on physics needs, limited by system performance, and algorithm performance and timing (and of course, funding) 9

10 HLT Racks in SDX1 Empty racks (with power) already installed in SDX1 47U on upper floor central switches, dataflow, online, will ultimately house EF2s for EF or L2 52U on lower floor (currently preseries ) will house only EF racks Rack coolers delivered for upper floor but only partially installed Cooling is main limitation studying options for higher performance 10

11 EF2 Racks EF2 racks can be allocated as either L2 or EF farm resources depending on (time varying) need Each EF2 rack has: 1 x10 GbE connection to Dataflow network for L2 function 2 x 1 GbE connections to Backend network for EF function 1 x 1 GbE connection to control network RTI proposal requests funding for 10 EF2 racks with an extended purchase profile: maintain maximum flexibility for HLT system, but in practice the price difference between L2, EF and EF2 racks is a few % GE Data Link 10GE Data Link ROS PC ROS PC GE Control Link DataFlow Network Dataflow network C tr l. C o n c. PUs D a ta c o n c. SVs SFIs L2PUs BackEnd Network Backend Network XPU rack EFPs EFPs EFPs 11

12 HLT Hardware HLT will be built from off-the-shelf computing hardware Late purchase to exploit price/performance improvements Various options are being considered as they become available: 2006 reference hardware was 3GHz 2xWoodcrest (dual-core Intel Xeon) In early GHz 2xClovertowns (quad-core Intel Xeon) purchased for testing CERN Tier-0 purchasing twinmotherboard 2xClovertown (2.33GHz) New AMD quad-core chips currently being evaluated Blade solution not currently competitive with U1, but this may change in the future 12

13 HLT Hardware Commissioning TDAQ technical runs are the primary commissioning exercises for the DAQ and HLT Since December 2006, seven ~1 week runs Seven ATLAS Canada members have so far worked shifts during these runs, many others indirectly involved via e.g. data quality and HLT algorithm development Specific set of objectives during each run designed to test the integration of new features, from basic run control, HLT supervision (i.e. handling of dummy HLT algorithms), HLT algorithm performance, overall trigger menu, data quality monitoring and data flow to mass storage 13

14 Technical runs Use realistic mix of simulated (L1-preselected) events injected into TDAQ front end format identical to real data coming off the detector validate DAQ software and HLT integration in realistic data-taking environment and with HLT preseries hardware study timing, resource usage, networking, memory leaks, stability etc refine wetware interface for run control, monitoring and data quality define procedures for shifters, as well as for integration of software at Point 1 (i.e. validation in online framework) shift handover procedures, checklists, shift logging etc 14

15 Technical Runs Technical runs also provide the opportunity to exercise the DAQ and run control system, and in particular are (currently) the main mechanism for training of expert shifters and TDAQ operations experts 15

16 Online Monitoring Strong coupling to Canadian HLT algorithm development and data quality activities (talks by R. Moore, B. Vachon) Online Histogram monitoring (May 2007 technical run): 16

17 November 2007 Run Highlights Most recent technical run took place Nov 19-27: 5 racks of 31 HLT preseries PCs available (~7%) of full system In addition to mixed Monte Carlo, specially prepared MC samples and real cosmics data (from earlier detector commissioning run) were used for specific tests full 1031 trigger menu implemented (200+ trigger chains) full HLT from L2 to CASTOR (mass storage) implemented up-to-date HLT and offline software (i.e. no cheating) integration of many software tools and functionalities: TriggerTool and ORACLE database for management of trigger menu, prescales etc. PartitionMaker for defining resources required for particular data-taking configuration (previously required hand-editing of xml files by experts) Histogram monitoring many new features in online and offline handling of dataquality histograms 17

18 Nov 2007 technical run highlights Several memory leaks located: ~2k/event in EF! long (60 hour) run resulted in hang of ~200 of 960 L2PUs Many algorithm performance studies by individual trigger slices, e.g jets and missing ET: compare the cell vs the FEB data unpacking for the L2 jets (first time FEB unpacking done online!) Compute Ex, Ey, Ez by summing 128 cells connected to one FEB (LAr) Cell unpacking FEB unpacking total processing time dominated by unpacking and data preparation time initial studies indicate factor of 2 improvement using FEB 18

19 Canadian HLT Testbed HLT testbed purchased in 2005 with Quebec FQRNT Equipe funds (grant held by Warburton, Robertson and Vachon) Commissioned primarily by UdeM postdocs and students ATLAS software infrastructure and configuration identical to that used in technical runs can reproduce full functionality of HLT software environment with single instance of each HLT element (e.g. EF PC) (obviously) some differences in networking etc compared with actual HLT UltraSPARC T1 2x(quad core) LVL2: 2.6GHz Opteron 2x(dual core) EF: 2.2GHz Opteron 1x(dual core) Gateway: 2.2GHz Opteron 1x(single core) 19

20 Canadian HLT Testbed Used for: Hardware performance testing (single/dual/quad core, memory etc) Release validation in advance of technical runs possible to perform much longer runs than ~1week Detailed performance checking and online integration of HLT algorithms avoid wasted time during technical run Memory leak and stability checking nodes can be reconfigured to check e.g. EF performance on single vs dual core without time-pressure of technical runs ATLAS DAQ and/or data quality monitoring shifter training DAQ screen-shots on pages 15 & 16 were actually from the testbed, not from a technical run! 20

21 Canadian HLT Testbed Test of L2 Jet algorithm performance in multi-threaded environment (on 2.6GHz dual-core Opteron 285): Total time (per ROI) for various L2 Jet trigger thresholds and different numbers of active threads VBF signal Monte Carlo simulation 21

22 Online integration HLT algorithm development typically takes place within the offline software framework but algorithms must run online... Time optimization, use of multi-threading Services e.g. online database Online histogram monitoring (in addition to offline monitoring) March 2007 Technical Run New Canadian HLT Testbed (2.6GHz Opteron 285) Old Factor of 2 improvement in total processing time by replacing pow(x,2) by x*x and removing unnecessary dereferencing 22

23 Conclusion Canadians are playing an active and increasing role in HLT commissioning activities This is a natural progression of HLT algorithm development, data quality monitoring, online integration and HLT testbed activities as ATLAS moves towards data taking Canadian leadership in several important areas of HLT development which are coupled to online operations, but desirable to extend the scope to these activities to encompass online operational responsibilities HLT RTI request represents a hardware contribution to ATLAS which is consistent with our ongoing Canadian HLT activities and which would help confirm our central role in HLT online operations 23

24 Control network Core online control network: Run control Online databases Monitoring traffic 10 GE Control Links Network root (to ATCN) 4 Online racks GE Control Links Management Servers Core 1 Conc. Core 2 Conc. Conc. 17 LVL2 racks 1 DC Control rack DataFlow Network 4 SFI racks Conc. Conc. 57 EFracks 3 SFO racks BackEnd Network SFO Switch 24

25 Back end network SFI to EF traffic and data output to mass storage via SFOs GE Data Link 10GE Data Link DataFlow Network GE Control Link ~100 SFIs SFIs ~30 SFOs Control Network SFOs Central 1 57 EF racks EF conc. EFs EF conc. EFs. c n o c O F S Mass storage EF conc. EFs 25

26 Data Collection network LVL2 Traffic Event Builder Traffic ~160 ROSs ROS PC ROS PC A ~20 ROS conc. switches B ROS PC ROS PC A ROS c onc. A B ROS PC ROS PC B ROS c onc. ROS c onc. B (A) B (A) B (A) A (B) A (B) Control Network A (B) A,B A,B A A,B Central 1 A,B B SVs L2PU c onc. A L2PUs Central 2 B A A A Control Network B A 17 L2PU racks ~550 L2PUs B L2PU c onc. B ~100 SFIs SFIs SFIs A B B L2PUs BackEnd Network L2PU c onc. A A L2PUs L2PU c onc. GE Da ta Link 10GE Da ta Link B B L2PUs GE Control Link 26

27 ATLAS Trigger System Yet another representation of the trigger: 27

28 Benchmarking: AMD 28

29 Benchmarking: Intel 29