CMS LHC-Computing Paolo Capiluppi Dept. of Physics and INFN Bologna P. Capiluppi - CSN1 Catania 18/09/2002
Outline Milestones and CMS-Italy Responsibilities CCS (Core Computing and Software) milestones Responsibilities (CMS Italy) Productions (Spring 2002) Goals and main issues Available resources Work done Data Challenge 04 Goals and plans CMS Italy participation and plans (preliminary) LCG role Tier1 and Tier2s (and Tier3s) LCG and Grid What s LCG Grid Real Results and Strategies Conclusion 2
Milestones (CCS and externals) Milestones (CCS and externals) DAQ TDR End EU-DataGrid / EU-DataTAG Projects End US-GriPhyN Project Data Challenge 04 (5%) LCG-1 phase 1 LCG-1 phase 2 End US-PPDG Project CCS TDR Physics TDR End LCG Phase 1 LCG-3 Data Challenge 05 November 2002 December 2003 December 2003 February 2004 June 2003 November 2003 December 2004 November 2004 December 2005 December 2005 December 2004 April 2005 Data Challenge 06 April 2006 3
CCS Level 2 milestones DC04 CCS Level 2 milestones (15 in total) 4 Most details defined at Level 1 (Level 2 milestones straightforward) See next slide
CCS Organigram (June 2002) CCS Organigram (June 2002) CCS PM David Stickland Technical Coordinator Lucas Taylor Resource Manager Ian Willers Regional Center Coordination Lothar Bauerdick Production Processing & Data Mgmt Tony Wildish Architecture Frameworks &Toolkits Vincenzo Innocente Computing & Software Infrastructure Nick Sinanis GRID Integration Claudio Grandi CMS Librarian Shaun Ashby 5
Boards of CCS Boards of CCS CMS CMS Collaboration Collaboration Board Board acts acts as as Institution Institution Board Board for for CCS CCS 4 meetings meetings per per year year CCS-TB CCS-TB (Open (Open mtg, mtg, 6/yr) 6/yr) Technical Technical Board Board Advises Advises PM PM L1,L2 L1,L2 managers managers + Cross Cross Project Project mgrs mgrs + CCS-SC T1/T2 CCS-SC T1/T2 Reps Reps (Closed (Closed mtg, mtg, weekly) weekly) Steering Steering Committee Committee L1,L2 L1,L2 Managers Managers + CCS-FB CCS-FB Cross-project Cross-project mgrs (Closed mgrs (Closed mtg, mtg, 6/yr) 6/yr) +Co-opted +Co-opted experts Finance experts Finance Board Board CCS CCS Management+ Management+ Funding Funding Agency Agency reps reps 6
CMS-Italy official Responsibilities CMS-Italy official Responsibilities CCS SC (Core Computing and Software Steering Committee) Grid Integration Level 2 manager (Claudio Grandi) INFN contact (Paolo Capiluppi) CCS FB (CCS Financial Board) INFN contact (Paolo Capiluppi) PRS (Physics Reconstruction and Software) Being recruited/refocused for the Physics TDR Muons (Ugo Gasparini) Tracker/b-tau (Lucia Silvestris) LCG (LHC Computing Grid Project) SC2 (Software and Computing Steering Committee) (Paolo Capiluppi alternate of David Stickland) Detector Geometry & Material Description RTAG (Requirements Technical Assessment Group) chairperson (Lucia Silvestris) HEPCAL (HEP Common Application Layer) RTAG (Claudio Grandi) CCS Production Team INFN contact (Giovanni Organtini) 7
Spring 2002 Production (and Summer extension) Goal of Spring 2002 Production: DAQ TDR Simulations and Studies ~6 million events simulated, then digitized at different luminosities NoPU (2.9M), 2x1033 (4.4M), 1034 (3.8M) CMSIM started in February with CMS125 Digitization with ORCA-6, started in April First analysis completed (just!) in time for the June CMS week Extension of activities: Summer 2002 Production Ongoing ntuple-only productions High-p t jets for the e-γ group (10 M) Non-recycled pileup for the JetMet group (300 K) Over 20 TB of data produced CMS-wide Most available at CERN, lots at FNAL, INFN FNAL, INFN, UK also hosting analysis Some samples analyzed at various T2s (Padova/Legnaro, Bologna, ) Production tools obligatory: IMPALA, BOSS, DAR, RefDB BOSS is an official CMS production Tool : INFN developed (A. Renzi and C. Grandi) and maintained (C. Grandi)! 8
(what are those acronyms?) (what are those acronyms?) IMPALA Uses RefDB assignments to create batch jobs locally in the RCs, uses BOSS to submit them BOSS Run-time tracking of job progress and interface to local scheduler DAR Distribution After Release, installs CMS software on farms RefDB Interface for PRS groups to request datasets, for production centres to update the status of their assignments, and for the production coordinator to monitor the overall progress of production 9
Spring02: CPU Resources 11 RCs (~20 sites) About 1000 CPUs and 30 people CMS-wide Some new sites & people, but lots of experience too UCSD 3% Moscow 10% INFN 18% UFL 5% Wisconsin 18% Bristol 3% RAL 6% HIP 1% Caltech 4% FNAL 8% IN2P3 10% IC 6% CERN 15% 10
2002 CMS Computing CMS-Italy available Resources as of August 2002 # CPUs # Boxes Average CPU (MHz) 2002 resources status Total Si2000 TBs on disk servers TBs on disks nodes Farm (% of use) Bari 18 9 900 8000 2.7 0.7 100 Bologna 28 14 1000 14000 3.3 0.5 100 Catania 4 2 866 1800 1.1 0.3 100 Firenze 42 21 1000 21000 1.6 0.9 30 Legnaro 136 68 1200 81600 3.3 8.0 100 Milano Napoli Padova 40 20 900 18000 0.4 1.0 100 Pavia Perugia 16 8 1000 8000 1.0 0.7 90 Pisa 25 15 670 11250 1.0 0.1 100 Roma1 21 12 870 9600 2.0 0.2 100 Torino 22 11 1180 12000 2.6 0.6 100 Total 352 180 185250 19.0 13.0 Tier1 CNAF 50 25 1000 25000 5.0 0.8 90 11
Production in the RCs Production in the RCs RC name CMSIM (K) 2x1033 (K) 1034 (K) Objy size (TB) CERN 870 1670 1970 10.4 Bristol/RAL 547 60 20 0.4 Caltech 214 146 0.5 Fermilab 345 251 332 2.5 INFN (9 sites) 1545 719 709 3.0 IN2P3 200 Moscow (4 sites) 425 0.2 UCSD 338 278 288 1.8 UFL 540 40 40 0.2 Wisconsin 67 54 0.3 Imperial College 878 147 121 1.4 Thanks to: Giovanni Organtini (Rm), Luciano Barone (Rm), Alessandra Fanfani (Bo), Daniele Bonacorsi (Bo), Stefano Lacaprara (Pd), Massimo Biasotto (LNL), Simone Gennai (Pi), Nicola Amapane (To), et al. 12
CMS-Italy 2003 Milestones (INFN) 2003 Milestones One half of the sites Grid enabled for production LCG production prototypes ready (Tier1+Tier2) New CMS analysis environment installed and working (selected sites) One half of the sites working with the new persistency 5% DataChallenge participation of Tier1 and half of Tier2 Scadenza Aprile Giugno Giugno Ottobre Dicembre RC Hit NoPU 2x1033PU 1034PU Bologna 0.420 0.300 0.126 0.126 Legnaro 0.593 0.000 0.593 0.584 Pisa 0.070 0.000 0.000 0.000 Roma 0.100 0.100 0.000 0.000 Nb of Events (M) 1.183 0.400 0.719 0.710 13
6 million events CMSIM 1.2 seconds per event for 4 months Feb. 8 th June 6 th 14
4 million events 2x1033 1.2 seconds per event, 2 months April 12 th June 6 th 15
3.5 million events 1034 1.4 seconds per event, 2 months April 10 th June 6 th 16
DC04, 5% Data Challenge DC04, 5% Data Challenge Definition Is 5% of 10 34 running, or 25% of 2x10 33 (Startup) One month data taking at Cern, 50 M events It represents a factor 4 over Spring 2002, consistent with the goal of doubling complexity each year to reach a full-scale (for LHC startup) test by Spring 2006 Called DC04 (and the others DC05, DC06) to get over the % confusion More importantly, Previous challenges have mostly been about doing the Digitization This one will concentrate on the reconstruction, data distribution and early analysis phase Move the issue of Analysis Model out of the classroom and into the spotlight 17
Setting the Goals of DC04 Setting the Goals of DC04 As defined to the LHCC, the milestone consists of: CS-1041 1 April 2004 5% Data challenge complete (Now called DC04) The purpose of this milestone is to demonstrate the validity of the software baseline to be used for the Physics TDR and in the preparation of the Computing TDR. The challenge comprises the completion of a 5% data challenge, which successfully copes with a sustained datataking rate equivalent to 25Hz at a luminosity of 0.2 x 10 34 cm -2 s -1 for a period of 1 month (approximately 5 x 10 7 events). The emphasis of the challenge is on the validation of the deployed grid model on a sufficient number of Tier-0, Tier-1, and Tier-2 sites. We assume that 2-3 of the Tier- 1 centers and 5-10 of the Tier-2 centers intending to supply computing to CMS in the 2007 first LHC run would participate to this challenge. 18
Pre-Challenge DC04: Two Phases DC04: Two Phases (Must be successful) Large scale simulation and digitization Will prepare the samples for the challenge Will prepare the samples for the Physics TDR work to get fully underway Progressive shakedown of tools and centers All centers taking part in challenge should participate to pre-challenge The Physics TDR and the Challenge depend on successful completion Ensure a solid baseline is available, worry less about being on the cutting edge Challenge Reconstruction at T0 (CERN) Distribution to T1s (Must be able to fail) Subsequent distribution to T2s Assign streams and analyses to people at T1 and T2 centers Some will be able to work entirely within one center Others will require analysis of data at multiple-centers GRID tools tested for data movement and job migration 19
DC04 Setting the Scale DC04 Setting the Scale Pre-Challenge Challenge Aim is 1 month of running at 25 Hz, 20 hours per day 50 Million reconstructed events (passing L1 Trigger and mostly passing HLT, but some background samples also required)) Simulation (GEANT4!) 100TB 300 ksi95.months 1GHz P3 is 50 SI95 Working assumption that most farms will be at 50SI95/CPU in late 2003 Six months running for 1000 CPUS (Worldwide) (Actually aim for more CPU s to get production time down) Digitization 75TB 15 ksi95.months 175MB/s Pileup bandwidth (if allow two months for digitization) Reconstruction at T0-CERN 25TB 23 ksi95 for 1 month (460 CPU @ 50SI95/CPU) Analysis at T1-T2s Design a set of tasks such that offsite requirement during challenge is about twice that of the T0 20
Building a Real Plan for DC04 Building a Real Plan for DC04 Organization: The sites that will participate to each stage of the challenge must be identified and their contributions quantified. The roles of the prototype T0, T1 and T2 in the challenge should be clarified. Goals: Establish Physics TDR goals of the Production Establish Analysis Model Goals of the Challenge Software: Establish the baseline for Persistency of Event Data and Meta-Data for this challenge Create SPROM work-plan to meet Simulation requirements Create RPROM work-plan to meet Reconstruction requirements LCG: Coordinate dates of Challenge with LCG Establish any additional requirements on LCG-1 functionality Production Establish a baseline production environment capable of managing the pre-challenge Establish with GRID Projects and LCG the extent of GRID products available for the prechallenge production. Establish and monitor milestones to track this. Establish a baseline production environment capable of managing DC04 Establish the Monitoring systems required to measure the performance of DC04 and to identify bottlenecks both during the challenge and in its subsequent assessment. Establish a deployment model for the monitoring 21
Overview of Resource Requirements Estimates for CPU and Storage Requirements for CMS Data Challenge DC04 Year.Quarter 03Q3 03Q4 04Q1 04Q2 Computing Power (ksi95 Months) Total Requirement for Simulation 100 200 Total Requirement for Digitization 15 Total Requirement for Reconstruction 25 Total Requirement for Analysis 50 50 Total Previewed CERN/LCG capacity (Eck) 100 100 100 210 CERN T0 33 72 25 CERN T1 (Challenge related only) 17 17 Offsite T1+T2 (Challenge only) 67 143 33 33 Storage (TeraBytes) Data Generated CERN 19 39 25 Data Generated Offsite 39 78 Data Transferred to CERN 17 33 Sum Data Stored CERN 36 108 133 133 Active Data at CERN 25 75 100 100 Assumed Number of Active Offsite T1 3 3 3 3 Sum Data Stored Offsite 39 117 192 192 22
CSM-Italy and DC04 CSM-Italy and DC04 Participation to the Challenge: ~ 20% contribution. Use of 1 Tier1 (common) and 3-4 Tier2s All Italian sites will possibly participate to pre-challenge phase Use all available and validated (CMS-certified) Grid tools for the pre-challenge phase Coordinate resources within LCG for both pre-challenge and challenge phases, where possible (Tier1/INFN must be fully functional: ~70 CPU Boxes, ~20 TB) Use the CMS Grid Integrated environment for the Challenge (February 2004) Participate to the preparation of: Build the necessary resources and define the Italian commitments Define the Data Flow Model Validation of Grid tools Integration of Grid and Production tools (review and re-design) 23
CMS-Italy DC04 Preparation CMS-Italy DC04 Preparation Use the tail of Summer Production to test and validate resources and tools (grid and non-grid) November/December 2002 Participate to the Production-Tools Review Now (Claudio Grandi, Massimo Biasotto) Hopefully contribute to the new tools development (early 2003) Make available the new software at all the Sites (T1, T2s, T3s) Use some of the resources to test and validate Grid Integration Already in progress at the Tier1 (CMS resources) and at Padova Commit and validate (for CMS) the resources for DC04 See following slide Define the participation to the LCG-1 system See following slide 24
CMS Italy DC04 preliminary plans All the current and coming resources of CMS Italy will be available for the DC04, possibly within the LCG Project Small amount of resources requested for 2003 Smoothly integrate the resources into LCG-1 Continue to use dedicated resources for tests of Grid and Production tools Integration Needs for the funding of the others 3-4 Tier2s Request for common CMS Italy sub-judice in 2003: Present a detailed plan and a clear Italian commitment to CMS 60 CPUs and 6 TBytes disk + Switches Will complete already existing Farms We are particularly low in disk storage availability Essential for physics analysis 25
Name & location of Regional Centre Experiments that are served by the resources noted below CMS Italy DC04 LCG preliminary plans INFN- Laboratori Nazionali di Legnaro (LNL) CMS Preliminary commitment of possibly available resources: years Processor Farm No. of processors planned installed Disk Storage Estimated total capacity (TB) 2002 2003 2004 2005 50 80 110 150 5 8 15 25 Tier1 plans common to all Experiments See F. Ruggieri s Presentation LNL partially funded in 2002 (24 CPUs, 3 TB) for LCG participation. The remaining resources are CMS directly funded. 26
DC04 Summary DC04 Summary With the DAQ TDR about to be completed, the focus moves to the next round of preparations The Data Challenge series to reach full scale tests in 2006 The baseline for the Physics TDR The prototypes required for CMS to write a CCS TDR in 2004 Start to address the analysis model Start to test the data and task distribution models Perform realistic tests of the LCG GRID implementations Build the distributed expertise required for LHC Computing DC04 will occupy us for most of the next 18 months 27
LCG LCG = LHC Computing Grid project (PM: Les Robertson) CERN-based coordination effort (hardware, personnel, software, middleware) for LHC Computing; Worldwide! (Tier0, Tier1s and Tiers2s) Funded by participating Agencies (INFN too) Two phases: 2002-2005 Preparation and setting-up (including tests, R&D and support for Experiments activities) 2006-2008 Commissioning of LHC Computing System Five (indeed four!) areas of activity for Phase 1: Applications (common software and tools) (Torre Wenaus) Fabrics (hardware, farms tools and architecture) (Bernd Panzer) Grid Technologies (middleware development) (Fabrizio Gagliardi) Grid Deployment (resources management and run) (Ian Bird) Grid Deployment Board (agreements and plans) (Mirco Mazzucato) Many Boards: POB(Funding), PEB(Executive), SC2(Advisory),. 28
LHCC Reports Reviews The LHC Computing Grid Project Structure The LHC Computing Grid Project Project Overview Board Common Computing RRB Resource Matters Project Manager Project Execution Board Requirements, Monitoring Software and Computing Committee (SC2) implementation teams RTAG 29
LCG Funding - Materials LCG Funding - Materials Changes to this table are again both positive and negative INFN provides now 150k CHF per contract year for 4 fellows Plans to add 6 more fellows for beginning 2003 Belgium has cut down its contribution to 400k Finland has pulled back its offer Chris Eck, 3 rd Sep 2002 Source Summary of Funding - Materials at CERN Materials in kchf SUM of Funding 2001 2002 2003 2004 2005 Materials Belgium 100 100 100 100 400 Germany 3742 3742 Greece 100 100 100 300 Italy-INFN 200 500 500 1200 Spain 300 300 600 UK-PPARC 1740 1560 3300 Enterasys 200 200 Intel (2) 180 180 CERN 2604 0 0 0 0 2604 SUM 6346 400 2740 2940 100 12526 30
Grid projects (CMS-Italy leading roles) Integration of Grid Tools and Production tools almost done (Italy, UK, Fr main contributions) (Thanks to CNAF people and DataTAG personnel) We can submit (production) jobs to the DataGrid testbed via the CMS Production tools (modified IMPALA/BOSS/RefDB) Prototypes working correctly on DataTAG test layout Will test large scale on DataGrid/LCG Production Testbed Will measure performances to compare with summer production classic jobs (November 2002) Integration of EU/US Grid/production tools Already in progress in the GLUE activity Most of the design (not only for CMS) is ready. Implementation in progress. Target for (first) delivery by end of 2002 31
Experiment Software Dataset Definition Logical components diagram Data Materializer Job Job Definition creation Software release New dataset request Production on demand Job submission Data Management System Dataset Input Specification Update dataset metadata Production monitoring Dataset Algorithm Specification Dataset Catalogue Input data location Workload Management System Job Catalogue Retrieve Resource status Software Release Manager Software Repository Data management operations Resource Monitoring System Resource Directory Job assignment to resources Job Monitoring System Job Monitoring Job type definition Definition Job Book-keeping Copy data Storage Service Data Publish Resource status Computing Service Job output filtering Write data Read data Push data or info Pull info 32 SW download & installation
CMKIN/SIM ORCA Dataset Definition IMPALA Job IMPALA scripts Schema Filter files creation Spring 2002 diagram Software release New dataset request Fetch request parameters Job submission Production web portal Dataset Input Specification Write dataset summary Production monitoring Job type definition Dataset Algorithm Specification RefDB Input data location Local Batch System (or Grid Scheduler) Scheduler Job catalog Retrieve Resource status SCRAM/DAR DAR files CVS repository Data management operations Web page with links to RC home pages Resource Directory Job assignment to resources BOSS BOSS DB Job submission Copy data AMS POSIX GDMP Data Publish Resource status Farm node (or GRAM) Job output filtering Write data Read data Push data or info Pull info 33 SW download & installation
Proposal for a DC04 diagram Experiment Software Dataset Definition VDT Planner IMPALA/MOP Job DAG/JDL +scripts Job Monitoring Definition creation Software release New dataset request Production on demand EDG UI VDT Client Job submission REPTOR/Giggle + Chimera? Dataset Input Specification Update dataset metadata Production monitoring Job type definition Dataset Algorithm Specification Dataset Catalogue Input data location EDG Workload Management System EDG L&B Retrieve Resource status REPTOR/Giggle? PACMAN? Dataset Catalogue Data management operations MDS LDAP Job assignment to resources BOSS&R-GMA BOSS-DB Copy data EDG SE VDT Server Data Publish Resource status EDG CE VDT server Job output filtering Write data Read data Push data or info Pull info 34 SW download & installation
Conclusion CMS Italia e leader nel Computing di CMS Pensiamo di averlo dimostrato e vogliamo continuare Chiediamo il supporto della CSN1 per realizzare il Data Challenge 04, e quelli che seguiranno 35