I Service Challenge e l'implementazione dell'architettura a Tier in WLCG per il calcolo nell'era LHC

Transcription

1 I Service Challenge e l'implementazione dell'architettura a Tier in WLCG per il calcolo nell'era LHC T. Ferrari (INFN-CNAF Tier-1) D. Bonacorsi (INFN-CNAF Tier-1 and CMS experiment) IFAE 2006 Incontri di Fisica delle Alte Energie Apr Pavia (Italy)

2 Outline WLCG and MoU Tiers' roles and responsibility in a WLCG-enabled world Technical participation of a Tier in WLCG Service Challenges (SC) Roles of Tiers in SCs Roles of LHC experiments in SCs SC achievements, open issues, and milestones in 2006 Conclusions 2

3 WLCG Worldwide LHC Computing Grid (WLCG) Purpose: to provide the computing resources needed to process and analyze the data gathered by the LHC experiments The LCG project is assembling at multiple computing centres the main offline data-storage and computing resources needed by LHC exps, and operating these resources in a shared grid-like manner Main goal: provide common tools and implement uniform means of accessing resources MoU [*] for collaboration in the Deployment and Exploitation of the Worldwide LHC Computing Grid between the Parties [#] [*] CERN-C-RRB /Rev. 21 March 2006 [#] see next slide After LCG Phase 1 (technology development and tests leading to a production prototype), the WLCG MoU governs the execution of the LCG Phase-2 (deployment and exploitation of LCG as a Service) Defines the program of work, distribution of duties and responsibilities to the Parties as well as the Computing Resources Levels they will offer to the LHC exps (also organizational, managerial and financial guidelines). 3

4 The WLCG MoU parties a) CERN: Tier-0: receives raw data from exps online computing farms and records them on permanent MSS + 1st-pass reco + distribute them to Tier-1 s CERN Analysis Facility functionality of a combined T1-T2 centre, except that it does not offer permanent storage of back-up copies of raw data a) all Institutions participating in the provision of the WLCG with a T1 and/or T2 computing centre (federations included): Tier-1 s: provide a distributed permanent back-up of the raw data, storage and mgmt of data needed during the analysis process + offer a grid-enabled data service data intensive analysis and reprocessing may undertake national or regional support tasks, and contribute to Grid Operations Services Tier-2 s provide well-managed, grid-enabled disk storage and concentrate on tasks such as simulation, end-user analysis and high-performance parallel analysis 4

5 Tiers architecture (it applies to all LHC exps, though) ~PBytes/sec Online System ~100 MBytes/sec 1 TIPS is approximately 25,000 SpecInt95 equivalents Tier 1 There is a bunch crossing every 25 nsecs. There are 100 triggers per second Each triggered event is ~1 MByte in size France Regional Centre ~622 Mbits/sec or Air Freight (deprecated) Germany Regional Centre Tier 0 Offline Processor Farm ~20 TIPS Italy Regional Centre ~100 MBytes/sec CERN Computer Centre FermiLab ~4 TIPS ~622 Mbits/sec ~622 Mbits/sec Tier 2 Caltech ~1 TIPS Tier2 Centre Tier2 Centre Tier2 Centre Tier2 Centre ~1 TIPS ~1 TIPS ~1 TIPS ~1 TIPS Physics data cache Institute Institute ~0.25TIPS Physicist workstations Institute ~1 MBytes/sec Institute Tier 4 Physicists work on analysis channels. Each institute will have ~10 physicists working on one or more channels; data for these channels should be cached by the institute server [ Image courtesy of Harvey Newman, Caltech ] 5

6 Minimal computing capacities for Tier-0 (i.e. minimal Computing Resources and Service Levels to qualify for WLCG membership) In support of offline computing systems of LHC exps according to their Computing Models, CERN shall mainly supply the following services: operation of the Tier-0 facility, providing: high bandwidth network connectivity from exp. area to offline computing facility; recording / permanent storage (MSS) of one copy of raw data throughout exp lifetime; distribution of an agreed share of raw data to each Tier-1 centre; 1st pass calibration and alignment processing, including sufficient buffer storage of associated calibration samples for up to 24 hrs + event reco according to policies agreed with exps; storage of the reco data on disk and in MSS; distribution of an agreed share of the reco data to each Tier-1 centre; operation of the CERN Analysis Facility, providing: data-intensive analysis, high-performance access to current versions of the exps real/simulated datasets; end-user analysis. i.e. all functionalities of a combined T1/T2, except from permanent storage of back-up copies of raw data provision of base services for Grid Coordination and Operation: Overall management and coordination of the LHC Grid integration, certification, distribution, support for software required for Grid operations support, at several levels: network issues, databases, tools, libraries, infrastructures, VOs-management, etc 6

7 Minimal computing capacities for Tier-1 s (i.e. minimal Computing Resources and Service Levels to qualify for WLCG membership) Tier-1 centres form an integral part of data handling service of LHC exps They undertake to provide services: on a long-term basis (initially at least 5 yrs) upgrading also, to keep pace with expected growth of LHC data volumes and analysis activities with high level of reliability/availability + rapid responsiveness to problems Wide pletora of services provided by each Tier-1 to LHC exps they serve: i. acceptance of agreed share of raw data from Tier-0, keeping up with DAQ; ii. acceptance of agreed share of 1st-pass reco data from Tier-0; iii. acceptance of processed and simulated data from other WLCG centres; iv. recording + archival storage of accepted share of raw data ( distributed back-up ); v. recording + maintenance of processed/simulated data on MSS vi. provision of managed storage space for permanent/tmp storage of files and dbs; vii. provision of access to the stored data by other WLCG centres; viii. operation of a data-intensive analysis facility; ix. provision of other services according to agreed exps requirements; x. ensure high-capacity network bandwidth and services for data exchange with Tier-0 (as part of an overall plan agreed amongst exps, Tier-0 and Tier-1 s); xi. ensure network bandwidth and services for data exchange with Tier-1 s and Tier-2 s (as part of an overall plan agreed amongst the exps, Tier-1 s and Tier-2 s); xii. administration of databases (and more ) required by exps at Tier-1 s. All storage/computational services shall be grid enabled according to standards agreed between LHC exps and the regional centres 7

8 Minimal computing capacities for Tier-2 s (i.e. minimal Computing Resources and Service Levels to qualify for WLCG membership) Tier-2 services shall be provided by centres (or federations of centres): as a guideline, Tier-2 s are expected to be capable of fulfilling at least a few % of the resource requirements of the LHC exps that they serve. Services provided by each Tier-2 are: i. provision of managed storage space for permanent/tmp storage of files and dbs; ii. provision of access to the stored data by other WLCG centres; iii. operation of an end-user analysis facility; iv. provision of other services, e.g. simulation, according to agreed exp requirements; v. ensure network bandwidth and services for data exchange with Tier-1 s (as part of an overall plan agreed amongst exps, Tier-1 s and Tier-2 s); As for T1 s: grid-enabling of all storage/computational services 8

9 Technical participation to WLCG The technical participation of the Parties is defined separately in terms of: Computing Resources levels that they pledge to provide to one/more LHC exps that they serve; Computing Services levels that they pledge to the WLCG collaboration. In both cases having secured the necessary funding for iron objects and warm bodies Crucial to define (and apply) metrics: associate with each element a set of key qualitative measures: e.g. Computing Resource Level associated to Networking shall include I/O throughput and average availability in terms of [time running]/[scheduled up-time] e.g. Computing Service level imply reliability, availability, responsiveness to problems, Let s dig into both resources and services 9

10 Resources : the pledged Computing Capacities 10

11 Services : required components at Tiers T0 Service Only some VOs? Class T1 Service Only some VOs? Class SRM 2.1 C SRM 2.1 H/M LFC LFC FTS CE RB LHCb ALICE, ATLAS C H C C C LFC FTS CE Site BDII ALICE, ATLAS H/M H/M H/M H/M Global BDII C R-GMA H/M Site BDII H T2 Service Only some VOs? Class Myproxy C SRM 2.1 M/L VOMS HC LFC ATLAS, ALICE M/L R-GMA H CE M/L databases C/H Site BDII M/L R-GMA M/L Class Description Downtime Reduced Degraded Availability SERVICE LEVELS definition: C H M L U Critical High Medium Low Unmanaged 1 hour 4 hours 6 hours 12 hours None 1 hour 6 hours 6 hours 24 hours None 4 hours 6 hours 12 hours 48 hours None 99% 99% 99% 98% None Downtime defines time between start of problem and restoration of service at min capacity (i.e. basic function but capacity < 50%) Reduced defines time between the start of the problem and the restoration of a reduced capacity service (i.e. >50%) Degraded defines the time between the start of the problem and the restoration of a degraded capacity service (i.e. >80%) Availability defines the sum of the time that the service is down compared with the total time during the calendar period for the service. Site wide failures are not considered as part of the availability calculations. 99% means a service can be down up to 3.6 days a year in total. 98% means up to a week in total. None means the service is running unattended 11

12 Services : the MoU availability targets (to be reviewed by the operational boards of the WLCG Collaboration) T0 T2 s T1 s The response times in the above table refer only to the maximum delay before action is taken to repair the problem. The mean time to repair is also a very important factor that is only covered in this table indirectly through the availability targets. All of these parameters will require an adequate level of staffing of the services, including on-call coverage outside of prime shift. 12

13 What s out of DAQ to MSS at CERN? CERN must be capable of accepting data from exps DAQ systems and recording it on magnetic tapes at a long-term sustained average rate of 1.6 GB/s during the p-p run and 1.85 GB/s during the heavy-ion run. The target DAQ-MMS rate achievable at Dec 05 is 750 MB/s, rising to the full nominal data rate of 1.6 GB/s by Aug 06. Experiment SIM SIMESD RAW Trigger RECO AOD TAG ALICE 400KB 40KB 1MB 100Hz 200KB 50KB 10KB ATLAS 2MB 500KB 1.6MB 200Hz 500KB 100KB 1KB CMS 2MB 400KB 1.5MB 150Hz 250KB 50KB 10KB LHCb 400KB 25KB 2KHz 75KB 25KB 1KB 13

14 Nominal MoU data rates at each T1 [ Note: throughput is measured network-tape at each T1, and disk-network at CERN ] VO-view different from WLCG-view acceptance of an agreed share of raw data from T0 (keeping up with DAQ) + 1st-pass reco data ALICE mostly rely on GridKa; ATLAS on BNL,IN2P3,NL; CMS on FNAL,CNAF; LHCb on IN2P3,NL 14

15 Service Challenges A mechanism by which the readiness of the overall LHC computing infrastructure to meet the exps requirements is measured and if(/where) necessary corrected understand what it takes to run a real and wide Grid service joined effort with Tiers community to trigger resources deployment, drive activity planning and encourage distrubuted know-how based on realistic use patterns ramp-up essential grid services to target levels of reliability, availability, scalability, end-to-end performance A long and hard path, done in several steps (see next slide) Service Challenge 1 and 2 building up the necessary data mgmt infrastructure to perform reliable T0-T1 transfers and permanent production services with the appropriate throughput: basically build up T0-T1 infrastructure and services to handle production transfers and data flows no T2 involved, no exp-specific sw and no offline use-cases included SC2 met its throughput goal (100 MB/s/site, 500 MB/s sustained out of CERN), but not its service goals Service Challenge 3 and 4 bringing T2s into loop and fully address exps use-cases towards full production services: add site/exps goals, e.g. focus on DM, batch prod and real data access addressing problems beyond set-up (throughput) goals: add a service phase 15

16 WLCG deployment schedule Apr05 SC2 Complete June05 - Technical Design Report Jul05 SC3 Throughput Test Sep05 - SC3 Service Phase Dec05 Tier-1 Network operational Apr06 SC4 Throughput Test May06 SC4 Service Phase starts Sep06 Initial LHC Service in stable operation Apr07 LHC Service commissioned SC2 SC3 preparation setup service SC4 LHC Service Operation cosmics You (IFAE 2006) are here First physics First beams Full physics run 16

17 Service Challenge 3 The first SC with exps-oriented objectives when: Jul 05 - Dec 05 (+ Jan 06) who: T0, all T1 s, small nb of T2 s Set-up ( throughput ) phase (Jul 05) Throughput targets for each T1 are 150 MB/s network-disk and 60 MB/s network/tape, with CERN capable of supporting 1 GB/s for the transfers to disk and 400 MB/s for those to tape at the T1s. Some T1s also support T2s, which may upload simulated data and download analysis data. Service phase (Sep-Dec 05) Stable operation during which exps are committed to carry out tests of their sw chains and computing models Includes additional sw components, including a grid WMS, Grid catalog, mass storage mgmt services and a file transfer service Re-run of the throughput phase (Jan 06) 17

18 SC3 throughput phase (Jul 05) Jul 05 throughput tests did not meet targets (~50% higher than SC2) on average: ~1/2 target, scarce stability Network-disk target: 150MB/s/T1 1GB/s (CERN) Network-tape target: 60MB/s/T1 (not all sites, though) important step to gain experience with the services before SC4 since then: SRM on each site, dcache , glite FTS 1.4, CASTOR fixes, a lot of debugging in transfers, network upgrades: all sw upgrades released and deployed 18

19 SC3 T0-T1 disk-disk re-run (Jan 06) substantial improvement in recent SC3 throughput re-run some sites exceeded the targets all sites clearly have a much better handle on the systems but still a lot of work to do daily average rate still far away, demostrate sustainability of met/exceeded targets, test recovery from problems, and get rid of heroic efforts! Limited by hw config at CERN to ~1 GB/s Example: CNAF inbound 19

20 + + + = Day-view Week-view 20

21 Current T0-T1 disk-disk transfer rates Centre ALICE ATLAS CMS LHCb Achieved (nominal) p-p data rates into T1 disk-disk (SRM) [MB/s] ASGC, Taipei 80 (100) (have hit 140) CNAF, Italy 200 PIC, Spain >30 (100) (network constraints) IN2P3, Lyon 200 GridKA, Germany 200 RAL, UK 200 (150) BNL, USA 150 (200) FNAL, USA >200 (200) TRIUMF, Canada 140 (50) NIKHEF/SARA, NL 250 (150) Nordic Data Grid Facility 150 (50) Meeting or exceeding nominal disk-disk rate Meeting target rate for SC3 disk-disk re-run These rates must be sustained to tape 24 hours a day, ~100 days/yr. So: not appropriate to focus on ~% discrepancies. Still missing are: stability + extra capacity required to cater for outages / Backlogs (namely running for 24-48h periods at ~2 x nominal). This is currently WLCG biggest data management challenge. 21

22 Tier-0 ESD1 0.5 GB/file 0.02 Hz 1.7K f/day 10 MB/s 0.8 TB/day ESD2 0.5 GB/file 0.02 Hz 1.7K f/day 10 MB/s 0.8 TB/day Other Tier-1s T1 T1 RAW 1.6 GB/file 0.02 Hz 1.7K f/day 32 MB/s 2.7 TB/day AODm1 500 MB/file 0.04 Hz 3.4K f/day 20 MB/s 1.6 TB/day AODm2 500 MB/file Hz 3.1K f/day 18 MB/s 1.44 TB/day RAW 1.6 GB/file 0.02 Hz 1.7K f/day 32 MB/s 2.7 TB/day ESD2 0.5 GB/file 0.02 Hz 1.7K f/day 10 MB/s 0.8 TB/day RAW ESD2 AODm Hz 3.74K f/day 44 MB/s 3.66 TB/day AOD2 10 MB/file 0.2 Hz 17K f/day 2 MB/s 0.16 TB/day AODm2 500 MB/file Hz 0.34K f/day 2 MB/s 0.16 TB/day T1-T1 transfers Take ATLAS average T1 data flow as example highest inter-t1 rates due to multiple ESD copies Reprocessing ~1 month after data taking (better calibs) + at the end of yr with better calibs+algos Tape disk buffer CPU farm disk storage ESD2 0.5 GB/file 0.02 Hz 1.7K f/day 10 MB/s 0.8 TB/day ATLAS T1 s at work Is T1-T1 traffic shaping a constant network load or only peaks? much exp-specific and Computing-Model dependent AOD2 AODm2 10 MB/file 500 MB/file 0.2 Hz Hz 17K f/day 0.34K f/day 2 MB/s 2 MB/s 0.16 TB/day 0.16 TB/day ESD2 0.5 GB/file 0.02 Hz 1.7K f/day 10 MB/s 0.8 TB/day AODm1 500 MB/file 0.04 Hz 3.4K f/day 20 MB/s 1.6 TB/day AODm2 500 MB/file Hz 3.1K f/day 18 MB/s 1.44 TB/day AODm2 500 MB/file 0.04 Hz 3.4K f/day 20 MB/s 1.6 TB/day Tier-2s T1 T1 Courtesy: D.Barberis (ATLAS) Other Tier-1s T1 T1 22

23 T1 T2 transfers Take CMS data flow as example CMS T1 T2 transfers likely to be bursty and driven by analysis demands ~10/100 MB/s for worst/best connected T2 s (2006) CMS T2 T1 transfers almost entirely fairly continuous simulation transfers ~ 10MB/s (NOTE: aggregate input rate into T1 s comparable to rate from T0!) tape 280 MB/s (RAW, RECO, AOD) 225 MB/s (RAW) Tier M-SI2K 0.4 PB disk 4.9 PB tape 5 Gbps WAN WNs 225 MB/s (RAW) 280 MB/s (RAW, RECO, AOD) Tier-1s Tier-1 40 MB/s (RAW, RECO, AOD) Tier-0 Tier-1s Tier-2s AOD AOD 48 MB/s (MC) 240 MB/s (skimmed AOD, Some RAW+RECO) 60 MB/s (skimmed AOD, Some RAW+RECO) 12 MB/s (MC) Tier MSI2K 1.0 PB disk 2.4 PB tape ~10 Gbps WAN WNs Tier M-SI2K 0.2 PB disk 1 Gbps WAN ~800 MB/s (AOD skimming, data reprocessing) Up to 1 GB/s (AOD analysis, calibration) NB: averaged sustained throughputs. Courtesy: J.Hernandez (CMS) WNs 23

24 INFN candidate Tier-2 sites at SC3 times (Oct 05) Torino (ALICE): FTS, LFC (LCG 2.6.0) Storage Space: 2 TB on dcache Milano (ATLAS): FTS, LFC Storage space: 5.29 TB on DPM Pisa (ATLAS/CMS): FTS (+PhEDEx), LFC, POOL f.c., PubDB Storage space: 5(+5) TB on DPM Legnaro (CMS): FTS (+PhEDEx), POOL f.c., PubDB Storage space: 4 TB on DPM Bari (ALICE/CMS): FTS (+PhEDEx), POOL f.c., PubDB, LFC, dcache, DPM Storage space: 1.4(+4) TB mostly on dcache Catania (ALICE): Storage space: 1.8 TB on DPM and classic SE LHCb CNAF 24

25 ALICE: job submissions (Jan-Feb 06) SUM CNAF 25

26 LHCb, ATLAS: FTS testing (Oct-Nov 05) LHCb: config of an entire CNAF-to-T1 s channel matrix for replication of ~1 TB of stripped DTSs data ATLAS: production phase started Nov 05 e.g. INFN T1: 5932 files copied and registered at CNAF: 89 Failed replication events, 14 No replicas found 26

27 CMS: data transfer (Jul 05 Sep-Nov 05) Intense data transfer via PhEDEx INFN: CNAF T1 (>10 TB), Bari proto-t2 (>7 TB), LNL T2 (>4 TB) CNAF: SRM/Castor-1, Bari: SRM/dCache, LNL: SRM/DPM Ex.: automatic daily report on the SC m.l. Job submission via CMS SC3 job robot, using CRAB Job destination pattern shows a 1st (very draft) ex. of load balancing depending on data availability (i.e. publishing) at Tiers Top 20 CEs (not only SC jobs here) >70k SC3 jobs submitted using CRAB (over >350k, by users in Q34/2005) ORCA exit status =0 ORCA exit status 0 Aborted due to Grid problems 27

28 Take CMS as example: T2 s in the game CMS SC3 traffic: LNL inbound A B B A B C D E CMS SC3 traffic: Bari inbound C D E C D E 28

29 Service Challenge 4 Aims to demonstrate that all of the offline data processing requirements expressed in the exps Computing Models, from raw data taking through to data access, can be handled within Grid at the full nominal data rate of the LHC when: Apr 06 - Sep 06 who: T0, all T1 s, majority of T2 s It will become the initial production service for LHC and made available to the exps for final testing, commissioning and processing of cosmic ray data Set-up ( throughput ) phase (Apr 06) Throughput demonstration sustaining for 3 weeks the target data rates at each site Target is a stable, reliable data transfer to T1 s at target rates to any supported SRM implementation (dcache, Castor, ) + factor 2 for backlogs/peaks. Service phase (May-Sep 06) get the basic sw components required for the initial LHC data processing service into the loop Target is to show capability to support full Computing Models of each LHC exp, from simulation to end-user batch analysis at Tier-2 s 29

30 Example: SC4-phase1 disk-tape rates Centre ALICE ATLAS CMS LHCb Target Data Rate MB/s Canada, TRIUMF 50 France, CC-IN2P3 75 Germany, GridKA 75 Italy, CNAF 75 Netherlands, NIKHEF/SARA 75 Nordic Data Grid Facility 50 Spain, PIC Barcelona 75 Taipei, ASGC 75 UK, RAL 75 USA, BNL 75 USA, FNAL 75 Apr 06: still using SRM 1.1 & current tape technology 30

31 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec SC milestones 2006 (by WLCG) SC3 disk-disk transfers repeat (nominal rates capped at 150 MB/s) full T1 involvement, ~1 GB/s out of CERN achieved, good rate to sites CHEP Workshop; T1-T1 Use Cases, SC3 disk-tape transfers repeat (~50 MB/s) all required sw baseline services deployed (SRM, LFC, FTS, CE, RB, bdii, ) target of 50 MB/s/site sustained with ~5 current-technology drives was met (not all sites, though) Detailed plan for SC4 service agreed (mw + DM service enhancements) Crucial period for Castor-2 at CERN. glite 3.0 release: beta testing plenty of testing of crucial components, also at T0: very important SC4 disk-disk (nominal rates) and disk-tape (reduced rates) tests [last was Feb06] glite 3.0 release: available for distribution disk-disk: all T1 s; BNL,FNAL,CNAF,FZK,IN2P3 go up to 200 MB/s; disk-tape: all sites, MB/s installation, configuration and testing of glite 3.0 release at sites mostly on preparation for Jun06 start of SC4 production tests by experiments, focus on T1 Use Cases CERN T2 Workshop: identification of key T2 s Use Cases and Milestones workshop will define T2 targets: a lot of work ahead for exps and Tiers in Q34/2006 T0-T1 disk-tape (full nominal rates) [last was Apr06] all T1 s; rates MB/s depending on VOs supported and resources provided start T1-T1, T1-T2, T2-T1; T2 Milestones (+ debugging of tape results if needed) all T1 s; T2 s; all VOs; all offline use cases LHCC review (+ rerun of disk-tape tests if required) WLCG Service officially opened. Computing capacities continue to build up IFAE 2006 is here 1st WLCG conference. All sites shall have network, tape hw ~ready for production Final service / mw review leading to early 2007 (+ upgrades for LHC data taking?) 31

32 Summary WLCG: more and more ready to provide infrastructure/components to LHC exps Tiers: more and more ready to operate WLCG-enabled services to realize the Computing Models of the LHC exps WLCG is approaching the regime through several SCs involving Tiers we need to practise repeatedly, and expand the scope of testing SC after SC full DAQ-T0-T1-T2 chain + tape backend + drive with exp software the services delivered by WLCG must match the MoU targets and the exps requirements Focus: prepare infrastructures for robust services delivery define metrics for success not good enough? re-implement, and re-measure until it works. First data will arrive next year. NOT an option to have things fully working later. 32

33 Acknowledgements This talk reflects the hard work of a large nb of people across many sites and within LHC exps Collaborations, so extended thanks goes to: all people in WLCG Collaboration all people in network, storage, farming, at WLCG sites all people (from exps and sites) specifically working on SCs INFN SC coordination: Tiziana Ferrari (CNAF), Michele Michelotto (PD) 33