Tier-2 Planning @ DESY Volker Gülzow, Peter Wegner DESY DV&IT 1
Outline LCG- requirements and concept Current status and plans for DESY Conclusion Plans for Zeuthen DESY DV&IT 2
LCG requirements and concepts DESY DV&IT 3
DESY DV&IT 4
Challenge: A large and distributed Community ATLAS Offline software effort: 1000 person-years per experiment CMS Storage Datennahmerate 0.1 1 GBytes/sec -> 5-8 Petabyte Processing 200,000 of today s fastest PCs Software life span: 20 years ~ 5000 Physicists around the world LHCb DESY DV&IT 5
LHC Computing Model Lab a Uni x Lab m ATLAS CERN Tier 1 USA UK Uni a CMS Tier3 physics department Tier2 The LHC Computing Centre Tier 1 Italy CERN CERN Tier 0 LHCb France. Uni n Desktop γ Lab b. Germany Lab c β α Uni y Uni b Tier 0 Centre at CERN DESY DV&IT 6
The LCG Project Approved by the CERN Council in September 2001 Phase 1 (2001-2004): Development and prototyping a distributed production prototype at CERN and elsewhere that will be operated as a platform for the data challenges - leading to a Technical Design Report, which will serve as a basis for agreeing the relations between the distributed Grid nodes and their co-ordinated deployment and exploitation. Phase 2 (2005-2007): Installation and operation of the full world-wide initial production Grid system, requiring continued manpower efforts and substantial material resources. DESY DV&IT 7
Organizational Structure for Phase 2 LHC Committee LHCC Scientific Review Computing Resources Review Board - C-RRB Funding Agencies Collaboration Board CB Experiments and Regional Centres Overview Board - OB Management Board - MB Management of the Project Grid Deployment Board Coordination of Grid Operation Architects Forum Coordination of Common Applications DESY DV&IT 8
The Hierarchical Model Tier-0 at CERN Record RAW data Distribute second copy to Tier-1s Calibrate and do first-pass reconstruction Tier-1 centres (11 defined) Manage permanent storage RAW, simulated, processed Capacity for reprocessing, bulk analysis Tier-2 centres (>~ 100 identified) Monte Carlo event simulation End-user analysis Tier-3 Facilities at universities and laboratories Access to data and processing in Tier-2s, Tier-1s Outside the scope of the project DESY DV&IT 9
Tier-1s Tier-1 Centre TRIUMF, Canada GridKA, Germany CC, IN2P3, France CNAF, Italy SARA/NIKHEF, NL Nordic Data Grid Facility (NDGF) ASCC, Taipei RAL, UK BNL, US FNAL, US PIC, Spain Experiments served with priority ALICE ATLAS CMS LHCb X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X DESY DV&IT 10
Tier-2s ~100 identified number still growing DESY DV&IT 11
The Eventflow Rate [Hz] RAW [MB] ESD rdst RECO [MB] AOD [kb] Monte Carlo [MB/evt] Monte Carlo % of real ALICE HI 100 12.5 2.5 250 300 100 ALICE pp 100 1 0.04 4 0.4 100 ATLAS 200 1.6 0.5 100 2 20 CMS 150 1.5 0.25 50 2 100 LHCb 2000 0.025 0.025 0.5 20 50 days running in 2007 10 7 seconds/year pp from 2008 on ~10 9 events/experiment 10 6 seconds/year heavy ion DESY DV&IT 12
CPU Requirements 350 MSI2000 300 250 200 150 100 50 0 2007 2008 2009 2010 Year 58% pledged CERN Tier-1 Tier-2 LHCb-Tier-2 CMS-Tier-2 ATLAS-Tier-2 ALICE-Tier-2 LHCb-Tier-1 CMS-Tier-1 ATLAS-Tier-1 ALICE-Tier-1 LHCb-CERN CMS-CERN ATLAS-CERN ALICE-CERN DESY DV&IT 13
Disk Requirements 160 PB 140 120 100 80 60 40 20 0 2007 2008 2009 2010 Year 54% pledged CERN Tier-1 Tier-2 LHCb-Tier-2 CMS-Tier-2 ATLAS-Tier-2 ALICE-Tier-2 LHCb-Tier-1 CMS-Tier-1 ATLAS-Tier-1 ALICE-Tier-1 LHCb-CERN CMS-CERN ATLAS-CERN ALICE-CERN DESY DV&IT 14
Tape Requirements 160 PB 140 120 100 80 60 40 20 0 2007 2008 2009 2010 75% pledged Year CERN Tier-1 LHCb-Tier-1 CMS-Tier-1 ATLAS-Tier-1 ALICE-Tier-1 LHCb-CERN CMS-CERN ATLAS-CERN ALICE-CERN DESY DV&IT 15
Typical Grid Components ssh UI JDL output RB RLS BDII GIIS CAT certs $HOME/.globus/ Computing Element ldap://ldap.desy.de VO RB CE SE /etc/grid-security/grid-mapfile CE PBS GRIS GRIS SE SRM world WN WN WN WN disk DESY DV&IT 16
Monitoring Accounting DESY DV&IT 17
Cooperation with other projects Network Services LCG will be one of the most demanding applications of national&international research networks such as GÉANT Grid Software Globus, Condor and VDT have provided key components of the middleware used. Key members participate in OSG and EGEE Enabling Grids for E-sciencE (EGEE) includes a substantial middleware activity. Grid Operational Groupings The majority of the resources used are made available as part of the EGEE Grid (~170 sites, 15,000 processors). The US LHC programmes contribute to and depend on the Open Science Grid (OSG). Formal relationship with LCG through US- Atlas and US-CMS computing projects. The Nordic Data Grid Facility (NDGF) will begin operation in 2006. Prototype work is based on the NorduGrid middleware ARC. DESY DV&IT 18
DESY DV&IT 19
Tier1/2 Summary Table Tier-1 Planning for 2008 ALICE ATLAS CMS LHCb SUM 2008 Offered 6.7 22.7 12.5 4.4 46.3 CPU - MSI2K Disk - PBytes Tape - PBytes Includes current planning for all Tier-1 centres TDR Requirements 12.3 24.0 15.2 4.4 55.9 Balance -46% -5% -18% -0% -17% Offered 2.8 12.5 5.7 2.2 23.2 TDR Requirements 7.4 14.4 7.0 2.4 31.2 Balance -62% -13% -18% -10% -25% Offered 3.2 9.1 8.1 1.9 22.3 TDR Requirements 6.9 9.0 16.7 2.1 34.7 Balance -54% 1% -51% -9% -36% Tier-2 Planning for 2008 ALICE ATLAS CMS LHCb SUM 2008 CPU - MSI2K Offered 5.0 19.5 17.4 4.4 46.3 TDR Requirements 14.4 19.9 19.3 7.7 61.3 Balance -65% -2% -10% -42% -24% Disk - PBytes Offered 1.4 5.9 4.5 0.8 12.6 TDR Requirements 3.5 8.7 4.9 0.023 17.1 Balance -59% -33% -8% n/a -26% # Tier-2 federations - included(expected) 12 (13) 20 (28) 17 (19) 11 (12) 28 (37) DESY DV&IT 20
Status and plans @ DESY DESY DV&IT 21
LHC and DESY DESY has decided to participate on an external experiment at LHC DESY will participate in Atlas and CMS DESY offers for both experiments to run a av. Tier 2 centre each DESY has to offer Tier 3 service to local groups A joint Hamburg-Zeuthen activity For DESY this is a long term committment DESY DV&IT 22
Where are the requirements stated? 23
From G. Quast, OB-meeting, May 2005 DESY DV&IT 24
From G. Quast, OB-meeting, May 2005 DESY DV&IT 25
Proposed Tier 2 Project @ DESY Under Consideration: Proposal for a 3 year project for ramp up Will become part of standard computer centres in Hamburg&Zeuthen Current key persons: Michael Ernst, Patrick Fuhrmann, Martin Gasthuber, Andreas Gellrich, Volker Gülzow, Andreas Haupt, Stefan Wiesand, Peter Wegner, Knut Woller et al. DESY DV&IT 26
Grids@desy Managed via Virtual Organisations (VOs) H1/Zeus Grids Lattice data Grid LHC-Tier 2 others Amanda/Ice Cube ILC DESY DV&IT 27
Plans for DESY-Tier2&3 Tier 2 is part of a larger grid infrastructure Tier 2 for CMS as a federated Tier 2 with RWTH Aachen Tier 2 for Atlas very likely as a federated Tier 2 with Freiburg and Wuppertal Efficient and shared setup for Hamburg&Zeuthen DESY DV&IT 28
Proposed hardware resources (total) As much as possible only one resource pool (per site) Distribution via Fair Share Scheduler 2005 2006 2007 2008 2009 CPU [ksi2k] 200 800 1400 1600 1800 Disk [TB] 30 200 600 600 800 Tape [TB]? 20 100 400 600 800 DESY DV&IT 29
Proposed hardware resources (total, A=Atlas, C=CMS) 2005 2006 2007 2008 2009 CPU [ksi2k] 100 A 400 A 700 A 700 A 900 A 100 C 400 C 700 C 900 C 1200 C Disk [TB] 15 A 100 A 340 A 340 A 570 A 15 C 100 C 200 C 200 C 300 C Tape [TB] 10 A 50 A 200 A 340 A 570 A (?) 10 C 50 C 100 C 200 C 300 C DESY DV&IT 30
DESY DV&IT 31
DESY DV&IT 32
Connectivity DESY-HH will have a in 2006 a 1 Gb/s ET XWIN-connection Bandwidth will be (according to the needs) at the beginning 300 Mb/s or 600 Mb/s Plan to have a 1Gb/s VPN: HH <-> Zeuthen We will have a VPN connection to GridKa (p2p) (likely 10 Gb/s in 2007) Atlas needs less bandwidth for Tier 2s (cf CM) than CMS DESY DV&IT 33
In Hamburg: ~ 250 kspecint2k ~ 70 TB Storage DESY DV&IT 34
Software Grid Infrastructure fully on LCG 2.6 VOMS will be available soon. currently 15 VO s are supported Atlas Software: done, CMS: done CMS: Successfull Analysis runs with CRAB ( Ernst, Rosemann) DESY DV&IT 35
Memorandum of Understanding for Collaboration in the Deployment and Exploitation of the LHC Computing Grid between The EUROPEAN ORGANIZATION FOR NUCLEAR RESEARCH ( CERN ), an intergovernmental Organization having its seat at Geneva, Switzerland, as the Host Laboratory, the provider of the Tier0 Centre and the CERN Analysis Facility, and as the coordinator of the LCG project, on the one hand, and all the Institutions participating in the provision of the LHC Computing Grid with a Tier1 and/or Tier2 Computing Centre (including federations of such Institutions with computer centres that together form a Tier1 or Tier2 Centre), as the case may be, represented by their Funding Agencies for the purposes of signature of this Memorandum of Understanding, on the other hand, (hereafter collectively referred to as the Parties ). DESY DV&IT 36
Tier-1 DESY DV&IT 37
Tier-2 Resources will be monitored under same conditions as the HERA resources (and much better then the min level) DESY DV&IT 38
Service Challenge DESY DV&IT 39
Service Challenge Goals An integration test for the next production system Full experiment software stack not a middleware test Stack = S/W required by transfers, data serving, processing jobs Main output for SC3 data transfer and data serving infrastructure known to work for realistic use Including testing the workload management components: the resource broker and computing elements Bulk data processing mode of operation Crucial step toward SC4, ATLAS DC, CMS DC06 and LHC Failure of any major component at any level would make it difficult to recover and still be on track with increased scale and complexity in SC4 and the ATLAS/CMS DCs Need to leave SC3 with functional system with room to scale DESY DV&IT 40
DESY 62 MB/s Hourly Averaged Throughput (from CERN CIC to DESY) 100 MB/s http://grid.desy.de/sc/sc3/desy-sc-status.htm (Ernst, Fuhrmann, Gellrich et al) DESY DV&IT 41
2006 Sep LHC service available The SC4 service becomes the permanent LHC service available for experiments testing, commissioning, processing of cosmic data, etc. All centres ramp-up to capacity needed at LHC startup TWICE nominal performance Milestone to demonstrate this 3 months before first physics data April 2007 2005 2006 2007 2008 SC2 SC3 SC4 LHC Service Operation cosmics First physics First beams Full physics run DESY DV&IT 42
Funding and personnel Money comes partly from DESY Additional funds via projects Existing computer centre staff for operation In cooperation with experiments Atlas and CMS specific Softwaresupport EU/national projects DESY DV&IT 43
Conclusion 3 year project planned for setup Joint activity between IT-HH and DV-Zn After 3 years in standard operation mode Tier 3 demands have to be considered Closed cooperation with the Atlas/CMS groups needed Tier 2 situation in Germany not settled Other research groups @DESY will profit DESY DV&IT 44
links http://grid.desy.de http://www-it.desy.de/physics/ http://www.dcache.org http://cern.ch/lcg/ http://goc.gridsupport.ac.uk/gridsite/accounting/index.htm DESY DV&IT 45
Farm Computing in Zeuthen installation Global Batch Farm based on Sun Gridengine starting from 2001 Batch: 60 Hosts amd64 (2.4 GHz Dual Opteron, 4 8 GB Memory) 96 Hosts ia32 (46 x 800 MHz Dual PIII + 50 x 3.06 GHz Dual Xeon) CPU Performance: 261.48 ksi2k Parallel Environment: 16 Nodes amd64 (2.6 GHz Dual Opteron) Infiniband Network 16 Nodes ia32 (1.7 GHz Dual Xeon) Myrinet Network Disk Raid Storage: 40 TB, 60% AFS data volumes, 30% dcache pools DESY DV&IT 46
Farm Computing in Zeuthen Performance in ksi2k (Kilo SpectInt2000) CPU ksi2k/cpu ksi2k/node (*1.7) Farm (46) PIII 800 MHz 0.344 0.585 15.73 (50) XEON 3 GHz 1.066 1.812 90.61 (60) Opteron 2.4 GHz 1.521 2.586 155.14 261.48 Average Tier 2 Center: 6% 2008 19 MSI2k / 30 = 633 (700) ksi2k 2010 52 MSI2k / 30 = 1733 ksi2k 59% 35% ice PIII 800 MHz globe Sun V65x, Xeon 3.06 GHz heliade Sun V20z, Opteron 250 DESY DV&IT 47
Farm Computing in Zeuthen installation Global Batch Farm based on Sun Gridengine starting from 2001 Batch: 60 Hosts amd64 (2.4 GHz Dual Opteron, 4 8 GB Memory) 96 Hosts ia32 (46 x 800 MHz Dual PIII + 50 x 3.06 GHz Dual Xeon) CPU Performance: 261.48 ksi2k Parallel Environment: 16 Nodes amd64 (2.6 GHz Dual Opteron) Infiniband Network 16 Nodes ia32 (1.7 GHz Dual Xeon) Myrinet Network Disk Raid Storage: 40 TB, 60% AFS data volumes, 30% dcache pools DESY DV&IT 48
Farm Computing in Zeuthen (statistics from July 2005 to December 2005) Amanda/IceCube/Baikal: 59% Theory/NIC: 35% Pitz/Tesla/LC: 5% Pitz,H1,Theory in addition dedicated systems CPU hours 300000 250000 4% 7% 9% 2% amanda apeuser baikal grid h1 herab hermes lc nic 200000 other pitz 150000 23% 52% rz sysprog theorie 100000 theorie_zn 50000 0 amanda apeuser baikal grid h1 herab hermes lc nic other pitz rz sysprog theorie theorie_zn DESY DV&IT 49
Current Grid environment @Zeuthen User Interface CE SGE master Resource Broker dcache door globe farm worker nodes BDII dcache head node VOBox File Catalog dcache pool dcache pool DESY DV&IT 50
Tier 2 / Grid plans Integrated Grid Installation Grid User Common GRID & Local environment Local User Farm management global farm worker nodes local farm Global/Local Storage (shared) DESY DV&IT 51
Tier 2 / Grid plans Dedicated Grid Installation Grid User Grid environment Local environment Local User Grid Farm management Torque Local Farm management SGE local farm global farm worker nodes Global/Local Storage DESY DV&IT 52
Tier 2 / Grid plans / ILDG International Lattice DataGrid Goal: Build-up infrastructure for long-term storage and global sharing of simulation data Participants: au, de, it, jp, fr, uk, usa Concept: grid-of-grids, webservice based interfaces DESY contribution: Implementation and operation of metadata catalogue for extensible XML documents Coordination of LCG-based grid infrastructure and operation of central information services DESY DV&IT 53
Farm, Storage Commitments to Experiments at Zeuthen IceCube (Maintenance and Operations Data Systems document): Offline data formatting & merging, filtering, re-processing, analysis, MC production ~ 400 nodes (2009), Storage not yet defined (30 TB raw data / year) NIC (LQCD): processing of configurations, simulations 32 50 nodes, O(10) TB Disk Space / year expected more due after full >2TFlops apenext installation DESY DV&IT 54
Tier 2/Grid plans in Zeuthen New computer room in the upper floor (2007) New UPS (uninterruptible power supply) systems (2006) New Cooling (attached to the PITZ cooling system) (2006) Replacement of old tape robot system (2007) WAN VPN to DESY HH, 1 GBit/s (2006) Tier2/Grid xx nodes, yy TB raid disk, zz tape Main Problem: Disk Storage integration DESY DV&IT 55