CMS Computing Model with Focus on German Tier1 Activities

Transcription

1 CMS Computing Model with Focus on German Tier1 Activities Seminar über Datenverarbeitung in der Hochenergiephysik DESY Hamburg,

2 Overview The Large Hadron Collider The Compact Muon Solenoid CMS Computing Model CMS in Germany CMS Workflow & Data Transfers CMS Service Challenges Monitoring HappyFace Project CMS Tier1 Computing - DESY DVSEM,

3 Large Hadron Collider ALICE CMS ATLAS LHCb Proton-Proton Collider Circumference: 27 km Beam Energy: 7 TeV/c² Below Surface: 100 m Temperature: -271 C Energy Use: 1 TWh/a 4 Large Experiments CMS (General-Purpose) Atlas (General-Purpose) LHCb (Physics of b-quarks) Alice (Lead Ion Collisions) 2 Smaller Experiments LHCf and Totem CMS Tier1 Computing - DESY DVSEM,

4 Compact Muon Solenoid Muon Chambers Hadronic Calorimetry Magnetic Retrun Yoke Technical Details: Total Weigh: Diameter: Total Length: Magnetic Field Solenoid: Yoke: t 15 m 21,5 m 4 Tesla 2 Tesla Silicon Tracker Readout Channels, e.g. Tracker: 10 Mio. Superconducting Solenoid Electromagnetic Calorimetry Forward Calorimetry Collission Rate: 40 MHz CMS Tier1 Computing - DESY DVSEM,

5 Pictures of CMS CMS Tier1 Computing - DESY DVSEM,

6 CMS Collaboration CMS 38 Nations 182 Institutions > 2000 Scientists & Engineers CMS Tier1 Computing - DESY DVSEM,

7 Physics Motivation Test of the Standard Model (at the TeV energy scale) Search for the Higgs-Boson In each proton-proton collision, more than 1000 particles are created. The decay products allow to conclude on underlying physics processes of the collision. Physics beyond the SM (e.g. SUSY, extra dimensions,...) CMS Tier1 Computing - DESY DVSEM,

8 Trigger and Event Rate 60 TB/sec Level 1 Trigger Reduction with ASICs (Hardware) Collision Rate: 40 MHz Event-Size: 1,5 MB 150 GB/sec Tape & HDD Storage for Offline- Analysis Recorded Events: 150 per second 225 MB/sec High Level Trigger Software Data Reduction CMS Tier1 Computing - DESY DVSEM,

9 [ TB ] [ TB ] [ MSI2k ] Expected Resources CPU Year Disk Space MSS Space Year Year CMS Tier1 Computing - DESY DVSEM,

10 CMS Computing Model LHC experiments have decided to use distributed computing and storage resources. The Worldwide LHC Computing Grid (WLCG). Grid services based on the glite middleware. Computing centres arranged in a four-tiered hierarchical structure. Availability and resources are regulated by MOU (e.g. Downtime per year, response time, etc.). CMS Tier1 Computing - DESY DVSEM,

11 CMS Tier Structure CMS Tier1 Computing - DESY DVSEM,

12 WLCG Resources Taken from the WLCG Real Time Monitor: CMS Tier1 Computing - DESY DVSEM,

13 Main Tier Responsibilities Tier0: Storage of RAW detector data. First reconstruction of physics objects after data-taking. Tier1: Host one dedicated copy of RAW and reconstructed data outside the Tier0. Re-processing of stored data. Skimming (creation of small sub data samples) Tier2: Monte Carlo production/simulation. Calibration activities. Resources for physics groups analyses. Tier3: Individual user analyses. Interactive logins (for development & debugging) CMS Tier1 Computing - DESY DVSEM,

14 German CMS Members Uni Hamburg DESY Hamburg/ Zeuthen RWTH Aachen FSP-CMS (BMBF) Forschungszentrum Karlsruhe Uni Karlsruhe CMS Tier1 Computing - DESY DVSEM,

15 Tier1 GridKa Located at Forschungszentrum Karlsruhe (KIT, Campus Nord) Part of and operated by the Steinbuch Centre for Computing (SCC, KIT) Multi-VO Tier centre (supports 8 experiments) 4 LHC experiments: CMS, ATLAS, ALICE, LHCb 4 non-lhc experiments: CDF, D0, BaBar, Compass CMS Tier1 Computing - DESY DVSEM,

16 GridKa Resources (2008) Network: CPU: 10 Gbit link CERN GridKa 3 x 10 Gbit link to 3 European Tier1s 10 Gbit link to DFN/X-Win (e.g. Tier2 connections) CMS Resources: Storage: Based on dcache (developed at DESY and FNAL) CMS Disk: ~ 650 TB + D-Grid: ~ 40 TB CMS Tape: ~ 900 TB ~ 800 cores, 2 GB Ram per core D-Grid Resources: ~ 500 cores, 2 GB Ram per core CMS Tier1 Computing - DESY DVSEM,

17 Tier2/3 and NAF Resources Besides the Tier1 resources, considerable CPU and disk capacities are available in Germany (August 2008): Tier2 (German CMS Tier2 federation): DESY: RWTH Aachen: 400 cores, 185 TB disk 360 cores, 99 TB disk Tier3: RWTH Aachen: Uni Karlsruhe: Uni Hamburg: 850 cores, 165 TB disk 500 cores, 150 TB disk 15 TB disk NAF (National Analysis Facility): DESY: 245 cores, 32 TB disk D-Grid resources, partially usable for FSP-CMS: RWTH Aachen: GridKA: 600 cores, 231 TB disk 500 cores, 40 TB disk CMS Tier1 Computing - DESY DVSEM,

18 CMS Workflow 1 Transfers 2 Transfers 3 CMS Tier1 Computing - DESY DVSEM,

19 PhEDEx Data Transfers Physics Experiment Data Export (CMS data placement tool): Various agents/daemons are run at the tier sites (depending on the tier level) upload, download, MSS stage-in/out, DB,... Provides automatic WAN data transfers, based on subscriptions Automatic load-balancing File transfer routing topology (FTS) Automatic bookkeeping (database entries, logging) Consistency checks (DB vs. filesystem) File integrity checks CMS Tier1 Computing - DESY DVSEM,

20 Other Involved Components Monte Carlo Production Agent (ProdAgent) CMS Remote Analysis Builder (CRAB) Dataset Bookkeeping System (DBS) Data Location Service (DLS) Trivial File Catalog (TFC) Grid middleware (glite) SE, CE, RB, UI,... DBS: Relates dataset/block and site DLS: Relates dataset/block and logical file name (LFN) TFC: Relates LFN and local file name Very complex system Needs intense testing, debugging and optimisation. CMS Tier1 Computing - DESY DVSEM,

21 WLCG Job Workflow User Interface Computing Element 6. Job execution Task Queue RB Storage glite WMS WM Proxy 2. Job Workload Manager 4. Job Job Controller CondorG 6. Grid enabled data transfer & access Information Service Match Maker Information Supermarket Information System Storage Element CMS Tier1 Computing - DESY DVSEM,

22 CMS Service Challenges - 1 Computing, Service and Analysis (CSA) Challenges: Test the readiness for data-taking CSA06, CSA07 and CSA08 were performed to test the computing and network infrastructure and the computing resources at a level of 25%, 50% and 100% required for the LHC start-up. The whole CMS workflow was tested: Production/Simulation Prompt reconstruction at the Tier0 Re-reconstruction at the Tier1s Data distribution to Tier1s and Tier2s Data Analyses CMS Tier1 Computing - DESY DVSEM,

23 CMS Service Challenges - 2 Cumulative data volume transferred during CSA08: > 2 PetaByte in 4 weeks Transfer rate from the Tier0 to the Tier1s: < 600 MB/sec Goal was achieved only partially, problems with: Storage systems (CASTOR at CERN) Problems identified and fixed afterwards. CMS Tier1 Computing - DESY DVSEM,

24 CMS Service Challenges - 3 CMS Grid-job statistics: May to August 2008 German centres GridKa, DESY and Aachen performed extremely well, compared to all other CMS tier sites. CMS Tier1 Computing - DESY DVSEM,

25 Results of CMS Challenges Service Goal 2008 Status 2008 Goal 2007 Status 2007 Goal 2006 Status 2006 Tier-0 Reco Rate Hz Achieved 100 Hz Only at bursts 50 Hz Achieved Tier-0 Tier-1 Transfer Rate 600 MB/sec Achieved partially 300 MB/sec Only at bursts 150 MB/sec Achieved Tier-1 Tier-2 Transfer Rate MB/sec Achieved MB/sec Achieved partially MB/sec Achieved Tier-1 Tier-1 Transfer Rate 100 MB/sec Achieved 50 MB/sec Achieved partially N/A - Tier-1 Job Submission jobs/day Achieved jobs/day Achieved jobs/day jobs/day Tier-2 Job Submission jobs/day Achieved jobs/day jobs/day jobs/day Achieved Monte Carlo Simulation 1.5 x 10 9 events/year Achieved 50 x 10 6 events/month Achieved N/A - CMS Tier1 Computing - DESY DVSEM,

26 First Real CMS Data 20 TB/day 10 TB/day Data Imports to GridKa of cosmic muon samples (per day) Stable running over large periods of time First real beam event: LHC Beam hitting collimators near CMS (Sept. 10th 2008). GridKa was the first CMS tier centre outside CERN with real beam data. CMS Tier1 Computing - DESY DVSEM,

27 Monitoring - Overview Distributed Grid resources are complex systems. To provide a stable and reliable service, monitoring of the different services and resources is indispensable. Use monitoring to discover failures before the actual services are affected. Different types of monitoring: Central monitoring of all Tier centres Status of the high level grid services and their interconnection Availability of the physical resources of the Tier centres. Central monitoring of experiment specific components Local monitoring at the different computing sites Local monitoring of experiment specific components Many different monitoring instances for different purposes! CMS Tier1 Computing - DESY DVSEM,

28 Grid Monitoring Central WLCG Monitoring: Service Availability Monitoring (SAM) Standardised Grid jobs, testing the basic grid functionality of a Tier centre. Includes all Grid services of a site. Centrally send to the Tiers (once per hour) Output available on a web page with history Once a critical SAM test has failed on a site, no user jobs will be send to it anymore. CMS Tier1 Computing - DESY DVSEM,

29 Experiment Monitoring Besides the basic grid functionality, experiment specific components and functionalities are tested. Access point to the results is the CMS ARDA Dashboard. SAM: Also tests of dedicated functionalities of the Tier centres requested by CMS. Job Summary: Type of the job and its exit code. History per centre and activity. PhEDEx data transfers: Transfer quality for CMS datasets. Collects error messages in case of failures. Gives overview on the transferred data samples available at the different sites. CMS Tier1 Computing - DESY DVSEM,

30 Local Site Monitoring Each computing centre has its own monitoring infrastructure: Nagios Ganglia... Offers monitoring for hardware, services and network connections. Adapted for the local particularities. Automated notification in case of failures by , instant messages, SMS,... Experiment specific local monitoring: Is the local software installation accessible? Are all required services properly configured? CMS Tier1 Computing - DESY DVSEM,

31 USCHI Ultra Sophisticated CMS Hardware Information System Provides specific tests of the local infrastructure of a site. Mainly driven by previously observed problems. Designed in a modular way to ease the implementation of new tests. Currently, the following tests are running locally at GridKa: Availability of the CMS production software area Sufficient free inodes of the file system Basic grid functionality (e.g. user interface commands) FTS transfer channels (are they enabled, working, etc.) Remaining space of the disk-only area Status of the dcache tape-write buffers Functionality of basic VOBox commands PhEDEx grid proxy validity Output of the test provided in xml format Developed in Karlsruhe A. Scheurer, A. Trunov CMS Tier1 Computing - DESY DVSEM,

32 Multi-Monitoring Drawbacks The current monitoring infrastructure is characterised by a multitude of different sources to query for information about one single site. Limitations of the current monitoring infrastructure: Information is not clearly arranged: Too much information not related to a dedicated site. Different site design and history plots / formats / technologies. Nearly impossible to correlate information from different sources. Uncomfortable to use: Access many different websites to get all needed information about one site. Different web interfaces require parameters (time range, site name,...) Often needs a long time to download / update plots (> 1 minute!) Increases administrational effort and slows down the identification of errors. A meta-monitoring system can help to improve this situation! CMS Tier1 Computing - DESY DVSEM,

33 Ideas for Meta-Monitoring Only one website to access all relevant information for one site Simple warning system (with symbols) Single point of access for all relevant information (+ their history) Caching of important external plots for a fast access Backend for implementation of own tests Easy implementation of new tests Self-defined rules for service status and warning-alerts triggering Correlation of data from different sources Fast, simple and up to date All monitoring information should be renewed every 15 minutes Short access / loading times of the web page Easy to install and use - no complex database Access to every test result in less then 3 mouse clicks Provide links to raw information from the data source The meta-monitoring system HappyFace Project offers this functionality. CMS Tier1 Computing - DESY DVSEM,

34 HappyFace Project CMS Tier1 Computing - DESY DVSEM,

35 HappyFace Project - 2 Our HappyFace instance is currently monitoring the following quantities at GridKa: Local CMS infrastructure: the USCHI test-set Incoming/outgoing PhEDEx transfers: General transfer quality. Classification of error type. Classification of destination and source errors. Status of the WLCG production environment Queries SAM test results and displays eventually failed tests including a hyperlink to the error report. CMS Dashboard: Search Job Summary Service for failed jobs, print statistics of the exit codes. Fairshare of CMS Displays the fraction of running CMS jobs Caching of all plots necessary for monitoring CMS Tier1 Computing - DESY DVSEM,

36 HappyFace Project - 3 Instances of HappyFace currently used by: Uni Karlsruhe RWTH Aachen Uni Hamburg Uni Göttingen Code and documentation available under: Developed in Karlsruhe V. Buege, V. Mauch, A. Trunov Future plans: Platform to share HappyFace modules Provide standardised output in xml format of all HappyFace instances Contact: Viktor Mauch: mauch@ekp.uni-karlsruhe.de Volker Buege: volker.buege@cern.ch CMS Tier1 Computing - DESY DVSEM,

37 Summary CMS uses Grid technology for data access and processing. Multiple service challenges have proven the readiness for first data. PhEDEx is used for reliable large-scale data management. Experience has shown, that monitoring is indespensable for the operation of such a heterogeneous and complex system. Multitude of monitoring services available but uncomfortable to use. Meta-monitoring HappyFace Project : Very useful for daily administration of services at e.g. GridKa CMS Tier1 Computing - DESY DVSEM,

38 Backup Backup CMS Tier1 Computing - DESY DVSEM,

39 GridKa Resource Requests *Numbers for 2013 uncertain. **Numbers for 2014 unofficial. CMS Tier1 Computing - DESY DVSEM,