CMS Computing Model with Focus on German Tier1 Activities Seminar über Datenverarbeitung in der Hochenergiephysik DESY Hamburg, 24.11.2008
Overview The Large Hadron Collider The Compact Muon Solenoid CMS Computing Model CMS in Germany CMS Workflow & Data Transfers CMS Service Challenges Monitoring HappyFace Project CMS Tier1 Computing - DESY DVSEM, 24.11.2008 2
Large Hadron Collider ALICE CMS ATLAS LHCb Proton-Proton Collider Circumference: 27 km Beam Energy: 7 TeV/c² Below Surface: 100 m Temperature: -271 C Energy Use: 1 TWh/a 4 Large Experiments CMS (General-Purpose) Atlas (General-Purpose) LHCb (Physics of b-quarks) Alice (Lead Ion Collisions) 2 Smaller Experiments LHCf and Totem CMS Tier1 Computing - DESY DVSEM, 24.11.2008 3
Compact Muon Solenoid Muon Chambers Hadronic Calorimetry Magnetic Retrun Yoke Technical Details: Total Weigh: Diameter: Total Length: Magnetic Field Solenoid: Yoke: 12 500 t 15 m 21,5 m 4 Tesla 2 Tesla Silicon Tracker Readout Channels, e.g. Tracker: 10 Mio. Superconducting Solenoid Electromagnetic Calorimetry Forward Calorimetry Collission Rate: 40 MHz CMS Tier1 Computing - DESY DVSEM, 24.11.2008 4
Pictures of CMS CMS Tier1 Computing - DESY DVSEM, 24.11.2008 5
CMS Collaboration CMS 38 Nations 182 Institutions > 2000 Scientists & Engineers CMS Tier1 Computing - DESY DVSEM, 24.11.2008 6
Physics Motivation Test of the Standard Model (at the TeV energy scale) Search for the Higgs-Boson In each proton-proton collision, more than 1000 particles are created. The decay products allow to conclude on underlying physics processes of the collision. Physics beyond the SM (e.g. SUSY, extra dimensions,...) CMS Tier1 Computing - DESY DVSEM, 24.11.2008 7
Trigger and Event Rate 60 TB/sec Level 1 Trigger Reduction with ASICs (Hardware) Collision Rate: 40 MHz Event-Size: 1,5 MB 150 GB/sec Tape & HDD Storage for Offline- Analysis Recorded Events: 150 per second 225 MB/sec High Level Trigger Software Data Reduction CMS Tier1 Computing - DESY DVSEM, 24.11.2008 8
[ TB ] [ TB ] [ MSI2k ] Expected Resources CPU 160 140 120 100 80 60 40 20 0 2007 2008 2009 2010 Year Disk Space MSS Space 80000 70000 70000 60000 50000 40000 30000 20000 10000 60000 50000 40000 30000 20000 10000 0 0 2007 2008 2009 2010 2007 2008 2009 2010 Year Year CMS Tier1 Computing - DESY DVSEM, 24.11.2008 9
CMS Computing Model LHC experiments have decided to use distributed computing and storage resources. The Worldwide LHC Computing Grid (WLCG). Grid services based on the glite middleware. Computing centres arranged in a four-tiered hierarchical structure. Availability and resources are regulated by MOU (e.g. Downtime per year, response time, etc.). CMS Tier1 Computing - DESY DVSEM, 24.11.2008 10
CMS Tier Structure CMS Tier1 Computing - DESY DVSEM, 24.11.2008 11
WLCG Resources Taken from the WLCG Real Time Monitor: http://gridportal.hep.ph.ic.ac.uk/rtm/ CMS Tier1 Computing - DESY DVSEM, 24.11.2008 12
Main Tier Responsibilities Tier0: Storage of RAW detector data. First reconstruction of physics objects after data-taking. Tier1: Host one dedicated copy of RAW and reconstructed data outside the Tier0. Re-processing of stored data. Skimming (creation of small sub data samples) Tier2: Monte Carlo production/simulation. Calibration activities. Resources for physics groups analyses. Tier3: Individual user analyses. Interactive logins (for development & debugging) CMS Tier1 Computing - DESY DVSEM, 24.11.2008 13
German CMS Members Uni Hamburg DESY Hamburg/ Zeuthen RWTH Aachen FSP-CMS (BMBF) Forschungszentrum Karlsruhe Uni Karlsruhe CMS Tier1 Computing - DESY DVSEM, 24.11.2008 14
Tier1 GridKa Located at Forschungszentrum Karlsruhe (KIT, Campus Nord) Part of and operated by the Steinbuch Centre for Computing (SCC, KIT) Multi-VO Tier centre (supports 8 experiments) 4 LHC experiments: CMS, ATLAS, ALICE, LHCb 4 non-lhc experiments: CDF, D0, BaBar, Compass CMS Tier1 Computing - DESY DVSEM, 24.11.2008 15
GridKa Resources (2008) Network: CPU: 10 Gbit link CERN GridKa 3 x 10 Gbit link to 3 European Tier1s 10 Gbit link to DFN/X-Win (e.g. Tier2 connections) CMS Resources: Storage: Based on dcache (developed at DESY and FNAL) CMS Disk: ~ 650 TB + D-Grid: ~ 40 TB CMS Tape: ~ 900 TB ~ 800 cores, 2 GB Ram per core D-Grid Resources: ~ 500 cores, 2 GB Ram per core CMS Tier1 Computing - DESY DVSEM, 24.11.2008 16
Tier2/3 and NAF Resources Besides the Tier1 resources, considerable CPU and disk capacities are available in Germany (August 2008): Tier2 (German CMS Tier2 federation): DESY: RWTH Aachen: 400 cores, 185 TB disk 360 cores, 99 TB disk Tier3: RWTH Aachen: Uni Karlsruhe: Uni Hamburg: 850 cores, 165 TB disk 500 cores, 150 TB disk 15 TB disk NAF (National Analysis Facility): DESY: 245 cores, 32 TB disk D-Grid resources, partially usable for FSP-CMS: RWTH Aachen: GridKA: 600 cores, 231 TB disk 500 cores, 40 TB disk CMS Tier1 Computing - DESY DVSEM, 24.11.2008 17
CMS Workflow 1 Transfers 2 Transfers 3 CMS Tier1 Computing - DESY DVSEM, 24.11.2008 18
PhEDEx Data Transfers Physics Experiment Data Export (CMS data placement tool): Various agents/daemons are run at the tier sites (depending on the tier level) upload, download, MSS stage-in/out, DB,... Provides automatic WAN data transfers, based on subscriptions Automatic load-balancing File transfer routing topology (FTS) Automatic bookkeeping (database entries, logging) Consistency checks (DB vs. filesystem) File integrity checks CMS Tier1 Computing - DESY DVSEM, 24.11.2008 19
Other Involved Components Monte Carlo Production Agent (ProdAgent) CMS Remote Analysis Builder (CRAB) Dataset Bookkeeping System (DBS) Data Location Service (DLS) Trivial File Catalog (TFC) Grid middleware (glite) SE, CE, RB, UI,... DBS: Relates dataset/block and site DLS: Relates dataset/block and logical file name (LFN) TFC: Relates LFN and local file name Very complex system Needs intense testing, debugging and optimisation. CMS Tier1 Computing - DESY DVSEM, 24.11.2008 20
WLCG Job Workflow User Interface Computing Element 6. Job execution Task Queue RB Storage glite WMS WM Proxy 2. Job Workload Manager 4. Job Job Controller CondorG 6. Grid enabled data transfer & access Information Service Match Maker Information Supermarket Information System Storage Element CMS Tier1 Computing - DESY DVSEM, 24.11.2008 21
CMS Service Challenges - 1 Computing, Service and Analysis (CSA) Challenges: Test the readiness for data-taking CSA06, CSA07 and CSA08 were performed to test the computing and network infrastructure and the computing resources at a level of 25%, 50% and 100% required for the LHC start-up. The whole CMS workflow was tested: Production/Simulation Prompt reconstruction at the Tier0 Re-reconstruction at the Tier1s Data distribution to Tier1s and Tier2s Data Analyses CMS Tier1 Computing - DESY DVSEM, 24.11.2008 22
CMS Service Challenges - 2 Cumulative data volume transferred during CSA08: > 2 PetaByte in 4 weeks Transfer rate from the Tier0 to the Tier1s: < 600 MB/sec Goal was achieved only partially, problems with: Storage systems (CASTOR at CERN) Problems identified and fixed afterwards. CMS Tier1 Computing - DESY DVSEM, 24.11.2008 23
CMS Service Challenges - 3 CMS Grid-job statistics: May to August 2008 German centres GridKa, DESY and Aachen performed extremely well, compared to all other CMS tier sites. CMS Tier1 Computing - DESY DVSEM, 24.11.2008 24
Results of CMS Challenges Service Goal 2008 Status 2008 Goal 2007 Status 2007 Goal 2006 Status 2006 Tier-0 Reco Rate 150-300 Hz Achieved 100 Hz Only at bursts 50 Hz Achieved Tier-0 Tier-1 Transfer Rate 600 MB/sec Achieved partially 300 MB/sec Only at bursts 150 MB/sec Achieved Tier-1 Tier-2 Transfer Rate 50-500 MB/sec Achieved 20-200 MB/sec Achieved partially 10-100 MB/sec Achieved Tier-1 Tier-1 Transfer Rate 100 MB/sec Achieved 50 MB/sec Achieved partially N/A - Tier-1 Job Submission 50 000 jobs/day Achieved 25 000 jobs/day Achieved 12 000 jobs/day 3 000 jobs/day Tier-2 Job Submission 150 000 jobs/day Achieved 75 000 jobs/day 20 000 jobs/day 48 000 jobs/day Achieved Monte Carlo Simulation 1.5 x 10 9 events/year Achieved 50 x 10 6 events/month Achieved N/A - CMS Tier1 Computing - DESY DVSEM, 24.11.2008 25
First Real CMS Data 20 TB/day 10 TB/day Data Imports to GridKa of cosmic muon samples (per day) Stable running over large periods of time First real beam event: LHC Beam hitting collimators near CMS (Sept. 10th 2008). GridKa was the first CMS tier centre outside CERN with real beam data. CMS Tier1 Computing - DESY DVSEM, 24.11.2008 26
Monitoring - Overview Distributed Grid resources are complex systems. To provide a stable and reliable service, monitoring of the different services and resources is indispensable. Use monitoring to discover failures before the actual services are affected. Different types of monitoring: Central monitoring of all Tier centres Status of the high level grid services and their interconnection Availability of the physical resources of the Tier centres. Central monitoring of experiment specific components Local monitoring at the different computing sites Local monitoring of experiment specific components Many different monitoring instances for different purposes! CMS Tier1 Computing - DESY DVSEM, 24.11.2008 27
Grid Monitoring Central WLCG Monitoring: Service Availability Monitoring (SAM) Standardised Grid jobs, testing the basic grid functionality of a Tier centre. Includes all Grid services of a site. Centrally send to the Tiers (once per hour) Output available on a web page with history Once a critical SAM test has failed on a site, no user jobs will be send to it anymore. CMS Tier1 Computing - DESY DVSEM, 24.11.2008 28
Experiment Monitoring Besides the basic grid functionality, experiment specific components and functionalities are tested. Access point to the results is the CMS ARDA Dashboard. http://arda-dashboard.cern.ch/cms/ SAM: Also tests of dedicated functionalities of the Tier centres requested by CMS. Job Summary: Type of the job and its exit code. History per centre and activity. PhEDEx data transfers: Transfer quality for CMS datasets. Collects error messages in case of failures. Gives overview on the transferred data samples available at the different sites. CMS Tier1 Computing - DESY DVSEM, 24.11.2008 29
Local Site Monitoring Each computing centre has its own monitoring infrastructure: Nagios Ganglia... Offers monitoring for hardware, services and network connections. Adapted for the local particularities. Automated notification in case of failures by email, instant messages, SMS,... Experiment specific local monitoring: Is the local software installation accessible? Are all required services properly configured? CMS Tier1 Computing - DESY DVSEM, 24.11.2008 30
USCHI Ultra Sophisticated CMS Hardware Information System Provides specific tests of the local infrastructure of a site. Mainly driven by previously observed problems. Designed in a modular way to ease the implementation of new tests. Currently, the following tests are running locally at GridKa: Availability of the CMS production software area Sufficient free inodes of the file system Basic grid functionality (e.g. user interface commands) FTS transfer channels (are they enabled, working, etc.) Remaining space of the disk-only area Status of the dcache tape-write buffers Functionality of basic VOBox commands PhEDEx grid proxy validity Output of the test provided in xml format Developed in Karlsruhe A. Scheurer, A. Trunov CMS Tier1 Computing - DESY DVSEM, 24.11.2008 31
Multi-Monitoring Drawbacks The current monitoring infrastructure is characterised by a multitude of different sources to query for information about one single site. Limitations of the current monitoring infrastructure: Information is not clearly arranged: Too much information not related to a dedicated site. Different site design and history plots / formats / technologies. Nearly impossible to correlate information from different sources. Uncomfortable to use: Access many different websites to get all needed information about one site. Different web interfaces require parameters (time range, site name,...) Often needs a long time to download / update plots (> 1 minute!) Increases administrational effort and slows down the identification of errors. A meta-monitoring system can help to improve this situation! CMS Tier1 Computing - DESY DVSEM, 24.11.2008 32
Ideas for Meta-Monitoring Only one website to access all relevant information for one site Simple warning system (with symbols) Single point of access for all relevant information (+ their history) Caching of important external plots for a fast access Backend for implementation of own tests Easy implementation of new tests Self-defined rules for service status and warning-alerts triggering Correlation of data from different sources Fast, simple and up to date All monitoring information should be renewed every 15 minutes Short access / loading times of the web page Easy to install and use - no complex database Access to every test result in less then 3 mouse clicks Provide links to raw information from the data source The meta-monitoring system HappyFace Project offers this functionality. CMS Tier1 Computing - DESY DVSEM, 24.11.2008 33
HappyFace Project - 1 http://ekpganglia.physik.uni-karlsruhe.de/~happyface/happyface/ CMS Tier1 Computing - DESY DVSEM, 24.11.2008 34
HappyFace Project - 2 Our HappyFace instance is currently monitoring the following quantities at GridKa: Local CMS infrastructure: the USCHI test-set Incoming/outgoing PhEDEx transfers: General transfer quality. Classification of error type. Classification of destination and source errors. Status of the WLCG production environment Queries SAM test results and displays eventually failed tests including a hyperlink to the error report. CMS Dashboard: Search Job Summary Service for failed jobs, print statistics of the exit codes. Fairshare of CMS Displays the fraction of running CMS jobs Caching of all plots necessary for monitoring CMS Tier1 Computing - DESY DVSEM, 24.11.2008 35
HappyFace Project - 3 Instances of HappyFace currently used by: Uni Karlsruhe RWTH Aachen Uni Hamburg Uni Göttingen Code and documentation available under: https://ekptrac.physik.uni-karlsruhe.de/happyface Developed in Karlsruhe V. Buege, V. Mauch, A. Trunov Future plans: Platform to share HappyFace modules Provide standardised output in xml format of all HappyFace instances Contact: Viktor Mauch: mauch@ekp.uni-karlsruhe.de Volker Buege: volker.buege@cern.ch CMS Tier1 Computing - DESY DVSEM, 24.11.2008 36
Summary CMS uses Grid technology for data access and processing. Multiple service challenges have proven the readiness for first data. PhEDEx is used for reliable large-scale data management. Experience has shown, that monitoring is indespensable for the operation of such a heterogeneous and complex system. Multitude of monitoring services available but uncomfortable to use. Meta-monitoring HappyFace Project : Very useful for daily administration of services at e.g. GridKa CMS Tier1 Computing - DESY DVSEM, 24.11.2008 37
Backup Backup CMS Tier1 Computing - DESY DVSEM, 24.11.2008 38
GridKa Resource Requests *Numbers for 2013 uncertain. **Numbers for 2014 unofficial. CMS Tier1 Computing - DESY DVSEM, 24.11.2008 39