高能物理分布式计算技术 张晓梅

Transcription

1 高能物理分布式计算技术 张晓梅

2 What is distributed computing in HEP?(1) Definition (Computing) A distributed system is a collection of autonomous hosts that are connected through a computer network. Each host executes components and operates a distribution middleware. Middleware enables the components to coordinate their activities. Users perceive the system as a single, integrated computing facility. Definition(HEP application) Coordinated resource sharing and problem solving in dynamic, multiinstitutional virtual organisations (I. Foster)

3 What is distributed computing in HEP?(2) Distributed computing originates from grid computing in HEP WLCG (World Wide LHC Computing Grid) EGI, OSG, ChinaGrid Not just grid, but other any possible resources Cloud Cluster HPC

4 Why distributed computing in HEP?(1) HEP Background Real need for very high performance infrastructures Large collaborations to share resources 8000 Physicists, 170 Sites, 34 Countries (LHC) 实验原始数据量总数据量 (10 年 ) CPU 资源 BESIII 100TB/ 年 10PB 1 万 LHC BELLEII 25PB/ 年 (6 年后 150PB/ 年 ) 10PB/ 年 (10 年后 200PB/ 年 ) 1000PB 500PB 60 万 13 万 ILC 10PB/ 年 100PB 8 万 CEPC 20PB/ 年 (15 年 ) 200PB(25 年后 ) 16 万核 Z 工厂 100PB/ 年 (25 100PB(25 年后 ) 10 万核

5 Why distributed computing in HEP?(2) Main reasons Funding Required computing power was far beyond the funding capacity available Use free resources from national or regional computing facilities or local funding Sharing Multiple copies of data can be kept in different sites, ensuring access for all scientists involved, independent of geographical location Having computer centres in multiple time zones eases round-the-clock monitoring and the availability of expert support. Robust and flexible No single points of failure. The system can be easily reconfigured to face new challenges, making it able to dynamically evolve throughout the life of the LHC, growing in capacity to meet the rising demand as more data is collected each year

6 Grid Initiatives Worldwide Australia Nimrod-G Gridbus GrangeNet. APACGrid ARC eresearch Brazil OurGrid, EasyGrid LNCC-Grid + many others China ChinaGrid Education CNGrid - application Europe UK escience EU Grids.. and many more... India Garuda Japan NAREGI Korea... N*Grid Singapore NGP USA Globus GridSec AccessGrid TeraGrid Cyberinfrasture and many more... Industry Initiatives IBM On Demand Computing HP Adaptive Computing Sun N1 Microsoft -.NET Oracle 10g Infosys Enterprise Grid Satyam Business Grid StorageTek Grid.. and many more Public Forums Open Grid Forum Australian Grid Forum Conferences: CCGrid Grid HPDC E-Science

7 WLCG A global collaboration of computer centers, launched in 2002 The world's largest computing grid based on two main grids European Grid Infrastructure in Europe, and Open Science Grid in the US many associated regional and national grids, such as TWGrid in Taiwan and EU-IndiaGrid Collaboration through an MoU between CERN (LCG project and experiments) and 50 funding agencies Resources and service quality are pledged to experiments as part of the MoU

8 Grid Challenges Security Computational Economy Uniform Access System Management Resource Discovery Resource Allocation & Scheduling Data locality Application Construction Network Management

9 WLCG

10 WLCG

11 Grid services layer

12 Grid middleware Globus Argonne National Lab and ISI Gridbus University of Melbourne Unicore European Project (Germany ) Legion University of Virginia Globus Applications Third Party User-Level Middleware Grid Resource Management (GRAM, GASS) Grid Information Services (MDS) Grid Data Management (GridFTP, Replica Catalog) GSI Security Layer Grid Resources and Local Services

13 Globus Toolkit Globus Toolkit is the source of many of the protocols described in Grid architecture Globus Toolkit is a fundamental enabling technology for building grids that allow distributed computing power, storage resources, scientific instruments, and other tools to be shared securely across corporate, institutional, and geographic boundaries. Adopted by almost all major Grid projects worldwide as a source of infrastructure Open source, open architecture framework encourages community development Active R&D program continues to move technology forward Developers at ANL, USC/ISI, NCSA, LBNL, and other institutions

14 Globus Toolkit Services Security (GSI) PKI-based(public key infrastructure) Security (Authentication) Service Job submission and management (GRAM) Uniform Job Submission Information services (MDS) LDAP-based Information Service Remote file management (GASS) Remote Storage Access Service Remote Data Catalogue and Management Tools LFC Recently move to: WSRF (Web Services Resource Framework)

15 GRAM Components Client MDS client API calls to locate resources MDS: Grid Index Info Server MDS client API calls to get resource info Site boundary GRAM client API calls to request resource allocation and process creation. GRAM client API state Globus change callbacks Security Infrastructure Gatekeeper Create Job Manager Parse RSL Library MDS: Grid Resource Info Server Query current status of resource Local Resource Manager Request Monitor & control Process Process Process Allocate & create processes

16 Grid Components

17

18

19

20

21

22

23

24 Common services

25

26 Tiered structure 4 layers Computing Model Tier-0: CERN: accelerator Data Acquisition and Reconstruction Data Distribution to Tier-1 (~online) Tier-1 24x7 Access and Availability, Quasi-online data Acquisition Data Service on the Grid Heavy Analysis of the data Tier-2 Simulation Final User, Analysis of the data (batch and interactive modes) Tier-3 Final User, Scientific analysis Tier-0 (1) Tier-1 (11) Tier-2 (160) Tier-3 (End User)

27

28 LHCOPN and perfsonar T0<->T1 10Gbps~100Gbps T2 > 1Gbps LHCOPN- Large Hadron Collider Optical Private Network for T0~T1 perfsonar is a web services-based infrastructure for collecting and publishing network performance monitoring. A primary goal of perfsonar is making it easier to solve end-to-end performance problems on paths crossing several networks

29 CMS 1. Physics Goals: search for the Higgs boson, extra dimensions, and particles that could make up dark matter. 2. Compact Muon Solenoid experiment (CMS) Compact: compared to weight Muon : high momentum particles Solenoid: the largest superconducting magnet, strong and large magnetic field for detecting Muon

30

31 CMS computing model Data to be processed 300MB/s DAQ BES(70~80MB/s) 2PB/year raw data Tier architecture 1 T0 (cern), 8 T1, T2, T3 Resource, data and job distribution, operation mode Computing resource(2010) Resource distribution: 20% CERN, 40% T1, 40% T2 T0: 11.5 MSI2K, 12Gb/s, 0.6PB/disk,12PB/tape, T1: 6.8MSI2K/T1, 16Gb/s, 2.6PB/disk, 7PB/tape

32 CMS data management Feature: Distributed system, Based on Dataset Subsystems: DBS(Data Booking System) -- what is the dataset DBS DLS(Data Locating System) -- where is the dataset DLS+TFC DTS(Data Transferring System) -- transfer dataset PhEDEx DSS(Data Storage System) -- store mass data dcache, Castor, DPM, StoRM, BeStMan.. DAS(Non-event Data Accessing System) -- non-event data accessing system Frontier/Squid

33 PhEDEx

34 CMS Job Management System Job Submission System Crab : user analysis jobs submission tools ProdAgent: official production jobs submission tools Job Scheduling System CRAB Server, GlideinWMS Job Monitoring System

35

36 BESIII experiment Located in Beijing, shijingshan district Physics Goals: study electron-positron collisions in the tau-charm threshold region ( Energy: 1.0~2.3 Gev) Accelerator: BEPCII Detector: BESIII The luminosity of BESIII is 100 time of BESII one 2009 ~now Start of physics run data taking In 2012, about 0.6PB data have been aggregated 2020 is estimated to have about 7 PB

37 BESIII collaboration China, Germany, Italy, Japan, Korea, Netherlands, Pakistan, Russia, Sweden, Turkey, USA

38 Resource Why distributed computing IHEP computing center 6600 cores, 5PB tape Need extra computing resources as a supplement Increasing data volume Peek needs in some periods Opportunistic resources Resources from BESIII collaboration Cloud, volunteer computing IHEP resources Time

39 BESIII data processing Main BESIII offline data processing: simulation, reconstruction, calibration, physics analysis Simulation with no input is most suitable to do in the distributed sites Analysis with small inputs can be considered

40 BESIII Distributed Computing Model Data taking at IHEP IHEP as central site Raw data processing, bulk reconstruction, analysis. Central storage for all the data Remote sites MC production, analysis Data flow Simulation data produced in remote sites transferred back by transfer tools or directly written back to IHEP by jobs for permanent storage Reconstructed data (DST) transferred to remote sites for particular analysis

41 Workload Management Main Components DIRAC (Distributed Infrastructure with Remote Agent Control) interware to cope with heterogeneous resources GANGA Massive job submission user interface CVMFS (CERN VM File System) deploy BOSS to remote sites GANGA DIRAC grid cluster cloud

42 DIRAC as a Interware DIRAC builds a layer between the users and the resources Providing users an optimized, transparent and reliable usage of the resources Resources supported Grid, cluster, cloud Components services, agents, interface Workload scheduling scheme pull scheduling and pilot agent paradigm Pilots get resources and pull jobs from center task queue fault tolerance, flexible and efficient

43 GANGA and GangaBOSS GANGA is a user-friendly framework that handles mass job definition and management with modular architecture GangaBOSS is the package developed based on basic modules Closely integrated with BOSS applications Auto handle complete life cycle of BOSS jobs in grid and cluster Hide grid complexity to end users Well work with DIRAC and local clusters to complete job management Build Configure Split Submit Monitor Merge

44 JSUB

45 CVMFS Current size of BESIII offline software ~7GB It is not convenient to frequently update BOSS version for each sites It is not possible to carry it in jobs CERN Virtual Machine File System Based on FUSE, mount web directory as local disk Local cache to fasten access Sites can get all the BOSS versions with CVMFS client installed Commonly used in grid and different experiments can share the same service

46 Data Management Most of data processing and transfer are based on dataset Badger (BESIII Advanced Data Manager) being developed for BESIII dataset and metadata management Replica Catalogue - map logical file names (LFNs) to physical file names (PFNs) at different sites Metadata Catalogue define metadata for datasets and files Dataset Catalogue define datasets and related operations Purpose of Badger Export distributed data to users with a uniform interface Allow production users to register output as datasets Allow analysis users to readily find and access datasets

47 BADGER in Job flow BADGER work closely with workload management to complete data processing Jobs can automatically register output files as datasets in BADGER Jobs can easily find input files with metadata query or with dataset name from BADGER

48 Data transfer Goals Transfer DST datasets to sites for analysis Copy back MC production job outputs to IHEP central SE Tools have been developed based on DIRAC to support dataset-based transfer Users can submit and monitor transfer requests through web interface Transfer service take care of bulk transfer automatically with dataset name Glite File Transfer Service (FTS2) has been established to manage multichannels transfers The channel between JINR and IHEP is tested, with maxium speed about 23MB/s

49 Action-based Monitoring Motivation Improve stability of system Ease the life of admins Clearly expose site status to end users Components Decision System Action Executor Policies Actions Status Evaluator Information Collection and Display Decision and actions Both Active and Passive info collected Various display to have a complete view of site status (next slide) Policies can be defined for automatic actions taken in case of problems Sending warning messages, ban sites Collection System Monitoring Information Information Collector Passive Monitoring Site Status Active Monitoring Real time history Information Display Information Sources

50 Monitoring Monitoring information display

51 backup

52 Site Contribution About 7 sites including IHEP joined now 5 sites from china, 1 from Russia, 1 from U.S. 1 glite site, others are cluster Peek CPU resources used can reach 2169 cores Maximum resources are not stable from irregular shape of waves resources are shared by other applications as well in most of sites

53 First production , production of 200 million y bhabha events 5 remote sites joined More than 8000 jobs are submitted and run More than 1600 CPU days are used Data transferred back to IHEP SE About 237GB production data are transferred back to IHEP SE Other parts of data failed to transfer back because of SE space

54 Second production ~9.23 produced 800 million Jpsi inclusive events with Use BOSS sites joined Almost jobs were submitted and done About 6TB data has been transferred back to IHEP SE

55 Third production Psi(3770) MC simulation 1,352.3 million Events have been simulated Status 69,904 Jobs has been completed TB data has been generated and transferred 20,943.5 CPU days has been used Success rate reach 90% Main failure from problems of work node

56 Summary The system has been put into production Have proved to be working Contribute to BESIII simulation tasks More sites are joining and resources are increasing More efforts and developments are needed to make system more robust and make operation life easier