GRID AND HPC SUPPORT FOR NATIONAL PARTICIPATION IN LARGE-SCALE COLLABORATIONS

Transcription

1 GRID AND HPC SUPPORT FOR NATIONAL PARTICIPATION IN LARGE-SCALE COLLABORATIONS M. Dulea, S. Constantinescu, M. Ciubancan, T. Ivanoaica, C. Placinta, I.T. Vasile, D. Ciobanu-Zabet Department of Computational Physics and Information Technologies, 'Horia Hulubei' National Institute for R&D in Physics and Nuclear Engineering (IFIN-HH), Magurele, Romania , GRID2012, Dubna 1

2 NATIONAL GRID INFRASTRUCTURE Resource centres: 11 active sites hosted by 7 institutions Total number of cores: 5830 Total disk capacity: 1,8 PB Network infrastructure provided by NREN: RoEduNet (min. 10 Gbps) , GRID2012, Dubna 2

3 SUPPORTED VOs AND GRID USAGE WLCG VOs: cover 98% of the national Grid production alice - 54% (supported by 3 centres) atlas - 41% (supported by 4 centres) lhcb - 3% (supported by 3 centres) (reported period ) 2%: envirogrids.vo.eu-egee.org - models & scenarios for the environment of the Black Sea Catchment gridifin - provides framework for running non- HEP applications supported by the National Grid for Physics and Related Areas (GriNFiC) see, seegrid - regional (South East Europe) hone ilc , GRID2012, Dubna 3

4 Supported VOs and Grid usage ( ) Tier-2 Normalised CPU time per country (HEPSPEC06-hours) 1,58% 10 th position / 33; Total: ALICE: (6 th ) ATLAS: LHCb: , GRID2012, Dubna 4

5 Supported VOs and Grid usage ( ) Tier-2 number of jobs 3,63% 9 th position / 33 Total: alice: (first) atlas: (9 th ) 90.6% CPU efficiency (4 th ) lhcb: Years Tier-2 RO-LCG / ,74 2, / ,44 2, , GRID2012, Dubna 5

6 (DFCTI) COLLABORATIONS AND PROJECTS Worldwide LHC Computing Grid High-Performance Computing Infrastructure for South East Europe s Research Communities Coordination of the Romanian Tier-2 Federation RO-LCG, - composed of 5 institutions that participate in the WLCG collaboration with CERN (since 2006) Coordination of the Romanian HPC consortium that participates in the HP-SEE project (FP7-RI , ) Processing of PetaByte-Scale FR Cloud GriCeFCo Collaboration IRFU/CEA - IFIN-HH, on Efficient Handling and Processing Petabite Scale Data for the Computing Centres within the French Cloud (HaPPSDag) Coordination of the National Grid for Research in Physics and Related Areas, built through EU structural funds (SOP IEC Grid) RO - LIT-JINR / Dubna collaboration Hulubei-Meshcheriakov programme ( ), project Optimization Investigations of the GRID and Parallel Computing Facilities at LIT-JINR and Magurele Campus , GRID2012, Dubna 6

7 GRID AND HPC DFCTI Data Center ^ APC InfraStruXure TM IFIN_Bio > Myrinet cores COMPUTING CENTER IFIN_BC > Infiniband 1040 cores , GRID2012, Dubna 7

8 LOCAL NETWORK TOPOLOGY DFCTI hosts 3 grid sites: RO-07-NIPNE ( alice, atlas, lhcb) RO-11-NIPNE (lhcb) IFIN GRID (GriNFiC) DD: dpm disk storage servers 2 CREAM CE, VO-Box, etc. Running atlas analysis jobs required a scalable solution for handling the increasing data transfers between the SE and WNs. A convenient solution was the stacking of switches, wich can provide the required minimum bandwidth when the number of simultaneous transfers grows. RO-07 IFIN GRID , GRID2012, Dubna 8

9 MONITORING Provided at 4 levels: 1) utility and support equipment (electric power, UPS, cooling, etc) 2) data traffic on main switches 3) grid activity: number of jobs of different types, Grid production ( normalised CPU time ), the traffic on the main servers (SE, GW, etc.), on a common interface: , GRID2012, Dubna 9

10 SERVICE AVAILABILITY MONITORING SAM provided by IFIN GRID, through ifops VO; results transferred to WMS and published by Nagios ifops is dedicated to the monitoring of the sites of GriNFiC, including those of RO-LCG The GUI allows to compare the results of the ifops tests with those of the ops tests performed by NGI and published on the EGI monitoring portal , GRID2012, Dubna 10

11 NETWORK SUPPORT from the sublime to the ridiculous... Extensive investigation of the connectivity in HaPPSDaG Most of the RO-LCG resources are connected to the NREN's NOC through a 12 km FO aerial cable (10 Gbps). perfsonar tests make no sense if this connection is cut This begins to happen rather frequently Below: reaching 8,5 Gbps throughput, during alice files transfer, just before the link being cut , GRID2012, Dubna 11

12 HPC INFRASTRUCTURE The development of the HPC infrastructure started in 2006 and benefited of the cooperation with LIT-JINR. There are now 4 parallel clusters in technology Infiniband, Myrinet 2G and GbE. The large-scale computations are performed on the IFIN_BC cluster (IBM Blade Center) Server type: IBM QS22 IBM LS22 IBM HS22 Total CPU IBM PowerXCell 8i AMD Opteron 2376 Intel Xeon X5650 Clock frequency 3,2 GHz 2,3 GHz 2,67 GHz Core no. / cpu 1x PPE + 8x SPE 4 6 Cache level2/cpu 512 KB 512 KB 6x256 KB FSB frequency MHz HDD / node 8 GB SSD 76 / 146 GB SAS 500 GB SAS RAM / node 32 GB 8 GB 24 / 36 / 48 Total RAM 512 GB 80 GB 21x24+23x36+12x48 GB 1908 GB Nodes total CPUs total Cores total 32x PPE + 256x SPE Interconnects Infiniband 4x QDR 40 Gbps , GRID2012, Dubna 12

13 HPC SUPPORT FOR RESEARCH PROJECTS Simulation and modeling of large biomolecular systems by means of molecular dynamics codes (e.g. NAMD) (collab. with Fac. of Biology / Univ. of Bucharest) Modeling drug - efflux pump inhibitors interaction dynamics: Activity modeling and simulation of efflux pump inhibitors based on advanced laser methods (collaboration with INFLPR) Dynamics of Bose-Einstein condensates (collaboration with IPB, Belgrade) Modeling of seismic events (INCDFP collaboration) Ab-initio investigation of charge transport in nanostructures (collab. with Physics Faculty / Univ. of Bucharest) , GRID2012, Dubna 13

14 HPC SUPPORT FOR REGIONAL RESEARCH COMMUNITIES 1/2 High-Performance Computing Infrastructure for South East Europe s Research Communities (HP-SEE) ( ) Coordinator: GRNET, Greece 14 partners DFCTI leads the Romanian consortium: IFIN-HH UVT UPB (NCIT) ISS Contribution to the realization of an common integrated HPC infrastructure in SEE (11 HPC resource centres). Grid middleware is used in in order to hide the various ways of accessing HPC sites provided by the local resource management systems (LRMS). Several middleware solutions can be used, but even this access layer can also be generalized by application specific, graphical portal solutions. Example: WS-PGRADE - can communicate with all the EMI legacy middleware solutions: ARC, dcache, glite, and UNICORE ( , GRID2012, Dubna 14

15 HPC SUPPORT FOR REGIONAL RESEARCH COMMUNITIES 2/2 ISyMAB - Integrated System for Modeling and data Analysis of complex Biomolecules The application provides a remote access framework on NAMD clusters which offers the users an integrated interface with analysis tools The user with access rights can launch jobs through pbs and execute shell scripts on various HP-SEE clusters In figure: the ISyMAB user can choose to work on IFIN_BC or on IPB, Belgrade. The working directory of the user on any cluster can be synchronized with the ISyMAB directory using the Sync buttons , GRID2012, Dubna 15

16 PROSPECTS 1/2 DFCTI will use a new building (in construction): 316 sqm datacenter, electric power: 750 kw , GRID2012, Dubna 16

17 PROSPECTS 2/2 SHORT- AND MID-TERM TASKS Realization of a redundant network connection (in collaboration with NREN), improving availability, and the integration in LHCONE Federated storage for the atlas sites RO-07-NIPNE, RO-02-NIPNE Simplifying the structure, operation, and management of the RO-LCG data centres LONG-TERM TASKS: Providing computing support for large-scale collaborations, on long term, based on international agreements: WLCG: implementing the post-ls1 strategy ELI-NP project (Extreme Light Infrastructure - Nuclear Physics, FAIR-GSI experiments (CBM, PANDA); ITER EURATOM programme, Integrated Tokamak Modelling Task Force (ITM-TF); etc , GRID2012, Dubna 17

18 CONCLUSIONS During the first years of LHC data taking, RO-LCG provided better production results than the Tier-2 average Scalable solutions were designed and implemented for improving the transfer, processing and storage of large datasets at the DFCTI center in the framework of the computing support of the ATLAS experiment at LHC-CERN. A significant contribution to the performance management of the Grid system came from the NGI-independent set of tools implemented for monitoring the data transfer, storage efficacy, and service availability in the resource centres; this avoids possible inconvenients related to the end of the EGI support. An integrated system for modeling, production runs, and data analysis of complex biomolecules was programmed for the molecular dynamics simulations performed in the framework of the HP-SEE infrastructure Urgent measures are required for improving the network connectivity, in order to participate to LHCONE , GRID2012, Dubna 18

19 INVITATION: RO-LCG CONFERENCE Grid, Cloud & High Performance Computing in Science Deadlines: : full paper submission , GRID2012, Dubna 19

20 ACKNOWLEDGEMENTS National contribution to the development of the LCG computing grid for elementary particle physics, project funded by the National Authority for Scientific Research (ANCS) under contract 8EU/2012 CEA - IFA Partnership, R&D project: Efficient Handling and Processing Petabite Scale Data for the Computing Centres within the French Cloud, contract C1-06/2010, co-funded by ANCS Collaboration with LIT-JINR / Dubna in the framework of the Hulubei- Meshcheriakov programme, project: Optimization Investigations of the Grid and Parallel Computing Facilities at LIT-JINR and Magurele Campus FP7 HP-SEE project: High-Performance Computing Infrastructure for South East Europe s Research Communities, contract FP7-RI , , GRID2012, Dubna 20

21 The 5th International Conference "Distributed Computing and Grid technologies in Science and Education" July 16-21, 2012 Dubna, Russia THANK YOU FOR YOUR ATTENTION! , GRID2012, Dubna 21