Distributing storage of LHC data - in the nordic countries Gerd Behrmann INTEGRATE ASG Lund, May 11th, 2016
Agenda WLCG: A world wide computing grid for the LHC NDGF: The Nordic Tier 1 dcache: Distributed data storage for e-science 2
Graphics extremetech.com 3
Graphics CERN 4
Graphics CERN 5
Graphics CERN 6
7
World wide LHC computing grid (WLCG) 8
Graphics CERN 9
585,530 cores 536 PB disk 540 PB tape Graphics CERN 9
585,530 cores 536 PB disk 540 PB tape Used to be strictly hierarchical with clearly defined data flows and roles for what is computed where (the MONARCH model). Becoming more dynamic, with remote data access, caching, federation. Still a clear distinction between Tier 1 and Tier 2 in terms of Service level Data retention Contact with experiments Regional support role Graphics CERN 9
Graphics GEANT 10
Agenda WLCG: A world wide computing grid for the LHC NDGF: The Nordic Tier 1 dcache: Distributed data storage for e-science 11
A Tier 1 in the Nordic Countries Computing Element (batch system with remote access) Storage Element (a locally and remotely accessible file system) A tape system A File Transfer Service (transfers files between your tier 1 and other sites) 24x7 monitoring, with 4 hour response time for alarms. A liaison for each supported experiment. Enough man power to join lots of periodic meetings. Enough man power to provide expert support for your region. LHC runtime + 15 years commitment. 12
NT1 aka the Nordic Data Grid Facility A distributed Tier 1-4 countries, 6 sites (7 originally) [ + Slovenia]. Build on the foundations of NorduGrid. First production scale instance in 2006. Since 2012 as part of the Nordic e- Infrastructure Collaboration. Physical resources at existing super computing centers - run by local staff. A virtual team of people to fulfill Tier 1 obligations and to maintain the illusion of a single site. 13
What s in a Name? NorduGrid Loose collaboration of Nordic HEP community. Founded in 2001 as Scandinavian Grid. Formed around the Advanced Resource Connector (ARC) software, developed by the NorduGrid community. Now over 80 sites in 19 countries. Nordic Data Grid Facility (NDGF) The Nordic Tier 1 site founded in 2006, layered on top of a slice of NorduGrid. Nordic e-infrastructure Collaboration (NeIC) Nordic collaboration founded in 2012 to facilitate development of IT tools and services in areas of importance to Nordic researchers. Nordic Tier 1 (NT1) An area of NeIC maintaining and operating NDGF. 14
CE CE CE CE Finishing Batch farm Other Tier 2s SE Preparing ARC Data taking Job Cache & session data Other Tier 1s AcT LHC PANDA 15
The queue after preparing and before execution isolates the job from the SE. Allows downtime on the SE without draining batch farms or having idle compute nodes. SE can be designed for density and cache/session disk for IOPS. CE CE CE CE Finishing Batch farm SE ARC Other Tier 2s Preparing Data taking Job Cache & session data Other Tier 1s AcT LHC PANDA 15
NDGF Storage System stats 94 million files on disk 6 million files on tape 3.5 PB on tape 7.5 PB on disk 16
Agenda WLCG: A world wide computing grid for the LHC NDGF: The Nordic Tier 1 dcache: Distributed data storage for e-science 17
dcache is... software for providing scalable, managed storage for huge amounts of data. deployed at research institutes throughout the world and used by a diverse collection of user communities. supported through the dcache.org collaboration, which provides: regular feature releases that are maintained with subsequent bugfix releases. Support and advice through a variety of channels. dcache is open source (AGPL 3). 18
Graphics dcache.org 19
Graphics dcache.org 20
Graphics dcache.org 21
Physical content (object store) Logical organization (database) End user access mechanism (protocol and endpoint) 22
Pool Physical content (object store) Logical organization (database) End user access mechanism (protocol and endpoint) 22
Pool Name space Physical content (object store) Logical organization (database) End user access mechanism (protocol and endpoint) 22
Pool Name space Physical content (object store) Logical organization (database) End user access mechanism (protocol and endpoint) FTP HTTP NFS 22
Pool Name space Physical content (object store) Request scheduling Monitoring Authorization Administration Message passing Logical organization (database) End user access mechanism (protocol and endpoint) FTP HTTP NFS 22
Graphics dcache.org 23
24
Graphics dcache.org 25