The LCG 3D Project. Maria Girone, CERN. The 23rd Open Grid Forum - OGF23 4th June 2008, Barcelona. CERN IT Department CH-1211 Genève 23 Switzerland

Transcription

1 The LCG 3D Project Maria Girone, CERN The rd Open Grid Forum - OGF 4th June 2008, Barcelona

2 Outline Introduction The Distributed Database (3D) Project Streams Replication Technology and Performance Availability and Performance Monitoring Conclusions 2

3 Introduction A wide-range of mission-critical applications that are part of the Grid infrastructure rely on databases middleware and storage-related services, but also key infrastructure and operations services A large proportion of non-event data, including detector conditions, calibration, geometry and production book-keeping meta-data, are stored in (distributed) relational databases These services - together with data and storage management services - are rated consistently by the LHC VOs as amongst the most critical 3

4 3D Project Mandate Set up a Distributed Database Infrastructure for the WLCG (according to the WLCG MoU) Oracle Real Application Clusters as building-block architecture Tier0: 25 clusters (125 servers, 450 cores, 150 disk arrays, 2000 disks) Tens of servers at Tier1 10 sites Oracle streams replication used to form a database backbone between Online and offline at Tier0 ATLAS, CMS and LHCb Tier0 and 10 Tier1 sites ATLAS and LHCb Frontier/SQUID from Fermilab for distributing and caching database data via a web protocol is used by CMS Database services listed among the critical services by the experiments 4

5 Database set-up at CERN RAC 1 RAC 2 RAC 5 RAC 3 RA C6 RAC 4

6 Oracle Streams Replication Technology for sharing information between databases Database changes captured from the redo-log and propagated asynchronously as Logical Change Records (LCRs) Source Database Target Database Propagate Redo Logs Capture Apply 6

7 Downstream Capture Downstream capture to de-couple Tier 0 production databases from destination or network problems source database performance and availability is our highest priority Optimizing redo log retention on downstream database to allow for sufficient re-synchronisation window we use 5 days retention to avoid tape access Source Database TCP and Oracle protocol optimisations yielded significant throughput improvements (almost factor 10) Redo Logs Target Database Downstream Database Propagate Redo Transport method Capture Apply 7

8 Streams setup for ATLAS Online Offline Tier1 sites (10) Real-Time downstream capture Oracle Use of rules Database size: 1.44 TB 8

9 Online to Offline replication ATLAS PVSS tests (online -> offline) 6 GB per day LCRs per second 9

10 Replication to Tier1 sites ATLAS Conditions data (Tier0->Tier1s) 2 GB per day LCRs per second 10

11 Streams Setup for LHCb (Conditions and LFC) 11

12 Database & Streams Monitoring, WLCG procedures Dashboard of current DB activity server usage, availability and performance streams replication activity Streams topology Status of streams connections Error notifications Streams performance (latency, throughput, etc.) 3D monitoring and diagnostics integrated in the experiments dashboards and with WLCG procedures and tools Interventions announced according to the established WLCG procedures EGEE broadcasts, GOCDB entries 12

13 DB availability and performance page

14 Streams Monitoring 14

15 Summary The LCG Database Deployment Project (LCG 3D) has set up a world-wide distributed database infrastructure for LHC some 124 RAC nodes = 450 CPU cores at CERN + several tens of nodes at 10 partner sites are in production now Large scale tests have validated that experiments requirements are implemented by the RAC & streams based set-up Database & streams availability and performance monitoring has been implemented key tool for the LCG community Distributed database infrastructure is ready for accelerator turn-on Smooth running during CCRC 08 Experiments are ramping-up to full use of the Tier1 infrastructure 15

16 More Details LCG 3D wiki interventions, performance summaries Physics Database at CERN wiki Support: 16