LCG data management at IN2P3 CC FTS SRM dcache HPSS

Transcription

1 jeudi 26 avril 2007 LCG data management at IN2P3 CC FTS SRM dcache HPSS Jonathan Schaeffer / Lionel Schwarz dcachemaster@cc.in2p3.fr

2 dcache Joint development by FNAL and DESY Cache disk manager with unique namespace Not open source, support provided Interfaced with various MSS (Enstore, TSM, OSM...) Most of Tier 1 sites in production 9

3 dcache in details Features Name space and data storage separation Multiple copied of the same file Automatic file replication on heavy load Catalog handled by PNFS (NFS2 implementation) Meta data in PostgreSQL database Local access: dcap + API POSIX, gsidcap, Xrootd, nfs Remote access: GFTP, HTTP, SRM (web service) 10

4 Namespace vs data storage Credit: Patrick Fuhrmann 11

5 The Pnfs Manager Manages the files/directories identification A file/directory is identified by a unique id called pnfsid, looks like A27D68 Manages meta data Size Pnfs path usable by end user /pnfs/in2p3.fr/data/cms/... Storage class State (precious, cached...) 12

6 The Pool Manager Decides which pools are eligible when a requests arrives with the following input information: Origin of the request (client IP or subnet) Path of the file in the PNFS namespace Direction of the request (WRITE or READ) Protocol used Then find the best pool to use (the one with the lowest cost) 13

7 Setting up pools Pools are entities that keeps a physical copy of a file Many pools on different hosts can be grouped to serve a large volume Too few pools: the service is degraded if 1 pool/host is down Pool node Pool node Too many pools: difficult to manage 14

8 Separating resources Storage groups (~=storage class) is defined in the PNFS namespace (this is called a tag ): /pnfs/in2p3.fr/data/atlas/raw > raw:atlas Storage Groups used to: Separate resources for a better usage Separate data for a better performance Make the maintaining easier Multi queues for different protocols (GFTP, dcap, Xrootd) 16

9 Read vs Write pools Write pools receive files from other sites with a high (but very well known) throughput Disk and network tuned for high throughput High quality hardware required for precious files Read pools are used by local jobs Disk access tuned for random access by lots of jobs High quality is not required but large volumes are needed RW pools are also used for traffic in both directions 17

10 CMS dcache pools setup Credit: Artem Trunov 18

11 Atlas pools setup (not implemented yet) RAW_MC 6.5TB 18 MB/s RAW HPSS 43 MB/s Tier-0 RAW ESD RAW RAW_MC Xx 26 MB/s 15 MB/s Credit: Ghita Rahal ESD Yy 51 MB/s Other T1 T1 Tier-1s Tier-2s AOD ESD 12TB he Cac d ns r da e t s Buffer 3 jours(avec facteur d efficacité): A te 1.5 TB pour 10MB/s AOD 2.5TB Xx ou Yy= taille(raw/esd) x #fichiers (RAW/ESD) par job x #max de jobs en machine 19

12 dcache protocols gridftp / http Pool node srm Head node dcap gsidcap Xrootd nfs pnfs Pool node 20

13 More details on protocols dcap is the native dcache protocol for local access rfcp /scratch/myfile dcap://dcache.in2p3.fr:22125/pnfs/in2p3.fr... fd=dcap_open( dcap://dcache.in2p3.fr:22125/pnfs/in2p3.fr... ); SRM is the standard grid access to the system srmcp file://scratch/myfile srm://dcache.in2p3.fr:8443/pnfs/in2p3.fr... GridFTP is the grid transport protocol g u c file://scratch/myfile gsiftp://dcache.in2p3.fr:2811/pnfs/in2p3.fr... Direct GridFTP access is not supported at IN2P3 CC GridFTP can be selected by SRM negociation as transport protocol 21

14 Local access: dcap vs SRM SRM is not recommended for local access (e.g. copy to local disk in a job): SRM is less efficient than dcap because it does a lot of things that are useless for a job SRM queues are centralized in SRM server and usually fit normal bulk transfers activity (T0 T1, T1 T1...). Using SRM in jobs would harm this activity. Secure dcap (gsidcap) should be used in local grid jobs Needs to be integrated in high level software 22

15 Local access: what about Xrootd? Xrootd is an optimized protocol for random access Used at IN2P3 CC for BaBar (interfaced with HPSS) Already chosen by Alice Using Xrootd at IN2P3 CC for LCG data implies: Using Xrootd dcache HPSS integration done for Alice Needs HPSS as storage reference system Using Xrootd door provided by dcache Not really well tested and needs some more developments Performance needs to be tested 23

16 Xrootd SRM HPSS integration for Alice HPSS (5) (4) dcache pnfsnode T1.root (3) Client T1.root? (1) (2) Data server: Xrootd Data server: Xrootd 14 data servers 150 TB of disk cache (6) Redirector server: Xrootd (etc ) 2 servers Credit: Jean-Yves Nief 24

17 Connection with HSM (1/2) Asynchronous migration (flushing) The file stays precious until is it migrated But stays on disk until the system needs space Retries in case of migration failures Flushing policy (minimum # of files, minimum volume...) Central flushing (not tested yet) 25

18 Connection with HSM (2/2) Synchronous staging A file is retrieved from HPSS when needed (triggered by a READ or CACHE operation) If a file already exists in a pool which is down, it is staged on another pool (if there is one available) Asynchronous deletion Done by a garbage collector on a regular basis 26

19 Separating resources in HPSS Storage Class in HPSS COS (Class Of Service) is not defined in HPSS namespace Associated with resources (type of disk, type of tape) Associated with policies (filesize, migration, purge...) Families are defined in the HPSS namespace Allow to group files from a family on the same tapes Subsystems Independent HPSS systems running on the same instance 28

20 Storage Classes levels PNFS Namespace /pnfs/in2p3.fr/data/atlas/tape/... /pnfs/in2p3.fr/data/cms/data/... dcache storage group tape-sc4:atlas hpssdata:cms HPSS Namespace HPSS storage class HPSS resources and policies /hpss/atlas/tape-sc4/... Family=142 Purge policy /hpss/cms/hpssdata/... COS=42 Family=23 Migration policy COS=0 Tape/disk ressources 29

21 HPSS connection Pool l) rfcp (contro RFIO server Mover rfcp (d ata) Head node ) a t a d ( rfcp Mover ) rfcp (data Pool rfcp (c ontrol) RFIO server 30

22 SRM/dCache/HPSS setup 33

23 dcache hardware setup (LCG system) 3 core nodes (PNFS, SRM, dcache) Head node: AMD bi opteron 2GHz, 2GB RAM SRM, Pnfs nodes: AMD quad pro 4GHz, 8GB RAM 25 disk servers (pools+gftp door) Various hardware, latest are SUN X4500 (Thumpers) Typical hardware: quad pro with 16GB RAM 270 TB allocated 240 TB on X4500 (20TB per host, 16 hosts) 34

24 dcache software setup SL3.0.5 or Solaris10 SL4 not tested yet XFS on Linux, ZFS on Solaris dcache installed in November 2006 PostgreSQL for pnfs, companion, billing, SRM Backup: DB dump+archive in local system (TSM) 35

25 FTS setup and configuration One single node for all agents (VO + channels) Bi pro 2GHz, 2GB RAM DB backend on Oracle cluster 10 T1 IN2P3 channels 27 IN2P3< >T2 channels (18 for Atlas) 12 STAR xxx channels (9 for Atlas) 38

26 Monitoring and operation dcache: 2 people, FTS: +1 (part time) Web monitoring (pool usage, current SRM requests, current errors, srmwatch...), dcache portal GUI for daily operation Integrated with local monitoring and logging systems Database query web pages integrated with local FTS database information Operation scripts for on duty people 41

27 dcache portal 42

28 Monitoring errors 43

29 Daily operation with dcache GUI 44

30 FTS monitor 45

31 FTS: monitoring a channel 46

32 FTS: monitoring a job 47

33 Central Logging System 48

34 SRM/dCache service Plans to describe different levels of intervention on the SRM/dCache system for example: administrator : all commands allowed operator : operations commands allowed only vo manager : basic commands allowed for VO only on duty : minimum commands for week end fixes Need to provide better diagnosis to map with other systems (HPSS, batch, FTS...) Need to automate operation commands ( mover kill...) 61

35 What next? Improve monitoring and operation tools Improve HSM staging process (group HPSS requests) Test and implement file hopping, central flushing, resilient pools... Deployment of SRM v2.2 (dcache 1.8) Testbed in PPS expected mid may Production system upgrade not expected before september 62