Deployment and Testing of Storage Management software, for CMS experiment

Transcription

1 Deployment and Testing of Storage Management software, for CMS experiment G.Donvito INFN Bari EGEE is a project funded by the European Union under contract IST

2 Introduction on SRM CMS Requirements Outline Experiment requirements (Numbers and use cases ) Users and administrators experiences and requirements DPM dcache STORM Summary on features and performances Conclusion IPRD06, 1-5 October

3 SRM Overview Storage Resource Manager SRM is a Control protocol What it does: Ask to make file ready for upload/download Basic metadata (size, checksum, ) Many components optional Web service (over GSI HTTP) What it doesn t: Data transfer However it can do third party transfer Access control & permissions However some implementation have already been tried IPRD06, 1-5 October

4 SRM functionalities Features from version 1.1 Get Put copy getfilemetadata getrequeststatus getprotocols AdvisoryDelete Critical subset of version 3 Tape-resident with system-managed disk cache: Tape1Disk0 == CUSTODIAL + NEARLINE Tape-resident with guaranteed copy on disk: Tape1Disk1 == CUSTODIAL + ONLINE Disk-resident, user-managed: Tape0Disk1 == REPLICA + ONLINE Critical subset of version 2.2 File types Space reservation Permission functions Directory functions Data transfer control functions Relative paths Query supported protocols IPRD06, 1-5 October

5 DATA TYPE STORAGE TYPE Tape: 1 Disk: 0 Tape: 1 Disk: 1 Tape: 0 Disk: 1 IPRD06, 1-5 October

6 CMS Requirements Services requirements: Reliable SRM Storage Elements Providing almost all needed features FTS and glite Services compliance SRM interoperability File access compliant with LHC software (2008) Network transfers between T0-T1 centers 2008 scale is ~300 MB/s (provision roughly twice that) Network transfers between T1-T2 centers 2008 peak rates from Tier-1 to all related Tiers-2 of MB/s (from 50 to 100 MB/sec per Tier2) Selection Submissions at Tier-1 centers ~800 MB/s to all WNs Analysis Submissions at Tier-2 centers Up to 1 GB/s to all WN IPRD06, 1-5 October

7 CMS Requirements Administrators requirements: Powerful configuration tools continued Powerful monitoring and debugging tools Powerful tools for handling with metadata and with files Capability of handling very large number of parallel requests User friendly installation and configuration procedures Tools to recover or prevent the failure Painless Back-up and Restore (data and metadata ) procedures (in case of failure) IPRD06, 1-5 October

8 DPM IPRD06, 1-5 October

9 DPM overview Developed mainly by Jean Philippe Bode at CERN It is the SRM supported by LCG Provides file access via (authenticated) RFIO and GridFTP It is possible to unify more physical pool under the same virtual namespace (browsable via dedicated command) It consists of a thin layer of software (C++) over MySQL DB easy to manage Not too heavy for CPU It is compatible with CLASSIC SE (it is possible to transform a CLASSIC SE in a DPM POOL) It has an implementation of SRM2.2 Each pool can be configured to serve just one or all VO (unix groups) It has a VOMS implementation in the latest release There is support for VOMS based ACL Accounting system based only on logs and space used per VO IPRD06, 1-5 October

10 Example: SRM put processing (1) SRM Server MySQL Server IPRD06, 1-5 October

11 Example: SRM put processing (2) SRM Server MySQL Server IPRD06, 1-5 October

12 Example: SRM put processing (3) SRM Server MySQL Server IPRD06, 1-5 October

13 Example: SRM put processing (4) SRM Server MySQL Server IPRD06, 1-5 October

14 Example: SRM put processing (5) SRM Server MySQL Server IPRD06, 1-5 October

15 Example: SRM put processing (6) SRM Server MySQL Server IPRD06, 1-5 October

16 DPM Tests Installed at many Tier2 in a production environment Simple to install and manage Simple to dedicate pools to VO (unix groups) Some problems with CMS software (RFIO library incompatibilities) Good performances with RFIO (LAN accesses) Problems with srmcp -pushmode= true (causes load on the other end-point) Installed at Bari to test the new functionalities (developed an automatic tests suite to test many SRMv2.x functionalities) Some new functionalities not yet at production level IPRD06, 1-5 October

17 DPM issues (Thank s to DPM Tiers2) SRMv2.2 not yet at production level (see back-up slides for details) Poor advanced functionalities for management remove files from root vacating a pool for hw problems Load limiting on a pool (Pool overload issue) Configurable Match-making Possibility to choose file destination Poor advanced functionalities for big sites move and replicate files between pools (both for managements and performance issue) Queue for local files requests (all requests must be served or rejected) Scalability to hundreds of TB to be proven More control on pool before writing Quota support has only Unix groups granularity It is not possible to associate dpm path to group of pools Monitoring and rate-mesuring functionalities are not available Developers Man Power IPRD06, 1-5 October

18 DPM Future plans Support for Xrootd Support for advanced SRMv2 functionalities Support for SRMCP Support for file replication between pools Support for cheap back-up system IPRD06, 1-5 October

19 dcache IPRD06, 1-5 October

20 dcache overview It is developed in a large collaboration between Desy and FNAL (plus some other minor contributions) GOALS: To make a distributed storage system that can use cheap disk-server to gain high performance and high-availability To provide an abstraction of whole disk space under a unique NFS like file-system (just for metadata operations) To possibly add the support for its own MSS system They are needed only 2 or 3 scripts (put/get/remove) To provide a system that scale at hundreds of TB of disk cache hundreds of pool nodes hundreds of TB per day to clients File access: provides local and remote access (posix like) with many protocols (dcap, ftp) both with and without authentication (gsi or kerberos) IPRD06, 1-5 October

21 dcache overview (2) Access management: access priority and load balancing obtained trough the use of different queue Allows multiple copy of files spread over different pools to improve performance and HA pool-2-pool automatic (or manually) transfers Allows dynamic match-making between pools According to the parameters chosen by the administrator (they can be based on disk space, load, network, type of access etc.) It is possible to split different type of access point (doors) on different nodes It is possible to move all the files in a pool to put it in a scheduled downtime Or just to choose which file you want to move and where. Also the central services can be split on different nodes to improve the scalability IPRD06, 1-5 October

22 Pool management: dcache overview (3) gives the possibility to create groups of pools named storage class (read, write, cache, or per VO and user bases or use bases) Can be useful for quota management Web monitoring, statistical module (also with rate-plot) The SRM layer can be used as stand-alone software (on standard Unix file-system) It is possible to choose the space used by dcache pool in a partition (you can host many services on the same partition) JAVA GUI for administration Also Xrootd protocol is supported (read only) Accounting system flat-files or DB based (not user friendly but there are many information) and space used per VO It is possible to use WN (or other not reliable space) disks to improve performance for local access IPRD06, 1-5 October

23 dcache overview (4) IPRD06, 1-5 October

24 dcache field test Used in production in many CMS Tiers1 and Tiers2 Used in production since may 2005 at INFN-BARI Good stability Achieved the performance needed for a tier2 in 2006 Good behavior with CMS software (both old and new framework) Performances measured at the CNAF test-bed Reached the performances and the scale needed for a Tier2 in 2008 Advanced functionalities tried (successfully) in production environment at INFN-BARI A solution to improve the scalability for a large number of concurrent accesses (splitting of central services over many machines) A solution to improve the managements of the accesses Splitting read-write access on different nodes Different queue for different type of access (priority and limit can be arranged on type-access bases) IPRD06, 1-5 October

25 dcache tests results CERN CNAF WAN CNAF LAN accesses (with ReplicaManager enabled) IPRD06, 1-5 October

26 dcache issues It is written in JAVA CPU and memory issues The configuration of the advanced features is not so easy The documentation has been improved, but still lacking The support is on best-effort bases The license is free but not completely Open-Source IPRD06, 1-5 October

27 dcache future plans Support SRMv2.2 Support VOMS (gplazma) Improve the pools management Improve the I/O queue management Improve the file replication functionalities Support Multiple PNFS server on different machines IPRD06, 1-5 October

28 STORM/GPFS IPRD06, 1-5 October

29 Storm Overview Is an Italian project (developed by INFN and EGRID) It is a lightweight SRM interface on a posix file-system It can exploit all the potentiality of every distributed file-system It provides VOMS authentication It provides ACL (permanent and on the fly) It will provide all needed SRMv2.2 functionality It is simply to install and manage It provides file access through different protocols: GsiFTP, and local file (i.e. by mounting GPFS file-system) It is possible to implement a load-balance between SRM and GsiFTP servers IPRD06, 1-5 October

30 Storm Overview Using to GPFS as back-end it is possible to: To vacate a pool (putting a machine off-line) To mirror files (and directory) for HA To associate some pool to some Storage Classes To provide an abstraction of whole disk space under a unique mount point To avoid single point of failure High performance, both in reading and writing, are yet proved in big installation (in the order of hundreds node and tens of terabyte) It is obtained by balancing both writing and reading operation between all pools IPRD06, 1-5 October

31 Storm/GPFS performances In this test the network infrastructure is limited to 3Gbit/sec Transfer rate between: 4 servers and 34 clients Time to open a file IPRD06, 1-5 October

32 STORM Issue Does not provide backward compatibility for SRMv1.1 Some high scale test are still needed in order to prove the stability and scalability Does not provide a high configurable match-making between the available SRM and GsiFTP servers Does not provide a possibility to configure a max number of connection per server or to queue the requests Monitoring and rate-measuring functionalities are not available Accounting is available only for GsiFTP access IPRD06, 1-5 October

33 Summary Functionality Feature dcache DPM Storm/GPFS FileReplication Yes No Yes PoolVacation Yes No Yes VOMS support Alpha Yes Yes ACL-VOMS Alpha Yes Yes SlotLimit Yes Partially No MatchMaking Yes Hard Coded Partially Group of pool per use type Yes No Yes VO quota Yes Yes Yes Advanced Quota Yes No Yes Services Splitting Yes Partially Yes Monitoring Features Yes No No MSS Support Yes No No Accounting Features Yes No No IPRD06, 1-5 October

34 Some good test of Performance: SC4 The graph shown the interference between local job activity and WAN transfers A long run example of trasfer Tier1->Tier2 (CNAF->BARI) IPRD06, 1-5 October

35 Acknowledgement & Links Acknowledgements: Vincenzo Spinoso, Vincenzo Vagnoni, Daniele Bonacorsi, Piergiorgio Cerello, Alessandra Doria Massimo Biasotto, Simone Badoer, Emidio Giorgio Giuseppe Lo Re, Pierpaolo Ricci, Vladimir Saputenko Jean Philippe Baud, Patrick Fuhrman Link: Wiki Storage Group: Test report and installation Guide LCG Baseline Services Group Report Storage Resource Management Working Group: Tests on Storage Managers SC4 / pilot WLCG Service Workshop IPRD06, 1-5 October

36 Conclusions The requirements coming from experiments are quite difficult to be addressed The storage system is one of the most critical part of the computing model of a farm There are several software solution at different level of maturity Each of these products is constantly evolving Each of them has positive and negative aspects The final choice may be driven by local-site needs and experience Many people are working to find out limit and solve problem IPRD06, 1-5 October

37 BACK-UP Slides IPRD06, 1-5 October

38 DPM Tests Total Rate MB/sec WN 2 WN 3 WN 4 WN 8 WN 20 WN Node Number RFIO Write RFIO Read SRM Write SRM Read LNL LAN LNL WAN IPRD06, 1-5 October

39 dcache overview (4) IPRD06, 1-5 October

40 dcache tests results (1) Transfer test done with both WAN Transfers ~50MB/sec Local I/O of 66 CMS analysis jobs ~100MB/sec analysis jobs and WAN transfer concurrently CNAF->BARI using PHEDEX IPRD06, 1-5 October

41 CNAF test Installation Schema Admin node dcache CORE PNFS Server Postgres DB Pool node gridftp door SRM door Gsi-dcap door Pool service Pool node gridftp door SRM door Gsi-dcap door Pool service Pool node gridftp door SRM door Gsi-dcap door Pool service Pool node gridftp door SRM door Gsi-dcap door Pool service IPRD06, 1-5 October

42 dcache Advanced Installation Schema DB Server Postgres DB Admin node dcache CORE Pool node gridftp door SRM door Gsi-dcap door Pool service (read) DNS Aliased Pool node gridftp door SRM door Gsi-dcap door Pool service (write) PNFS Server PNFS Server Pool node gridftp door SRM door Gsi-dcap door Pool service (xrootd) IPRD06, 1-5 October

43 SC4 / pilot WLCG Service Workshop (1/2) IPRD06, 1-5 October

44 SC4 / pilot WLCG Service Workshop (2/2) IPRD06, 1-5 October

45 DPM issues 1. Test sulle funzionalità di directories: Mkdir, Rmdir, Mv, Rm funzionavano e funzionano correttamente. Ls non funzionava e non funziona ancora (non fa il listing, ma fa "stat" della directory) 2. Test su "prepare to put": funzionava e funziona 3. Test su "prepare to get": funzionava e funziona 4. Test sul pinning: rimozione durante globus-url-copy, dopo prepare to get/put: non funzionavano entrambi, e non funzionano ancora, nel senso che durante GridFTP transfer noi ci aspettiamo che non sia possibile rimuovere il file; io riesco a rimuovere il file via SRM (Rm) 5. Test "putoversize": il test indica che è possibile effettuare una put di un file le cui dimensioni superano quelle dichiarate nella preparetoput; questo problema c'era e c'è ancora 6. Test "putoverspace": il test effettua successive operazioni di preparetoput fino a saturare lo spazio disponibile sull'srm; mentre prima non si riusciva a disallocare (AbortRequest) lo spazio allocato senza trasferire corrispondentemente il file, adesso AbortRequest funziona. E' pertanto adesso possibile disallocare tutte le preparetoput effettuate dal test. 7. NEW! Test sulla lifetime: provando ad effettuare il test di putoverspace senza effettuarne il "rollback", abbiamo aspettato il lifetime delle preparetoput, superato il quale lo spazio doveva essere reso nuovamente disponibile. Osserviamo che anche se lo spazio non viene disallocato subito, almeno deve essere disallocato in conseguenza di una richiesta di preparetoput. Lo spazio rimane purtroppo allocato, e disallocabile per mezzo di un esplicito AbortRequest. IPRD06, 1-5 October

46 Summary Functionality IPRD06, 1-5 October

47 IPRD06, 1-5 October

48 Transfer activities Tier1-Tier2 for SC4 Tier-1 to Tier-2: very bursty and driven by analysis Goal is to reach from 10MB/s (worst Tier-2s) to 100MB/s (best Tier- 2s) by June Tier-2 to Tier-1: continuous simulation transfers Goal is to reach 10MB/s from Tier-2s to Tier-1 centers (1TB per day) The PhEDEx FTS integration has been reached Two tools (Heartbeat and transfer activity) help CMS with the continuous transfer CMS distributed analysis uses CMS Remote Analysis Builder (CRAB), now interfaced to CMSSW Also trivial file catalogs work The goal is kjobs/day IPRD06, 1-5 October