Jörn Adamczewski-Musch. GSI / EEL 5 December 2017

Transcription

1 Jörn Adamczewski-Musch GSI / EEL 5 December 2017

2 DAQ storage for HADES (dabc plug-in) LTSM gateway for MBS systems Summary 5th December 2017 Jörn Adamczewski-Musch 2

3 /data0i RFIO /lustre storage interface QA monitoring HYDRA2 EPICS run sync, GUI 5th December 2017 Jörn Adamczewski-Musch 3

4 LTSM /lustre /data0i storage interface QA monitoring HYDRA2 EPICS run sync, GUI 5th December 2017 Jörn Adamczewski-Musch 4

5 Data Acquisition Backbone Core (://dabc.gsi.de) Linux, C++, multithreading, modular, extensable plug-ins, multi-node DAQ LTSM plugin : Implements dabc::fileinterface with fopen, fwrite, etc. methods (POSIX like) Wraps and uses LTSM library by T.Stibor and tsm client by IBM May write any file format (HADES., MBS.lmd,...) generally configurable in dabc application by xml files for HADES configurable by existing DAQ/EB scripts and setup files 5th December 2017 Jörn Adamczewski-Musch 5

6 mbs> con rfio lxg0546:2345 disk X86L-10:transport :Connected RFIO server! mbs> open file /data.local1/myaccount/test_files_ first=10 size=1000 -auto -rfiofile test.lmd rfio client write data rawdisprfio64 rfio dispach fork rawservrfio64 rfio server local harddisk widely used at standalone mbs traveling systems with local disk servers for several years. 5th December 2017 Jörn Adamczewski-Musch 6

7 rawdispltsm64 mbs> con rfio x86l-59:2345 ltsm X86L-10:transport :Connected RFIO server! mbs> open file /lustre/hebe/mbs first=10 size=1000 -auto rfiofile test.lmd rfio client write data rfio dispach fork rawservltsm64 rfio server ltsm API new development for MBS v6.3 benefit: MBS host does not need TSM client library (<- not available for LynxOS, PPC Linux etc.) TSM server ltsm01_test lustre ltsmsync + copytool tape 5th December 2017 Jörn Adamczewski-Musch 7

8 Wrote data in parallel with HADES/dabc in September 2017: (12 files x 3 MB/s), (8 files x 5 MB/s) from HADES DAQ pulser trigger (1 x 75 MB/s), (2 x 40 MB/s) (4 x 18 MB/s) self-generated data 10 GbE at HADES servers, but 1 TSM server only! MBS with via RFIO/LTSM gateway (both on x86l-59) in August 2017: single file 40 MB/s at 5.5 khz (without file: 70MB/s at 7 khz) 1 GbE at MBS node, single TSM server 5th December 2017 Jörn Adamczewski-Musch 8

9 Data writing with LTSM has been implemented for GSI major DAQ systems (MBS, HADES/dabc) Data access from batch farm possible via transparent /lustre file Data reading/copying possible via ltsmc (or easy wrapper script?) To do: HADES needs 4 TSM servers at 10 GbE, optimization for 2018 beamtime Test of MBS LTSM interface software rawdispltsm64 / rawservltsm64 Deployment and test of LTSM gateways for MBS? dedicated gateway machine for each experiment?? common gateway servers provided by GSI IT department? 5th December 2017 Jörn Adamczewski-Musch 9

10 5th December 2017 Jörn Adamczewski-Musch 10

11 The old gstore/rfio system will no longer be supported by GSI Writing to RFIO from HADES DAQ has failed already since 2016 New System: Ligtweight Tivoli Storage Manager (LTSM) Development by T.Stibor, GSI-IT (s:/github.com/tstibor/ltsm) Write to tape archive via sockets, like RFIO (but use IBM client directly) Transparent synchronization between tape and lustre file system archive file name is lustre file path, file is retrieved on first lustre read access plug-in for LTSM has been developed in 2017 Successfully wrote data with HADES EB in September 2017: (12 files x 3 MB/s), (8 files x 5 MB/s) from HADES DAQ pulser trigger (1 x 75 MB/s), (2 x 40 MB/s) (4 x 18 MB/s) self-generated data, 1 TSM server only! 5th December 2017 Jörn Adamczewski-Musch 11

12 Data Acquisition Backbone Core (://dabc.gsi.de) Linux, C++, multithreading, modular, extensable plug-ins, multi-node DAQ hadaq plugin : UDP subevent receiver, event builder functionality file to RFIO and disk EPICS interface via intermediate shared memory writes same run info for ORACLE logbook configurable by existing DAQ/EB scripts and setup files QA monitoring events provided by server socket HTTP server for EB monitoring and control May optionally run pre-analysis processing (TDC calibration etc) 5th December 2017 Jörn Adamczewski-Musch 12

13 performance (August 2014) 29 subsystems, 25 khz pulser, 3 EB, RFIO and local disk file => 38 Mbyte/s per EB process, max CPU load 78%, no event loss => would achieve 608 Mbyte/s with all 16 EB processes (?) typical pion beam data rate 2014: 9 Mbyte/s at <2kHz trigger rate) netmem/evtbuild performance (August 2011) 28 subsystems, Au-beam trigger, 8 EB, RFIO and local disk file => typical 17 Mbyte/s per EB process without event loss => overall 135 Mbyte/s 5th December 2017 Jörn Adamczewski-Musch 13