Entry Stage for the CBM First-level Event Selector

Transcription

1 Entry Stage for the CBM First-level Event Selector Dirk Hutter for the CBM Collaboration FIAS Frankfurt Institute for Advanced Studies Goethe-Universität Frankfurt am Main, Germany CBM CHEP 2018 Conference in Sofia, Bulgaria

2 The CBM Experiment at FAIR FAIR CBM (Electron-Hadron Setup) RICH TRD STS Fixed target heavy ion experiment at FAIR Physics goal: exploration of the QCD phase diagram Extreme reaction rates of up to 10 MHz and track densities up to 1000 tracks in aperture Conventional trigger architecture not feasible Full online event reconstruction needed Self-triggering free-streaming readout electronics Darmstadt, Germany MVD Dipole magnet MUCH TOF PSD ECAL Event selection exclusively done in FLES HPC cluster!2

3 First-level Event Selector (FLES) FLES is designed as an HPC cluster Commodity PC hardware First phase: full input connectivity, but limited processing and networking Green IT Cube FPGA-based custom PCIe input interface Total input data rate > 1 TB/s e c n a t s i d h r t a g e n n e i l l e l m b 0 a 5 c 3 ~ m 0 ~70 Timeslice building via InfiniBand network Combines the data from all input links to self-contained overlapping processing intervals and distributes them to compute nodes CBM Service Building RDMA data transfer very convenient for timeslice building Located in the Green IT Cube data center Cost-efficient infrastructure sharing Maximum CBM online computing power only needed in a fraction of time combine and share computing resources Consequences Transmit 10 TBit/s over 700 m distance Needs single-mode optics Increased link latency!3

4 Initial CBM DAQ/FLES Architecture on detector counting room green cube FLES FEE GBT DPB custom opt. Link FLIB PCIe entry InfiniBand ~700m node FLES proc. node single FLES cluster Single flat cluster design Two FPGA-based stages Data Processing Board (DPB): aggregation and preprocessing FLES Interface Board (FLIB): data preparation and interface to standard COTS hardware Long-range connection to Green Cube via custom optical links Revisited design in context of CBM modularized start version and usage of GBTx frontends Consider future upgradability Maximize use of standard hardware!4

5 Optimized DAQ/FLES Architecture on detector counting room green cube FLES FEE GBT DPB custom opt. Link FLIB PCIe entry InfiniBand ~700m node FLES proc. node Interface to standard hardware via PCIe single FLES cluster Long-range InfiniBand FEE GBT CRI PCIe FLES entry node long range InfiniBand ~700m entry cluster Small specialized cluster Combine DPB and FLIB to single FPGA board, similar to ATLAS, LHCb and ALICE Split computing into 2 dedicated clusters Only small entry cluster holds custom hardware Relaxed requirements on processing cluster design FLES proc. node compute cluster Long-range connection to Green Cube via standard network equipment (e.g., long-range InfiniBand)!5

6 FLES Entry Stage Design CRI prototype candidates: Input from detector systems 4800 GBT GBit/s (21,5 TBit/s total bandwith) < 50% link occupancy 10 Tbit/s peak 100 CRI with 48 links and PCIe 4.0 x8 (3.0 x16) 100 GBit/s card Output to processing cluster FLX-712 No significant data reduction in entry cluster possible Averaging in spill intensity variations in entry stage can smooth data rate HTG-Z920 Averaging machine duty cycle foreseen in processing cluster Connectivity to ~ 600 processing nodes Absolut worst case 10 Tbit/s peak Current entry node prototype: ASUS ESC8000 Possible entry node configuration 2 CRI + 2 HCAs per node feasible 50 Gbit/s or 100

7 Reminder: InfiniBand Networks High-throughput, low latency switched fabric network Widely used in HPC applications RDMA semantics perfectly suitable for FLES timeslice building Lossless fabric, i.e. packets are not routinely dropped Credit based flow control on link-level Each (logical) link supplies credit to the sending device denoting the available buffer space Quality of Service features Virtual lanes (VL) are separate logical communication links that share a single physical link Up to 15 virtual lanes (VL) and one management lane per link fat-tree Clos network THE SOLUTION: MELLANOX SOCKET DIRECT Mellanox Socket Direct is a unique form-fact as two PCIe cards, wherein the PCIe lanes are A key benefit that this adapter card brings to m eliminating the network traffic traversing over sockets, significantly reducing overhead and la Figure 2 shows a photograph of the Socket Dir effectively integrates a single network adapter together with an auxiliary board and a harness Next generation: HDR (200 Gbit/s) 40 port switches, e.g. QM87xx series Allows port splitting HDR100, 80 ports per switch Figure 2. Socket-Direct Adapter (Front!7

8 Long-haul connections Challenge: bridge 700m between entry and processing cluster Buffer = Bandwidth Time round trip Physical communication Typical intra data center techniques reach m Use medium or long range single-mode optics, e.g. CWDM4 up to 2km Protocol All reliable communication protocols suffer from high bandwidth-delay products Higher delays account for more buffers or lower bandwidth Mellanox InfiniBand switches allow to switch off VLs Collapses buffers form unused VLs into larger buffer Idea endorsed by Mellanox To be tested in real lab setup Bandwidth (Gbit/s) Theory: 100 GBit/s link, 0,75 µs allowed delay 1 x RTT (µs)!8

9 Long-haul Connection Test Stand QSFP Transceiver Box 8-fach ribbon 400m 8-fach ribbon 800m Testing requires connection with given delay Box with 8x 400m + 8x 800m single mode fiber G.657.A1 bend-insensitive fiber to allow compact size Each box is sufficient for: 1 PSM4 connection 4 LR4 / CWDM4 connections using MPO-LC breakouts Allows measurement in 400m steps e.g. 400m, 800m, 1200m until patching exceeds loss budget, ~3,5 db First lab tests with InfiniBand EDR very promising!9

10 Possible FLES Network 18 (2) 18 (2) used switch ports p hs num. of cables 18 m ) e n c a e - 10,8 Tbit/s inter cluster - 1,2 Tbit/s local - Up to 120 Gb/s - Up to 612 Gb/s ~1/6 blk switches, cables fibers < 600m OS2 local spine (6 fro i 10,8T 10,8T total bandwidth ,8T+1,2T 12T 40x HDR100 Fat-tree like network 6 34 x9 10,8T 61,2T 68x HDR EN nodes HDR PN nodes HDR100 Sub-clusters share a common spine Compute cluster can have 1/6 blocking ratio Timeslices are built across the long range connection Local spine for minimal connectivity w/o compute cluster!10

11 Network with CBM-local Timeslice Building could be routers T 20T fibers 5-20 Tbit/s local (shared) - 12 Tbis/s inter cluster - Up to Gb/s in switches, cables < 600m OS2 20T 20T x HDR 100 EN nodes HDR Build timeslices to EN memory send fully built timeslices to GC Additionally needed network resources feasible with HDR Timeslice building decoupled from GC network More control over critical building process, no long link delay Independent of GC network architecture, flexible against changes Better scaling to more PNs Infiniband routers additionally allow to separate subnet control!11

12 Thanks for your attention Dirk Hutter CBM!12

13 Local TS Building + Application Level Routing 30 (10) 120 EN nodes 2x HDR T 12T x HDR100 - Application level routing - 12 Tbit/s local - 12 Tbis/s inter cluster - Up to Gb/s in switches, cables 12T 12T T 12T fibers < 600m OS2 Use separate HCA (port) instead of TS building network to send timeslices to GC Full separation between networks Mix of different network technologies possible Less flexible in terms of link balancing to GC!13

14 FLES Data Management Framework RDMA-based timeslice building (flesnet) Works in close conjunction with FLIB hardware design Paradigms: Do not copy data in memory Maximize throughput Based on microslices, configurable overlap Delivers fully built timeslice to reconstruction code 10 GBit/s custom optical link FEE FLIM Direct DMA to InfiniBand send buffers Shared memory interface FLIB Server Device Driver FLIB SHM HCA Prototype implementation available C++, Boost, IB verbs Measured flesnet timeslice building (8+8 nodes, including ring buffer synchronization, overlapping timeslices): ~5 GByte/s throughput per node Timeslice building InfiniBand RDMA, true zero-copy TS- Building IN IB Verbs TS- Building CN IB Verbs HCA SHM Indexed access to timeslice data Reco/ Ana Prototype software successfully used in several CBM beam tests!14

15 RDMA Timeslice Building Dual Ring Buffer in Input Node Dual Ring Buffer in Compute Node (per Input Link) TS 1 TS 0 TS 0 TS 1000 FLIB DATA CONTENT MC 0 CONTENT MC 1 CONTENT MC 99 CONTENT MC 100 CONTENT MC Example: 2 GB ~200k microslices ~2k timeslices DATA DESC MC 0 DESC MC 1... DESC MC 99 DESC MC 100 CONTENT MC 0 CONTENT MC 1... CONTENT MC 99 CONTENT MC 100 DESC MC DESC MC Example: 10 MB ~1k microslices ~10 timeslices Analysis ADDR DESC MC 0 DESC MC 1 DESC MC 99 DESC MC 100 DESC MC Example: 8 MB ~1M microslices ~10k timeslices DESC DESC TS 0 DESC TS Example: 1 kb ~80 timeslices Two pairs of ring buffers for each input link Second buffer: index table to variable-sized data in first buffer Copy contiguous block of microslices via RDMA (exception: borders) Lazy update of buffer status between nodes, reduce transaction rate!15

16 FLES Input Data Path Dual Ring Buffer in Shared Memory... CONTENT MC 1 FLES Input Module HDR MC 1 CONTENT MC 0 HDR MC 0 MC handling DESC MC 1 DESC MC 0 CONTENT MC 1 CONTENT MC 0 DMA channel... per channel DMA Mux PCIe DATA ADDR CONTENT MC 0 DESC MC 0 DESC MC 1 CONTENT MC 1 TS 0 DESC MC 99 CONTENT MC 99 CONTENT MC 100 CONTENT MC DESC MC 100 DESC MC TS 1 Example: 8 MB ~1M microslices ~10k timeslices Example: 2 GB ~200k microslices ~2k timeslices Microslice Timeslice substructure Constant in experiment time Allow overlapping timeslices Full offload DMA engine Transmit microslices via PCIe/DMA directly to userspace buffers Buffer placed in Posix shared memory, can be registered in parallel for InfiniBand RDMA Pair of ring buffers for each link Data buffer for microslice data content Descriptor buffer for index table and microslice meta data!16

17 Interface to Online Reconstruction Code Timeslice Component 0 Component 1 Time DESC MC 0 CONTENT MC 0 DESC MC 0 CONTENT MC 0 DESC MC 1 DESC MC 1 CONTENT MC 1 CONTENT MC DESC MC 99 DESC MC 99 CONTENT MC 99 CONTENT MC 99 DESC MC 100 DESC MC 100 CONTENT MC 100 CONTENT MC 100 DESC MC 101 CONTENT MC 101 DESC MC 102 CONTENT MC 102 Basic idea: For each timeslice, an instance of the reconstruction code......is given direct indexed access to all corresponding data...uses detector-specific code to understand the contents of the microslices...applies adjustments (fine calibration) to detector time stamps if necessary...finds, reconstructs and analyzes the contained events... Core region Overlap region Timeslice data management concept Timeslice Two-dimensional indexed access to microslices Overlap according to detector time precision Interface to online reconstruction software Timeslice is self-contained Calibration and configuration data distributed to all nodes No network communication required during reconstruction and analysis!17