Read-Out Driver (ROD) and Data Challenges

Transcription

1 Read-Out Driver (ROD) and Data Challenges SLUO LHC Workshop 17 July, 2009 SLAC

2 Outline The slhc Data Challenge Generic DAQ Building Blocks Current Installation RCE Training Workshop Looking Ahead Reconfigurable Cluster Element (RCE) Cluster Interconnect (CI) The ATCA Packaging Standard The RCE/CIM Test Platform ATLAS Read-Out Bandwidth Requirements for the Phase II Upgrade A Hypothetical 48-Channel Read Out Module Bandwidth Projections Summary 2

3 Upgrade Read-Out Challenge Super LHC (slhc) The ATLAS Phase-II Upgrade Implies: All-new detector front-end electronics All-new Read-Out System... Read-Out Demands are Formidable... Aims at 10x luminosity increase to 1035 cm-2s-1 Expect 400 interactions per bunch crossing (every 50 ns) All-silicon tracker will have 2x the area and 18x the data rate compared to current system LAr and Tile Calorimeter aim to read out every bunch crossing... but the Concrete Path is yet far from Clear 3

4 DAQ Upgrade R&D One R&D path towards a significantly improved DAQ readout architecture and bandwidth has started at SLAC Based on experience from other experiments Developed by Mike Huffer and colleagues Uses a modular design of three fundamental building blocks May replace and combine existing ROD+ROS functionality LSST, LCLS, PetaCache Also addresses concern with many flavors of RODs We aim to leverage on this generic high-performance DAQ research to explore future DAQ architectures for the ATLAS Phase-II upgrade 4

5 Example: the LAr Challenge ROS For each L1 accept, samples of the signal (between 3 and 32) are digitized in the FEB and transmitted to the ROD The ROD calculates energy using optimal filtering coefficients E = ai (Si pedestal) i=1..5 If E>threshold, also time and pulse quality 5 samples 0.5" 5

6 Current LAr Bandwidth Present System There are 192 ROD boards in 16 crates Each ROD is connected to 8 Front-End Boards (FEB) by optical fibers (160 MB/s S-Link) Each FEB processes 128 detector channels Input Input link of 160 MB/s allows 5 samples at 100 khz L1 rate Total ROD input bandwidth is ~ 1560 Gb/s 128 channels x 16 bits x 5 samples x 100 khz = 128 MB/s 1524 FEB x 128 MB/s per FEB = 195 GB/s Output Calculation for 5 samples is done in < 10 us, so no limitation 4 output links of 160 MB/s connect each ROD to the Read-OutSystem downstream (1 output for 2 inputs) Data needs to be reduced by a factor of 2 to sustain input rate 6

7 Upgraded LAr Front-End New LAr Front-End Board would digitize and transmit data of every bunch crossing from its 128 channels at 40 MHz 128 channels x 40 MHz x 16 bits/channel = 82 Gb/s Gb/s with 8b/10b encoding overhead Entire LAr: 1524 FEB * Gb/s > 150 Tb/s Energy calculation and L1 trigger sums would be performed at bunch crossing rate This bandwidth is the equivalent of streaming 40,000 uncompressed HDTV movies at once Analog trigger towers to be replaced with digital computation Finer granularity, added flexibility from decoupling L1 and FEB Such an approach will require (Extremely) high-bandwidth I/O, low latency Computational capability Interconnect capability 7

8 Generic DAQ Building Blocks 8

9 Three Building Block Concepts Computational Elements System-On-Chip technology Virtex 4, 5 & 6 Mechanism to Connect Together these Elements Must be low-cost $$$, footprint, power Must support variety of computational models Must have both flexible and performant I/O The Reconfigurable Cluster Element (RCE) based on: The Must be low-cost The Cluster Must provide low-latency/high bandwidth I/O Interconnect (CI) Must be based on commodity (industry) protocol based on: 10 Gb Ethernet Must support a variety of interconnect topologies switching Hierarchical, peer-to-peer, fan-in & fan-out Packaging Solution for Both Element and Interconnect Must provide high availability Must allow scaling Must support different physical I/O interfaces Preferably based on a commercial standard ATCA: Crate based Serial backplane 9

10 (Reconfigurable) Cluster Element (RCE) Combinatoric Logic reset & bootstrap options MGTs DSP tiles DX Ports DSP tiles Core Boot Options 450 MHZ PPC-405 Cross-Bar Processor Combinatoric Logic DSP tiles Memory Subsystem 512 MByte RLDRAM-II Combinatoric Logic Configuration 32 MByte Flash DX Ports Resources Combinatoric Logic MGTs DSP tiles 10

11 Software & Development Cross-Development GNU cross-development environment (C & C++) Remote (network) GDB debugger Network console Operating System Support Bootstrap loader Open-Source Real-Time Kernel (RTEMS) POSIX-compliant interfaces Standard IP network stack Exception handling support Object-Oriented Emphasis Class libraries (C++) Plug-in support Configuration interface 11

12 Resources Multi-Gigabit Transceivers (MGTs) Up to 12 channels of Each channel can operate up to 6.5 Gb/s Channels may be bound together for greater aggregate speed Combinatoric Logic SER/DES Input/output buffering Clock recovery 8b/10b encoder/decoder 64b/66b encoder/decoder Gates Flip-flops (block RAM) I/O pins DSP Support Contains up to 192 Multiple-Accumulate-Add (MAC) units 12

13 Plug-ins PGP plug-in (3.125 Gb/s) Ethernet MAC plug-in XAUI (4 x Gb/s) Boot Options 450 MHZ PPC-405 Processor Cross-Bar Memory Subsystem 512 MByte RLDRAM-II Configuration 32 MByte Flash 13 14

14 The Cluster Interconnect (CI) Q0 Even side Q1 10-GE L2 switch Management bus 10-GE L2 switch Odd side RCE Q2 Q3 Based on two Fulcrum FM224s 24-port 10-GE switch Is an ASIC (packaging in 1433-ball BGA) 10-GE XAUI interface, however, supports multiple speeds 100-BaseT, 1-GE, and 2.5 Gb/s Less than 24 Watts at full capacity Cut-through architecture (packet ingress/egress < 200 ns) Full Layer-2 functionality (VLAN, multiple spanning tree etc.) Configuration can be managed or unmanaged 14

15 RCE Board + RTM (Block Diagram) P3 RCE MFD slice0 slice1 E0 slice2 fabric slice3 E1 flash memory Fiber-optic transceivers P2 slice4 E1 slice5 base slice6 E0 slice7 MFD RCE P3 Payload RTM 15

16 RCE Board + RTM transceivers RTM Zone 1 (power) Zone 2 RCE Zone 3 Media Slice controller Media Carrier with flash (1 TB) 16

17 Cluster Interconnect Board + RTM P3 XFP XFP XFP (fabric) 1-GE 10-GE XFP Q0 CI 10-GE (base) XFP P2 base 1-GE (base) 10-GE (fabric) XFP Q2 fabric MFD 10-GE XFP Q1 Q3 XFP XFP XFP P3 Payload RTM 17

18 CIM Board and RTM RTM XFP Zone 1 10 GE switch Zone 3 1 GE CI RCE XFP 18

19 RCE Development Lab 19

20 Networking Setup Two isolated Networks One public/one private Infrastructure server and all development servers are dual-homed ATCA crates only visible on private subnet 20

21 RCE/CIM Development Platform CERN Bldg 32 Development/Test Machines RCE/CIM ATCA Crates 2x 1-GE NIC DAQ Switch 48x1GE + 2x10GE 2x Dual-3GHz Xeon GPN Switch 48x1GE Infrastructure Server for DHCP, DDNS NFS, NTP HP ProCurve 3500yl 2x 10GE X2 SR PowerEdge

22 RCE Training Workshop Took place at CERN on June 15 & 16 Tutorials, Demonstrations Discussion Towards Future Collaborations Hands-On Session 33 participants 18 people requested accounts Everyone learned to program and execute Hello World! Some learned quite a bit more than that... After the Workshop Accounts are still in use 12 RCEs can be used independently Contact me if you are interested in using one This training focused specifically on software Will hold a future workshop on FPGA firmware 22

23 Case Study: Pixel FE Calibration Current ATLAS Setup Martin Kocian's Setup Pixel FE Digital Test was ported from Pixel ROD to the RCE Modified DSP code runs on PowerPC processor Controlled by a linux host that communicates with the RCE Front-end communication through fiber at Gb/s Runs successfully... 23

24 Pixel FE Digital Test on the RCE Pixel Digital Test runs on the RCE Martin ran his setup in front of the workshop audience This is a concrete example of replacing the BOC/ROD/SBC(VME) chain with RTM/RCE/CIM(ATCA) and a Linux host The original DSP code could be ported without major changes finished running... One major complication was byteswapping between PowerPC (bigendian) and DSP (little-endian) This example focused on preserving functionality Future test cases will explore & compare bandwidth 24

25 More Information RCE Training Workshop Page Links to workshop presentations on e.g. (cross-)development cycle, RTEMS, class libraries and APIs, code examples RCE Lab TWiki Description and setup of the RCE lab infrastructure, host names, instructions, external material, e.g., ATCA manuals etc. RCE High Lumi Mailing List Discussions on Phase II upgrade studies using the RCE platform, announcements of future workshops 25

26 A Hypothetical 48-Channel Read-Out Module for ATLAS 26

27 Hypothetical 48-Channel Read-Out Module From detector FEE SNAP-12 xmt rcv gb/s x4 xmt rcv xmt rcv xmt GBT plug-in rcv Rear Transition Module P3 Cluster Elements x 12 Ethernet MAC plug-in 10-GE XAUI TTC fanout CIM switch management Synchronization clock interface Rainer Bartoldus Introduction and Overview fabric 10-GE x 4 ROM P2 ROC backplane base 1-GE x

28 Hypothetical Read-Out Crate (ROC) To L2 & Event Building To monitoring/control 10-GE x 12 From CTP Base Fabric Fabric Base Rear Transition Module Rear Transition Module P3 P3 SNAP-12 CIM TTC fanout TTC fanout switch management switch management CIM Backplane 10-GE x2 ROMs 1-GE Shelf Management SLUO To LHC Workshop, 17 July 2009 monitoring & control 28 28

29 Current (Future) RCE Bandwidth Future ROM Board Assume to be able to host 12 RCEs in 6 FPGAs RTM Back Panel (ATCA board: 322mm high) Going from XFP to Snap-12 (~15 mm) should fit 2x8x12 fibers, pairs of one up and one down, or 96 detector channels GBT at 5 GHz delivers 3.2 Gb/s usable bandwidth (at 10 GHz may hope for 6.4 Gb/s?) ATCA Zone 3 Connector 307 Gb/s (614 Gb/s) per RTM Can go up to 400x2 pairs and run 3 Gb/s per pair today (possibly 10 Gb/s in the future) so not a bottleneck RCE/ROM Input Four MGT lanes per RCE, 6.8 Gb/s today (10 Gb/s in the future) 27 Gb/s (40 Gb/s) per RCE 326 Gb/s (480 Gb/s) per ROM Matches today's GBT rates (would be limiting factor in the future) 29

30 Current (Future) RCE Output RCE Processing Power Implementation dependent, and on how much can be offloaded to DSPs or gates RCE Output Four lanes of Gb/s MGT for 10 Gb/s (10 GE) Gb/s per RCE or Gb/s per ROM might be a good guess, which matches the input bandwidth 1/4 to 1/3 of full input rate ROM Crate (ROC) Output ROM output is 2 x 10 Gb/s for each star 40 Gb/s per ROM on dual star 15 % of L1 rate for silicon tracker with in:out = 1:1 full L1 rate for calorimeters Each star supports up to 6 boards 240 Gb/s per ROC 30

31 Future DAQ Plant on a Napkin Balance of Real Estate and Throughput For silicon tracker may not push GBT beyond 3.2 Gb/s and RTM to full density but rather add more ROMs to scale to more than 15% L1 accepts for L2 For calorimeter, can minimize plant size while still easily serving full rate of L1 For muon system, might need to scale for computational power LAr ROMs need only Gb/s per FEB for 5 samples at 100 khz Input 1524 FEBs of 102 Gb/s each; at 6.4 Gb/s GBT this is 16 fibers per FEB or 6 FEBs per 96-channel ROM or 254 ROMs Pixel ROMs slhc Pixel has 18x data rate of current detector with ~800 fibers of 3.2 Gb/s With 48 fibers per ROM (half) one has an 18 ROMs system that could output 20% of the L1 accepts Compare that to current 134 ROD plus 12 ROS system 31

32 Summary We have started to explore a new generation DAQ platform Strategy is based on the idea of modular building blocks Architecture is now relatively mature Inexpensive computational element (the RCE) Interconnect mechanism (the CI) Industry standard packaging (ATCA) Both types of demo boards (and corresponding RTMs) functional RTEMs ported and operating Network stack fully tested and functional Performance and scaling meet expectations Documentation is a work-in-progress This technology strongly leverages off industry innovation System-On-Chip High speed serial transmission Low-cost, small footprint, high-speed switching (10 GE) Packaging standardization (serial backplanes and RTM) 32

33 Conclusions Gained experience with these innovations will itself be valuable This technology offers a ready-today vehicle to explore both alternate architectures and different performance regimes Test platforms exist at SLAC and now also at CERN with RCE/CIM building blocks mounted on prototype ATCA boards Workshop material is available that gives a tutorial of the platform with ready-to-use examples As an example, the Pixel FE test was successfully run on the RCE Back of the envelope calculations indicate that a system made of these building blocks could meet the ATLAS phase II upgrade requirements Teststands are available at SLAC and at CERN and everybody is welcome to explore! 33

34 Extra Material 34

35 Gigabit Glossary XAUI (10 Gigabit Attachment Unit Interface) - standard for extending the physical separation possible in connecting full duplex 10 GE ports on a printed circuit board XFP (10 Gigabit Small Form Factor Pluggable) hot-swappable, protocol-independent optical transceiver 10 GE 10 Gigabit (per second) Ethernet IEEE standard MGT Multi-Gigabit Transceivers GBT (Gigabit Bi-Directional Trigger and Data Link) radiation hard optical link, high-bandwidth & error correction SNAP Channel snap-on pluggable parallel optical transceiver module 35