Using Pulsar as an upgrade for L2 decision crate Ted Liu, FNAL (for CDF Pulsar group)

Transcription

1 Using Pulsar as an upgrade for 2 decision crate ed iu, FNA (for CDF Pulsar group) For more information about Pulsar board:

2 Back to Basic: What does Global 2 really do? racks Jets make decisions based on objects already found upstream 1 trk svt clist Iso reces mu umet,met electron photon muon aus Met umet physics object examples Combines/matches trigger objects into e, muon, jets Count physics objects above thresholds, or, Cut on kinematics quantities echnical requirement: need a FA way to collect many data inputs Global 2 2

3 Current 2 Global Crate: echnical requirement: need a FA way to collect many data inputs With the technology available back then, had to design custom backplane (magicbus) and processor data is either pushed(via DMA) or pulled (via Programed IO) to ease the bandwidth demands In addition, one has to deal with the fact that each data input was implemented in a different way 6 different type of custom interface boards Magic bus 1 eces x 4 a(s) X P V Only one MB crate C I I O M U O N largest data size Global 2 3

4 Basic idea on possible new approach using modern technology: Convert/merge different data inputs into a standard link interface with commodity PC(s) high bandwidth, commercially available link to PCI interface cards: CEN -INK is designed just for this purpose for HC and other experiments. eal question: can we design an universal interface board s s (and the rest are all commercial products)? Concept Pulsar is designed to be able to do more than this. ss s s s PCI link to PCI mem CPU 4

5 INK format example: INK interface mezzanine card AA INK data format 5

6 Pulsar design P1 9U VME (VME and CDF ctrl signals are visible to all three FPGAs) K 3 Altera APEX 20K400_652 (BGA) FPGAs 502 user IO pins each Data IO Mezz card connectors P2 INK signal lines Control/ Merger AM 128K x 36 bits P3 spare lines 1 K Data IO AM 3 APEX20K400 FPGAs on board = 3 Million system gates/80kb AM per board K x 36 pipelined AMs with No Bus atency: 1 MB AM (~5ns access time)

7 Front-panel (double width) VD connectors Mezzanine cards 1 1 Mezzanine Card Hotlink/ axi/ -INK PUA design Each mezzanine card can have up to 4 (hotlink/axi) fiber channels IO VME Ctrl optional user defined signal connection from P2/P3 Optional V input V V IO INK spares -INK x or x -INK x or x hree FPGAs: Atlera APEX20K400 component side he mezzanine card connectors can be used either for user I/O or INK cards o/from Pulsar or 7 a PC

8 Pulsar Design methodology A major fraction of the design effort was dedicated to core firmware development and extensive design verifications by using state-of-the-art CAD tools: eonardo pectrum for VHD synthesis; Quartus II for place and route, and FPGA level simulation; developed core firmware (VHD) first to guide and optimize the board design Quickim for board and multi-board level simulation; used to carefully verify board schematics Interconnect ynthesis tool for trace and cross talk analysis; I_MultiBoard tool for signal integrity checks between motherboard and mezzanine cards; used to carefully verify board layout All details can be found at: 8

9 Performed multi-board simulation to verify the design 9

10 Board design work started Nov Pulsar prototypes received: ept 19 th, Initial VME access to all three FPGAs are successful: ept. 21th,

11 11

12 Hotlink mezzanine card prototypes have been tested in July, Hotlink x mezzanine card prototype (for muon and cluster data paths) Hotlink x mezzanine card prototype 12

13 Pulsar with AUX card INK Mezzanine card 13

14 Pulsar approach: Goal: only build one type of custom interface board and the rest are all commercial products. Use mezzanine cards to take care optical data paths (Cluster,Isolation, Muon and eces) P1 P2 INK P3 Pulsar design I Control/ Merger 1 K K Optical IO Optical IO Mezz card connectors Extra features: 1 trigger bits and trk/svt input are visible to all three FPGAs 14 for flexibility

15 racks Jets electron photon muon aus Met umet 1 trk svt clist Iso reces mu umet,met P1 What does evel 2 really do? Combine/matches trigger objects into e,muon Count objects above thresholds, or, Cut on kinematics quantities K Data IO Mezz card connectors Most trigger objects need 1 and track/svt trigger information, this is reflected in Pulsar design: for flexibility INK I/O P2 P3 user ctrl spare lines Control/ Merger 1 K Data IO 15

16 One possible simple configuration: link to PCI CPU PCI to link or use V_PCI_ Global Processor Controller 1 2 decision V eces/trk 160MB/s INK INK Cluster/trk INK Muon/trk ume,me (can plug into one of other boards) All 8 motherboards are physically identical 4 eces eces Pre-processor Pre-processor + 1 merger X 3 Cluster Pre-processor Muon Pre-processor eces: 12 x4 =48 taxi fibers Cluster: 6 hotlink fibers Iso: 7 taxi fibers Muon: 16 hotlink fibers XP 16 1 bits

17 Possible New 2 Decision Crate O C A C E All 8 motherboards are physically identical M U O N C U E E C E Muon/trk E C E E C E M E G E Cluster/trk eces/trk umet,met Each Pulsar board take two slots (due to mezzanine cards) otal: 8 pulsar boards = 16 slots Baseline design: use pre-processor Pulsars to simply suppress/organize data, use Processor Controller Pulsar to simply pass data to CPU via link to PCI and also handshake with. All trigger algorithm will be handled by CPU. E C E 160MB/s P O C C 160MB/s link to PCI PCI to link/v/ GHz PC or mem CPU 17 VME processor

18 New 32-bit INK to 64 bit PCI interface card: 32PCI64 highly autonomous data reception 32-bit INK, 64-bit PCI bus 33MHz and 66 MHz PCI clock speed up to 520MByte/s raw bandwidth CEN -INK to to PCI-64 interface I II N I N K N K K to 32 to to 64 to 64 map 64 map INPU INPU BUFFE BUFFE FIFO FIFO (1024 (1024 x x 64-bit) x 64-bit) PCI PCI BU PCI BU FIFO FIFO 128x 128x 64-bit 128x 64-bit DMA DMA ENGINE ENGINE High-speed follow up of the imple INK to PCI interface card BACKEND BACKEND CONO CONO OGIC OGIC EQUE EQUE FIFO FIFO (address, (address, length) length) ACKNOWEDGE ACKNOWEDGE FIFO FIFO (ctl FIFO (ctl words, (ctl words, length) words, length) length) CONO, CONO, AU AU & & INEUP & INEUP EGIE EGIE 64-BI 64-BI PCI PCI PCI COE COE 18 33/66 33/66 MHz 33/66 MHz 32/64-bit MH 32/64-bit PCI PCI PCI Erik Erik van van der der Bij Bij division division EP EP AE/DQ AE/DQ

19 New 32-bit INK to 64 bit PCI interface card: 32PCI64 highly autonomous data reception 32-bit INK, 64-bit PCI bus 33MHz and 66 MHz PCI clock speed up to 520MByte/s raw bandwidth High-speed follow up of the imple INK to PCI interface card 19

20 2 decision from CPU: how to handshake with rigger upervisor? here could be a few ways to achieve this: (1) could use PCI to INK card: send a INK message back to Pulsar Processor Controller, then Pulsar handshakes with ; (2) Or use V_PCI_ daughter card built by Pisa group. V in V out I link to PCI PCI to link/v/ mem CPU 20

21 PCI daughter card: V-PCI- which can be plugged into Altera PCI board he time to send a decision from CPU to Pulsar/ can be short: ~ 1us measured value 21

22 Baseline Design (with the simple configuration) could be: PreProcessor/Interface versions of the board gather data from each subsystem and package the data. his can include sparsification based on 1 trigger bits, tracking, or the data itself. Processor controller/merger version of the board merges data from interface boards and packages the data for transfer to a CPU. he data can be further sparsified at this stage. his board also provide the interface between 2 and the rigger upervisor. Both types of PreProcesor board can be used to readout data he default operation would have the final decisions made in code in the CPU. Pulsar is designed to be able to do more if necessary. Designed to be quite flexible board level and system level Pulsar flexibility 22

23 P1 P2 P3 Control/ Merger Pulsar as Muon Pre-processor (interface board) Can create physics objects (i.e., muons) at this stage (optional) INK signals spare lines A muon data available with rack info: muons 1 K Data IO Info available: 1, XP, Muon matchbox data (0-240 degree) DataIO Info available: 1, XP, Muon matchbox, Pre-matchbox AM AM data from Matchbox (16 hotlink fibers) degree degree degree data from pre-match box (4 hotlink fibers) Board level flexibility 23

24 Pulsar as eces data PreProcessor (merger) P1 P2 P3 Could use AM as U to match XP and eces info to create electrons (optional) Control/ Merger INK signal lines spare lines A Info for XP and eces available: electrons 1 K Data IO Info available: 1, XP, eces electrons DataIO Info available: 1, XP, eces electrons Assume 4 Pulsars upstream AM AM East wegdes 1-12 West wedges 1-12 East wedges West wedges Board level flexibility 24

25 P1 P2 P3 Control/ Merger Pulsar as Processor Controller Pulsar design is driven by physics needs possible to combine trigger objects here (optional) INK signal lines spare lines A Info available for 2 decisions, 1 V Data IO Info available: 1, V, XP, eces, Cluster, Iso racks, Jets, e, photon,tau etc DataIO Info available: 1, V,XP, Muon,Met, umet racks, muon, Met,umEt etc AM AM eces/trk Cluster/Iso Muon/trk Met/umEt Board evel Flexibility 25

26 One possible configuration one PC use two PCI slots to receive INK data: One receives normal trigger data, the other is dedicated for V data (arrives late). CPU can start working on the early arrival data first -INK Pulsar Custom Mezzanine cards Pulsar -INK -INK Pulsar -INK Normal data V data V Pulsar 2 decision link to PCI link to PCI PCI to link/v/ ystem level flexibility mem CPU Pulsar 26

27 Another possible configuration using four PCs: Each PC(i) gets its own event(say buffer i), each works on a different event. ay PC 0 is working on buffer 0, PC 2 is working on buffer 1 etc. his way one long tail event would not prevent other three events to be processed. -INK Pulsar Custom Mezzanine cards -INK Pulsar -INK Buffer 0,2 events -INK Pulsar PC 0 PC 2 Pulsar Pulsar -INK PC 1 PC 3 -INK Buffer 1,3 events Pulsar ystem level flexibility 27

28 Current firmware status: we have written firmware to record data into a PC for the teststand. Work applicable to upgrade needs. Pulsar can convert any user data (via custom mezzanine cards) into INK format then transfer the data into a PC K Optical IO Mezzanine card connectors CDF ctrl Merger data PC INK to PCI INK 1 K Optical IO ctrl Pulsar in (general purpose) recorder mode (directly into a PC) 28

29 1A Buffer# Firmware for Pulsar in recorder mode(into a PC): 1A(x4) queue 1A egister 1 trigger bits mezzanine cards inputs FIFO FIFO 32-bit FIFO 32-bit 1 egister (pulls one event worth of data at a time) Output FIFO *Checks data consistence merges and stamp data INK Formatter 32-bit his formatter is based on existing code from CEN tate machine 29

30 Firmware design is similar in all three FPGA s on all versions of the Pulsar DataIO FPGA 1A 1 Control/merger FPGA DataIO FPGA P3 Data package (per 1A) in INK format sent 30

31 In ummary Pulsar design is powerful, modular and universal Distributed processing motivated by physics ufficient safety margin in raw bandwidth and flexibility at both board- and system-level to handle unexpected un II B challenges Can be used as a test-stand as well as an upgrade path uitable to develop and tune an upgrade in stand-alone mode educes impact on running experiment during commissioning phase Pulsar is useful for un II A maintenance and any 2 upgrade Built-in maintenance capability ystem relies heavily on commercial resources Only one custom board Firmware design is similar in all FPGA s on all incarnations of the Pulsar Easier long-term maintenance he rest is commercially available Easily upgradeable CPU, PCI boards Widely used link well documented HC standard Knowledge transferable to and from HC community 31