AIDA Advanced European Infrastructures for Detectors at Accelerators. Presentation

Transcription

1 AIDA-SLIDE AIDA Advanced European Infrastructures for Detectors at Accelerators Presentation A scalable gigabit data acquisition system for calorimeters for linear collider Gastaldi, F (CNRS) 02 June 2014 The research leading to these results has received funding from the European Commission under the FP7 Research Infrastructures project AIDA, grant agreement no This work is part of AIDA Work Package 9: Advanced infrastructures for detector R&D. The electronic version of this AIDA Publication is available via the AIDA web site < or on the CERN Document Server at the following URL: < AIDA-SLIDE

2 Grant ANR A scalable gigabit data acquisition system for calorimeters for linear collider GASTALDI Franck On behalf of the electronic & software team Monday, June TIPP Amsterdam 1

3 Introduction: ILC detectors Method: Imaging calorimetry ~ channels/detectors Issues: Integration Power consumption Ideas: Detectors prototypes Power pulsing (1% duty cycle ~25µW/ch) allowed due to the beam structure (5 Hz spills) Switched on during > ~1ms of ILC bunch train and data acquisition Bias currents shut down between bunch trains Data acquisition and control A single cable for everything Scalable architecture Reliable protocols & simplicity Monday, June TIPP Amsterdam 2

4 Introduction: generic DAQ In most cases, detectors and associated readout systems are designed, tested and approved before DAQ effort is undertaken Our idea for this project is to design as a generic, scalable, and a self contained system, build around commercial components where possible. This DAQ is then configured towards multiple use-cases. ILC calorimetry might not be the only customer Remark : This work follows a R&D from Univ.College London, Manchester Univ and Cambridge Univ that continued at LLR-Ecole Polytechnique / IN2P3-CNRS Monday, June TIPP Amsterdam 3

5 Calorimeter DAQ: overview Machine clock DAQ2 PC Networ k card 1 Gb/s GDCC Clock & Control 50 Mb/s 7 DCC (optiona l) : DCC (optiona l) DIFs Slabs 50 Mb/s 8 50 Mb/s 50 Mb/s 8 50 Mb/s n layers GDCC 50 Mb/s Digital (Config, Control, Data) Clock & Sync Optic GigE or copper Debug USB Slabs = detector unit : detector with integrated front-end electronics and sensors DIFs: Detector InterFace, servicing the detector unit GDCC: Giga-Data-Concentrator-Card: Concentrates data, fanin/fanout for clock and control data CCC: Clock & Control card: Fanout of clock and fast controls DCC: Data concentrator Card: optionnal extra level of data concentration Monday, June TIPP Amsterdam 4

6 Calorimeter DAQ: Serial Link (cont d) HDMI connectors between DIF-DCC-GDCC-CCC - Commercial standard for consumer electronics - High-bandwith connection at low cost 3 twisted pairs + 2 optional Reference clock (50 MHz), fan-out from CCC Data in (fast control, slow-control) Data out (slow control, data readout) Monday, June TIPP Amsterdam 5

7 DAQ: The DIF card The DIF concept is generic in firmware, running on detector specific hardware Based on low cost FPGA Compact (73mm x 50 mm) Control up to 10K channels Functionalities are simple VFE chip management (power pulsing, SC, DAQ) with a common interface Local storage of SC data (Flash Ram) Architecture of the DIF FPGA Monday, June TIPP Amsterdam 6

8 DAQ: The GDCC card Format : VME 6U (chassis with only J1 connector used for power distribution) Format shared in 2 part (1/3 2/3) 1/3 is the mezzanine with the HDMI connections Reliability of mezzanine by a specific Samtec connector (SEAM and SEAF series: 160 pins) Until 28 differential signals and 19 single ended 2/3 is the GDCC heart with the main functionalities Based around a Xilinx Spartan XC6SLX75 + Marvell component USB is used to an extra access to the GDCC (debug for example) VME USB RJ45 & sfp fiber Main part Mezzanine part 7 x DIFs HDMI CCC HDMI Monday, June TIPP Amsterdam 7

9 DAQ: the GDCC card (cont d) Functionalities: Aggregate data from many DIF links and send it to the PC over Gigabit Ethernet link The PHY layers is made by a specific component MARVELL88E1111 Signaling between the DIF and the GDCC is made by 5 differential LVDS pairs in HDMI cable Extract packet from the PC and execute the command sent (R/W register, DIF configuration packet, fast command) Encapsulate data from DIF in Ethernet frame and send them to the PC E T H FPGA E R N E T mclk MARVELL component Gemac lithe Homemade (from xilinx reference design) Totally free CCC interface Main Interface (based on several FSM And few Xilinx reference design) DIFs Links (Protocol fsm ser-des 8b/10b) T O D I F trig Single architecture of GDCC card Monday, June TIPP Amsterdam 8

10 DAQ: The GDCC card (cont d) GDCC frame to the PC is based on standard Ethernet format Dst MAC Src MAC Ethernet Type GDCC_type GDCC_modifier GDCC_pktID GDCC_dataLength GDCC_Data PAD CRC32 6 Bytes 6 Bytes 2 Bytes 2 Bytes 2 Bytes 2 Bytes 2 Bytes Variable Min size Eth 4 Bytes 3 kinds of frame Fast-command with a special Ethernet type 0x809 (GDCC DIF) Control data with a special Ethernet type 0x810 (GDCC DIF) Read-out data with the Ethernet type 0x811 (DIF GDCC) GDCC Header Content of the DIF structure Data CLK DIF SOF DIF EOF Example of sending data from GDCC to DIF Data are sampled on rising-edge of clock Register packet from DIF Monday, June TIPP Amsterdam 9

11 DAQ: GDCC improvement Put in place an UDP interface Simple and fast protocol Easy to be implemented in hardware Does not require big resources Reveive RX Header_Filter Header UDP (3) Header IP (2) Header ETH (1) Flags REGs Frame_is Valid DATA_OUT GOOD dest Header src Link type Data Header RX BAD FRM Version & length Service Frame length Header IP Time of ID Flag Offset Protocole Checksum life IP src IP dest Data Header generator MEMORISE source adress IP, MAC, Port Length IP, UDP Header TX Checksum header, frame DATAs + Nb count words Port src Header UDP Port Length Checksum dest Data Packet switch Queue Buffer TX FIFO 9x1k FIFO 9x1k DATA_IN (UDP frame) Ethernet frame structure with UDP Header Send Architecture of UDP bloc DATA_IN (ETH frame) Currently under the first tests and after 3 days of sending a command to the DIF to read some registers (~ times), there is no error. Monday, June TIPP Amsterdam 10

12 DAQ: The DCC card (optional) VME format VME only used for the card power supply 1 HDMI connection for the GDCC Until 8 connections for the DIFs Identical data rate at the input and output (50 Mb/s) Advantage: This card can be connected or disconnected in DAQ chain without modification of behavior. Architecture of DCC Monday, June TIPP Amsterdam 11

13 Calicoes Software The Acquisition chain Ecal dedicated software suite Based on the Pyrame framework (LLR) Based on XML language Allow to prototype rapidly a on-line system Multi-media distribution(files, sockets and shared memories) Online event-building Acquisition chain: software architecture Monday, June TIPP Amsterdam 12

14 Calicoes Software The Control-Command Highly modular and distributed Control the Ecal electronics but also the peripheral devices (Power supply, pulse generator, ) Provides a high level state machine for final user Scripting language (Python) Good stability Global control-command architecture Monday, June TIPP Amsterdam 13

15 The system: beam test This DAQ has been used on the SiW-Ecal technical prototype for two years It has been used successfully for 4 test beams at DESY Typical setup is : (~2.5K Channels) 10 layers of detection, 10 DIFs, 2 GDCC 250 GBytes of data have been generated This system has been validated for 10 Hz of spill frequency (ILC requirement is 5 Hz) Exemple of event display 1e- (5GeV) 5 W plates between layers S/N > 14 DAQ chassis SLAB structure Monday, June TIPP Amsterdam 14

16 Conclusion The aim who was to develop a DAQ system generic in nature, using commercial components where possible has been in most part attained The tests had shown the ability of the DAQ to take a lot of data (~250 GB) During the last beam test, configurations have been injected in the system in three weeks. It remained stable during all this time. Currently, we improve our system with the implantation of a UDP block on GDCC. Next step Connect the DAQ to a real calorimeter system 16 ASICs per ASU (under test today), will be up to 160 ASICs per layer Perspectives for ECAL(ILC) With this actual configuration and for 100M channels ECAL for example the setup will be: DCC, 2000 GDCC and 200 PC For reducing the number of card, the main work must be done on front end modules for easiness of integration Monday, June TIPP Amsterdam 15

17 Monday, June TIPP Amsterdam 16

18 Back up Monday, June TIPP Amsterdam 17

19 Time line Physics prototype ILD? Technological prototype Proof of concept Feasibility of design options Construction layers 4000 channels Linearity Resolution Sensors Very front-end Compactness Granularity Front-end Power pulsing Long SLAB Integration Environement Services Industrialization Tooling Project org channels/dm channels/dm channels/dm 3 ~24 X0, 20 cm thick ~2500 m 2 active detectors ~100M readout channels S/N ~ 7.5 S/N ~ 15 Monday, June TIPP Amsterdam 18

20 Slab details ASUS with 16 Asics (180 x 180 mm) 1 Si Wafer with 256 pixels of 5X5 mm2 and thickness of 325 µm 190 mm 70 mm 180 mm Battery charger application AVX BestCap BZ01 After regulator Slab overview 360 mm Monday, June TIPP Amsterdam 19

21 DIF card Slow control and read-out Sent from the DAQ/control PC as a raw Ethernet frame Passed to/from the DIF via GDCC/DCC with the following structure (protocol) Exemple of a fast decodind command at the DIF level Internally decoded frame (test pin) IDLE SOF header data EOF DIF input:standard packet DIF output: Here: read out of 13x16b status registers (Reshaped into GDCC frame) Exemple of a decoding frame at the DIF level Monday, June TIPP Amsterdam 20

22 GDCC: some plots Trigger = start spill DIF SOF Data from DIF 0xfcff = start spill Example of data readout 0xfdff = start chip data Example of readout packet spied by wireshark RJ ~23 ps DJ ~166ps eye width ~19.75ns Example of Result of eyes diagram and jitter on data readout Monday, June TIPP Amsterdam 21

23 CCC card Supplied by University of Cambridge in 2009 Synchronize all sub-systems upon pre-spill warning Until 8 HDMI connection Distribute asynchronous fast trigger and/or busy signals Capable to run stand-alone for distribute clock (50 MHz) and spill from an external trigger Monday, June TIPP Amsterdam 22