Cerntech founded in 1997

Transcription

1 20 20 Cerntech founded in 1997 Mérnökök és s fizikusok együttm ttműködése a nagyenergiás s fizikában Kiss Tivadar Mérnökök és fizikusok együttműködése a nagyenergiás fizikában 1

2 Introduction Cerntech Engineering, Ltd. is a Budapest based spin-off company founded in 1997 by engineers and physicists of KFKI-RMKI and TU Budapest Since then we closely co-operate with academic institutes and industry Experience in designing of reliable, high-performance data transmission systems for the scientific market serial optical communication design from system to board level turning prototypes into commercial products Work with several subcontractors, especially in production (e.g. PCB manuf., assembly) More than high-speed optical links and other interface card units have been sold to CERN, its collaborators, and other scientific (mainly High Energy Physics, HEP) institutes Developments based on single or multiple of 2-10 Gb/s optical links and FPGAs Read-out cards, data concentrators for HEP Intelligent digital camera electronics for video diagnostics in fusion experiments Mérnökök és fizikusok együttműködése a nagyenergiás fizikában 2

3 The First Success Story: the ALICE DDL Project, Developing and production of Detector Data Link (DDL) subsystem for the data acquisition of the ALICE detector of LHC at CERN project started at 1996, completed in January 2007 major Hungarian engineering contribution to ALICE Collaboration between CERN, KFKI-RMKI (now Wigner RCP), and Cerntech Ltd. supported partly by the Hungarian state funding agency (OMFB, NKTH) through Cerntech; and CERN Conceptual and logic design, hardware, software development close work relation with collagues at CERN and KFKI-RMKI Successful tendering Production and testing in Hungary, delivery to CERN Full responsibility for the ALICE read-out links development, production, maintenance, and support (for years of ALICE) /19/2016 Mérnökök és fizikusok együttműködése a nagyenergiás fizikában 3 3

4 DDL in ALICE DAQ ALICE sub-detectors Sub-Detectors TPC Sub-Detector Readout Electronics Readout Electronics 487 DDL optical links Source Interface Units (SIU cards) Duplex, multimode optical fibers max. 200m Destination Interface Units (integrated on D-RORCs) DIU SIU DIU SIU 144 DDL DIU SIU DIU SIU 25 GB/s total SIU DIU DIU 216 DDL DIU SIU DIU 127 DDL DIU DIU H-RORC H-RORC FEP FEP SIU HLT Farm 10 DDLs SIU DIU DIU High- Level Trigger DAQ- Readout Receiver Card D-RORC D-RORC D-RORC D-RORC D-RORC D-RORC Local Data Concentrators (server computers) LDC LDC LDC LDC LDC LDC (Gigabit Ethernet and FibreChannel switches) Event Building Network Global Data Collectors (server computers) GDC GDC GDC (disk farms, tapes) Storage Network /19/2016 Mérnökök és fizikusok együttműködése a nagyenergiás fizikában 4 4

5 The Surce Interface Unit (SIU) of a Gb/s (200 MB/s) DDL link Duplex LC optical connector 32-bit parallel interface /19/2016 Mérnökök és fizikusok együttműködése a nagyenergiás fizikában 5 5

6 DDL Radiation Tolerance DDL SIU cards and fibers are exposed to irradiation in the detector Up to 5 krad TID, and 4 x neutron x cm -2 (1 MeV n equiv.) Several irradiation tests were carried out to check components and full SIU cards. Tests were carried out at: ATOMKI, Debrecen, Hungary (TID measurements (gamma irradiation), and neutron irradiation (5-15 MeV) Teo Swedberg Laboratory, Uppsala, Sweden (proton irradiation, MeV) We found two problems to be solved Configuration loss in SRAM based FPGA devices (e.g. ALTERA or XILINX FPGAs) permanent error, card configuration needs to be reloaded Transmission bit errors caused by single event upsets (SEU) less serious problem, however errors must be detected! Design modifications were implemented Changing to flash memory based FPGA (e.g. ACTEL ProASIC) from SRAM based ones (ALTERA) Additional parity-check on internal FIFO memories, and more robust control logic to detect/handle SEU errors /19/2016 Mérnökök és fizikusok együttműködése a nagyenergiás fizikában 6 6

7 100 MHz, 64-bit PCI-X X Read-out Receiver Card (RORC) for the Gb/s (200MB/S) bidirectional DDL link duplex LC optical connector SFP optical transceiver (pluggable) 100 MHz, 64-bit PCI-X interface /19/2016 Mérnökök és fizikusok együttműködése a nagyenergiás fizikában 7 7

8 DDL in other DAQ Systems STAR detector upgrade at BNL (Brookhaven) The new DAQ is based on 160 DDL links produced by Cerntech SHINE experiment (NA61) at CERN Based on the upgrade of the former NA49 TPC detector front-end cards have to be read-out with a 10x speed! A new read-out and DAQ system were designed by KFKI-RMKI and Cerntech New Read-out Motherboards (250 pcs) and Data Concentrators (10 pcs) had to be designed and produced (Cerntech) Data is highly concentrated and sent to DAQ via 10 DDL links Now all three major TPC detectors in the world (ALICE, STAR, and SHINE) use DDL from Cerntech as the data link from the detector to the DAQ computers! /19/2016 Mérnökök és fizikusok együttműködése a nagyenergiás fizikában 8 8

9 Evolution of the ALICE Read-out Data Links Standardised detector data links (DDL) as the common interface between the detectors read-out and the DAQ (online system) D-RORC ALICE / LHC Run1: 2x Gb/s custom DDL & D-RORC C-RORC ALICE / LHC Run2: 12x 4.25 Gb/s custom DDL2 & C-RORC CRU ALICE / LHC Run3: developed by: CPPM (Marseille) Wigner RCP (Bp) VECC (Kolkata) Common Read-out Units (CRUs) as common detector, an trigger, and control interface 48x 10 Gb/s optical links (3.2 Gb/s radiation hard) x16, Gen3 PCI Express (Cerntech is not involved) /19/2016 Mérnökök és fizikusok együttműködése a nagyenergiás fizikában 9

10 ALICE DAQ Upgrade - LHC Run 2 During LS1, the TPC Detector Consolidation Project has been decided. The present Read-out Concentrator Units (RCU) to be completely re-designed Instead of having 3 different FPGAs, we will have only one, radiation tolerant, system-on-chip FPGA Splitting up of the two front-end branches to four more parallel read-out Increasing the DDL speed to 4.25 Gb/s ( DDL2) New Read-out cards, the C-RORC will read out the DDL2 links The new RCU2 cards were designed by Cerntech and Univ. Bergen (for ALICE TPC) /19/ cards got produced by Cerntech by 2015 Mérnökök és fizikusok együttműködése a nagyenergiás fizikában 10 10

11 SmartFusion2 SoC FPGA (Flash memory based) 2 SERDES interface: 4 lanes /each SERDES interface working modes: XAUI, Gen2 PCI Express end-point, Custom /19/2016 FPGAs in Radiation - T. Kiss (Wigner RCP) 11

12 ALICE TPC Detector Consolidation /19/2016 Mérnökök és fizikusok együttműködése a nagyenergiás fizikában 12 12

13 New Read-out Receiver Card for ALICE DAQ and HLT...now also used by ATLAS For LHC Run2, a new, 4.25 Gb/s ALICE Detector Data Link (DDL2) has been developed. This required new PCI Express read-out cards, too. A common read-out card called (C-RORC) for the DAQ and for the HLT has been developed by Frankfurt University, CERN and Cerntech Ltd. Conceptual design: CERN DAQ&HLT teams, Heiko Engel (IRI, University of Frankfurt) Schematics: Heiko Engel (IRI, UNiversity of frankfurt) PCB design: Cerntech Ltd. (Budapest) Firmware: Heiko Engel (IRI, Univ. Frankfurt), Filippo Costa (CERN) SI and PI simulations: Cerntech Ltd, Csaba Soos (CERN) Produced in: HU (small series), NO (final series) 12/19/ FPGAs in Radiation - T. Kiss (Wigner RCP) 13

14 12 x 6 Gb/s optical link C-RORC FMC interface SD card x8, Gen2 PCI Express 1156-pin FPGA DDR3 SDRAM SODIMM, 2x /19/2016 FPGAs in Radiation - T. Kiss (Wigner RCP) 14

15 10G with SmartFusion2 FPGA SIU+ : 10 Gb/s interface unit With careful PCB design......we will have a lot of 10G interafce options: SFP+ QSFP ipass (PCIe External cabling) Even two in the same time! /19/2016 FPGAs in Radiation - T. Kiss (Wigner RCP) 15

16 SIU+ Logic Diagram Pattern Generator PCIe Gen2 4x 5 Gb/s XAUI 4x Gb/s SFI Gb/s SIU interface PG SERDES IF SmartFusion2 SERDES IF A B PHY QSFP or SFP+ QSFP or ipass SoC Bus AMBA AXI4 (if needed) PCIe Gen2 or Custom 4x 5 Gb/s 4x 2.5 Gb/s /19/2016 FPGAs in Radiation - T. Kiss (Wigner RCP) 16

17 Other Interesting Configurations Test bed for protocol developments Data SIU Gen UDP/IP, 10G PCIe, 16G and / or UDP/IP, 10G protocol converter (CRU proto) PCIe, 16G detector side converter and concentrator (CRU proto) 4 x DDL, 2.5G UDP/IP, 10G server side converter and concentrator (CRU proto) 4 x GBT, 4.8G PCIe 16G 4 x copper, 1.25G ITS? converter and concentrator (CRU proto) UDP/IP 10G, or PCIe 16G 16x copper, LVDS, ITS? converter and concentrator (CRU proto) UDP/IP 10G, or PCIe 16G /19/2016 FPGAs in Radiation - T. Kiss (Wigner RCP) 17

18 Other Developments EDICAM (Event Detection Intelligent Camera) Cerntech also works for the plasma physics department of the Wigner Reserach Center, Budapest in designing of a special high-speed digital camera (EDICAM) for the video observing of plasma in fusion experiments High-speed CMOS sensor, needing 10 Gb/s data flow Adaptive camera control based on the image content (i.e. motion detection and prediction) Need for hardware signal procesing (fast and complex digital circuits) Radiation tolerance is an issue, again. FlexRIO Adapter module based on very high-speed Texas Instruments ADC and DAC chips ADC: 2 ch, 400 MS/s, 500 MS/s per channel DAC: 2 ch, 500 MS/s per channel External reference and sampling clocks, general digital I/Os /19/2016 Mérnökök és fizikusok együttműködése a nagyenergiás fizikában 18 18

19 CMS Barrel Pixel Detector Supply Tube Electronics Cerntech: Subcontracting for Wigner RCP in CMS Phase I Upgrade Strong 2-years co-operation between electronics engineers and physicists Connectivity boards to provide data, control, low voltage, and high-voltage connections in a very dense environment for the CMS Pixel Detectors ECAD MCAD co-design /19/2016 Mérnökök és fizikusok együttműködése a nagyenergiás fizikában 19 19

20 CMS Barrel Pixel Detector Supply Tube Electronics A few mm stack of PCBs (0.5 mm thick per each) /19/2016 Mérnökök és fizikusok együttműködése a nagyenergiás fizikában 20 20

21 CMS Barrel Pixel Detector Supply Tube Electronics Digital and Pixel Optohybrids (DOH, POH) in DOH-Motherboard, and POH-Motherboard Radiation tolerance at very high levels Using of commercial integarted circuits, FPGAs is not possible Passive components, ASICs only Productions of prototypes, and 11x 50 final PCBs (known as Budapest Boards in CMS) /19/2016 Mérnökök és fizikusok együttműködése a nagyenergiás fizikában 21 21

22 New Collaborations Subcontracting for Sagax Kft. in designing of high perfomance data acquisition cards (multi channel ADC/DAC cards) with radio signal processing in the field of microwave/rf communications first collaboration out of the scientific market very similar design challanges as in scientific / HEP data acquisition high-speed, high data volume channel-to-channel and card-to-card synchronization in-line hardware data processing with FPGAs Subcontracting for Wigner RCP in different new collaborations FAIR / CBM (at GSI, Darmstadt) Designing and prototyping Read-out Boards (ROBs) with the CERN radiation hard GBT chipset IMP (China) - Wigner RCP MAP Joint Laboratory Developing of DAQ components for a new beam monitoring detector /19/2016 Mérnökök és fizikusok együttműködése a nagyenergiás fizikában 22 22

23 Summary Cerntech Engineering Ltd. is a HEP spin-off company founded in 1997 by engineers working on development of reliable, high performance data transmission links and data acquisition systems for CERN and other HEP institutes. The present main activities are designing and producing high-speed data links that move high data volumes using up-to-date serial, optical transmission; and interface cards for the data source (front-end electronics, imaging sensors, digital cameras, etc.) and the data destination (server computers, RAID systems, etc.) Our hardware products operate mainly in Linux environment. As a research and development company we collaborate with Hungarian research institutes, universities, and other companies to be able to develope and produce complete systems for the scientific market. Our solutions are based on such technologies like the physical layers of Fibre Channel, Gigabit Ethernet and 10 Gigabit Ethernet standards; PCI-X, and PCI Express, utca, PXIe and designing of complex digital circuits with powerful FPGA-s. We also gained expertice in designing of radiation tolerant electronics and are able to design and - with our partners - carry out radiation tolerance tests and measurements Mérnökök és fizikusok együttműködése a nagyenergiás fizikában 23

24 References Main users of our high-speed data transmission products are: S-LINK cards for CERN and collaborators (ATLAS, CMS, COMPASS, etc.) different versions (altogehter cca. 700 units) DDL, D-RORC, and other DAQ hardware for ALICE at CERN different versions (altogehter cca units) DDL and RORC cards for STAR at BNL (altogether cca. 260 units) DDL, RORC (and other read-out cards) for SHINE (NA61) at CERN (altogether cca. 15 units, plus cca. 260 other interface cards) DDL and S-LINK interface cards produced by Cerntech also bought at: CEA (Paris) INFN (Roma Torino Bologna Napoli Pisa Pavia) IPN (Orsay, Nantes) NIKHEF (Amsterdam) Max-Planck Institute (München) Wigner RCP (Budapest) Stockholm University Mérnökök és fizikusok együttműködése a nagyenergiás fizikában 24 IFAE (Univ. of Barcelona) Univ. of Valencia Univ. of Lausanne TU München