Takeo Higuchi IPNS, KEK Belle DAQ group. 2009/01/07 SLAC Advanced Instrumentation Seminars

Transcription

1 Belle DAQ System Takeo Higuchi IPNS, KEK Belle DAQ group 2009/01/07 SLAC Advanced Instrumentation Seminars

2 Introduction to the Belle DAQ

3 Belle DAQ Overview trigger decision system clock trigger clock F/O Main coverage by DAQ Data flow: signal digitizer storage Timing: trigger timing distribution detector frontend TDC TDC TDC Readout PC detector frontend TDC TDC TDC Readout PC Event builder detector frontend TDC TDC TDC Readout PC Event builder Online Analysis PC detector frontend Readout Event TDC TDC TDC PC builder

4 Belle DAQ Diagram Coverage list Event building Run control Signal digitization Online data analysis HV control Digitized data readout Timing distribution Data quality monitor

5 Belle DAQ Virtual Tour 1 Signal digitization and data transmission Signal digitizer (TDC) PMC-sized CPU signal digital via LAN

6 Belle DAQ Virtual Tour 2 Trigger timing redistribution signal digital timing timing Trigger Timing Redistributor

7 Belle DAQ Virtual Tour 3 COPPER TDC module 2 Trigger timing redistributor PMC-sized CPU VME9U-sized Platform COPPER Detail of the COPPER is described later.

8 Belle DAQ Virtual Tour 4 COPPER crate + = After cabling of input signals

9 Belle DAQ Virtual Tour 5 Crate readout Digitized data readout by the PMC-sized CPU are sent to crate readout PCs via a network switch.

10 Belle DAQ Virtual Tour 6 Event building PCs / Online analysis PCs Fragmented data from each crate readout PC are sent to event building PCs to be combined to a single event record. Finally, the data are sent to online analysis PCs and recorded to hard disks.

11 Belle DAQ Virtual Tour 7 Trigger timing distribution again timing Trigger timing distribution System clock distribution ib ti Busy collection tim ming Fanout timing crate rear

12 Q-to-T System Drift chamber readout Drift charge to the sense wire Trigger latency = 21µs 2.1 t1 event Q L1-trigger T(Q) t2 0 t DRIFT t Q-to-T conv. t STOP t t1 = Drift time t1 t2 = Drift charge Q-to-T enables ADC+TDC simultaneously. Belle readout is based on Q-to-T + TDC

13 Introduction to the COPPER

14 COPPER System We have developed several excellent DAQ technologies by ourselves. Among them, today, we emphasize the COPPER System for its High flexibility to fit your experiment, Wide acceptance of L1 rate up to 30kHz, Broad bandwidth of > 80MB/s, Less DAQ deadtime with equipped pp pipeline, and Less requirement of knowledge in writing readout software.

15 Why We Developed COPPER? KEKB / Belle upgrade plan KEKB luminosity increases: cm -2 s -1 more L1 rate: 500 Hz khz Belle upgrades more readout channels: 40 kb/ev kb/ev Limit of the present Belle DAQ Deadtime fraction of the present FASTBUS-based DAQ will be extrapolated to ~20% L1=1kHz. Call for new DAQ accommodates with L1=30kHz.

16 COPPER FINESSE 4 Add-on modules to the COPPER. Responsible to digitize the input signals and output t them to pipeline FIFOs on the COPPER. Online CPU Add-on module to the COPPER. Responsible to readout the COPPER FIFO, to process and format the data, andto send the data to external PC via the network. RadiSys EPC-6315 Intel P3 800 MHz 256 MB memory Network boot RedHat Linux 9 100BaseT port 2 Data transmission line and control line. Trigger timing module Add-on module to the COPPER. Responsible to receive trigger timing and system clock from upstream and to deliver them to the FINESSEs.

17 COPPER Block Diagram add-on modules COPPER FIFO PMC modules fr rom su ub-det tector FINESSE FINESSE FINESSE FINESSE Loc cal bus dge Bri PC CI bus PMC CPU Network I/F Timing Module to ex xterna al PC

18 FINESSE SuperBelle: composite of different kinds of sub-detectors Detector A Requires ADC ADC FINESSE Detector B Requires TDC TDC FINESSE Detector C Requires buffer Buffer FINESSE COPPER COPPER COPPER The COPPER/FINESSE system enables one to concentrate on the digitizer part without worrying about implementation of the readout part. design the dete ector Flexible fitting to Predefined protocol Com mmon re eadout afte er the FIN NESSE

19 FINESSE Catalogue 2008 Several functions of FINESSEs are ready for hitchhikers. 65 MHz 8ch Digital Buffer 65 MHz 8ch FADC FINESSE Digital Buffer FINESSE TDC FINESSE 500 MHz 2ch USB I/F FINESSE w/ AMT3 TDC chip FADC FINESSE (Debug use) TDC FINESSE w/ HPTDC chip TDC FINESSE w/ Vertrx5 FPGA Pixel Readout FINESSE Dry run FINESSE

20 Inside the COPPER FINESSE Typical FINESSE design from detector Analog / digital signal FINESSE Sampling clock digi tizer 186x76 mm 2 L1 pipeline Untriggered garbage FIFO full to COPPER FIFO Local bus System clock 42.3 MHz L1 trigger Busy response

21 Inside the COPPER FINESSE I/F Data Transfer from FINESSE to COPPER Data are fetched at FWCLK timing during FWEN ==L - FWCLK and FWEN are operated by the FINESSE to the COPPER FIFO. FWEN also indicates a single event sequence. - Event-end marker is needed to build up the 4-FIFO data to an single event record. - # of FWCLK during FWEN ==L is also counted by the COPPER FPGA to determine the DMA start timing from the FIFO to CPU. No-word word event is also OK. FWEN FWCLK FF00-FF31 Firmware limit #-of words / FIFO / ev < 256kB

22 Inside the COPPER Local Bus 1MB 32 Local bus DMA PCI-local bridge PLX-9054 PCI bus 8 MB addr 33 MH z DMA controller FIFO ctrl Cascaded FIFO JP Cyclone EP1C12Q240C6 bypas ss IDT72V36110 FINESSE Address decoder FIFO empty/full monitor 4-FIFO event builder Data formatter DMA start interrupter 512k IDT72V k

23 Inside the COPPER PCI Bus CPU Ethernett Intel Base-T CPU Bridge EPC-6315 DMA A32D32 33 MHz PCI-VME bridge Special DPM 4kB dual port memory PCI-PCI Bridge TI PCI-2050 PCI bus CPU boot, Run control Ethernet Intel Base-T Local bus PCI-local bridge PLX-9054 DMA controller L1 trigger PCI-local bridge PLX-9054 TT-RX

24 PMC: PCI Mezzanine Card Standard 100% Compliant w/ PCI RadiSys EPC-6315 Good for a high density applications Many commercial products Ethernet cards, CPUs, GbE cards, memory modules, etc. PMC sized CPU module. Equipped with Intel PentiumIII 800 MHz w/ 512 MB memory. RedHat Linux 7.3 or 9, or FedoraCore 1 Linux run on it. Price = ~ 120,000 / module. PCI NIC PMC NIC Bootable from CF card or from network.

25 Inside the COPPER DMA Data FIFOs FWCLK FWEN FWCLK FWEN FWCLK FWEN FWCLK FWEN #-of-words Almost full PCI intr DMA start interrupter #-of-words #-of-events counter The Cyclone FPGA issues DMA start interrupt onto PCI bus when One of FIFOs is almost full, #-of events in FIFOs exceeds a threshold, or #-of words in FIFOs exceeds a threshold (blocked until all event chunk has transferred to the FIFO.) #-of-words FIFO

26 Inside the COPPER DMA The CPU starts DMA upon the PCI interrupt. Counts up #-of words in the FIFO together with header/trailer words. Setup DMA and start. Then, the Cyclone FPGA controls the FIFO RE/RCLK. Chained readout to realize chained DMA.

27 Timing Distribution for COPPER VME6U VME6U VME6U FINESSE TR RG S EQ TT T-IO CAT-5 serial bus connection TT- -SW TT T-RX FINESSE FINESSE SEQ Event sequence controller. TT-IO Clock / trigger repeater from the SEQ. Busy collection. TT-SW Clock / trigger fan-out. Busy fan-in. TT-SW can be cascaded. TT-RX Clock / trigger fan-out to the FINESSEs. Busy collection from FINESSEs. COPPER TT-RX TT-RX TT-RX FINESSE KEK-VME 9U Crate M.Nakao

28 Trigger-Busy Handshake Every L1 trigger to the FINESSE must be replied by busy. To avoid possible event loss. Until the busy is cleared, the next L1 trigger will be blocked. - L1 trigger delivery resumes 2 SCLKs after the busy cleared. FINESSE L1 trigger Busy Busy response: no later than 2 sec or no busy error happens. Busy duration: 1.5 SCLKs or more, but less than 5 sec or keeping busy error happens. In case of FIFO almost full (remained buffer == event size max) TT-RX Do not issue busy immediately or unexpected busy error happens, but keep busy raised at the response to the next L1 trigger.

29 COPPER Performance Study COPPER Dry run FINESSE Dry run FINESSE Dry run FINESSE Dry run FINESSE Compressed by 10% CPU Trigger Timing Mod. 416 bytes/ev/finesse (typical data size for SuperBelle CDC) Accep pted L1 rate [khz z] Design Max L1 SuperBelle Typical L1 SuperBellee e Input L1 rate [khz] The COPPER works well even under the severest L1 rate (>30kHz) of the SuperBelle.

30 AMT-3 FINESSE

31 AMT-3 FINESSE Motivations First practicable FINESSE. COPPER in-situ study in the working DAQ. Boundary conditions of R&D Compatibility with the working DAQ (FASTBUS TDC). Tolerance under the SuperBelle operation. Our choice of the TDC chip = AMT3

32 What s AMT-3? ATLAS MUON TDC A TDC chip to readout the ATLAS muon chamber. Y. Arai, M. Ikeno, S. Iri, T. Sofue, M. Sagara and M. Ohta, Development of a new TDC LSI and a VME module, IEEE Trans. Nucl. Sci. 49, 1164 (2002). Y. Arai, Development of front-end electronics and TDC LSI for the ATLAS MDT, Nucl. Instrum. Meth. A 453, 365 (2000).

33 Why AMT-3? The AMT-3 satisfies all requirements for the SuperBelle drift chamber. For Tracking Requirement AMT-3 Position i resolution < 130 µm 27 µm For de/dx measurement Dynamic range 10 bit 17 bit Linearity < % 0.49% Other boundary conditions Single rate 200 khz Total channel # ~ 15 k 24 ch/chip

34 Snapshot of the AMT-3 FINESSE Tandem FINESSE FASTBUS TDC (manufactured by LeCroy) has 6 cable-connectors 16 channels = 96 channels. Not to change the cables configuration, we developed tandem FINESSE (occupying 2 slots), with 3 connectors. AMT-3 TDC chip Spartan3 FPGA AMT3 register control. Data readout from the AMT-3 output FIFO. Data formatting (header/footer etc.). Data output to the COPPER FIFO. I/F to the COPPER local bus.

35 FINESSE Connector / Input Connector Normal pitch 17x2 pins 3 connectors Input 48 ECL inputs / tandem - Receiver chip = SN65LVDT348PW 3 GNDs Connector pin assignment is completely compatible with the LeCroy 1877S. COP PPER sid de IN+ IN- ch#00 ch#01 ch#02 ch#03 ch#14 ch#15 GND LED ch#16 ch#17 ch#18 ch#19 ch#30 ch#31 GND LED ch#32 ch#33 ch#34 ch#35 ch#46 ch#47 GND

36 Pipelines in AMT-3 AMT-3 output AMT-3 Readout FIFO (64 edges) Never gets full L1 pipeline FIFO (256 edges) 24 Channel buffer (4 (4 edges) Channel buffer (4 edges) Trigger matching Trigger timing FIFO (8 events) 24-channel inputs Level-1 trigger inputs

37 System Clock Bunch crossing rate LHC = 40 MHz KEKB = 508 MHz To operate the AMT-3 with a similar SCLK as the LHC, we choose SuperBelle SCLK = MHz MHz = 508 MHz / 12.

38 AMT-3 Timing Measurement Dynamic range = 17 bit: coarse + fine counters Coarse counter: 12-bit bunch crossing counter LHC Belle) Fine counter: 5-bit time memory cell 1 SCLK / 2 4 Edge timings are measured with coarse + fine. Ti Trigger timing i is synchronous to coarse. LSB = 074ns@ 42 3 MHz SCLK LSB = MHz SCLK Comparable to the LeCroy TDC (0.5ns)

39 AMT-3 Trigger Matching How it works? preset L1 latency preset search window L1 pipeline t=0 L1 Edges Output L1 Search Determine trigger are the edges comes stored found t=0 in edges to within from the L1-pipeline AMT-3 to a preset the readout search L1 latency FIFO window t

40 Preset Parameters for CDC Preset L1 latency (virtual latency) 8.1 us Edge search region 8.1 us == whole coverage Real L1 laetncy 2.2 us Delay by TTRX 1.9 us t=0 SEQ L1 trigger TT-IO to AMT-3 TT-RX Red numbers are preset based on a request by the CDC group.

41 AMT-3 Readout DREADY AMT-3 CS, DSPACE GETDATA Spartan3 0xa0... 0x xc0... D t t f i t ll d b Data transfer is controlled by DREADY/GETDATA handshake.

42 AMT-3 Event Format header edge data #0 edge data #1 edge data #2 : start with 0xa TDC id event # trigger timing start with 0x3 TDC id edge type (up/down) edge timing (relative from L1) error edge data #n start with 0x6 TDC id error (if happens) error type (most likely L1 full) trailer start with 0xc TDC id The AMT-3 can output more information event # in principle, but the Spartan3 does # of words not recognize other format than this. (incl. header & trailer)

43 AMT-3 Spartan3 COPPER AMT-3 Spartan3 The AMT-3 outputs are stored in a 32-bit FIFO within the Spartan3. Spartan3 COPPER FINESSE header AMT-3 outputs FINESSE trailer w/ check sum COPPER_FIFO_WENB COPPER_FIFO_WCKL Driven by the Spartan3 SCLK # of WCLKs within WENB gate is considered as # of words / ev by the COPPER FPGA.

44 Spartan3 Output Format CPU. ware OPPER y a softw every CO moved by ning on e Rem runn 0xffaa 0x0000 event# tag header edge data #0 : edge data #n error (if happens) trailer 0xff55 checksum Event# (24) counts from 0 / Event tag (8) XOR checksum (16) XOR s of all Spartan3 outputs ([31..16] xor [15..0]) word=0 xor ([31..16] xor [15..0]) word=1 :

45 AMT-3 Device Driver When loaded by insmod Registers the driver itself to the Linux kernel. Resets the AMT-3. Presets AMT-3 registers to default values. When unloaded by rmmod Unregisters the driver. open() close() close(fd); Does nothing special. ioctl() ioctl(fd, COMMAND, arg); Resets the AMT-3. Manipulates AMT-3 parameters. Views AMT-3 status registers. Manipulates Spartan3 registers. int fd = open( /dev/copper/amt3:?, O_RDWR); Rejects duplicated device open. Does nothing else.

46 System Test

47 Full Scale Test Bench Network switches Pulse Generator (signal & trigger source) COPPER/AMT3 6 crates Full-scale data flow simulation from the PG to the readout PCs. Detailed study of the AMT3 behavior (including debug). Establishment of the global control scheme.

48 In-Situ Study with Belle CDC Word-by-word comparison of readout data by the FASTBUS and the COPPER-II/AMT3. CDC STBUS 3 LSB) T3 FAS it: AMT3 AMT (uni from CDC Q-to-T Residual distribution Resolution AMT LSB (RMS) Daisy-chain FASTBUS COPPER 16 AMT3 FASTBUS LeCroy TDC 16

49 Full Replacement of CDC DAQ 6 COPPER crates 89 COPPERs Readout PCs Q-to-T TX SW Boot SW Q-to-T Q-to-T TX SW Boot SW TX SW Boot SW Local event building PC Q-to-T TX SW Boot SW Q-to-T TX SW Boot SW Q-to-T TX SW Boot SW

50 Reduction of DAQ Deadtime Typical data size S.Y.Suzuki Deadtim me per event (µs) 29.5 µs FASTBUS We achieved deadtime reduction by 90% by installing pipelined DAQ with COPPER. 2.8 µs COPPER-II/AMT3 # of hits/tdc

51 Performance: x-t Curve ft time [n ns] Dri x-t curve w/ COPPER/AMT3 Belle real data Exp=55, Run=1526 (Dec.11, :13 -) Memo x-t curve w/ LeCroy Track-wire distance [cm]

52 Future

53 COPPER-3 COPPER-II COPPER-3 Replacement of terminating parts with recent ones. Fix jumper patches by pattern layout. Upgrade onboard Ethernet chip to Gbit-Ethernet. and others. We confirmed the COPPER 3 is fully compatible We confirmed the COPPER-3 is fully compatible with the COPPER-II using AMT3 FINESSE.

54 Brief Prospects of SuperBelle DAQ Preliminary design Detector Digitizer Location FPGA link FINESSE Pixel? On hybrid Yes Link RX SVD APV25 EH No Special CDC ASD based In detector Yes Link RX RICH/TOP Special ASIC In detector Yes Link RX ECL Waveform Sa. On detector Yes Link RX KLM? In detector Yes Link RX On/In detector digitizer FPGA Data and timing link Rocket I/O? Electronics hut COPPER Link RX FINESSE

55 Advertisement

56 COPPER Is Not Difficult Several functions of FINESSEs are ready for you. Thanks to an excellent device driver, the COPPER can be read out quite easily. int main(int c, char *v[]) { int fd; fd = open( /dev/copper, O_RDONLY); /* put your FINESSE initialization here */ while(1){ static char buf[1024*1024]; read(fd, buf, sizeof(buf)); /* data read */ } } That s all.

57 COPPER Users Belle: SuperBelle: T2K: musr Half of the DAQ is COPPER ized Planning to utilize the COPPER. Beam monitor Please join us and enjoy data taking with the COPPER system. Contact me at

58 Summary We developed a new readout system COPPER toward the higher luminosity HEP experiment. We developed a TDC FINESSE equipped with an AMT3 chip to readout the Belle CDC; In the in-situ study, we found the COPPER/FINESSE/AMT3 system showed high compatibility to the FASTBUS DAQ system. The COPPER is quite easy to operate. We introduce you to join us and to be a member of COPPER user team. We are designing COPPER-3. Design of a new FINESSE for SuperBelle has also been started. In SuperBelle, one of key roles of the FINESSE will be data RX.