Event-Synchronized Data Acquisition System of 5 Giga-bps Data Rate for User Experiment at the XFEL Facility, SACLA Mitsuhiro YAMAGA JASRI Oct.11, 2011 @ICALEPCS2011
Contents: Introduction Data Acquisition for User Experiment Future Prospects Summary 2
SACLA : XFEL facility at SPring-8 (SPring-8 Angstrom Compact free electron LAser) SPring-8 ~700m SACLA Experimental Hall esee next talk TUDAUST01 for the SACLA Began operation in February 2011. Aiming for < 1Å of wavelength of FEL by 8-GeV linac. First lasing achieved on June 2011 at 1.2Å. User experiment will begin on April 2012. 3
Courtesy of M. Yabashi Anticipating a lot of Type of Experiments Generation of extreme state High peak brilliance x10 9 Atomic-level imaging for noncrystalline objects phage flagella, cilia Spatial coherence 100% XFEL Ultrafast pulse < 100 fs Probing of ultrafast chemical reaction & phase transition 4
Coherent X-ray Diffraction e.g. Coherent X-ray Diffraction Imaging Differential Cross Section dσ 2 2 A lot of = Pr images e F(K ) must dω Structure Factor be taken by a large i r F K r e K pixel sensor ( ) = ρ( to ) dr reconstruct the original structure. ρ (r) : Electron Density Distribution Sample Fourier Transform Phase Retrieval Sample Image Reconstruction without the aid of Lenses Unstained Human Chromosome Courtesy of Y. Nishino 5
Requirements for Data Acquisition(DAQ) Shot-by-shot data acquisition in synchronization with the beam operation cycle. In order to correlate the beam characteristics with the data. 60Hz at present, higher for future. Accumulate data of large image sensors into storage without data loss. >5 Gbps of data rate with 6M pixels at 60Hz. Data compression to reduce the bandwidth and storage size. On-line data-quality monitor for efficient data collection. Any type of users instrument specific to their experiment can be attached to our DAQ to work with. Like a Plug-and-play device. 6
Schematic of DAQ Network distributed system. MADOCA as the control framework. Front-end Electronics Image Sensors CCD cameras Beam line Instruments Monitors Users Experimental Instruments Detectors Front-end Computer VME : Solaris PC : Linux Data Buffering VME : Solaris PC : Linux Data Data Buffering Windows LabVIEW TCP socket client vi On-line Database MySQL 1G/10G Ethernet Accelerator Status Database Sybase DAQ Control GUI GUI On-line Data Monitor High Speed Storage 京 (K) Super Computer PC farm for Analysis 7
Schematic of DAQ Data transfer via TCP/IP over 1- and 10-Gbps Ethernet. FIFO buffering for non-real-time components. Front-end Electronics Image Sensors CCD cameras Beam line Instruments Monitors Users Experimental Instruments Detectors Front-end Computer VME : Solaris PC : Linux Data Buffering VME : Solaris PC : Linux Data Data Buffering Windows LabVIEW TCP socket client vi On-line Database MySQL TCP/IP 1G/10G Ethernet Accelerator Status Database Sybase DAQ Control GUI GUI On-line Data Monitor High Speed Storage 京 (K) Super Computer PC farm for Analysis 8
Schematic of DAQ Provide TCP socket client module of LabVIEW as a part of our DAQ. Easily communicate with our DAQ with only a small modification. Front-end Electronics Image Sensors CCD cameras Beam line Instruments Monitors Users Experimental Instruments Detectors Front-end Computer VME : Solaris PC : Linux Data Buffering VME : Solaris PC : Linux Data Data Buffering Windows LabVIEW TCP socket client vi On-line Database MySQL 1G/10G Ethernet Accelerator Status Database Sybase DAQ Control GUI GUI On-line Data Monitor High Speed Storage 京 (K) Super Computer PC farm for Analysis 9
Event Synchronization: Tag Number To reconstruct the data set in the same beam shot, a tag number is recorded in every data to identify which one is related to which beam shot. Sequential number of master trigger pulse. Counted by the trigger-counter module at each station. Delivered to users instruments as well. Encoded serial pulse-train, parallel bit-pattern. Master 1,2,3,.. 60Hz Pulse 1,2,3,.. 1,2,3,.. 1,2,3,.. 1,2,3,.. 1,2,3,.. 1,2,3,.. Gun Accelerator Components Monitor System Beamline Components Experimental Detectors Users Instruments 10
CCD array 4M pixels 60Hz DAQ for Large Image Sensors : >5Gbps >5 Gbps CPU board in total Custom-made VME Camera Camera FPG Link A FPG Camera Link Board Camera Link カード Camera A FPG CameraLink Link データカード Camera A FPG Link 圧縮データカード Camera A FPG > 2Gbps/connection Link 圧縮データカード Camera A FPG Link A FPG Camera 圧縮データカード Link 圧縮データカード Grabber A FPGA 圧縮データカード Lossless 圧縮データ Compression 圧縮 Linux K Supercomputer 10 Pflops, 2012.4~ 10 Gbps Ethernet control 10 Gbps Ethernet High Speed Storage PC Cluster Preprocess 10Tflops Reduced data for on-line monitor Data-handling Servers 11
High Speed Storage : DDN + StorNext Parallel-writing to a single file system Achieve over 5 Gbps in total. Convenient to handle data files in a centralized repository. additional sensor Data of CCD sensor #1 #2 #3 #4 CameraLink #5 #6 #7 #8 #9 #10 #11 #12 1G/10G Ethernet > 5Gbps in total Data file servers SAN >100Gbps Data Direct Network S2A9900 Quantum StorNext file system ~180TB >GB/sec Single file system 12
DAQ System Worked as Expected Intensively used during beam tuning of SACLA to view the beam profile. Ready for user experiments. June 7, 2011 First lasing at SACLA Beam energy: 7GeV Wavelength: 1.2Å MPCCD image 1 mm 13
Future Prospects In near future: Image sensor: 6M 76M pixels Beam repetition: 60Hz 120Hz 300Hz Data rate ~ x26 : ~150Gbps! ~ x65 : ~380Gbps! We must find: Faster data transfer and storage system RocketIO? FlexRIO? Distributed storage? Effective data compression / suppression Dedicated algorithm? Prism? JPEG2000? Ideas, suggestions are welcome. 14
Summary We have constructed an event-synchronized data acquisition system for >5 Gbps of data rate for the SACLA experiment. Shot-by-shot data acquisition up to 60Hz. Expandable for user instruments. Ready for user experiments. Further upgrades for larger detector size and higher repetition rate will be necessary in future. Faster data transfer and storage system. Effective data compression / suppression. Start fighting them. 15