Data Handling and Transfer for German LOFAR Stations

Transcription

1 Data Handling and Transfer for German LOFAR Stations On behalf of LOFAR and GLOW (slides stolen from many other LOFAR presentations by other people) 1/24

2 Some Initial Comments LOFAR network details still under development Continue to evolve for international partners as single-station (and stand-alone operation) needs are developed and implemented GLOW meeting in Garching next week Now mostly frozen within the Netherlands Will discuss many issues of single-station operation (and subarray interferometry) and network issues for achieving that within GLOW GLOW partners will discuss observing options, plus use of other stations (Uw, Ju) Many other network experts within GLOW who know more about the network details than I do I know more about the station and data issues than I do the network configuration Many network experts at the various stations 2/24

3 View of a LOFAR Station No complete German stations yet... Each dipole/tile polarization transmitted along a cable back to the station cabinet Each signal digitized by 12 bit A/D converter at 160 or 200 MHz rate 3/24

4 Inside a Station 4/24

5 Control Overview 5/24

6 Types of Data Flow for LOFAR Stations In: Station control commands Out: Beamformed data Main LOFAR observational data product Summation of antenna signals for specific direction(s) on sky (beams) 248 subbands maximum (subbands reused for 8 and 4 bit modes) Up to 3.1 Gb/s Out: TBB data Typically a small load, < 1 Gb/s May be up to 5 Gb/s for certain single-station applications Out: Monitor information A/D time series for each antenna/polarization, or polyphase filter values as function of time for each antenna/polarization Receiver temperatures, coarse station spectra, GPS information, etc. Out: Station visibilities 1 subband per second, for all receiver units 6/24

7 Keeping Things Organized: VLANs In ASTRON's LOFAR plan, all LOFAR data travels down a single 1x10 GE fiber connection Each data type separated by different VLAN IDs About 11 standard LOFAR VLAN IDs in use Many different types of monitoring data in previous slide are separated 7/24

8 QinQ 12 bit VLAN ID IDs for the entire world VLAN IDs chosen by ASTRON for LOFAR are often standard IDs used for various applications 1 through 100 filled with other uses VLAN IDs must be unique for security reasons LOFAR VLAN IDs cannot make it through the various German stations and Juelich Solution is to use QinQ Bundle all LOFAR VLAN traffic inside of another layer, with a different VLAN ID GLOW allocated range 2000 through 2100 for itself, to be kept free by all GLOW stations and Juelich 8/24

9 10 GE Network Following the upgrade of the LOFAR correlator (Blue Gene supercomputer) from the L to P class, ASTRON decided to switch to using 10 GE technology for LOFAR Blue Gene/P only has 10 GE connections 10 GE bid cheaper than 3x1 GE bid Extra bandwidth allows more things to be done International partners told to use 10 GE if at all possible Currently only a (money) problem for Tb and Uw Note that the fiber connection provides 10 Gb/s each in both directions Can also shove 10 Gb/s to a station 9/24

10 Dutch Core Solution Fibers from nearly all Dutch stations are first sent to a central collector buiding near the Exloo core Data then sent along a LOFAR fiber path to Groningen 10/24

11 International Network German stations send all LOFAR data through Juelich supercomputing center Data from different stations bundled onto common fibers Allows German stations to efficiently operate a German subarray Cb, On, and Nc will have LOFAR data go through Amsterdam 11/24

12 German Network Continued 12/24

13 German Network and Juelich 13/24

14 Juelich Research Center near Dutch boarder GLOW member, participating in networking/ storage/computation work in LOFAR Will also host the German university station Providing 1 PB of archive storage for early LOFAR work Can potentially house computational and extra storage for LOFAR/GLOW 14/24

15 Groningen Overview Main processing center for LOFAR (correlation and initial calibration and imaging pipelines) Blue Gene used as correlator and data formatter Calibration, RFI flagging, imaging, and so on happens in the off-line cluster for now 15/24

16 16/24

17 Basic Network Topology for GLOW 17/24

18 Notes on GLOW Topology Nearly every (TBD) GLOW station will have local data storage capability, with 10s of TB each Many stations will have local processing capability for singlestation operation Processing power needs (# CPUs) to be worked out in the future as we figure out how much we need Also have the ability to gather station data in Juelich Tap off data-stream to Groningen for monitoring Recording and/or processing at Juelich planned for the future Multiple single-station or subarray uses Can redirect data from other stations In principle possible to do at both Juelich and Groningen, but Groningen will likely have ability first Can directly pass other International or Dutch station data to Juelich or to a German station 18/24

19 Station Data: Individual Antenna Spectra Station hardware records average spectra over ~1 min intervals for individual antennas Example LBA spectra shown here Relatively low data rate, but also poor frequency and time resolution 19/24

20 LBA Example Dynamic Spectrum Made from a time series from an antenna on the previous plot Can see Milky Way rise and fall over several days Low frequency RFI also shows diurnal patern 20/24

21 Station Data: Visibilities Example Tb LBA image shown Can get visibilities for 1 subband (~ 200 khz) once per second All-sky imaging for LBA Can image full tile beam for HBA If certain antennas within tiles are shut off, can also make allsky images for HBA, but at reduced sensitivity Limited ability to remove RFI, low data rate 21/24

22 Station Data: Beamformed Data Station electronics coherently sums antenna signals together for specific sky direction Summation done in ~200 khz polyphase filter subbands Up to 248 subbands can be sent out from the station, with subband samples every ~5 μs Can process outside of station to get higher frequency resolution, with corresponding time resolution loss 22/24

23 Beamformed Data Data comes out of station in a raw format Software currently running on Blue Gene to convert the raw format to HDF5 files for further processing Documentation on current raw beamformed data format made available during Technical Status Meeting in May GLOW effort to port software to run on standard Linux computer needs to be finished Many single-station and interferometer subarray applications will probably want to read the raw beamformed data coming off the network to process in realtime Lots of development work ahead 23/24

24 The End 24/24