Pre-Processing and Calibration for Million Source Shallow Survey

Transcription

1 Pre-Processing and Calibration for Million Source Shallow Survey V.N. Pandey(Kapteyn Institute/ASTRON) for LOFAR Offline Processing Team April 1 st, 2009 CALIM 09, Socorro

2 Outline MSSS (MS 3 ) and LOFAR processing chain Preprocessing (DP 3 ) - Role + Capabilities - Ongoing development/open Issues Calibration (BBS) Role + Capabilities 5 - Ongoing development/open issues DP 3 - Default Pre-Processing Pipeline BBS - Black Board Selfcal System MS 3 - Million Source Shallow Survey (60MHz and 150MHz)

3 MSSS - Specifications *To be carried out with LOFAR 20(13+7) LBA (60MHz) HBA (150MHz) Sky Coverage (Ω) 2π str 2π str FOV 120 deg sqr 20 deg sqr Observation time 45 min x 4 15 min x 4 Max Baseline 10 km 10 km PSF Integration time 1 s 1 s Freq resln, BW 0.76 KHz, 8MHz 0.76 KHz, 8MHz Total data size 407 Tbyte 2.3 Pbyte Point source sensitivity(σ) I 5 mjy 0.5 mjy N(S>30σ ) 2.7x x10 5

4 Processing Chain - LOFAR OFFLINE CLUSTER

5 DP 3, BBS in Automatic Offline Processing Y Y Data stream from Stations Y Y CEP (BLUE GENE/P) Correlated Data Copied to Offline Storage/Cluster SB0 SB1 SB2 subbands spread along frequency axis SubBand N DP 3 (Default Pre-Processing Pipeline)!! Data -> Preprocessed, Flagged and Compressed SB0 SB1 SB2 SB N GDS BBS (Black Board System) DISTRIBUTED + Multi-Threaded Processing Data ->Calibrated SB0 SB1 SB2 SB N MWI (Master Worker Imager CImager) DISTRIBUTED PROCESSING Images, Sources..

6 DP 3 Aims/Objectives Carry out all Pre-Calibration common default steps in an efficient way. Software 1. Flagging Pre-Flags and Algorithm based 2. Phase Correction due to diff clocks at stns 3. Sub-Band Pass correction 4. Global Band Pass correction 5. Compression along Frequency Axis 6. Compression along Time Axis 7. Combining Different Sub Bands into a single Mset. I N T E G R A T E D D I S T R I B U T E D Source Code (C++) Parameter Set File, Cluster Description file, Runs in a Distributed (Purely Data) way on different compute nodes Each Measurement Set processed on one processor Reads the Measurement set only once for all steps Single Parset file for all the subbands

7 1. Flagging DP 3 Capabilities APRIOR Knowledge Based REGION CUM CRITERIA APPROACH (under progress!!) REGION: - Time, Freq, baseline, Correlation, Target, Direction CRITERIA: - A condition which Pixel in the region should satisfy Emphasis on using all instrument specific information regarding RFI environment ALGORITHM BASED Generic code to accommodate - different flagging algorithms - multiple cascade flaggers Currently used - MAD flagger (Hampel filter) (Two dimensional freq & time) - Mirroring on the ends- avoids edge effects - Can run on data column desired * - Multiple flaggers implemented Advantage: Provides region specific suitable flagging Ex. Threshold a function of correlation, baseline etc. R R1 R2 R3 R1- C1, C2,. R2 C1, C6 Satisfactorily results on actual data May be just good enough for MSSS (Surely not for Transients) Later Hierarchical system of flaggers

8 Pre-Flagging -> Region cum criteria Pre-Flagging refers to flagging bad data based on aprior information/estimates before RFI detection Algorithms take charge - Estimate must not use neighborhood visibilities Region Time range, Freq range, Correlation, Baseline, Target source, Direction (can be many) Time range Sidereal time, local time, UTC, Integration number, time since start Freq Range MHz, Subband number, Channel number Baseline (ant1,ant2), (baseline length, Direction) Direction any direction in sky, Zenith baseline length for example can be an expression (u>0λ & u<30λ, direction1) Criteria - a condition visibility should satisfy but should not involve neighborhood visibilities example ALL, amp>0.8, amp> 0.8 f(ν) amp<0.1f(ν) All operations expressed for entire observation expressed as region cum criteria statements Regions can overlap and have multiple criteria example R1 C1&C2, C1 C2 Use as much information as possible: Station beam calibration meta data, Satellite bands, Transmitter, ionosphere information, Bootstrap based on user inputs

9 DP 3 example: Flagger MS9315 (40MHzOct 24,2008) Frequency -> Frequency -> Before Flag After Flag Time (4hours) Integ 30s Time(4 hours) CS1_us0 and CS1_us1 (XX corrleation) No absolute Threshold Flagging

10 DP 3 example Typical RFI Cases which can be handled with ease One channel A single Baseline

11 DP 3 - Flagging Example Amplitude (XX) Time index

12 DP 3 Capabilities 2. Correcting for clock phases, and cable delays Correct for phase differences introduced due to different clocks at stations Predetermined Table of corrections to be applied At this stage exact algorithm is under investigation 3. Correcting for sub-band shape Bandpass shape within each Sub-band due to Polyphase filter bank Predetermined table as a function of frequency Already implemented at CEP (Blue Gene) In case we need improvements, correction may be implemented. 4. Correcting for Global Bandpass Global Bandpass correction due to Antennae response to Pre-determined table as a function of frequency Presently Estimation using BBS global solver under investigation Implementation to have multiple tables of corrections in one step

13 5. Compression along frequency axis 6. Compression along time axis DP 3 : Data Compression Implemented together, any one may be selected or switched off Compressed pixel = mean ( unflagged values ) Weight column appropriately modified depending on number of pixels flagged Time and Frequency of compressed (Averaged pixel) ** - MASK? (Long baseline work?) Multiple stage compression allowed Freq Time Performance of DP 3 >97% CPU usage most of the time (Data sets of ~ 5Gbyte)

14 Flag Category Multiple Bits Instead of single flag for each visibility, Use multiple bits to store flags by flag category Each bit tagged by User/KSP, Parset/algorithm log, dependencies of other flag bits, time.. log warning when conflicting conditions An intermediate file produced which expresses the parset in terms of condensed statements. A1 0 A2 1 A3 0 A4 0 A few advantages For being able to select desired combination of flag categories based on their performance - Comparative study of different flagging algorithms - Flagging during calibration ex. based on gain solutions can be stored - Flagging based on residual data can be stored - Each KSP may have different strategy for flags, so all can be accommodated - Minimal increased disk space (few bits compared to 8 bytes per correlation visibility) - One can combine algorithms of more than one user - For EoR it may be critical to have this information Data can reside at one place and users can have only model and corrected data..?

15 Inferences, issues Complete DP 3 Pipeline has been tested, works successfully. Computational Speed improved steps integrated in one. Integrated in the offline Pipeline. Code Generic support Multiple flaggers and Multiple levels of compression. Flags stored bitwise based on flag category Time and frequency centroid of averaged visibilities in compressed MS under discussion. Performance? Regularly used from? RFI for Transients.. Does not support flagging using data across different subbands/days together.

16 LOFAR Calibration - BBS Commonly used reduction packages for aperture synthesis data AIPS : VLA, WSRT, GMRT, ATCA, VLBI, Miriad : VLA, ATCA, WSRT, NEWSTAR : WSRT AIPS++ : WSRT, VLA, and Now CASA For LOFAR, with all it novel /complicated aspects, we need to do much better. Two packages have been, and continue to be, developed: MeqTrees is being used to develop/simulate our understanding BBS will be implementing efficiently/optimally what we have learned BBS (Based on Black Board Design Pattern for Distributed computing) Pool of independent processes operate on shared memory A central control process examines the black board and decides what is to be done next depending on the current state. Software to do matrix self-calibration Emphasis on performance, batch-mode operation, and distributed processing of large data volumes. Well integrated with SAS/MAC

17 Environment Processing strategy Observed visibility data BBS Calibrated visibility data Sky model parameters Instrument model parameters

18 BBS Capabilities Processing strategy List of processing steps Processing steps have the following attributes: Data selection Operation to perform Sky model Instrument model Supported operations Simulation Subtraction, Addition Correction Parameter fitting Easy to add other operations

19 BBS - Capabilities Supported parameter types Constants Polynomials of frequency and/or time Supported source models Point source Elliptical Gaussian Supported model components Bandpass (Directional) Gain Basic SPAM Dipole beam Analytical model (S. Yatawatta) Semi-analytical model (J.P. Hamaker) Dipole beam

20 Processing strategy Data will be read only once to minimize disk I/O Processing strategy List of processing steps to perform Hierarchical specification Root Root.Steps = [Peel0, Peel1] Peel0.Model.Sources = [CasA] Peel0.Steps = [Solve, Subtract] Peel1.Model.Sources = [CygA] Peel1.Steps = [Solve, Subtract] Peel0 Peel1 Solve Subtract Solve Subtract

21 Data distribution The visibility data of an observation is stored in a distributed fashion Individual subbands are distributed over the cluster This is only an issue when fitting parameters Parameter fitting combines visibility data from a certain domain in frequency and time All other operations are embarrassingly parallel BBS supports parameter fitting across subbands using separate solver processes

22 Calibration group 0 Parameter fitting SB0 Calibration process 0 Solver Equations Solver process 0 SB1 SB2 Calibration process 1 Calibration process 2 Solver Solver Model parameters Merge equations Solve SB3 SB4 Calibration group 1 Calibration process 3 Calibration process 4 Solver Solver Equations Model parameters Solver process 1 Merge equations Solve SB5 SB6 Calibration group 2 Calibration process 5 Calibration process 6 Solver Solver Equations Model parameters Solver process 2 Merge equations Solve Use of Storage node CPUs?

23 Inspecting the output Parameter values are stored in a (distributed) parameter database Python interface (Parm Facade) is available -Enables easy plotting, analysis of solutions-speed up commissioning A M P L I T U D E P h a s e Time Time

24 LBA image using BBS >500 sources Average { 24hrs, 36* Sub Bands: 0-35} MHz 00 h CygA (residual) 1% level 18 h 12 h Tycho Taurus Res ~ Rms ~ 1 Jy Band width(5 MHz) 06 h Sun Initial deep all sky wide field (full hemisphere centered on North Celestial Pole)

25 Open Issues High quality images with test stations routinely obtained Source positions agree - <1% of HPBW(0.50) on different days (avg) - <3% of HPBW with NVSS Technical - Control framework general enough - Scaling, Multi-Threading, Performance - Scheduling/Failing nodes? Algorithms Simultaneous solution v. peeling Solver robustness Alternative solver algorithms Models Beam Ionosphere, Mosaicing V.N.Pandey et. al. DPPP and BBS for MSSS CALIM 2009, Socorro

26 Thank you