High dynamic range imaging, computing & I/O load

Transcription

1 High dynamic range imaging, computing & I/O load RMS ~15µJy/beam RMS ~1µJy/beam S. Bhatnagar NRAO, Socorro

2 Parameterized Measurement Equation Generalized Measurement Equation Obs [ S M V ij = J ij, t W ij E ij s,, t [ F. I s, ] ] Improvements in sensitivity is achieved by larger band-widths and collecting area Combined RHS determines the time constant over which averaging helps Unknowns Direction independent Ji: Complex gains assumed constant across the FoV Direction dependent EiS: Aperture plane effects, ionosphere/atmosphere IM: Sky brightness distribution

3 Parameterized Measurement Equation Implications for high dynamic range imaging Smaller scale variations over larger parameters space needs to be accounted for Very significant increase in the number of samples for the required SNR per DoF Need more sophisticated parameterization of the ME Better parameterization (of the Jis, EiSs and the Sky (IM)) Solver for the (unknown) parameters Forward and reverse transform that account for the DD terms Efficient runtime implementation Useful parameterization: Which models the effects well and with minimum DoF For which efficient solvers can be implemented

4 Example: Imaging extended emission Conventional imaging Imaging with A-Projection Run time for both is comparable. Stokes-V imaging of extended emission Algorithms designed for point sources will not work

5 Parameterized Measurement Equation Solvers for DD terms can be as expensive as imaging, or more Three fundamentally different approaches being pursued Corrections in the data domain (FFT based transforms) AW-Projection, Pointing SelfCal, Mosaicking,... Challenges: Controlling the propagation of errors Corrections in the image plane (DFT based transforms) Theory: Rau, Bhatnagar, Voronkov & Conrwell (IEEE, in press) Peeling / Faceting Challenges: Curse of dimensionality, runtime efficiency with realistic data volumes Linear Algebra methods (Least-Square All Sky Imaging) Hybrid: Image- (or UV-) plane faceting + Projection algorithms Facets on a few strong sources only

6 Recent advances Better parameterization for the sky MS-, Asp-Clean (TJC, 2003/SB, 2004) Shapelets (LOFAR), Compressed sensing MS-MFS (Rau et al., ) Correction for DD terms Known apriori: W-Projection (Cornwell et al. 2004/08) Measured/modeled: A-Projection (Bhatnagar et al., 2005/08) ~1000 Components Solvers for unknown DD terms Pointing SelfCal (Bhatnagar et al., 2004) Aperture illumination (work in progress) Ionospheric effects (Cotton et al., LOFAR Team) Peeling (LOFAR Team) Computing - Stream processing : LOFAR, ASKAP, MeerKAT

7 Full beam imaging Limits due to the rotation of asymmetric PB Max. temporal gain ~10% point DR limit: few X 104:1 Limits due to antenna pointing errors In-beam error signal 50% point DR limit: few X 104:1 Limits for mosaicking would be worse Significant flux at half-power point Significant flux in the side-lobes for most pointing Approach taken Algorithm R&D (SNR per DoF, error propagation, computing requirements,...) Test the fundamental performance with realistic simulation Apply to real data

8 Time varying DD gains due to PB Wide band PB d M I =PSF PB. I

9 Generalization of SelfCal: Pointing SelfCal Typical antenna pointing offsets for VLA as a function of time Over-plotted data: Solutions at longer integration time Noise per baseline as expected from EVLA (Bhatnagar et al., EVLA Memo 84, 2004) Minimize : V Oij Eij V M ij w.r.t. Ei Sources from NVSS. Flux range ~2-200 mjy/beam

10 A-Projection algorithm: PB corrections Before Correction (Bhatnagar, Cornwell & Golap, EVLA Memo 100 (2006)) After Correction Minimize : V Oij Eij [ F I M ] w.r.t. I M Extend for PB corrections in wide-band imaging (Rau et al.) Overlapping main- and slide-lobes at 2:1 bandwidth ratio Tricky to manage aliasing and computing load

11 VLA L-band imaging: Stokes-I and -V Stokes-I Stokes-V (10x improvement) A-Projection, Bhatnagar et al., A&A,487, 2008 EVLA Instrumental Stokes-V pattern

12 Pointing SelfCal: VLA L-band VLA L-band data Solutions are noise limited Single Channel (24KHz) Tests with wide-band EVLA in progress

13 Primary beam effects: Wide band Model for the EVLA Power Pattern PB variation across the band EVLA 2:1 Bandwidth ratio: Sources move from main-lobe to side-lobes 10% 50% 90% Avg. PB Spectral Index (1-2GHz) Sources of time variability PB rotationally asymmetric PB rotation with PA CrossPB hand power scaling with frequency patternantenna pointing errors (Rau s talk for more details) Cross polar power pattern

14 PB effects in mosaicking: Wide(r) field

15 Computing load Scaling laws for imaging Non co-planar baseline correction W-Projection: (N2wproj+ N2GCF)NVIS Faceting: N2facetsN2GCFNvis PB Correction AW-Projection: N2GCF * Nvis Peeling: Ncomp * Nvis * (~10) Combine with Scale-sensitive deconvolution Nvis: , N2GCF: 7-50, Ncomp: 104-5

16 I/O load Near future data volume (1-2 years) GB/12hr by mid-2010 Effective data i/o: few 100 TB Next 5 years 100X increase (in volume and effective I/O) Non-streaming data processing Expect passes through the data (flagging + calibration + imaging) Exploit data parallelism Distribute normal equations (SPMD paradigm looks promising) Deploy computationally efficient algorithms ( P of SPMD) on a cluster

17 Computing challenges Direction dependent terms As expensive as imaging Significant increase in computing for wide-field wide-band imaging E.g. convolution kernels are larger (up to 50x50 for single facet EVLA A-array, Lband imaging) E.g. Multiple terms for modeling sky and aperture for wide-band widths Terabyte Initiative: 4K x 4K x 512 x 1Pol tests using 200 GB data set Timing Simple flagging : 1h Calibration (G-Jones) : 2h15m Calibration (B-Jones) : 2h35m Correction : 2h Imaging : 20h Compute : I/O ratio : 2:3

18 Parallelization: Initial results Spectral line imaging: (8GB RAM per node) Strong scaling with number of nodes & cube size Dominated by data I/O and handling of image cubes in the memory 1024 x 1024 x 1024 imaging Run-time with 1-Node : 50hr Run-time with 16-nodes : 1.5 hr Continuum imaging: (No PB-correction or MFS) Requires inter-node I/O (Distribution of normal equations) Dominated by data I/O 1024 x 1024 imaging: 1-node run-time : 9hr 16-node run-time : 70min (can be reduced up to 50%)

19 Parallelization: Initial results Image deconvolution is the most expensive step Most of the time spent in the forward and reverse transforms (gridding/de-gridding) Matching data access and in-memory grid access patterns is critical Optimal data access pattern for imaging and calibration are in conflict Freq-Time ordered data optimal for imaging Time-ordered data optimal for calibration DD calibration comparable to imaging in computing SS deconvolution + MSF might make FLOPS per I/O higher: A good thing!

20 Work in progress System integration Imaging Wide-band imaging: Integrate algorithms for various DD terms (W-term, PBcorrection, MFS) Wider-field, full-polarization imaging Imaging extended emission / wide-band mosaicking Return of Scale-Sensitive Deconvolution Calibration Testing pointing SelfCal Testing ideas for modeling aperture illumination Parallelization of all of the above!