CASA. Algorithms R&D. S. Bhatnagar. NRAO, Socorro

Transcription

1 Algorithms R&D S. Bhatnagar NRAO, Socorro

2 Outline Broad areas of work 1. Processing for wide-field wide-band imaging Full-beam, Mosaic, wide-band, full-polarization Wide-band continuum and spectral-line imaging 2. Related High Performance Computing (HPC) Multi-threading, Cluster computing, GPU, Pipeline processing (imaging) 4. Establish cost-performance equation Relatively low-fte effort FTEs spread across 3 4 scientists 2

3 Workload Determine the landscape Characterize the problem, survey existing solutions, estimate the parameter space, etc. Develop a path with scientifically useful intermediate stops R&D for solution (start simple), stabilize the implementation, scientific testing, characterize the algorithm, etc. Publish papers in appropriate refereed journals Integrate with production software Overheads of issues related to new code in software system Usable HPC is necessary: So also involved with HPC effort(s) Write document, maintain code, even user-support... 3

4 Current relevant activities Algorithms for wide-band wide-field imaging Frequency dependence of the sky-brightness distribution Instrumental: Time- and frequency-dependent PB Heterogeneous PBs Scientific testing, characterization of limits Some results & details in later slides Full-polarization imaging In-beam polarization (full-mueller (?) Imaging) Numerical characterization of the problem Extend WB PB corrections to full-pol Wide-band full-pol. MT-MFS or Cube imaging RM synthesis 4

5 Current relevant activities Related HPC activities Large data + High Computing load Data parallelism on Cluster + Multi-threading GPU computing Determine balance between computing resources and imaging performance Needs scientific testers with domain expertise in advanced algorithms Establish cost-performance equation Important for development going forward Crucial for usable and reliable pipeline processing Of great interest for the larger RA community and algorithms R&D 5

6 Current relevant activities Pipeline processing Develop heuristics to determine an optimal path through the imager parameter space Needs understanding and characterization of limits, estimate of cost-performance equations, scaling laws, etc. Software development [Details in presentations later] Re-factor imager framework Integrate with existing parallelization framework, re-integrate with new parallelization framework when it is ready Test for correctness, performance Many overheads + inherently time-consuming 6

7 Some terminology / definitions Wide-band : Frequency dependent effects are significant Fractional bandwidth used for imaging > ~20% High Spectral Index sources Wide-field imaging: Imaging FoV requires PB or W-term corrections Imaging beyond the 50% of the PB at a reference frequency Single pointing wide-band imaging at lower EVLA bands Mosaicking (by definition!) at any of the EVLA or ALMA bands Imaging when Bλ >1 2 fd (error due to the W-term is significant) 7

8 Some terminology / definitions MT-MFS: Multi-term Multi-Frequency Synthesis algorithm To account for the frequency dependence of the sky brightness distribution Important for fractional bandwidth of > ~20% and dynamic range (DR) > ~103 A-Projection: Algorithm to correct for Direction-dependent (DD) effects (PB effects) as a function of time and polarization Useful for sensitive spectral-line imaging a.k.a Narrow Band A-Projection or NB A-Projection WB A-Projection: Algorithm to also account for frequency dependent DD effects (frequency dependent PB) PB corrections beyond 50% point in single-pointing imaging For accurate mosaic imaging at DR in the range of few x 103 Probably at even lower DR for full-pol imaging at any of the ALMA or EVLA bands 8

9 Wide band imaging MT-MFS for frequency dependence of sky brightness [Rau & Cornwell, A&A, 2010] S ( ν, l ) 3C286, BW= GHz No wide-band modeling of the sky emission DR: 1600 α (ν, l ) ν S ( ν o, l ) ( ν ) o 3C286, BW= GHz With MS-MFS (freq. Dependent model for the sky emission) DR: >110,000 9

10 PB effects: Characterization A-Projection for in-beam time and pol effects WB A-Projection for frequency dependence I Continuum ( l, Pol) = I ( l, ν) PB( l, ν,t, Pol) Time-dependent DD effects d ν dt Pol-dependent DD effects 10

11 Time and polarization dependence Effects of time and polarization dependence of the PB Stokes-I Errors due to PB Squint + Rotation + Pointing errors Purely instrumental Stokes-V artifacts Stokes-V Due to avg. PB 11

12 PB Effects: Frequency dependence WB A-Projection for in-beam frequency dependence ( l, Pol) = I ( l, ν) PB( l, ν, t, Pol) ( ν, l, Pol) = I ( ν, l ) PB ( l, ν, t, Pol) I Continuum I Spectral d ν dt dt PB Freq. dependence (blue curve) 12

13 Wide band wide field imaging: Characterization Effect of instrumental frequency dependence Pulsar Sp. Ndx -3.0 Artificially steep Spectral Index 13

14 Wide band wide field imaging: Performance evaluation MT-MFS + Standard Imaging MT-MFS + WB A-Projection WB A-Projection MFS + Standard Imaging Combined MT-MFS and WB A-Projection algorithm MT-MFS + NB A-Projection Ap.J.,

15 Wide band wide field imaging: Performance evaluation Characterize performance, limits [Rau & Bhatnagar, in prep.] Heterogeneous PB correction [Kundert & Rau, Masters thesis] Time-, shape-dependence, in-beam effects important at DR > 104 Size-dependent functions sufficient for ALMA for now (usable already) Size-dependent full-pol support for ALMA may be required next Brightest Source :100 mjy MT-MFS 6 ujy rms* peak res : 15 ujy Cube 4 ujy rms peak res : 20 ujy MT-MFS + WB A-Projection 2 ujy rms Cube + NB A-Projection 3 ujy rms 15

16 Wide band Mosaic Imaging Characterize performance, limits [Rau & Bhatnagar, in prep.] Intensity : Reconstructed / True Alpha : Reconstructed - True 16

17 Wide band Mosaic Imaging Characterize performance, limits [Rau & Bhatnagar, in prep.] RMS : 0.3 ujy Intensity : Reconstructed / True Alpha : Reconstructed - True 17

18 Full polarization imaging: Work in progress Extend WB A-Projection to full-polarization (full-mueller?) [PhD Thesis project of P. Jagannathan] [] [ ] I I I I o I o Q o U o V I I = I I obs I obs Q obs U obs V The Direction-dependent Mueller matrix (in Stokes basis) 18

19 Memory Parameter Space for HPC More memory per FLOP Computing Lesser memory per FLOP I/O Compute-to-I/O Ratio In terms of algorithm design Move towards higher compute-to-i/o ratio Minimize memory footprint Remain inside the green box 19

20 Related HPC Large data volumes (few 100 GB to few TB), higher computing load, higher memory footprint Distributed major-cycle on compute cluster + Luster FS [EVLA Memo #132,133, 2009] Favorable compute-to-i/o ratio Good scaling: 60 70% efficiency Memory foot-print an issue beyond a certain scale. Solutions: Multi-threaded gridding A single instance of gridder utilizing all available cores Optimal W-Projection planes [Golap] [Golap] Determine number of w-planes from the data rather than FoV Scientific testing in progress Frequency resolution for wide-band PB correction Rotation with PA: interpolation vs. Caching Oversampling 20

21 Testing: MT MFS + WB AWP + Mosaic + HPC Result Stokes-I 80-pointing EVLA WB MTMFS + WB A-P using ~40 processes [Rau & Bhatnagar, (work in progress)] 21

22 Testing: MT MFS + WB AWP + Mosaic + HPC Result Intensity Weighted Sp. Ndx. 80-pointing EVLA WB MTMFS + WB A-P using ~40 processes [Rau & Bhatnagar, (work in progress)] Unresolved issues Numerical noise with wide-band and large number of pointings TODO Evaluate solutions for WB OTFM 22

23 Imaging pipeline The imager can be configured into large number of states with vastly varying computing cost and imaging performance Staged development: Heuristics to determine imager parameters minimize computing costs and maximize imaging performance Auto-flagging: Existing algorithms (tfcrop, rflag) a good start, but need heuristics to use in a pipeline 23

24 Other, longer term activities Asp-Clean type deconvolution algorithm [L. Zhang s PhD thesis] Positively impacts memory footprint and Spectral Index imaging performance CompSens ideas built-in Evaluate other similar ideas ~60K Clean 50 MEM ~15K MS-Clean ~1 K Asp-Clean Image Id-BIM Niter VTrue- VModel Visibility 24

25 Other, longer term activities GPU computing Collaboration with NVIDIA Dev. Tech. Division + Univ. group Efficient for computing OTF convolution function computation and multi-scale computations Not yet clear if useful for gridding (the dominant cost) PB measurements/modeling Total FFT Mulit-scale image computation 25

26 Summary Pace of work is resource limited Resources of the right skill-set is crucial Following the 2009 memo plan and EVLA/ALMA requirements Wide-band continuum imaging requires MT-MFS for DR > few x 103 PB effects DR > few x few x 103 for full-pol. MT-MFS + WB A-P required for mosaic and spectral index mapping Work in progress Test combined WB imaging algorithm (including mosaicking) in production code Test deployment on HPC platforms (necessary for practical usability) Characterizing effects of in-beam polarization for full-pol imaging Towards developing wide-field full-band full-pol imaging capability Research deconvolution algorithms with smaller memory footprint 26