What might astronomers want? Flexible data products and remotely-steered pipelines

Transcription

1 What might astronomers want? Flexible data products and remotely-steered pipelines Anita Richards AstroGrid / ALMA Regional Centre JBCA, University of Manchester thanks to e-merlin, ALMA, RadioNet ALBiUS and EuroVO teams and CALIM organisers

2 From sky to Nature paper What are interferometry data products? PIs's eyes' views of observational output B2E VO tools and standards "I don't trust anyone to reduce my data" "I just want an image" Metadata Stages in data processing Well defined, heuristic and goal-dependent steps Finding data Extracting customised ('virtual') products Interoperability and VO tools Examples biased towards cm/long-baseline p2

3 Available data products Raw visibilities MERLIN, VLA, (ALMA) Extracted info EVN, MERLIN, VLApipeline +Target self-cal solutions Target images, (cubes) older EVN, VLBA + Phase-ref etc. solutions older VLA, most spectral line + Instrumental calibration LOFAR source list p3 Associate target with cals Mostly welldefined Targetdependent requirements Need uv data to change resolution etc. Flexible timeaveraging etc?

4 Data cube PolarizationFirst Moment Image Interferometry visibility amplitude v. baseline length Calibrated, binned visibility amplitudes Data products p4

5 Observatory Domain Instrumental effects Strategy depends on array, observing parameters, otherwise ~independent of specific target May need updating in early operations Derive/apply reference source calibration e.g. as per George Moellenbrock EVLA polarization! Degree of rigour depends on target & goals Flux scale for comparisons or just detection? Can target be self-calibrated? Editing - iterative Some hard to automate e.g. line + continuum Pre-apply instrumental/unique calibration Hardest for non-radio astronomers to understand p5

6 Astronomer Domain Heuristics determine self-calibrate-ability e.g. Competing methods for sensitive wide fields Always calibrate polarization for completeness? Imaging etc. depends on goals Do you keep confusing sources for posterity? Image & spectral resolution/noise tradeoff Combining data from different arrays Resampling in time Extracting spectra, measuring sources etc. Spectral indices etc. require matching resolution May need to go back to visibility data Astronomers want to use favourite packages p6

7 Range of user expectations Know-all Will hand-craft everything the first time Becomes happy part-calibrated data saves time! Important to provide good metadata and tools Encourage their feedback/innovation Keen novice Pipeline uv cal, guided by proposal metadata Tackles refining self-cal/editing Prefers to use one, familiar package Give recipes with well-defined scope Interoperability essential Can't FT, won't FT Jargon-free pipeline interface Steer within instrumental constraints Clear explanation of pitfalls/artefacts p7

8 Range of user expectations Know-all Will redo everything the first time Becomes happy part-calibrated data saves time! Important to provide good metadata Encourage their feedback/innovation Keen novice Pipeline uv cal, guided by proposal metadata Tackles refining self-cal/editing Prefers to use one, familiar package Give recipes with well-defined scope Interoperability essential Can't FT, won't FT Jargon-free pipeline interface Steer within instrumental constraints Clear explanation of pitfalls/artefacts p8

9 Range of user expectations Know-all Will redo everything the first time Becomes happy part-calibrated data saves time! Important to provide good metadata Encourage their feedback/innovation Keen novice Pipeline uv cal, guided by proposal metadata Tackles refining self-cal/editing Prefers to use one, familiar package Give recipes with well-defined scope Interoperability essential Can't FT, won't FT Jargon-free pipeline interface Steer within instrumental constraints Clear explanation of pitfalls/artefacts p9

10 Range of user expectations Always provide full processing history Autopsy Tools Example parameters for further processing Troubleshooting postmortem p10

11 ARRAY model measurements logs Pipelines Metadata Optimum pipeline models self calibration science products User and steerable versions Occasional updates CAL SOURCES TARGET Average pipelines models calibration Advanced products with metadata Science pipeline ARCHIVE ready made products error information PI/public domain access self calibration, reprocessing etc. VO access

12 B2E Metadata flow Phase 1 Science proposal Phase 2 Science products Schedule e.g. ALMA proposal tool Archive Processing history, metadata Steerable pipeline Standard calibration pipeline Observations Quality control Basic irreversible correlation, calibration etc. Metadata: two varieties Key projects Major surveys Standard products documented methods Public access Re extraction recalibration etc To control processing inputs For humans to understand

13 incremental antenna, baselin e calibration Depends on: nce y t b varia us igh no we mple ge sa mo Ho na us ten as no an nn e n te og ter A He cing i a s o m ra te,p,a Ba sel ine len gth so li n t How will you choose cal strategy? a n e p th p y a& c n e u q e r F p13 Frequency Baseline length Nature of antennas (also e.g. uv coverage, available cal sources) Which steps unique, or conditional?

14 e-merlin UK radio interferometer , 4-8, GHz wavebands 2 GHz bw ~fills aperture at <8 GHz mas angular resolution Upgrade to e-merlin 10x-30x continuum sensitivity 3 ~5 Jy/12 hr at 5-7 GHz Spectral line sensitivity more than doubled Due for completion 2010 Optical fibre, L-band lenses, Lovell resurfacing etc. done Rx, signal path upgrades well advanced Configuring WIDAR station and baseline boards First fringes any day now! Legacy science program awarded, next call fall p14

15 MERLINImager MERLIN radio interferomety archive Massive visibility data sets Each dataset can provide a range of resolutions Total field of view >108 pixels - unwieldy Data/calibration on-line in multi-source files Specialised software e.g. AIPS to extract images RadioNet Parseltongue python wrapper to AIPS Python script executes local database query etc. Managed by AstroGrid Universal Worker Service Jargon-free interface No need for massive data sets/special software Tweak image in any FITS-handling package p15

16 MERLINImager User AstroGrid Desktop VO (Registry) explorer MERLINImager Position, size,, resoln, date Download Aladin, GAIA VOSpace UWS standard service MERLIN archive RadioNet Parseltongue/AIPS FITS images VOTable description

17 MERLINImager Simple inputs Defaults within available limits p17 Download or display directly using SAMP messaging protocol

18 VOTable SIAP-compliant VOTable Basic metadata (size, position, waveband etc.) Accref - URL of image, loaded when required p18

19 IVOA Interoperability standards International Virtual Observatory Alliance Simple Image, Spectral (etc.) Access Protocols Being extended to cover multi-dimensional data Incl. Healpix & non-positional searches Models under development: Polarization Visibility data parameters affecting science quality VO visualisation tools Connected via SAMP New image formats? 'MS'-like CASA images? p19

20 Aladin interprets VOtable p20

21 Qualitative manipulation in Aladin 5 GHz WR GHz non- Spectral index thermal image from matched resolution inputs (static archive images are optimum thermal at different p21 resolution per freq.)

22 MERLINImager pipeline Find via VO Registry (e.g. VOExplorer) User inputs: AstroGrid workbench provides dialogue box Or, set parameters in python script Required: Target name or position Optional: Freq. and date ranges, resolution, size Destination: cache, VOSpace or download Pipeline heuristics Supply data closest to requirements e.g. resolution: request 2"; 0".5 Up to 5 data sets with most visibilities processed Calibration applied, image CLEANed p22

23 Data returned to user Data service emulates IVOA SIAP standard Simple Image Access Protocol (being expanded) Returns pointers to data plus metadata Standard VOTable format VO-enabled tools display image properties Images stored at data centre, URL provided Download or send directly to VO tool/visualiser Other services VO: HDF(N) MERLIN+VLA cut-out server Web: Multi-source uv data + cal, flag tables Easier to implement in CASA than in AIPS? But Bourke's AIPS-Lite might be better still p23

24 Extracting radio variability curves X-ray binary peaks vary on timescales of hrs Search for known XRB in MERLIN Archive Often offset from pointing centre Tony Rushton developed data-mining pipeline uv data are stored in 1-month blocks per config. Use MERLINImager script to make target image Shift phase centre of target uv data to image peak Average all baselines in e.g. 3-min time intervals Plot 'light' curve Implement service for variability on all scales Eyres seeking summer student funding Phase-ref variability/structure database for e-merlin p24

25 VO tools in data processing NGC3351 Confusion all over 40' primary beam pixel maps Too slow to clean all Faint central sources hidden by sidelobes AIPS + TopCat: Coarse, dirty facets Capture maxima values VO Tool for Operating on Plot, shade by intensity Catalogues TOPCAT Select brightest subset Write out facet file! p25

26 Interoperability and metadata Metadata standards Can't change observatory engineering dialects Translation layer e.g. EuroVO DMMapper Standard i/o: SSAP, TAP; SIAP being generalised Interoperability for data products Many users will only learm one of AIPS, CASA etc. ALBiUS identifying priority stages and solutions Liaison with package developers &/or Py wrappers Should VO tools recognise MS-like images? Extension/metadata conservation in FITS MS Hard to find format definition or even name! How will direction-dep. calibration be transfered? How will spindex/curve. be attatched to CC? p26

27 Conclusions Identify scope of algorithms and strategies Mainly instrumental effects in domain of experts Automate/pipeline Target-dependent effects Deduce parameters from proposal/scheduling Provide examples and scope of applicability Capture metadata to allow flexible re-use Will often require going back to uv data Ground-breaking techniques still need specialists Robust non-jargon pipelines for all astronomers Package interoperability as required (e.g. ALBiUS) Use and contribute to VO standards & tools Graduate schools supply innovative new experts But 'real' users will break everything at first! p27