Data Inspection Tutorial. Hiroshi Nagai

Transcription

1 Data Inspection Tutorial Hiroshi Nagai

2 Scope of this tutorial This tutorial demonstrates the step-by-step procedure how to see/check/understand the ALMA data using an archive dataset. How to check the data products and investigate the data quality How to reproduce the calibration/imaging results Some tips which should be checked by users Data: Pipeline-reduced data Level: Beginners - Intermediate

3 References ALMA Data Products Document ALMA Technical handbook Presentation at Mitaka Town Meeting Data Reduction Tips by H. Nagai (mainly for manuallycalibdated dataset, only in Japanese)

4 Preparation Install CASA Download the presentation file and data package from tutorial webpage Install analysisutils

5 What is the ALMA Archive Data? ALMA archives (a) raw data (b) calibration scripts, final image products (FITS image), and supporting materials. Part (a) consists of visibility datasets in ALMA's native format, the ASDM, which are only needed if the PI would like to perform custom data calibration and imaging. Part (b), the QA2 Products, consists of the QA2 data reduction scripts, log files, calibration, and flagging tables, and the imaging products on which the quality assurance decision was based. The calibration/imaging is done at ARCs and JAO by either manual or ALMA Pipeline (Quality Assurance) Pipeline calibration + manual imaging : Standard mode observation Manual calibration and imaging: Non-standard mode observation, standard mode observation which needs significant manual intervention (low SNR, too many flagging, temporal pipeline bugs, etc) (What s standard and non-standard? See PG/TH.) Roughly speaking, PL:Manual = 6:4

6 Recent update NAOJ provides calibrated Measurement Sets to PIs at Japanese institutes as an optional service. Dear XXXX XXXX, As you received a notification , the data of the member ObsUnitSet uid://xxxx/xxxx/xxxx of your project XXXX.S are now available to you for download. EA-ARC has also prepared calibrated Measurement Sets (MSs) for this SB. If you would like to download the calibrated MSs, please visit following ftp site. ftp location: ftp://af0xxx@alma-dl.mtk.nao.ac.jp/calibrated.tar.gz user name: AF0XXX password: XXXXXXXXXXXX Available until: :54:47 JST ALMA Imaging Pipeline has been in operation. Roughly 15 MOUSs imaged with PL has been delivered to PIs. Note the size of delivery package can be very large (>100GB).

7 QA Philosophy The goal of ALMA Quality Assurance (QA) is to ensure that a reliable final data product is delivered to the PI that has reached the desired control parameters outlined in the science goals, is calibrated to the desired accuracy and contains no significant calibration or imaging artifacts. For Cycle 4, as for all previous Cycles, QA will be done on a best effort basis, covering all the major issues affecting data quality. The ALMA QA2 data reduction team performs for each ALMA science data set a detailed analysis to confirm that the observations have achieved the science goals requested by the PI.

8 QA Philosophy If the requirements are met within the cycledependent tolerances (see chapter 11 of the ALMA Technical Handbook), the data are declared QA2 pass, packaged in a standardized way, and delivered to the PI. If the requirements are not met but additional observations of the SB are no longer possible within the cycle, the data are delivered as QA2 semipass.

9 Example Data (public data) Project: S (PI: R. Indebetouw) MOUS id: uid://a001/x12e/x280 Scheduling Block: sn1987a_a_07_te

10 Data Search Go to Science Portal Click Archive Query

11 Archive Query & Request Handler Search by Project code

12 Project Structure Tree Project code Science Goal OUS Group OUS Member OUS Scheduling block Product (tar package containing image products and supplement materials) ASDMs

13 Data Package Package name should be in the format of ProjectCode_MOUSuid_001_of_001.tar e.g., S_uid A001_X12e_X280_001_of_001.tar Project code MOUS uid Untar the package, i.e., tar xvf S_uid A001_X12e_X280_001_of_001.t ar S/science_goal.uid A001_X12e_X27e/group. uid A001_X12e_X27f/member.uid A001_X12e_X280

14 Check README There also should be raw/ directory if you downloaded asdm packages and untar them. member.uid A001_X12e_X280]$ ls calibrated calibration log product qa README script member.uid A001_X12e_X280]$ less README NB: calibrated directory can be created by running scriptforpi.py. This was already done for this tutorial.. Use same (or newer) version of CASA and pipeline

15 Check Weblog Weblog is a system of webpages containing all the diagnostic plots and other information generated by the pipeline For this data, there are two execution blocks.

16 QA2 Report for Manual Calibration Instead of weblog, several png images and a text file for each EB, so-called QA2 report, is provided under qa/ directory.

17 QA2 report (png images)

18 QA2 report (text file)

19 Let s check weblog together [nagai@alma-proc01 member.uid A001_X12e_X280]$ ls calibrated calibration log product qa README script [nagai@alma-proc01 member.uid A001_X12e_X280]$ firefox qa/pipeline t113145/html/index.html & 1. Check observation summary 2. Check flagging summaries 3. Check calibrated data If any suspicion, check relevant calibrations 4. Check flux consistency Search calibrator fluxes in ALMA database using au.getalmaflux

20 Observation Summary Click here Click here

21 Check flagging summaries Go to By Topic This page shows how much fraction of the data are flagged by pipeline per antenna.

22 Check calibrated data Go to By Task, click 15. hifa_applycal Click here If any suspicious feature, check relevant calibrations. e.g., strange bandpass shape -> check bandpass/tsys calibration Time variable phase -> check gain/wvr calibration

23 Check flux consistency Go to 13.hfa_fluxscale For this data, a quasar J was chosen as a flux calibrator. The pipeline obtained the flux mode of this calibrator from the measurement in ALMA database close in time. See 10.hifa_setjy for the flux model. This page shows the flux densities of other calibrators derived from the flux scaling using the model of J

24 Retrieve Calibrator Flux in database CASA <2>: au.getalmaflux(sourcename='j ',date=' ',frequency=' ghz') Using Band 3 measurement: (age=1 days) GHz Using Band 7 measurement: (age=2 days) GHz exact value: , 1-sigma extrema: , , mean unc= Median Monte-Carlo result for = (scaled MAD = ) Result using spectral index of for GHz from GHz = Jy Out[2]: {'agedifference': 1.0, 'fluxdensity': , 'fluxdensityuncertainty': , 'meanage': 1.5, 'montecarlofluxdensity': , 'spectralindex': , 'spectralindexuncertainty': }

25 Produce Calibrated MS Download tarred asdms Once untar them, asdms will be placed under raw/ directory Go to script/ directory and run sciprtforpi.py It can take a few - several hours to finish In this tutorial, we already created calibrated MSs and provide them. Please download from the tutorial webpage. Check the calibrated amplitude and phase as a function of time, freq., uv-distance for calibrators (and targets if SN is enough) using plotms For this data set, we see a dip in the spectrum of phase calibrator shown in weblog. Please confirm if you see the same feature in the calibrated data using plotms.

26 Check bad data calibrated]$ pwd /work/nagai/tutorial/ s/science_goal.uid A001_X12e_X27e/group.uid A001_X12e_X27f/member.u id A001_X12e_X280/calibrated calibrated]$ ls casa log products uid A002_Xa48b1f_X1b0.ms.split.cal working ipython log rawdata uid A002_Xaa96da_Xb77.ms.split.cal calibrated]$ casa CASA <9>: plotms(vis='uid A002_Xa48b1f_X1b0.ms.split.cal',xaxis='chan',yaxis='amp',avgtime='1e9',avgscan=T,avgantenna=T,spw='1',c oloraxis='ant1',field='2')

27 sciprtforpi.py vs. casa_pipescript.py Running scriptforpi.py reproduces the pipeline calibration done at JAO/ARCs. Rerun the pipeline calibration with additional flagging can be performed running casa_pipescript.py Copy casa_pipescript.py from the script directory to the raw directory Copy flux.csv and *flagtemplate.txt from the calibration directory to the raw directory

28 Introducing Additional Flagging Additional manual flagging can be introduced to any Pipeline reduction by editing the *flagtemplate.txt files under calibration/ directory. # User flagging commands file # # Examples # Note: Do not put spaces inside the reason string! # The data of DA56 in spw=17 was flagged by data reducer because of amplitude outlier. # mode=manual correlation=yy antenna=dv01;dv08;da43;da48&dv23 spw=21:1920~2880 autocorr=false reason='bad_channels' # mode=manual spw='25:0~3;122~127' reason='stage8_2' # mode=manual antenna='dv07' timerange='2013/01/31/08:09:55.248~2013/01/31/08:10:01.296' reason='quack' # mode=manual antenna='da56' spw='17' reason='timegaincal_outlier_amp' For this data, one may want to flag channels affected by the atmospheric absorption. Add following line: mode=manual spw= 19 channel= 34~38 reason= bad_channel Rerun casa_pipescript.py

29 Editing Calibrator Fluxes For some reason, you may want to redo the flux calibration Referenced calibrator flux is old Quasar which is used for the flux calibrator is time variable Check Calibrator database i) Edit flux.csv and rerun casa_pipescript.py or ii) give flux rescaling task in sciptforimagingprep.py and run it on the calibrated MSs. Note QA2 data reducers normally checked the flux consistency of calibrators and did the flux rescaling if necessary at EA-ARC. However, about 20% of EA data is processed at JAO. Those may not be checked as careful as what we do. The criteria to apply/un-apply flux rescaling could be different among ARCs.

30 Example of scriptforimagingprep.py setjy(vis = '../calibrated/uid A002_Xb0ebd1_X6594.ms.split.cal', standard = 'manual', field = 'J ', fluxdensity = [1.89, 0, 0, 0], spix = -0.43, reffreq = '233.0GHz') os.system('rm -rf../calibrated/uid A002_Xb0ebd1_X6594.ms.split.cal.ampli_inf') gaincal(vis = '../calibrated/uid A002_Xb0ebd1_X6594.ms.split.cal', caltable = '../calibrated/uid A002_Xb0ebd1_X6594.ms.split.cal.ampli_inf', field = 'J ', solint = 'inf', combine = 'scan', refant = 'DV16', gaintype = 'T', calmode = 'a') applycal(vis = '../calibrated/uid A002_Xb0ebd1_X6594.ms.split.cal', field = '0,3', # J ,NGC5064 gaintable = '../calibrated/uid A002_Xb0ebd1_X6594.ms.split.cal.ampli_inf', gainfield = '0', # J calwt = F, flagbackup = F) Give a correct flux density to the phase calibrator Run amplitude gain calibration using the given flux model Apply new gain calibration to the phase calibrator and target print "# Concatenating the data." concat(vis = ['../calibrated/uid A002_Xb020f7_X32f5.ms.split.cal', '../calibrated/uid A002_Xb0ebd1_X6594.ms.split.cal'], concatvis = '../calibrated/calibrated.ms')

31 Imaging Products Imaging products (FITS of primary-beam corrected image namely *.pbcor.fits and primary beam coverage namely *.flux.fits) are found under product/ directory product]$ pwd /work/nagai/tutorial/ s/science_goal.uid A001_X12e_X27e/group.uid A001_X12e_X27f/member.uid A001_X12e_ X280/product product]$ ls calibrated_final_cont.flux.fits SN1987A_SiO76R2.pbcor.fits calibrated_final_cont.mask uid A001_X12e_X280.J _bp.spw17.I.fits calibrated_final_cont.pbcor.fits uid A001_X12e_X280.J _bp.spw19.I.fits cont.mask uid A001_X12e_X280.J _bp.spw21.I.fits SN1987A_SiO76R1.flux.fits uid A001_X12e_X280.J _bp.spw23.I.fits SN1987A_SiO76R1.mask uid A001_X12e_X280.J _ph.spw17.I.fits SN1987A_SiO76R1.pbcor.fits uid A001_X12e_X280.J _ph.spw19.I.fits SN1987A_SiO76R2.flux.fits uid A001_X12e_X280.J _ph.spw21.I.fits SN1987A_SiO76R2.mask uid A001_X12e_X280.J _ph.spw23.I.fits Imaging script used for produce these products can be found under script/ directory (scriptforimaging.py).

32 Check Image Click here to zoom-up CASA <2>: viewer('calibrated_final_cont.pbcor.fits') CASA <4>: viewer('sn1987a_sio76r2.pbcor.fits') CH=0 CH=2 CH=4 Will explain how to display - beam size and other parameters - integrated spectrum in a given region - image statistics - source size - Gaussian model fitting

33 Parameter control in scriptforimaging.py scriptforimaging.py # imaging control # # The cleaning below is done interactively, so niter and threshold can # be controlled within clean. weighting = 'briggs' robust=0.5 niter=1000 threshold = '0.0mJy' ############################################# # Imaging the Continuuum # Set the ms and continuum image name. contvis = 'calibrated_final_cont.ms' contimagename = 'calibrated_final_cont' # If necessary, run the following commands to get rid of older clean # data. #clearcal(vis=contvis) #delmod(vis=contvis) for ext in ['.flux','.image','.mask','.model','.pbcor','.psf','.residual','.flux.pbcoverage']: rmtables(contimagename+ext) clean(vis=contvis, imagename=contimagename, field=field, mode='mfs', psfmode='clark', imsize = imsize, cell= cell, weighting = weighting, robust = robust, niter = niter, threshold = threshold, interactive = True, imagermode = imagermode) weighting = 'briggs' robust=0.5 niter=1000 threshold = '0.0mJy' Choice of robust parameter changes the beam size and image quality. Smaller value, higher resolution Larger value, less sidelobe (if the uv-coverage is very bad) But, optimal parameter always depends on the uvcoverage

34 Trial of different robust parameter execfile( scriptforcontimaging_rb2.0.py )

35 Want to do manual calibration? If you would like to do calibration manually instead of the pipeline, try Eric s script generator (developed by Eric Villard in JAO). Type es.generatereducscript( asdm_name ) on CASA term analysisutils should be installed. This will return a set of calibration tasks An example script uid A002_Xa48b1f_X1b0.ms.scriptForCalibration.py is found under calibrated/ directory