ALMA$Science$Pipeline$Guide:$CASA$4.5.1$

Transcription

1 Doc 3.13, ver. 1 January,2016 ALMASciencePipelineGuide:CASA4.5.1 InterferometryandSingle?DishData ALMA,aninternationalastronomyfacility,isapartnershipofESO(representingitsmemberstates),NSF(USA)andNINS (Japan),togetherwithNRC(Canada),NSCandASIAA(Taiwan),andKASI(RepublicofKorea),incooperationwiththe RepublicofChile.TheJointALMAObservatoryisoperatedbyESO,AUI/NRAOandNAOJ.

2 Forfurtherinformationortocommentonthisdocument,pleasecontactyourregionalHelpdeskthroughtheALMAUser Portalatwww.almascience.org.HelpdeskticketswillbedirectedtotheappropriateALMARegionalCenteratESO, NAOJorNRAO. Version Date Editors 1.0 January2016 PipelineTeam Addnamesandlogo sforanyadditionalcontributionsbeyondthealmaexecutives Inpublications,pleaserefertothisdocumentas: ALMAPipelineTeam,2016,ALMASciencePipelineQuickstartGuide,Version1,ALMA

3 TABLEOFCONTENTS 1 PURPOSE(...(3 1.1 NEW(SINCE(THE(PREVIOUS(VERSION(OF(THIS(DOCUMENT(...(3 2 OBTAINING(THE(PIPELINE(...(4 3 PIPELINE;RELATED(DOCUMENTATION(...(4 4 ALMA(INTERFEROMETRY(DATA(...(4 4.1 PRODUCING(CALIBRATED(MEASUREMENT(SETS(FROM(ALMA(PIPELINE(DELIVERIES:( SCRIPTFORPI.PY(...(4 4.2 THE(PIPELINE(INTERFEROMETRY(CALIBRATION(SCRIPT:(CASA_PIPESCRIPT.PY(...( Runningcasa_pipescript.pyfromALMADeliveryPackages Imagingafterusingcasa_pipescript.py THE(PIPELINE(MODIFYING(THE(PIPELINE(RUN(...( NoteonthePipelineMode RemovingASDMsfromtheprocessing Introducingadditionalflagging EditingcalibratorfluxesusedbythePipeline ManipulatingthePipelineContext THE(PIPELINE(WEBLOG(FOR(INTERFEROMETRY(DATA(...( Navigation HomePage ObservationSummary ByTopic ByTask Selectingandfilteringplotdisplays WeblogQualityAssessment(QA)Scoring

4 4.5 CASA(COMMANDS(LOG(...(15 5 ALMA(TOTAL(POWER((SINGLE(DISH)(DATA(...( THE(SINGLE(DISH(PIPELINE(CALIBRATION(SCRIPT:(CASA_PIPESCRIPT.PY(...( Runningcasa_pipescript.pyfromALMADeliveryPackages ManualImagingafterrunningcasa_pipescript.py MODIFYING(THE(PIPELINE(RUN(...( FlaggingAutoUcorrelationdata WEBLOG(QUALITY(ASSESSMENT(SCORING(FOR(SINGLE;DISH(DATA(...( NAVIGATION(OF(THE(WEBLOG(FOR(SINGLE;DISH(DATA(...( hsd_importdata hsd_flagdata hifa_tsyscal hifa_tsysflag hsd_mstoscantable hsd_inspectdata hsd_calsky hsd_applycal hsd_baseline hsd_blflag hsd_imaging

5 1 Purpose The ALMA Science Pipeline is being developed for the automated processing of ALMA interferometryandsingleudishdata,andforthecombinationofdatatakenusingmultiplealma arrays. At present, the Pipeline is commissioned for the calibration of the majority of ALMA interferometry datasets and for the provision of diagnostic calibrator images. It is also commissioned for the endutouend calibration and imaging of ALMA SingleUDish data acquired from Cycle 3 onwards. Data combination from multiple arrays and science target imaging for interferometric data are not yet commissioned. This document describes how to obtain the ALMAPipeline,howtouseittocalibrateALMAinterferometricandsingleUdishdataanduseof thepipelineweblog,thesetofplotsoutputbythepipeline. The Pipeline calibrates and images ALMA data automatically using Pipeline tasks. Pipeline tasksusecasataskswhereverpossibletoperformthedatareduction,andpipelinetaskscan beviewedandexecutedwithincasainexactlythesamewayascasatasks.forexample,in a CASA version also containing the Pipeline, to view the inputs possible for Pipeline task hifa_importdata type: inp hifa_importdata. To see all the tasks available in CASA type tasklist. The Pipeline is dataudriven i.e. the characteristics of each dataset drives the calibration and imaging strategy (the Pipeline Heuristics). During the Pipeline run, information on which calibrationtablesshouldbeusedetcarestoredinthepipelinecontext.boththeheuristics andthecontextareimplementedaspythonclasses. In order to determine if the Pipeline was used in the processing of an ALMA dataset, please consultthereadmefileinthedatadeliverypackage.someprojectsmaycontainamixofboth manuallyandplcalibrateddata. 1.1 NewSincethePreviousVersionofThisDocument There is now a Chapter on Pipeline Processing in the ALMA Cycle 3 Technical Handbook The Pipeline WebLog has a new structure on the By Task subupages when multiple executionblockshavebeenprocessed NewsectiononManipulatingthePipelineContext NewWebLogQuickTipsinformation NewsectionontheCASACommandsLogwrittenbythePipeline NewsectiononprocessingALMASingleUDishdatausingthePipeline 3

6 2 ObtainingthePipeline TheCycle3PipelineisreleasedaspartofCASA4.5.1.WhileearlierversionsofthePipeline neededspecialcasainstallations(forcasa4.2.2and4.3.1),thisisnolongernecessary:the Pipeline is a standard part of CASA CASA is available, along with installation instructions, from the Documents & Tools Section of the ALMA Science Portal: ( contactthealmahelpdeskviathelinkonthealmascienceportal.almacycle3datamust beprocessedusingcasa4.5orlater. 3 PipelineNrelatedDocumentation TheUserdocumentationcurrentlyrelatingtoPipeline,availablefromtheALMASciencePortal, is: ALMASciencePipelineQuickstartGuide:OverviewoftheALMASciencePipeline (thisdocument) ALMASciencePipelineReferenceManual:DescriptionofindividualPipelinetasks ALMATechnicalHandbook:Chapter13isdevotedtoPipelineProcessing ALMAQA2DataProducts:DescriptionoffilesincludedinALMAdeliveries 4 ALMAInterferometryData 4.1 ProducingCalibratedMeasurementSetsfromALMAPipeline Deliveries:scriptForPI.py IfthePipelinewasusedduringtheALMAQualityAssuranceProcess,thiswillbenotedinthe README file of an ALMA delivery. In order to convert ALMA raw data into calibrated Measurement Sets, for example before reuimaging the data, then the script scriptforpi.py should be used. scriptforpi.py directly applies calibration and flagging tables determined duringthealmaqualityassuranceprocesstorawdatatocreatecalibratedmeasurementsets. The instructions for using this script will be in the README file of the ALMA delivery. Using scriptforpi.py is the recommended and fastest method of obtaining calibrated ALMA datafromthedelivery.fordeliveriesthatusedthealmapipeline,scriptforpi.pymustbe runfromwithinaversionofcasathatcontainsthepipeline. scriptforpi.pyusesthescriptcasa_piperestorescript.py.casa_piperestorescript.pyshould not be executed directly/standalone at present. When data calibrated by the Pipeline are restoredusingscriptforpi.py,notethatadditionaldirectoriessuchasworkingandrawdataare created.thesedirectoriesandtheircontentsshouldbeignored. 4.2 ThePipelineInterferometryCalibrationScript:casa_pipescript.py To make any alterations to the calibration and flagging performed during the ALMA Quality Assurance Process, data have to be fully reucalibrated using the script casa_pipescript.py (insteadofusingscriptforpi.py).casa_pipescript.pyisapythonscriptforperformingfulldata 4

7 reucalibrationusingthepipeline(typicalscriptshowninfigure1).notethatthepipelineworks on completed Observing Unit Sets (defined in the ALMA Technical Handbook), such that multipleexecutionsofthesameschedulingblock(executionblocks,asdms)canbecalibrated using a single script to produce a calibrated Measurement Set per ASDM. Information on individual Pipeline tasks is in the ALMA Science Pipeline Reference Manual. The default processingrecipecanchangebetweendifferentpipelinereleases Runningcasa_pipescript.pyfromALMADeliveryPackages ToreproducethePipelinecalibrationperformedaspartoftheALMAQualityAssurance Process: Copycasa_pipescript.pyfromthescriptdirectorytotherawdirectory Copyflux.csvand*flagtemplate.txtandantennapos.csv(ifpresent)fromthe calibrationdirectorytotherawdirectory Intherawdirectory: MakesurethenamingoftherawALMAdataisconsistentwiththoseprovidedinthe script(e.g.ifthedataendsin.asdm.sdmthenmovetonameswhichdonothavethis suffix) StarttheversionofCASAcontainingPipelineusingcasapyNNpipeline,thentype execfile( casa_pipescript.py ) Runningthescriptwillcreate: acalibratedmsforeachasdminthesamedirectory testimagesofthebandpassandphasecalibrator(1perspectralwindowin*.image format.toviewa*.imagefilee.g.usecasaviewerimage_file_name) apipeline*/htmldirectorycontaining o thepipelinewebloguthehtmlpagesofplotsfromthecalibrationprocess. Accessusinge.g.firefoxindex.html o casa_commands.log alistofthecasatasksusedbypipelinetasksduring calibration NotethattoreNrunthePipelinemultipletimes,itisrecommendedtostarteachtimefrom acleandirectorycontainingonlytherawdata,flux.csv,antennapos.csvfile(ifpresent) and *flagtemplate.txt files and the casa_pipescript.py script. If the flux.csv and *flagtemplate.txtfilesarenotpresentinthedirectory,pipelinewillcreatenewdefaultversions ofthesefiles,whichwillnotcontainanyeditsmadetothembyalmastaffduringthequality Assurance Process. Also note that it is advisable to have 8U10 GB RAM and 50U75 GB disk space per ALMA raw data file (ASDM) available to perform the Pipeline calibration. Please contactalmaviathehelpdeskifassistanceisneededwithdatareprocessing Imagingafterusingcasa_pipescript.py Inordertousetheimagingscript(scriptForImaging.py)providedbytheALMAObservatoryafter using casa_pipescript.py, then the science spectral windows must be split out from the calibratedmeasurementsetsandthemeasurementsetsoutputwitha.split.calsuffix. 5

8 rethrow_casa_exceptions = True h_init() try: hifa_importdata(dbservice=false, vis=['uid A002_X877e41_X452'], session=['session_1']) finally: hifa_flagdata(pipelinemode=automatic) hifa_fluxcalflag(pipelinemode=automatic) hif_rawflagchans(pipelinemode=automatic) hif_refant(pipelinemode=automatic) hifa_tsyscal(pipelinemode=automatic) hifa_tsysflag(pipelinemode=automatic) hifa_antpos(pipelinemode= automatic ) hifa_wvrgcalflag(pipelinemode=automatic) hif_lowgainflag(pipelinemode=automatic) hif_gainflag(pipelinemode= automatic ) hif_setjy(pipelinemode=automatic) hifa_bandpass(pipelinemode=automatic) hifa_spwphaseup(pipelinemode=automatic) hifa_gfluxscale(pipelinemode=automatic) hifa_timegaincal(pipelinemode=automatic) hif_applycal(pipelinemode=automatic) hif_makeimlist(intent='phase,bandpass,check') hif_makeimages(tlimit=2.0) hif_exportdata(pipelinemode=automatic) h_save() Figure1:ExamplePipelinecalibrationscriptcasa_pipescript.py 6

9 ThesciencespectralwindowsarespecifiedinthePipelineWeblog(Home>Observation Summary>MeasurementSetName>SpectralSetup,intheIDcolumn)orcanbedetermined usingthecasatasklistobse.g.listobs( uid A002_X89252c_X852.ms )andresultswillbein thecasalogger.toperformthesplit,incasae.g.: split('uid A002_X89252c_X852.ms',outputvis='uid A002_X89252c_X852.ms.split.cal',spw='1 7,19,21,23') IfthescriptnamedscriptForFluxCalibration.pyispresentinthescriptdirectory,thismustalso thenbeexecutedpriortorunningtheimagingscript. 4.3 ThePipelineModifyingthePipelineRun (NoteonthePipelineMode InthePipelinestandardcalibrationscript,thePipelineModeisdefinedas automatic foreach task.inthismode,thetasktakesthedefaultsettingsofpipelineandonlyalimitednumberof parameters are exposed for editing by a user. Setting the pipeline mode to interactive will usually enable the values of a larger number of parameters to be changed. See the ALMA SciencePipelineReferenceManualformoredetails (RemovingASDMsfromtheprocessing ASDMscanberemovedfromtheprocessingbyeditingthevis=andsession=listsin hifa_importdataincasa_pipescript.py Introducingadditionalflagging Additional manual flagging can be introduced to any Pipeline reduction by editing the *flagtemplate.txtfiles.examplesofthesyntaxtouseineditingthesefilesaregivenatthetop ofthefiles.theflagsareappliedwhenhifa_flagdataisrun (EditingcalibratorfluxesusedbythePipeline Pipelinecurrentlyreadsquasarcalibratorfluxesfromtheflux.csvfile.IfaregularlyUmonitored quasar has been used as the flux calibrator, the flux of this provided from the ALMA Quality AssuranceprocesscanbeoverUriddenbyeditingitinthisfile.Onlyvaluesforthefluxcalibrator canbeoveruridden,sincevaluesforthebandpassandphasecalibratorsarederivedduringdata calibration (ManipulatingthePipelineContext ItisrecommendedtoalwaysrunthePipelineusingpythonscripts.NewPipelineruns/scripts needtobeinitialisedusingh_initinordertocreateanemptycontext. IfitiswantedtohaltaPipelinerunbeforetheendofprocessing,theContextshouldbesaved atthatpointusingh_save.inordertoresumetherun,useh_resumetoloadthesaved Contextbeforeresumingtherun.SeetheALMASciencePipelineReferenceManualfor moreinformation. 7

10 IfitiswantedtomainlyusethePipelinetocalibrateadatasetbuttoe.g.insertadifferent bandpasstableintotheprocessing,thefollowingprocedureshouldbefollowed: 1. Runthepipelineuntiltheendofthebandpasstablecreationtaskhifa_bandpass 2. ViewthecalibrationtablesthatPipelinewillusewithhif_show_calstate 3. ExportthecalibrationtablesPipelineusestoafileondiskusinghif_export_calstate 4. EditthefiletoreplacethenameofthePipelineUcreatedbandpasstablewiththeoneitis wantedtouseinstead 5. ImportthefilebacktotheContextusinghif_import_calstateandresumethe processing 4.4 ThePipelineWebLogforInterferometryData TheWebLogisasetofhtmlpagesthatgiveasummaryofhowthecalibrationofALMAdata proceeded.theweblogproducedduringthealmaqualityassuranceprocesswillbeinthe qadirectoryofanalmadelivery.toviewtheweblog,untarandunzipthefileusinge.g.tar zxvf*weblog.tar.gz.thiswillprovideapipeline*/htmldirectorycontainingtheweblog,which canbeviewedusingawebbrowsere.g.firefoxindex.html.theweblogaimstoprovideboth aquickoverviewofdatasetsandalsogivemethodsforexploringthecalibrationandflaggingin detail.thereforemostcalibrationpagesoftheweblogwillfirstgiveasingle representative view of that calibration (or flagging) step. Users can then choose whether to further select a detailed view of all the plots associated with that calibration step. These plots can easily be filtered by a combination of outlier, antenna and spectral window criteria. Calibration and flagging steps also undergo an automated scoring to give an at a glance indication of any troublepoints.throughouttheweblog,linksaredenotedbytextwritteninblueanditis usuallypossibletoclickonthumbnailplotstoenlargethem.wherehistogramsaredisplayed,in modern web browsers it is possible to draw boxes on multiple histograms to select the plots associatedwiththosedatapoints Navigation To navigate the main pages of the WebLog, click on items given in the bar at the top of the WebLog home page. Also use the Back button provided at the upper right on some of the WebLogsubUpages.Avoidusing back/previouspage onyourwebbrowser(althoughthiscan workonmodernbrowsers). 8

11 4.4.2 HomePage ThefirstpageintheWebLoggivesanoverviewoftheobservations(proposalcode,datacodes, PI, observation start and end time), a pipeline execution summary, and an observation summary.clickingonthebaratthetopofthehomepageenablesnavigationtobytopicorby Task. WebLogQuickTips Anytextwritteninblue,includingheadings,isalinktofurtherinformation Togostraighttoviewingcalibratedsciencetargetplots,gotoByTask>hif_applycal andscrolldowntothebottom Histogramscanhaveselectorboxesdrawnonthemusingthemouse CommandsforreUcreatingsomeWebLogplotsinCASAarenowprovided ObservationSummary The Observation Summary table lists all the measurement sets included in the pipeline processing. Each measurement set is calibrated independently by the pipeline. The table provides a quick overview of the ALMA receiver band used, the number of antennas, the start/enddateandtime,thetimespentonsource,thearrayminimumandmaximumbaseline length,thermsbaselinelengthandthesizeofthatmeasurementset.toviewtheobservational setupofeachmeasurementsetinmoredetail,clickonthenameofittogotoitsoverviewpage. Figure2:WebLogHomePage.TheNavigationBariscircledinred Measurementsetoverviewpages Eachmeasurementsetoverviewpagehasanumberoftables:ObservationExecutionTime, SpatialSetup(includesmosaicpointings),AntennaSetup,SpectralSetupandSkySetup (includeselevationvs.timeplot).formoreinformationonthetablestitledinbluetext,click on these links. There are additionally links to Weather, PWV and Scans information. Two thumbnail plots, which can be enlarged by clicking on them, show the observation structure 9

12 eitherasfieldsourceintentvstimeorfieldsourceidvstime.toviewthecasalistobs outputfromtheobservationclickonlistobsoutput. Figure3:AMeasurementSetOverviewPage.Clickonthetableheadingsinbluetosee moreinformationabouteach ByTopic The By Topic page provides an overview of all Warnings and Errors triggered, a Quality AssessmentoverviewinTasksbyTopicandFlaggingSummariesfortheprocessing ByTask TheByTaskSummarypagegivesalistofthecalibrationandflaggingstepsperformedonthe dataset.itisnotdisplayedpermeasurementsetasthepipelineperformseachsteponevery measurementsetsequentiallybeforeproceedingtothenextstepe.g.itwillimportandregister all measurement sets with the Pipeline before proceeding to perform the ALMA deterministic flaggingsteponeachmeasurementset.thenameofeachsteponthebytaskpageisalinkto moreinformation.ontherighthandsideofthepage,thebarsandscoresindicatehowwellthe Pipeline processing of that stage went. All green bars should indicate a fairly problemufree dataset.symbolsnexttothetasks,suchas?,aregivenonthebasisofthepipelineqascore TasksubNpages ThetasksubUpages(accessedbyclickingone.g.hifa_importdata)providetheoutcome,orthe representative outcome, of each Pipeline task executed. Additionally by clicking on the appropriatebluelinksoneachsubupage,thecasalog,pipelineqa,inputparametersand 10

13 TaskExecutionTimescanbeaccessed.MostsubUpageshavefurtherlinksinordertoaccess a more detailed view of the outcome of each task. These links are often labelled by the measurement set name. For a fast assessment of results, go straight to the By Task > hif_applycalpage Selectingandfilteringplotdisplays UsingtheByTask>hifa_tsysflag:FlagTsyscalibrationpagesasanexample. Figure5:FlagTsysCalibrationOverviewPage.ArepresentativeviewoftheTsysforeachspectral windowisshown.toseethemeasurementsforindividualantenna/spectralwindows,clickonthe measurementsetname,circledhereinred. 11

14 Figure6:TheTsysplotsforallantennas.Abovetheplotsarehistogramsindicatingthe distributionsofmetricsbasedonthemediantsys. Figure7:Filteringtotheplotofinterestbydrawinggreyboxesontwoofthehistogramsusinga mousebutton,andaddingadditionalspectralwindowandantennafilters.toclearthegreybox filtersonthehistograms,clickonwhitespaceinthehistograms WeblogQualityAssessment(QA)Scoring Pipelinetaskshavescoresassociatedwiththeminordertoquantifythequalityofthedataset andthecalibration.thescoresarebetween0.0and1.0andarecolourizedsuchthat: 12

15 Score Colour Comment 0.90U1.00 Green Standard/Good 0.66U0.90 Blue Belowstandard 0.33U0.66 Yellow Warning 0.00U0.33 Red Error PipelineTask hifa_importdata PipelineQAScoring Metric Checkingthatthe requiredcalibratorsare present Score 1.0allpresent 0.1subtractedformissingbandpassorflux calibrator 1.0subtractedformissingphasecalibratoror Tsyscalibration 0.5subtractedforexistingprocessinghistory hifa_flagdata hifa_fluxcalflag hif_rawflagchans hif_refant hifa_tsysflag hifa_antpos Determiningpercentage ofincrementalflagging Determiningpercentage ofincrementalflagging Determiningpercentage ofdataflaggeddueto deviantchannelsin rawdata Determiningifa referenceantennacould befound Determiningpercentage ofincrementalflagging Determiningfractionof antennaswithout positioncorrections 0<score<1===60<fractionflagged<5 (for'online','shadow','qa0','before'and 'applycal')where0<score<1===high< fractionflagged<lowmeans Scoreis0ifflagfractionis>=HIGH Scoreis1ifflagfractionis<=LOW Scoreislinearlyinterpolatedbetween0and1 forfractionsbetweenhighandlow 0U5U>1.0,5U50U> ,>50U> 0.0 0U>1.0,100U> ifreferenceantennacouldbedetermined, 0.0otherwise 0U5U>1.0,5U50U> ,>50U> ifnocorrectionsneededs0.0ifallantennas corrected.scoreisequaltothefractionof antennaswithnooffsets 13

16 14 hifa_wvrgcalflag CheckingphaseRMS improvement 0.0ifRMS(before)/RMS(after)<1, forratiosbetween1and2,and1.0forratios> 2 hif_lowgainflag Determiningpercentage ofincrementalflagging 0U5U>1.0,5U50U> ,>50U> 0.0 hif_gainflag Determiningpercentage ofincrementalflagging Score1:0U5U>1.0,5U50U> , >50U>0.0 Score2:ifaflaggingviewcouldbecreated then1.0,otherwise0.0 hifa_bandpass Judgingphaseand amplitudesolution flatnessperantenna, spectralwindowand polarization(interim measure,futurescoring willbebasedondata withsolutionsapplied) twoalgorithms:wienerentropyandderivative deviation,andsignalutounoiseratio(scores: Wienerentropy:errorfunctionwith1Usigma deviationof0.001from1.0sderivative deviation:errorfunctionwith1usigmadeviation of0.03fortheoutlierfractionssignalutounoise ratio:errorfunctionwith1usigmadeviationof 1.0forthesignalUtoUnoiseratio) hifa_spwphaseup Determiningfractionof spectralwindowswithout phasesolutions transferredfromother windows Scoreisthefractionofspectralwindowsfor whichphasesolutionsareunmappedto expectednumberofspectralwindows hifa_gfluxscale DeterminingSNRof fittedfluxvalues FittedfluxvalueswithSNR<5.0areassigned ascoreof0.0,snr>20.0ascoreof1.0,and alinearlyscaledvalueinbetween hifa_timegaincal DeterminingXUY/X2UX1 phasesolution deviations StandarddeviationofXUYphasedifference convertedtopathlength:1.0iflowerthan 4.25eU6m,0.0ifhigherthan7.955eU2m,with anexponentialdecreaseinbetween.standard deviationofx2ux1phasedifferencesof subsequentintegrationsconvertedtopath length:1.0iflowerthan3.08ueu5m,0.0if higherthan2.24eu2m,withanexponential decreaseinbetween.nb:thehighlimitsare currentlydummies.determiningtheirrealistic valuesisstillunderdevelopment. hifa_applycal Determiningpercentage ofincrementalflagging 0U5U>1.0,5U50U> ,>50U> 0.0 hif_exportdata DeterminingPipeline productshavebeen exported 1.0whenfilessuccessfullyexported

17 4.5 CASACommandsLog TheCASACommandsLogiswrittenbythePipelineandisprovidedaspartofanALMA delivery.thisisnotthesameasthecasapylog.thecasacommandslogprovidesallofthe CASAtaskcommands(asopposedtoPipelinetasks)usedbythePipelinetoprocessadataset. WhilethislogcannotbeusedtocreateaCASAreductionscriptthatisequivalenttothePipeline processing,itisveryusefulindeedforseeingwhatcasacommandsandparametersettingsin thesecommandswereused.thelogiscommentedtoindicatewhichpipelinestagethetasks werecalledfromandwhy. 5 ALMATotalPower(SingleDish)Data ThissectionexplainshowtousethePipelineforALMATotalPower(TP)dataandhowto navigatethepipelineweblog. ForTPdata,thereiscurrentlynoscripttorestorerawALMAdatatocalibratedmeasurement sets(mses)bydirectlyapplyingtheobservatoryudeterminedcalibrationtables.instead,for ALMATPdatathathavebeencalibratedandimagedbytheSingleDishPipeline,when scriptforpi.pyisused,itwillcallcasa_pipescript.pytoperformafullreucalibrationofthedata (seesecion5.1).pleasenotethatprocessingmaytakeasignificantamountoftimetoreu calibratethetpdata.toknowhowlongitwilltake,seethe ExecutionDuration whichis shownonthetopupageoftheweblog. Notethattheimagesincludedinadeliverypacakgehaveanativefrequencyresolutionanda cellsizeofoneninthsofbeamsize,asthecasaguidesrecomend.ifyoujustwanttochange themtoyourpreferablefrequencyresolutionandcellsize,werecommendyoutoimportthe deliveredfitscubestocasaandregriditusingthecasataskimregrid.ifyouwantto performbaselinesubtractionusingyourpreferablemaskrangeratherthanthepipelineudefined range,werecommendtodoitontheimagesusingthecasataskimcontsuborrecommendto doitusingthecasatasktsdbaselineduringyourownmanualcalibration(refertothecasa Guides). 5.1 TheSingleDishPipelineCalibrationScript:casa_pipescript.py ThescriptForPI.pycallscasa_pipescript.py.AtypicalscriptforprocessingCycle3datais showninfigure8. Figure8:ExampleSingleDishPipelinecalibrationscriptcasa_pipescript.py rethrow_casa_exceptions=true h_init() hsd_importdata(vis = ['uid A002_X877e41_X452']) hsd_flagdata(pipelinemode= automatic ) hifa_tsyscal(pipelinemode= automatic ) 15

18 hifa_tsysflag(pipelinemode= automatic ) hsd_mstoscantable(pipelinemode= automatic ) hsd_inspectdata(pipelinemode= automatic ) hsd_calsky(pipelinemode= automatic ) hsd_applycal(pipelinemode= automatic ) hsd_baseline(pipelinemode= automatic ) hsd_blflag(pipelinemode= automatic ) hsd_baseline(pipelinemode= automatic ) hsd_blflag(pipelinemode= automatic ) hsd_imaging(pipelinemode= automatic ) hsd_exportdata(pipelinemode= automatic ) h_save() AlthoughCASA4.5.1PipelineisoptimizedtoprocessCycle3data,incasethatthedata ofearliercyclethan3isprocessedinpipeline,userswillbeawarethatthereisone additionalcommand, hsd_simplescale(pipelinemode= automatic ) afterthe hsd_applycalcommandincasa_pipescript.py Runningcasa_pipescript.pyfromALMADeliveryPackages ForTPdata,theeasiestwaytoreproducethePipelinecalibrationfromALMADelivery PackagesistorunscriptForPI.py.Alternativelythefollowingstepscanbeperformedmanually butitisessentiallythesame. ToreproducetheSingleDishPipelinecalibrationperformedaspartoftheALMAQuality AssuranceProcess: Copycasa_pipescript.pyfromthescriptdirectorytotherawdirectory Copyjyperk.csvand*flagtemplate.txtfromcalibrationdirectorytotherawdirectory Intherawdirectory: MakesurethenamingoftherawALMAdataisconsistentwiththoseprovidedinthe script(e.g.ifthedataendsin.asdm.sdm thenmovetonameswhichdonothave thissuffix) Start the version of CASA containing Pipeline using casapy NNpipeline, then type execfile( casa_pipescript.py ) 16

19 Runningthescriptwillcreate: AcalibratedMSperantennaforeachASDMinthesamedirectory ApipelineN*/htmldirectorycontaining o ThePipelineWebLog o TheCASACommandsLog ManualImagingafterrunningcasa_pipescript.py Aftercalibratingwithcasa_pipescript.py,itispossibletoreUimageusingtheCASASingleDish task,sdimaging,withuserudefinedparameters.singledishpipelinecreatesacalibratedms perantennaforeachasdm,whicharetobeselectedasinfilesparameterinsdimaging command.forotherparametersinsdimaging,refertoscript/*casa_commands.logunderthe deliveredpackage. 5.2 ModifyingthePipelineRun FlaggingAutoNcorrelationdata Itispossibletoinsertmanually(inatextfile,flagtemplate.txt)additionalflaggingnotidentifiedby pipelineifnecessary.pleasepayattentionthatyouneedtoadd &&* (whichmeanscrossuand autoucorrelation)or &&& (whichmeansautoucorrelation)in antenna keywordwiththenameof theantennatoflagabadantenna.youcanfindanexampleinfigure9. Figure9:Weblogviewofhsd_flagdatashowinganexampleofflagtemplate.txt( 5.3 WebLogQualityAssessmentScoringforSingleNDishData PipelineTask PipelineQAScoring Metric Score hsd_importdata Checkingthattherequired calibratorsarepresent 1.0ATMOSPHEREintentsare present 0.5subtractedforexistingprocessing history 0.5subtractedforexistingmodeldata 1.0onecontinuousobservingsession 17

20 18 1.0allsourcecoordinatearepresent hsd_flagdata Determiningpercentageof incrementalflagging 0<score<1===60<fraction flagged<5(for'online','shadow', 'qa0','before'and'applycal') where0<score<1===high< fractionflagged<lowmeans Scoreis0ifflagfraction>=HIGH Scoreis1ifflagfraction<=LOW Scoreislinearlyinterpolatedbetween 0and1forfractionsbetweenHIGH andlow hsd_baseline Checkingthatoneormore thanoneemissionlineis detectedbylineufinder. 1.0thereismorethanoneemission linedetectedinatleastonespw. 0.0Nolineisdetectedinallspw. hsd_blflag Determiningpercentageof incrementalflagging 1.0<5flagged U50flagged 0.0>50flagged hsd_imaging CheckingthatallKelvinUtoU Jyconversionfactorsare provided 0.0MissingKelvinUtoUJyconversion factorforsomedata hsd_exportdata Checkingthattherequired filesareexported 1.0pipelineprocessingrequestfileis exported 1.0pipelineweblogfileisexported 1.0pipelinescriptfileisexported 1.0pipelinecommandslogfileis exported 1.0Nomissingfinalflagversionfiles 1.0Nomissingfinalapplycommands files 1.0Nomissingcaltablesfiles

21 5.4 NavigationoftheWebLogforSingleNDishdata ThissectiondescribesnavigationoftheTasksubUpagesforeachSingleDishPipelinetask startingfrom ByTask tab.generalexplanationfor Home, ObservationSummary, By Topic and ByTask pages,refertosection4.4.fortaskdescription,referto ALMAScience PipelineReferenceManual hsd_importdata Theweblogforhsd_importdatataskshowsthesummaryofimportedMSes,andalsoflux densitiesofcalibrators(forpointing,sidebandratio,orwvr).singledishpipelineneverusethe valuesimportedhere hsd_flagdata Theweblogforhsd_flagdatashowssummaryofpercentageofflaggeddataperMSdueto binarydataandonlineflagging,manuallyinsertedtemplate,shadow,unwantedintents,and edgeofchannel.notethatthevaluein BeforeTask columncorrespondstothepercentageof flaggeddatabybinarydataflagging,whichisappliedwhenimportingasdms(at hsd_importdata) hifa_tsyscal Theweblogforhifa_tsyscaltaskshowsalistofpairofTsysandsciencespectralwindowstobe usedfortsys(amplitudeuscale)correction,andalsoshowanoriginaltsysspectraperspectral window hifa_tsysflag Theweblogforhifa_tsysflagtaskshowsflaggedTsysspectraperspectralwindowafterheuristic flaggingisapplied hsd_mstoscantable Theweblogforhsd_mstoscantabletaskshowsatableofMSanditsconvertedScantables(with ansuffixof.asap ).NewlygeneratedTsystables(perantennaperspectralwindow)atthis stagearealsoshownwiththelistoftheiroriginaltsystablefromhifa_tsyscal hsd_inspectdata Theweblogforhsd_inspectdatashowsplotsofTelescopePointing,AzimuthandElevation, Weather,andWVR.Bothflaggedandunflaggeddataaredisplayedindifferentsymboland color:i.e.,flaggeddataisalwaysshownasgraycolordots. NoteonWVRPlots TheplotsshowthepowerdirectlyreadfromtheWVRreceivervstime. 19

22 Figure10:thedetailedpageofTelescopePointingforhsd_inspectdata.Twotypesof plotsappearinthehsd_inspectdataweblogpageforeachantenna.oneisfortelescope pointingincludingonandoffsourcepointings,whiletheotherisonlyforonsource pointings(alsoseefigure11).thesetwotypesofplotscanbefilteredbychoosing full pointing or onsourcepointing. Figure11:Anexampleof ONsourcepointing plot.theredcircleindicatesthebeam sizeoftheantennasanditslocationisthestartingpositionoftherasterscan.thered (small)dotindicatesthelastpositionoftheraster.thegreenlineisthemotionofthe antenna,andthus,thegreenlinefromthetopleftcornertothebottomrightcorner indicatesthattheantennagoestothelastscanandreturnstothestartposition hsd_calsky Theweblogforhsd_calskyshowstheintegratedOFFspectraperspectralwindow(Figure12). AtthetopoftheplotsthecoordinatesoftheOFFpositionareshown.TheyUaxisisthedirect outputfromthecorrelator,whichmeansthevaluesaredominatedbysignalsfromboththe atmosphereandreceivers. 20

23 Figure12:AnexampleofOFFspectra hsd_applycal ItshowsalistofthecalibratedScantableswithanameoftheappliedTsysandSkytablesper antennaandperspectralwindow hsd_baseline Theweblogofhsd_baselineshowsthespectralmaps(whichsometimeswecall sparseprofile map )beforeandafterbaselinesubtraction,andplotsforlinedetectionbyclusteringanalysis. Inthesummarypage,thespectralmapsforbeforeandafterbaselinesubtraction(topand bottomlinesinfigure13,respectively).thedetailpagehasseveralsimilarplotsbutforeach EB,antenna,andpolarization.Notethatthe*representative*spectralmapforeachspectral windowinthesummarypagewasrandomlychosenoutofallspectralmapsinthedetailpages. NoteonR.A.vsDec.plotsinLineDetectionbyClusteringAnalysis Therearefourdifferentplotsperspectralwindow,i.e. clustering_detection, clustering_validation, clustering_smoothing,and clustering_final (Figure15).Thenumber ofplotsineachfigureisthesameasthatofcandidatelinecomponents. ThelinefindingalgorithmfirstintegratesspectrumperXYUgrid(perbeamsize)perthreetimeU domaingroups(i.e.pervoxel),andsearchesforemissionlinecandidateshavingabrightness aboveacertainthreshold.the cluster_detection plot(figure16)showsthegridcellshaving emissionlineexceedingthethreshold.thenthealgorithmcalculateshowmanyspectrawith emissionlinesareineachvoxel,andjudgeswhetherthegridcellpossiblycontainstrue emissionlines.atthislinedetectionvalidationstep,theratioofthenumberofspectrahaving detectedemissionlines(definedas Nmember )pergridcellandthenumberoftotalspectra belongingtothegridcell( Nspectra )iscalculated.theratio,nmember/nspectra,isshownin the clustering_validation plot(seefigure17).thegridcellismarkedas1) Validated if Nmember/Nspectra>0.5(Bluesquares),2) Marginallyvalidated ifnmember/nspectra>0.3 (Cyansquares),and3) Questionable ifnmember/nspectra>0.2(yellowsquares). Afterthevalidationstep,thegridcontainingtheNmember/Nspectraratiopergridcellis smoothedbyagaussianulikegridfunction.thisistoeliminatetheisolatedgridcellshaving emissionlinecandidate,whiletoenhancethegridcellshavingemissionlinethatcontinuesto theneighboringgridcells.figure18showsanexampleof clustering_smoothing,displayinga 21

24 mapofsmoothednmember/nspectraratio. Finally,themaskchannelrangeisestimatedperspectrum.Inthevalidatedareathrough validationandsmoothingstep(bluesquaresinfigure18,andgreensquaresinfigure20), emissionulinechannels(mask)arecalculatedoverthespatialdomain,extrapolatedper spectrumandappliedtothebaselinesubtraction.themaskchannelrangeisdeterminedand usedinbaselinesubtractionalsointhegreensquaresinfigure20( clustering_final ).The maskrangeissetasthesameastheoneofthespectraatthenearestedgeofvalidatedarea: i.e.aspectraataredcircleinthestraightlinebetweenthecenterofabluesquareandthe centerofthenearestgreensquareinfigure19. ThedeterminedlineUwidthandlineUcenteroftheidentifiedemissionsareanalyzedbyclustering analysis(figure21).theneachcombinationofclustersisscoredbasedontheclustersizeon the linecenter U linewidth space.thescoringisempiricallydefinedsothatthescoregets better(smaller)when[1]theclustersizeissmaller,[2]thenumberofclustersissmaller,and[3] thenumberofoutliersisfewerthanthoseofotherclusters.theuserswillknowwhichnumberof clustersismoreplausiblebysearchingforthenumberofclusterswiththelowestscore.this plotisbasicallyfordevelopers. Figure13:Thesummarypageofhsd_baseline. 22

25 Figure14:Anexampleofaplotofaveragedspectrumandspectralmap.Onthetopofthe spectralmap,aspatiallyaveragedspectrumpereb,antenna,spectralwindowand polarizationisshown.thecyanfilledregionsindicatethemaskchannelscontaining identifiedemissionline.theredthickbarindicatesadditionallymaskedchannelsto avoidfromfittingunwantedfeaturessuchaslinesatthebandedgeandatmospheric absorptionlines.onthebottom,eachsmallpanelshows*one*representativespectrum pergridcell.theredlinesineachsmallpanelindicatethefittedbaselineintheplotsof BeforeBaselineSubtraction,andthezeroNlevelintheplotsof AfterBaseline Subtraction,respectively. Figure15:Detailspageofresultsofclusteringanalysis(hsd_baseline) 23

26 Figure16:Exampleof clustering_detection plot.theyellowgridcellsshowaregion thatthereisone(pipelinendefined)timendomaingrouphavingdetectedemissionlines. CyansquaresindicateagridcellthattherearemorethanonetimeNdomaingroups havingdetectedemissionlines. Figure17:Exampleof clustering_validation.bluesquaresrepresentagridcellwith Nmember/Nspectra>0.5(Validated). Figure18:Exampleof clustering_smoothing.bluesquaresrepresentthegridcellswith pointsexceedingadefinedthreshold,i.e.,thegridcellshavingpromisingemissionlines thathavesmoothlydistributedtotheneighboringgridcells.cyanandyellowsquaresare thegridcellswithpointsslightlybelowthethreshold(border),andlowerthanthreshold (Questionable),respectively. 24

27 Figure19:Inthebluesquares,themaskchannelrangeisusedoftheoneforthe spectrumatthenearestedgeofvalidatedarea(whitenfilledredcircle).greensquares indicatefinalgridcellswherethemaskedemissionlinechannelsarecalculatedand appliedtothebaselinesubtraction.bluesquaresrepresentfinalgridcellswherethe calculatedlineprotectionchannelsareappliedtothebaselinesubtraction. Figure20:Anexampleof clustering_final.colorindicationsarethesameasfigure19. Figure21:Exampleoflinedetectionclusteringinthelinewidthvs.linecenterparameter space.thesmalldotsindicatespectracontainingidentifiedemissionline.theredovals showeachclusteringregionwithasizeoftheclusterradius. 25

28 hsd_blflag Theweblogforhsd_blflagtaskshowssummaryofpercentageofflaggeddataperMSper antennaperspwperpolarization.notethat Online columnrepresentsthepercentageof flaggeddatabeforethehsd_blflagtask.thepercentageofflaggeddatabythistaskareshown inthe BaselineRMS, Runningmean and ExpectedRMS column. Therearesevenfiguresbyclickingthe details page.theyareplotsof(1)tsys(figure22),(2) baselinermsbeforebaselinesubtraction,(3)baselinermsafterbaselinesubtraction(figure 23),(4)deviationofmovingaverageofbaselineRMSbeforebaselinesubtraction,(5)deviation ofmovingaverageofbaselinermsafterbaselinesubtraction,(6)baselinermsbefore baselinesubtractionwithexpectedrms,and(7)baselinermsbeforebaselinesubtractionwith expectedrmsasafunctionoftimeseriesspectrum.forthedetailedderivationofthese parameters,refertothealmapipelinereferencemanual. Figure22:AnexampleofTsysplot.Theblue,redandgraydotsrepresentdatapoints, outliers,andpreviouslyflaggeddatarespectively.cyanlineisathreshold. Figure23:AnexampleofbaselineRMSplot,before(toppanel)andafter(bottompanel) thebaselinesubtraction.thesymbolsandlinesarethesameasinfigure22. 26

29 hsd_imaging Thesummarypageforhsd_imagingshowsalistoftheappliedKelvinUtoUJyconversionfactors, spectralmaps(figure24),channelmaps(figure25),baselinermsmap,andintegrated intensitymap. NoteonSpectralMaps(ProfileMap) Threetypesofspectralmapsareavailableintheweblogforhsd_imaging(Figure24):1)a simplifiedprofilemapofthecombinedimageperspectralwindowatthetop,2)asimplified profilemapperantenna,and3)adetailedprofilemap.toaccessthesimplifiedprofilemapper antenna,clickthecorresponding SpectralWindow.Eachspectrumof(1and2above)the simplifiedprofilemapcorrespondstoanaveragedspectruminareawhichcorrespondstooneu ninthsoftheimagesize(imsize),sothatthetotalnumberofspectraintheprofilemapis8times 8(Ifthenumberofpixels(alongxUoryUaxis)islessthaneight,itshowsallspectrumperpixel.) Toseethedetailedprofilemaps,clicktheiconwithasymbolofpolarizationinthepolarization column(seefigure24).eachbinoftheprofilemapisequivalenttospectrumperpixel,butwith anintervalofthreecells.duetothelimitationoftheavailablenumberofplotspossibleper page(max5times5plotsinapage),therestoftheplotsisdisplayedinotherpages. NoteonChannelMaps Inthedetailpage,thenumberofchannelmapscorrespondstothenumberofemissionlines thathavebeenidentifiedbytheclusteringanalysis. NoteonBaselineRMSMap TheBaselineRMSMapiscreatedusingbaselineRMSstoredinthebaselinetables.The baselinermsiscalculatedbyhsd_baselineusingemissionfreechannels. NoteonIntegratedIntensityMap TheIntegratedIntensityMapforeachspectralwindowisgeneratedusingimmomentstask usingallavailablechannelrange. Figure24:Summarypageofhsd_imaging 27

30 Figure25:Anexampleofchannelmapperdetectedline.Topmiddle)theidentified emissionlinewithadeterminedlinewidth(sandwichedbytworedverticallines), overplottedontheaveragedfluxspectrum(injy)asafunctionoffrequency(inghz). Topleft)thezoomNupviewoftheidentifiedemissionline,butwithvelocityaxis.(Please beawareofthatthehorizontalaxisofthetopnleftpanelisconvertedfromfrequencyof thetopnmiddlepaneltovelocity.)theverticalaxisistheaveraged(overtheentiremap) fluxinjyandthehorizontalaxisisinunitsofkm/s.the(center)velocityof0km/s correspondstothecenterfrequencyoftheemissionline,whilethevelocityranges equivalenttothemaskedregionasidentifiedemissionline.thelinevelocitywidthis griddedinto15bins,whichareshownasredverticallines.topright)thetotalintegrated intensitymap(injy/beamkm/s)overthechannelrangeofthedeterminedlinewidth. Bottom)thechannelmapswithinthevelocityrangeoftheidentifiedemissionline.Each panelcorrespondstoeachbinonthetopnleftpanel. 28

31