NHSC/PACS Web Tutorials Running PACS spectrometer pipelines

Size: px

Start display at page:

Download "NHSC/PACS Web Tutorials Running PACS spectrometer pipelines"

Dale Cannon
5 years ago
Views:

1 NHSC/PACS s Running PACS spectrometer pipelines PACS- 302 Level 1 to Level 2 and beyond: From Sliced Cubes to Rebinned and Projected Spectral Line Cubes, and 1- D Spectra Original Tutorial by Philip Appleton Updated Sept 2012 by Steve Lord Updated Feb 2013 by Steve Lord Updated Oct 2013 by Steve Lord Updated May 2014 for HIPE by David Shupe Updated June 2015 for HIPE by Roberta Paladini - page 1

Introduc=on This tutorial presents the main steps of the standard pipeline starmng from the

It describes the processing steps needed to create a set of Pacs Rebinned Cubes and (especially if

Numerous break points are shown, where one can interacmvely examine the intermediate results of the

2 Introduc=on This tutorial presents the main steps of the standard pipeline starmng from the calibrated Frames (Level 1) and pacscube (also Level 1). It describes the processing steps needed to create a set of Pacs Rebinned Cubes and (especially if the observamon is a raster) a Projected, Drizzled or specinterpolated Cube. Numerous break points are shown, where one can interacmvely examine the intermediate results of the pipeline. The PACS Spectroscopy pipelines are documented here: PACS Data ReducAon Guide: Spectroscopy, e.g., Chapter 3 These documents can be accessed through the HIPE help pages. - page 2

3 Important addimonal material for each release is found online at the PACS public Twiki: hyp://herschel.esac.esa.int/twiki/bin/view/public/pacscalibramonweb Also, find out what is new for pipeline 13 here: hyp://herschel.esac.esa.int/twiki/bin/view/public/hipewhatsnew13x#pipeline - page 3

4 Tutorial Pre- requisites: 1. HIPE is running version (or user build ) or later 2. You have completed the following tutorial:. PACS- 301 PACS Spectroscopy Level 0 to 1 Processing - page 4

5 At this point in the processing we had just completed the last step of tutorial PACS 301. We last performed the flat field step a`er having created a set of sliced Cubes (step 3 below). We also did some cleanup (steps 4 & 5 below) In this tutorial we conmnue working with obsid = camera = blue Our last steps which brought us to Level 1: - page 5

6 Let s recap the structure of the data products so far, and see where we are going: Level 0-1, and Level 2 Products PACS PIPELINE CUBE BUILDING RAW Level 0 To Level 0.5 SINGS background FRAMES To calibrated L1 FRAMES Projected to 3 x 3 pixels CALIBRATED PACSCUBE REBINNED PROJECTED FRAMES CUBE CUBE CUBE - page 6 Level 1 Level 2

7 Let s recap the structure of the data products so far, and see where we are going: Near the end of the Level 1 pipeline, we created a set of pacscubes. There are 6 pacscubes in our example, because this blue AOR contains 6 slices (Nod A and Nod B for each cycle posimon of the only observed line) PACS PIPELINE RAW Level 0 To Level 0.5 SINGS background FRAMES CUBE BUILDING To calibrated L1 FRAMES Projected to 3 x 3 pixels CALIBRATED PACSCUBE REBINNED PROJECTED FRAMES CUBE CUBE CUBE - page 7 Level 1 Level 2

8 Level 1 to 2 overview: Step 1 Check number of Frame and Cube Slices from level 1 Step 2 Create a wavelength grid for binning Step 3 Flag data with user- controlled sigma- clipping Step 4 Create the rebinned cube and average the NODS Step 5 Inspect the Rebinned Cube Spaxel- by- Spaxel Step 6 (for Raster or dither paiern mapping of extended sources) Project Rebinned Cubes (via specproject, Drizzling or specinterpolate) onto a WCS Step 6 (for point sources and single poinangs) Extract spectra from the central pixel Step 7 Explore Rebinned and Projected cubes in Spectrum Explorer - page 8

9 Step 1 Check that you have converted your frames to pacscubes (5 x 5 x N) product, and that you have the correct number of both - page 9

A pacscube is simply a reorganized Frame with dimensions which are 5 x 5 x N where the 5 x 5 refers to the IFU spaxels of PACS, and N is the number of Mme samples which translate into gramng posimon

In a sense you can think of the pacscube as a spamal representamon of sky (5 x 5) and the third dimension can be translated into wavelength forming the basic dataset or cloud of wavelength samples as

10 A pacscube is simply a reorganized Frame with dimensions which are 5 x 5 x N where the 5 x 5 refers to the IFU spaxels of PACS, and N is the number of Mme samples which translate into gramng posimon and wavelength. In a sense you can think of the pacscube as a spamal representamon of sky (5 x 5) and the third dimension can be translated into wavelength forming the basic dataset or cloud of wavelength samples as a funcmon of posimon. In our example we had 6 slices of Frames. Now, having executed the last step, we have 6 slices of Cubes. These slicedcube and slicedframe enmmes (objects) are simply collecmons of Frames and Cubes sliced by raster posimon (if relevant), Nod posimon (Nod A and B) and Line ID. In this case we have one line observed with two Nod observamons and three Nod cycles. - page 10

11 Check One Let s check that we have the correct number of level1 Frames and pacscubes Type into the Console window the following slicedsummary(slicedframes) slicedsummary(slicedcubes) - page 11

12 Summary of Level 1 Products HIPE> slicedsummary(slicedframes) noslices: 6 nocalslices: 0 noscienceslices: 6 slice# isscience nodposition nodcycle rasterid lineid band dimensions wavelengths 0 true ["B"] [2] ["B3A"] [18,25,1536] true ["A"] [2] ["B3A"] [18,25,1536] true [ A"] [2] ["B3A"] [18,25,1536] true [ B"] [2] ["B3A"] [18,25,1536] true [ A"] [2] ["B3A"] [18,25,1536] true [ B"] [2] ["B3A"] [18,25,1536] Above are 6 Frames (18x25x1536) represenmng (=18) spectral pixels, 25 spamal pixels (the 5 x 5 array) and 1536 Mme samples. Note the 16 spectral pix plus 2 extra (a open channel and an overscan ) making 18 spectral pixels total HIPE> slicedsummary(slicedcubes) noslices: 6 nocalslices: 0 noscienceslices: 6 slice# isscience nodposition nodcycle rasterid lineid band dimensions wavelengths 0 true ["B"] [2] ["B3A"] [24576,5,5] true ["A"] [2] ["B3A"] [24576,5,5] true [ A"] [2] ["B3A"] [24576,5,5] true [ B"] [2] ["B3A"] [24576,5,5] true ["A"] [2] ["B3A"] [24576,5,5] true [ B"] [2] ["B3A"] [24576,5,5] A`er conversion to slicedcube we now have 6 pacscubes, organized as Mme sample*16 (=24576), 5 x 5 in this case. Note that the open and overscan spectral pixels have been stripped off in the pacscube format. - page 12

13 Step 2 Create a wavelength grid. This will be used to grid the spectrum in wavelength- space and is the first step in construc=ng a re- binned cube. In the default case the grid is Nyquist- sampled. However, the user can select a finer grid or, alterna=vely, a courser grid to effec=vely provide addi=onal smoothing. - page 13

14 CreaMng a wavegrid: the grid resolumon and number of phased samplings are governed by the parameters oversample and upsample, respecmvely. The default values of these are oversample = 2, upsample = 4. The default values provide a grid which oversamples the intrinsic spectral resolumon element by a factor of 2 (i.e., Nyquist sampling). An upsample value of 4 means that four conmguous sets of wavelength bins are laid- out. When oversample is m, the binwidth used is the intrinsic resolumon element/m. When upsample is n, n sets of wavlength bins are laid- out, each phased by the binwidth/n. wavegrid=wavelengthgrid(slicedcubes.refs[0].product, oversample=2, upsample=4, caltree = caltree) IllustraMve examples follow. - page 14

15 Examples of Oversample and Upsample Parameter Pairs - page 15

16 Oversample = 2 Upsample = 1 bins = ½ spectral resolumon at that wavelength spectral resolumon = 0.02µm BINS widths are ½ the spectral resolumon element = Nyquist Sampled SAMPLES TAKEN HERE - page 16

17 Oversample = 4, Upsample = 1 bins = 1/4 spectral resolumon at that wavelength Oversample = 4 means 4 Mmes narrower bins that the spectral resolumon (2 x beyer than Nyquist) spectral resolumon = 0.02µm - page 17

18 Oversample = 2 Upsample = 2 bins = ½ spectral resolumon at that wavelength, but bins are shi`ed by ½ a bin and resampled, thus the final spectrum contains a finer grid (although the values are not independent). Sample with oversample = 2 (oversample = 2) then shi` by ½ bin then sample again (upsample = 2) BINS are ½ spectral resolumon = Nyquist Sampled spectral resolumon = 0.02µm - page 18

Thus, the final spectrum contains a fine- spaced grid, but one with highly correlated samples.

19 Oversample = 2, Upsample = 3 ; Binwidths are ½ the spectral resolumon at their wavelength, and bins are shi`ed by 1/3 and 2/3 of a binwidth and resampled. Thus, the final spectrum contains a fine- spaced grid, but one with highly correlated samples. BINS are ½ spectral resolumon = Nyquist Sampled spectral resolumon = 0.02µm Sample with oversample = 2 (oversample = 2) then shi` by 1/3 bin then sample again (upsample = 3 - page 19 Sample centers are spaced at intervals 1/3 the binwidth.

20 Step 3 Before regridding, we apply OUTLIERS rejec=on flagging to the spectra. This process is quite robust and is found to work well for PACS spectra. If the user suspects that the Deglitcher is over- flagging with GLITCH flags, the effect of these GLITCH flags can be turned- off and the de- glitching can instead be done with the OUTLIERS rejec=on flag method - page 20

Next we acmve the masks and apply a sigma- clipping roumne to the spectra (See PACS Data ReducMon Guide: Spectroscopy, 5.2.8.

21 Next we acmve the masks and apply a sigma- clipping roumne to the spectra (See PACS Data ReducMon Guide: Spectroscopy, for details) # AcMvate all masks slicedcubes = acmvatemasks(slicedcubes, String1d([str(i) for i in slicedcubes.get(0).masktypes \ if i not in ["INLINE", "OUTLIERS_B4FF"]]), exclusive = True) # Flag the remaining outliers, (sigma- clipping in wavelength domain) slicedcubes = specflagoutliers(slicedcubes, wavegrid, nsigma=5, niter=1) specflagoutliers has the task of defining a new flag called OUTLIERS created using a simple sigma- clip algorithm on each bin. niter=1 means that the algorithm has been applied once and then again, iteramvely rejecmng points > nsigma. The user can experiment with these two parameters. The output of this roumne is to create a mask. The mask is later applied when the actual rebinning of the spectra onto the wavegrid takes place. The mask is called OUTLIERS and is acmvated in the next step. NOTE ON EXCLUSION OF GLITCH MASK: If you choose to exclude the glitch detecmon from your results, remove the GLITCH string from the acmvatemasks command above. This will ignore the glitch mask. We have had good results by doing this. - page 21

Now we will create our first set of rebinned cubes one for each pacscube: in our lineid=2 case, we have six pacscubes: i.e., for each repemmon (we have three), one pacscube for Nod A and one for Nod B PACS PIPELINE CUBE BUILDING RAW Level 0 To Level 0.

22 Now we will create our first set of rebinned cubes one for each pacscube: in our lineid=2 case, we have six pacscubes: i.e., for each repemmon (we have three), one pacscube for Nod A and one for Nod B PACS PIPELINE CUBE BUILDING RAW Level 0 To Level 0.5 SINGS background FRAMES To calibrated L1 FRAMES Projected to 3 x 3 pixels CALIBRATED PACSCUBE REBINNED PROJECTED FRAME CUBE CUBE CUBE - page 22 Level 1 Level 2

23 Step 4 Create the Rebinned Cube of dimensions 5 x 5 x rebinned wavelength elements - page 23

Now Create the Rebinned Cube We acmvate the OUTLIER mask for the first Mme.

$masktypes \ if i not in ["INLINE", "OUTLIERS_B4FF"]]), exclusive = True) # Rebin all selected cubes on the same wavelength$ (mandatory for specaddnod) slicedrebinnedcubes = specwaverebin(slicedcubes, wavegrid) - page 24 This step produces a set of rebinned

(mandatory for specaddnod) slicedrebinnedcubes = specwaverebin(slicedcubes, wavegrid) - page 24 This step produces a set of rebinned

GLITCH string to the acmvatemasks command above. This will ignore the glitch mask. We have had good results by doing this.

24 Now Create the Rebinned Cube We acmvate the OUTLIER mask for the first Mme. This happens before rebinning # AcMvate all masks slicedcubes = acmvatemasks(slicedcubes, String1d([str(i) for i in slicedcubes.get(0).masktypes \ if i not in ["INLINE", "OUTLIERS_B4FF"]]), exclusive = True) # Rebin all selected cubes on the same wavelength (mandatory for specaddnod) slicedrebinnedcubes = specwaverebin(slicedcubes, wavegrid) - page 24 This step produces a set of rebinned cubes (one slice per cube) NOTE ON EXCLUSION OF GLITCH MASK: If you choose to exclude the glitch detecmon from your results, add the GLITCH string to the acmvatemasks command above. This will ignore the glitch mask. We have had good results by doing this. This is the second and last Mme you need to worry about ignoring the glitch mask. If you performed this step and the one before specflagoutliers then you will have a rebinned cube which excludes the glitch detecmon and relies on the sigmaclipping for removal of outliers. In general, we have found sigmaclipping to produce the beyer results.

25 Step 4 (con t) Average the Nods - page 25

26 Now we average together the two Nod posimons. For an observamon with a set of raster posimons, there smll will be more than one final cube (store as slices) one for each raster posimon # Average the nod-a & nod-b rebinned cubes. # All cubes at the same raster position are averaged. # This is the final science-grade product for single pointing and spatially undersampled # rasters. slicedfinalcubes = specaddnodcubes(slicedrebinnedcubes) - page 26

27 Step 5 Inspect the rebinned spectra spaxel by spaxel using various diagnos=c tools - page 27

28 With the verbose=1 diagnostics, we can view (e.g.) the central (2,2) spaxel profile along with its estimated line flux and continuum RMS. Draw a box to zoom in - page 28

29 - page 29 In the case of a raster observation, you can plot the profiles for each slice separately.

30 Step 6 (for point sources and single poin:ng) Extract spectra from the central spaxel - page 30

31 ExtracMng the spectrum for Point Sources Caveats: The following instructions assume that the observation consists of a single pointing of a target which is a point source*, and that the target peak position is centered on the central spaxel [2,2] (i.e., the flux peaks at spaxel [2,2])**. *If there are multiple pointings, these should be handled separately following the instructions below using the individual rebinned cubes. (The PACS point source corrections operate only on individual rebinned cubes.) **If the source is in a spaxel other than the central spaxel, use the PACS Point Source Loss Correction (any spaxel) script found in HIPE->Scripts->PACS useful scripts. extractcentralspectrum will return three spectra: a) Spectrum of the central pixel corrected for the part of the beam that the central spaxel misses. This is known as the c1 spectrum. b) Spectrum of the sum of the central 9 pixels, with the (much smaller) correction for the part of the beam that this superspaxel misses: the c9 spectrum. c) Spectrum of the central pixel, with the 3x3 correction that compares the ratio of c1 to c9 to the ratio expected from the beam profile; this is intended to correct for small mis-pointings and is the c129 spectrum ( c one-to-nine, get it?). To trust this correction the source continuum must be of order 10 Jy. Discussions of these corrections are found in the PACS Data Reduction Guide: Spectroscopy section 8.3 and page 31

32 ExtracMng the spectrum for Point Sources (con t) for slice in range(len(slicedfinalcubes.refs)): if verbose: print # a. Extract central spectrum, incl. point source correction (c1) # b. Extract SUM(central_3x3 spaxels), incl. point source correction (c9) # c. Scale central spaxel to the level of the central_3x3 (c129 -> See PDRG & print extractcentralspectrum. doc ) c1,c9,c129 = extractcentralspectrum(slicedfinalcubes, slice=slice, smoothing='median', islinescan=-1, caltree=caltree, verbose=verbose) # # Save to Fits if saveoutput: name = namebasis + "_centralspaxel_pointsourcecorrected_corrected3x3no_slice_" simplefitswriter(product=c1,file = outputdir + name + str(slice).zfill(2)+".fits") name = namebasis + "_central9spaxels_pointsourcecorrected_slice_" simplefitswriter(product=c9,file = outputdir + name + str(slice).zfill(2)+".fits") name = namebasis + "_centralspaxel_pointsourcecorrected_corrected3x3yes_slice_" simplefitswriter(product=c129,file = outputdir + name + str(slice).zfill(2)+".fits") On the next page we show how this can found in the variable list and opened in SpectrumExplorer. - page 32

33 ExtracMng the spectrum for Point Sources (con t) In the variable list right- click on, e.g., c1 and either open with Spectrum Explorer or send to a FITS file - page 33

34 By execumng the command lines on page 32, we get. - page 34

35 Step 6 (for Raster Observa:ons of Extended Sources) In order to mosaic your cubes onto a uniform World Coordinate System (WCS) grid, users with raster observa=ons* should select between the three methods of mosaicking: 1) specproject or 2) Drizzling, for oversampled maps; 3) specinterpolate, for undersampled maps. *As opposed to the single- poin=ng example data set of this tutorial - page 35

36 The projected cube is only recommended for raster observamons where you have many rebinned cubes and these are combined into a final projected cube PACS PIPELINE CUBE BUILDING RAW Level 0 To Level 0.5 SINGS background FRAMES To calibrated L1 FRAMES Projected to 3 x 3 pixels - page 36 CALIBRATED PACSCUBE REBINNED PROJECTED FRAME CUBE CUBE CUBE Level 1 Level 2

37 specproject (Refer to sec=on 7.8 of the PACS Data Reduc=on Guide for Spectroscopy ). For its rela=ve speed, specproject is recommended for large wavelength range observa=ons and SEDS. Each output spaxel is formed from a weighted combina=on of overlapping and/or undersampled spaxels from the slicedfinalcubes (which have been rebinned in wavelength) for each spa=al and wavelength point onto a (default) 3 grid. - page 37

Invoking specproject # specproject works on the final rebinned cubes. slicedprojectedcubes = specproject(slicedfinalcubes,outputpixelsize=3.

38 Invoking specproject # specproject works on the final rebinned cubes. slicedprojectedcubes = specproject(slicedfinalcubes,outputpixelsize=3.0) # Display the projected cube in the Spectrum Explorer if verbose: slice = 0 oneprojectedslice = slicedprojectedcubes.refs[slice].product openvariable("oneprojectedslice", "Spectrum Explorer") The present script makes a slicedprojectedcube and even opens up a slice in Spectrum Explorer. Also available are other viewers (by right- clicking on the variable slicedprojectedcube ) as well as the same FITS wrimng capability as earlier described. - page 38

Caveat: Making a projected cube from a single poinmng Making a projected cube (using specproject or Drizzling) from a single poinmng is not recommended, but it is also not prohibited.

39 Caveat: Making a projected cube from a single poinmng Making a projected cube (using specproject or Drizzling) from a single poinmng is not recommended, but it is also not prohibited. However, the formal final product from a single poinmng is considered to be a rebinned cube, not any projected cube. One may ayempt to project a single poinmng for visualizamon purposes (such as is done in this tutorial), but, because the PACS IFU footprints are distorted, flux conservamon is typically not maintained in the output map. Only in the case of a fully sampled raster (spamal dithered mapping) can one expect to recover proper fluxes from a projected cube. - page 39

40 Drizzling The Drizzling projec=on is so- called because the total drops of flux collected by the input pixels are divided into smaller droplets and then collected/weighted into the output pixels. It differs from specproject in that it operates on the Level 1 slicedpacscubes prior to wavelength rebinning (not Level 2 cubes). (Primary drizzle document: - page 40

41 Invoking Drizzling For parameter details, refer to sec=on 7.7 of the PACS Data Reduc=on Guide (Spectroscopy) # Drizzling works from the not-rebinned cubes oversamplewave = 2 upsamplewave = 3 wavegrid = wavelengthgrid(slicedcubes, oversample=oversamplewave, upsample=upsamplewave, caltree = caltree) oversamplespace = 3 upsamplespace = 2 pixfrac = 0.6 spacegrid = spatialgrid(slicedcubes, wavelengthgrid=wavegrid, oversample=oversamplespace, upsample=upsamplespace, pixfrac=pixfrac, caltree=caltree) sliceddrizzledcubes = drizzle(slicedcubes, wavelengthgrid=wavegrid, spatialgrid=spacegrid)[0] These Drizzled Cubes may be viewed in Spectrum Explorer as well. - page 41

42 Invoking Drizzling (WARNING) Warning: In Pipeline 13 DRIZZLING is run by default that is the drizzling script lines shown in the previous slide are not commented-out. The user should be sure that he/she wants to run drizzle before invoking it. The drizzle routine is an excellent optimized and powerful projection method (when sufficient oversampling has been obtained within the observation) but it can require many hours of execution time and large amounts of computer memory, especially for large maps or line ranges. It is best run on a large and perhaps remote virtual machine (via a VNC session) and utilizing multithreading. The user s alternative, specproject will, in general, yield a fairly good projected map in much shorter processing times. - page 42

43 specinterpolate For undersampled raster maps or single poin=ng maps of extended sources, the best projec=on op=on is specinterpolate, which is based on Delaunay triangula=on. In this projec=on, the default output pixel size is page 43

44 Invoking specinterpolate For parameter details, refer to sec=on 7.9 of the PACS Data Reduc=on Guide (Spectroscopy) # The final science grade products are the previously produced #"slicedfinalcubes" # You can also interpolate these cubes on a regular spatial grid using # specinterpolate outputpixelsize = 4.7 slicedinterpolatedcubes = specinterpolate(slicedfinalcubes, outputpixelsize=outputpixelsize, conserveflux=true) if saveoutput: name = namebasis+"_slicedinterpolatedcubes" saveslicedcopy(slicedinterpolatedcubes, name, poollocation=outputdir) - page 44

45 Step 7 Explore your final projected or rebinned cube with the Spectrum Explorer - page 45

The Spectrum Explorer provides many visualizamon opmons as well as line fi}ng You may display and examine the final spectral cubes with the Spectrum Explorer.

46 The Spectrum Explorer provides many visualizamon opmons as well as line fi}ng You may display and examine the final spectral cubes with the Spectrum Explorer. Further processing can be performed in this environment using, e.g., the Cube Tool Box and the Spectrum Fitter GUI. These allow line extraction in a selected area, baseline fitting, the generation of integrated line maps, etc.. - page 46

pointing), use the last script in the pulldown menu, that operates on the rebinned cubes.

47 Useful Scripts are provided for making integrated line maps, and for extracmng point source spectra from spaxels other than the central one For the case shown in this tutorial (single pointing), use the last script in the pulldown menu, that operates on the rebinned cubes. For well-sampled raster maps, use Fitting and making fit-images, Level 2 projected/drizzled cubes. - page 47

NHSC/PACS Web Tutorials Running PACS spectrometer pipelines

NHSC/PACS Web Tutorials Running PACS spectrometer pipelines NHSC/PACS s Running PACS spectrometer pipelines PACS- 302 Level 1 to Level 2 and beyond: From Sliced Cubes to Rebinned and Projected Spectral Line Cubes, and 1- D Spectra Original Tutorial by Philip Appleton