Implementation of the Temperature-Dependent Dark Correction in CalnicA

Transcription

1 Implementation of the Temperature-Dependent Dark Correction in CalnicA R. Jedrzejewski December 1, 2002 ABSTRACT The pipeline implementation of the temperature-dependent dark correction, previously only available to users as a WWW tool, is described. The algorithm used to generate the darks is described, along with a description of how the IDL code and data that are used in the WWW tool were incorporated into the usual HST pipeline model of code+reference files. This implementation necessitated the creation of a new reference file type: the Temperature-Dependent Dark (TDD) file. The design of this file type allowed many of the hard constants to be removed from the code and put into the reference files, allowing more convenient and flexible updating of the calibration parameters, if necessary. Introduction As described in NICMOS ISR NICMOS Temperature-specific darks (B. Monroe), the dark current in the NICMOS detectors is a function of temperature. That ISR describes the implementation of an algorithm to determine the dark correction in real time, given the instrumental setup and a temperature measure. The method was implemented as a WWW tool, allowing users to enter their exposure characteristics (camera, sample sequence and temperature), and the tool would return a dark reference file appropriate for those parameters. The user could apply this dark reference file, either by editing the header parameter DARKFILE to point to this newly-generated file and re-running the CalnicA pipeline, or by doing the same operations by hand in, for example, IRAF or IDL. While the results of using the darks generated by this tool were a big improvement over Copyright 2002 The Association of Universities for Research in Astronomy, Inc. All Rights Reserved.

2 using static dark reference files, the process necessary to generate and use the required reference files was rather cumbersome. There was a clear need to make the process more transparent to the user, especially with On-the-Fly Reprocessing being introduced for NICMOS. Components of the Dark Current Signal The dark current in NICMOS has three components: the linear dark signal, proportion to cumulative exposure time the amplifier glow, proportional to the number of reads the shading component, which depends on the time since the last readout and the temperature in a complex fashion. The first two components are straightforward to understand and implement. The linear dark signal is basically the true dark signal, proportional to the time since the last detector reset. It depends on pixel location, so can be modelled as an image that would be obtained in a 1 second exposure. Typical values are a few hundredths of a count per second per pixel, with some increase near the corners where the readout amplifiers are located. The linear dark level varies from quadrant to quadrant, and there are several hot pixels with elevated linear dark current that can be as large as several counts/second/pixel. The linear dark component is believed to vary with temperature (since it is basically due to thermally-generated electrons), but this effect is quite small and difficult to measure, so the temperature dependence is not currently included in the characterization of this component. The amplifier glow component arises because the amplifiers emit a low level of infrared radiation when they are reading out the detector pixels. This radiation is detected by the pixels, and appears as a signal. Again, the signal depends on pixel location, as well as on the number of reads that have been performed since the last detector reset. Typically, the amp glow is 1-2 counts/read over most of the image, with a rise to approximately 10 counts/read towards the corners where the amplifiers are. The amp glow can be represented as an image of the counts for 1 read, and then the amp glow contribution for a given image can be calculated by multiplying this unit image by the number of reads. The shading component is more complicated. Here the shading signal depends on both the time since the last read (DELTATIM in NICMOS parlance) and the temperature. The dependence on delta time is approximately logarithmic. The temperature dependence depends strongly on pixel position, in a way that is different for different regions of the detector. It is the implementation of this correction that is the main subject of this report. 2

3 The final dark signal is just the sum of these components: dark = lineardark + ampglow + shading The IDL algorithm The WWW tool, available at is implemented as an html form that calls a perl script. The script is basically a wrapper around a set of IDL calls. At the time the CalnicA code was being designed, the IDL code lived in /data/bendyjack3/ httpd/cgi-bin/darks/ and was served by the http server run on the machine bendyjack. Now it has been moved to the STScI Zope server, but the content should be identical. The perl script generates IDL code that looks like the aa=getset( /data/cdbs5/nref/hat1336fn_drk.fits ) syndark=syndark2_2(tstep8,1,67.67) newdark=float(aa) newdark(*,*,*,0)=syndark write_multi, tmp/step8_1_6767_drk.fits,newdark,header_from= /data/ cdbs5/nref/hat1336fn_drk.fits,history=[ file created by STScI web interface, on date listed above in DATE keyword, B. Monroe and E. Bergeron ],pedigree= MODEL 10/08/1999,useafter= 22/08/1997,descrip= MODEL NIC1 SAMP_SEQ step8 dark exit This code was created by the perl script from a request that specified NICMOS Camera 1, SAMP-SEQ=STEP8, and a temperature of degrees K. The first line is a call to a routine to do Camera 1-specific startup operations, such as defining common data blocks and restoring an IDL SAV file. This file holds, among other things, the A234_TIME data structure and the amplifier glow image. The former is just an array of FLOATs that holds the list of NICMOS DELTATIMEs for which a temperature-dependent calibration has been performed. The list of calibrated DELTATIMEs is given below in Table 1. Table 1. List of DELTATIMEs for which a temperature-dependent dark calibration has been performed. Calibrated DELTATIMEs (s)

4 Calibrated DELTATIMEs (s) This list covers almost all of the exposure times specified in the allowed sample sequences; the only delta times that are not covered are the tenth readouts in the MIF sequences, much of the MIF3072 sequence, the fourth reads in the SPARS sequences (although the difference between the actual DELTATIME and the calibrated DELTATIME is probably negligible in this case), and the SCAMRR sequence. The only sequence where the lack of calibration would be significant is the SCAMRR sequence, where delta times are all s and the shading correction is a strong function of delta time for these short exposure times. The amplifier glow image is stored in the variable C[123]AMPARR as a array of FLOATS, with each image plane equal to the glow produced by a single read. This means that the amplifier glow component for a given extension can be extracted as the plane for that extension (modulo the usual zero-indexing of IDL arrays). The next line, aa=getset( /data/cdbs5/nref/hat1336fn_drk.fits ) just sets up a structure that looks like the CDBS reference file for that Camera and sample sequence. All of the real work is done in the next line: syndark=syndark2_2(tstep8,1,67.67) where the synthetic dark file for the STEP8 sequence for Camera 1 is calculated for a temperature of The syndark2_2 routine grabs the amplifier glow array from the common block initialized previously, then locates the appropriate linear dark array image by restoring the lineardark[123].sav file for the relevant camera. This array, C[123]_B_DARK, is then scaled by the exposure times of the sample sequence to make a multi-plane array where each plane corresponds to a readout. 4

5 With the easy work out of the way, the shading component is calculated in the routine makeshade[123](delta, temp), where delta is the array of delta times and temp is the temperature. The makeshade routine starts by restoring the polynomial coefficients that describe the variation of shading signal with both pixel position and temperature from the file brianshade[123].sav. These coefficients are stored in arrays, with the names of the arrays dependent on the implementation for that camera. The temperature-dependent shading signal is calculated in the following way: First, the delta times for each read in the sample sequence are calculated. The index of the calibrated delta time in the A234_TIME array that most closely matches the delta time of the group under consideration is calculated. Then, the detector area is divided up into a patchwork of rectangular regions. Each region has its own set of polynomial coefficients. For a given region, the shading signal is a polynomial function of position in one of the dimensions (X or Y), and constant in the other dimension. The polynomial coefficients of position are themselves polynomial coefficients of temperature. Thus, for example, for a hypothetical region A: X range = X1 to X2 Y range = Y1 to Y2 Function varies along the X direction, constant in the Y direction Shading signal is a polynomial function of X pixel, shifted to start at P1 and end at P2 (where P2 P1 = X2 X1). The shading signal at pixel with coordinates (i,j) is given by m S( ( i+ X1 1), j) = at ( )( i+ P1 1) n where p at ( ) = b n T n n = 0 n = 0 This is coded in the makeshade routine in blocks like the following: (1) (2) pp = [poly(temp,shadefit0(*,0,deltaindex)), poly(temp,shadefit1(*,0,deltaindex)), poly(temp,shadefit2(*,0,deltaindex)), poly(temp,shadefit3(*,0,deltaindex)), poly(temp,shadefit4(*,0,deltaindex))] where the array of polynomial coefficients a(t) is calculated as polynomials in the temperature temp, using the coefficients stored in the shadefit[01234] array. The 5

6 deltaindex parameter is the index of the delta time calibration used. The actual shading signal is calculated in the next line: kk_l = poly(findgen 1 (21),pp) where the signal is a polynomial function of pixel coordinate in the patch under consideration. This creates a 1-dimensional vector of the shading signal. This is converted into a two dimensional signal by replicating in the perpendicular direction for the extent of the patch. The total signal is calculated as the sum of all the patches; in this way both X and Y variation are possible. Finally, some patches are left blank because there is no temperature dependence in that region. In that case, a static component, which depends on pixel location and delta time only, is added to the overall shading signal. The last operation is of course to add the linear dark, amplifier glow and shading components into the final dark image for each read appropriate for that sample sequence. CalnicA Reference File Design The main problem with implementing this algorithm in the CalnicA pipeline is that many of the parameters describing the fit are hardwired into the code. This does not map well to the standard model of pipeline processing; make the code as general as possible and put the things that are likely to change (due to improved calibration) in the reference files. That way, when a new calibration is performed, only the reference files need to be updated (with USEAFTER date processing to take care of time-dependent calibrations), not the code itself. This seemed particularly important for NICMOS, since its cryogenicallycooled life was over, and its mechanically-cooled life is in its infancy. The significantly different temperature régime of the CryoCooled era could well make the earlier calibrations invalid. Similarly, the IDL code looks quite different for each of the three cameras, and yet a method is needed to process each camera with the same code, moving the camera-specific parameters to the reference files. After much thought, a solution was developed in which all of the calibration parameters could be moved to the reference files. A reference file format was chosen that is a combination of images and binary tables, which fits in well within the FITS format used for reference files. The format looks like this: 1. The IDL function findgen(n) creates a 1-d array of integers with values from 0 to (n-1). So in the example above, findgen(21) is the same as [0,1,2,3,...,18,19,20] 6

7 Primary Header Unit LIN extension AMPGLOW extension SHAD binary table STATIC extension SHAD binary table STATIC extension SHAD binary table STATIC extension Figure 1: Structure of the Temperature-Dependent Dark reference file The Primary Header Unit (PHU) contains basic information about the reference file. The PHU for the NICMOS Camera 1 reference file is shown below: SIMPLE = T / Fits standard BITPIX = 16 / Bits per pixel 7

8 NAXIS = 0 / Number of axes ORIGIN = NOAO-IRAF FITS Image Kernel July 1999 / FITS file originator IRAF-TLM= 11:01:26 (15/02/2002) / Time of last modification DATE = T18:31:33 / Creation date (CCYY-MM-DD) of FITS header EXTEND = T / File May Contain Extensions TELESCOP= HST /telescope used to acquire data INSTRUME= NICMOS /instrument in use DARKMETH= TEMPERATURE-DEPENDENT /Dark calculation method TEMPKEY = NDWTMP11 /Temperature key CAMERA = 1 /NICMOS camera (1 2 3) NDELTA = 12 /#different delta-times DELTA1 = / DELTA2 = / DELTA3 = / DELTA4 = / DELTA5 = / DELTA6 = / DELTA7 = / DELTA8 = / DELTA9 = / DELTA10 = / DELTA11 = / DELTA12 = / PEDIGREE= MODEL 18/02/2002 / USEAFTER= Aug / DESCRIP = Camera 1 temperature-dependent dark reference file / FILETYPE= TDD HISTORY cam1_drk.fits renamed to m3b1331gn_tdd.fits on Mar The DARKMETH keyword was originally included as a way of distinguishing temperature-dependent dark reference files from regular static dark files, since the intention was to make the temperature-dependent dark reference file a _drk.fits file, just like the static dark reference file. However, the fact that the dark reference file would then need to have a different format depending on type (not supported by CDBS), and that the dark correction reference file would then not be backward compatible with earlier versions of CalnicA, led to the decision to abandon the plan of using _drk.fits files to store the temperature-dependent calibration data. Instead, a new type of NICMOS reference file was created: the _tdd.fits file. The rest of the reference file format is split into several parts. Firstly, there is the linear dark signal, stored in the LIN extension. This is just an image of the linear dark signal for a 1s exposure. To determine the linear dark component for a given group, this component is multiplied by the exposure time (keyword SAMPTIME from the extension header). The next component is the amplifier glow signal, stored in the AMPGLOW extension. This is an image of the amplifier glow for a single read. To determine the amplifier glow component for a given group, this component is multiplied by the number of reads (nreads = nsamp ngroup, where nsamp is the value of the NSAMP keyword from the primary header and ngroup is the number of the group in question; remember NIC- MOS _raw files have the order of groups reversed so that the first read is the last group). 8

9 The rest of the extensions are in pairs, with one pair for each calibrated delta time. The STATIC components give the static shading signal for the delta time in the list of delta times in the PHU of the dark reference file; for example, the [STATIC,3] extension is the static component with delta time equal to s. The SHAD component is where the coefficients for the temperature-dependent shading signal are stored. This component is a binary table with seven columns: Column Name Format Description xstart int Initial X value for region xend int Final X value for region ystart int Initial Y value for region yend int Final Y value for region axisvary int Direction of polynomial variation (1=x, 2=y) pstart int Initial value for polynomial evaluation pend int Final value for polynomial evaluation coeff Float array Coefficients Table 2. Column names in SHAD binary table There is one row for each rectangular patch that the detector area is divided into. The columns follow the structure outlined in the previous section, where the parameters describing a rectangular patch are described. The patches are described in a coordinate system where the X axis is the same as the FITS AXIS1, and the Y axis is the same as the FITS AXIS2. Both coordinates are 1-indexed, unlike IDL arrays. The PSTART and PEND parameters are generally equal to 0 and (XEND XSTART 1); however, there are cases where PSTART is not zero. This arises from patches where the shading polynomial is evaluated with an expression like the following (from makeshade1.pro): kk_r=poly(findgen(22)-2,pp) 9

10 This particular construct was used in the process whereby the polynomial coefficients were determined; it allowed additional pixels to the left of the region to be fitted with the polynomial even though they weren t part of the original fit. The polynomial coefficients themselves are stored in the COEFF array. This is a simple 2- d array of floating point numbers. Each row has the set of coefficients b n that are used to determine the coefficients of the polynomial a(t) that are used in equation (1) to calculate the shading signal S. The final shading signal is determined by adding the signals from each patch, along with the static signal from the STATIC extension. In order to ensure that the static signal is not contaminated by a temperature-dependent component, a rectangular patch that covers the static region is specified in the SHAD table with zeros for the coefficients. The overall dark signal is calculated by adding together the linear dark, amplifier glow, static and temperature-dependent components. Creating the Reference Files The reference files are created using three IDL procedures, one for each camera. They are stored in the directory /data/zork4/rij/ssg/nicmos/tempdependdark/cgi/ and have the names mkref1.pro, mkref2.pro and mkref3.pro to make reference files for NICMOS cameras 1, 2 and 3. The procedures were created specifically to work with the IDL save sets and procedures that the WWW tool uses, so if the details of the procedures change, then it will be necessary to edit the reference file generators accordingly. The following listing shows the procedure used to create the Camera 1 reference file: ; This program will return a reference file for Camera 1 ; Get the linear dark current array restore, lineardark1.sav ; Get the amp glow array and the reference deltatime vector restore, dark1.sav ; Get the temperature-dependent shading coefficients ; and the static dark component restore, brianshade1.sav ; make the primary FITS header mkhdr, header,, /EXTEND orderedelements = [11, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0, 12] ndelta = 12 10

11 fxaddpar, header, TELESCOP, HST, telescope used to acquire data fxaddpar, header, INSTRUME, NICMOS, instrument in use fxaddpar, header, DARKMETH, TEMPERATURE-DEPENDENT, Dark calculation method fxaddpar, header, TEMPKEY, NDWTMP11, Temperature key fxaddpar, header, CAMERA, 1, NICMOS camera (1 2 3) fxaddpar, header, NDELTA, 12, #different delta-times for j=0, ndelta-1 do begin index = orderedelements[j] deltatime = a234_time[index] keyword = DELTA +strcompress(string(j+1),/rem) fxaddpar, header, keyword, deltatime endfor fxaddpar, header, PEDIGREE, MODEL 2/18/2002 fxaddpar, header, USEAFTER, 21/08/1997 fxaddpar, header, DESCRIP, Camera 1 temperature-dependent dark reference file ; open the output FITS file writefits, cam1_drk.fits,, header lineardark = fltarr(256, 256) lineardark = float (c1_b_dark) mkhdr, header, lineardark, /image fxaddpar, header, EXTNAME, LIN fxaddpar, header, EXTVER, 1 ; write the linear dark array writefits, cam1_drk.fits, lineardark, header,/append mkhdr, header, c1amparr(*,*,24), /image fxaddpar, header, EXTNAME, AMPGLOW fxaddpar, header, EXTVER, 1 ; write the amp glow array writefits, cam1_drk.fits,c1amparr(*,*,24),header,/append ; Define the columns we will use ntables = 12 xstart=1 xend=256 ystart=1 yend=21 axisvary=2 polstart=0 polend=20 coefficients = fltarr(3,5,12) staticshading = fltarr (256, 256) ; loop over delta times for j = 0, ndelta-1 do begin index = orderedelements[j] 11

12 deltatime = a234_time[index] ; Write the static component first staticshading[0:255, 0:20] = bottoms[0:255, 0:20, index] staticshading[0:255, 128:148] = bottoms[0:255, 128:148, index] mkhdr, header, staticshading, /image fxaddpar, header, EXTNAME, STATIC fxaddpar, header, EXTVER, j+1 fxaddpar, header, DELTIME, deltatime writefits, cam1_drk.fits, staticshading, header, /append coefficients[*,*,0]=[[shadefit0[*,0,index]],[shadefit1[*,0,index]],[sha defit2[*,0,index]],[shadefit3[*,0,index]],[shadefit4[*,0,index]]] coefficients[*,*,1]=[[shadefit0[*,1,index]],[shadefit1[*,1,index]],[sha defit2[*,1,index]],[shadefit3[*,1,index]],[shadefit4[*,1,index]]] coefficients[*,*,2]=[[shadefit0[*,2,index]],[shadefit1[*,2,index]],[sha defit2[*,2,index]],[shadefit3[*,2,index]],[shadefit4[*,2,index]]] coefficients[*,*,3]=[[shadefit0[*,3,index]],[shadefit1[*,3,index]],[sha defit2[*,3,index]],[shadefit3[*,3,index]],[shadefit4[*,3,index]]] coefficients[*,*,4]=[[shadefit0[*,4,index]],[shadefit1[*,4,index]],[sha defit2[*,4,index]],[shadefit3[*,4,index]],[shadefit4[*,4,index]]] coefficients[*,*,5]=[[shadefit0[*,5,index]],[shadefit1[*,5,index]],[sha defit2[*,5,index]],[shadefit3[*,5,index]],[shadefit4[*,5,index]]] coefficients[*,0:1,6] = [[yshadefit0[*,2,index]],[yshadefit1[*,2,index]]] coefficients[*,0:1,7] = [[yshadefit0[*,6,index]],[yshadefit1[*,6,index]]] coefficients[*,0:1,8] = [[yshadefit0[*,3,index]],[yshadefit1[*,3,index]]] coefficients[*,0:1,9] = [[yshadefit0[*,7,index]],[yshadefit1[*,7,index]]] headerstring = Shading correction for deltatime= +strcompress (string(deltatime))+ s fxbhmake,header,12, SHAD, headerstring fxaddpar, header, EXTVER, j+1 fxaddpar, header, DELTIME, deltatime fxbaddcol, xscol, header, xstart[0], XSTART fxbaddcol, xecol, header, xend[0], XEND fxbaddcol, yscol, header, ystart[0], YSTART fxbaddcol, yecol, header, yend[0], YEND fxbaddcol, avcol, header, axisvary[0], AXISVARY fxbaddcol, pscol, header, polstart[0], PSTART fxbaddcol, pecol, header, polend[0], PEND fxbaddcol, cocol, header, coefficients[*,*,0], COEFF fxbcreate, unit, cam1_drk.fits, header xstart = [1, 1, 1, 1, 1, 1, 1, 1, 129, 129, 1, 1] xend = [256, 256, 256, 256, 256, 256, 128, 128, 256, 256, 256, 256] ystart = [1, 22, 33, 128, 150, 161, 22, 150, 22, 150, 1, 129] yend = [21, 32, 127, 149, 160, 256, 128, 256, 128, 256, 21, 149] axisvary = [2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1] polstart = [0, 0, 0, -2, 0, 0, 0, 0, 0, 0, 0, 0] 12

13 polend = [20, 10, 94, 19, 10, 95, 127, 127, 127, 127, 255, 255] for i = 1, 12 do fxbwrite, unit, xstart[i-1], xscol, i for i = 1, 12 do fxbwrite, unit, xend[i-1], xecol, i for i = 1, 12 do fxbwrite, unit, ystart[i-1], yscol, i for i = 1, 12 do fxbwrite, unit, yend[i-1], yecol, i for i = 1, 12 do fxbwrite, unit, axisvary[i-1], avcol, i for i = 1, 12 do fxbwrite, unit, polstart[i-1], pscol, i for i = 1, 12 do fxbwrite, unit, polend[i-1], pecol, i for i = 1, 12 do fxbwrite, unit, coefficients[*,*,i-1], cocol, i endfor fxbfinish, unit end CalnicA Implementation The CalnicA code was inherited from Howard Bushouse. He had been working on a prototype Version 4, which involved changing the order of calibration so that each calibration step is applied to all of the groups before going on to the next step, in contrast to the thencurrent version which applied all the calibration steps to each group before going on to the next group. The new version that incorporates the temperature-dependent correction does the following steps, assuming the DARKCORR primary header keyword is set to PERFORM: Look for a temperature-dependent reference file specified using the primary header keyword TEMPFILE If the TEMPFILE keyword does not exist, or if it is set to N/A, use the static dark reference file pointed to by the primary header keyword DARKFILE If the TEMPFILE keyword points to a valid temperature-dependent reference file, use it to calculate the dark component for each group of the data file. The temperature sensor reading to be used by CalnicA is encoded in the reference file keyword TEMPKEY. For Cameras 1 and 2, this key is NDWTMP11, while for Camera 3 the key is NDWTMP13. The temperatures are read from the _spt file that accompany the data files. Each group has its own temperature measurement; the actual temperature used in the temperature-dependent correction is obtained by averaging the temperature readings from all the groups. If the writedark parameter is set to TRUE (or if the calnica program is run using the command-line parameter -write), a static dark reference file is created by CalnicA, with the name rootname_drk.fits. This file can be compared with that created using the WWW tool. The ability to write out a static reference file was extremely valuable in testing the code, since the true dark values could be obtained using the WWW tool. Agreement with the then-current version of CalnicA (v3.3), using static dark reference files created using the WWW tool, was reasonably good. The differences mainly arose from the differences in the order of calculation, not from the temperature-dependent dark correction step. 13

14 What s Missing, and What s Next The most obvious deficiency in this code is the lack of error terms. In principle, these can be determined by calculating the formal errors when calculating the dark components. The next step is to add the capability to calculate the temperature from the bias signal. This step has already been coded by Eddie Bergeron in IDL basically as a state machine calculation. Once again, it will be difficult to incorporate this calculation step into the code+reference file model. The most likely implementation will involve calculating the appropriate temperature from the data before CalnicA is run, and inserting this temperature into a _spt file keyword. CalnicA will then use this keyword in calculating the temperature-dependent dark signal. This coding has not started yet, since it seems wise to wait until some of the SMOV issues are sorted out and then a decision can be made on the bast way to tackle the problem. The flatfield is also known to be temperature dependent, and a WWW tool has been written to determine the temperature-dependent flatfield correction. This can also be incorporated into CalnicA in a similar fashion to how the dark correction was implemented. 14