OPUS Science Data Processing

Size: px

Start display at page:

Download "OPUS Science Data Processing"

Bruno Clarke
6 years ago
Views:

1 Science Data Processing of 13

2 OPUS Science Data Processing PMDB ASSIST PDB pod file Data Partitioning Support Schedule Data Validation EDT binary dataset EDT binary dataset + dgx EDT binary dataset, dgx + dvx check for bad segments insert fill data if telemetry dropouts extract proposal and scheduling information from PMDB dredge mnemonic values from telemetry and apply conversions if necessary compare actual vs. planned observation parameters World Coordinate System convert pointing informationinto standard FITS WCS keywords Keyword Database Keyword Exceptions Table Keyword Rules Generic Conversion Best Ref/ Switch EDT binary dataset, dgx, dvx + dcx uncalibrated FITS dataset uncalibrated FITS dataset convert data into FITS format properly orient images or spectra populate keyword values populate calibration reference file and switch keywords (also used by StarView/web) CDBS Data Collector hold individual exposures for association processing uncalibrated FITS dataset 2 of 13

3 OPUS Processes Œ Data Partitioning segments the telemetry stream into standard EDT dataset appends multiple dumps into a single exposure file > uses an index file to avoid sorting packets > flags packet errors caught by PACOR Œ Support Schedule gathers proposal information from PMDB test proposals required for development > test version of PMDB must be populated by TRANS 3 of 13

4 OPUS Processes (cont.) Œ Data Validation decodes the exposure and engineering parameters in the telemetry and compares them to the planned values internal header specification (from Ball) > PDB (EUDL.DAT, TDFD.DAT) must be fully populated and defined in DM-06 [DONE] Planned value comparisons ( flags and indicators ) need to be determined by the Instrument Scientists [09/2002] Œ World Coordinate System implements a translation from telescope coordinates through the instrument light-path to an astronomically valid pointing aperture positions must be defined [09/2002] 4 of 13

5 OPUS Processes (cont.) Œ Generic Conversion Generic Conversion outputs uncalibrated data > raw, asn, spt, trl data will be output in standard FITS format with image or table extensions primary header will contain keywords inherited by all extensions and a null data array Extensions > Images grouped by imsets consisting of science array, data quality array, and null error array. > data quality array will be null if no telemetry dropouts 5 of 13

6 OPUS Processes (cont.) Œ Generic Conversion (cont.) Required for development > DM-06 to develop algorithms for data formatting [DONE] > keyword definitions (ICD-19) are currently being worked by the Instrument Scientists world coordinate definitions [09/02] calibration switches and selection criteria [09/02] calibration file name keywords [09/02] 6 of 13

7 OPUS Processes (cont.) Œ Data Collector OPUS will ensure all component exposures are present before processing further association table contains information about the product dataset > dataset self-documenting only associations required for data processing will be constructed in the OPUS pipeline > CR splits > Repeat obs 7 of 13

8 Calibration Œ OPUS will use STSDAS calibration software Œ run on ALPHA / Tru64 UNIX platform in operations Œ expands size of dataset converts integer raw data to real possible expansion of data quality array. expansion of the error array. Œ Need calibration reference files for testing (at least dummies) 8 of 13

9 Other Science Data Modes Œ data content & keywords for other science data modes must be defined by Instrument Scientists microprocessor memory dump [09/02] engineering diagnostic data [09/02] 9 of 13

10 CODE REUSE Œ Core OPUS system (OPUS 13.2) ~90,000 lines of code 100% reuse Œ WFC3 specific processes Based on FUSE study (Rose et al. 1998, OPUS: The FUSE Data Pipeline, > 5076 lines of code > 71% reuse of existing OPUS modules (ACS, NICMOS) Œ Major effort in functional and throughput testing 10 of 13

11 Major Science Data Processing Requirements Summary Œ Internal header specification DM-06 to document content and format of science internal header [DONE] PDB (EUDL.DAT, TDFD.DAT) defined. [DONE] Œ Keyword definitions Œ Planned values flags and indicators for Data Validation Œ Aperture definitions [DONE] 11 of 13

12 Test Data Requirements Œ Test data from detectors on ops bench expected in March 2002 and from integrated instrument in June 2002 including all science and engineering modes Œ SMGT & IPT test data must include: PMDB population and PDB definition images that enable OPUS to determine proper orientation list of possible error conditions to simulate data that simulate those error conditions enough data for throughput test 12 of 13

13 Thermal Vac Support Œ.SDI files will trigger full dataset name: ipppssoot_csiiyydddhhmmss allows reuse of SMS Œ Optical Stimulus Snapshots named the same as the image file: csiiyydddhhmmss.oss allows corresponding OSC data to populate keywords Œ Support Schedule defaults since exposures will not have PMDB support, a default support schedule will be used. Auxilliary keyword values can be supplied in a separate exposure-specific ASCII file. 13 of 13

The New HST FITS Format, the FITS Kernel and OpenIRAF

1997 HST Calibration Workshop Space Telescope Science Institute, 1997 S. Casertano, et al., eds. The New HST FITS Format, the FITS Kernel and OpenIRAF P. Greenfield Space Telescope Science Institute, Baltimore,