WISE Science Data Center

Transcription

1 WISE R. Cutri Infrared Processing and Analysis Center RMC: 28-1

2 MOS Functional Block Diagram TDRSS WHITE SANDS COMPLEX TDRSS Uplink ENGINEERING OPERATIONS SYSTEM (JPL) AND ESMC EOS OPERATIONS Command Generation and Transmission FLIGHT SYSTEM ENGINEERING OPERATIONS MOM Review and Uplink Approval Command Load Generation WISE SCIENCE DATA CENTER (IPAC) Real Time and Stored Eng. TLM Science Survey Data Science Survey Data Console Ops Health/Safety Monitoring Data Capture Eng/Data Processing Mission Ops Manager Review/ Approval Sequence Generation SCIENCE OPERATIONS SYSTEM (UCLA) Science Review/ Approval Instrument Engineering Requests Quick Look Pipeline Processing Quick Look QA Phased Pipeline Processing DATA ARCHIVE SYSTEM (IRSA) WISE Project Database Data Receipt/ Verification On and Offsite Storage Pipeline QA TDRSS/WISE Ephemeris Instrument Engineering Requests Flight Rules Checker TDRSS Scheduling S/C Engineering Requests Science Survey Planning File Storage and Staging Final Product Generation Public Catalog and Image Atlases (IRSA) Instrument Engineering Support (SDL) S/C Engineering Support (BATC) ENGINEERING SUPPORT MISSION OPERATIONS AND GROUND DATA STSTEMS RMC: 28-2

3 WSDC Responsibilities Science Data Processing Convert raw imaging data into a photometrically and astrometrically calibrated Image Atlas and extracted Source Catalog Generate intermediate data products to support mission requirements Science Data Quality Assurance Quicklook QA feedback for on-orbit performance (i.e. scan mirror synch) Overall science data QA for survey planning and data product generation Science Data Archiving and Distribution Raw data archive during mission Long-term living archive for primary and intermediate data products Serve WISE science data products to project team, astronomical community and general public along with user s guide documentation and descriptive analysis RMC: 28-3

4 Driving Requirements (1 of 2) Higher Level Requirement L3MOS-366,374 (L1PP-34) L3MOS-355, 363 (L1.5SRD-50) L3MOS-417 (L1PP-010) L3MOS-363, 418 (L1PP-011) L3MOS-376 (L1PP-012) Level 4 Requirement Compliance Verification Method Data Release: The WSDC shall release to the public an image atlas and source catalog covering the full survey area within 17 months after the end of on-orbit data collection. Preliminary Data Release: The WSDC shall release to the public a preliminary image atlas and source catalog covering at least 50% of the surveyed area within 6 months after the end of on-orbit data collection Catalog Reliability: The reliability of sources in the WISE catalog shall be >99.9% for sources in unconfused regions detected in a single band with SNR>20. Catalog Completeness: The WISE catalog shall be >95% complete for sources in unconfused regions detected in a single band with SNR>20. Photometric Accuracy: The relative photometric accuracy of the WISE catalog shall be better than 7% in each band for unsaturated point sources in unconfused regions with signal to noise ratio greater than 100. Demonstration Demonstration Test: Comparison with external truth tables ; detection confirmation Test: Detection repeatability in deep coverage areas (e.g. the ecliptic poles) Test: Photometric repeatability; stellar color stability over sky RMC: 28-4

5 Driving Requirements (2 of 2) Higher Level Requirement L3MOS-370 (L1PP-013) L3MOS-420 (L1PP-4) L3MOS-272 Level 4 Requirement Compliance Verification Method Astrometric Accuracy: The RMS positional accuracy of sources in the Catalog shall be <0.5 with respect to the 2MASS Point Source Catalog Photometric Sensitivity: Source measurements in the WISE catalog shall achieve a signal to noise ratio of 5 or more in regions unconfused by Galactic sources, but including the effects of extragalactic source confusion, on point sources with fluxes of 0.12 mjy at 3.3 micrometers, 0.16 mjy at 4.7 micrometers, 0.65 mjy at 12 micrometers, and 2.6 mjy at 23 micrometers. Quicklook Quality Assurance: Approximately 3% of the science data from each orbit shall be processed and data quality summary reports posted to the WISE internal web site within 24 hours of receipt at the WSDC. L3MOS-286 Single Orbit Data Processing Latency: Within 3 days of receipt of a given data set, the WSDC shall process all single orbit data through the first stage of pipeline processing. L3MOS-381 Archive Longevity: The WSDC shall make the WISE Image Atlas and Source catalog available to the astronomical community in collaboration with the Infrared Science Archive to ensure long term availability and to insure interoperability with other NASA mission archives. Test: Comparison with 2MASS PSC and other astrometric catalogs (e.g. UCAC) Test: Photometric repeatability; Colors of normal stars, galaxies Demonstration Demonstration Demonstration RMC: 28-5

6 System Design Considerations Driving factors High data rate (51GB/day uncompressed), large data volume (11TB raw, ~80TB processed) Aggressive data release schedule (EoO+6m & EoO+17m) Strict Catalog reliability and completeness requirements Intellectual heritage from IPAC support of other missions (2MASS, IRAS, WIRE, Spitzer, GALEX) Demonstrated all aspects of WISE data processing, archiving and distribution tasks on past and existing missions Actual code re-use probably <10%, but many of the same engineers will develop subsystems similar to the ones they developed for other missions Take full advantage of excellent IPAC infrastructure: administrative, systems, network, archives RMC: 28-6

7 System Design Strategy High data rate, large volume Highly-automated, high-throughput pipeline processing system with very little opportunity for human intervention Extensive, automated quality assurance system Sophisticated data management system Release schedule Planned two stage data processing and product release: Can t get it right the first time Allows early data release, and time to apply lessons learned from first-stage processing Plan for rapid response to in-flight performance Ground testing cannot capture all performance aspects Parameter file control to minimize complete software rebuilds Modular software development Subsystems joined with Executive wrapper Enables flexibility in development and parallelization Strict interface control RMC: 28-7

8 Key External Interfaces GDS/EOS (receive and acknowledge) Spacecraft H/K and metadata Spacecraft ephemeris GDS/White Sands (receive and acknowledge) High-rate science data Science Operations (receive and send) Science observation plan Science data processing QA to close loop on survey planning Project Team (send) Intermediate processed data products and working databases Astronomical Community (send) Primary mission deliverables (Image Atlas and Source Catalogs) Derived data products (TBD) Data product documentation (on-line Explanatory Supplement) RMC: 28-8

9 Functional Block Diagram RMC: 28-9

10 WSDC Subsystems Ingest System Receive and verify high-rate Level 0 science data from White Sands, H/K and ephemeris data from EOS, Science observation planning from SOC (UCLA) Decompress and assemble completed frames, collect meta-data, integrate into FITS-format file Write raw data to tape archive (3 copies, 1 off-site), H/K to engineering archiving Data Reduction Pipelines Industrial strength automated systems to convert raw imaging and spacecraft H/K data into calibrated images and extracted source lists Orbit pipeline derives and applies instrumental calibration on frames from single orbit, extracts and characterizes high SNR sources, identifies artifacts, derives and applies astrometric and photometric calibrations, identifies known solar system objects. Output to frame image and extracted source Working Databases. Quicklook pipeline subset of orbit pipeline, but operates on ~5% of orbit data with scaled-back functionality. Provides image quality (scan mirror synch) and coverage assessment of each delivery within 24 hours of receipt. Multi-orbit pipeline combines images from multiple orbits, extracts and characterizes all sources, identifies artifact detections, updates calibrations. Output to image and extracted source Working Databases. RMC: 28-10

11 WSDC Subsystems Quality Assurance System Generate concise reports summarizing science data quality, based closely on successful 2MASS QA system Automatically reads QA summaries generated by INGEST, PIPELINES, FPG subsystems and assembles into condensed reports that present science-based performance metrics Web-based reports, with capability to drill-down to detailed image, graphical and tabular data Specialized tools to allow interactive QA QA scientists review each report and confirm/update automatically assigned quality score Final Product Generator Constructs WISE Preliminary and Final Image Atlas and Source Catalog from multi-orbit Image Archive and extracted source Working Databases Analysis and DB-intensive activity. Not an automated system Examine QA and select release-quality Images Generate value-added columns in Working Database. Run source selection DBMS query on Working Database to create Catalog. Analyze and validate final products RMC: 28-11

12 WSDC Subsystems Archive/Distribution System Archive raw and processed mission data. Serve Image Atlas, Source Catalog and mission metadata to WISE project team and astronomical community. Raw telemetry archive from INGEST. 3 copies to tape, on- and off-site storage. Mission metadata and engineering data in small, flat-file systems. Rapid access for processing control (from INGEST) Image data in file system and/or pixel database (from INGEST and PIPELINES). Extracted source information, image metadata, orbit metadata and QA results in DBMS integrated into Infrared Science Archive (IRSA) at IPAC WISE databases integrated in IRSA, so delivery to data center is implicit Partnership with IRSA: NASA s designated archive center for infrared and submm missions IRSA infrastructure optimized for handling and serving massive datasets Development minimized - use existing and augmented middleware, develop thin-layer interfaces IRSA/WISE web-based tools are primary interface to processed and final data products Public interfaces will be a subset of Project interfaces. Allows thorough vetting of public interfaces before release RMC: 28-12

13 Internal Data Flow and Operational Cycle RMC: 28-13

14 WSDC Implementation WSDC will be an independent task at IPAC Analogous to 2MASS, NHSC, IRSA, NED IPAC/WISE Task Lead reports to IPAC Executive Director and IPAC Manager Administrative, facilities and common infrastructure cost shared with all projects WSDC will use dedicated hardware that is integrated into IPAC network structure Resources and expertise shared with other IPAC tasks along with SSC, MSC which reside under a common administrative umbrella at IPAC Results in a substantial savings and improved quality because of access to expertise and synergy with other IPAC projects, including the Spitzer and Michelson Science Centers RMC: 28-14

15 System Development Phase-In Version 0-4/2007 Supports simulation, early data flow development, basic data analysis, algorithms and detector testing Version 1-4/2008 (instrument cal. - 1 month) Supports detailed instrument calibration data analysis including correlation with H.K and ancillary engineering data Version 2-9/2008 (end-end test - 1 month) Supports full telemetry input, end-to-end testing (MST) and WSDC product life cycle Version 3-3/2009 (launch - 3m) Supports ORT, Launch, IOC and on-orbit operations. Produces preliminary data release products. Version 4-4/2010 (end of on-orbit ops. + 4m) Supports final data processing, quality assurance and final product generation. Produces final data release products. RMC: 28-15

16 WSDC Concerns Surprises in actual instrument performance Mitigation: Modular S/W system with extensive parameter file control. Detector ground test. Lessons learned from Spitzer. Aggressive preliminary release schedule allows very little time for validation Mitigation: Relaxed requirements for preliminary release. Design for automated QA concurrent with processing, tied to science metrics (2MASS heritage) Inter-institutional network compatibility, integrity and reliability Mitigation: Comprehensive ICD. Early and frequent interaction between institutional system groups. Mission scenario testing. 30 day data buffer at White Sands. Handling deliveries of fragmented, out-of-order, and/or incomplete data frames Mitigation: Design for expected fragmentation. Insure Flight/Ground ICD has all necessary information to reassemble fragmented files. Final product analysis and validation plan and release readiness procedure Mitigation: Work with Project and Science Team to define this process. Make sure Science Team is funded and committed to participate in validation of final products. RMC: 28-16

17 Summary WISE data processing, archiving and distribution tasks are challenging, but we know how to do them The WSDC processing and archiving system design is based closely on the proven 2MASS system Scope and functionality of WISE data processing, quality assurance, final product generation and archiving tasks are similar to those of 2MASS. 2MASS photometric/astrometric accuracy, completeness and reliability requirements were more stringent than those for WISE, and all were met or surpassed. WISE instrument/observing environment will be much more stable than 2MASS (no weather!) WSDC Level 0 data ingest system design draws upon operational GALEX system (not strictly an IPAC institutional activity, but supported by IPAC staff detailed to GALEX) RMC: 28-17