JWST Pipeline & Data Products

Transcription

1 JWST Pipeline & Data Products Stage 1: Ramps-to-Slopes Karl D. Gordon JWST Calibration WG Lead Space Telescope Sci. Inst. Baltimore, MD, USA Stage 2: Calibrated Slopes Stage 3: Ensemble Processing 18 May 2017 STScI Archive

2 Outline Pipeline Philosophy User Experience High level pipeline structure How the pipeline is decided and built Baseline and Optimal versions Pipeline Availability Data Products

3 Learning from Past Efforts Previous Missions Hubble, Spitzer, Herschel, etc. Ground-based observatories/instruments Especially important for integral field spectroscopy (IFU), coronagraphy, and multi-object spectroscopy (MOS) What to do What not to do

4 Pipeline Philosophy Algorithms based on community best Input from instrument teams and mode specific expert teams Overall goal is best justified algorithms Use same code for different instruments Where possible Easier to maintain Takes advantage of strengths of all teams Provide pipeline directly to community

5 User Experience Pipeline automatically run on all data Default parameters for all pipeline steps Pipeline products produced and archived Final as well as raw and intermediate products User can download and run pipeline locally Change defaults Add customized reduction steps Pipeline may require internet connection for telemetry and reference file queries

6 Pipeline versus Data Analysis Tools Pipeline Automatically runs on all data Requires no human interaction Data Analysis Tools Requires science decisions human interaction Overlaps For example, parts of the pipeline can be re-run interactively with non-default options

7 Overall Pipeline Architecture CALDETECTOR1 CALIMAGE2 CALIMAGE3 CALCORON3 CALAMI3 ARCHIVE CALSPEC2 CALTSO3 CALSPEC3

8 NIRCam Team Workflow NIRSpec Team JWST Calibration WG NIRISS Team FGS Team JWST Data Management System WG MIRI Team Coronagraphic WG Time Series Obs WG Moving Targ WG Science Software Branch

9 NIRCam Team Real Workflow NIRSpec Team JWST Calibration WG NIRISS Team FGS Team JWST Data Management System WG MIRI Team Coronagraphic WG Time Series Obs WG Moving Targ WG Science Software Branch

10 Development Plan Baseline Pipeline All instruments and all modes Provides good science Meets requirements Algorithms defined implementation in progress Implementation done by Dec 31, 2017 Final testing to happen in early 2018 Optimal Pipeline Best possible reductions Highest quality science data This is the final goal (launch + many years) Start work after baseline pipeline done and continue for the mission lifetime and beyond Will need to prioritize effort

11 Potential Optimal Algorithms In development A few examples Upgrade for slope fitting Weighted linear least squares to Generalized Least Squares Optimal Spectral Extraction Use knowledge of spatial profile Flag outliers and improve S/N of extracted spectra MIR self-calibration Correct for small drift in MIR detectors using dithered data

12 Pipeline Availability Written in python (uses astropy) Freely available Configurable Users can rerun all or part of the pipeline Users can replace specific modules Hosted on github Part of astroconda Official release early 2018 Will include high level and detailed code documentation

13 Data Products Raw data Intermediate stage data Final data Best quality from an automated pipeline browse-quality data in Spitzer terminology Flux, wavelength, and position calibrated Baseline and Optimal versions very similar Optimal have improved quality Any user can rerun the pipeline offline Changed parameters to specific steps Replace a step with a user written version

14 Imaging Data Products Mosaic of all images with the same filter Catalog of sources In sky coordinates (e.g., ra, dec) automated generation Integration level All images fully corrected with flagging of CRs, etc. In detector & sky(rectified) coordinates

15 Imaging Data Products: Example Gordon et al. (2015, PASP, 127, 696) MIRI PASP paper X

16 Spectroscopic Data Products Slit & Slitless: 2D spectral image along-slit and wavelength Includes MOS targets (each processed separately) IFU: 3D spectral cube Extracted spectrum for the target source ra, dec, and wavelength will be able to advise pipeline if source is point/extended via APT Integration level All images fully corrected with flagging of CRs, etc. In detector & sky(rectified) coordinates 3D cubes or 2D images depending on observation type

17 Spectroscopic Data Products: IFU Example Gordon et al. (2015, PASP, 127, 696) MIRI PASP paper X

18 Coronagraphic Data Products Coadded images In detector coordinates After PSF subtraction Integration level All images fully calibrated with flagging of CRs, etc. In detector & sky(rectified) coordinates

19 Coronagraphic Data Products: MIRI 4QPM Example Gordon et al. (2015, PASP, 127, 696) MIRI PASP paper X

20 Aperture Masking Interferometry Data Products Fringe parameters Closure phases and amplitudes Reconstructed images

21 Time Series Observation Data Products Photometry Aperture photometry per integration Spectroscopy Extracted spectrum per integration White-light photometry per integration Integration level Image or spectral image as appropriate fully corrected with flagging of CRs, etc. In detector & sky(rectified) coordinates

22 Interested? Talk to me or any of the many members of the JWST Cal WG Instrument team members both at STScI and external Really motivated Join the JWST Calibration WG details including discussions are on a set of confluence(wiki) pages Can provide an account so you can view them and comment (full version control)

23 Stage 2: Calibrated Slopes e lin ts pe uc Pi rod ST P ta JW a D & Stage 1: Ramps-to-Slopes Stage 3: Ensemble Processing Archive

24 Backup Slides

25 NIRCam Team Algorithm Requests NIRSpec Team NIRISS Team FGS Team Majority of Inputs Instrument Experts Extensive Ground Testing MIRI Team Coronagraphic WG Time Series Obs WG Moving Targ WG Special Case Inputs Science Experts Includes Instrument Experts

26 NIRCam Team Decisions NIRSpec Team NIRISS Team JWST Calibration WG Composed of STScI and external members FGS Team MIRI Team Coronagraphic WG Time Series Obs WG Moving Targ WG Critical Review of Inputs Identify Common Steps Encourage cross pollination of ideas Encourage cross instrument development Decide on the algorithms for development (resource constrained)

27 NIRCam Team Implementation NIRSpec Team JWST Calibration WG NIRISS Team FGS Team JWST DMS WG MIRI Team Coronagraphic WG Time Series Obs WG Moving Targ WG Composed of scientists and programmers Work with programmers to implement algorithms Defines products, supports DMS design Testing/Validation of Pipeline Feedback to JCalWG

28 NIRCam Team Code! NIRSpec Team JWST Calibration WG NIRISS Team FGS Team JWST DMS WG MIRI Team Coronagraphic WG Science Software Branch Time Series Obs WG Moving Targ WG Write the pipeline Integrate into the Data Management System (DMS) Use code delivered in earlier stages (if in python)

29 Baseline Implementation Most of the baseline is implemented Build 7.0 (Coding finished by Dec 31, 2016) Testing in the 1st three months of 2017 Pipeline stages 1 & 2 basically complete Stage 3 for imaging and AMI complete Partial completion of stage 3 for other modes Complete baseline Build 7.1 ( Coding finished by Dec 31, 2017) Completing stage 3 of pipeline Spectroscopy, Coronography, and TSO

30 1 ge CALIMAGE2 CALSPEC2 St a ge 3 St a ge 2 St a CALDETECTOR1 CALIMAGE3 CALCORON3 CALAMI3 ARCHIVE CALTSO3 CALSPEC3

31 TSO TSO Saturation Check TSO TSO Error Initialization Error Initialization TSO Superbias Subtraction First Frame Correction Reference Pixel Correction RSCD Correction Nonlinearity Correction Nonlinearity Correction Persistence Correction Last Frame Correction Dark Correction Dark Correction Jump Detection Reference Pixel Correction TSO TSO Slope Fitting Persistence Correction TSO Bad Ramp Flagging Jump Detection TSO Slope Fitting TSO TSO TSO TSO TSO TSO TSO TSO Saturation Check TSO TSO CALDETECTOR1 Data Quality Initialization NIR/MIR NIR MIR Bad Ramp Flagging TSO TSO Data Quality Initialization

32 TSO CALIMAGE2 WCS Information Background Subtraction TSO NIR/MIR Flat Field Correction TSO Flux Calibration TSO NIR Rectified 2D/3D Product MIR

33 CALSPEC2 TSO Background Subtraction Imprint Subraction NIRSpec MSA & IFU MSA Failed Open Flagging NIRSpec MSA & IFU WCS Information TSO Subwindow Extraction NIRSpec MSA/Fixed Slit NIRCam/NIRISS WFSS NIRIS SOSS Flat Field Correction Stray Light Subtraction MIRI MRS Fringing Removal MIRI MRS TSO TSO Path-Loss Correction Flux Calibration Rectified 2D/3D Product NIR/MIR NIR MIR

34 CALIMAGE3 Refine Relative WCS Background Matching NIR/MIR Outlier Detection NIR Image Combination Create Exp Level Products Source Catalog MIR

35 CALCORON3 Assemble Reference PSFs Align Reference PSFs NIR/MIR Reference PSF Sutraction Outlier Detection Image Combination Create Exp Level Products NIR MIR

36 CALAMI3 Assemble Reference PSFs Calculate Fringe Parameters Final Fringe Parameters NIR/MIR NIR MIR

37 CALSPEC3 Master Background Subtraction NIRSpec & MIRI Background Matching Point vs Extended Decision Residual Fringe Correction MIRI MRS Outlier Detection NIR/MIR Cube Creation NIR Create Exp Level Products Spectral Extraction Spectral Leak Subtraction MIRI MRS MIR

38 CALTSO3 Outlier Detection Create Exp Level Products Extract Photometry Extract Spectroscopy White-Light Photometry Imaging Spectroscopy

39 CALDETECTOR1

40 CALIMAGE2

41 CALSPEC2

42 CALIMAGE3

43 CALCORON3

44 CALAMI3

45 CALSPEC3

46 CALTSO3