SDO EVE Data Processing Overview & Data Access

Transcription

1 SDO EVE Data Processing Overview & Data Access Don Woodraska (303) October 7, 2008 SDO EVE Science Team Meeting, Woodraska 1

2 Acronyms DDS - Data Distribution System DS3 - Digital Signal 3 (or T3) ESP - EUV Spectrophotometers FITS - Flexible Image Transport System LISIRD - LASP Interactive Solar Irradiance Datacenter IOC - Instrument Operations Center MEGS - Multiple EUV Grating Spectrograph SAM - Solar Aspect Monitor SOC - Science Operations Center (Comprised of IOC and SPOC) SPOC - Science Processing & Operations Center TLM - Telemetry WSC - White Sands Complex SDO EVE Science Team Meeting, Woodraska 2 of 16

3 EVE-SOC Interface Overview S-band Ka-band SDO in Geosynchronous Orbit MOC at GSFC DDS at WSC RT HK (S-band) near-rt science data files (Ka-band) Handshake files Retransmitted science data files EVE SOC at LASP RT HK (S-band) Commands Daily HK file Planning data Retransmissions SPOC ENG Data IOC Science Data Products WWW Eng Data USC RT HK SDO EVE Science Team Meeting, Woodraska 3 of 16

4 Data Processing Design Drivers Requirements Each SOC shall provide and coordinate all instrument science data collection, archiving, analysis, and science product generation / distribution capability for the instrument science team. The EVE SPOC shall receive and process EVE science data into solar irradiances. The SPOC shall archive all raw EVE science data, derived data products, and processing reports for the mission. EVE quick-look space weather products must be publicly available within 15 minutes of receipt under nominal operational conditions. The SPOC shall provide data to the scientific community in a searchable, easily accessible form. Instrument data rate is 7 Mbps or ~27 TB raw data annually Provide the sole archive for EVE data Special request received after design was mostly implemented NOAA desires data latency of < 1-minute (from photon detection) Producing spectra from CCD images is not possible 3 minute DDS latency, would need to build a new antenna/ground system Partly possible with photometers An S-band packet contains one set of 4-Hz samples every 10 seconds SDO EVE Science Team Meeting, Woodraska 4 of 16

5 EVE Science Channels Channel MEGS-A, SAM Description Multiple EUV Grating Spectrograph-A One grazing-incidence grating λ-range: 5-37 nm, Δλ~ nm Integration time: 10 seconds Solar Aspect Monitor (SAM) Pinhole camera, separate filter mech λ-range: nm, variable.1-1 nm MEGS-B, MEGS-P Multiple EUV Grating Spectrograph-B Two normal-incidence gratings λ-range: nm, Δλ~ nm Integration time: 10 seconds MEGS Photometer Lyman-alpha from first grating λ-range: nm Integration rate: 4 Hz ESP EUV Spectrophotometers λ-range: 0.1-7, 17-19, 25.6, 30.4, 36 nm + dark Integration rate: 4 Hz SDO EVE Science Team Meeting, Woodraska 5 of 16

6 Science Product Subsystem DDS Ka-band TLM data to SPOC Level 0A Legend Code Data Level 0A to 0C are time-critical space weather data flow Levels 1-3 are not near-rt Public Products Level 0A, 0B, 0D complete Most of level 1 is complete S-band TLM data from IOC RT Photo. Test Patterns Level 0C Level 0C Data Dissemination Subsystem VCDU files Level 0A Level 0B Level 0B Level 1 Level 0D Level 0D Level 3 Level 2 Level 2 Level 1 ESP, SAM, MEGS-P only Level 3 Reprocessing IOC ENG Cal data FOV & Cruciforms, Flatfields, Darks, Higher Orders, etc SDO EVE Science Team Meeting, Woodraska 6 of 16

7 Data Processing Timing and Version Numbering Space weather (S and Ka) Seconds latency and < 15 minutes Routine processing At 14 UT, the previous days routine processing begins (Level 1-3) Completed L0 data files from the MOC unavailable until this time. Routine reprocessing After 30 days, processing is repeated Holes are filled, we have all the data we will ever get Calibration/degradation corrections can be interpolated, higher quality products New Version release reprocessing When significant changes are made to the calibration corrections, or processing algorithms that impact the quality of the data, a new processing version number is created. Requires regenerating the entire mission The majority of compute time is spent doing this. Version Plan is subject to change Version 1 is the pre-flight, testing version Version 2 will be released on launch day Version 3 is expected after the first rocket calibration is applied (L+4 months) Version 4 is expected after the 2nd or 3rd rocket calibration is applied (L+12 months) Other versions SDO EVE Science Team Meeting, Woodraska 7 of 16

8 EVE Data System Highlights Spans 6 separate networks (not including ops) SDONet, UVNet, BlueNet, GreenNet, EVENet, DMZ 48 compute nodes, 10 servers, 4 workstations Dedicated DS3 to WSC (the first fiber connection to the building) 33 TB of disk, 170+ TB dedicated online tape storage capacity Sufficient hardware redundancy to minimize down time SDO EVE Science Team Meeting, Woodraska 8 of 16

9 Science Data Access Navigable web pages to download data product (FITS) files Experts Full access to all Level 2 and 3, Level 1 ESP, SAM, MEGS-P Scriptable (curl or wget) downloads possible (FTP or HTTP) Novices On-line tool to display FITS file content in ASCII Browse tools? EPO? SpWx pages with 1-minute updating plots and files Latest ASCII photometer irradiances S-band, < 1 minute ESP Flare location estimates Latest ASCII spectrum Ka-band, < 15 minutes QuickTime and a decompressor are needed to see this picture. QuickTime and a decompressor are needed to see this picture. SDO EVE Science Team Meeting, Woodraska 9 of 16

10 Science Data Discovery Two complimentary databases LISIRD for daily averages (contains multiple missions, tools) EVE Custom: 1-minute data with links to data product files Daily insertion of data, NO space weather Database can support search capability for finding files based on: Time range, possibly flare flags, quality flags Need community input on what is desired Support automated external queries Database and appropriate tools can provide interactive browse capability Event discovery, time scaling (pan / zoom) Flares, anything else? Tables contain selected bands, possibly lines, all 1-min avg. ESP, MEGS-P bands, MA1, MA2, SAM, MB bands, bg estimates, AIA bands, GOES-R bands, Q EUV, SEM bands?, GOES-13 bands? Possible lines (integrated, including background): MEGS-A: 15.0 O VI, 17.1 Fe IX, Fe X, Fe XI, 19.5 Fe XII, 25.6 He II, 28.4 Fe XV, 30.4 He II, Fe XVI MEGS-B: 36.8 Mg IX, 58.4 He I, 97.7 C III, H I, O VI Any other lines or bands useful for event discovery? Are movies of spectra useful? Mission-length? Daily? SDO EVE Science Team Meeting, Woodraska 10 of 16

11 SEE Movie example QuickTime and a decompressor are needed to see this picture. SDO EVE Science Team Meeting, Woodraska 11 of 16

12 Data File Details All data products are either ASCII or FITS Reader software ASCII files are readable by any text reader Some may have very long lines, where each line pertains to one date/time Fixed length? FITS standard readers FV (FITS View) is a graphical tool for opening/editing FITS files IDL software packages are available from several sources (like SolarSoft) EVE processing uses high-level mrdfits.pro/mwrfits.pro available from the GSFC IDL Astronomy Library (we also use the low-level cfitsio library) Header keywords are sparsely used Any requests for header keywords? Rather than treating metadata (like time, date, resolver, etc) as ASCII strings that need to be extracted from the header, it is included with the data and treated like data Limitations of FITS No convention for nested structures, so all internal structures are only 1 level or flat Level 1-3 files are all FITS Files contain an empty primary HDU (for compatibility), and a binary table A row in the table corresponds to a time Columns are data fields (scalars or arrays, no pointers, no objects, no structures) Each row pertains to one date/time Fixed length? variable number of integrations depending on clock drift, power cycles, Ka-comm switches, etc. SDO EVE Science Team Meeting, Woodraska 12 of 16

13 Example: fv Properly formed FITS files reserve the first (#0) HDU for image data The second (#1) HDU is the binary table *** Column names *** TAI, YYYYDOY, SOD, SLIT1_IRRADIANCE, SLIT2_IRRADIANCE, SLIT1_STDEV, SLIT1_PRECISION, SLIT1_ACCURACY, SLIT1_FLAGS, SLIT2_STDEV, SLIT2_PRECISION, SLIT2_ACCURACY, SLIT2_FLAGS, ALPHA, BETA, GAMMA, INT_TIME, RESOLVER, OBS_FLAGS SDO EVE Science Team Meeting, Woodraska 13 of 16

14 Spectral Product Resolution Comparison Uncalibrated rocket spectra are shown for MEGS-A and MEGS-B Plots are equivalent sampling to Level 1, 2, and 3 data product sampling. MEGS-A Level 1: 0.01 nm MEGS-B Level 1: 0.02 nm Level 2: 0.02 nm 5-bins per resolution element MEGS-A 2-bin avg MEGS-B same as Level 1 Level 3: 0.02 nm Additional 0.1 and 1.0 nm sampled daily average will be created in Level 3 in mission-merged files (and in LISIRD) All products are W/m^2/nm SDO EVE Science Team Meeting, Woodraska 14 of 16

15 Science Product Availability S-band latency is seconds Photometer file combining ESP & MEGS-P A single 4 Hz sample every 10-seconds for all photometer bands Time span? 15 minutes? 60 minutes? 24 hours? Ka-band latency is > 3 minutes and < 15 minutes Details of contents will be discussed in Frank s talk Routine products available after 14 UT for the previous day. Best products available after 30 days SDO EVE Science Team Meeting, Woodraska 15 of 16

16 Summary EVE SPOC contains sufficient hardware redundancy to produce products under most common failure scenarios EVE science processing will create files with several latencies Availability ranges from seconds to 30+ days File formats are ASCII and FITS Many standard readers exist for FITS, we use mrdfits.pro and fv Level 2 maintains the instrument spectral resolution Time cadence is maintained for CCD data Online accessible database (2) One for daily average data (LISIRD) One for 1-minute samples of bands and lines SDO EVE Science Team Meeting, Woodraska 16 of 16

17 Example of a Prototype Flare Catalog Possible EVE Flare Catalog Content Convention: All times UT To merge NOAA edited events info and help ensure the most complete EVE data, the catalog should only be updated daily with a 2-day lag NOAA info from the edited events report Year/Doy mo-dd, start time hhmm, peak time, stop time, Class, Long, Lat, Region, Event EVE Info ESP quad, ESP flare Long Lat, Raw SAM average image,? Will NOAA create an EUV flare index before SDO launches? SDO EVE Science Team Meeting, Woodraska 17 of 16