Experiences From The Fermi Data Archive. Dr. Thomas Stephens Wyle IS/Fermi Science Support Center

Transcription

1 Experiences From The Fermi Data Archive Dr. Thomas Stephens Wyle IS/Fermi Science Support Center

2 A Brief Outline Fermi Mission Architecture Science Support Center Data Systems Experiences GWODWS Oct 27,

3 What is Fermi Large Area Telescope (LAT) Two Instruments on board Large Area Telescope (LAT) (PI Peter Michelson, Stanford University; managing institution: SLAC) Primary instrument, gamma-ray (pair production) telescope 20 MeV >300 GeV Scanning mode sees 20% of the sky all the time; all parts of sky for ~30 min. every 3 hours. This is the primary mode of operation. GLAST Burst Monitor (GBM) (PI Charles Meegan, NASA/MSFC) Context instrument 8 kev 30 MeV GLAST Burst Monitor (GBM) Observes entire unocculted sky all the time searching for gamma ray bursts 5 year mission (10 year goal) no consumables on-board to be depleted GWODWS Oct 27,

4 Fermi Mission Data Flow Fermi Mission Data Flow GLAST MISSION ELEMENTS GPS µ sec Large Area Telescope & GBM Fermi Spacecraft Telemetry 1 kbps - DELTA 7920H - GN - S TDRSS SN S & Ku Mission Operations Center (MOC) Schedules Fermi Science Support Center LAT Instrument Science Operations Center White Sands HEASARC GSFC GRB Coordinates Network Alerts Schedules GBM Instrument Operations Center Data, Command Loads GWODWS Oct 27,

5 Fermi Science Support Center Archive Data ( GBM Daily data (CTIME and CSPEC) Trigger and burst data (Time Tagged Events) All provided through BROWSE LAT Photon (and extended) data Spacecraft position and pointing and live time history Provided through a custom web server Ancillary data Catalogs Background models Others Provided through BROWSE, FTP and web pages Science Tools GWODWS Oct 27,

6 Automated Processing All data received (from MOC, GIOC, or LISOC) is identified and processed by an automated data ingest pipeline About 30 different data products received Some are science data, others operations data for commanding the spacecraft Built using OPUS as a framework Custom Perl scripts and C++ programs do the actual data processing Data Validation Data archiving Creation of BROWSE tables Ingest into custom data server Pages ( s) sent out when problems are encountered GWODWS Oct 27,

7 LAT Data Server MySQL Database Web Interface Queue Manager Photon Database Event Database Ingest System HEASARC BROWSE GSSC Internal Tools Pointing and Livetime History Database GWODWS Oct 27,

8 Photon Database Details MySQL server Control Node Query status Web Interface Master Data Copy Node Status Query results Query requests New data QueueManager Ingest System Files to search Query parameters Search Nodes Local Data Copies GWODWS Oct 27,

9 Data Queries Over Time GWODWS Oct 27,

10 Data Served GWODWS Oct 27,

11 Fermi Data Experiences We ve been providing data to the general community for just over two years now (three years to the LAT Collaboration). No serious issues with the data servers Things that went well Data Challenges Training Workshops Things that have caused headaches Multiple data sources Data Reprocessing Data Monitoring and validation Science Analysis Tools GWODWS Oct 27,

12 Data Challenges Conceived as a method to check the usefulness of the data, data servers, and science analysis software before launch Staged testing 1 day (1 week) data 55 days data (one procession period of the spacecraft orbit) 1 year of data Analysis tools got more and more sophisticated as we went Data server was used to provide data for all three data challenges Provided experience ingesting and serving the data and gathered feedback on usefulness of the data server. Surprisingly very few changes were suggested. GWODWS Oct 27,

13 User Training Regularly hold workshops to teach users about the Fermi data and Science Tools Held around the country, usually in the summer Gives people hands on experience with experts there in person to answer questions Getting the data Using the various science tools Provides valuable feedback to FSSC on variety of topics Ease of use of server and tools Issues with the various OS s the tools are supported on Documentation GWODWS Oct 27,

14 Data From Every Direction FSSC receives data from multiple sources Detailed File Format Description document describes the data we should be receiving Almost a day doesn t go by when some data received violates this design. Have to decide if the violation is important or not Does it impact science or operations Is it more cosmetic Probably >90% of the time this is data from the GBM team Under funded to reliably build automated data processing system A lot of processing still done by hand. GWODWS Oct 27,

15 Data Reprocessing/Reingest Complete reprocessing of the entire mission data set for a given data type. Has happened three times now for the LAT data, and at least once for the GBM data Primarily this is a huge bookkeeping exercise Also have to monitor resource usage Bandwidth Processing power Disk Space Requires a bit of data validation work Something we need to be better about automating (my current project) GWODWS Oct 27,

16 Data Validation There are so many ways for the data to get messed up This needs to be as automated and resource light as you can make it Needs to work on lots of data Needs to run as often as possible Things that have bit us Missing data (gaps) Duplicate data (double ingests) Corrupted data (never did figure this one out) Need good tools for correcting errors as quickly and automatically as possible This is something we are still working on at the FSSC GWODWS Oct 27,

17 Science Tools The Fermi mission provides a set of Science Analysis Software tools for analyzing Fermi data Primarily tools for the LAT data GBM data can be processed using already existing tools such as Xspec Supposedly a single tool set for the instrument team and the general scientific community Suffered (still suffers?) from improper division of responsibility Instrument team responsible for tool development FSSC responsible for dealing with user issues GWODWS Oct 27,

18 Backup Slides

19 Large Area Telescope (LAT) Data Not images event lists Three different types of data Photon Data (most used dataset ) Events that have the highest probability of truly being photons and which have well defined Instrument Response Functions Table of detected photons and their properties (RA, Dec, Energy, Time, etc). Event Data Superset of Photon data Includes more events that are probably particles (e -, e +, p +, etc.) More data columns (x10) than the basic photon data Spacecraft Data Position and attitude history Instrument mode Use in calculating livetime and exposure of instrument GWODWS Oct 27,

20 Large Area Telescope (LAT) Data LAT Characteristics Large FOV (~2.4 sr = 1/6 th of the entire sky) Large energy range (30MeV to >300 GeV) with energy dependent PSF Always on sky survey operation Entire sky every two orbits (~3 hours) Any given position observed for about 30 minutes during the two orbits Constantly changing instrument response for a given position on the sky Photon Data Rate Currently coming in at about a rate of 4.4 Hz 120 million per year Represents 1% of total data rate New data available every 2-3 hours GWODWS Oct 27,

21 Issues for Serving LAT Data Continuous data stream (Define an observation please.) We re looking everywhere all the time The concept of an observation has no meaning in a traditional sense Source Confusion (Is this your photon or mine?) Energy Dependent PSF 3.5 at 100 MeV GeV This mean the sources overlap (especially at low energies) Analysis requires accounting for all nearby sources as well as the one you are interested in Usually need from a radius region around your source ~2% of the entire sky. GWODWS Oct 27,

22 System Design Considerations Continuously updating data stream Data downlinked from the spacecraft every 3-4 hours Processed data arriving from the LISOC every 2-4 hours Want fast response times to queries Loosely modeled on HEASARC Browse system NASA interface used for other high energy mission data Only stores metadata and provides immediate query response with lists of available data Delay tolerance for web response usually < 10s Need to provide rapid feedback on status Typical queries will be for some region of the sky over a given time period (possibly the entire mission) Three different types of coordinated data served GWODWS Oct 27,