NOAA Integrated Ocean Observing System (IOOS) Data Integration Framework Richard Crout NOAA NWS National Data Buoy Center

Transcription

1 NOAA Integrated Ocean Observing System (IOOS) Data Integration Framework Richard Crout NOAA NWS National Data Buoy Center GCOOS Board of Directors Meeting Corpus Christi, TX 19 August 2008

2 Outline NOAA s Integrated Ocean Observing System Data Integrated Framework Challenge Solution - DIF Goals and Objectives Approach Coordination Web Services Status Validation and Verification Regional Implementation Future Challenges

3 NOAA IOOS Is modernizing the way NOAA collects, shares, and uses ocean information Is increasing data interoperability across NOAA and the regions and efficiency of operations Is expediting access to data for improved decision making v To achieve these objectives, the Program focuses its contributions in two main areas: 1. Developing the IOOS Data Integration Framework (NOAA DIF-to-National DMAC) 2. Managing the regional IOOS partnership

4 NOAA IOOS Program Focus Standards Develop and maintain a process to document, coordinate, evaluate and accept data standards Regions Invest in Regional Associations efforts that contribute to the definition and implementation of IOOS Data Integration Framework (DIF) Conduct a limited scope implementation of IOOS within NOAA as a proof of principle and to gather lessons to inform the larger IOOS effort

5 Data Integration Framework (DIF) Months 0-12 Month 18 Month 24 Integration of 5 IOOS Core Variables NOAA 5 Core Variables Integrated Variable Ingest for Select Data Products Test & Evaluation Month 36 Benchmarked Product Improvements for Operational Use Temperature Salinity Sea Level Currents Color Hurricane Intensity Model Hurricane Intensity Model Enhanced decision support through: NOAA IOOS Data Integration Framework Data Standards Systems Engineering & Standards Development Data Access & Exchange Coastal Inundation Model Harmful Algal Bloom Model Integrated Ecosystem Assessment Systems Engineering: Product Enhancement Test & Evaluation Verification & Validation Coastal Inundation Model Harmful Algal Bloom Model Integrated Ecosystem Assessment Integrated information services for NOAA programs Identify observation gaps Validated enhanced data products NOAA s Data Integration Framework NOAA MISSION OBJECTIVES Temperature Salinity Sea Level Currents Color External sources of 5 Core Variables (consistent with NOAA standards) Future State: Regional-coastal data integration for Regional scale data and information products and services REGIONAL COASTAL NEEDS

6 IOOS Functional Areas = Data Discovery, Access, Transport (encodings, metadata, catalog services, access services, conversion services) Observing Systems Data Management and Communications (DMAC) Modeling and Analysis = Provider (observing system, data archive) = Customer (decision support tool, analysis tool, generic client)

7 Integration Challenge multiple, unique, independent data requests PLATFORM text Data Distribution Data Transfer/Extract/Load NOAA Ships CREIOS ARGO Profiling AOML ARGO Drifting Buoys C-MAN (1) * * * 10 distinct data requests 10 algorithms to convert to model format 10 different load functions Data provider changes have ripple effects Drifting Buoys VOS (xbt) Weather Buoys NDBC C-MAN (2,,n) Weather Buoys (1) * * HABS Tropical Moored Buoys Weather Buoys (2,,n) * C-MAN SWMP PORTS NERRS CDMO SWMP PORTS * * Inefficient Costly processing Time consuming Stovepipe highlight NWLON CO-OPS NWLON * SWIM NCDDC SEAWIFS * OSDPD Satellites COAST WATCH

8 Integration Solution DIF content and transport standards/services PLATFORM Data Distribution AUGUST text 2008 OCTOBER 2008 Modeling & Analysis NOAA Ships CREIOS AOML DIF ARGO Profiling Drifting Buoys VOS (xbt) Weather Buoys Tropical Moored Buoys NDBC DIF DIF Data Request HABS C-MAN SWMP PORTS DIF NWLON CO-OPS DIF SWIM NCDDC OSDPD Satellites COAST WATCH DIF

9 What is the Data Integrated Framework? First spiral of IOOS development Core IOOS project Limited scope Developing the IOOS Data Integration Framework DMAC functions Common data-sharing infrastructure Agreed-upon standards Aggregation services Web services Implement at selected provider & customer sites Focus on several core variables Standardize on small number of services & encodings Adopt, adapt, expand, develop Evaluate in FY 2010 (Graphic by i3 Aerospace Technologies Pty Ltd used with permission)

10 NOAA IOOS Data Integrated Framework Validate premise: data integration to measure improvements System engineering approach Functional Requirements from identified customers Concept of Operations Document Design Document in progress Architectural principles Service oriented architecture Distributed services operated by data stewards Adopt existing standards and specifications Goals and Objectives

11 Documentation Systems Engineering Approach NOAA IOOS Program Data Integration Framework (DIF) Master Project Plan Version 1.0 September 12, 2007 NOAA IOOS Program Data Integration Framework (DIF) Functional Requirements Document Version 1.0 September 17, 2007 NOAA IOOS Program Data Integration Framework (DIF) DIF As-Is Baseline Systems Document Version 1.0 September 19, 2007 NOAA IOOS Program Data Integration Framework (DIF) Concept of Operations Version 1.0 April 25, 2008 Master Project Plan Version 1.0 9/12/07 Functional Requirements Document Version 1.0 9/17/07 DIF As-Is Baseline Systems Document Version 1.0 9/19/07 Concept of Operations Version 1.0 4/25/08

12 NOAA IOOS DIF Office of National Environmental Satellite, Data, and Information Services (NESDIS) CoastWatch National Climate Data Center (NCDC) National Ocean Data Center (NODC) National Geophysical Data Center (NGDC) National Coastal Data Development Center (NCDDC) Office of Satellite Data Processing and Distribution (OSDPD) National Marine Fisheries Service Southwest Fisheries Science Center (SWFSC) National Ocean Service Coastal Services Center (CSC) National Centers for Coastal Ocean Science (NCCOS) Center for Operational Oceanographic Products and Services (CO-OPS) National Weather Service National Data Buoy Center (NDBC) National Centers for Environmental Prediction (NCEP) Oceanic and Atmospheric Research Pacific Marine Environmental Laboratory (PMEL)

13 IOOS Data Integration Framework Coordination CONOPS (completed) NESDIS/ORA NMFS/SWFSC NWS/NDBC NOS/CSDL NOS/CSC NWS/NCEP NWS/EMC Functional Requirements (completed) NESDIS/NCDDC NOS/CSC NWS/NDBC NMFS/SWFSC NESDIS/NGDC NWS/NCEP OAR/PMEL NOS/CO-OPS Data Standards (completed) NESDIS/NCDDC NESDIS/NGDC OAR/PMEL NOS/CSC Harmful Algal Blooms NOS/NCCOS NOS/CO-OPS NWS/NDBC Coastal Inundation NWS/TPC NOS/CO-OPS Integrated Products Team Chair Charles Alexander NOAA IOOS Program National Weather Service (6) National Environmental Satellite Data and Information Service (6 + 1) National Marine Fisheries Service (2) National Ocean Service (9 + 3) Oceanic and Atmospheric Research (1 + 1) Design NOS NWS NESDIS NOS NMFS METADATA NESDIS/NGDC NESDIS/NCDDC Web Services and Data Encodings Chair - Jeff de La Beaujardiere NOAA IOOS Program NESDIS (5) NWS (2) NOS (4) OAR (1) NMFS (1) RA s (5) Image Matters

14 DIF Implementation Schedule Developing the National IOOS Data Integration Framework Schema and Integration Final Report May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb OSMC Observing System Monitoring Center XML Entensible Markup Language DCS Data Content Standard WMS Web Map Standard 5/12/2008 Deliver XML Schema NOAA IOOS Program 6/12/2008 Integrated Currents CO-OPS & NDBC 6/26/2008 DCS Color NCDDC 8/18/2008 Integrated Sea Level Temperature, Salinity, Winds, & Waves CO-OPS & NDBC 9/25/2008 Integrated Color CoastWatch 10/31/2008 Integrated PORTS Currents (profile) CO-OPS 9/30/2008 Integrated HF Radar NDBC Global and Coastal Point Obs OSMC 11/30/2008 WMS Services Available NDBC 12/31/2008 PORTS Survey Currents (profile) CO-OPS Global Profile Obs OSMC May Jun Jul Aug Sep Oct Nov Dec Schema and Integration

15 IOOS Data Integration Framework Data Sources Gridded Data KULA01 CWAO CREX++ T A008 D EUREKA //// //// Imagery Data In-Situ Data Need for three services! Open Geospatial Consortium (OSG) provides them.

16 Recommended Web Services and Data Encodings Data Type Web Service Encoding In-situ data (buoys, piers, towed sensors) Gridded data (model outputs, satellite) OGC Sensor Observation Service (SOS) OpenDAP and/or OGC Web Coverage Service (WCS) XML based on OGC Observations and Measurements (O&M) NetCDF using Climate and Forecast (CF) conventions Images of data OGC Web Map Service (WMS) GeoTIFF, PNG etc. -possibly with standardized styles OGC: Open Geospatial Consortium XML: Extensible Markup Language OPeNDAP: Open-source Project for a Network Data Access Protocol NetCDF: Network Common Data Format GeoTIFF: Geospatial Tagged Image File Format PNG: Portable Network Graphics 16

17 XML Encoding of In-Situ Data (SOS GetObservation Result) XML Extensible Markup Language Generic method for structuring text data specialized by OGC GML Open Geospatial Consortium Geography Markup Language XML that can represent any geospatial feature specialized by OGC O&M Observations and Measurements Model GML that describes the act of measuring real-world phenomena and the result of the measurement Data providers currently beta-testing this specification ( 17

18 IOOS Data Integration Framework Sensor Observation Service NDBC is implementing SOS on a map interface page Data may be retrieved as: Comma separated values (CSV) NetCDF XML Data available: Sea level Currents Salinity Sea Temperature Winds & Waves [

19 IOOS Data Integration Framework Sensor Observation Service GetObservation for ocean current vertical profile

20 DIF Data Provider Status Expected by end CY2008 Data Integrated from One Provider NDBC NDBC DAC - buoys, C-MAN, VOS, NOS NERRS, etc. Sensor Observation Service IOOS DAC - Regional observations Data Content & Encoding Standards TAO DAC - Tropical Atmosphere Ocean DART DAC - Deep-Ocean Assessment and Reporting of Tsunamis Web Coverage Service & OPeNDAP High-Frequency Radar (HFR) surface currents NDBC Data Assembly Centers (DACs) CO-OPS Sensor Observation Service Data Content & Encoding Standards National Water Level Observation Network PORTS Data Integrated from One Provider User Requests CoastWatch Web Coverage Service & OPeNDAP Data Content & Encoding Standards Satellite Ocean Color (Aqua MODIS, Terra MODIS, SeaWiFS) Data Integrated from One Provider IOOS Variables: temperature, salinity, currents, water level, winds, waves, ocean color (chlorophyll)

21 Additional Providers & Services SOS WCS/OPeNDAP NDBC WCS/OPeNDAP CoastWatch SOS Region 1 WCS/OPeNDAP SOS CO-OPS SOS Data Archives (NODC?) WCS/OPeNDAP Region 2 Single User Request Registry/Catalog Service Data Integrated from Multiple Providers Data Integration Service Agency 1 SOS Agency 2 IOOS Variables + other variables t.b.d. 21

22 Jun - Feb Verification & Validation Verification and Validation Feb Final Report May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb April 08 February 10 Service Function Valid HTTP response on success and error Server load test Service Content XML is well-formed (obeys generic XML rules) XML is valid (obeys schema rules) XML matches Data Record Definition Service Operations Service can receive data queries Service can send data queries Controlling Documents OGC SOS Specification OGC O&M Specification Abstract Data Content Standard XML schema XML dictionaries Feature Type Record Definitions Data Validation Data returned matches query parameters Data complies with Abstract Data Content Standard Data is easily combined/integrated o expressions of geographic position are consistent o expressions of temporal position are consistent o expressions of measured parameters are consistent

23 DIF Service Layers Target for FY2010 Customers NOAA DSTs COTS/ GIS General Public Other Other Services Access Services Registry/Catalog Integration Translation Web Map Service Sensor Observation Service Web Coverage Service OpenDAP Data Providers NOAA Regional Partners Other

24 NOAA IOOS DIF Accomplishments Developing the IOOS Data Integration Framework 7 IOOS variables integrated by end of 2008 Focus on major NOAA data providers for greatest impact Example: 83% of NOAA s ocean temperature integrated and accessible in initial DIF implementation DIF Integrated Products Team (IPT) developing effective, cross-noaa integration solution DIF implementation software and protocols will be made available to regional partners NDBC provides 39 Regional temperature observations in August 2008 Regions to begin implementation in FY09

25 Regional Implementation

26 Regional Potential (GCOOS, SECOORA, Federal, Industry)

27 NOAA IOOS Data Integrated Framework Very short schedule driven by externals NDBC and CO-OPS are breaking new ground Limited IOOS-specific reference material Simultaneous implementation, testing, revision Sensor metadata descriptions (SensorML) not ready Process needed for working with OGC, SSML, etc. to revise/harmonize/promote in FY09 Need for better documentation, tools, converters, clients, reference implementation, test suites Need more customers/users Future Challenges

28 Questions?

29 Back-up Slides

30 NOAA Coordination Integrated Products Team Chair, Charles Alexander NOAA IOOS Program National Weather Service Chris Taylor Steve Baig Walter Smith Avichal Mehra Hendrick Tolman Bill Burnett National Env. Satellite Data & Informtn. Service Ken Casey Julie Bosch Tess Brandon Ted Haberman Kent Hughes Jennifer Fry Ken McDonald Lewis McCulloch Phil Keegstra Paul Digiacomo National Marine Fisheries Service Roy Mendelssohn Jim Sargent Dave Foley National Ocean Service Katie Fisher Andrea Hardy Shelley Tomlinson John Dandy Jack Harlan John Ulmer Daniel Martin Rebecca Love Katie Fisher Whitley Saumweber Rich Patchen Josh Pederson Anne Ball (CSC/Ocean.US) Oceanic and Atmospheric Research Steve Hankin Gustavo Goni Derrick Snowden 11

31 Temperature Integration Build PLATFORM Data Distribution text Modeling & Analysis NOAA Ships AOML/ (OSMC?) CREIOS DIF ARGO Profiling Drifting Buoys VOS (xbt) Weather Buoys Tropical Moored Buoys NDBC DIF DIF Data Request Model C-MAN SWMP W PORTS NWLON N SWIM Satellites S E DIF CO-OPS NCDDC OSDPD COAST WATCH DIF % of NOAA Integrated 83% FY08 100% FY09 (w/ funding) Under investigation

32 National DMAC Development Path (based on NOAA, NASA DoD Decision Processes) Initial Concept Studies & Work of Others 26 High-Level Functional Requirements Define requirements Use Cases Concept of Operations Functional & Technical Requirements Analysis of Alternatives (AoA) Analyze options and determine optimal approach NOAA Data Management Projects NSF Ocean Observing Initiative Federal SOA projects Data Integration Framework (DIF) Regional Coastal Ocean Observing Systems Industry RFI Analysis of Existing Approaches Economic Analysis of Preferred Alternative Capability Development Strategy Implementation Decision (~late FY2010) 32

33 IOOS Data Integration Framework Web Map Service (WMS) for Images of Data Map is a georeferenced picture of data GetCapabilities operation results in a table of contents in standardized format GetMap operation results in image of data customized according to: Variables of interest User-specified bounding box User-specified time Image size File format PNG, GetTIFF, JPEG, GIF May contain layers of data

34 IOOS Data Integration Framework Web Coverage Service for Gridded Data Coverage is an array of gridded data values Simplified viewpoint for this discussion, can be more complex GetCapabilities operation results in a table of contents GetCoverage operation results in a data file containing header and array(s) of numbers customized for: Variable of interest User-specified bounding box User-specified time File format netcdf, HDF, floating-point TIFF DescribeCoverage operation results in metadata about a specific dataset

35 IOOS Data Integration Framework Sensor Observation Service for in-situ Data Sampling feature discrete locations of measurements Point, vertical, or horizontal profile, trajectory and time series or collections thereof GetCapabilities operation results in a table of contents GetObservation operation results in an XML data file containing observation values for desired: Variables of interest Bounding box Time or named geographic feature of interest or a single sensor DescribeSensor operation results in XML providing detailed information about a specific sensor

36 IOOS Data Integration Framework Sensor Observation Service Sensor Ocean Service (SOS) One of a family of standards and specifications that constitute the Open Geospatial Consortium (OGC), Inc. Sensor Web Enablement (SWE) activity An application programming interface (API) for providing sensor and observation data via extensible markup language (XML). Supports: GetCapabilities This function returns metadata about this service DescribeSensor The function returns detailed sensor characteristics GetObservation This function returns observation data of the specified type for the stations within the specified station and date/time range. A bounding box will be added in the future. Currently, can get: Most recent observation Observation at a specific time Observation for a time range

37 IOOS Data Integration Framework Sensor Observation Service GetObservation for ocean current vertical profile

38 IOOS Data Integration Framework Sensor Observation Service GetObservation for ocean current vertical profile

39 Integration Challenge multiple, unique, independent data requests PLATFORM Data Transfer/Extract/Load text Data Distribution ARGO NOAA Ships Drifting Buoys CREIOS ARGO Profiling C-MAN (1) Drifting Buoys C-MAN (2,,n) VOS (xbt) Weather Buoys NDBC Weather Buoys (1) Weather Buoys (2,,n) Tropical Moored Buoys C-MAN SWMP SWMP * PORTS CO-OPS NWLON NCDDC OSDPD Satellites COAST WATCH NWLON SEAWIFS 10 distinct data requests 10 algorithms to convert to model format 10 different load functions Data provider changes have ripple effects * * * HABS * * NERRS CDMO PORTS SWIM * AOML * * * Inefficient Costly processing Time consuming Stovepipe highlight

40 Integration Solution DIF content and transport standards/services Data Distribution PLATFORM Modeling & Analysis AUGUST text 2008 OCTOBER 2008 NOAA Ships AOML DIF CREIOS ARGO Profiling Drifting Buoys VOS (xbt) Weather Buoys Tropical Moored Buoys DIF content and transport standards/services NDBC DIF C-MAN SWMP PORTS DIF CO-OPS NWLON DIF SWIM NCDDC OSDPD Satellites COAST WATCH DIF DIF Data Request HABS

41 Documentation: Systems Engineering Approach NOAA IOOS Program Data Integration Framework (DIF) NOAA IOOS Program Data Integration Framework (DIF) Master Project Plan Version 1.0 September 12, 2007 Master Project Plan Version 1.0 9/12/07 Functional Requirements Document Version 1.0 September 17, 2007 Functional Requirements Document Version 1.0 9/17/07 NOAA IOOS Program Data Integration Framework (DIF) NOAA IOOS Program Data Integration Framework (DIF) DIF As-Is Baseline Systems Document Version 1.0 September 19, 2007 Concept of Operations Version 1.0 April 25, 2008 DIF As-Is Baseline Systems Document Concept of Operations Version 1.0 9/19/07 Version 1.0 4/25/08 16

42 Data Integration Framework (DIF) Months 0-12 Integration of 5 IOOS Core Variables Month 18 Month 24 Integrated Variable Ingest for Select Data Products Test & Evaluation Month 36 Benchmarked Product Improvements for Operational Use NOAA 5 Core Variables Temperature Salinity Sea Level Currents Color Hurricane Intensity Model Hurricane Intensity Model Enhanced decision support through: Systems Engineering: Coastal Inundation Model NOAA IOOS Data Integration Framework Temperature Salinity Sea Level Currents Product Enhancement Test & Evaluation Harmful Algal Bloom Model Data Access & Exchange Data Standards Systems Engineering & Standards Development Coastal Inundation Model Verification & Validation Integrated Ecosystem Assessment Harmful Algal Bloom Model Integrated Ecosystem Assessment Integrated information services for NOAA programs Identify observation gaps Validated enhanced data products NOAA s Data Integration Framework NOAA MISSION OBJECTIVES Color REGIONAL COASTAL NEEDS Future State: External sources of 5 Core Variables (consistent with NOAA standards) Regional-coastal data integration for Regional scale data and information products and services